Can my 17-year-old daughter stay in Thailand on my retirement visa extension while she waits for university?

You may be able to change the reason for your extension to being over 50 inside Thailand. Check with your local immigration office for detail.

Your daughter can be dependent on your extension as well until she will turn 20 yo. BUT she may need to get an initial visa outside Thailand once she cancels her Non-ED before she leaves. You need to talk with your local IO to see if she can apply for changing the reason inside Thailand or not first.

Your wife can be a trailing spouse to your extension. If she can do it without leaving Thailand depends on what visa/extension she is on now. You need to check with your local IO for detail about if she can do it inside Thailand.