Can I enter Thailand on a tourist visa and apply for a non-O retirement visa, or must I use my non-O 90 days visa for this?

You can get a re-entry permit at the immigration office where you live/stay or at the airport before you fly out

IF you can't get back into the country before your 90 day Non-O entry stamp would run out you can apply at most immigration offices for a 90 day non-O visa in country by meeting the requirements

If you're not going to make it back before that 90 day Non-O stamp runs out, once you get here free stamp GO to the immigration office where you live/stay, if they can issue in country Non-O visas get the hand out listing the requirements.