Can I transition from an O-A visa to an O visa using a monthly income requirement and what happens if I leave Thailand without a reentry permit?

1 you would have to transfer in 65K baht a month each month every month for the previous 12 months before you apply for a yearly extension using monthly income. and that means you can't do it for the first year extension on an O visa you'd have to use banked money

2 - the whole purpose of bouncing out and back just when the OA visa is expiring is to get off it because you can't get an O visa inside the country when you're on an OA entry stamp or extension.

So yes, you either come back in on a free visa exempt entry OR you buy a tourist visa at a nearby thai consulate before you come back in. Then apply in country for the 90 day Non-O visa and after that the year extension.