Can I change a Non-O visa to a Non-B visa by leaving Thailand and re-entering on a tourist visa?

You are not changing anything. Once you leave Thailand without a re-entry permit, your current stamp will be no longer valid.

When you enter on tourist status, you will apply for a changing type of visa and it will be a brand new process.

Now, there are two different factors regarding your case whether you can do as you posted or not.

1. Your employer: Whether they would provide the documents for your initial in-country Non-B visa.

This depends on your new employer if they can meet the requirements for you to apply in-country visa. Not all employers can or would provide the documents enough for your in-country Non-B. You need to ask your employer for details.

2. Your current stamp: Whether you need to apply for cancellation of your current stamp.

This depends on the reason for your current non-O stamp and if you are on an extension or on the first 90 days or not. If you are on an extension, you may need to officially cancel your current extension depending on your reason so that it wouldn't be an issue when you apply for an in-country visa/extension based on employment.

Your question cannot be answered so easily. You need to talk with your employer and your local immigration office.

It'd be much easier to apply for the non-B visa at a Thai embassy or consulate when you leave. Have you confirmed with your job and their immigration office that they allow and can meet the requirements to convert to a non-B visa inside the country? It requires a large amount of paperwork that some companies cannot or will not go through the process of obtaining and providing. Most employers send new employees to a nearby country to apply for the non-B there.

A non-o based on what ?