Can I apply for a Non-B visa in Vietnam while waiting in Thailand?

If you go back to Thailand immediately after applying, your visa will not be activated upon return. It will only get activated when you cross a border again after your visa is approved. So indeed you could do a little "hack", apply for the visa, come back, and then bounce a border when it gets approved, to minimize your time out of the country (in exchange for doing two trips).

It is however not entirely free of risk; sometimes visas are refused if you leave the country from where you applied. They could notice if you entered Thailand. I do not know the embassy in Vietnam's standards on this.

The requirement is that you are in the country from the time you apply until you receive the visa. They don't know where you are, but there have been recent reports of embassies asking for proof of your location again a few days after the application was submitted. So far I've only seen that reported for the DTV visa, but it could potentially happen for any visa.