Can You Open a Company with a Non-Imm O Retired Visa in Thailand?

There is no such thing as a "Non-O family" visa. There is an "Extension of stay based on being married to a Thai woman", and "Extension of stay based on having half Thai children" (both allow employment).

For visas themselves, you can get a "[Single or Multiple Entry] Non-O visa based on marriage to a Thai wife" or a "[Single or multiple entry] Non-O visa based on supporting a half-Thai child".