If you're doing a self transfer, which per my understanding is collecting your luggage, going through immigration and then checking in to your next flight, then you're going to be entering Thailand.
As such, you might need a visa depending upon your nationality as well as a Thai Pass.
said you can only enter as part of a tour group. With regards to re-entering Thailand, you'd need to meet all of the entry requirements in place including getting a Thai Pass.