Unless you and your brother are both willing to buy an Elite visa there isn’t really any visa that suits, at least not easily. An Education Visa might be a possibility if either are willing to register for an appropriate course, but they are looked at with suspicion by Immigration due to frequent abuse. Another option is to try and get a job with a company that will sponsor you. Otherwise, you’ll need to get Tourist Visas and leave the country periodically hoping g that immigration don’t flag you.
It would be much easier for your mother to get a Non O and she can then stay permanently. This can be got in your own country if you wait until she’s 50 or can also be got inside Thailand if she enters on a Tourist Visa and is 50 when she tries to convert it. She will need
*****
0 baht in a Thai bank account to do it herself, unless she’s willing to use an agent to help. Your brother is 18 so can’t go on her visa.