Should I enroll in school before applying for an education visa or extend my stay first?

Have you asked your school? Some can convert inside Thailand, and some cannot. For those that can, it can take more than a month between them having to apply to the Ministry of Education for approval, and then you must have between 14-21 days left on your existing stamp at the time that you apply for the ED Visa at immigration. If you don't have plenty of time left available to you, you might have to leave regardless.