Can I apply for a Non-B visa while my ED visa is still active and unpaid fees are outstanding?

Pay the school

Pay them.

That depends on the embassy. It's not likely that your stamp is still active if you left without a re-entry permit. But if the embassy sees that you didn't cancel your extension, then they might not issue a new visa to you. If you only had the initial 90 day non-ED visa and then left without ever getting an extension, you didn't have anything to cancel as it's only the extension that you can cancel.

But if you were on an extension, you have a bigger problem. Even if you can get the non-B visa, it's very possible immigration will not let you apply for an extension until you cancel your previous education extension to close out the loop. And they'll also fine you 500 baht per day since you stopped studying, to a maximum of 20,000 baht.