You can not "just leave". They probably would not even let you out at the border. You have to cancel the "Non B visa". That is funnily done with an "extension of stay" form, where you give as reason "visa termination/cancelation". Then you get a 7 days extension.
Yes, some nitpickers will point out "the vis already USED. You are on an extension of stay, based on an Non B visa" - no worries, the immigration office knows what you mean with "cancel work visa, please!"
Regarding the work permit, I do not know. It is basically automatically void, when the visa/extension is canceled. However you are (and your employer is) supposed to inform the administrations in less than 10 or so days. You have to check that.
I guess that depends on what you call ridiculous and how good you are in keeping your papers together. I consider it ridiculous, because I would not easily be able to "procure" the papers again.