I took this screenshot this morning - every single time someone comes on here and says it's ok to overstay even for a short time, just pay the fine on the way out, Im going to post it.
what you've heard is wrong. In and out of Chiang Mai immigration for over 5 years - tshirt, shorts and flipflops and Ive never once had a problem. There are some immigration offices that are apparently anal about dress - Chiang Mai isnt one of then
In terms of the visa, it is is canceled from the date advised by the school in the letter - source, I did it one hour ago. In my case, it cancels May 14, so I have to leave on or by May 14 or I'm on overstay.
The rest - as to is it required or not - my new employer said no, my school (also your school) says yes. My take - if in doubt, get it done anyway. With the letter you'll need your usual passport copies and it's counter 8 at Promenada.