Depending on your arrival time you could make it to the port in under 3 hours and it's a very short boat ride to the island..
Or if you don't mind a quiet village type atmosphere then Bang Saray is very nice and good beach and only around 2 hours or so from Bangkok.. (about 30 minutes past Pattaya) but nothing at all like Pattaya..
It's got a few hotels and accommodation options and some restaurants etc
I think they even have a
****
there (of course) but very much a fishing village atmosphere.