depends how much you are changing. The difference adds up on bigger sums and I always carry £50 and $100 bills anyway.
And no super rich aren't the only ones. The ones with the best rates always offer more for the biggest notes. If they don't then their rates are rubbish.