actually my understanding is the opposite, but maybe because my impression was from the time before e-visas, where your passport was kept by the embassy for physical endorsement, and thus you couldn't travel anywhere that needed your passport as ID to travel.