Oh, I do, I do, but with a phone, it's like, just unplug it and go... I usually don't think about it. I have a SLVR which mounts it's card on the Mac and in iTunes, and you can just unplug and go with no trouble.
Trust me, Firewire drives ALWAYS get dismounted before unplugging.
Anyway, I've done it a few times now, and dragging the SD card/virtual disk icon to the trash before unplugging seems to prevent the dreaded pcsd hang.