I can't say I've had anyone try to physcially steal it, though I live in a pretty tame part of town.
As for downloading viruses or whatever, AFAIK there isn't any yet. I suppose someone could exploit the TIFF vulnerability and put something on your phone, but I don't know of any cases where that has happened yet (except for jailbreaking it).
I just read an article about smart phones being vulnerable to viruses. Anything from being able to monitor your calls and emails, to redirecting calls and charging outrageous fees.
Plus, being the Jesus Phone, I figured there may have been some thefts.