That's a really good point for anyone not familiar. Every Catholic priest must carry on them their celebret, it's kind of our universal ID card. Mine looks like a credit card and has my picture and the contact information from my archbishop.
If you ever suspect someone is impersonating being a priest, ask to see their celebret. We have to produce them to be examined. If the guy looks confused or doesn't have it, there's more than a good chance he's being insincere.