Hey!
I think that for a lot of us, it's sort of just instinct. You know a troll when you see one!
I think that many of us have kind of been raised on the internet, so it is just something that you naturally pick up from doing a lot of stuff online! Forums, chat rooms, reddit, tumblr, etc. all have trolls and when you've been hanging out online like that for most of your life, some stuff just becomes intuition. (Though seriously some trolls are really good and very tricky, so experience and intuition will totally not always help you out.)
Here are some things that might help give you some more insight!
http://knowyourmeme.com/memes/subcultures/trolling
http://thebloggess.com/2012/06/how-to-t ... e-a-troll/
http://www.linkedin.com/today/post/arti ... ith-trolls
KnowYourMeme.com is always great for figuring out what a meme means and learning more about internet subcultures and stuff! It's an awesome resource! Sometimes I check it out just to see how some of my favorite memes started.
You can usually tell a troll if they are trying to get a rise out of you and turn everything you say against you. Trolls will often trick you into arguing with them or making you feel silly in some way.
The best advice ever regarding Trolls:
"DO NOT FEED THE TROLLS!!"
This means, completely ignore them. Don't respond to them. Give them nothing to work with. You can either ban them or let them get bored and they might leave on their own. (But seriously you have to not respond to them AT ALL. Even one word replies or acknowledgement in the slightest is enough to feed them.)
If you yourself have some trolling tendencies (I do, but only for the purposes of good), then you can troll them back and have fun with them! This is an advanced technique though, so don't try this unless you're REALLY comfortable because it could backfire!