P4Golden was my first Persona game. Got it in October so I'm kinda late to the game(pun intended).I have since then become obsessed with the series and finished four of the five games(not counting Persona Q but I hear it's great!)...Anyway I played Persona 4 Golden for about 20 hours but I wasn't used to the social aspects and just went to the "TV world" every afternoon to grind levels. So I figured that I fucked up my save file and put the game in my backlog for a while.
I got Persona 3 FES on PSN as an impulse buy on the thanksgiving sale they where having there. And was glued to the TV all through December, the game actually helped me coping with the loss of my dog so that might cloud my judgement some. But I think the story was amazing, it's really emotionally dark at times(well the whole theme of the game is "memento mori"). On the other hand gameplay can be annoying since you can't control you companions in combat, just give vague orders.
I strongly recommend Persona 3 FES if you can get a hold of that version, used for PS2 or on PS3 via PSN. If not then there is a solid version for PSP that is available on PSN(works on PS Vita), though this one lacks the extra epilogue scenario that was added to FES. P3 and P4 share a lot gameplay-wise, so you will probably recognize a lot if you decide to try it.
The different versions of Persona 3 can be confusing, if you want I can do my best to explain them.
As for Persona 1 and the two Persona 2 games. They are very different. No social aspects and instead of shadows you battle demons and have to literally negotiate with them to give you Tarot Cards that let you summon demons to use as Personas in the velvet room with good old Igor. Persona 1 feels a bit like a run-of-the-mill 90's JRPG, but the story is interesting with parallel world stuff and Carl Jung psychology-concepts thrown in. And the two Persona 2 games has a bit of the same 90's JRPG feel and with a balls to the walls crazy story(just for a taste of that: a certain dictator comes back to life due to the power of rumors).
I've so far enjoyed all the games and do recommend them all. But beware that the first games are very different than 3 and 4. And I apologize for the wall of semi-coherent text, I got a bit carried away there...