I think it depends what you look for, really. Fallout 3, New Vegas and 4 definetly scratch different itches.
3 has a really great post apocalypse atmosphere, although the fact it has been 200 years is a bit of a plot hole you just kind of have to see over to accept it. It's really the only one of the 5 that I think manages to have a really good creepy factor, not neccesarily outright scary but really does keep up a great atmosphere. The main story, well, to not spoil it, gets a bit rubbish bit by bit by bit, and the original ending is notorious for how bad it is. Some of the side quests are pretty great though. DLC is a bit of a mixed bag, since each DLC is actually quite small compared to the standard of DLC you see nowadays. Point Lookout is pretty great, The Pitt is good, and the other 3 I can take or leave.
New Vegas does make a lot of improvements, guns now have crosshairs and combat feels less reliant on the usage of VATS. Roleplaying is a bigger aspect, with lots more skill checks in various quests, and quest designs are alot more itneresting, not to mention there is something like twice the number of quests Fallout 3 had. I do think the game has its flaws, especially when some fans out there really overhype how great the game is. The main quest isn't that fantastic in my opinion, and the way it is designed, I feel like the game forces you to side with the NCR for the first playthrough because of the number of side quests the NCR gives out which you would otherwise miss out on. DLCs are for the most part better as well, Old World Blues is great, Lonesome Road and Honest Hearts are good, but Dead Money is atrocious, the story of that DLC being the only saving grace.
I think the issue with Fallout 4 is, well, sadly, that Bethesda seems more interested in emulating companies like Rockstar and Ubisoft with their open world titles, than any interest in making a full RPG, and as such, 4 is by far the worst as an RPG. But 4 is easily the best as an FPS, the crafting system is really well implimented, and the shooting is genuinely great compared to being just passable in 3 and NV.
I think all 3 games are worth playing, as well as going back to give the originals a chance too.