It's 250 years old, and is probably second only- a distant second, it's true- to the Prado, and the list of directors and fellows it has had over the years is very impressive. The building is also of some architectural interest.
You can read about all this by following the links, so I'm not going to describe it in detail, just give a bit of a taster and point out a few things which caught my eye, some unexpectedly.
On the 3rd floor you will find the modern stuff, much of it abstract, though representational and allegorical work is also there. You might find Muñoz Degraín's 'The Collossus of Rhodes' interesting. It's a extraordinary, vast fantasy on the celebration of a festival by the statue. And there is Eduardo Chicharros' 'The Temptations of Buda', a modern allegory showing the great man resisting the undoubted charms of a woman presented in many different ways, some involving animals, and very few involving clothes. You have to wonder why he resisted, but it made a better story that way.
The 2nd floor is largely dedicated to portraits of Academicians by other Academicians, and while some are of interest, they suffer from the usual problems of portraits; constrained by the whims of the sitter, the artist has little freedom of subject, medium, composition, colour, symbolism, technique or expression, and was probably only doing it as a favour in the first place.
The 1st floor, much the largest, has a number of fine Greco's, including a magnificent St Jerome, Zurbarán's, which include the exquisite 'Agnus Dei', a large room dedicated entirely to Alonso Cano and José de Rivera (if you don't know much about Spanish renaissance painting check those two out). Rivera has a scene of the taking of Christ, a group of figures, little more than faces, lit only by a hidden candle or torch which makes the face of Christ a reddish-orange focus of the scene, the intense emotion brought out by the intimacy of the lighting. Worth a visit by itself.
There are some Goya's, including a series of 'Children at Play', which is as good as the holiday scenes that hang in the Prado, and a couple of Rubens', perhaps not among his finest. Some Flemish painters who came to Spain to live and work, and some lesser known Spanish painters. Pedro Orrente, Andrés de la Calleja, Genaro Pérez Villaamil, Joaquín Sorolla, and many others, there to be discovered.
If you happen to be in Madrid, once you've got tired of the Prado, go and have a look at the Academy of St. Fernando. It's worth a morning.