The Accademia Gallery stands on a site formerly occupied by the ancient Monastery of San Matteo and Convent of San Niccolò di Cafaggio whose grounds covered the entire area lying between Piazza Santissima Annunziata and Via Ricasoli.

This huge building has the form of a Latin-cross church.
The side wings and the Galleria dei Prigioni are located in what was formerly the medieval Hospital of San Matteo, while the Tribune was built between 1873 and 1882 to house the statue of David according to a design by the architect Emilio De Fabris.