In the last decades, most efforts to catalog and characterize the built environment for multi-hazard risk assessment have focused on the exploration of census data, cadastral data sets, and local surveys. Typically, these sources of information are not updated regularly and lack sufficient information to characterize the seismic vulnerability of the building stock. Some recent efforts have demonstrated how machine learning algorithms can be used to automatically recognize specific architectural and structural features of buildings. However, such methods require large sets of labeled images to train, verify, and test the algorithms. This article presents a database of 5276 building images from a parish in Lisbon (Alvalade), whose buildings have been classified according to a uniform taxonomy. This database can be used for the testing and calibration of machine learning algorithms, as well as for the direct assessment of earthquake risk in Alvalade. The data are accessible through an open Github repository (DOI: 10.5281/zenodo.7625940).