According to the 2011 National Housing Census, masonry buildings comprise more than 50% of the Portuguese building stock. This type of construction has demonstrated a poor performance during the destructive events that struck Portugal in the last century (e.g., 1909 M6.3 Benavente, 1969 M7.8 Algarve, 1980 M6.9 Azores). While some efforts have been made towards the vulnerability assessment of individual masonry buildings or the residential building stock at the national scale, this building class has been the target of limited research. In this study, a comprehensive database of material and geometric properties of granite and limestone buildings was used to generate complex 3D numerical models. These models were subjected to nonlinear time history analysis to derive fragility curves considering the record-to-record and building-to-building variability. The structural damage was defined using two novel engineering demand parameters, and the resulting fragility functions were compared against other existing models. These results can be used directly to generate earthquake scenarios at the country scale or in probabilistic seismic risk analysis to identify regions where risk reduction measures should be prioritized.