[en] In Emission Tomography images are reconstructed by solving an inverse problem: the three-dimensional activity map producing the signal observed by the gamma camera or the Positron Emission Tomography (PET) detector is estimated given a model of the imaging system response. This model gives the set of probabilities R_i_j that a γ or β+ emission occurring at point j in the volume of interest be detected in detection element i. Thorough modeling of this R system matrix (SM) is essential for ensuring the most accurate possible estimate of the activity distribution within the object of interest. Thirty years from now, it was proposed to calculate the system matrix R based on Monte Carlo simulations, as opposed to using analytical geometrical models, for increased accuracy. A lot of progress has been made since the initial idea and using simulations for enhanced SPECT and PET image reconstruction has become a reality. In this paper, we review the rationale for this approach, explain the advantages and limitations, the performance that can be achieved, and the challenges that remain to be solved. - Highlights: • ET images are obtained using an image reconstruction involving a system matrix (SM). • Analytical and Monte Carlo simulations can be used to calculate this SM. • Accurate modeling of the detector response for SM calculation is essential. • The accuracy of SM directly impacts the reliability of reconstructed images. • Comprehensive modeling of all detector response components raises large challenges.