[en] We study models of lepton masses and mixing based on broken modular invariance. We consider invariance under the finite modular group ${\mathrm{\Gamma }}_4\simeq S_4$ and focus on the minimal scenario where the expectation value of the modulus is the only source of symmetry breaking, such that no flavons need to be introduced. After constructing a basis for the lowest weight modular forms, we build two minimal models, one of which successfully accommodates charged lepton masses and neutrino oscillation data, while predicting the values of the Dirac and Majorana CPV phases.