Coherent and incoherent diffractive hadron production in pA collisions and gluon saturation

We study coherent and incoherent diffractive hadron production in high-energy quarkonium-heavy nucleus collisions as a probe of the gluon saturation regime of quantum chromodynamics. Taking this process as a model for pA collisions, we argue that the coherent diffractive gluon production, in which the target nucleus stays intact, exhibits a remarkable sensitivity to the energy, rapidity, and atomic number dependence. The incoherent diffractive gluon production is less sensitive to the details of the low-x dynamics but can serve as a probe of fluctuations in the color glass condensate. As a quantitative measure of the nuclear effects on diffractive hadron production we introduce a new observable--the diffractive nuclear modification factor. We discuss possible signatures of gluon saturation in diffractive gluon production at the Relativistic Heavy Ion Collider, the Large Hadron Collider, and Electron Ion Collider.