Spectral problems in open quantum chaos

We present an overview of mathematical results and methods relevant for the spectral study of semiclassical Schrödinger (or wave) operators of scattering systems, in cases where the corresponding classical dynamics is chaotic; more precisely, we assume that in some energy range, the classical Hamiltonian flow admits a fractal set of trapped trajectories, which hosts chaotic (hyperbolic) dynamics. The aim is then to connect the information on this trapped set with the distribution of quantum resonances in the semiclassical limit. Our study encompasses several models sharing these dynamical characteristics: free motion outside a union of convex hard obstacles, scattering by certain families of compactly supported potentials, geometric scattering on manifolds with (constant or variable) negative curvature. We also consider the toy model of open quantum maps, and sketch the construction of quantum monodromy operators associated with a Poincaré section for a scattering flow. The semiclassical density of long-living resonances exhibits a fractal Weyl law, related to the fact that the corresponding metastable states are 'supported' by the fractal trapped set (and its outgoing tail). We also describe a classical condition for the presence of a gap in the resonance spectrum, equivalently a uniform lower bound on the quantum decay rates, and present a proof of this gap in a rather general situation, using quantum monodromy operators. (invited article)