[en] In order to detect high frequency gravitational waves, we need a new detection method. In this paper, we develop a formalism for a gravitational wave detector using magnons in a cavity. Using Fermi normal coordinates and taking the non-relativistic limit, we obtain a Hamiltonian for magnons in gravitational wave backgrounds. Given the Hamiltonian, we show how to use the magnons for detecting high frequency gravitational waves. Furthermore, as a demonstration of the magnon gravitational wave detector, we give upper limits on GHz gravitational waves by utilizing known results of magnon experiments for an axion dark matter search.