Topological phase transition in anisotropic square-octagon lattice with spin–orbit coupling and exchange field

Highlights: • Each topological phase is indexed by the Chern number and the spin Chern number. • The interplay of spin–orbit coupling and exchange field yields a number of topologically distinct phases. • Topological phase transition can occur when tuning the exchange field. • In a quantum spin Hall system, the edge states can be gapless in the absence of time-reversal and inversion symmetries. - Abstract: We investigate the topological phase transitions in an anisotropic square-octagon lattice in the presence of spin–orbit coupling and exchange field. On the basis of the Chern number and spin Chern number, we find a number of topologically distinct phases with tuning the exchange field, including time-reversal-symmetry-broken quantum spin Hall phases, quantum anomalous Hall phases and a topologically trivial phase. Particularly, we observe a coexistent state of both the quantum spin Hall effect and quantum anomalous Hall effect. Besides, by adjusting the exchange filed, we find the phase transition from time-reversal-symmetry-broken quantum spin Hall phase to spin-imbalanced and spin-polarized quantum anomalous Hall phases, providing an opportunity for quantum spin manipulation. The bulk band gap closes when topological phase transitions occur between different topological phases. Furthermore, the energy and spin spectra of the edge states corresponding to different topological phases are consistent with the topological characterization based on the Chern and spin Chern numbers.