Quantum phase transition between spin liquid and spin nematics in spin-1 Kitaev honeycomb model

Abstract

Besides the exactly solvable spin-1/2 Kitaev model, higher spin-S ones, not exactly solvable, are promising playgrounds for researches on the quantum spin liquid as well. One of the main interests in higher spin-S cases is the interplay between the Kitaev spin liquid (KSL) and spin nematics. We probe this interplay in a spin-1 model on the honeycomb lattice with competing bilinear-biquadratic and Kitaev interactions. Utilizing the 2D infinite projected entangled-pair state (iPEPS), we map out the phase diagram for the ferro-biquadratic interaction. In the phase diagram, we discover the direct phase transitions between the KSL phases and ferro-quadrupolar ordered spin nematic phase, which we call the FQ phase. It has been revealed that the ferro KSL exhibits robustness against perturbations from ferro-quadrupolar interactions. Also, the FQ phase is extended to the parameter region near the antiferro-Kitaev limit. Published by the American Physical Society 2024