Loss of superfluidity in the Bose-Einstein condensate in an optical lattice with cubic and quintic nonlinearity

In a one-dimensional shallow optical lattice, in the presence of both cubic and quintic nonlinearity, a superfluid density wave is identified in a Bose-Einstein condensate. Interestingly, it ceases to exist when only one of these interactions is operative. We predict the loss of superfluidity through a classical dynamical phase transition, where modulational instability leads to the loss of phase coherence. In a certain parameter domain, the competition between lattice potential and the interactions is shown to give rise to a stripe phase, where atoms are confined in finite domains. In a pure two-body case, apart from the known superfluid and insulating phases, a density wave insulating phase is found to exist, possessing two frequency modulations commensurate with the lattice potential.