[en] Various nonlinear behaviors caused by nonlinear boundary conditions have been observed, and it is feared that in large vessels like FBRs, the instability phenomena such as self-exciting sloshing may occur in the free liquid surface of coolant. In this research, the nonlinear instability phenomena in free liquid surface were examined by the basic experiment and the analysis. As to the self-exciting oscillation 'jet flutter' of upward plane jet that collides against liquid surface, in order to know the mechanism of determining the frequency and supplying energy, the amplitude and phase relation of various variable quantities were investigated. The simplified model for calculating the displacement of jet was made, and compared with the experiment. The jet flutter phenomena are explained. The interaction of free liquid surface and turbulent flow, which is important for considering the nonlinearity in free liquid surface, was measured by LDV and visualization, and the turbulent flow phenomena in free liquid surface were investigated. In the experiment, turbulent flow energy was given to the free liquid surfaces of water and polymers, and the effect that the Toms effect exerted to interface turbulent flow was observed. The results of these studies are reported. (K.I.) studies are reported. (K.I.)