[en] In this investigation, we examine theoretically modulational instability (MI) in a dual-core optical fiber with higher order dispersion as well as higher order nonlinear effects. For dual-core or multicore optical fiber the investigation of MI analysis extends with parameters like linear coupling coefficient and coupling coefficient dispersion using extended coupled nonlinear Schrödinger equation. We analyze the impact of these parameters on MI gain spectrum at anomalous and normal dispersion regime along with higher order parameters. Our results show that the combined effect of higher order dispersion like fourth order dispersion and nonlinear parameter like self-steepening effect due to cubic–quintic nonlinearity, stimulated Raman response which is inevitable for the study of ultrashort pulse propagation and generation of the soliton. The study reveals that the frequency selection of MI gain is primarily controlled by the dispersion parameters while nonlinear parameters restrict the intensity and bandwidth of the MI gain. Though most of the distinguishable effects are observed in anomalous dispersion regime due to the presence of higher order parameters decent gain is also visible in the normal dispersion regime. Graphical abstract: .