The stability properties of a coastal current composed of a deep flow and of an intense counterflowing surface jet are investigated with a linear quasi-geostrophic model and with a nonlinear isopycnic shallow-water model (MICOM), both in three-layer configurations. The currents are modeled by strips of uniform potential vorticity anomaly (PVA), with opposite signs. The linear stability analysis of the current is performed with exponentially-growing modes, varying the ratio of layer thicknesses, the potential vorticity distribution (width and intensity of the PVA strips) and the perturbation wavelength. The effect of a sloping bottom is also investigated. The various nonlinear regimes are described and interpreted in terms of growth of the modal perturbations, of critical layer distributions and of interactions of PVA poles. The linear growth rates of the perturbations are essential for enough PVA of the currents to reach the critical layers before wave breaking can occur. The positions of the critical layers determine where cut-off of the PVA contours occurs. This position is shown to depend only on the lower layer thickness (the upper layer one being kept constant). The intermediate layer thickness determines only the growth rates of the perturbation. Finally, the long-time nonlinear evolutions are governed by the interaction of detached PVA poles. An oceanographic application is performed using a 4-layer configuration, representative of the Mediterranean water undercurrents flowing under the South Portugal coastal upwelling jet. The analysis of this configuration shows that even for a stable upper layer, instability-driven eddies in the lower layers can disturb the surface jet and generate large meanders and filaments, similar to the observations from south of the Iberian peninsula.