0.3gmsk, qpsk,8-psk were popular 2G and 3G modulation schemes then QAM with OFDM was introduced in LTE, now which modulation technique will be used in future mobile communication.
Well 5G is still an open issue. nothing yet is standardized. But in my opinion new modulation schemes with high spectral efficiencies would be explored e.g some hybrid schemes like F-QAM or FSK-QAM.
5G radio access will be built upon both new radio access technologies (RAT) and evolved
existing wireless technologies (LTE, HSPA, GSM and WiFi)......according to Huawei.....5G will be worldwide by 2020. As Aamir said who knows some new schemes may come with hybrid technologies.
For 5G, in context to modulation schemes the objectives include the following
New modulation schemes targeting at reducing the physical (PHY) overhead,
New modulation and coding schemes (MCSs) to increase the spectrum efficiency such as in small-cell deployment scenarios having more favorable signal-to noise ratio (SNR) conditions than typical macrocell deployments (like Aamir has already mentioned).
I do support Sandeep's view that it will be built on new Radio Access Technologies and Evolved Existing Wireless Technologies. For Further information
At present 4G LTE uses QAM with OFDM as modulation and OFDMA as access scheme. The challenge in 5G is high bit rate with mobility. Therefore, spectral efficiency is utmost important. The following indicate modulation schemes, which may find use in 5G implementations.
QAM with OFDM is a still contender.
Filter-bank multi-carrier (FBMC/OQAM) modulation. It exhibits better spectrum shape compared to the OFDM (Orthogonal Frequency-Division Multiplexing) and enables better spectrum usage and mobility support.
Single-carrier modulations (SCMs): This is still active and contender for 5G. The requirement is optimal equalization. The optimal equalization of a single carrier system is much more involved and essentially requires the use of a Viterbi algorithm.
Faster-than-Nyquist (FTN)/Time-frequency-packed (TFS) signaling: FTN multi-carrier is named as TFS. The TFS system has most of the parameters identical to FBMC. The TFS is implemented either with QAM or OQAM.
Wave modulation (WAM): WAM patented by MagnaComa is pure digital technique; a set of algorithms that implement a form of spectral compression. It fits into the signal processing path after the FEC, but before the digital front-end of the radio. WAM gives an overall 10% system gain advantage, up to 400% increase in range, a 50% spectrum savings, improved noise tolerance, and increase in data speed. One key benefit is the ability of the system to work with non-linear (such as class C) power amplifiers (PAs). It can be implemented in a DSP, but is more efficient in an FPGA or ASIC.
Cohere Technologies patented new modulation technology called Orthogonal Time Frequency and Space (OTFS), to increase the capacity.