I have not worked with the various 8-QAM formats at 43 Gbaud, but I can say that you need to supply more details for a useful answer.

What type of fibre is used, and in particular what is the effective area and chromatic dispersion coefficient?

Is this for an upgrade to an existing system, or for a new design?  Is there a single uniform fibre span, or multiple optically amplified sections in series?  Is there any in-line dispersion compensation or terminal dispersion compensation, or are chromatic dispersion and polarisation mode dispersion corrected by analogue or digital signal processing in the transmitter or receiver?

What is your tolerance to noise and distortion of the received constellation?  If chromatic dispersion in the link is low, then non-linear distortion accumulates coherently.  This means that distortion is large but deterministic, and amenable to compensation by digital processing of the received signal.  Conversely with high in-line dispersion, mean square non-linear distortion is smaller (at an equivalent power level), but more noise-like and more difficult to compensate.

In a multi-span 43 GBaud system, non-linear noise, crosstalk and distortion is likely to accumulate with an increasing number of amplified sections as (N P2 / Aeff2)  where N is the number of amplified spans, P is the per-channel launch power and Aeff is the fibre effective area.  The scaling is sensitive to chromatic dispersion and dispersion compensation strategy, and in very low dispersion systems, non-linear distortion can scale as N2.

Are the sub-carriers co-polarised at  the transmitter?  Orthogonally polarised sub-carriers typically suffer less non-linear distortion due to cross-phase modulation.

Is this a single channel system or wavelength multiplexed?  In a WDM system, cross-phase modulation (XPM) and four wave mixing (FWM) can be significant, and may dominate over single channel distortion at narrow channel spacings, especially when fibre dispersion is low.  Stimulated Raman scattering (SRS) introduces amplitude crosstalk between more widely spaced channels, but is usually second order in typical submarine links.

Do you have access to simulation software such as VPI or Optisystem?  Alternatively, it is not difficult to build a split-step Fourier propagation simulation in Matlab or Octave.  This would allow you to investigate the sensitivity to launch power, fibre parameters and dispersion compensation strategy.

Hi Alan. I thank you so much by your complete response. 

Yes, you're right. I forgot to put more details about the WDM system. Actually, I work with VPI software but I did not yet have implemented the appropriated QAM Transmitter/Receiver module. The system that I am studying is a new submarine link (2200 km) to be deployed with around 22 EDFA repeaters (100 km per span).  I am only estimating  the final OSNR (some guesswork). As the OSNR dependents on the power per channel and, simultaneously, it is limited by the non-linear (NL) impairments, I really need additional information about the NL-power/channel threshold. The 100G-submarine link is going to be made of 2200 km of SM Standard Fiber, using 4.8 THz-bandwidth EDFA amplifiers (C-Band) and DSP at the receiver. Supposing an 100 GHz  channel occupancy (for 8-QAM modulation format), the link is going to be able to carry ~ 48 channels. 

Would you know where I can find information about the NL power threshold for 8-QAM modulation format?

Thanks a lot.

With 2200 km standard single mode fibre at 43 GBd, you are working in a high dispersion regime where the power envelope will change substantially within the 20 km non-linear interaction length of each amplified span.  A reasonable assumption is that non-linear distortion will accumulate incoherently.  To a first approximation it can be treated as an additional source of noise.

There is not a hard threshold where non-linear impairments begin.  Non-linear noise power increases as the cube of the channel power, P, so:

Noise to Signal ratio ~ ANL P2 + ALIN / P

This is a minimum when the non-linear noise power ratio is equal to 1/3 of the tolerable NSR in the absence of non-linear distortion.  Performance degrades rapidly at higher powers, so a conservative design will usually aim to operate at slightly lower powers.

The optimum power will depend on the raw bit error rate you can tolerate.  If you need a pre-FEC BER of 10-9 or 10-12,  the power will be much lower than if you allocate a proportion of the payload to forward error correction, and can operate at BER 10-3 or lower.

Do you have a distortion limit for the 16-QAM case?  If so, you may be able to estimate how much more power you can launch using 8-QAM. 

At the same symbol rate, the spectral content of the signals will be similar.  Neglecting interactions between different channels or different sub-carriers, we expect similar non-linear distortion for the same sub-carrier power.  However at the same power, constellation points for 8-QAM are approximately sqrt(2) = 1.4 times further apart than for 16-QAM, so we can tolerate more distortion.

For the same sub-carrier spacing (172 Gbit/s using 2 sub-carriers at 8-QAM, and 215 Gbit/s using 2 sub-carriers at 16-QAM), then we can tolerate twice the non-linear NSR power, and +1.5 dB higher launch power using 8-QAM.

The comparison is more difficult between 2-sub-carrier 8-QAM and a single 16-QAM carrier, each supporting the same raw bit rate of 172 Gbit/s in a 100 GHz channel.  Without inter-carrier crosstalk, we might launch +4.5 dB higher power into the 2x 8-QAM composite channel.  In practice, the two sub-carriers will interact via cross-phase modulation with each other and with neighbouring sub-carriers in adjacent channels.  For the single 16-QAM carrier, XPM will be less important, as adjacent channels are 100 GHz apart, and faster walk-off due to chromatic dispersion will reduce the impact.  FWM would be an issue in lower dispersion fibre (DSF or NZDSF), but should be second order here.

My guess is 2 +/- 2 dB higher power using 2x 8-QAM, compared with 1x 16-QAM, but this does not take account of potential eye-closure due to inter-symbol interference and linear crosstalk between sub-carriers.  With sufficiently low polarisation mode dispersion in the cable, you may see some advantage in arranging for adjacent sub-carriers to have orthogonal polarisation states.

Does this help?  You should get more reliable answers once you have your simulation up and running.

Is this a simulation exercise are are you developing a real product?  Is the cable design fixed?  Have you considered using one of the specialist large effective area fibres available, for example from Corning or Sumitomo.

https://www.corning.com/emea/en/products/communication-networks/products/fiber/vascade-fibers.html

http://global-sei.com/fttx/images_n/TR-16042SumitomoSumarineFibers-1.pdf

I have work with 4 and 16QAM all-optical OFDM system at symbol rate of 25GSymbol/sec and number of subcarriers of 29. Furthermore, I investigate the system by VPI software. I have found that the optimum power has been occurred at -3dBm and -1 dBm for 4 and 16QAM, respectively. The communication channel contents 10 spans, each span consist of 50 km standard SMF,  DCF and EDFA. You know that the OFDM system highly suffers form phase noise that induced by fiber nonlinearities because the channel spacing is equal to symbol frequency. However, Your system (WDM) is lower sensitive for phase noise than AO-OFDM system, therefore , I expect that the optimum power will be higher.

Hi Alan.

Thank you very much for your suggestions. They are going to be really useful for my project. Yes, the cable is a design fixed. I need to assume that the fiber is a SMF. I'll try to updated my simulation for incluiding the QAM modulation formats in order to get more reliable results. Thx.

Hello Jassim.

I thank you for your answer. The number for optimum power are close to those reported in the literature. I will take them into account.