You need to look up dynamic gain tilt (DGT).

Basically as the gain of the EDFA increases there is a tilt in the spectral gain characteristic, if you keep the internal gain constant (potentially including a variable optical attenuator after the amplifier to avoid receiver saturation, or one before the amplifier to avoid gain saturation) the gain spectrum will be constant and you can flatten by filtering. The DGT is also affected by the erbium length, so some compensation can be achieved by changing this if you have access, provided it doesn't affect other parameters important to you.

That said with a 300Ghz span the gain variation should not be massive.

There are various EDFA parameters to be optimized in order to maximize gain and minimize noise figure. Some of EDFA's parameters are its doping core radius, Erbium ion concentration, Numerical Aperture, Pump wavelength, Pump power, signal power and selection of proper pumping configuration.  You have to overcome ASE to minimize Noise Figure. 

"I want to optimize the EDFA output for my system ... (EDFA used is Alnair labs EFA200C)"

It is difficult offer robust advice without more information, such as:

• What range of optical powers is expected at the receiver?
• What range of optical powers can your photodiode accept?
• Is there a single photodiode, or will you use a DMUX filter and 4 independent detectors?
• What is the insertion loss of the passive components prior to the photodiode?
• Which version of the EFA200C will you use?

Typically, the noise figure, gain and output power of a well-designed EDFA improve with increasing pump power, but this need not be the case, particularly if there are significant internal reflections or Rayleigh backscatter within the EDFA,

The EFA200C seems designed for moderate output powers up to 20 dBm, so you should check that it will meet your requirements as a pre-amplifier. In particular the data sheet refers to a firmware upgrade with "Safety Interlock (automatic output power-down when input falls below -10dBm)". -10 dBm is a relatively high input power for a receiver, so this could be a problem if you have less than -10dBm optical power at the end of your single span, unless this feature can be disabled.

For best noise performance, the EDFA should be located at the input to the receiver, and before passive components such as demultiplexer filters.

The amplifier probably has a reasonably flat gain spectrum with 0 dBm input and 20 dBm output.  For lower output powers expect significantly higher gain, but with increasing gain tilt as the gain deviates from 20 dB. As Mark Bray points out, this is probably not be a problem with signals spread over only 300 GHz (2.4 nm).

Can your photodiode accept optical powers greater than +10 dBm?  You can limit this by reducing amplifier pump power, or with additional passive loss (such as an optical attenuator) between amplifier output and photodiode.

If you have highly variable input powers, then amplifier saturation can be beneficial, and act as an automatic gain control, with more consistent optical powers reaching the photodiode - provided that some variation in gain tilt is acceptable.

http://www.alnair-labs.com/download/EFA-200C.pdf

Thank You Mark, Subhash, Zbigniew and Alan for your answers.

I try to answer some of your questions.

• I will you use a DMUX filter and 4 independent detectors.
• Insertion Loss around 7db of Dmux Filter
• Edfa is EFA-200C-20
• Receiver is a Sevensix Inc. 12.5 Gb/s Optical Receiver whose datasheet I couldn't find online (so unable to exactly know its parameters)    

I've just looked up a datasheet for a Tektronix 12.5 Gb/s receiver OI1125 • OI2125

This has an input power range between -16 and 0dBm. If this is representative it will allow you to consider your power budget.

... and whether you need the EDFA.