I am transmitting my RF signal through atmosphere at a range of 10 kms. But it seems range is limited by environmental noise floor. Is there any way through signal processing that can mitigate the effect of environmental noise floor.
What do you mean by "environmental noise floor"? Thermal noise from antenna (kTB noise)? Receiver noise? Interference from other transmitters in the area? There are methods both analog and digital that must be applied to improve signal to noise ratio. These include band and channel filtering, antenna gain, low-noise amplifiers, transmitter power, forward error correction & coding, etc.
Have you compiled a link budget? This is a first step in determining the system requirements.
You can reduce the noise as the colleagues said by reducing the noise of the rf amplifier to be a low noise amplifier. Also, you can limit the bandwidth of the reception by using rf band pass filter.
Also in the detection you can use matched filters or correlators which maximize the the signal to noise ratio.
The most effective technique to detect a signal impeded in the noise is by using spread spectrum transmission system in which the bits to be transmitted is chopped to a much higher rate chips. In this techniques the power is traded by increases bandwidth.