Some metrics used to measure the efficiency of the system:

1.     Mean Square Error.

2.     Peak Signal–to–Noise-Ratio.

3.     Bit Error Rate (BER).

4.     Normalized Correlation (NC).

5.     RGB Histogram.

thank you Dr. Qasim for your reply

I had used some of this metrics such

MSE and PSNR between the cover and the stego image.

Bit Error Ratio between the secret message and the extracted message.

but I need to evaluate my proposed system against the steganalysis techniques

You may want to look at Rich Steganalysis features proposed by Jessica Fridrich. These features are highly robust.

Another popular method is based on Linear Binary Pattern (LBP) features. 

 You can use the ensemble classifier to test the robustness of you steganography method

It seems that many of the responders use `steganography' to refer to what I would call robust watermarking.  I will consider steganography where the attacker's objective is to detect the presence of a secret communication.  Ingemar Cox has discussed the distinction in illuminating detail.

You can only study or describe the robustness relative to a certain context, i.e. assuming a specific application scenario and a specific attack scenario.

Aspects of the context includes

1.  Choice of cover data.

2.  Size of the cover data and size of the payload.  Note that the problem does not scale linearily, as you will see from Andrew Ker's work on the square root law of steganography.

3.  Steganalysis algorithm used by the attacker.  

4.  Frequency of steganogrammes in the medium where you distribute your steganogrammes, as this effects the likelihood that a given message is a steganogramme both prior to and after steganalysis.

Virtually all work on steganalysis and steganographic robostness has focused on a completely artificial context (lab conditions) with little regard for real application. Typically that means a given pool of covers, a small set of attacks, a given relative payload, and assuming that only a single steganogramme is ever intercepted. Andrew Ker's work on batch steganography and pooled steganalysis is a notable exception.  I also discuss the challenge of addressing real life problems based on lab experiments in my book on Machine Learning in Image Steganalsysis, but I do not have a solution.