I would recommend using the panchromatic band (i.e. one single channel that covers almost the entire visible spectrum) in the process of pansharpening solely for image interpretation and vizualization purposes. As PS will modify the radiometric information contained in the bands, you should not use these in quantitative spectral analyses (e.g. calculation of a vegetation index) thereafter.

How much the spectral information is impacted by pan sharpening is dependent on the method used; take look at the three papers below for examples of what I have used with various digital image data sets.  Keep in mind, that if you are pan sharpening bands who's spectral window is outside the pan's spectral window then there is the possibility of 'edge reversals' that will impact the results (i.e., a boundary / line may be dark to light in the visible bands which includes the pan band BUT light to dark in the nir or swir bands so will be affected by the pan sharpening).  Pay particular attention to the first paper / link:

Comparison of Three Different Methods to Merge Multiresolution and Multispectral Data: Landsat TM and SPOT Panchromatic (1991)

https://www.researchgate.net/publication/200458993_Comparison_of_Three_Different_Methods_to_Merge_Multiresolution_and_Multispectral_Data_Landsat_TM_and_SPOT_Panchromatic

Digital merging of Landsat TM and digitized NHAP data for 1: 24,000-scale image mapping((National High Altitude Program))  (1986)

https://www.researchgate.net/publication/240321338_Digital_merging_of_Landsat_TM_and_digitized_NHAP_data_for_1_24000-scale_image_mappingNational_High_Altitude_Program

An automatic optimum kernel-size selection technique for edge enhancement (1982)

https://www.researchgate.net/publication/223377841_An_automatic_optimum_kernel-size_selection_technique_for_edge_enhancement

Good luck with your work.

Pat

Article Comparison of Three Different Methods to Merge Multiresoluti...

Article Digital merging of Landsat TM and digitized NHAP data for 1:...

Article An automatic optimum kernel-size selection technique for edg...

Pan sharpened images are good for visual analysis, but if you intent to do some other calculations which involves spectral characteristics of that band then use bands as they are. Pixel values tend to vary in pan-sharpened image as compared to actual values

For qualitatve assessment yes. But for quantitative geoinformation extraction, NO.

@Julian Ijumulana Ah! hard comment indeed! Could you please cite some reference please whereas I can be on the right track at least at the literature review.

Hello Sharif, what do you want to do with end product of image classification you are doing? Just a simple understanding on the pixel size and shape: You want see an object on a image unless the dimensions of that object are equal or greater than the sensor spatial resolution. Usually, there is a trade-off between spectral and spatial resolution. Fine spectral resolution image is limited in space. If a given object cannot be detected on image A, even if you pan sharpen it, you won't get accurate dimensions  from pansharpened image. Take a look at this link may be you can understand what am talking about: http://www.tandfonline.com/doi/abs/10.1080/014311698215748