I calculated PSD of my function, but the PSD output has too many oscillations; therefore, it's hard to find -10dB threshold. So, my question: Is there a correct way to smooth the PSD or do I just need to take the envelope into account?
Depending on what your "noise" looks like, there may a more optimal way to filter it out, but an easy thing to do is as follows. Setup a loop over the length of your data. Calculate the value at each point as the average value of it and some number of surrounding points. Be sure to calculate the averages all from the same original data set. Don't mix in your newly calculated values and old values. The more points you include in your averages, the more smoothing you will get. So if your data is stored in any array F(n), you cancalculate a new set of values F2(n) using three points as
F2(n) = ( F(n-1) + F(n) + F(n+1) )/3
A more sophisticated thing to do is to calculate the FFT and plot it. You may find that your noise is concentrated over some span of frequencies. You can set these to zero, and recover your filtered data using in inverse-FFT.
There are also a bunch of digital filtering toolboxes available that can do even more sophisticated filtering. I suspect, however, that the simple averaging technique (which is a simple digital filter) will work for you.
The best way to filter your “noise” really depends on the properties of the noise. The simplest thing to do is setup a loop over your data. For each data point, you will calculate a new value by averaging it with some number of surrounding points. The more points you use in your averages, the smoother your second set of data will be. Be sure to use only the values from your original set of data when calculating your averages. Don’t mix new and old values. So for example, you will calculate a new set of data F2(n) from the old set of data F(n) according to:
F2(n) = ( F(n-1) + F(n) + F(n+1) )/3
A more sophisticated way is to calculate the FFT of your data. You may find that the noise is band limited and that you can zero out the noise in your FFT. After setting the noise components to zero, you can calculate the inverse-FFT to recover your filtered data.
There are also a lot of digital filtering toolboxes that you can use to filter your data in other ways. I suspect the simple averaging technique will work fine for you. Start with a window size of maybe 5 points.
Thank you for your comments. Instead of using FFT, I simply used moving average filter to smooth the data, as you said; however, I am afraid that some information in PSD may be lost due to this filter, but for now it seems working.
Using the moving average is roughly equivalent to low pass filtering your raw data. The resulting smoothing you obtain on your spectrum is due to filtering out other high frequencies out. Your spectrum do not reflect your raw data any more, I believe this is not your main intention.