I have a doubt in plotting an ROC , I have following parameters
1) Probablity of detection Theoritical (Pd_theory)
2) Probablity of false alarm assumed (Pfa)
3) Probablity of detection simulated (Pd_sim)
4) Probablity of false alarm simulated (Pf_sim)
And I want to plot Pd_sim with respect to false alarm. But I don't know which false alarm would be more correct to take. Pfa or Pf_sim and why?
Hi Imtiyaz
I recently answered a similar question elsewhere. Here it is again for you. I hope it helps ...
It sounds like you might be wanting to generate a "receiver" operating characteristic (ROC) curve. This has the probability of false alarm (P_F) along the x axis and the probability of detection (P_D) along the y axis - for a given system configuration. As the system is re-tuned or tweaked the curve will change. That is, for a given P_F, what is the P_D? Ideally, the designer would want the curve to go straight up from the lower left corner, to the upper left corner, then straight over to the upper right corner.
To generate this curve, using simulated data, you would do a large series of Monte-Carlo (MC) runs for a given set of algorithm tuning parameters. If you are designing a detector, you presumably have a detection threshold that you are applying to some kind of detection metric that you are generating. Save all of the results from your MC runs. When they are complete, generate the ROC curve by varying your detection threshold from the smallest value to the largest value in small increments. For each value of the threshold, count the number of false detections and the number of true detections, for all of your runs, convert these to probabilities and plot.
For a considered PfA, plot PD (sim) and Pd(Theo). For Theoretical Pd you have to first find the threshold for the required Pf, then use that threshold to find Pd. For simulation you have to find the threshold for the given (same) Pf using the distribution (or histogram) obtained from Monte Carlo realizations. Using that threshold you can find Pd from the distribution (or histogram) obtained.
@ Hugh @ Rajesh
I appreciate your answer but my question is specific i just want to plot Pd_sim with respect to false alarm. But I don't know which false alarm would be more correct to take. Pfa or Pf_sim and why?
Hi Imtiyaz,
May be due to your approach, you PF values in simulation are not exactly the same as in Theory. for example pf theory (0,0.1,0.2, 0.3 0.4 ....) and Pf, sim ( 0, 0.05, 0.15, 0.25, 0.35) , then the Pf values are not same. Then your question gives a sense. In such case plot simply Pd, sim vs Pf; sim. If you want to compare theory and simulation then you need to merge two separate Matlab figures. Although the axis points are now different, but the overall closeness between the theoretical ROC and simulated ROC should be needed for your simulation validation. you can find the difference in Pd for your desired Pf by reading (approximately) from the graph.
I hope it is now clear.
If you perform large number of Montecarlo simulations (i.e., 10 ^5 perhaps) you can use the Pf_simulations. If you have a closed form solution for the probability of false alarm or for the probability of detection than use the theory one. You can also verify the theory and the simulated ROC performance and they should be very close to each other
Hello. You should either plot theoretical PD and PFA, based on assumed statistical models, or plot simulated PD and simulated PFA obtained from e.g. a Monte Carlo simulation. To mix these up in the same graph doesn't make sense to me.
BTW a good way to do ROC curves without having to detect false alarms, especially at low PFA, is to use Forced Choice - a test developed in the 1960s. Some references are in my paper below.
Also, you don't have to plot PD versus PFA for varying SNR, you can also plot PD versus SNR for fixed PFA (which comes out naturally from the forced choice method). P.S. I am not selling this method, it's just another way to do things that doesn't seem to have a lot of publicity in the signal processing literature.
Conference Paper Bias analysis of the forced choice detection test for Raylei...
Let me make my answer more clear. Take Pf as a parameter you choose (your requirement). For example let PF=0.1. Now find threshold for this Pf from analytical expressions for H0. Then find Pd for this threshold using analytical expression.
For simulation, find the threshold for given PF (90th percentile value from the histogram of test statistic under H0 obtained from Monte Carlo). Count the outcomes above this threshold for the distribution (histogram) under H1, which gives your Pd (sim).
Now you have Pd(analytical) and Pd(Sim) for same Pf=0.1. Plot in the same graph and see whether your simulation and analysis match. This is needed for validation of your work.
If set of Pf in analysis and simulation were chosen to be different, then consider my previous answer, Make comparison from the whole trend of the graph. The graphs should definitely coincide if you take large (say more than 10000 trails in Monte Carlo).
most of the statistical packages like spss, sigma plot, sas etc has option to plot roc curve
i think you should plot Pd(sim) versus Pfa (sim), and Pd(theo) versu Pfa (theo) and then compare the curves.
Or, plot Pd(sim) versus SNR or threshold and Pd(theo) versus SNR or threshold in the same figure and compare the results, SNR or threshold must be the same for both simulation.
can you please give more information about your simulation? ( sensing method type, simulation way....
Thanks,
Fatima
Hello,
Pf (Probability of false alarm - simulation)values are taken in the x-axis and Pm(Probability of missed detection - simulation) values are taken in the y-axis. I think you found those values based on SNR threshold. Then plot it using matlab.
Thank you
@Salahdine
I had done many methods like energy detection, feautre detection, matched filter detection method. What you want to know about simulation?
Use the formulas to find the values of Pf and Pm. Afterwards you do the simulation.
Thank you
My Pfa and Pf_sim are not matching? What should be the probable error?
@ salahdine
About which detection technique simulation method you want to know ?
From the graphs, how we will identify values Probability of detection and probability of false alarm?
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Probability
- Probability Theory