There is a paper that perfectly answers what you asked, just check:

T. Fawcett, An introduction to ROC analysis, http://people.inf.elte.hu/kiss/13dwhdm/roc.pdf

If you use the R tool, you can install the rminer package: 

https://cran.r-project.org/package=rminer 

The package includes functions mmetric and mgraph that implement the algorithm of Fawcett and allows to build and plot ROC curves.

No,

Usually the classifier is a function output=f(x), if the output of the classifier is higher than a threshold the x is classified as postive, if is lower then is classified negative.

What you should do is taking a look at your output values (before taking the decision among positive and negative samples) and evaluate the classifier using different values for the threshold.

In python you have functinons to use the output and the true labels as parameters and plot the roc or evaluate the area under the roc curve (auc):

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.auc.html

.

you have two different things here :

- a ROC curve depends on one given classifier only (say a 5-NN classifier) and shows the dependence of your performance (TPR, FPR) on the theshold value

(you may wonder why there is a threshold when dealing with k-NN classifiers : there is a threshold, even in the simplest case as you could classifiy an instance in class "0" if it has at least n neighbours of class "0" in its k nearest neighbours and you can vary n between 0 and k ; so even in this simple k-NN classifier there is a notion of threshold)

- different classifiers (say a 5-NN classifier and a 10-NN classifier) will have different ROC curves and these ROC curves allow you to compare the performances of your classiifers

.

a ROC curve considers all threshold values ; when you want to set the threshold to its "optimal" value, you need more information such as a cost matrix (cost of true positive, true negative, false positive ans false negative) : combining the ROC curve and the costs will give you a cost vs threshold curve from which you can choose an optimal threshold

.

The ROC curve is the plot of TPR against FPR for different threshold values. Generally in classification the classifier outputs a score which is compared with a threshold value such that if

score < threshold class = -1

or

score > threshold class = 1

Therefore to plot ROC curve the vary threshold value in steps (may say 0.1 or 0,2 depends on your values) such that min(scores)

I think that this is getting confusing in respect to your question.

First, your approach is somewhat ok. What you are doing is called optimization of algorithm parameters, but you should limit your testing instances for optimization to e.g. 10% of all instances (there is no point to test it against all instances and then say optimal parameters for this model WERE this and that, it is something you need beforehand). This is especially handy if you have a large set...

In optimization you can compare the performance of different setting just by comparing values of TPR, FPR or AUC. Extensive programming should not be involved. In WEKA for example, you have Experimenter mode, which is used for optimizing algorithm parameters (you select your parameters and your algorithms, and the metrics you desire, e.g TPR and FPR), after that you can get your results per each combination of the parameters (in kNN case just one parameter) as table or make a graphic out of it (but not ROC curve)...

If you really want to plot the curve, here are some tips.

- If your classifier does not involve probability of a hit (discrete classifier), then your curve will not be exactly "curvy". In ROC space your ROC result will be one single point in binary classification case. kNN is an example of discrete classifier.

- If your classifier is probabilistic then a point in ROC space is assigned per each threshold value of probability, e.g. 0.1, 0.2, ... , 1.0.

Please read Fawcett 2006 for further clarifications.