It is strongly depends of your purpose. For virtual screening validation in most cases, you may use DUD test set  (a directory of useful decoys http://dud.docking.org/ ). If you want to develop specific  decoy set for chosen target you may use this article (http://www.ncbi.nlm.nih.gov/pubmed/22539671)

your decoy set should contain compounds with oveall the same property profile for some basic properties like molecular weight, logD, donors, acceptors, halogen atoms ring types. If I say profile, I mean not mean values but value distribution. It should consist of about the same types of functional groups, rings etc, and the difference to your hit set should be connectivity of those features.

The size of your decoy set will also play a role, but that depends a litlle bit on the compute power for your evaluations.

Finally, there are different metrics which are more trimmed to find singletons or clusters or to find as early as possible as many as possible representatives of each cluster. There are papers and reviews from, if I remeber right, Mark Mackay, Robert D. Clark, Nicholls, etc.

Is it compulsory to have decoy set?

To answer your second question: it is HIGHLY recommended.

If there's one universal rule in modeling, it is "Crap in = crap out".

You can always run your protocols in a virtual screening exercise without first running a validation against a set of decoys. Of course, it will give you output files, numbers to put in a table, and rankings... But how can you then trust your results, since you haven't validated your protocols?

Running your virtual screening without prior validation against a decoy set is actually exactly the same as running a biological assay without proper validation with blanks, positive controls and negative controls...  Such results would not be trusted either.

It might be useful to run your virtual screening protocol on a diverse selection of your screening set to assess its sensitivity. For example, if the protocol assigns10% of compounds as potential hits (that is, 100,000 hits for a screening collection of 1 million), it definitely needs to be improved.