If I understand correctly, you want to categorize answers to an (online) quiz into three domain levels using Machine Learning techniques? Unless you already have some metric in place, a supervised method will require example results with known domain levels (so you can compare the answers from known beginners to known intermediates, etc).

Once you have that kind of data, the problem resolves into a simple classifier problem, which depends somewhat on the type of questions and answers to solve. I would guess a high-dimensionality SVM or a SLP/MLP would be quite sufficient; in fact, a simple Naive Bayes classifier would probably do just fine already.

If you want to do unsupervised learning, the problem becomes somewhat trickier. You can use a K-means or SOM algorithm to find three groups of answers, but those do not necessarily correspond to domain level knowledge groups. I'm afraid I don't know how that could best be solved; interested to hear what you come up with!

As far as making this a supervised learning, i know i will end up with simple classifier problems that can be resolved with feed forward Neural network, however my challenge begins from here: once the classification is done, how to move/supply questions from a huge question base and carefully selecting the level of questions that the students has just been classified as. If i go for SOM like u said, it becomes a whole lot trickier, decided how to establish a mapping for each domain level. More clarification from more experts are invited please.

Then, your real problem is classification of new questions by difficulty. If you have some history of the "new questions" and the answers by a number of students, I think it is a conventional classification problem. By if the questions are brand new, no history, then text mining may be the solution (clustering or classification by keyowrds, areas, etc)

If you have a static knowledge base and information on the past success rate for each question, then the problem is not really difficult.

For example, I use the "tactics trainer" on chess.com - which gives you a chess problem and a fixed time to solve it (sometimes seconds, sometimes minutes) - and you have to find the correct move sequence. Every time you solve a problem correctly, your own rating goes up and the rating (difficulty) of the problem goes down. The amplitude of change depends on the absolute difference in your rating and the problem's rating. The process is, obviously - incremental. There is a default initialization value for all problems and all participants, so you could always simulate something like this for your question database.

If, on the other hand - you want to be able to estimate the difficulty of new questions that have not yet been asked, it's a slightly more difficult problem.

I would still suggest using the above described approach for those questions that have an answer history. Then you could try and - for each new question - find the most similar questions that do have previous history - and do some sort of regression to try and guess its difficulty. Alternatively, you could do text mining in order to cluster or classify questions - and assign the average difficulty of the cluster/category (= topic) to be the initial difficulty of the new question.