Given students in large courses can game the system, as Gary pointed out, it might be worth turning this to your advantage. The 'PeerWise' tool from University of Auckland provides an environment for students to author the questions and answers, comment on each other's questions and rank each other's questions. In this way the best questions rise to the top. It is a great way to get students thinking deeply about the course. It plays on the idea that the best way to learn something to teach it to someone else.

According to my opinion, this is more philosophical than technical problem. But to (at least partially) answer your question, I can write my experience from technical point of view. We use LMS Moodle four testing at our faculty. We use randomly selected question in random order, each question has also randomly shuffled aswers. Each test can have its time limit. You can find more technical information in attached link.

https://docs.moodle.org/23/en/Quiz_settings

Thanks for you comments Václav

Although I use Blackboard rather Moodle I am well aware of using question pools and random re-orderings to limit crowd sourcing by students. But I write all my own questions and do not use publishers questions, I am aware of other educators who do the same in my discipline area what we do not have is a way of sharing our questions to crowd source even bigger pools in a secure environment.

A number of years ago myself and a number of colleagues generated questions on a weekly basis relating stories in the previous weeks news to academic theories. This was for for an introductory business course with about 700 students. The news quiz only ran for one year as the students gamed the quizzes (crowd sourcing the answers) much more effectively than we managed to crowd source the questions. So, rather than reading the news feeds for the FT and Economist as we had asked them to do, they compiled large word documents of questions the the early attempts generated and used find to jump to questions including certain key words.  This means random question order and random shuffled answers have no effect only the number of completely unique questions and hence the probability of someone getting a question  that they had already collected.

In our extensive reading program, there are question banks for over 3000 graded readers for EFL learners--crowd sourced from hundreds of teachers around the world.  With the MoodleReader plugin to Moodle LMS, teachers contribute questions and these questions are vetted and stored in a collective question bank for free download by institutions. Each book quiz has 40 questions contributed to the bank and 10 questions are randomly selected by the program when a student takes a quiz. More information is available at http://moodlereader.org.

Given students in large courses can game the system, as Gary pointed out, it might be worth turning this to your advantage. The 'PeerWise' tool from University of Auckland provides an environment for students to author the questions and answers, comment on each other's questions and rank each other's questions. In this way the best questions rise to the top. It is a great way to get students thinking deeply about the course. It plays on the idea that the best way to learn something to teach it to someone else.

Gary, the logistics of collecting questions from multiple courses is easy enough in most LMS, so I guess the tough part of your question is about evaluating questions.

Student evaluation. PeerWise delegates both the creation and evaluation of questions to the students. There is little reason to keep questions secret, because there are so many.

Empirical evaluation: Socrative (like most LMS) automatically shows the distribution of answers on quiz questions. Questions answered correctly by 100% or 0% of students are useless. There is little reason to keep questions secret, because they seem ephemeral.

Decontextualised evaluation: Evaluation of a question needs to be qualified by its context. For example, "The last word of the national anthem" has a different meaning for you and me in 2014 (though it did not in 1970). This is not a problem for publisher's question banks, which sit in the context of a particular textbook designed for a cohort of similar students.

Concealed solutions: Moodle has a forum in which a student can see (and rate and reply to) her peers' answer only after she has posted her own. Of course, students can subvert this by talking to each other, so questions should be designed to be educational when discussed.

That's cool--students creating and evaluating their own questions. I went to the Peerwise site but I couldn't understand how to get the questions into my LMS.  Are we locked into using one giant Auckland site?  That's no go.

Don, good point about the lock-in at Peerwise. The inability to, at least, export student contributions and results has really deterred uptake.

I like the look of Peerwise, but as well as the lock-in, it also appears to be limited to multiple choice questions. I have lots of questions that are numerical or fill in the blank and I think I would like my students to have a wider selection of question types.

Even for multiple choice, it appears to be a single explanation system rather than giving context sensitive help depending on which wrong answer is chosen.

If you are still interested look at this paper, you may find something similar "STEP: A Scalable Testing and Evaluation Platform" it basically a platform that uses the crowd and generates a continuous test questions. It depends on Q&As site StackOverFlow to re-design the questions already answered into a context useful in online testing environments in this case for Java programming and of course you can twist it for any other topic giving the fact that there are many online resources similar (e.g. Q&As) .The system depends on the crowd to edit questions and it also require them to asses the questions and test them for quality purposes. The system always looks for similarity of it's generated questions into Search Egnines and if found it considers them leaked and mark them off the system. Hope that is useful.