We use Mooshak in several courses and it works pretty well.

For Java, Python, C++, and a few other languages, we use Web-CAT (http://web-cat.org/), which is one of the most widely used open-source tools specifically for program grading. For C, assembly, and other low-level programming, it doesn't offer plug-ins so you can't use it there without doing some work.

Basically, I use Eclipse in my Java Programming lab. I use JUnit to evaluate student codes (in essence, I write a testcase and I verify if the testcase ends with success), the Checkstyle plugin to verify the code style (compliance to Sun Code Conventions). Finally, I use the Eclipse views to verify the compliance of code to some specifications required in the assignments.

There has been some research done that investigates the approach that Corrado mentions in Australia by Cardell-Oliver see http://www.csse.uwa.edu.au/~rachel/ and Java programming. You might find the papers interesting given your question.

@Corrado Mencar, I agree that the tools you mention can be very helpful. That's why Web-CAT (a) allows submissions directly from within Eclipse using a simple plug-in (also from a few other IDEs, or directly via a web browser), (b) allows instructors to use JUnit tests to check student solutions, (c) allows instructors to use Checkstyle and PMD to perform static analysis checks on student solutions, (d) allows students to write and submit their own JUnit tests (instructors can even require it), and (e) uses Clover to measure code coverage from student-written tests and give students feedback about which parts of their code are untested (good when instructors require students to test their own code). This is all for Java, of course--the open-source analysis ecosystem isn't equally mature for all programming languages.

@Stephen seems to be a very interesting tool. Thanks.

I've used Web-CAT to examine Java programming assignments and it is excellent, especially the integration with Eclipse and BlueJ. From recollection I think it requires a relatively old version of Tomcat to get it up and running. If you want to get up and running quickly I have a private AMI for Amazon EC2 that I would be happy to make public if there was any interest in it.

Thanks for the endorsement, Chris. On our main server, Web-CAT is running on the latest yum-available version of Tomcat (on CentOS), which I believe is 6.x.

Very happy to Stephen! The whole Tomcat thing could well have been me hitting a limit in my Linux skills when I was trying it out a couple of years ago on Ubuntu server :)

Just learned about an automatic grading tool for programming assignments coming from MIT and Microsoft Research:

http://bit.ly/18OVoQK

The system grew out of work on program synthesis. The authors (Rishabh Singh, Solar-Lezama and Sumit Gulwani) will present a paper about this at the Association for Computing Machinery’s Programming Language Design and Implementation conference.

This might be useful for MOOCs. TAs at MIT already are using the system.

Has Web-CAT been integrated with Moodle?

We tried Moodle integration in the past, but Moodle was too much of a moving target.  Web-CAT supports grade exporting in a Moodle-compatible CSV format that is easy for uploading into Moodle's gradebook.  Also, we're considering supporting LTI v.2 for better integration with other LMSes, but I believe Moodle only supports LTI v1 (addresses single sign-on, but not gradebook integration).

Thanks for your reply.  We need to integrate something for partial evaluation of code snippets as well as grading full submissions.

Are Bob & Bev Williges and Debbie Hix and Rex Hartson still at Tech?

I'm not sure what you mean by "partial evaluation".  While there are some systems that handle evaluating code for small-scale coding questions (usually where you write a single function/method), and there are also systems that handle full-scale programming assignments, I'm not aware of a system that does both and that can be easily integrated into an LMS.  I'm just not sure if that's what you're targeting, though.  I'm not aware of anything that is widely used that has deeper moodle integration support in terms of moodle's activities, but I'd like to know of anything you find.

I'll let you know if I find (or build) an all-purpose evaluator that can evaluate both partial and full programs (and be configured easily).

We use Stepic.org to create and host programming assignments. More info how to create them is here – https://stepic.org/lesson/Step-Code-9173

https://stepic.org

There are two grading systems I've come across in the last year worth looking at - one is http://www.inginious.org which uses Docker to have arbitrary code evaluation, multiple languages, etc. It's easy to install, has LDAP authentication and Moodle LTI integration including grade reporting. Short video showing an extension (using c-tutor) is at https://www.youtube.com/watch?v=l2hn_xq6B_g -- they don't have a video tutorial, but good docs.

Another is 'autolab' https://github.com/autolab - I haven't installed this yet, but the group of folks maintaining it at CMU are good. They also have an LTI interface, but haven't used it.

We explored using Web-Cat but the student involved didn't have the background in WebObjects. It would be useful to package this up as a Docker image so people can deploy it with less fuss.

After having researched grading practices for programming courses, my senior capstone focused on streamlining this process. This ultimately yielded in a full-scale web based platform -  AutoGradr.com which runs student code against test cases set up by professors. It's language agnostic, supports programs that read from/write to file, supports drag and drop of code (multiple files), integrates with Git 

Drexel's CS department has been using it  as the primary grading and submission tool for introductory C++ courses, and data structures courses. 

Professors can request free access anytime through the link, and get started immediately 

AutoGradr.com

I just started using stepik.org too.

It is easy to use and like it that we can integrate assignments into lessons. That way, I can lead the students to the assignment, providing them with the content they need to solve it right on the same platform. 

My only problem is that the grading is absolute. If I give 10 input/output pairs, I would  like the students to see for which of these pairs their code passed or failed and be graded accordingly. But right now, it gives them either 100% or 0 points.

 @Nikolay: do you have the same issue as well?

Another tool we are trying to set up and use is AutoLab. I think it will have more options to tailor it to our needs than stepik, but the setting up is definitely harder.