I don't know about any validated instruments, but fundamentally, you should assess the students based on the skills you want to see. For instance, you might ask them to write a program that adds 2 and 3. You can then grade them based on how advanced they perform this task, e.g. from lowest marks to highest marks (I won't bother with syntax since I don't know which language you're teaching):

b=3

c=a+b

Depending on the programming language you're using, there is support to help you set up an online testing system which automatically grades codes.

In truth, measuring programming skills is the easy part (it is mostly objective: does the code do what it's meant to), although discerning where bugs reside is only straightforward for expert programmers. The real challenge is helping students develop their programming skills, because that is very labour-intensive. Last semester, I trialled using a peer-assessment system where up to 5 other students critiqued each student's work. This was mostly successful: the students mostly learned how to do things and how not to do things based on what other students did and it improved their confidence. I gave 0.5% overall marks each week for participating in that week's peer assessment.

I taught beginning, intermediate and advanced programming for several years and one flaw that I have seen during this time is trying to isolate learning by modules. While module assessment is useful for learning, programming progresses in an incremental fashion. My approach was to have the evaluations divided into module(had to do it because of departmental requisites), module comprehension on top of previous material and problem solving and troubleshooting sections. At the end of each course (even introduction to programming) the students needed to hand in a project of complexity commensurate to the course level.

I have found from working in the industry that graduates who come seeking work based on module based programming learning to come with huge deficiencies in understanding problem solving. They know the syntax but do not know how to solve problems.

Thank you very much for all your help!

I use a rubric that is not validated but  it checks basic Delivery, Coding Standards, Documentation and Runtime.  I use this is a CS 0 and CS 1.  As they progress in the courses I get tougher on what column they fall into.  I also write notes and do screen captures when they do not meet certain criteria.    Along with this at the beginning of the semester they are given coding standards (CS0 is very rudimentary coding standards), Expected Documentation, and Expected Runtime of each project.  I only use this for programs they do on their own.  I grade labs on do they meet the expect learnng objective and grade 0,25,75,85,95,100.  I use this is C++ and Java.

Dear Dr. Jennifer Pollack,

Thank you very much for your answer. Your rubric is really helpful. Could you please provide me more information? Do you use your rubric for individual and/or group evaluation?

Joana Martinho Costa I only use this is classes that are individual project.  I don't do group projects in CS0 or CS1.  When I have group projects I use this sheet plus a peer review sheet and they must take all code with who wrote what, who tested what (shouldn't be the person who wrote it) and who contributed support (didn't wite the entire code but gave help to solve issues or answer to small details). 

you may want to look at the literature of learning and teaching novice programmers. There is no single metric, but many papers use the SOLO taxonomy to classify learners at different levels of that scale. Students can be assessed on reading skills (tracing or explaining what a code does) and writing skills  (coding only or  design + code).

Note the progress is not linear but spiral, the same concept is revisited and extended in later courses.  

Since I asked this question, we have been developing an instrument to measure the initial programming knowledge. It is now available here (in portuguese language): https://www.researchgate.net/publication/321945649_Desenvolvimento_e_validacao_de_uma_prova_de_avaliacao_das_competencias_iniciais_de_programacao

There is a body of work on developing a validated instrument to measure core CS 1 programming knowledge called the FCS1with full details of the development process and background in Allison Elliott Tew's PhD dissertation

https://smartech.gatech.edu/bitstream/handle/1853/37090/tew_allison_e_201012_phd.pdf?sequence=1&isAllowed=y

There are a number of shorter published papers on it including

http://people.cs.vt.edu/~kafura/CS6604/Papers/FCS1-Assessment-CS1-Knowledge.pdf

And a replication of the original instrument called the SCS1 by Miranda Parker here

https://dl.acm.org/citation.cfm?id=2960316

Following.