The methods you suggest will show whether it is reliable (the inter-rater agreement) and to some extent fair (across sites). So that is two of the big three from the Standards (http://ncme.org/publications/testing-standards/). As far as validity do you having any other measures (i.e., gold(ish) standards)?

The observational element of the study runs concurrently with a repeated measures student survey. they survey contains domains of the observational metric. I was looking at possibly looking at the correlation between the two. However i have been unable to find any literature around this concept thou.

The correlation will help to argue for validity since the true value should influence both of these. If you have a definition of a class activity happening and can measure that in some situations, then using your method and comparing the results will be useful.

The foundation for reliability and validity is based on  classical test theory, which assumes that each person has a true score that would be obtained if there were no errors in measurement. However, because measuring instruments are imperfect, the score observed for each person almost always different from the person’s true ability or characteristics. The difference between the true score and the observed score results from measurement error.The relationship between a test and a criterion is usually expressed as a correlation, called the correlation coefficient. The coefficient tells the extent to which the test is valid for making statements about the criterion. Inter-rater validity is a good predictor. A good instrument or test is one that has both high reliability and validity and one that can measure as accurately as possible, attributes presumed to be characteristics of students.

• I used Flanders interaction analysis system, and for validity usingCoefficient of agreement between observers.

You're working on an interesting project, Terry. I am looking for a way to study classroom facilitation across a few different instructors in college level mathematics courses that are being taught in the fall. I saw that you were looking for a way to validate your work. By any chance, has your instrument been published? 

A presentation of the pilot study was presented at the 21st Century forum at Harvard. The video of the presentation can be seen at

https://youtu.be/_48M8lBouqA

Terry:

Just in case you have not looked in RG, here is a link to several studies dealing with evaluating and validating instruments in different disciplines:

https://www.researchgate.net/publicliterature.PublicLiterature.search.html?type=keyword&search-keyword=Validating+a+Classroom+Observational+Metric&search-abstract=&search=Search

Many thanks,

Debra

Many thanks Debra.