Forget about tests and "significance" - it won't tell you anything about the validity. The validity in your case is defined as the agreement of the diary with the reference (accelometer). You simply have to calculate this agreement. A second aspect of the question may be the precision, so you may further calulate and compare the precisions of the two diaries.

Since you are talking about "counts" as your measure, I'd suggest to use a Poisson-regression or a negative binomial model. It will geve you estimates (and their precisions) of the differences in counts between the diaries and the reference.

You then must judge if the estimated differences are relevant.

It appears that you are comparing diaries to accelerometers with the assumption that the accelerometer is accurate. You intend to test the diaries to compare the differences in validity between the diaries. Is this like saying the mean bias of A is greater than the mean bias of B? Is the intent of the comparison to show that diaries are unreliable? Is the intent to show that A's diary is more reliable than B's? 'Compare the validity differences' sounds like the latter. What is the import of this comparison? The import of the comparison will decide how much bias can be tolerated. Are you interested in the distribution of bias? Does the import require that you validate the accelerometers?

A significance test will not answer your question. It will only show that the bias is different from something based on the test metric.

You can simply compare test statistics to see which one is larger (for example F for ANOVA or t-student for t-test and etc. In fact, higher value of test statistics means there is higher probability that H1 is true. 

No, Hormoz, not at all.

It depends on how large and how skewed (when compared to the normal distribution) your data set is. With a large, not too extremely skewed sample the Central Limit Theorem will come into to play and the MEAN of your sample will adopt a normal distribution. Then you can use the typical t-test or ANOVA.

You can quickly check it out by bootstrapping your data and running a test for normality.

Rachael -

Hi. - If you have the accelerometer as a standard, say x, and plot it with y' on a scatterplot, where y' is the corresponding result from one diary method, and have another scatterplot where (x,y'') is plotted, where y'' is the other diary method, it may be as simple as looking at two linear regressions through the origin, to see which has closer to slope, b, equal to 1. (Attached is a link to a spreadsheet example that might be useful for you, though you only likely need to see b and standard error of b.)

However, do not assume linearity if the scatterplots clearly contradict this.

You'd want to look at standard error of b in each case, and you can also see what the standard error is for the difference in the two related b-values. (An internet search could help you there.) But I only like to look at confidence intervals or even just compare estimates to their estimated standard errors, as opposed to looking at hypothesis or significance tests, as a p-value is a function of sample size, and can be very misleading.

Hope that was helpful.

Jim

Data CRE Prediction 'Bounds' and Graphs Example for Section 4 of ...

Rachael - 

The attached spreadsheet here may be more helpful for you than the one above.  Each person has an accelerometer value and a corresponding diary value, correct?  No one fills out both diary forms, do they?  If so, then there would be dependence which you would need to account for with a covariance in the standard error of the difference in slopes. 

The nice thing about a confidence interval around the difference in slopes is that if it seems that the difference is acceptably close to zero, then your diaries each do about as well.  But you can look at the b-values and their standard errors and see how well the diaries seem to work.  Remember, you want an individual b in your case here to be close to 1.  

Scatterplots themselves can be very informative. 

Jim 

Technical Report A Note on Estimating Impact of Data Collection Mode Change f...