All of variables/factors/effects should just be called 'factors', but 'test person ID' is just an ID and not a factor (but still random). As a general rule, you need at least 30 participants. Each participant must perform the 3 categories ONCE each :)

Hi Andrew, thanks for answer. Yes, seems to be the best to have the 3 categories once in random order to eliminate "improved" search image. I think it will be feasible to have more than 30 participants, maybe as many as 50 will be possible. 

Hi Ronny. Random order of the categories is a very good idea and 50 is better than 30 lol :)

Unless the images that you are using were used in previous projects, you are trying a new experimental approach. As such you do not know if the differences in the objects are sufficient to show the trend that you expect. You may not even know what mix of ages you will get. Thus you should expect that you might have to run the experiment at least twice. The first time is a learning exercise for you, the second time (with modifications) is the real experiment.

What do you envision the data looking like? You have gone out and gotten 30 data points. You have ages 5, 12, 21, 22, 22, 23, 23, 23, 23, 24, 24, 25, 25 25, 25, 25, 25 , 25, 25, 25, 25, 26, 26, 26, 26, 28, 47, 79. What will this tell you?

We all age at different rates. By that I mean that at age 60 I might still be in good health, but another person of the same age might have more arthritis, or some other ailment. So one person at each age from 10 to 40 might not tell you a great deal because the biological noise will be larger than the effect you are trying to measure.

So may be the target would be something like I want 30 people in each age for ages 10, 30, and 60. After two weeks gathering data you find that you succeeded in getting 5 people age 10, 30 age 30, and 12 age 60. This is causes more problems because of the 5 people age 10, you got 4 males and 1 female. I will then return to my office and analyze the data. If it looks good, and there have been no problems, then I might (or might not) return and get an additional 30 people in each of the following ages 20, 40, 50, and 70. If things don't look good, then redesign the test based on your experience in the field and try again.

You might ask a few more questions. 1) are you an avid video game buff, and if so what is your favorite game? 2) Do you do any sport, and if so which is your current favorite (up to two answers)?

Thanks for answer @Timothy. It's a good point to control for previous experience with gaming. People being used operate tablets and smartphones will have an advantage, as the participants has to tap and hold the moving object. Since I'm only interested in the differences between the 3 moving objects and just controlling for variations associated with the participants, I might just narrowing down to use students from my classes (about the same age).

Hi Ronny,

What an interesting scenario : )

I will respond to each of your questions individually.

You ask:

1.            How many survey participants (test persons) do I need?

In order to figure out how many people you will need in your study, you can do something called an a priori power analysis. The term (a priori) means to come before, in this case the power analysis comes before the actual study.

In order to conduct an a pirori power analysis to determine sample size you will need three pieces of information. First, the alpha level (i.e., typically this is set to .05). Second, you need to determine the statistical power of your test (i.e., this is 1 – β, and is typically set to .80). Finally, you will need to know the effect size of the phenomenon (i.e., object detection) you are investigating. In recent peer – reviewed articles it is common for authors to report the effect size, so it is possible that you can find the effect size in the articles you are using for your literature review. However, if you absolutely can find no reference for effect size, the general rule of thumb is to assume a medium effect size (i.e., .50). But, it is very important to try and determine the exact effect size, because this number can have a tremendous impact on the number of people you will need for your sample. The larger the effect size, the smaller your sample will need to be in order to achieve sufficient statistical power to ascertain if you have a statistically significant result.

In order to run the a priori power analysis I recommend using a free software program called G*Power. You simply plug in the numbers you have (i.e., alpha level, statistical power, and effect size) and this program will show you what sample size you need.

You ask:   

2.      How many times should each object category be repeated?

From your description, I am assuming that you intend to run a within subjects (i.e., repeated measures) design. One important question to ask yourself when designing a study, is how will you analyze the data?

The main objective for anyone designing a study is to keep both the design and the analysis as simple as possible. This way, interpreting your data will not only be easier on you, but it will be clearer for your readers to understand.

Based on your description, it sounds like you want to include age, gender and object shape (i.e., 3 different shapes) as your independent variables. And your dependent variable is success vs. no successes (i.e., binomial).

In order to determine which test statistic is appropriate, and which is not appropriate, here is a breakdown of what (i.e., it looks as though) you are attempting to determine.

1)      ANOVA (Analysis of Variance)

a.       This analysis is ideal if you have 1 – 3 independent variables which can all be measured on a categorical scale (i.e., nominal or ordinal)

b.      The dependent variable must be continuous (i.e., measured on the interval of ratio scale).

                                                              i.      Since your dependent variable is categorical (i.e., success vs. no successes), you CANNOT conduct an ANOVA.

2)      ANCOVA (Analysis of Covariance)

a.       This analysis is ideal if you have 1 – 3 independent variables where at least one of these variables is continuous (i.e., interval or ratio scale of measurement). The continuous variable is your covariate.

                                                              i.      None of the covariates you suggest are measured on a continuous scale.

b.      The dependent variable must be continuous (i.e., measured on the interval of ratio scale).

                                                              i.      Since your dependent variable is categorical (i.e., success vs. no successes), you CANNOT conduct an ANCOVA.

3)      Multiple Regression

a.       This analysis is ideal if you have more than 3 independent variables, which can be either continuous or nominal. This is because, once you have more than 3 independent variables in an analysis it starts to become VERY difficult to properly interpret the results of your analysis.

b.      You can run a multiple regression with 2 or 3 independent variables as well, but here’s the thing, with a multiple regression you can only make a prediction about the level of an outcome variable. (i.e., whereas with an ANOVA or ANCOVA, you can establish whether or not there is a causal link between your independent and dependent variables). So if you are looking to make a causal inference with your data, you really want to set up your study (i.e., carefully choose variables) so that you can conduct an ANOVA, ANCOVA, or even a t – test.

c.       The dependent variable must be continuous (i.e., measured on the interval of ratio scale).

                                                              i.      Since your dependent variable is categorical (i.e., success vs. no successes), you CANNOT conduct a Multiple Regression.

d.       

4)      Binomial Logistic Regression

a.       This analysis is ideal if you have 1 or more independent variables, which can be either continuous or nominal.

b.      You have one binomial dependent variable, which is what you have. YAY!

Based on your description, it sounds as though a binomial logistic regression is the best analysis. Given the type of data you want to collect.

So, what does this mean for your study design?

You do NOT have a repeated measures design.

You only need to run your participants once.

Collect the data and use it to build a binomial logistic regression model that can predict whether or not an individual of a certain age and gender, given a particular object shape, will be able to either successfully vs. not successfully detect the object.

The main point is this, if you want to conduct a repeated measures design, you have to rethink how you are operationally defining and measuring your dependent variable.

Good luck,

Jolie     

Hi Jolie,

Thanks for your recommendations. I will dig into priori power analysis, that is new for me. As you suggest I will use  Binomial Logistic Regression with random effect, by the use of lme4 package in R. 

Just some thoughts on the design. Given that I have about 50 participants and that each only perform the test once with the 3 different forms presented. To avoid "predator learning", i.e the participant become more efficient on the third compared to the first due to learning, I consider the following:

What wwill be best? 

1. The 3 different forms comes in random order for all participants, with random each combination will not be 100% equally represented in the final data set. 

OR

2. The 3 forms comes in a fixed order, although the order change for each participant (the order is hidden for the participants ). Constructed so that each combination is equally represented in the final data set.