Doruk Engür , hello:

It appears to me that the study which you propose does not require any correlation analysis at all, as the variables appear to be all ordinal. You appear to have the following variables: (1) Question correctly answered (yes/no), as you appear to use a multiple choice approach; (2) Question design (A/B/C/...); (3) Gender. You might simply add up the results and provide a table. You could calculate sums within the matrix, and get to results like: "If the question is asked '1+1=?' (design A), 53% of the men (n = 53) provided correct answers, while only 23% (n = 23) answered correctly if the question was designed as 'Tell me the sum of one plus one' (design B). The design A question was correctly answered by 54% of the women (n = 108), and 46% of the women (n = 92) answered the design B quesion in the right manner."

You see, in this case, you get results which you can build the analysis upon without needing correlation, statistical power, or the definition of moderators. When assessing the quality of your results, you would have to provide information on the composition and way of acquision of your test persons, how representative they are for whatever larger group of persons (without that information, only the institution where the test took place, or the participants themselves, or, in case of minors, their parents might find your study interesting), and why.

In summary, the gender appears to simply be the variable which splits the total group into two.

But if I misunderstood you, and you want to perform a correlation analysis, please allow some remarks:

In a correlation analysis, variables are employed in a "symmetric" fashion: you test whether variable A correlates with variable B, in a one-tailed or two-tailed test (in your example, the two-tailed test appears appropriate, see here: https://stats.oarc.ucla.edu/other/mult-pkg/faq/general/faq-what-are-the-differences-between-one-tailed-and-two-tailed-tests/). Therefore, you would not worry - in this context - about what variable is dependent, and what variable is independent, but you should know it for the interpretation of the results at a later stage.

Let me provide an example of a math test: Imagine you take the single participants as cases in your analysis, and imagine they should write down some self-formulated answer, so they receive points from 0 to 10 for the quality of the answer (represented in the dataset as single variables). And imagine you put five questions. And assume that, as an independent variable, you would use the time of the day, the times ranging from 9 in the morning to 6 (18:00) in the evening. And then you start the correlation analysis on the results in your software. You would not be surprised that you find correlations between the time of the day and the quality of the answers (at least, I assume that, without having in-depth field knowledge), but that there are also some correlations between the dependent variables found (meaning, the points received from the answers). Apart from the fact that you did not perform the analysis to test that, but your software simply performs that test collaterally, you would also not be too surprised. The quality of the answers will depend on the individual ability to solve mathematical problems. Here, your knowledge of the field comes into play. You probably even would not report such correlation in a paper, unless that finding is not considered trivial in your field. In contrast, what you would report is the correlation between the quality of the answers to the single questions and the time of the day. Here, you could discover that some sorts of questions would be answered in the same manner at all daytimes, while the differences are remarkable with regard to other questions.

And here lies one big danger of correlation analyses, that is: potentially undetected nonsense correlations, moderated by other variables. You might find (I phantasize here) that the density of barber shops in cities correlates with the public transport prices. This not so unrealistic correlation might depend on several moderators, which relate to the structure of cities and local socioeconomic factors. Changes of the moderator or moderators (those factors) would lead to a change of the outcome (barber shop density and transport ticket prices). Therefore, this implies that the moderator is subject to change, at least theoretically. Otherwise, it is a constant value.

The gender of a person taking a test at a given time is a constant value in that sense. It could be a moderator if you inspect groups. But then, not the individuals are the cases, but the groups are.

Let me elaborate on that, with regard to the inclusion of the gender of the participants: Considering some gender based variable as a moderator might be possible if that variable can theoretically be subject to modification. For example, if you inspect the respective mean or average result of participant groups (e.g. mixed gender school classes), and if each group has a gender composition, you could consider that specific composition of each group (constructed, for example, as the number of men divided through the total number of group members) as a possible moderator. For testing your hypothesis, you might then check the correlation of the mean or median outcome of the tests of each group with the value of the gender composition variable of the respective group. In your data matrix, which you use as the basis of the correlation analysis, each case would represent one group, the first variable representing a (median or mean) test result, and the second variable representing the gender composition.

But this design has its pitfalls: For such a correlation analysis, you require sufficient statistical power - in short, you need enough data points / cases. And these data points are the single groups then, and not the single participants. It is normal to require a statistical power of 80%, some fields, journals, or supervisors require 90%, depending on the field. Tools like G*DATA (Article Statistical Power Analyses Using G*Power 3.1: Tests for Corr...

However, there might be room for a correlation analysis, though, even if you use individual participants' results as cases. In such desgins, the typical approach would be to split the set of participants into at least two sub-groups, men and women (and possibly a third diverse gender, depending on your participants). You would then start two different correlation analyses: One for the correlation of test results of men with whatever other variable, and one for the correlation of test results of women with whatever other variable. If "whatever other variable" is not present, you would not perform any correlation analysis. Beware, if you want to test whatever correlations for these sub-groups, you would have to have sufficient statistical power for each of the sub-groups. If you perform several correlation tests, check if you have to apply the Bonferroni correction.

Another option, if you have a number of independent variables, could be to perform a cluster analysis. For example, if you have data like age, gender, household income, distance of the home from school, and what else might be possible, you could use a smart statistics program to calculate clusters for you, based (only) on these independent variables. You might end up with a cluster of rich participants living far away and being predominantly women, a cluster of older men who live nearby in medium income households, and the like. And then, you could test how the collective test results of these groups differ from each other (the latter also not being a correlation analysis).

I hope that this has helped a bit; if I misunderstood you or was wrong at some point, please feel free to correct me.

Oliver Maor, I think I couldn't express what I wanted to ask properly. The term "correlational research" in the question I asked was not related to statistical correlational analysis. When we examine the relationship between variables without manipulating any of them, I call it correlational reserach. Some refer to it as causal comparative if they examine the relationship between two variables using a t-test. For example, if they calculate the Pearson correlation coefficient for the sleep duration of students and their math scores, they call it correlational research; if they divide these students into two groups, those who sleep little and those who sleep a lot, and compare the math scores of the groups using a t-test, they call it causal comparison. I call both of these correlational research.

However, my question was not about this. To emphasize the part I want to open for discussion, let me ask the following three questions:

1- In order to check whether people's pupils dilate when light is shined on them and whether this enlargement varies according to gender, you shine light on the subjects' eyes in some cases and not in others. Is this research experimental or correlational? Or if you use another name, what do you use?

2- To investigate whether math scores vary by gender, you asked for student scores from a math teacher and analyzed the scores of female and male students using a t-test. Is this research experimental or correlational? Or, if you use another term, what is it?

3- You are investigating whether people answer math questions more accurately without seeing them or by seeing them, and you are curious if the effect of seeing the questions on giving correct answers varies by gender. For this, you sometimes show the questions and sometimes do not. Is this research experimental or correlational? Or, if you use another term, what is it?

Hello Doruk,

in the field of social sciences, we would normally differentiate between observational (non-experimental) designs and experimental designs. The term "correlational" seems odd because a correlation is a statistical parameter. Perhaps "associational" would work and concur with the "observational" above.

The key difference is twofold (and can be got by answering two questions):

1) Did YOU administer the treatment (or more general: "influence the value in the IV") or did you only passively measure it?

2) When you did influence the values of the IV: Did you have full control over the assignment or could other forces (interest, ability etc.) have an influence?

In total, the question is about exogeneity of the IV. In the random experiment, both questions would be answered with "yes" if there is no endogenous selection (i.e., the dependent variable forced participants out of the study).

If the first is "yes" and the second is "no", it would be a quasi-experiment. This would be the case if you do a training, and the training participants participated at own will. This implies the danger of unobserved confounding.

If the answer is "no" to both questions, than you have an observational design with the need to come up with a causal identification strategy (using control variables or instruments).

I don't know if this helps :)

Best,

Holger

Thank you for your reply, Holger Steinmetz ,

I use the "correlational research" term as Field (2018) defined "a form of research in which you observe what naturally goes on in the world without directly interfering with it. This term implies that data will be analyzed so as to look at relationships between naturally occurring variables rather than making statements about cause and effect."

As I said before, some people use causal-comparative when one variable is binary, making two groups.

For example, Creswell gives the name according to the statistical analysis which I don't like. he says, "One type of nonexperimental quantitative research is causal-comparative research in which the investigator compares two or more groups in terms of a cause (or independent variable) that has already happened. Another nonexperimental form of research is the correlational design in which investigators use the correlational statistic."

Fraenkel et al. use the correlational and causal-comparative distinction, too.

Gravetter & Forzano use the terms "differential research" and "correlational research" and say, "Many researchers place differential research in the same category as correlational research." They define "differential research" just like the ones who uses causal-comparative. (But I'm one of those who place them in the same category)

As it's written in the APA manual, "Nonexperimental studies (in which no variable is manipulated) are sometimes called, among other things, "observational," "correlational," or "natural history" studies."

You said, "The term correlational seems odd because a correlation is a statistical parameter."

There is no need to get hung up on the name, but it is not right to consider the word correlation as a word specific to statistical analysis. Nothing prohibits us from using this word, which means "together relationship," as a research design because it is used in statistics.

The sentence "Correlation is a statistical parameter" is not entirely accurate. Correlation is a statistical measure or method rather than a parameter itself. In statistics, a parameter typically refers to a characteristic or numerical measure that describes a population.

I don't know why we ended up here. I know what is experimental, what is nonexperimental,

All I wanted to start a discussion about this:

When we give an exam to a student, we assume that we are measuring their characteristics without any manipulation. However, to make the measurement, we are giving a stimulus (exam questions) that can reveal their knowledge. In this case, how much do we "passively measure it" and or do we "influence the value in IV."

Suppose we are just "passively measuring it" and comparing boys' and girls' scores. In that case, this is observational, causal-comparative, correlational, or whatever you call it, but it is nonexperimental since there is no manipulation. But is it really just passively measuring?

Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Upper Saddle River, NJ: Merrill.

Field, A. (2018). Discovering statistics using IBM SPSS Statistics. SAGE.to describe a mutual relationship or connection between things.

Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education. McGraw-Hill.

Gravetter, F. J., & Forzano, L. B. (2012). Research Methods for the Behavioral Sciences (4th ed.). Belmont, CA: Wadsworth.

Hi,

1) I would use correlation indeed as a statistical parameter (defined as the standardied covariance) but I can agree that there is broader more qualitative interpreation (in the sense of any kind of relationship. Ok

2) I like Andy Field but is statment is nonsensical. Apart from purely predicative modeling (such as machine learning) were the goal is only to infer the unknown value of y given a set of predictors we have always causal interests. In observational designs, this is harder to support (due to the danger of unobserved confounding and reverse causation) but it nonetheless is our goal. Simply knowning a "correlation" does not teach you anything from a theoretical perspective.

3) With regard to "statements of cause and effects". As noted we ALWAYS conclude causal effects but the basis for that is sometime weaker, sometimes stronger and depends on your assumptions, whether the assumptions were plausible, testable and in line with predictions

4) Your example of measuring Y for boys and girls is passively measuring Y although you present the measurement instrument IF you do not affect the values of Y. If your presentation of the measurement instruments indeed affected the students, than I wouldn't call that experimental (as this was unintentional and not systematic) but indeed a form of manipulation (defined as the effect on X).

5) The use of names for design clouds the very basic issue namely the one about the model and the assumptions. You correctly describe the scenerio of comparing boys and girls as observational--and yet you could make a valid causal conclusions due to the exogeneity of the gender variable.

Best,

Holger