In his very helpful online book on structural equation modeling, Jon Lefcheck writes the following concerning d-separation tests for SEMs:
"Once the model is fit, statistical independence is assessed with a t-, F-, or other test. If the resulting P-value is >0.05, then we fail to reject the null hypothesis that the two variables are conditionally independent. In this case, a high P-value is a good thing: it indicates that we were justified in excluding that relationship from our path diagram in the first place, because the data don’t support a strong linkage between those variables within some tolerance for error."
https://jslefche.github.io/sem_book/local-estimation.html
My understanding is that this is a standard procedure in SEM and more broadly in DAG-data consistency checks, not a quirk of Lefcheck's workflow. My question is, doesn't this amount to an attempt to use p-values to control type II error rate?
Setting aside the inherent limitations of p-values even when used correctly, the fundamental problem is that p-values simply don't control type II error rate --- they control type I error rate.
The language in the documentation of the R package bnlearn, a suite of Bayesian causal discovery tools, is even more explicit in equating the failure to reject the null hypothesis of zero correlation with the acceptance of the null hypothesis, i.e. the conclusion that correlation is "in fact" zero:
"Now, let’s check whether the global Markov property is satisfied in this example. We will use the survey data to check the Null hypothesis that S is independent of T given O and R. In R, we can use ci.test function from bnlearn to find conditional dependence. [...] As you can see, the p-value is greater than 0.05 threshold, hence we do not reject the Null hypothesis and conclude that in fact, S is independent of T given O and R."
https://rpubs.com/sarataheri/bnlearnCGM
As a convert to causal inference and Bayesian logic from the cult of cause-blind null-hypothesis statistical testing, I find it frustrating that when it comes time to validate our causal graphs --- after all our effort to construct principled causal models and estimate parameters informatively --- we fall back on p-values, and a dubious use of p-values at that.
Am I missing something? I hope to be corrected.
I think here you are talking about the "basis set”. They (basis tests) all are INDEPENDENT tests. So, these INDEPENDENT PROBABILITY LEVELS (null probabilities) can be combined using the Fisher’s C test and then test whether ALL the d-separation claims in your DAG are true or false. In a DAG, many of the d-separation claims are logically entailed by other d-separation claims, which means they are not independent tests. However, the D-SEPARATION CLAIMS IN THE BASIS SET ARE MUTUALLY INDEPENDENT, so, we can test them using the Fisher’s C test – if all the d-separation claims in the basis set are true, then all the other ones are also true. You can choose whatever the probability levels you want. If the null hypothesis is true (i.e. each of the predicted d-separation claims in the DAG is true), then the C statistics should not be unusually large.
Tharaka S. Priyadarshana thanks for your answer. I understand the concept of the basis set and I have heard of (but do not fully understand) Shipley's global test (Fisher's C).
What I had in mind is the sort of itemized testing provided, for example, by the R package `dagitty`, which calculates Pearson's correlation coefficient for each path in the basis set. The idea is that you are supposed to use the p-values of these tests to decide whether each path is "significant", and wherever a path is "non-significant", you can justify the assumption of (condition) independence.
My problem with this is that it inverts the meaning of p-values. A p-value of 0.05 means that there is a long-run type I error rate of 5%. In the context of testing basis set paths, it means that the long-run rate of wrongly concluding non-independence is 5%. But that is not what we care about. What we want to control (if we're committed to a frequentist framework) is the long run rate of wrongly concluding independence along a basis set path.
To illustrate, let's adopt for the moment the decidedly wrong interpretation that a p-value of 0.06 means that there is a 94% chance that the alternative hypothesis is true. This is, of course, a falsely Bayesian interpretation of a frequentist statistic, but let's just say it's close enough to what you would get from a Bayes factor (or whatever). In the usual mode of hypothesis testing, we would hang our heads and fail to reject the null hypothesis...then proceed to p-hack the model until we got something significant. When testing basis set paths, though, we celebrate a p-value of 0.06 --- we can assume that the correlation is zero! But wait --- there is still (again, bear with me --- Fisher is rolling over in his grave) a 94% chance that the correlation is not zero. How does this amount to a validation of our basis set? To reverse the logic, it's like celebrating a p-value of 0.94 because we failed to reject the null hypothesis at an alpha of 95%.
As I said, I don't understand Shipley's test well enough to know whether the same critique applies to it. But what I describe above is, so far as I can tell, what people are doing with the most widely used tools for evaluating DAG-data consistency.
Douglas B. Sponsler thank you for this discussion. My answer may be a bit long-winded, so please bear with me. First of all, there is no way of inferring a DAG (causal relationships) from observational data until the d-separation claims are translated into a probability distribution, i.e., translating the graph theory to the probability theory. Testing for the probabilities associated with Pearson's correlation coefficient for each path of the basis set is NOT really accurate. Instead, we are testing for conditional independence and these tests can differ for each d-separation claim in the basis set depending on the nature of your data and the other relationships (typically we use regression tests). When you do a correlation test you can assign a probability to test whether that correlation coefficient is different from zero (i.e., null hypothesis testing the independence), but you cannot get a probability value to say that the correlation is not equal to zero. For example, you can test whether the correlation is different from 0.1, but it is different from saying that it is greater than zero.
If I understand correctly your main point is about the errors associated with the probability when conducting multiple testing, i.e., testing for multiple d-separation claims. As you correctly said, such multiple testing "inverts the meaning of p-values". This is why we do not test for all the d-separation claims in a DAG ONE BY ONE, rather we do that through the basis set. The basis set only includes d-separation claims evaluated as TRUE, and it has a special property, i.e., all the d-separation claims in it are MUTUALLY INDEPENDENT. So, now we can use the Fisher's C test to test whether the patterns of (conditional) independence in the sample data are consistent with the predicted patterns of (conditional) independence predicted by d-separation in the DAG.
How the Fisher's C calculated: Let’s say you run the “psem” function and its summary printed out (you have to use “conditioning = TURE” to print the conditioning variables) your first d-separation claim as, A ~ B + C + D. This means that A is d-separated from B conditioning on C and D, so you take the probability value for this claim. Note that you do not have to test for the probabilities associated with the correlations between each pair of variables in this claim. Let’s say you have five d-separation claims, so you will have five INDEPENDENT probability values. Now you take the log of each of those five probability values, sum them up, and multiply by (–2), and this will be the Fisher's C. If all the null hypotheses are true, then this C statistics should follow a chi-square distribution with a (2k) (k = number of d-separation claims in the basis set) degree of freedom. This null probability is the one you have to check with the arbitrary cut-off value of 0.05 (or another "meaningful" value). Again your decision on ALL the d-separation claims should not be based on the probabilities associated with each of the individual d-separation claims or the probability associated with the Pairwise correlation between each variable in the basis set, rather it depends on the null probability associated with the C statistics. It is ALWAYS better to simulate data and confirm your DAG before you test it with your research data, so you can be VERY sure about your DAG and d-separation claims, and then you do not have to worry if you get a “marginal” null probability for the C statistics. P.S. This logic is the same for the covariance-based SEM.
Tharaka S. Priyadarshana thanks for that nice description of Fisher's C. I think I understand it better now.
I'm still not satisfied, though, with the logic of p-values in tests of d-separation. It's not the issue of multiple comparisons that worries me. It's the question of what kind of error a p-value does and does not control. My understanding is that a p-value does exactly one thing: it controls type I error, i.e. the long run rate of wrongly rejecting the null hypothesis.
But what is our null hypothesis? When validating a DAG, the null hypothesis should be that the DAG is invalid, right? That is, we are obliged to demonstrate that our data are inconsistent with the null hypothesis of (conditional) dependence, i.e. of C (or ρ) ≠ 0. What we need is a test statistic that controls the long-run rate of wrong rejecting this null hypothesis, the long-run rate of wrongly concluding that our DAG is valid.
What a p-value does, though --- and again, I don't think it matters whether the p-value comes from Fisher's C or Pearson's ρ or Spearman's ρ or Tukey's turkey --- is control the long run rate of wrongly rejecting a different null hypothesis, the null hypothesis of (conditional) independence.
So, let me clarify what I said earlier. Test of d-separation attempt to invert the meaning of the p-value because the whole notion of d-separation has already inverted the null hypothesis.
I realize that there is no way to use a p-value to test an null hypothesis of C ≠ 0. This leads me to conclude that there is no way to use p-values to test claims of d-separation.
« p-value of 0.05 means that there is a long-run type I error rate of 5% » : this is wrong, and the similar claims that p-values control the Type-I error rate are also wrong.
p-value is the probability of the observed data (or data even more in disagreement with the null hypothesis) assuming the null hypothesis is true. Nothing more. It does not control any risk at all. In other word, p-value = 0.05 means that if H0 was true, you had one chance other 100 to observe the results you had (or results even less likely).
The decision rule « reject H0 if p ≤ alpha » controls the Type I error rate, at alpha. This is very different! Using this decision rule, for alpha = 0.05, a p-value of 0.9999 or of 0.050001 will have exactly the same interpretation, lead to the same conclusion ("do not reject H0"); just like a p-value of 0.04999 and a p-value of 10^-5 ("reject H0"). In other words, when using the decision rule, the exact value of the p-value is meaningless, only its comparison to alpha has a meaning.
Note that this means that you are wrong on your Type I error rate control, but it also means that you are right in not trusting the « p-value is high, so we can conclude that H0 is true », which is definitely wrong.
Douglas B. Sponsler, I think Emmanuel Curis has provided a very comprehensive answer to all your concerns. Another point I thought worth mentioning is that accepting (i.e., provisionally accepting) a DAG does not mean your data have proved the proposed DAG is true. One obvious reason is due to the statistical power to reject the null hypothesis, and another reason is the presence of equivalent models (i.e., models that have different causal claims, but with the same basis sets).
Emmanuel Curis and Tharaka S. Priyadarshana
Thanks for the stimulating discussion. Is it fair to say that we have converged on the conclusion that p-value-based NHST simply cannot achieve the stated purpose of DAG validation? Since DAG validation entails the (provisional) acceptance of independence along the paths of the basis set, and because p-value-based NHST by its very nature cannot quantify evidence for independence (or, strictly speaking, for anything at all), we need an entirely different framework. No amount of refinement can make p-values serve the purpose of DAG validation.
This leads me to conclude that only Bayesian inference can provide the kind of evidence needed to validate a DAG. The most straightforward approach, I think, is to develop bespoke Bayesian models for the paths of the basis set, and interpret the posterior probability distributions relative to some region of practical equivalence (ROPE) around zero. Maybe there could also be a place for Bayes factors, but they worry me.
This is a lot more work than (a) ignoring causal structure or (b) testing DAGs trivially with p-values, but is arguably better than being wrong about important things. One could even argue that the validation of the DAG is more important than the details of its parameterization, that DAGs themselves --- rather than the coefficient we estimate for them --- should be the chief object of inference and interpretation.
Any objections to this conclusion?
Douglas B. Sponsler
No objection, but a few comments:
1) there is no way to prove that a model, whatever it is, is correct. You can only prove it is wrong when observations contradict its predictions. That what p-values were meant for (with "contradict" meaning "event with too small probability to happen"), but it remains true for all approaches.
2) you may, as a workaround, consider the "equivalence" approach, that is define a region of acceptable departure from the model, and try to proove that indeed, departure is in this region. This is the idea behind equivalence tests (and around the ROPE), and in this case p-values can be used to reject the null hypothesis that the model makes unacceptable prediction. Of course with a lot of limitations for generability.
Practically, that would mean for your case that "correlations below 0.01 can be ignored" (in absolute value), and you may want to test that indeed, the true correlation coefficient is included in [-0.01 ; 0.01]. The choice of 0.01 or any other value is of course subjective, debatable and so on, and must be done before any model building. ROPE is the bayesian version of that, but frequentist equivalence tests work also very well... The usage is not to use associated p-values in this case, but it's just a matter of usage.
In other words: p-values for usual tests may be useful to « prove » that an edge does exist; to « prove » that an edge can be ignored, you may use p-values of equivalence tests. In both case, beware of the pitfalls of false positives and false negatives, not only "by luck" (Type I error rate) but also because of various biases, experimental design limits etc.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Bioinformatics