This is not a priori "wrong". Many health studies will dichotomize what are actually non-binary outcomes in logistic analysis, or, for convenience, will dichotomize continuous or multi-level independent variables in a regression to conserve analytic power (by reducing the number of variables required) or for interpretive convenience.

What comes up for me from your question is why you proposed to use the median as the cut-off value rather than the mean, or an a priori cut-off that has some external validity? Obviously there are distributional considerations at play and these are relevant too. If one can articulate the choice in terms that are meaningful and relevant to the question at hand then one can include that in how the question is framed, the analytic method chosen, and the results interpreted.

Since it is the DV you wish to dichotomize on, can one attach an interpretation to having performed above or below this median that is both externally valid (i.e., meaningful outside this sample) and useful to know? To some degree, that is fundamentally implied if the median DV value in one sample is being used to compare with another.

It also depends what you're doing analytically. For instance, if you intended to use contingency table type analysis then it matters that the separation into two classes due to being median performance is capturing something meaningful and has decent discriminatory properties. A somewhat orthogonal example to make a point, if you are looking at health behaviour in young people and your sample median age is near 25 and you have specific youth-oriented programs/services for youth aged under 25 then this externally valid, real-world consequential, policy-based cut-off of under-25 versus 25-and-older might be more relevant than basing a cut-off on having had a median age of 25 in your sample.

How much of the sample performs close to the median performance? If it is a lot then the categorization may be so noisy that misclassification issues could dominate group assignment. If your distribution of DV values is strongly bimodal and you're looking for a means to split the groups then there might be better ways to separate them than using a median. Overall, it's hard to say without more information on what performance the DV measures, what the distribution of values is, and whether it is meaningful to dichotomize them at all.

There are many parametric and non-parametric methods for comparing distributions that might be more powerful than simply comparing relationship to a central measure (such as a median or mean) across samples; e.g. Kolmogorov-Smirnoff, Wilcoxon-Mann-Whitney, Kruskal Wallis, etc. The best one depends on whether your DV meets the distributional assumptions for the test and whether you have many tied values or not. Simple comparisons of measures of central tendency might also suffice, e.g. t-test or ANOVA.

The median is a popular option for splitting threshold. What happens if there is no effect based on the median-split? Do any other splitting ideas come to mind?

The ODA machine-learning algorithm identifies cutpoint (threshold) values, or category assignment rule (for qualitative variables) that explicitly maximize the discrimination between (two or more) groups.

Here are some recent articles that describe this procedure in various contexts in the field of causal inference.

• Linden, A., & Yarnold, P.R. (2016). Combining machine learning and propensity score weighting to estimate causal effects in multivalued treatments. Journal of Evaluation in Clinical Practice, 6, 875-885.
• Linden, A., & Yarnold, P.R. (2016). Combining machine learning and matching techniques to improve causal inference in program evaluation. Journal of Evaluation in Clinical Practice, 6, 868-874.
• Linden, A., Yarnold, P.R., & Nallamothu, B.K. (2016). Using machine learning to model dose-response relationships. Journal of Evaluation in Clinical Practice, 6, 860-867.
• Linden, A., & Yarnold, P.R. (2016). Using machine learning to identify structural breaks in single-group interrupted time series designs. Journal of Evaluation in Clinical Practice, 6, 855-859.
• Linden, A., & Yarnold, P.R. (2016). Using machine learning to assess covariate balance in matching studies. Journal of Evaluation in Clinical Practice, 6, 848-854.
• ·Linden, A., & Yarnold, P.R. (2016). Using data mining techniques to characterize participation in observational studies. Journal of Evaluation in Clinical Practice, 6, 839-847.

 If you would like to see some open access articles, here is one that was published several weeks ago:

•  https://doi.org/10.7717/peerj.3020

Plenty of examples may be found in the open access eJournal that specifically addresses this matter:

• www.ODAJournal.com 

For example, using threshold values (for DV or IV) that don’t explicitly maximize classification accuracy can reduce model accuracy:

•  https://odajournal.com/2014/03/30/breaking-up-an-ordinal-variable-can-reduce-model-classification-accuracy/

And, optimizing thresholds used in regression models usually greatly improves their predictive accuracy:

• https://odajournal.com/2013/09/20/maximizing-the-accuracy-of-multiple-regression-models-using-unioda-regression-away-from-the-mean/

IMO, carving continuous variables into categories (which often have somewhat arbitrary cut-points) is usually a bad idea that typically results in loss of power.  Rather than go at length here, I've attached the slides for a departmental seminar talk I gave on this issue a few years ago.  HTH.

Binning is a risky practice.

Draw a normal distribution/bell shaped curve on a piece of paper. Select any two points, label them treatment A and B respectively. Draw a straight line between them. What do you conclude about the relationship between those two points based on the line? The next question is how many points do you need to get the true relationship?

Alternatively, maybe your DV is clearly bimodal. However, you don't have sufficient data to model each of the modes. Splitting this variable into two states would be one very reasonable option.

There is one other possibility. The DV that you are using will be split but not recoded. That is, you now have two variables DV1 and DV2 where DV1 has all the values from DV that were less than the median. In this case I think you have created more problems than you have solved. How do interpret the mean of the left half of a normal distribution? Of course all the statistical packages will happily spit out results for you, but that isn't an indication that the output has any meaning.

I'll add my voice to the others wondering why you would want to use a median split for a continuously distributed variable. Analyses involving continuously distributed data will be more statistically powerful.

And If I understand your question, you want to split your sample into groups based on the DV, then compare them on the same DV?? You will almost surely find a difference, but what does that mean when you assigned cases to groups on that variable in the first place? 

For example, if your DV was number of days absent from school, and you classifed participants into "low absence" and "high absence" from a median split, is there any reason you would not expect the two groups to be different on number of days absent? 

To Michael Seto: I am comparing the two subgroups with a reference/control group. Primary purpose of doing the median split. Without the reference group, there's no purpose of the median split.

See:

Robert C. MacCallum, Shaobo Zhang, Kristopher J. Preacher, and Derek D. Rucker. (2002). On the practice of dichotomization of quantitative variables.Psychological Methods, Vol. 7, No. 1, 19–40.

Abstract

The authors examine the practice of dichotomization of quantitative measures, wherein relationships among variables are examined after 1 or more variables have been converted to dichotomous variables by splitting the sample at some point on the scale(s) of measurement. A common form of dichotomization is the median split, where the independent variable is split at the median to form high and low groups, which are then compared with respect to their means on the dependent variable. The consequences of dichotomization for measurement and statistical analyses are illustrated and discussed. The use of dichotomization in practice is described, and justifications that are offered for such usage are examined. The authors present the case that dichotomization is rarely defensible and often will yield misleading results.

I'm with Michael Seto in trying to understand what such an analysis would tell you. If I understand correctly, you have a control group and one experimental group. Doing a median split on the DV in the experimental group would set you up with three groups: (a) top half of experimental group; (b) bottom half of experimental group; (c) control group. Comparing (a) and (b) doesn't make sense, as you said, but I'm not sure what type of research question you could answer by comparing either of them to (c). Do you expect the experimental manipulation to increase the variance in performance, so that some people in the experimental group perform much better and some perform much worse than the control group? If so, I don't think comparing the three groups' means is the most direct way to address the question (since it would seem that you really want to compare the variances, and/or try to identify a variable that's interacting with the manipulation, which this kind of split wouldn't help with). If you're not expecting increased variance, and instead expect that the manipulation will affect performance overall in one direction (say increase it), then it seems likely that you'll end up with a significant difference between the control group and the top group, and not between the control group and the bottom group. But it's unclear what that would mean, because all you did was remove the lower performers and put them in a different group.

YES, the median-split actually increased the total variance in performance, primarily explained by the comparison of the "top" group and the reference group. The median split is based on empirical precedents. There doesn't appear to be any confounding variables affecting the median-split, as long as regression analyses (with and without the split) are concerned. [FYI: regression analyses were done as additional analyses--to counter arguments that the median-split reduced variance, effect size, power, etc.] There are papers by MacCallum, Zhang, Rucker, et al., arguing against the median split. However, there are others, Dawn Iacobucci et al., who argued that the median split is find as long multicollinearity is taken care of among the predictors, which is obviously not an issue for my case. There're also something that I didn't add to the question details. The median split is considered because the two subgroups that emerged are well-matched/yoked in other variables apart from the continuous DV of interest. So they are regarded as "high" and "low" performers on the DV under analysis, and not confounded by any other variables. Now, I am acutely aware of the pitfalls of the median split technique; I perceive the median split as an offshoot of the taximetric methodology and all related works about the need for closeness/certainty, classifying personality traits, etc., but there are good instances when median split is actually a good idea--that is, to answer a certain research question at hand.

Those papers talk about doing a mean split on an *independent* variable to make two groups, which are then compared to a different, *dependent* variable. For example, a researcher might give people a test of working memory capacity and a vocabulary learning test. If the researcher for some reason doesn't want to do a regression analysis, they could look at the relationship between working memory capacity and vocabulary learning by splitting the group into high and low working memory capacity groups (with WMC functioning as kind of a pseudo-IV) and then comparing the two groups' performance on the language test. This is not uncommon, though there are the problems that are mentioned in the papers and by others here.

Unless you've got the terms mixed up or I'm totally misunderstanding what you're saying, though, you're describing something different. You're proposing splitting people on the outcome (DV -- like the language test in the above example), not on a predictor (IV -- WMC in the example), and then comparing groups on that same outcome that you did the split on (language test). I'm having trouble imagining a research question that this could address. It has nothing to do with statistical issues like power -- it's that the setup doesn't make sense. You need to look at two variables, not one. (Whatever manipulation differentiates the control group from the experimental group doesn't count here -- there's no manipulation within that group and that wouldn't be relevant for a median split anyway. The point of a median split is to make a continuous variable into a categorical one so that you can do a t-test or ANOVA instead of a regression when you're looking at its relationship with *another* variable.)

What statistics are you looking at to make the claim that "the median split increased the total variance in performance"? The median split itself surely didn't, since you're dividing the sample in high/low halves. Is there more variance in the experimental group than in the control group (greater variance or standard deviation in the full experimental group than in the control group)? Comparing the means between the high/low groups and the control group won't tell you this; you have to look at the variance or standard deviations.

To Victor:

Thanks for the lengthy response, buddy! YES, you're right, the setup is ambiguous. The idea wasn't mine, take note; I'm just trying to follow the proposal of my collaborator. The regression I'm referring to is a categorical regression (dummy variables with other continuous predictors). It's not the median split per se, but one of the dummy variable is accounting for almost half of the variance in the DV ("high" group versus the reference group). Yes, I've looked at variance, SEs/SDs, and Cohen's d. The stats may make sense but I agree, the method doesn't ascribe to the conventional notion of a median split since you require a separate continuous variable (a predictor) for the split. Like I said, I am not the proponent of the median split--that's why I'm asking this question to figure out whether this idea of my colleague is sound. I would call the "median split" in question a quasi-median split.

The dummy coding you're referring to is for group membership (high, low, and reference)? If I understand correctly, that would just be the regression version of a t-test. What does your collaborator think the analysis would tell you? All it says is that the best performers in the experimental group are better than the control group, which certainly doesn't suggest a strong effect of the manipulation, if you have to only look at the best performers in the experimental group.

Depending on who your collaborator is, you probably can't say "It just doesn't make sense," so here's what I see as the core statistical reason why this shouldn't happen for the situation you describe (which, note, is not necessarily exactly the same as the reasons not to do a median split the "normal" way, that is, by splitting on an IV). A basic assumption behind designs involving comparisons between groups is that observations are centered around a mean. There's a population mean for the control group and one for the experimental group. But there's variability around those means, both true variability and error. You hope that your sample represents both the population mean and population variability adequately. The question that you answer with a t-test or its regression equivalent is whether the means of the two groups are different, in the context of how scattered around the mean the observations are. Assuming a normal distribution, the mean is at about the same point as the median. If you do a median split, then you split the experimental group into two groups, high and low, right around the mean, i.e., your estimate of the "true" population mean of the experimental group. You can still calculate the mean of each group, obviously, but it's not meaningful, because you only have half of the full distribution that is centered around the actual mean of the experimental group. You don't have a full distribution anymore; you have two half-distributions. So when you then compare the means of these two half-distributions to the control group, neither one of them represents the effect of the intervention: the mean of the high group is artificially high, and the mean of the low group is artificially low, compared to the overall experimental group mean that represents the effect of your IV. (The variability will also be reduced, which will affect a t-test or regression equivalent as well, because the tests consider the difference between means in the context of the standard deviation; when the standard deviation is lower, the same difference between means seems larger).

Note that this is somewhat different from the typical arguments about a median split on the IV (e.g., loss of power, no reason to expect someone just above and just below the median to be meaningfully different, etc.). The situation here is a more basic question of what do the numbers even represent. If you median split on the DV, then you no longer have a number that represents the effect of the IV (i.e., the difference between the experimental group mean and the control group mean).

This is the most straightforward statistical argument I can see as to why not to do a median split on the DV.

To Victor:

Thanks for the lengthy account again! Yes, I see your reasoning. I have also considered them many weeks ago. However, the quagmire is that my collaborator is from the medical field and this method of performing a median split (or dichotomization with +1 SD and -1SD of the mean/median) on a performance DV (not the IV) has been done and papers have been published (believe it or not, in solid Neuroscience journals) [and we're citing those papers]. Ostensibly, this quasi-median split is fine for some researchers in his field of expertise. I shall remember and convey your ideas, which is really quite similar to mine. Wish me luck!

FYI: There may be something I missed. Analyses were performed and presented WITHOUT the median split. The median split was an additional analysis. For the regression, it is of course similar to any t-test or ANOVA since ANOVA and regression are two sides of the same coin. Regression is performed due to the request of reviewers.