Recently, I have read two articles that are trying to figure out why some students do well in their education, while others do not. The common factors that are considered include: a) social-economic advantage/disadvantage; b) scores on SAT or ACT; c) being first in family to go to college; d) being a racial or ethnic minority; and e) scoring well/poorly in certain courses. These would constitute, say, 5 factors.
But, we know that there are many, many more potential factors that might possibly explain why some students succeed while others do poorly.
The curse of dimensionality has traditionally restricted how many factors we can include in our statistical analyses. (The number of data needed goes up exponentially with the number of factors.)
However, there are new statistically-based techniques that can be used to handle many factors, one or a few at a time, in order to circumvent this curse. One such technique that is now used is BIG DATA analysis.
One example of this is work being done at Harvard by Raj Chetty.
I personally have not done BIG DATA analysis, but back in the 1970s I did use similar techniques for spanning high-dimensional data, based on mentoring I received from Leo Breiman, the famous statistician. It was initially based on generating a binary decision tree which reduced entropy at each decision node. Later, reducing entropy was replaced with increasing a measure called "mutual information."
Modern research into socio-economic-cultural issues that can explain the outcomes in peoples' lives must now include far more than 5 factors.
I would like to recommend you to read some reviews on agent-based modeling of socio-economic systems. It is an exciting area of research. In my project, Complexity Digest & Complexity in Medicine, you can find some examples.
There is Illachinski's books on Artificial war (it goes to the core) and cellular automata that can help you understand massive parallel computations. There are some reviews on the topic of MPCs from myself -- general ones.
Another good source of information is Dirk Helbing and his reviews and books. He started in pure physical modeling of societal interactions -- disaster escape from buildings -- which later extended to society and economy.
Those things give you a completely different -- massively parallel -- point of view.
In those models, you can reach complex systemic responses through quite simple local processes. The trick is to know what simple processes should be added to the simulating system.
In the context of this issue, not only do the factors need to be deconstructed into their components; but also the definitions and criteria for "success" need to be expanded. Success means different things to different people, for example: a) having advanced the state of knowledge; b) being financially secure; c) having helped other people; or d) self-realization.
Too often in this modern world success is conveniently and simplistically defined in terms of how much money you make.
I love your question, Antonio! I've had the same experience as yours, with more than two articles on this topic.
Undoubtedly the answer cannot be "constraints resulting from high dimensionality", because this problem has been solved years ago :-). I've coded machine learning solutions using Terabytes of high-dimensional data and can attest that current tools (EMR, Spark, etc.) make it easy and fast.
Unless I'm missing something, Jiří is offering an alternative approach to the one adopted in the articles you mentioned. I hope researchers that have participated in studies like the ones you described will chime in.
In the absence of different explanations, my bet is that the choice is mostly based on the "publish or perish" pressure (fewer factors can significantly speed up the data collection and analysis process, and consequently time to publish).
- Badges
- Science topic
- Similar topics
- Psychology