My answer refers to teaching applied statistics, for students with stats as subsidiary subject (not for math or stats majors - that's a different topic!)

Let me give an allegory, "doing statistics" as "driving a car":

Students should learn to drive a car. This can be taught technically quite simple, and it is surely helpful to actually drive a real car (i.e. use some statistics software and do some data analysis). However, still most courses focus on engine design, fluel pump efficiencies, thermodynamics, gear-box friction coefficients and so on. Gear boxes are opened and closed, ingnition bulbs are exchanged etc. (i.e. teaching formulas, matrix algebra, calculus, doing calculations "by hand"), presumably because it seems to be too simple "just teach how to drive (technically)".

However, it is absolutely correct that "just teaching how to drive (technically)" is far too less. But adding technical details about the engine and the thermordynamics is the wrong way to go. It would be so eagerly required to understand how car-driving relates to the problems the students (will) have: traffic jams, running costs, environmental pollution, social status, resource depletion etc. For this the students need to get a thorough understanding of the whole context of which car-driving is a part. But this is a difficult and time-consuming part, and to my experience the teachers rarely understand this context themselves (instead they usually understand quite a lot of all these uninteresting and unhelpful engine and thermodynamics details; teachers are skilled engineers, but they often have no experience in actually driving a car, they might not even have a driving licence, they never looked at the context [because this is not their job!] - this is a severe practical problem when the focus of statistics courses should be changed).

To my opinion, (applied) statistics should teach 99.9% of the time this context (what I call "statistical thinking", comprising an understanding of what is empirical science, data, information, learning, knowledge) and can spend 0.1% of the time with teaching some software. It is good that students do not need to learn or practise "stupid" calculations (note that noone today draws a chart or diagram "by hand" as well, even when a simple sketch would be absolutly sufficient to show the relevant aspect), so more of the available time can be used to understand the context.

Yes, I think more time should be spent for students to grasp and understand the reasoning processes of t-test, anova, post hoc tests, correlations and the other statistics.  I appreciated the working out (with a calculator, before using SPSS) and writing it out as well.  That took time and perhaps allowed constructivism links to form in the brain.

My answer refers to teaching applied statistics, for students with stats as subsidiary subject (not for math or stats majors - that's a different topic!)

Let me give an allegory, "doing statistics" as "driving a car":

Students should learn to drive a car. This can be taught technically quite simple, and it is surely helpful to actually drive a real car (i.e. use some statistics software and do some data analysis). However, still most courses focus on engine design, fluel pump efficiencies, thermodynamics, gear-box friction coefficients and so on. Gear boxes are opened and closed, ingnition bulbs are exchanged etc. (i.e. teaching formulas, matrix algebra, calculus, doing calculations "by hand"), presumably because it seems to be too simple "just teach how to drive (technically)".

However, it is absolutely correct that "just teaching how to drive (technically)" is far too less. But adding technical details about the engine and the thermordynamics is the wrong way to go. It would be so eagerly required to understand how car-driving relates to the problems the students (will) have: traffic jams, running costs, environmental pollution, social status, resource depletion etc. For this the students need to get a thorough understanding of the whole context of which car-driving is a part. But this is a difficult and time-consuming part, and to my experience the teachers rarely understand this context themselves (instead they usually understand quite a lot of all these uninteresting and unhelpful engine and thermodynamics details; teachers are skilled engineers, but they often have no experience in actually driving a car, they might not even have a driving licence, they never looked at the context [because this is not their job!] - this is a severe practical problem when the focus of statistics courses should be changed).

To my opinion, (applied) statistics should teach 99.9% of the time this context (what I call "statistical thinking", comprising an understanding of what is empirical science, data, information, learning, knowledge) and can spend 0.1% of the time with teaching some software. It is good that students do not need to learn or practise "stupid" calculations (note that noone today draws a chart or diagram "by hand" as well, even when a simple sketch would be absolutly sufficient to show the relevant aspect), so more of the available time can be used to understand the context.

I totally agree with Jochen Wilhelm and his comments. Traditional methods of statistical teaching becomes boring with the applied part. In fact, I suffered severely during my Masters and PhD courses in statistics learning all the difficult theories and failing to understand what all these formulas and maths has to do with my data. Now, when I am analyzing data myself, I learn what is needed and this works best for me.

Yes, Dear Eddie, there must be a shift in classroom teaching from "calculation" to "explanation", since the old methods consume more time.

  I do not have a happy answer for this problem.  

  Unfortunately, although it is true that a solid education in statistics - and probability- is time-consuming, there is no way to learn without putting in those hours. Correlated results often depend on this analysis, and some of the concepts (criteria of underlying populations for specific statistical tests, restrictions of small data sets, etc. ) are too easily overlooked even by experienced researchers when using software without the theory behind it.  

       The SNR (signal to noise ratio) problems in modern published science result in confusion about published results - and then additional time is needed to figure out whether what one is reading is valid. 

I agree with Jochen as well, and I think a balance of both is necessary. To use Jochen's "doing statistics as driving a car" analogy, it's great to be able to simply be able to drive the car (i.e., do the analyses on a particular software and interpret the results), but what if the car breaks down and you're stuck? You could call someone to come and help you out, like a mechanic, but it would be much more efficient and less costly to be able to do it yourself, or at least identify the problem and understand what you need to do to fix it so you can help the mechanic out when they get there. Likewise, with statistics, it's great to have outside resources (statisticians) to help you out, but if you can grasp some of the nuts and bolts of statistics and understand the theory and limitations, then efficiency can be optimized for your statistical needs.

So, I don't think we should we stop teaching the "nuts and bolts" in the classroom completely, but we can probably sop focusing on them TOO much. My introductory statistics course ended up teaching me nothing about application and everything about theory - this is something I would consider insufficient. However, many institutional departments are now implementing "quantitative research" based courses, in which you learn to actually run statistics, interpret the output, and derive evidence-based conclusions from the results in the context of a particular field. For example, in my undergraduate degree, I took a "Quantitative Biological Research" course, which taught application and interpretation of a wide range of statistical tests relevant to biology/ecology and was a fantastic complement to the introductory statistics courses that I took early on. The same type of course is taught here at the University of New Brunswick, where I'm currently completing my dissertation. I think such courses are a fantastic way to complement the "nuts and bolts" of strict statistics courses and can bring everything full circle. 

Jeff, I honestly think we are beyond the phase where we (drivers!) should all be able to repair a car when it's broken (to stay in the analogy).The cars are of excellent quality, and we have a huge (open-source and free) car-sharing pool, so to say. If your car at hand doesn't take your luggage then you can simply take another one. No need to take the welding equipment and try to modify your car.

If your car get's stuck or broken then you obviousely made something wrong. That's not the problem of the car and it is not solved by repairing - you would do the same mistake again. You should understand enough of cars not to break them. But if this is the case then I really wonder where you would get by maltreat the nuts and bolts...

*switch*

If this would be really true: "[...] courses, in which you learn to [...] derive evidence-based conclusions from the results in the context of a particular field." - this would be a really great achievement. To my (limited) experience, "evidence based" is a big keyword right now, especially in medicine. But the people holding these flags high could never explain me what precisely is meant by "evidence". Most of them think that this is related to p-values from hypothesis tests, not recognizing that this is utter nonsense (additionally they are generally unable even to give a correct definition of a p-value!). So I am quite sceptic when people talk about such things. As you wrote, "in the context of a particular field" is a key point. Often this is considered as "the collection of methods that are typically used in the particular field", what is again far off the mark. In the correct sense it means that you really have to be an expert in this field, you really have to have background knowledge and experience in order to judge the evidence from data (what is per se only possible in a given context - the context of the particular field).

It takes a hell lot of effort and time to know enough of a particular field to be able to judge evidence. That's the way to go, for sure. In courses one could train this for very simple subjects. And still this would take so much more time than what is usually given to statistics courses. Therefore statistics should be more integrated to other (experimental, practical) courses, where expertise in a subject is taught and data is generated and has to be interpreted. When students then start asking what they can learn from the data, and two groups have a "significant" result and three other groups have a "non-significant" result, then they might recognize that the whole concept of testing significance is not related to evidence and does not answer the research questions...

I think I will partially agree with the suggestion of the question.

OK, we are not doing computations by hand now (especially those 'marvellous' summations of the old time), but this does not prevent us from studying more.

On the contrary, the fact that a software will give you almost always an answer, even if you have entered an obviously ill defined problem, must make us more suspicious and more theoretically qualified than yesterday.

So, in order to understand when the program gives just nonsenses and when it gives a robust answer, you have to do

• a combined study of theory and applications based on the chosen platform, which does not reduce the calculations but makes them sometimes more, since you can handle more sophisticated kind of data now.

"On the contrary, the fact that a software will give you almost always an answer,"

Yes, and this answer is 42 :)

Douglas Adams realy hit the mark!

And another one (also taken from the "The Hitchhiker's Guide to the Galaxy":

“We demand rigidly defined areas of doubt and uncertainty!”

The car and driver analogy seems to work pretty well, so I'll try to extend it to add my 2 cents. So far we've defined drivers as ones interested in using a particular statistical machinery and mechanics who more or less understand the nuts and bolts. I would be temped to define several more groups, and additionally redefine mechanics. I'll also point out before hand that no single player in the story that is about to unfold will be capable of correctly designing, building and operating anything more complicated than a horse drawn carriage. 

The analogy of a driver/operator/pilot is pretty good for the typical user of your typical statistics software package such as R, SAS, or SPSS. They are tasked with understanding a fair amount of detail that is not really related to statistics at all. Operating the machinery safely and efficiently, that is designing and running experiments, collecting and curating data, and performing a useful and valid analysis of the data, could ostensibly have significant variation in difficulty. One could perhaps imagine the difference between a horse drawn carriage and an advanced fighter jet. 

Back to driving, I would compare choosing a good statistical technique to renting a car or generalized vehicle. The average user acting as a mechanic would be perhaps comparable to renting a prius and modifying it to perform a certain task, say moving your household across town, when one would have been better off renting a u-hall or hiring professional movers. The probability of failure is high, and even if the effort is successful, it probably will probably take more time and effort.

I would compare statisticians more with engineers than mechanics. Perhaps the fighter jet is a better mental image here. Certainly aerospace engineers and jet fighter pilots have different skill sets, but it's not necessarily clear who has the more difficult job, but rather the skill sets are disjoint and difficult to compare. It seems difficult to imagine a pilot being willing to climb into a jet he designed, but also difficult to imagine and engineer piloting one that he designed. I'm sure there are exceptions, but I would be willing to bet that an analysis of their life expectancy would reveal some sort of interaction term.

I apologize for the next comparison, I don't mean to be unkind towards software developers or computer scientists, but while software is important, I can't think of a less offensive way to incorporate it into the analogy than by comparing building computational tools to an automotive manufacturing facility. A good method poorly implemented is in fact not a useful piece of equipment at best being inefficient, and possibly failing spectacularly. 

Mechanics, I think, are more closely comparable to statistical consultants or data scientists. In the car analogy they are the only party that's not strictly necessary. It might not be the optimal solution, but you could get away with buying a new car every 3 months rather than changing to oil. And while, perhaps not strictly necessary, taking your car to a mechanic if you don't know how to replace the oil (or the engine), or your data to a statistical consultant, is often worth the time. 

Miranda, Sheikh Mohammed, Mahfuz, Jeff, Demetris, Shane, Susan and Jochem, thank you for reasoned responses. I think all of us in this forum concur to the analogy made by Jochen on the driver and mechanic. I do believe that a thesis student using statistics, or a researcher applying statistics, or a non-major student taking statistics, or a professional statistician, or a professor in statistics will need different calculation times and explanation times because of their different "priorities" and focus.

But I also believe that the advent of these software will have or did already have a tremendous effect on the time balance between calculation and explanation to many statistics users and professers.

If you are going to make your students do "hand calculations" the problems should be fairly easy. Especially if you are going to make them do it on a test. Dropping 10 numbers into a calculator and finding the mean and std dev is fairly easy. Forcing your students to calculate, Xi-X, (Xi-X)^2 to get a std dev for the 10 numbers above or the expected value for a multi-tier lottery on an exam is uncalled for. Understanding if the results the calculator gave are useful or correct, is called for.

You can spend a lot of time dealing with the theory behind the stats. Sometimes you will run into problems with competing theories, then what do you do? I have nested data where any software you choose will determine that one of the variance components is negative. But, variance can't be negative. So, the software will go back and recalculate the negative variance as 0.00 and scale the other variances accordingly. George Box ran into some of these types of issues with CUMSUM charts and QC stats. He was practical enough to realize his theory/calculation was wrong. 

In my first Design of Experiments class, our prof decided to give us a 8x8 matrix from which we were to calculate the the coefficients for the experimental data he gave us. He wanted to see every step of the process as we went row by row creating the X, X' matrix, matrix multiplications, etc. (No calculators allowed) We had 90 mins to do 6 problems. The highest grade on the exam was about 50%. (No curves). Meanwhile, math profs in the linear algebra class would do a 3x3 matrix with the RREF method and tell the class, after they made a couple mistakes, "This is why we use software. Does anyone here have so much free time that they can spend 20-30 mins per problem on their homework? Is anyone that silly?" (Oddly, students hated the DOE prof and loved the math prof. Wonder why?" 

Math and stats profs tend to have this insane idea that, "If you can't "prove it" you can't use it." To them I say, "Draw out the metabolic pathway of oxygen in the krebs cycle. Remember, if you can't "prove it" you can't use it. So no breathing!" Or better yet, "Explain the metabolism of caffeine in the human body at different dose levels. if you can't, no coffee, tea or cola drinks for you!"    

A note about Jochen's analogy, and what Steve C was saying:

The Cub/Boy Scouts of America give out a badge for automotive repair and maintenance. In an interview, a mechanic told a visiting troupe that the best thing they can do is have a cell phone and a AAA(roadside assistance) card. The reason, if a typical car breaks down on the road, even a trained mechanic would have a real hard time fixing it roadside.

If you watch "Wheeler Dealers" you know that even a trained mechanic in a shop can have a hard time changing spark plugs.  Changing a light bulb in the dash often requires removing large sections of the dashboard. The easiest part of replacing the serpentine belt is replacing the serpentine belt. If you don't have everything lined up properly to begin with, putting on a new serpentine belt is useless and possibly damaging to the car. Some cars don't even have dip sticks to check the oil. You have to trust the oil life monitor. ( I guess it is a good thing sensors never wear out on a car;-) 

Explain nothing!

Apart from students in mathematics and statistics, it seems unhelpful, tedious, and a waste of time to explain the intricacies of statistics to students, say, psychology (my part). What they need to learn is to use the right statistical tools, that is to say the right tests, according to their data and hypotheses they want to test. What students need is to acquire an expert system capable of making decisions to run a good statistical software with their data. Do not forget that in the modern form of scientific publications, the figured data are rarely presented. What is shown are the results of tests. No more writing "mean = 12.23 (P = 0.005)," one directly writes "0.005." It is therefore clear that the results of the study will be understood as conclusions in a statistical test and less as a calculation on data. So the choice of the test and the results of the test will prevail.

That said, nothing is perfect and it is true that research (in psychology, I do not want to talk about something else) becomes a bit routine and formatted. What teachers must teach is the art of using the right tests, the limits of significance of tests, incorrect interpretation of tests, but not necessary entering into a detailed understanding of statistics. The best method, in my opinion, is to use a good scientific paper as example, which presents sophisticated statistical tests and  training to understand how it all works and how to interpret it.

Finally, to learn to meet students and professional in statistics is also a good learning for non statistics students. I remember, when I was in my thesis, that my meeting with a statistician was very helpful. After all, every man to his trade!

Eric, there will never be any progress towards better science if we'd follow your advise (although I well see its merits).

If teachers and students were trained in statistics, they would not even consider "tests" as something useful for their research.

"What teachers must teach is the art of using the right tests, the limits of significance of tests, incorrect interpretation of tests [...]"

Excuse me, what teacher is able to do this?

The first part is to explain when a hypothesis test and when a significance test is to be done. Then it will be understood that a significance test is nothing to be published but to help in judging if one should start a project on this. So the only "hard and objective" way to decide about a hypothesis is a hypothesis test. But then a whole bunch of problems starts: how a reasonable alpha and beta is chosen? Referring to "common practise" is nothing but an oath of disclosure. Where do we ever (except, for instance, in clinical trials) have sufficient background information to design an experiment so that a sensible hypothesis test is possibe? Wouldn't the students recognize that the procedure in infeasable, and that common work-arounds (to somehow make it feasable) are logically unsound?

But even if everything is in its correct place, it still has to be explained that tests are about the probability of the data. And this is the least interesting thing one wants to know (the data is there - no need to ask for its probability). We want to learn something about the hypothesis / the model. But how should this be possible without explaining the meaning of probability, and Bayesian logic, and large-sample theory and all this.

No, teaching students the "art of using the right tests" is like teaching a blind man the "art of using the right paint". My world is full of such "blind people" who can name any color and paint, but who will never be able to produce an artful painting. Those few who learned to "see" notice that they won't need any particular paint to be an artist.

Eddie, statistics may be introduced gradually to students. First teach them to create a measuring context as a data source - I agree in this point with Jochen-. Then, to estimate the average of small datasets. Then to estimate and graph the Lorenz curve, so they may conclude, for example, that half of the total mass is asigned to 25% of the receptors population. When they master calculus and sets theory, we may teach them that the derivate of the Lorenz curve is the Cumulative Distribution Function, so they may graph them from data and later from a good model of LC. This may be done at ages 12-16, and later students may estimate questions as "What´s is the probability of having a variable bigger than 0.5 medias?"  All this would create context gradually and statistical interest and criteria among them so they catch "the sense and utility of matter". I think that it is a great opportunity to gather the best of calculus and graphs, sets theory (boolean), measurement and randomness with simple activities like weighting, recording and ordering 20 books taken randomly from library, to be subject of simple statistical analysis graphs and gradual models according to their age and level. Thanks, emilio

Andrew, Eric, Jochen, and Emilio, thanks for sharing your varied but no doubt meaningful self-experienced opinions on this matter. I could be wrong, but I believe the views you shared somewhat leads to an openness towards statistics or mathematics crossing or blending with other disciplines, such as research, such that a statistician for instance, would have to understand the peculiarities of research methods so that statistics' application would be facilitative. 

Yes, Eddie, this is exactly the point. There is a "thechnical aspect" of statistics, the mathematical side, so to say. This has to be understood and developed by mathematical statisticians, and this is a science and huge field and with its on right to exist. One may see it as a side-product that tools, procedures and algorithms are produced that are (or can be) extremely useful for research workers / empirical scientists. I think we all supposed that this question was not related to teaching mathematical statistics but to teach statistics to students of some research field (of the empirical sciences).

Nowadays, students learn the theories related to their study subject, and they also learn how experiments are practically conducted. The data analysis is somehow "outsourced" to the "field of statistics" that is seen as a mere toolbox, often execrated, yet at best as necessary evil. Teaching statistics is completely separated from teaching the main subject. This way, students inevitably learn that statistics has nothing to do with science and research - it is only required to "put numbers in form".

This practice obscures a very important relation between statistics and (empirical) science. It is completely missed that science is about accumulation knowledge, and there is almost no student (or teacher) who ever even thought about what knowledge actually is, and how it can be altered by empirical data. Thinking about this one would recognize that empirial science and statistics are deeply interwoven, like speech is interwoven with verbal communication. I think this is a rather nice allegory. Children do not learn speech from theoretical lessons about alphabets, vowles, consonants, and orthography. They learn it through communication. It is not possible to separate it. However, in (university) curricula we actually and unfortunately do this (artificial) separation.

Jochen, your supposition "that this question was not related to teaching mathematical statistics but to teach statistics to students of some research field (of the empirical sciences)" is indeed the point of my question. The interweaving of statistics and empirical science as you have clearly elucidated, I believe is really a need. I just do not know if there is a necessity to "formalize" the solution- a subject blending "statistics" and empirical science, incorporated into to the curriculum. Thanks. Ed

  Thank you, all of you, for this discussion.  I think the point is that there IS a discussion-because clearly there is a problem. In fact, there are MANY problems, particularly with statistics used in small data sets. 

   I agree with Jochen : blending this with empirical science is 'a need.'   My own thought would be that we should devise other tests to understand hypotheses which have been formulated, incorporating 'if - then' and logical decision trees to double check for reliable conjectures.  Good scientists do this anyway.  It is a well known  that the inventor of the 'p-value'  was unhappy about 'the monster he had created.'

I agree Susan. Maybe we can suggest to the producers of Statistical Packages the inclusion of a "caution" on the use of tests.

Interesting discussion.

Reading the posts so far it seems to me that an important point is missing. The student. As a statistical educator I am regularly confronted with the problem, that the reasons to learn statistics are not obvious to beginners in some applied field (say psychology, as this is my field of expertise). Now the Problem is to motivate the students to be motivated beyond "learn this stuff or you wont pass the exam". Usually Students are motivated by the fields topics. So, from my point of view, it is a good starting point to teach statistics by using it to actually do inference on some interesting Data analysis question.

I feel that at that stage Students profit from nontechnicel treatments of the subject. Thus there may be a process of linking data analysis to relevant concepts (most of which may be presented without much technical treatment, e.g. with prepared simulations, see http://elder.uni-psych.gwdg.de:3838/acordes/multReg/ for an example on multiple Regression (in german)).

For many students, especially those who do not intend to become researchers, that level may be sufficient to empower them to understand the key methods of their field and thus to understand literature and have some general statistical literacy.

However, in my opinion the training of researchers needs more technical detail. Students who are entering science will usually find out that the statistical training they got is insufficient. They will need the ability to learn Methods from literature. Mathematics is the main language iin the relevant literature. Though it may be in principal possible to study such methods on a mere conceptuel - practical methods, it may be hard to find appropriate material. Consequently students need to be fammiliarized with technical details, to be able to read technical literature and to translate the symbolic information into concepts. In my view this level of detail may be backed by working with simulation methods and "stochastic experimentation".

However, in my view the motivation to learn the technical sides will depend on the perceived usefulness of this knowledge, which will probably be higher if it poses solutions to real world problems.

Making extensive use of software and teaching technical details may not  be mutually exklusive, but can go together well. Especially when using software that easily allows low level Data Manipilation (like R, Matlab but also Excel) one can merge software based data analysis with exploring technical details , e.g. by programming test procedures instead of using predefined functions, or stochastic experimentation using simulation methods.

Eddie -  FYI, the "Future of Statistical Software" was considered (link below) by the (US) National Academy of Sciences quite a few years ago. Perhaps it is time for an update. - I have no idea why "chemistry" is in the URL. But if this URL did not copy correctly, the report can be found by searching on the Internet.  - Jim 

I'll try that attachment one more time. 

http://202.117.122.45:85/dmtzy/yy/7/wwdzs/Chemistry/index11.pdf

I liked Andreas' comments above, but I'd like to add that personally, when I was first introduced to probability and statistics, I enjoyed knowing how a procedure worked, and had no interest in memorizing one.  Late in my career, it still greatly irritated me for someone to use the word "magic" with regard to my work. I wanted everyone, on at least some level, to understand the concepts. And the biggest problems I had at work were at least partly due to people using ill-conceived estimators that they did not understand, that were largely inappropriate, that they tried to fudge to get what they thought they wanted. Trying to change that was difficult, as it had a momentum that was amazingly powerful. I think that the more people understand concepts, the better. Unfortunately, that will probably never be enough to overcome office politics.  Tooany things are done for the wrong reasons, anyway.