In addition to a mathematical basis, there are philosophical foundations. A good statistician may know formulas and equations (mathematics), but there's also a need to know "why" they matter and how to interpret results in real-world contexts (philosophy)

One concept that seems to differentiate statistics from its parent discipline, mathematics, is that of randomization. A deterministic model may be considered a mathematical model, where a stochastic model is a probabilistic model. 

In the field of finite population sampling, there are two major approaches to inference, but both involve randomization. The probability-of-selection-based (design-based) sampling approach may be the more obvious case. However for the prediction-based (model-based) approach, randomization is found in the distribution(s) of residuals or estimated residuals.  In both cases, we have "random variables."

Also, similar to Daniel's mention of "why" above, when one samples from a finite population, Ken Brewer once told me that his mentor, Ken Foreman, said that there is no substitute for knowing your data. Currently there is an increased need for modeling, and that would be very true for attempting to see that each model is only applied to data for which it is appropriate, but they dealt a great deal with probability (design-based) sampling. It occurs to me that you need to know your data to even properly stratify your population.

Randomization is also the key to clinical experiments and other applications.

So, as indicated earlier, I'd say that randomization is a key concept which identifies a statistical application from a "merely" mathematical one.  🙂