A first point I would like to mention:

For linear and polynomial regression, the normal errors hypothesis is by no mean necessary to get a reasonable fit. It is, however, to get meaningful inference, i.e. coefficients ans standard errors. This means that if we are not particularly interested in interpreting the resulting model per se, non-normal errors are not a problem and least-squares can be seen as an optimization problem with a quadratic objective function (instead of the solution maximizing the likelihood).

Thus, although loess is based on least-squares polynomial regression, "non-parametric" refers to the fact that:

I believe the following paragraph from Wikipedia explains it very well:

https://en.wikipedia.org/wiki/Nonparametric_statistics#Definitions

Thank you @Gianluca Frasso for your response. As being a "Lead data scientist" with papers on P-Spline based approaches I would pretty much consider you as an expert ;)

Maybe another question: is there a good definition of "coefficients" vs. "parameters". I read something at StackOverflow, but that wasn't all too convincing. In a statistical model, there are factors to weight the effects of predictors (usually called "betas"). In a mathematical formula, as far as I know, these factors are called coefficients when they are constant, whereas variable factors are called parameters. Like:

f(x) = 3*x² - 2*x + 5: the values 3,-2 and 5 are coefficients

f(x) = ax² + bx + c: the values a, b and c are parameters

Now what are the betas in statistical models? To my unqualified opinion, these should be coefficients, because they are considered constant and fixed, only that we don't know their values. What is then the correct way to name these factors? In the given model, I would call them parameters, but whe it comes to fit a model, these are coefficients... or are the "betas" parameters and the "beta-hats" coefficients?

Terminology may vary from one field to another, but I believe "parameter" is a more generic term whereas "coefficient" refers to values directly interacting with the predictor variables.

In your example, a, b, and c could be called either. I would call 3, -2 and 5 the "parameter values" or "coefficient values".

An relevant example could be linear (for simplicity) splines. We define the a two-piece linear spline as follows:

f(x) = a*x + b if x = x_0

(With a, b, c and d such that a*x_0 + b = c*x_0 + d)

Here I would call x_0 (the knot position) a parameter because it does not directly interact with the data x. However, both part of the splines are defined by their respective coefficients a, b and c, d.

For your last point, betas usually denote the true parameters values from the population and are generally unknown. Beta-hats denote estimators of these true parameters i.e. value we can infer with the data at hand.

The Wikipedia page referenced above looks a good summary of present terminology regarding "(non-)parametric" and "distribution-free".

If one were to seek to group lo(w)ess methodology with other statistical tools, it would perhaps be useful to consider it as a smoothing method, to be considered as similar in what it targets to methods in:

(1) (time-series) spectral-density estimation;

(2) the distribution-free estimation of probability density functions;

(3) estimating the varying rate of occurrence in a point-process.

Here, the required outcome is essentially just a visual representation. Not only are the specific details of any underlying structure irrelevant, but also the visual appearance of outcome would not change much with small changes to those details

I don't get the term "distribution-free". Is that a misnomer, too?

Whenever we talk about probability (in whatever context), there is a probability distribution (except possibly for degenerate distributions with a point mass of 1 - but taht's surely a strange theoretical case). It may be that we do have a (parametric!) model to describe the form of the distribution. It may also be that we can not express the form with any known model and instead use some tables or other [heuristic] descriptions, in which case the description of the distribution is non-parametric. But I don't see how a statistical procedure that involves probabilities can be "distribution-free"?

"Distribution-free" seems entirely appropriate to the very basic types of rank-based tests, since the theory depends only on the properties of the ranks, not of the underlying data. The properties of ranks don't depend on the form of the underlying distribution. Whether such tests are "non-parametric" depends on what one means by a "parameter" here. Specifically, a basic rank-based test is effectively a test for the difference between medians. Is the median a "parameter". It is directly a parameter of a specific parametric family of distributions, but rather it is a descriptor applicable across all distributions.

Other (perhaps clearly) distribution-free methods are permutation tests and bootstrapping and jackknifing. Again ,there is no reliance on a parameter from a particular parametric family of distributions, but rather attention is centred on a "statistic" or comparison of "statistics" derived from the data, where these "statistics" may be considered as sample-versions of some general property that could be defined for any distribution.

So, possibly, there is confusion between "parameter" in the sense of a parameter within a particular family of distributions, and "parameter" in the sense of a generally applicable descriptor.

But rank sums, just for instance, do have a particular distribution under H0. It's only that we don't have a closed formula with parameters to express it. I attached a figure showing the probability (mass) distribution of the wilcoxon rank sum statistik for two samples of size n=4. The figure is generated with R and the probabilities are calculated with dwilcox(). The R help writes to this:

This distribution is obtained as follows. Let x and y be two random, independent samples of size m and n. Then the Wilcoxon rank sum statistic is the number of all pairs (xi, yj) for which yj is not greater than xi.

As the term suggests, "non-parametric" means the absence of parameters in the conditional mean and in the probability distribution corresponding to any random variables.

The term "semi-parametric" refers to the presence of parameters in either the conditional mean or in the probability distribution corresponding to any random variables, but not both.

All of this is discussed here:

https://www.amazon.com/Generalized-Additive-Monographs-Statistics-Probability/dp/0412343908/ref=sr_1_2?keywords=generalized+additive+models&qid=1571632661&s=books&sr=1-2

Best, D. Booth