In addition to Ofoghi's answer:

As you must know, C is the penalty hyperparameter of the error term in the training process.

With a high C value, the algorithm will try very hard to reduce the error in the training phase and, to achieve this, a complex surface (hyperplane) will be formed to separate the classes.

If this surface is too complex (i.e. very high C), it's going to be shaped to cover each pattern of the training data, even noise. In this case, the classifier is likely have a very good performance on the training data and lose in terms of generalization, resulting in a poor performance on the test set. This is a phenomena known as overfitting, as you mentioned.

On the other hand, if this hyperplane is too simple, it may not recognize important patterns of the data, leading to high train and test errors (i.e. underfitting occurs)

Let me know if something isn't clear enought.

Last doubt to have clear ideas. I get what is the overfitting of the classifier and i understood what the hyper parameter is. However i cannot explain why on the training set the error decrease while on the test error increase. What is the behaviour of the classifier on the two sets. For example i have a Gaussian Mixture Model classifier if i increase the number K of Multivariate Distributions then even the ''noise points'' will have a proper distribution, i'm wrong? But why on the training set the error rate more or less still descrease while on the testing set increase. How does K affect the overfitting? I'm sorry for the number of questions but i will like to learn more this relation. Thank you in advance for your understanding.

I'm not aware of GMM classifiers and its hyperparameter K, but let me talk in general terms so you can come to your own conclusions about the specific case.

Imagine a scenario: a classifier is being trained, so it can optimize its internal parameters (support vectors in SVM, weights in ANN, etc). In the task of training, the objective of the optimization is to decrease the error observed in the data (training data). When you adjust these internal parameters, the generalization error (error on data not seen in the training phase) will decrease together with the training error, but not forever. If this process continues, the training error will still be optimized, reaching the point where the parameters of the classifier are so specific about decreasing the training error that the peculiarities of data that were taken into account in the training process are no longer general patterns of the data as a whole. This is the point where generalization (test) error begins to increase, because the classifier now is just too specific to that piece of data it was trained on.

This is the phenomena of overfitting in the first level of inference, when adjusting the internal parameters of a classifier. It has been shown that another form of overfitting can occour, the overfitting at the second level of inference, i.e. overfitting in model selection.

"I get what is the overfitting of the classifier and i understood what the hyper parameter is". I don't think you do. Victor Costa's answers are excellent, let me just put them another way:

1) The fact that the training error decreases while the validation error increases is universally known in machine learning as the symptom of overfitting.

2) Regularization is universally known in machine learning as a the remedy to overfitting.

3) From the definition of C for SVMs, the larger the value of C, the smaller the regularization.

4) Therefore, the larger the value of C, the stronger the overfitting, which is exactly what you observe.

Yes, when C increases SVM over fits to the training data. C is affecting the regularization term. When C increases that means it does not penalize theta parameters. So, over fitting occurs. it the inverse of lambda in logistic regression