LET’S START WITH SIMPLE DEFINITIONS:

• Machine learning is an algorithm that can learn from data without relying on rules-based programming.
• Statistical modeling is the formalization of relationships between variables in the form of mathematical equations.

The mathematical fundamentals of Statistics and Machine Learning are mostly similar.

Statistics is more traditional, fixed, and was not originally designed to have self-improving models. It is more academically formal and meticulous as a field, and uses smaller amounts of data. Statistics focuses on understanding the data in terms of models. Example: interpretability, hypothesis testing.

Whereas, Machine Learning focuses on prediction and the analysis of learning algorithms. Machine learning models need more data than statistical models to perform well. It emphasizes on optimization and performance over inference (which is what statistics is concerned about).

However, both machine learning and statistics have the same objective-

According to Larry Wasserman:

They are both concerned with the same question: how do we learn from data?

Optimization is used in machine learning during the training phase. Parameters are chosen so as to optimize some function that combines error rate with, in some cases, a term that acts as a penalty for overfitting.