This is a very brief explanation and there is more information in the links:
Parametric tests has an underlying normal distribution and they are based on central limit theorem. So we use parametric tests when we have continuous normally distributed variables or large samples, so we can assume normal distribution according to CL theorem. When we have categorical vars. or small samples and non-normal distributions, we apply non-parametric tests. In general, every common parametric test has its non-parametric equivalent. For example, Pearson corr. Spearman corr., Independent samples T-test Mann Whitney U test, Paired samples T-test Wilcoxon test, One way ANOVA Kruskall Wallis test, and so on...