Just run Your regression and look at the attached papers and the references within. The regression tells you all you want to know if you look at it carefully. David Booth

David Eugene Booth I can use the G statistics as well as z-test(Wald test) to check for significance for each and every independent categorical variable. But that would be a bit time consuming in a case where there are 20-30 independent variables. Is there a short way (like heatmap when considering numerical variables) which I can use?

Also I tried using Chi square test for testing the correlation between independent variable "JobType"(8 levels) and the dependent variable"SalStat"(2 levels) and the p value came out to be 0.99. So according to Null Hypothesis there is no correlation between them and hence can be discarded. But that is obviously wrong. When thinking about it normally, obviously the job you do affects the salary you get.

Well if you model is predictive you could possibly do something like the methods in the attached in the attachment for variable selection perhaps. I don't understand your chi-square tests. Best wishes, David Booth

Make a contingency table (frequency) and then use the Chi-square test.

Contingency table:

#preprocessing: text to categorical variable categorical_var_1

base.loc[base['categorical_var_1'] == 0, 'categorical_var_1'] = "Text transform 1"

base.loc[base['categorical_var_1'] == 1, 'categorical_var_1'] = "Text transform 2"

base.loc[base['categorical_var_1'] == 2, 'categorical_var_1'] = "Text transform 3"

data_crosstab = pd.crosstab(base['categorical_var_2'],

base['categorical_var_1'],

margins = False)

print(data_crosstab)

Chi-square test:

# chi-squared test with similar proportions

from scipy.stats import chi2_contingency

from scipy.stats import chi2

# contingency table

table = data_crosstab

print(table)

stat, p, dof, expected = chi2_contingency(table)

print('dof=%d' % dof)

print(expected)

# interpret test-statistic

prob = 0.95

critical = chi2.ppf(prob, dof)

print("")

print('critical-value probability=%.3f, critical=%.3f, stat=%.3f' % (prob, critical, stat))

if abs(stat) >= critical:

print('Dependent (reject H0)')

else:

print('Independent (fail to reject H0)')

print("")

# interpret p-value

alpha = 1.0 - prob

print('p-value significance=%.3f, p=%.3f' % (alpha, p))

if p