Hi Olga,

those are good questions. From what I've come to understand there are really no hard and fast rules to help you choose predictors. There are some automatic procedures for predictor selection that you can try, but they really should be used with care. For a small data set, I think expert domain knowledge will get you far.

There are a couple of things you might want to try to figure out which interactions to include in your model (if you're not doing it already), however:

1) Try running a regression / classification tree (with all your predictors): they are good tools for spotting interactions among categorical predictors. In my dissertation (available from my profile), I discuss using the "rpart" package in R. I also really like the ctree() function from the "party" package in R. The trees produced by those functions are a nice way to see how the predictors interact with each other.

2) Keep checking the regression model goodness of fit, and especially pay close attention to residuals vs. fitted plots. With a binary response value I'm assuming you're using some kind of glm regression model, and for those models it's possible to create binned residual plots which will show how well (or badly) your model is fitting your data (again, check out my dissertation for some examples using a function from the "arm" package in R). You can also compare numerical model summaries (AIC, pseudo-R squared) but in my experience the binned residuals plots are very good when it comes to glm models for linguistic data.

Gard

What you describe suggests that your two predictors are (strongly) correlated, hence carrying almost the same information: adding one may then remove the significant of the other, because it becomes superfluous. It will also depends on the exact test you are carying out and, eventually, the order of your variables (if you use Type I sums of square-like tests for instance).

This correlation may be due to the nature of  the predictors (imagine one is the number of letters in the words and the second one is the number of syllabes...), or by « chance » and the results of the data (imagine one is sex, the other one is « smoking », and in the sample all women are non-smoking, all men are smoking...)

In both case, this suggests you have to choose which one is more important for the model (mostly on interpretation background) or think carefully about alternative ways to combine them or forget about model selection and care about prediction quality of the model or...

This is not necessarily related to interactions between them...

I'm not into linguistics, so can give specific advice on methods, just some general observation based on general statistical principles.

How many predictors are you trying out?

What validation steps are you following?

With a binomial response you have a very high risk of overfitting and what you are describing sounds like a classic case of overfitting. Do you do any data reduction methods to reduce the number of input variables while retaining maximal explanatory information? In datasets with properties similar to what you describe I would use a multi-layered validation procedure - start off with a subset of data (75-80%) doing cross validation to identify the most promising predictors, then a validation set (the remainder) to test the performance of those predictors. Ideally you would then test that model on a new set of collected data.

I'm not a statistician but I've worked with categorical data quite a bit. I always find looking at the descriptives helpful. Break down by categories, and then create a chart in excel so that you can visually look at all the categories. I find it helpful to understand what's going on.

In terms of determining what/which predictors to include, I find it best to select predictors based upon, in descending order of importantance, prior research resutls, theoretical rationale, clinical observation. If, as someone pointed out earlier, two of the predictors are correlated (easy enough to inspect a correlation matrix of predictor variables), choose the one that's most interpretable and make sure that one is entered first. If as a result, the second one is not significant, remove it from the final model

As to interactions, I thought that SPSS would now allow you to model interactions directly, but back in the day, we would create interaction effects by multiplying one variable by another. Once done, I'd make sure that the main effects (singleton variables) were entered first, then the interaction term(s); again those that have no predictive potency should be removed from the final model.

Hope this is helpful.

Additionnal remark about interactions: if you have numeric, quantitative predictors (like Age and Weight, for instance), if you add interactions between them, don't forget to add also squared terms (Age2, Weight2) --- all of this after centering, since in that case interaction means second order Taylor expansion, so should have the complete Taylor expansion...

Yes, Emmanuel is quite right. A significant squared (or cubed) term will indicate that the relationship between the predictor and the dependent variable is curvilinear rather than linear. My guess is that you don't have enough data to support an analysis with first order, interaction and quadratic terms. With small data sets, simpler is better. Use this data to gather baseline information, then build from there when you have a larger data set.