There is a assumption all predictor's are  independent in a regression. Under assumption, the specific conclusion on the effect of individual factors from a multiple regression equation depends on whether the predictor is correlated with other predictors . The coefficient of a predictor measures the change of the response variable when the predictor changes by one unit while other predictors are held constant. When one or more predictors are removed from the equation, the coefficient of the remaining predictors should not change. However,

in practice, the interpretation may not be valid because predictors are often correlated making it difficult to change one while holding the others constant. In addition, if a strong association exists among predictors, referred to as collinearity, the interpretation of regression coefficients becomes unreliable. When collinearity exists, even minor changes such as removing or adding a predictor or deleting a few records may lead to unpredictable changes in the estimated coefficients, or even changes in sign or an increase in the estimated standard errors.

Hence you need check the collinearity among the markers. If all markers are continuous variable, you may use principal component analysis, if an eigenvalue is near zero, those makers corresponding are related,

PCA, ridge regression, etc. can handle collinearity data analysis, You may separated the markers into different groups, such that all marker in the same group are roughly independent, to pick up important markers. For logistic regression with continuous maker predictor, you may see:

Biomarker identification and effect estimation on schizophrenia – a high dimensional data analysis,  Front. Public Health, 05 May 2015 | http://dx.doi.org/10.3389/fpubh.2015.00075

Dear Ye,

The problem with your data is multi-collinearity. Transformation or orthogonalization of correlated variables could be a solution. 

Agreed.  Your independent variables are correlated.  I recommend determining which variables are correlated to which other ones.  And then using a logical approach for determining which variables to include in your model.  You might use AIC, BIC, or a stepwise approach adding one variable at a time and looking at the p-value for each term.  The best approach is to look at which variables are correlated and use logic or theory to include those that are most meaningful.

How many variables are there in your model? How many observations are there in your data? Do you perform separate logistic regressions one for each variable beforehand? What is the ratio of events (1's) to non-events (0's) in your dependent variable?

I have 42 observations and 5 variables, 4 categorical and one continuous. The ratio ov events to non-events is 1,8. And yes, I have performed separate logistic regressions for each variable, and coefficients differ comparing to the model when I use them together at the same time.

Your sample size is rather small. It would be better to increase your sample size. Another good alternative may be trying Lasso.

On the other hand do not expect that a significant variable will preserve its significance when modeled with other variables. There is no such rule. You might search for concept of confounding.

I agree with Mehmet. Too few observations to use PCA.... and to rely on asymptotical properties of estimators of your logistic regression. Reducing the number of regressors (solving muticollinearity) could help together with increasing your sample size.

A very good handbook is https://www.crcpress.com/Modelling-Binary-Data-Second-Edition/Collett/p/book/9781584883241

Agree that sample size is too small to allow for more than an exploratory analysis. Any findings will be quite uncertain, even if a sensible technique for multivariable model selection such as Lasso is used.

Furthermore I was puzzled by the remark '65-75% level of correct prediction'. First, the quality of predictions can be quantified in various ways; accuracy (% correct classifications) is a poor measure. Better are measures such as the area under the ROC curve (or c statistic), or overall measures such as Brier score or R^2 based on log likelihood. Second, given the small sample size, no attempt should be made to quantify performance. Some insight on promising genetic and epigenetic markers of cancer is the most one can get from this data I suppose.

Ye,

Since 4 predictors are categorical, you can't use PCA,. You may consider latent analysis. The sample size is small, you should not have too many parameters in the regression. The total number of parameters in regression is counted as:

if the outcome is binary: say yes or no

Interception: count 1

Continuous factor: 1

Categorical: if it has k levels, count k-1,

say race: white, black, Spanish and other, count 3, i.e you need 3 parameters in the regression to describe the Race Effect (difference from the reference group).

In general, the sample size should be 10 times of the total number of parameters, at least 5 times.

Definitely, you don't have enough sample size to use all predictors.

If the outcome has m level, multinomial outcome, the total number of parameters needs multiply the above by m-1.

Using more parameters, it will increase the regression standard error, hence it reduces significance.

Hence,

1. You need study the categorical factor with multi levels, to see if you can combine some levels together.

2. After combining, do contingency table analysis for those categorical factors, if you see a lot of zero counts, it means those are correlated, you can't use them in the same model. You can do it by pairs of the factors. Or do decomposition for three or more ways contingency table analysis,  

3. after select the categorical factors, do contingency table analysis for them and outcome. If you see zero cell or near zero cell, (1 ,2,

42 observations are to few to use the method of logistic regression. At most, you have 21 events (if there is a 50-50 in your dependent variable). This means that even two predictors is pushing it. First you need at least 10-20 events per variable (EVP) (i.e. no less than 40 positive AND 40 negative observations), then you must still figure out colinearities. After that you may model.