You can do cross validation to determine the optimal number of ensembles for the given problem.  For the imbalance problem, random under sampling of negative samples would work fairly well.

As said by Kwon, using a cross-validation technique (e.g., stratified 10-fold CV)  seems the best one can do in such a situation.  For the skew in class distribution, I suggest applying an over-sampling method (i.e., adding new synthetic cases instead of eliminating real ones) via the Synthetic Minority Over-sampling Technique (SMOTE), which was introduced by Chawla et al. (2002).

N. Chawla, K. Bowyer, L. Hall, and W. Kegelmeyer. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321–357, 2002.

Hi Andreau and Junghyun. Thank for your answer!

I have done a lot of trials with oversampling vs. undersampling and I also ran into Chawla et al. I did however not implement the approach in my code. I will have a second look at it. I am doing text mining with alot of variables so I need to keep computation times in mind, which speaks for the undersampling solution.

The solution I have come up with for now is to under sample the majority class, and apply the full set of positive cases on the undersampled negative cases. Example:

Training set size: 3000 cases

Positive: cases: 100

Negative cases: 2900

My procedure then does the following with replacement:

   For each ensemble(

      100 random negative cases with replacement vs. 100 positive cases

       Predict on a validation set

       )

Given this procedure, how can I then optimally decide on number of ensembles and also number of positive and negative cases that goes into each ensemble...?

Hope it makes sense...

Your approach looks similar to:

X.-Y. Liu, J. Wu, and Z.-H. Zhou. Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man and Cybernetics - Part B: Cybernetics, 2009, 39(2): 539-550.