If assumptions are reasonably satisfied with your data set, there is nothing to prevent you from running your anova to compare groups. However, do be aware that the power to detect group differences will not track the mean group size, but instead will track more closely the harmonic mean of the group sizes (which usually implies lower than expected statistical power).
If assumptions (specifically, homogeneity of variance) aren't satisfied and you can't ameliorate the situation via data transformation, just know that ordinary anova is likely to yield a biased F-ratio. Whether the bias is positive or negative depends on the sample variance of the larger vs. the smaller size groups in your data set. There are some well-known adjustments (like the Welch_Aspin) to correct for this.