You somehow seem to confuse homoscedasticity with the fitted values. Clusters of the fitted values only mean that not all fitted values are equally frequent. This happens when the model function has segments that are more "horizontal" (parallel to the predictor dimensions) then other segments. For these "flat" segments, all fitted values are very similar, leading to a cluster in the fittes vs. residual plot (in your case it should be the interaction of continuous predictors allowing for such a "flat" region in the response surface). Homosecdasticity is a feature of the residuals. The variance of the residuals should be similar, independent of anything else, including the fitted values. This means that the scattering in the vertical direction in the residuals vs. fitted plot should be similar along the horizontal direction. It is irrelevant if and where the values on the horizontal direction cluster.

In your plot I don't see a severe violation of the assumption of homoscedasticity.

The QQ plot shows that there is in fact some more or less considerable information in the data that your model is not aware of. Hard to say if this unused information is relevant to your problem.

I would make (many different) plots of the fitted function together with the observed data to get an impression how the model looks like, if it is reasonable and where it has problems to describe the data well. Added variable plots can also be helpful here.

Possible remedies are: i) adding interactions, ii) modelling non-linear relationships.

i) You could try to fit a model with at least all pair-wise interactions

~ (bout_progress + bout_length + WAMP_t + numbr_chucks)^2

or even with all triple-interactions

~ (bout_progress + bout_length + WAMP_t + numbr_chucks)^3

to see if this may be due to a missed interaction.

ii) If variables are included "just" to adjust the effects of other variables, you may consider to use splines rather than linear relationships for these variables (include package splines, then use ... + ns(x, df=3) + ... to include a nonlinear relationship of the variable x in your model. "df" is the number of degrees of freedom of the spline and has a similar meaning like the order of a polynomial (like poly(x,3)), but splines are more flexible (e.g. a spline with 3 d.f. does not need to be symmetric, but a cubic polynomial must be symmetric).

Hi Henning Teickner

yes, seems I could have saved time if I'd seen this question a few minutes later :)

You mentioned that the response might be a count variable, in which case a negative binomial model (Poisson is almost always way too restrictive for biological data!) would be appropriate. I agree. Although the amount of data (some hundreds of values, given only 6 df of the model plus one random intercept term) is likely to be enough to render the differenc to a gaussian model negligible (even more so when the counts are "large" and if there are not many counts close to zero).

Luke Larter ,

Going throug one more, I think that using MALE_NUM as a random intercept is not a very good idea. This is also a count variable (I suppose), so it's NOT a factor (=categorical variable), and random effects are defined over factors (R will interpret the numbers as the levels of factor here).

If the response variable C1AMP_t is a count variable (as Henning mentioned), I think it would be better to include MALE_NUM in a negative binomial model as offset, to model the proportion of calls, relative to the number of males than can call (otherwise one could model the log-ratio). This assumes that the calls were recorded not for individual frogs but for the observed group of MALE_NUM frogs in the respective habitat (if there are several groups observen in one habitat, one might use habitat as a randpom factor!). What do you think, Henning?

Jochen Wilhelm Henning Teickner

Thank you both for your thorough and insightful feedback. You have both given me a lot to think about in moving forward with my analysis! I will try what you have suggested.

C1AMP (the dependent variable) is actually the amplitude (loudness) of a certain part of the call, and so it is a continuous positive value, and the 'MALE_NUM' variable which I included as a random effect is just a unique identification number for each individual frog sampled.

Thank you for the feedback. If you have several measurements for an individual amimal (labelled with MALE_NUM), your approach with the random effect is absolutely ok and my idea is nonsense (because I assumed a wrong meaning of the variables).

Given the response is an amplitude, which is a strictly positive variable, I would intuitively think of modelling proportional effects, either by using the logarithms or by using a GLM of the gamma family with log-link.

Henning Teickner

Redundancy in answers can be re-assuring that the idea/solution is okayish (or even good), but our sample size is way too small to put any confidence in this conclusion ;)

Jochen Wilhelm Henning Teickner

Thank you both again, all of your comments have been exceedingly helpful!