First, I believe that the interaction between a fixed and a random effect will be a random effect.

Now I am not sure to understand exactly what you want to do, in particular because such a model sounds extremely complicated to me and you maybe should think carefully about whether you really need that (why do you want to reduce the residual variance?), and how you would interpret it. I'm just giving the basic principle. Maybe you can give more details later and we can refine.

Assuming that your initial model looked like that (I'm using an approximate lme4 syntax, without checking how to code the nesting, and it's also probably different from nlme):

lmer(x~1+temp+(1|Cont:Pop:Line))

you would probably have to change it to

lmer(x~1+temp+(temp|Cont:Pop:Line))  which estimates nested variances in intercepts, nested variances in the slope of temperatures and correlations between the two

or

lmer(x~1+temp+(temp-1|Cont:Pop:Line)) which should be equivalent to lmer(x~1+temp+(0+temp|Cont:Pop:Line))

which estimates only nested variances in the slope.

or

lmer(x~1+temp+(0+temp|Cont:Pop:Line) + (1|Cont:Pop:Line)) )

which estimates both the nested variances in the slope and in the intercept, but without their correlations.

Also, a related thread on r-sig-mixed models:

https://stat.ethz.ch/pipermail/r-sig-mixed-models/2010q2/003594.html

Good luck

https://stat.ethz.ch/pipermail/r-sig-mixed-models/2010q2/003594.html

Thank you for your help! 

So, first, I would have assumed it to be random too, but the lme per default takes it as fixed and I haven't found the proper syntax to make it random yet. 

Second, I don't really understand what the difference between the lmer from lme4 and lme from nlme is (but from what I have read there is one).

I am not very familiar with the lmer syntax, but I would use the last proposition (with out the correlation, because I don't exactly understand what it means). Intercept and slope are easier to understand alone.

So I was trying to answer the question of whether different populations or even continents are reacting stronger on different temperatures, I got a set of shape components of which I can (using this model) say how much is contributed by each level, and then maybe make connections of origin climate to reaction strength to temperature differences.

I am not particularly interested in reducing the residual variance, but because this is considered as the error, I assume that the biggest part of the variance explained by the interaction would come out of this error. (Is this assumption incorrect?)

Okay, that kind of make sense to me (although I would need to see more to be sure). Maybe you already did it, but I would also try and fit smaller models, maybe continent by continent..., to check that you big model is doing what you want.

I have tried it on a dummy example, the syntax for a simple random interaction works fine for me in nlme (it gives a random interaction):

lme(fixed = y~x, random=~x|id)

Could you show me the syntax you use with lme?

About the differences between lme and lmer, there are some for sure, but I don't think that they will matter so much for the model you are trying to fit, apart from the syntax and probably the speed.

Right, I also think that the random interactions should remove some residual variance.

(also, if you think these are useful answers, you can vote for them with the green arrow at the left bottom of this frame to increase their visibility)