Hi Hans,

There are quite a few ways to attack this problem. The most complex and informative one I know would be to use a differential equation which you believe this data follows and estimate the parameters using a nested style MCMC analysis. The easiest method, and perhaps most appropriate for what you are considering, would be to use a nonlinear mixed effects regression model nesting observations within individuals. I believe the "nlme" package in R can do this. These types of models can handle a bit of unbalance.

Vonesh, E., & Carter, R. (1992). Mixed-Effects Nonlinear Regression for Unbalanced Repeated Measures. Biometrics, 48(1), 1-17. doi:1. Retrieved from http://www.jstor.org/stable/2532734 doi:1

Consider building your foundation on Singer and Willett's classic text: https://www.amazon.com/Applied-Longitudinal-Data-Analysis-Occurrence/dp/0195152964. The multi-level growth model is quite flexible. The good folks at UCLA have worked out the accompanying R code: http://www.ats.ucla.edu/stat/examples/alda/

Thanks for the hints! That should help me to figure it out. One question which I am uncertain about though: In the comparison of generalized linear mixed models and marginal models (estimated by GEE), caution is always given that the effect estimates have different interpretations (effects in individuals vs. in the population as a whole), as opposed to the situation in continuous outcome linear models, where individual and population-average effects coincide.

But what is the situation in models of nonlinear relations between continuous variables, where the outcome is clustered (as described above)? Is it also necessary to differentiate between individual-specific and population-average effects in this case?

Other than once in a class, I have not used GEE, which places contrasting it and generalized linear mixed models outside of the topics I’m comfortable discussing in a public forum. But at least with mixed models, you may differentiate between group and individual-level effects to the extent that you care to. The same goes for nonlinear models.

For example, you could have a quadratic growth model for which the linear slope of time is random (i.e., differs across individuals) but the quadratic slope is fixed (i.e., the same for all individuals within the analysis). If you wanted, both could be fixed, both could be random, and so forth. The popular lme4 and nlme packages in R make all of this very easy to specify. And if you prefer Bayesian estimation, the brms and rstanarm packages are great for these kinds of models, too.

1. To deal with imbalance: there are R packages that oversample the rare responses, e.g., SMOTE, unbalanced, and so on. But this strategy is more prevalent among CS guys. Social science journal reviewer may question this practice. Or you can creatively regroup or categorize one variable to make it more balanced (for example, there are few less than 40 year-old people, then you categorize age_group1 ranging from 18 to 50, and age_group 2 higher than 50). How to categorize to lose least information and to get more balanced responses is important. 

2. To deal with nonlinearity. One method i(assuming you wish to stick with general linear modeling) is to transform the variable. For example, through plotting, you speculate the relationship between Y and X is nonlinear, specifically, Y= b + aX(squared). Then you can define Z= X(squared). Then you we can see the relationship between Y and Z (the transformed variable of X) is linear. 

Another method is to pick a model that accommodates nonlinear relationship, such as decision tree, SVM, and so on. These techniques are common in machine learning and CS.

3."Population average" estimates can be easily calculated by using margins and dydx commands in Stata.