You could transform this quantitative variables into a qualitative one: An ordered factor with as first levels : "No user, or NA", and some other levels depending on the distribution of your variable.

Hope this help

cheers

Pierre

That could work. At least in terms of a correlation I could use Spearman. But, if I were to include the new variable in linear regression I would have to create dummy variables for each on of the levels. Problem is that they are many.

You could have 4 levels, the quartiles of your quantitative variable (e.g. using cut() ) and the 5th levels for NA.

I don't know what is your aim., but in your regression model, use this factor as a factor.

Yes! Great idea about using cut. Worked like a charm. Thank so much!

Zero is also a data. If you are using a non parametric test, the test itself would transform your data to categories.

Looks like you have some good answers already. I think, however, that to answer you question properly, it may be necessary to know what question it is that you are wanting to answer with your data? Anyway, if you are working with proportions or ratios, be careful with your regression; a linear regression may not be a valid option,

If you have percentages, then you have a binomial problem, which you could model directly by logistic regression. The dependent variable would then become a compound of two numbers, the number of trials (total users) and number of successes (registered users). Both can be zero.

Here is an example in R:

totalusers

I think this last suggestion is getting close to the appropriate way of dealing with these tyupes of data, rather than degrading them to some ordered factor or similar. It may not get around the problem of excessive numbers of zeros though. There are various ways of dealing with this (all available in R or other software with R interfaces). Zero-inflated models can explicitly model this over-representation in the overall model, giving you an actual estimate of this over-representation. For instance, there are various methods in R or linked with R that allows you to fit zero-inflated binomial distributions to your data. You can for instance search for 'zibinom' in your "favourite" R web search engine.

@Biuw: Yes, although this only is an issue when the number of total users is the dependent variable. We don't know. But, valid point! When dealing with count data one must carefully check the residual distribution. Besides excesses in zeroes, overdispersion is a frequent issue. If uncorrected, Type I error is inflated.

To answer your first question 'is this accurate'? Yes, if you have zero registered users and divide that by any number it is always zero.

1) 0 describes a condition - yes this is a factor, you can call it whatever you like, just as long as R is ok with it.

2) do not use NA for zeros, as people have said zero is meaningful.

3) negative numbers are ok in regression, especially for explanatory variables and are common when using a 'normalised' scale.

I'm not sure from your question what your response variable is. Is it a count? a precentage? or a probability? I guess your metric is a continuous variable going from 0 to 100 percent or 0 to 1 in a proportional sense. There are important properties here, it can't be less than zero or greater than 100, so you should not apply methods based on gaussian distribution!

However, it sounds like a GLM with a binomial distribution. You may or may not have zero inflation, I don't think there are hard and fast rules about this. If you fit a model with this distribution you can get over dispersion and this needs checking, the high number of zeros may contribute to this. I think dividing the residual deviance by the degrees of freedom is one way of checking this (not 100% sure though) if it's >1.5 then you have some overdispersion. Plotting residuals against each of the explanatory variables in the model and the ones you chose not to put in, if there are any, should help you track down the cause of overdispersion.

good luck

Hi all, the answers proposed by both Martin (Biuw and Schmettow) just above seem to me the "way to go"... in the past and previously to using the more "statistically elaborated" (and complete) generalized additive models and GLM to the full data set (in the way described above), I approached an equivalent problem in a two-step "procedure": 1st modelled presence/absence (0/1) considering a Poisson distrib and then just the cases where data existed considering a binomial distr/logistic regression.

Thank you all for your answers.

I should have clarified. My dependent variable is a continuous variable. The metric described in my problem is my explanatory variable. Converting it into e.g., 5 level factor works in lm model.

In my case the percentage metric does not linearly correlated with my outcome variable but forms somewhat the shape of a normal distribution. Converting it into a 5 level factor allows me to include it in the model even though it does not correlate linearly.

I still think the binning approach degrades your data and makes your test less powerful. The zero-inflated approach to these problems can actually be extended to proportion data (as opposed to true binomial data). One relatively simple approach is something along the lines of what Eduardo Esteves writes. Do a two-step model where the first model is a logistic GLM of zero vs. non-zero values (i.e. tests for the probability of obtaining a zero). Then, to model the spread of all non-zero proportions, you can either fit a GLM with quasibinomial family (logit link), or you could simply logit transform your proportions and run a simple lm.

....and to account for the curvilinear relationship you can even include a polynomial term in your models.... but if the simple approach of binning gives you what you're looking for I guess it's OK too :o)

@Martin can you provide an example of the two step model you describe in R. My only experience with two-step models is using heckman (heckit function in R).

Now that I read your original post a bit better, your original data are actually binomial (as Martin Schmettow already pointed out in his first post), but the question really is what your zero users really mean. In your case, I would think a two-stage modelling approach, using consecutive GLMs, is sufficient. Say your data are stored in a data rame called 'data', and you have variables 'totusers' and 'regusers':

Model 1)

data$users 0 ## Create binary users/no users variable

mod1

Martin,

if I well understood, your dependant variable "y" is continuous (gaussian?)

one of your independant varialbe is the ratio regusers/totusers.

Hence, the 2 step approach would be:

1/ anova for comparing y between presence and abscence of user

userPA=ifelse(totusers>0,TRUE,FALSE)

lm(y~userPA)

2/ When user are present, linear, or non-linear (see nls) regression

lm(y~I(regusers/totusers),data=subset(yourdata,totusers>0))

about nls, you can choose functions like asymptote or sigmoid. This might be suitable if for instance, change in "y" accelerate from a threshold value of the ratio.

Note that depending on the distribution of the continuous dependant variable, you could have to fit glm/gnls instead of lm/nls, or to consider robust or resampling approaches.

I hope I understood you problem,

Cheers,

Pierre

@Martin thank you for the answer. In the models you gave here, the user variables are the predicted. In my case however this variable is the predictor. For example:

lm(dollarsspent ~ user_percentage_metric, data=data)

I forgot something. As I red above, weighting samples (with totusers) in regression could be suitable.

Another approach for "exploring" the putative effect of totuser could be to fit a multiple regression model.

In your case, you could fit a smoothed 3D surface (x=I(regusers/totusers),y=totusers,z="y dependent variable") using gam.

You could detect putative totuser effect and its interaction with regusers/totusers.

Of course, relevance of all of this depends on your hypotheses, which I don't know.

cheers

Michael,

thank you for clarifying that the percentage is your predictor. Predictors in linear modelling can have any shape you want. You can safely forget about binomials, zero inflation, over-dispersion, and all that stuff. You may still consider using total users and registered users as separate predictors. Also test the interaction.

But, without knowing more about your data and your research questions this is all highly speculative.

--Martin

P.S. Would someone be so nice ...

Hi Michael

As Martin and I have pointed out, unless you provide more detail of what it is you want to know, everyone will be trying to second-guess you. As you can probably see by now this is a slow way of going about finding the "best" answer. This comment is not meant as a criticism, and I am in admiration of all the help and sensible contributions. Bo

As a general example(sort of fictional) to give you an idea. I am expecting that the number of unregistered users leaving feedback contributes positively or negatively on the overall dollars spent on a website. So a table that looks something like this:

Item dollars registered total

item1 $10 10 20

Item2 $20 15 30

...

The interaction, or directly the variables registered and total have almost no effect (to total is slightly significant).

Call:

lm(formula = dataset$dollars ~ dataset$total +

dataset$totalbyreg + I(dataset$tota *

dataset$totalbyreg))

Residuals:

Min 1Q Median 3Q Max

-2.7187 -0.7888 -0.7586 0.4068 5.2272

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) 7.586e-01 4.203e-02 18.052 < 2e-16 ***

dataset$total -1.923e-03 9.522e-04 -2.019 0.0437 *

dataset$totalbyreg 9.951e-03 1.231e-03 8.081 1.72e-15 ***

I(dataset$total * dataset$totalbyreg) -3.351e-06 4.754e-07 -7.049 3.21e-12 ***

---

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.214 on 1074 degrees of freedom

Multiple R-squared: 0.1703, Adjusted R-squared: 0.168

F-statistic: 73.48 on 3 and 1074 DF, p-value: < 2.2e-16

However, when I input the percentage metric, converted to a factor that incorporates the 0s then I get this

Call:

lm(formula = dataset$dollars ~ as.factor(tempvalue) + dataset$total +

dataset$totalbyreg + I(dataset$total *

dataset$totalbyreg))

Residuals:

Min 1Q Median 3Q Max

-2.4409 -0.3376 -0.0067 0.1946 4.7580

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) -9.567e-03 5.744e-02 -0.167 0.867757

as.factor(tempvalue)(-0.00075,0.25] 9.603e-01 2.656e-01 3.616 0.000313 ***

as.factor(tempvalue)(0.25,0.5] 1.773e+00 2.010e-01 8.822 < 2e-16 ***

as.factor(tempvalue)(0.5,0.751] 1.690e+00 1.768e-01 9.558 < 2e-16 ***

as.factor(tempvalue)(0.751,1] 1.245e+00 7.471e-02 16.658 < 2e-16 ***

dataset$total -3.842e-03 9.020e-04 -4.260 2.23e-05 ***

dataset$totalbyreg 9.272e-03 1.141e-03 8.123 1.24e-15 ***

I(dataset$total * dataset$totalbyreg) -1.770e-06 4.274e-07 -4.141 3.73e-05 ***

---

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.067 on 1070 degrees of freedom

Multiple R-squared: 0.3619, Adjusted R-squared: 0.3577

F-statistic: 86.7 on 7 and 1070 DF, p-value: < 2.2e-16

If I remove the other variables and keep just the tempvalue to the adjusted r-squared remains more or less the same.

PS: the missing level between -0.25--0.00075 is ommitted automatically probably because the coefficient is 0. All pages that had 0 users where assigned -0.25 score.

Michael,

do not include these predictors at the same time. It is very likely, that they are strongly correlated, imposing the risk of colinearity. How about using as predictors the total number of users and the percentage, instead?

I have the intuition that you should just exclude the cases where the total number of users is zero. They simply do not contain any information on the users' tendency to register.

--Martin

I agree its better to remove most and I'll test for multicolinearity later on. The interesting part is that while tempvalue as numerical does not make any sense to keep due to the fact that it is not linear (its not even significant), but as a factor it is incorporated well and contributes to the model significantly judging by the coefficients.

Call:

lm(formula = dataset$dollars ~ as.factor(tempvalue) + dataset$total)

Residuals:

Min 1Q Median 3Q Max

-3.1422 -0.3870 -0.0054 0.3850 4.6537

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) -0.0064032 0.0591536 -0.108 0.913819

as.factor(tempvalue)(-0.00075,0.25] 0.9198357 0.2733522 3.365 0.000792 ***

as.factor(tempvalue)(0.25,0.5] 1.7414962 0.2064923 8.434 < 2e-16 ***

as.factor(tempvalue)(0.5,0.751] 1.4218767 0.1738424 8.179 8.03e-16 ***

as.factor(tempvalue)(0.751,1] 1.3468405 0.0751906 17.912 < 2e-16 ***

dataset$total 0.0019622 0.0002538 7.730 2.46e-14 ***

---

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.1 on 1072 degrees of freedom

Multiple R-squared: 0.3206, Adjusted R-squared: 0.3175

F-statistic: 101.2 on 5 and 1072 DF, p-value: < 2.2e-16