Dear Bin

I think if the independent variable is categorical, why u r not using logistic regression. Random sampling provide unbiased results, that is why one prefers. 

Thanks.

Regards

Thank you, dear Khandker and Rajiv!

The independent variable is not categorical, it is greenness of street scenes. 

Bin - Hi.  -  May it be taken then that your independent variable is expected to have a different relationship to the dependent variable in different ranges of the IV?  Perhaps you are using a spline, or just a linear regression in each range that differs in each group. Thus this is basically nonlinear?  In each group, you have data available, but more than you think you want to handle in one of them?  Would each range group be from a finite population, or theoretically infinite or superpopulation - which would impact accuracy considerations?  Measurement and other nonsampling error considerations are important.  -  So, you might want to discuss a few of these details.  -  -  As for R2, it is sometimes a good general gauge, but can not be considered too strictly.  Too few points can look better than they should, by chance, and Willett and Singer (The American Statistician) pointed out you cannot compare R2 between two different models directly. As for p-values, they are worse. They are sample size dependent.  Just changing your sample size may change your conclusion.  It is better to look at 'predictions' and/or other parameters and compare them to their standard errors to decide on their usefulness. Confidence intervals can be good, but you cannot always assume normality. There are formulas for using standard errors assuming distributional forms other than normality, and there is Chebyshev's inequality when you have no distributional assumption.  -  If you are using, for example, SAS PROC REG to run  linear regression in each of the three groups, you could look at the predicted values in each relevant case, and the standard error for each, which for SAS PROC REG would be the STDI.  That is the square root of the variance of the prediction error, found in say, Introduction to Econometrics by GS Maddala (latest edition with Lahiri). (More advanced than 'introductory.')   -  SAS PROC REG also gives other standard errors. I think that STDP just drops the residual part, as if that weren't important to your study, but I'd think usually it would be. Other software may give similar information.  -  Anyway, I suggest you look at standard errors, and perhaps confidence intervals, more than R2 or p-values. -  Thanks. -  Jim 

I agree with James, and I would like to emphasize his last point not to use R2 or p-values to make comparisons between models. Model fit metrics can be influenced by several factors, not just the ones you are interested in.

It might be more interesting to ask how the regression coefficients change for IV at low versus medium versus high levels. So how about a randomization test. Put all 140 samples into a giant pot. Randomly withdraw without replacement three sets of samples of size 75, 29, and 35. Calculate the regression coefficients for each DV. For each variable subtract the coefficient in low from medium, low from high, and medium from high. Store each difference in a separate file. Do this about 10,000 to 20,000 times. Then count the number of times you find a difference that is greater than your observed difference. Divide this by the number of randomizations to get your p-value.

Thanks, dear James and Timothy. 

I will run some analysis then come back here to discuss more. 

Dear James, Khandker, Rajiv, and Timothy,

The original relationship can be best described by a curvy line (quadratic or power line). Now I plan to take following steps to look into the relationship between IV and DV as IV is divided in low, medium, and high levels. I use IBM SPSS software.

1. Log10 transformation to make all variables has a normal distribution

2. Then run linear regression for each of three levels

3. Run bootstrap for linear regression (1000 bootstrap samples) and report p value, 95% confidence interval, standard error, and what else?

4. point out that relationship between IV and DV is becoming weak (p value from 0.001 to 0.3) as IV increases from low to high level.

Does this plan make sense to you?

You should spend some time asking if the log transformation is appropriate. The standard choices are log, square root, and inverse. They can be applied to IV, DV, or both. The choice can be determined using the data, or it may also be determined by a conceptual model.Test for normality?

Make a scatter plot. are the data linear?

Figure out why the p value becomes "weak." Is variance changing with IV, or is the relationship between IV and DV changing, or both?

Don't rely on 1000 bootstrap samples, or at least try rerunning the analysis several times and compare the outcomes from each analysis. If with 5 runs, your answer changes by a few thousandths of your mean, then all is probably ok (if your mean is 1000, then you are looking for differences of 5 or less. If your mean is 1 then you are looking for differences of 0.005 or less).

Dear Timothy,

Thanks! The scatter plot is linear for the whole group of samples after log-transformation. As I divide IV as three levels, the association becomes weaker and weaker, which matches our guess ( See attached file).

You mean run the bootstrap for a few times? How do I report findings if the results are different? 

I see variance in ParaPTC% decreasing as STC% increases.

I see STC%

Dear Timothy,

Thanks a lot! Do you have any reference articles (implication of methods, not theoretical or too deep) for this issue? 

The attached manuscript was my contribution to the literature. It is a fairly simple example. See page 762, top of right hand column. Also, Figure 2 looks fairly nice, but it does so only because I used a large number of randomizations. The very obvious shape to the data becomes nearly unrecognizable if you only use a few hundred randomizations. The raw data are in the appendix, so you can play with the numbers yourself. You could also make up fake data using a random number generator and start working with the method. It can be as simple or as complex as you have time and interest.

Article Randomization Tests: Example Using Morphological Differences...

Dear Timothy,

Appreciate so much!

Best wishes!

Timothy has nice observations on your scatterplots, Bin.

The only concern that I see remaining is that of deciding what is a 'significant' level for a p-value. That is why I like instead to see that each important regression coefficient is at least a few times larger than its standard error. Oh, and the heteroscedasticity Timothy noted should be accounted for. SAS PROC REG may or may not be one of the better options here, with its option to use regression weight as a formula, but I have only used it in the first quadrant where w = 1/x might be a good choice, for example (and without transformation).  As Timothy noted, the variance of y was getting smaller with increased x, not larger.

The scatterplots are a good idea. They can tell you a great deal.

If you someday want to read more on transformation and weighting in regression, Carroll and Ruppert have 

Strange that the reference was cut off. 

I was just noting that Carroll and Ruppert(1988) have a book, Transformation and Weighting in Regression, first published by Chapman & Hall, and later CRC Press, I think, that may be helpful for your future work. I read the whole book, and it has a good deal of useful information in a relatively small book, I think.  

Dear scholars,

Thanks for your great comments

The attached is a sample table. I used IBM SPSS 22.0, do you have any comments on it? I present many results but I am struggling how to compare results. 

So what do you think causes this relationship? Think of this only in terms of numbers, and forget the context for the moment.

1) Numbers have range: The values at the extreme ends of this range have a great influence on the regression.

2) Numbers have dispersion: This is variance, skewness, and kurtosis.

I can change any of these to get the results that you show. Once you figure out which attribute of the numbers is causing the observed relationship, then you can develop a test based on that cause.

What do you see if you take the range for "High" and select only those data in "low" that have the same span. So, if the range of data for "high" is 1.5 to 1.8 then try only using the data from low between -0.3 to 0 (assuming that there are five or more values in Low that fall in this range). Try a different location in Low. How often do you observe your result now that each level has the same range? If the answer is "not often" then you might be seeing a sampling artifact in your original data.

What do the residuals look like for each model? Do the residuals have the same mean and standard deviation for each level? What happens if you do a multiple comparison test using the residuals?

Dear Timothy, thanks a lot. The way I divide three categories are based on original data (equally as 0-15%, 15-30%, and >30%). after a log-transformation the ranges become unequal. 

Ok, so your experiment was designed for untransformed data, but it can only be analyzed using a transformation. What happens if you arcsine(square root(value)) transform your percentages rather than log transform them? It will make the categories more equal.

Is there something in the original data and methodology that forces you to analyze the data using these three categories?