Haiyan Hao , you say that its for validating your "method output". Yet, estimation is much, much better suited for validation than testing. Your method should make some prediction, and you can check how close the estimate is to that prediction.

When dealing with large sample sizes it might always be a good idea to randomly draw a small sample and perform calculation and exploration with this data. In a second step, you could verify patterns you found in the smaller sample with the original data.

You also raised the concern of the sample being to big and thus always creating significant correlation rendering information about correlation patterns useless. Keep in mind, that most correlation coefficients are meaningful in their interpretation as well and thus they can speak for themselves.

Additionally, when you consider my advice of drawing a random sample, you can also integrate your assumptions about reasonable corellation coefficient values by means of a power analysis and thus only draw a sample big enough to detect correlation patterns of a certain size.

Hope that helps,

Sven

Thanks, Sven! Thanks very much for your help.

However, I got this problem from a reviewer and he wants me to validate my method output quantitatively. In this case, I think using a portion of my result may possibly raise more questions. For example, they may ask me why you use this portion of data rather than that portion. Is there a method that can be used to get all my method output evaluated?

Also, I have an ordinal variable, i.e., level 1 and level 2. There are drastic differences between the number of samples for different levels. Do I need to apply some technique to balance the data samples in order to get more prudent results?

I presume that you know which is a independent variable (X) and which is a dependent variable (Y) in your data for which you wish to attempt the correlation. Yes, for large data, generally calculation of correlation is not attempted. However, your interest is to show the relationship between X and Y that you can still do it in a different fashion.

Divide the X data (inundated days) and Y data (Level of damage in the building) into 10 categories, each. Now, for varying class of X, see how the level of damage is changing.

For more clarity, let there are following 10 categories of X labelled as X1,X2,X3,...... X10 and X1 < X2 < X3 < ...

Testing does not make much sense for large samples. Ignoring or dividing data just toget smaller sub-samples on which testing might make sense (given this was the only data available) seems like a stupid detour to me (please correct me when I am wrong).

In large data sets, the emphasis can and should be placed on estimation. It should be possible to get estimates with reasonable precisions and to interpret these estimates in the scientific context.

Haiyan Hao, if I understand the analyses you want to perform, you have two kinds on your plate. First, it seems you want to relate the number of days that buildings have been inundated with the extent of damage (which I assume is measured on some kind of ordinal scale with a number of gradations on it if you want to work with correlation). In that case, I think you could simply calculate a correlation - probably Spearman's if you think you have measured extent of damage at a level that could be regarded as merely ordinal (not equal interval).

In calculating a correlation, it is indeed the case that it's easy to get statistical significance with small coefficients if the data set is large. However, it's important to regard the correlation coefficient as providing an indication of effect, and not rely just on statistical significance - or even take account of significance at all. In light of that, correlations of .1 indicate little relationship, of .30 indicate a medium relationship, and .5 a substantial relationship - and that's regardless of sample size. I suggest you google "correlation and effect size" to check that kind of thing out.

Second, it seems you want to see whether the buildings that you have categorized as not damaged or damaged have different numbers of days of inundation. I think you could perform a simple independent-samples t-test to investigate that.

I hope I've understood your question / situation clearly. If not, please forgive me.

Dear Haiyan Hao

According to the article, “Sample Size Guideline for Correlation Analysis” Sample size estimation for correlation analysis should be in line with the study objective. Researchers who are not statistician need simpler guideline to determine the sufficient sample size for correlation analysis. Therefore, this study aims to tabulate tables that show sample size calculation based on desired correlation coefficient, power and type 1 error (p-value) values.

https://core.ac.uk/download/pdf/268085221.pdf

Ramnath Takiar Thanks so much for the answer. Actually, I plotted a (violin) figure to show the distribution for different levels in my before revision. Are there any statistics that can be output together with this approach to show the statistical evidence?

Jochen Wilhelm Thanks so much for the answer. However, my purpose is not to discover new knowledge. It is for the validation of my method output...

Pervaiz Ahmad Thanks so much for your answer. However, I intend to validate my research outcome instead of discovering new knowledge.. So I need an approach to get all my samples (outputs) tested.

Haiyan Hao , you say that its for validating your "method output". Yet, estimation is much, much better suited for validation than testing. Your method should make some prediction, and you can check how close the estimate is to that prediction.

Jochen Wilhelm Yes! I can construct a model with my method output (i.e., number of inundated days) and use the model to predict the ground-truth results (building damage severity). If the model has good performance then my method output is also validated. Why I did not think of this approach! Thanks so much for this brilient idea.

Jochen Wilhelm I think I can also resolve the class imbalance (there are many more buildings without any damage) by applying weights to different classes. This solution does solve my problem.

I am going to reaffirm the validation of the model and what was proposed by Jochen

in the method of Neural Networks and AI, large amounts of data are required to validate the model. This is necessary, as data must be available to simulate new scenarios. If you take all your data into creating the model, how will you do it to prove it works? use bootstrapping to generate your test samples

As a tip, please observe the assumptions carefully, since when you have thousands of data, the problem may be heteroscedasticity, given the existence of latent variables.

For the Damaged and Undamaged Buildings model, you could use a probit or logit model

I wish you success!!!