Hi all, I've got 100 realizations for 100 years of a climate model data (daily values) which I intend to use for water risk assessment and want to draw some logical conclusions. Could anyone suggest which statistical tools are the most suited to analyse such huge data set?
You can quantify risk changes using R packages. SAS is also good for large datasets. One challenge you will face is combining horizontal observation with the time dimension. You can try quantifying within year variability and see how this is changing over time. There are also some non-parametric tests could give good insights. Recommend doing some preliminary analysis on the nature of the dataset using simple analyses of variability within a year and over 100 years.
A basic resource is http://www.iwr.usace.army.mil/Portals/70/docs/iwrreports/92r1.pdf
Hey Vinod,
I suggest using a time series approach with clear a priori hypotheses to test. I also suggest using R for this (see the link) and the key is to very clearly define your models for decreasing trend, increasing trend, or stasis. You can define multiple levels for each or simply have a threshold for the increasing or decreasing risk models. Stasis can have it's own thresholds on both sides (like a two-sided t-test) or you can simply document the slope if you want to go that way. Basically, the key with time-series is to set your hypotheses before you run the tests and not data mine to look for a trend you 'want' to see. Hope this helps!
https://a-little-book-of-r-for-time-series.readthedocs.io/en/latest/
Dear Vinod,
In general, R programming is a good tool for analysing complex data and relationship between independent and dependent variables. However, if you have already formulated the model, then you can input your model in to MATLAB software to test a preliminary set of data to see whether or not it works .
Dear,
may be check mintabe software , it s good for time series data analysis
I am quite confused about the question. What you wish to know - software package or statistical method for analysis. Different software packages have inbuilt modules that use variety of statistical methods to analyse the long term data. Mann-Kendall and Sens Slope are the tow statistical techniques that can help in this. These are available in a excel based program named as MAKESENS. 'Excell-stat' can be other better option.
Thanks everyone for valuable suggestions. Though I'm a novice about R package, however, I've a fair bit of knowledge on Mann-Kendall and Sens slope. Santosh and Ken, I do not know why you confused about the data set as it is clearly mentioned that there are 100 sets of daily data (e.g evaporation) for 100 years (GCM's down scaled data) which i need to analyse for some logical conclusions. It doesn't matter whether it is a software of a statistical test or excel spread sheet. Mann-Kendall and Sens slope only provides one way variability in a time series either can be used to examine annual data for each realization or daily/monthly variability in one realization. Probably, I should try R package which can involve multivariate variability at the same time. Thanks Tim, David and Hadi for valuable suggestion and providing useful link for R package.
It is a bit confusing, indeed...maybe what you are trying to find out is about the best and the most suitable variables (either dependent and independent) to be used in the data analysis so as to prove really significant correlations and eventually, the true impact of climate change in the area....and this is not that easy, agree.
Hi Vinod,
One tool that many find useful for subsetting and processing gridded time-series climate data is the USGS Geo Data Portal. The tool can perform both weighted and weighted area grid statistics and may be of use.
http://cida.usgs.gov/gdp/
Article Description of the U.S. Geological Survey Geo Data Portal Da...
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software