Hello,
I have a historical time series of 72-year monthly inflows. I need to generate, say 100, synthetic scenarios using the historical data.
Simple resampling (by reordering annual blocks of inflows) is not the goal and not accepted. I want synthetic scenarios to have different monthly values, but all summing up to the same value of the annual inflow as in the historical one (e.g. if in the historical data 2015 inflow is 2400 MCM, I want the synthetic scenarios to have the same value for 2015 inflow but with different monthly values). Surely these synthetic scenarios must satisfy some statistical metrics such as autocorrelation etc.
Does any one know of an existing code/script for this, preferably written in Excel, Python, or Matlab? I've heard of ARMA, KNN, etc. but none suits my purpose.
Thanks
Perhaps the simplest implementation for the result you want is a stochastic Markov chain approach, though it really depends on what the data you have is, e.g. does the state of month 2 have any relationship to month 1.
You create a transition matrices, whereby element (1,1) contains the number of transitions that a value of 1 transitioned to a value of 1, (1,2) is a value of 1 transitioning to a value of 2 etc.
You then have the probability of your transitions.
You can then scale all data to match monthly or annual means.
This is the simplest. I do not know of a matlab package that does this, though I could help you with the coding should you try this and get stuck.
Perhaps upload an example time series.
I am reminded of a paper by R. J. Aguiar (Article Simple procedure for generating sequences of daily radiation...
)that applied a similar logic to solar irradiance using Markov Chains that helped move from monthly means to daily means. ( a similar principle applies to our yearly means down to monthly values).
The same scientist also employed autoregressive methods to move from those daily to hourly values. Same principle (Article TAG: A time-dependent, autoregressive, Gaussian model for ge...
)Best,
Jamie
I think you have to be careful with the series correlation. For inflow the correlation can be strong within seasons for some systems while for others a stochastic Markov chain approach may be acceptable. I don't see the need for recreating a single year with the same accumulated inflow but with different profiles. You could make new series by combining seasons of the individual years but this would still be challenging as the trend in at least some countries has been increasing inflow. I don't know of freely available code for such activities.
The method based on a combinatorial optimization algorithm with the simulated annealing can perhaps meet your requirements. This technique is described in the recent paper by Borgomeo et al. (2015): “Numerical rivers: A synthetic streamflow generator for water resources vulnerability assessments”, the text is available at:
https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1002/2014WR016827
Ultimately, considering the climate change, and the recent drastic shift in the precipitation patterns observed in Europe and in North America, I’m not sure if the standard time series generation methods are applicable. Perhaps a better solution is to consider a synthetic weather data to generate the flow scenarios.
You want to do forecasting and understand the trend, autocorrelation, autoregressive process and behavor of white noise. SARIMAX may help. if there is reigme shift, you may apply a mixture of SARIMAX. For model validation, you may consider blocked 10 fold cross validation and blocked forward validation. For forecasting, you may consider Bayesian method because of the mixture component. Hope this can help. I would not consider machine learning, deep learning or AI to build the model. Unless there exists hidden process, I would not use MCMC method to build the model.
Thank you all for your answers.
I wanted to find a ready code/tool that generates a synthetic ensemble based on a historical time-series whilst satisfying correlation auto-correlation, and any other statistical factors that need to be considered-just like Yuk and Olav mentioned. And I wanted to use it to generate an ensemble of streamflow scenarios.
And I found it, and I recommend it to anyone who has the same use. It's here: https://github.com/julianneq/Kirsch-Nowak_Streamflow_Generator
It's build using methodology proposed by Kirsch et al. (2013; Article Evaluating the Impact of Alternative Hydro-Climate Scenarios...
) and Nowak et al. (2010;Article A nonparametric stochastic approach for multisite disaggrega...
). There are some pros and cons though. PROS: the code (written in Matlab) generates a fully synthetic ensemble of any size you want with the input of the historical data. Also, it has some python codes to create some plots describing those statistical factors around the generated ensemble such as flow duration curve (FDC), spatial correlation, auto-correlation, etc. CONS: the input historical data must be in daily time-step; so if your's is in longer time-steps (e.g. monthly) as in my case, then you have to interpolate daily values and don't forget to divide by 86400 to get the same monthly value at the end. Also, the code is pretty much hard-coded i.e. you have to go through them line by line to change bits that is relevant to your problem (it's not much though).
Dear Majed
Just came across to your question and in case if you are still struggling with this I can help you. I have developed a novel model for simulating synthetic flow sequences and would be very to collaborate with you and share my algorithms.
You can read about my model Article Stochastic modelling of flow sequences for improved predicti...
Please get back to me at [email protected] if you have any questions.
Cheers
Sandhya
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python