Hi,

Part 1:

It does make sense to use GPs to model time series data.

In the simplest setting, we can consider having a set of (x,y) pairs, where x is the time and y is an observation/output at that time. To train a GP model, we just need to pick a mean and a covariance function for the x's. It is common to use 0 for the mean (and subtract off the empirical mean before training). There are many valid choices for the covariance function (see, e.g., Rasmussen and Williams, 2006), but a common one is the squared exponential function (commonly called the radial basis function in the context of NNs). Intuitively, this function implies that we expect observations close together in time (i.e., close x values) to have similar observed values (y's). If you expect strong periodic behavior of the inputs, covariance functions based on trigonometric functions are a reasonable choice.

While quite a bit of theory has been developed to explain what is really happening in training/inference, I recommend the GP chapter (17 for version 2.15.0) in the Stan user manual: http://mc-stan.org/documentation/ for a thoroughly explained implementation, supplemented with the Rasumussen and Williams book mentioned above for more detailed theoretical explanations.

Generalizing the model to multivariate inputs (such as the 18-dimensional space you mention in part 2 of your question) is straightforward (at least with the squared exponential covariance function). The Stan manual covers this (pp. 245-246 in version 2.15.0 of the manual).

---

Part 2: Unfortunately, I have only GPs for time series in low-dimensional settings, so I have not really experienced this problem. You may try searching for "online" or "sequential" training of GPs. For example, [Csató, L. & Opper, M. Sparse On-Line Gaussian Processes. Neural Computations, 2002, 14, 641-668] is highly cited and seems like it could be relevant.

---

I hope this helps at least get you pointed in the right direction.

Have a good day,

Brandon

Hi,

Part 1:

It does make sense to use GPs to model time series data.

In the simplest setting, we can consider having a set of (x,y) pairs, where x is the time and y is an observation/output at that time. To train a GP model, we just need to pick a mean and a covariance function for the x's. It is common to use 0 for the mean (and subtract off the empirical mean before training). There are many valid choices for the covariance function (see, e.g., Rasmussen and Williams, 2006), but a common one is the squared exponential function (commonly called the radial basis function in the context of NNs). Intuitively, this function implies that we expect observations close together in time (i.e., close x values) to have similar observed values (y's). If you expect strong periodic behavior of the inputs, covariance functions based on trigonometric functions are a reasonable choice.

While quite a bit of theory has been developed to explain what is really happening in training/inference, I recommend the GP chapter (17 for version 2.15.0) in the Stan user manual: http://mc-stan.org/documentation/ for a thoroughly explained implementation, supplemented with the Rasumussen and Williams book mentioned above for more detailed theoretical explanations.

Generalizing the model to multivariate inputs (such as the 18-dimensional space you mention in part 2 of your question) is straightforward (at least with the squared exponential covariance function). The Stan manual covers this (pp. 245-246 in version 2.15.0 of the manual).

---

Part 2: Unfortunately, I have only GPs for time series in low-dimensional settings, so I have not really experienced this problem. You may try searching for "online" or "sequential" training of GPs. For example, [Csató, L. & Opper, M. Sparse On-Line Gaussian Processes. Neural Computations, 2002, 14, 641-668] is highly cited and seems like it could be relevant.

---

I hope this helps at least get you pointed in the right direction.

Have a good day,

Brandon

Dear Kko Kko,

In response to your question "Do Gaussian processes have such a structure?", the answer is Yes. I suggest you to follow these links:

• http://www.gaussianprocess.org/gpml/chapters/RW.pdf
• http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=3552729EA88E7D246230164D5156EA09?doi=10.1.1.335.9713&rep=rep1&type=pdf
• https://arxiv.org/pdf/1601.08068.pdf

Here is some excerpts from http://www.gaussianprocess.org/gpml/chapters/RW.pdf :

"One straightforward method to speed up GP regression is to note that the linear system (K + σn2.I)v = y can be solved by an iterative method, for example conjugate gradients (CG)"

"One way that the CG method can be speeded up is by using an approximate rather than exact matrix-vector multiplication. For example, recent work by Yang et al. [2005] uses the improved fast Gauss transform for this purpose"

"Chapter 2 contains the definition of Gaussian processes, in particular for the use in regression. It also discusses the computations needed to make predictions for regression. Under the assumption of Gaussian observation noise the computations needed to make predictions are tractable and are dominated by the inversion of a n × n matrix"

In relation with the latter excerpt and as an answer to your second question, here is a valuable reference: https://arxiv.org/pdf/1403.6015.pdf

Finally, some alternative approaches of interest:

• http://users.informatik.uni-halle.de/~hinnebur/PS_Files/onlineDFT_pkdd06_streamWS.pdf
• http://charuaggarwal.net/streambook.pdf (chapter 12)

Hoping it will be helpful

For time series analysis and prediction, you also have the state-space based methods as implemented in, for example, the KFAS and MARSS packages in R. I recommend the book "Time series analysis by state space methods" by Durbin and Koopman. They also touch upon non.Gaussian distributions and how to handle them.

Hello,

In addition to above comments, you may check below implementation of Gaussian Process(GP), where it has been successfully applied for classification and regression problem. So, you can also modify this tool to model your problem.

http://www.gaussianprocess.org/gpml/code/matlab/doc/

And a relevant book of GP  http://www.gaussianprocess.org/gpml/chapters/

It is definitely fine to use GP for the time series analysis. If the time series is too simple and the nature of correlation between the target variables are not complex. Using time series analysis techniques such as ARIMA or GARCH are more straightforward. But such TSA techniques are not very good at handling mixed-type feature inputs and also are sensitive to outliers. For those that are non-stationary and has high volatility, you may have to use whitening tech first. Thus the model fitting process can be complicated. But why here you may use TSA techniques over GPR? The main reason is GPR (the full GPR) has the computational complexity of O(N^3) for model training. If you training sample reaches few thousands. The inference can be way slow.

But you can use sparse GPR which you approximate the joint prior and the likelihood using inducing set, which yields much smaller computational cost if the inducing set size is small.

Also, you can use distributed GPR.

For your question #1:

There is no structural similarity like RNNs, and also, the way model infers are also different. But the learning concept can be similar as the state of the model can be recursively updated the model with the new observations.

For your question #2:

You can use sparse GPR or distributed GPR.

Hope it works.

Dear Kko Kko,

Some other additional sources of information are available by following this feed on ResearchGate (in particular by checking out Probabilistic Interpretations of Recurrent Neural Networks):

https://www.researchgate.net/post/What_are_other_modern_alternatives_to_Markov_Chain_models

Regards

One way is to add gaussian noise to improve the system capacity to learn patterns.