I'm no C# expert (not even a basic user) but you might have limit either within the int32 type for elements (so no more elements than int32 maximum), or, what is more likely, a 2GB limit for object (array).

Do you need to have all of the array in memory at the same time?

Can you do partial processing or request information directly from a binary file? It's going to be much slower but if you want to take advantage of the algorithm you already have you may want to try something like memory mapping.

.

you might consider training your last layer with a "standard" backpropagation algorithm ; it sure will take longer than computing the Penrose pseudo-inverse but it requires much less space

moreover, because you certainly have a very large layer of hidden units (just acting as a set of random projections), you might also consider the "dropout" option for this backpropagation training

.

Hi Fabrice. Thx for your suggestion. Could you please specify what you mean by "dopout" option?

.

see for instance :

http://jmlr.org/papers/v15/srivastava14a.html

http://papers.nips.cc/paper/4878-understanding-dropout.pdf

this last reference might be quite relevant to your case since it treats the case of a simple linear network "as a starter"

.

Fabrice's suggestion of using gradient descent is a good one.  An alternative approach would be to use a linear SVM  for the output layer weights, which again will hopefully solve the problem with the pseudoinverse.  There are good methods for this kind of problem in liblinear http://www.jmlr.org/papers/v9/fan08a.html  ELM is basically just another method of cheaply constructing a feature space in which a linear classifier is likely to work, so using an SVM for the output layer seems still within the spirit of ELM.

From experience, C# may not be your best bet.  I wrote a Neural Network library in C++ with Python bindings.  I've used it on some pretty large data sets and it has been very successful.  Its part of a more general package called "AILib", and its available for free at the link.  (Not tested in windows)

http://kovachtech.com/downloads.html

Whithout knowing the details of your problem, you may want to check out a swarm-based approach for training your NN: (e.g: "Training high-dimensional neural networks with cooperative particle swarm optimiser" (Engelbrecht et.al.on IEEE).

On a technical note, some ideas:

By default, the maximum size of an Array is 2 gigabytes (GB).

In a 64-bit environment, you can avoid the size restriction by setting the enabled attribute of the gcAllowVeryLargeObjects configuration element to true in the run-time environment.

However, the array will still be limited to a total of 4 billion elements.

Refer Here http://msdn.microsoft.com/en-us/library/System.Array(v=vs.110).aspx

Hope this helps.

I recommend you check section B, titled "Computationally efficient methods for training an ELM: iterative methods for large training sets" of this paper

http://arxiv.org/abs/1412.8307

Hi Sven and all guys, the so-called ELM is nothing at all but a renamed stuff. Please read the following articles and you will understand where it came from:

REFERENCES

[1] W. F. Schmidt, M. A. Kraaijveld, and R. P. W. Duin, “Feedforward

neural networks with random weights,” in Proc. of 11th IAPR

International Conference on Pattern Recognition, Conference B: Pattern

Recognition Methodology and Systems, vol. II. IEEE, 1992, pp. 1–4.

[2] C. L. P. Chen, “A rapid supervised learning neural network for

function interpolation and approximation,” IEEE Transactions on

Neural Networks, vol. 7, no. 5, pp. 1220–1230, 1996.

[3] H. Li, C. L. P. Chen, and H.-P. Huang, Fuzzy neural intelligent

systems: Mathematical foundation and the applications in engineering.

CRC Press, 2000.

[4] Y.-H. Pao and Y. Takefuji, “Functional-link net computing,” IEEE

Computer, vol. 25, no. 5, pp. 76–79, 1992.

[5] L. P. Wang and C. R. Wan, “Comments on ”the extreme learning

machine”,” IEEE Transactions on Neural Networks, vol. 19, no. 8, p.

1495, 2008.

[6] G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, “Extreme learning machine:

a new learning scheme of feedforward neural networks,” in

Proc. IEEE International Joint Conference on Neural Networks, vol. 2,

July 2004, pp. 985–990.

There exist randomized algorithms which may be useful to handle large scale data sets.  

Ran into this same issue w/ neural net in A.I. class project.

Assuming that your input and output node counts are constant? (input and output nodes)

This only leaves the hidden node layers and hidden node counts as possibly being variable.

In order for me to be able to use the most out of that 2GB array/vector size limit, I made my hidden node layer count as well as hidden node count user definable, increasing the layers count and/or node count w/i each layer until I hit that 2GB limit.  This way I was able to maximize the use of resources allowed.

You may or may not be able to use this technique depending on the speed and clarity you're desiring from your application.

I did this in C++ btw.

Let us know how you wind up resolving this issue please. Interested.

Todd

Simple constant preset w/i code itself:

// User defined:

#define numInputs 17

#define numHidden 21

.

a paper related to the option mentionned by Gavin above :

https://www.researchgate.net/publication/270651724_Online_Sequential_Extreme_Learning_Machine_With_Kernels

.

Article Online Sequential Extreme Learning Machine With Kernels

Thanks for citing my recent publication. :-) Allow me a few comments:

1) Suppose you have N examples, each of dimensionality D. If you expand them in B hidden basis, then you have an hidden matrix of dimensionality NxB. From your question, I assume you are computing the optimal weight as (using MATLAB notation):

Weights --> H*(H*H' + lambda*I)\y

However, for a positive lambda, you can rewrite the previous equation as:

Weights --> (H'*H + lambda*I)\H'*y

In this case, the matrix to be inverted is BxB, which might simplify your computation.

2) If you still have troubles computing this in batch, you can compute it recursively starting from many subsets of your training data, using the recursive least-square (RLS) algorithm. I wrote very briefly its description in the attached PDF.

3) After publishing the paper cited by Fabrice Clerot, I came to agree to the conclusion stated in a previous comment by Prof. Wang, i.e., ELM is in good part equivalent to much previous work, such as the Random Vector Functional-Link by Prof. Pao and his associates. For that reason, I now try to use these original denominations. For example, the algorithm described in point (2) is known as "OS-ELM" (actually, its non-regularized version), although its origins as RLS goes as far back as Gauss.

I hope this can be of help.

Dear Sven,

Why do not you try to substitute the Moore-Penrose pseudoinverse by Lagrangian multipliers? Using Lagrangian multipliers you can reduce your problem to N linear equation systems, which can be solved using iterative methods like Gauss-Seidel or Gauss-Jacobi. The solution of each linear system gives you the weight vectors of each output neuron.

Solving using such a suggested solution will help you to reduce the amount of memory you use when implementing the Moore-Penrose pseudoinverse, once C# has strong memory limitations. Generally, C# is not as precise as Java when you deal with mathematical operations and memory manipulation.

Best regards.

Dear Sven,

Do you have a clear mind why you called a single layer FNN as a ELM, where the idea of random assignment of input weights and biases came from other publications in 1992? To provide you a better understanding on randomized learning techniques for FNNs, I list some related work below:

[1]. W. F. Schmidt, M. A. Kraaijveld, and R. P. W. Duin, “Feedforward neural networks with random weights,” in Proc. of 11th IAPR International Conference on Pattern Recognition, Conference B: Pattern Recognition Methodology and Systems, vol. II. IEEE, pp. 1–4, 1992.

[2] Y. H. Pao and Y. Takefji. Functional-link net computing, IEEE Computer Journal, vol.25, no.5, pp.76-79, 1992.

[3] Y. H. Pao, G. H. Park and D. J. Sobajic, Learning and generalization characteristics of the random vector Functional-link net, Neurocomputing, vol. 6, 163-180, 1994.

[4] B. Igelnik and Y. H. Pao, Stochastic Choice of Basis Functions in Adaptive Function Approximation and the Functional-link Net, IEEE Trans. Neural Networks, vol. 6, no. 6, 1995.

[5]. L. P. Wang and Chunru Wan, “Comments on “the extreme learning machine”,” IEEE Trans. Neural Networks, vol.19, no.8, pp. 1494-1495.

I think that we need to respect the original contributors and term this type of FNNs as RVFLs (from [2], and further studied in [3] and [4] where you can find a mathematical proof on the universal approximation theorem in probability) or FNNs with random weights (NNRWs) (from [1], just an idea and without any scientific justification). 

Hope this clarifies and helps.

Chapter Pseudoinverse Learners: New Trend and Applications to Big Data