Using 10 billion features to train a prediction model with 100 inputs seems like an overkill. The data can be better utilized by splitting into multiple batches, training several models and ensembling them.

Thank you for your answer. For classification and regression I would like to use supervised models. Anyhow, can you propose any implementation of random forest, decision tree, etc. for my dataset size? The conventional implementations do not cope with this amount of data.

To deal with the curse of dimensionality you should do pre-processing of your data first, then think about ANN, DT, etc.

Basically, there are two general approaches used for dataset volume reduction: 1) dimensionality reduction such as PCA and 2) size deduction such as K-means clustering, etc.

The following links are useful:

http://featureselection.asu.edu/software.php

https://www.researchgate.net/post/Are_these_feature_selection_methods_the_state_of_the_art/3

https://www.researchgate.net/post/How_to_beat_the_curse_of_dimensionality?_tpcectx=profile_questions

Thanks. I already thought about dataset size reduction. But is it convenient to cluster the datapoints before applying machine learning algorithms?

And, is there a clustering implementation (library), which can deal with this huge amount of data?

The following article (PDF is available online) and the cited references there provide some answers and  give you new ideas. 

M.A. Shayegan and S. Aghabozorgi. A New Dataset Size Reduction Approach for PCA-Based Classification in OCR Application, Mathematical Problems in Engineering,  Volume 2014, Article ID 537428, 14 pages. 

Good Luck.

-M.O.

http://dx.doi.org/10.1155/2014/537428

Extreme Learning Machines 

http://www.ntu.edu.sg/home/egbhuang/

https://www.researchgate.net/publication/40684804_OP-ELM_optimally_pruned_extreme_learning_machine

Article OP-ELM: Optimally pruned extreme learning machine

Thank you for your answers. From my point of view online machine learning methods and online clustering methods are most promising.

Can you suggest any online machine learning or online clustering methods?