I am running a genome data which contains the 1200 Column and 5 lac Rows in a Matrix form for classification via random forest. Software is able to read the data in x matrix and transpose of this data has been used for classifications using ntree =2 only. We also increase the the ncels and vcells memory by using following commands: R --min-vsize=100GB --max-vsize=1000GB --min-nsize=10GB --max-nsize=100GB.
After running the data in R error is throwing error out of memory. Please help me out from this problem?
Thank you
you have to consider the memory use after looking at the memory use by your linux distribution, for eg my windows machine uses 1GB for page file, another 1-2 gb is used by the system and running programs. so out of 8gb total ram I am left only with additional 1-2 gb with which i can play with.
You did not mentioned if you were able to run your analysis at the default settings.
if you are using a desktop PC then you will benefit by running this large dataset on a server
You can allocate whatever you want via these parameters but you cannot violate the system memory availability. 1000GB you are asking for is 1TB of available memory! Only a tiny minority of machines have beyond 256GB of RAM and perhaps double that in swap space and you cannot just magic that memory into existence by specifying it!
I suggest you are truly, really running out of memory on a physical level and you either find a bigger machine or find a more memory-efficient way (check that you are using whatever package you are using properly) of solving your problem. Run top while your R code is running and watch memory usage rise. Did the machine freeze from a swap storm before R failed?
Dear David S Huen Sir,
I have install R in our SMP server which has 1.5 TB Ram that why I am running via above said command and when I am checking the use memory by R through
>gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 181610 9.7 407500 21.8 350000 18.7
Vcells 276931 2.2 13421772800 102400.0 9160227 69.9
after that I am running our script which contain following code
library("randomForest")
y
You may want to increase your protection stack size to circumvent this problem:-
https://www.mail-archive.com/[email protected]/msg40383.html
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Software
- R Statistical Package