I am running an R script that downloads and preprocesses all the available methylation data sets from TCGA. I'm using the Bioconductor package MethylMix for this. However, when I try to process the 450K breast cancer methylation data set (size ~13GB), I get a "Cannot allocate vector of size 12.8 GB" error.
I am running R 3.4.0 on 64-bit x86_64_pc-linux-gnu using my school's computing cluster, and each node has the following properties:
- Dual Socket
- Xeon E5-2690 v3 (Haswell) : 12 cores per socket (24 cores/node), 2.6 GHz
- 64 GB DDR4-2133 (8 x 8GB dual rank x8 DIMMS)
- No local disk
- Hyperthreading Enabled - 48 threads (logical CPUs) per node
so it seems as though there should be enough memory for this operation. The operating system is Linux, so I thought that R will just use all available memory, unlike on Windows? And checking the process memory using ulimit returns "unlimited." I am not sure where the problem lies. My script is a loop that iterates over all cancers available on TCGA, if that makes any difference.
You may want to look into R packages for handling big data, such as bigmemory and ff:
http://www.bigmemory.org/
http://ff.r-forge.r-project.org/
You can also try the command "memory.limit()" to see how much memory R is allocated.
The 64-bit version of R should be able to adress all available memory, even under Windows (64 bit).
A possible issue is that you may keep several copies of the object in memory. Also, every time you make a change to an object, the entire object is copied. Make sure that you don't have any copies and call the garbage collector before starting a memory-demanding process.
However, if you anyway process it in a loop, you can divide it in chunks and process them one after the other.
And having a look at Emilys hint is also surely a good idea.
Dear Lon
It is not a direct solution, but maybe will be still possible for You - work with local IT department to organize so-called High Computational Custer. I had the same issue, and now very happy to have this access to HCL.
It is coming under the category of Bigdata. I think u can perform analysis using 'R'.
you can try to distribute the jobs using the "foreach" package and a parallel backend instead of a classical loop.
Here is one solution by forking a process in the background which also involves reading large files in chunks and creating temporary files and merging the results in the main R shell you are working in.
These are a set of 2 related scripts one is the main script and the second is the forked process.
https://github.com/uhkniazi/BRC_NeuralTube_Miho/blob/master/S135/09_importMethylationExtractor.R
https://github.com/uhkniazi/BRC_NeuralTube_Miho/blob/master/S135/09.2_importMethylationExtractorFork.R
good luck
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Analysis
- Data Analysis
- Dataset
- Big Data