Hi folks, I am handling large spatial data sets (around 15GB raster files) in R. For any kind of computation (running programme) R shows an error message 'cannot allocate vector of size ---GB.' I know it is a problem of RAM size of the PC as R use RAM memory for computation of any analysis by default. However, is there any other alternative process to solve this issue without increasing the RAM of a computer?
Error message is because You try to load into RAM memory more data than computer can handle. Solution is:
1- add RAM memory and use 64 bit version of operating system an 64bit R
2- use R library to work with big data like:
-bigmemory
- ff
You can try extending the Virtual Memory in your OS. This may allow you to allocate enough memory for this object your application works with, but it will certainly be very slow. The best solution is, as Michal suggested, to add more RAM.
Michał Combik I´ve checked 'bigmemory' documentation but I couldn´t realize which function should one use in order to decrease R objects size. Could you tell me more about it?
If you use windows operator, R can only use 25% of RAM . While in Linux it will use more than 70% RAM. It is one of weak point of R.
Hi!
If possible, could you provide some specs of the machine you're running the analysis on?
Anyway, it may be of value to you to try using memory.limit() in your R session to see how much memory is allocated to R. It returns the value in Mb, so if this value is lower than 1100, that means you need to allocate more memory to R.
If you're using 32-bit R, you may also want to consider switching to 64-bit R, as its max memory allocation is lower than 64-bit, especially on Windows machines.
More details can be found from the R manual and this Reddit thread by another user with the same issue
I'm using a server with 128 GB RAM, but I still cannot create a wordcloud because of the same error
Error: cannot allocate vector of size 221269.0 Gb
Wow Lale Madahali ... that's about 221 Tb.... I don't think your server will even have enough space for that, not mentioning costs and handling of such amount of RAM.
I'm having the same problem, but with vectors of about 200 Gb. I really don't know what to do either. I'm using a Linux machine, and I realize that it also creates huge temporary files. Given that the TEMP folder is limited by the disk space, I thought about moving the TEMP directory to an external HD, but I don't know if that would a solution. Maybe a combination of that with a huge SWAP partition could solve it. But that's in Linux, in Windows I couldn't say
Another way could be to use R packages that uses space from your hard drive instead of RAM, like ff, R.huge, bigmemory, filehash, etc. I haven't tried any of them, but it seems it could be a solution for some cases.
Put the following script to see if your problem gets resolved?
Change memory as required.
install.packages("devtools", dependencies = TRUE)
devtools::install_github("krlmlr/ulimit")
ulimit::memory_limit(2000)
Ahmed Mehdi
hi
devtools::install_github("krlmlr/ulimit")
Error in loadNamespace(name) : there is no package called ‘devtools’
> ulimit::memory_limit(2000)
Error in loadNamespace(name) : there is no package called ‘ulimit’
>
kindly guide
After conducting devtools::install_github("krlmlr/ulimit") , I was successful to perform the command in R.
- Badges
- Science topic
- Similar topics
- Biological Science
- Kinesiology
- Running