Hi Levent,

How big is the data file, what is the format, what function you use to read it and what is the organization of the data you would like to achieve (e.g. data frame)?

You can try setting a larger limit like

memory.limit(size = 2500)

where the number for the size is provided in megabytes. But... R cannot exceed the physical memory of the RAM you have and the OS also uses quite of it...

What you could try is using packages like ff, LaF or filehash to import in a database and/or read the file in chunks. These might help:

http://stackoverflow.com/questions/18720036/reading-big-data-with-fixed-width

http://stackoverflow.com/questions/1727772/quickly-reading-very-large-tables-as-dataframes-in-r

Or use the data.table package which is supposed to work with large tables in memory. Here is a concise tutorial:

http://www.marketingdistillery.com/2014/05/05/working-with-large-data-sets-in-r-the-data-table-module/

Best of luck,

Plamen

Since you are not providing us with detail about your data or approach, I'd suggest you reading the Memory usage topic from Advanced R book.

You'll get much more feedback and information from r channel at Stackoverflow.

http://adv-r.had.co.nz/memory.html#memory

http://stackoverflow.com/questions/1358003/tricks-to-manage-the-available-memory-in-an-r-session

https://stat.ethz.ch/R-manual/R-devel/library/base/html/Memory-limits.html

R taps entirely on your RAM, and that's the reason you may encounter trouble analyzing big datasets with R. 

If your OS is Windows, you can try to use a pendrive as additional RAM. It won't be as fast as an actual RAM, of course, but it's storage capacity that is limiting you here. I think it's worth a try:

http://www.wikihow.com/Use-Pen-Drive-As-RAM

I have recently ran into issues regarding large data sets causing memory allocation problems in R. One solution tied to the previous commenter's response (Marcello), would be to just increase the RAM in your computer. RAM cards are much cheaper than they used to be - but you must be sure to install the correct type of RAM for your computer, which isn't difficult to determine.

It may not help depending on the size of your data set and the RAM that you have available, but my computer performed much more smoothly during the analyses and I ran into fewer memory allocation issues.

I hope this helps!

Thank you for all replies.

You can use these commands for allocating required vector size,  if you are running your script on Linux environment. 

For R interactive sessions : 23048 is the required vector size

       bsub -R "rusage[mem=23048]" -Is -q interactive R

Requested memory will be allocated within the server limits. 

On LSF: XXXX is  size of memory needed for scratch space  

        bsub -R "rusage[scratch=XXXX]

The server will allocate the requested memory for you (according to the server limits, but with good server - hugefiles can be used)

I'm facing the same problem. I have installed bigmemory package as well but didn't work. I have around 17 million rows that needs to be processed using R. Any solution to this. I'm using 64 bit-windows 8 system.

I have encountered this issue before, but you can use PostgreSQL and Spark to get around this issue. The packages you will need is dplyer and sparkly

it's too late, but have a look at affy documentation, they suggested to use

eset

Put the following in your R script for MAC.

Change memory as required.

install.packages("devtools", dependencies = TRUE)

devtools::install_github("krlmlr/ulimit")

ulimit::memory_limit(2000)

Ahmed I have done this but I have the following error message:

package ‘ulimit’ is not available (for R version 3.3.3)

Is there any other straight forward way?

Cheers

The gitgub page of the package ulimit (https://github.com/krlmlr/ulimit) says " Obsolete, use this instead: https://github.com/jeroen/unix".

Ahmed and Plamen,

I could not download unix

Here the message I got:

> devtools::install_github("jeroen/unix")

Downloading GitHub repo jeroen/unix@HEAD

Erreur : Failed to install 'unix' from GitHub:

Could not find tools necessary to compile a package

Call `pkgbuild::check_build_tools(debug = TRUE)` to diagnose the problem.

> unix::rlimit_as(2000)

Error in loadNamespace(name) : aucun package nommé ‘unix’ n'est trouvé

Do you have another idea?

Thanks.

Hi Sophie,

You should have the latest version of the devtools package installed. If you already have it, try reinstalling it.

Best,

Plamen

Thank you, Plammen, but it seems there is another cause. I got 1 message asking me to install some softwares and another that tells me that it is not available :-(

Unfortunately, I don't speak French and don't use Mac. This looks something very specific to Macc.

Plamen Vladkov Mirazchiyski

Yes, Indeed ;-)

But I found the problem for my specific situation but am afraid next time it will happen again and I would need the solution! I will follow the questions ;-) thanks a lot for trying to help me.

One option is before and after running your command that causes high memory consumption to do a "garbage collection" by running the gc() command, this will free up memory for your analyzes, in addition to using the memory.limit() command.

Example:

gc()

memory.limit(9999999999)

fit