I just happen to work with similar problems on polymer simulation. My first approach is that it would be best if you could use R on a cluster, which makes everything much easier. If you do not have access to a computer cluster. A budget approach is to use Amazon Web service (AWS) with about $1/hr for (I could not remember detailed specs but it provides  decent computation capability) .

1.My first approach is that it would be best if you could use R on a cluster, which makes everything much easier.

2.If you do not have access to a computer cluster. A budget approach is to use Amazon Web service (AWS) with about $1/hr for (I could not remember detailed specs but it provides  decent computation capability) .

3.If the above two approaches failed, you could always buy more RAM, which is expensive but solves the problem directly.

4.If you could not afford AWS not RAM, you could use bigmemory  package(and some other big family packages ) to release stress on system RAM and a combination of doParallel and foreach package for parallelized computation for faster computation. This takes me about two days to figure things out. 

That is my answer based on your information provided.

You have to be more specific about size of data, and some of your code for optimization of computation, especially what went wrong with your calculation.

I would be glad to help you if you have any further questions.

how much memory you have? Is it an option to buy more memory? do you need all data? If not, you may connect with R to a data base where you store your data. You may google for RSQLite and related examples. 

Hi Mayur,

As Mathias mentioned, RMySQL or RSQLite  could be useful in your case.

Here a couple of  additional "pure R" solutions:

A) If you have a lot of columns and can preform your analysis in a column-by- column fashion, you can use the colClasses argument of read.table()  in a loop that skips certain columns in each step:

http://stackoverflow.com/questions/5788117/only-read-limited-number-of-columns-in-r

B) Similarly, if  you have too many rows and and can preform a row-by-row analysis, you should take a look at the skip an nrows arguments of read.table()

C) If your data is relatively sparse,  the functionality in package Matrix might be helpful:

http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/

Good Luck,

Amir

@Matthias Templ :Currently I have 32GB of RAM. and I dont want to buy memory as after loading data it consumes 25GB of Total RAM.

The problem i am facing when i perform operation on data after getting loaded into RAM, it sometimes gives correct output and sometimes failed to work. My code contains one for loop which is going to iterate for 40K times.

Thanks for your suggestion.

Hello,

You can try to increase virtual memory with RAM size. You may have some improvement of memory overflow problem.

Are you sure that the problem is memory? If the dataset "only" consumes 25GB of RAM, you are left with 7GB of RAM for the OS and R. Assuming you are not using any other big application concurrently, it seem to me that the amount of available RAM is plenty enough to hold the system together.

I would monitor the RAM usage during the experiments, if it does not grow I would start to take into consideration the possibility that some weird bug is lurking in your code. If it does grow I would try to understand if you can change things so that it uses less ram. If this is not possible I would try the techniques suggested by @Amir. 

R will copy most probably your data or subsets of your data when doing operations on your data set, so you will run out of memory. 

Maybe you could look on the data.table package which handle this more clever than base R. It is even much faster, if you once specified a data.table.

@Matthias Templ, @Amir B K Foroushani, @Sujan Kumar Roy, @Roberto Esposito : Thank you all for your suggestions. I appreciate your help. your solution helped me in my problem.

Hi @Mayur

I classify the solutions in four groups: external memory (bgml bigmemory), divide and Recombine (rmr, rhype, etc.), query languages (RMySQL, RSQLLite would be here) and distributed collection manipulation (Presto, SparkR). My slide on it: http://www.slideshare.net/jlpino/thesis-defense-38619252/30

If you want to do data mining, more than likely query languages are not enough. I would start with distributed collection manipulation packages.

http://www.slideshare.net/jlpino/thesis-defense-38619252/30

Following link provides solution to such type of problems:

http://www.r-bloggers.com/build-predictive-model-on-big-data-using-r-and-mysql-part-1/

Hi Mayur,

maybe I'm late,

but not knowing your code,

I would recommend you to take a look at the following link from  H. Wickham's "AdvancedR". 

Pay attention the section 4 "Modification in place" and to the subsection "Loops"

http://adv-r.had.co.nz/memory.html

The overall idea is to avoid modifying copy od the data, performing "inlace substitutions" instead. 

data.table is the fast and easy way to perform computations on various structures in R, It allows very much substantial access to structures.

You need to subset your data to avoid the machine paging out to disk.  I've found that any object larger than something like 1/3 of the RAM capacity causes problems if the algorithms being used involve copying the data (which most do, e.g on function calls).

How to subset the data?  Change your code to only read part of the data.  Different methods must be used for different types of data, e.g for text files, many reading functions have arguments for skipping lines, for binary files you can "seek" a certain byte, and for SQL-type databases you have very precise control.

1- Use LaF package to read the data.

install.packages("LaF")

library(LaF)

d

@Dan: Could you tell me how can we specifically skip selected column or include some from a large dataset text file in R?

A one line command would be more helpful

Thanks in advance

Doing operations on entire data set is unproductive thought, but big data analysts might refute this claim. If the big data analytics is true, then the entire edifice of sampling theory collapse. If we have two alternatives one: using samples and infer the population, the two: scaling up resources. The first one seems to be more cleverly. In R there are several approaches but there is no ground for those operations. For instance, the fact that R can scale up to maximum capacity  is not true due to the reason that the host i.e. Windows doesn't allow applications to use full capacity. The only solution is using HDD. That way the only option is to mix hadoop with R. The other packages like foreach, distributedR, snow, parallel only takes care of multi-core processing but doesn't work through HDD. 

I just happen to work with similar problems on polymer simulation. My first approach is that it would be best if you could use R on a cluster, which makes everything much easier. If you do not have access to a computer cluster. A budget approach is to use Amazon Web service (AWS) with about $1/hr for (I could not remember detailed specs but it provides  decent computation capability) .

1.My first approach is that it would be best if you could use R on a cluster, which makes everything much easier.

2.If you do not have access to a computer cluster. A budget approach is to use Amazon Web service (AWS) with about $1/hr for (I could not remember detailed specs but it provides  decent computation capability) .

3.If the above two approaches failed, you could always buy more RAM, which is expensive but solves the problem directly.

4.If you could not afford AWS not RAM, you could use bigmemory  package(and some other big family packages ) to release stress on system RAM and a combination of doParallel and foreach package for parallelized computation for faster computation. This takes me about two days to figure things out. 

That is my answer based on your information provided.

You have to be more specific about size of data, and some of your code for optimization of computation, especially what went wrong with your calculation.

I would be glad to help you if you have any further questions.

Depending on the nature of your data and the efficiency of different methods of slurping, then you might find readlines of use. This acts to read a subset and assists with testing code on a reasonably sized sample before upscaling to the entire data set. Alternatively as per your request, where the data set exceeds your RAM:

https://stackoverflow.com/questions/19204917/how-can-i-read-selected-rows-from-a-large-file-using-the-r-readlines-command-a

Dear Mayur,

I used to work with large data sets and I had the same problem in R. I was going to increase the RAM but I could workaround the problem with the following code:

memory.limit(size = 16150)

this code increase the memory allocation from system hard disk and compensate the limitation on RAM memory size.

Try this.

Best,

Mehdi

https://www.researchgate.net/post/How_can_I_increase_memory_size_and_memory_limit_in_R

http://www.stat.yale.edu/~mjk56/temp/bigmemory-vignette.pdf

https://www.quora.com/How-can-one-open-and-handle-very-large-data-sets-Greater-than-5-GB-with-a-standard-computer

http://www.columbia.edu/~sjm2186/EPIC_R/EPIC_R_BigData.pdf

http://www.bytemining.com/wp-content/uploads/2010/08/r_hpc_II.pdf

use bigmemory package