I've used 6 - 12 GB data sets using 64-bit R. You can partition your hard drive or install Linux alongside windows. Your processor is probably 64 bit, unless it is more than 10 years old. If you are a Linux noob, I suggest Linux Mint. It has a Windows installer. Your university might have server space. You an also purchase cloud computing time on Amazon Web Services for mere pennies. If your dataset is in a database format (e.g. MS Access), MDBtools can be used to read in only the parts of the data set you need to analyze.

This might be a primer:

http://stat.ethz.ch/R-manual/R-devel/library/base/html/Memory-limits.html

- if you have a 64bit OS, are you using a 64bit version of R?

- can you specify/modify memory limits?

- possible work-around: create chunks of the data and process them sequentially

@René Schenkendorf :Thanks for your advice.I will try it

@Jochen Wilhelm Thanks a lot.I am new to R.

I use windows with 32bit and R version is also 32bit

I try to modify memory limits as max values supported by system and nothing changes indeed.

I also divide my data into two parts and process was still limited by memory. More chunks used may affect the integrality and accuracy of result for my data.

Allways a good site to look for answers concerning problems with R is R bloggers. You're definitely not the only one to have problems with huge dataframes in R, as this: "http://www.r-bloggers.com/five-ways-to-handle-big-data-in-r/" appeared at the top of the list for "top 9 articles of the week".

@ Berlinda Verdoodt:thanks a lot. such website is just what i need.

I would advise switching to a 64 bit system with 64 bit R, 32 bit systems are limited to a maximum of 4gb of RAM much of which is used by the operating system itself. Out of curiosity how big is the dataset you are trying to load?

@Craig Duffy:Thanks for your advice. I wonder whether 64bit system with 64bit version R can break the limit of 4gb memory since I see some books about R saying such system is also limited to 4gb. my data is about 1.3GB, but i find that continuous operation have to be quited for the memory limit although i rm() and gc() in time.

@ Lesley Hoyles:Great thanks for your advice. I will try it

At least on 64-bit Linux, I can assure you that R will use memory beyond 4 GB. My instance blows up at 32 GB whn it's used all available RAM and swap :-)

It depends what dataset you are using e.g if yopu are dealing with bif matrix then you can use Hadoop alogwith R. Or use big machines.

I've used 6 - 12 GB data sets using 64-bit R. You can partition your hard drive or install Linux alongside windows. Your processor is probably 64 bit, unless it is more than 10 years old. If you are a Linux noob, I suggest Linux Mint. It has a Windows installer. Your university might have server space. You an also purchase cloud computing time on Amazon Web Services for mere pennies. If your dataset is in a database format (e.g. MS Access), MDBtools can be used to read in only the parts of the data set you need to analyze.

@Richard Smith:Thanks a lot for your advice. I have improve my computer and use ubuntu as my second operation system. Now I can deal with my dataset. Your several suggestions greatly inspire me. Since I am a new one in both linux and R,I am glad to hear more advices from you. Glad to follow you.

You can use bigmemory or ff packages that allows you to work with very large amount of data.

use a package designed to store objects on hard drives rather than RAM (ff, filehash, R.huge, or bigmemory), or use a library designed to perform linear regression by using sparse matrices such as t(X)*X rather than X (big.lm - haven't used this yet). For example, package bigmemory helps create, store, access, and manipulate massive matrices. Matrices are allocated to shared memory and may use memory-mapped files. Thus, bigmemory provides a convenient structure for use with parallel computing tools (SNOW, NWS, multicore, foreach/iterators, etc...) and either in-memory or larger-than-RAM matrices. I have yet to delve into the RSqlite library, which allows an interface between R and the SQLite database system (thus, you only bring in the portion of the database you need to work with).

from the link below:

http://www.matthewckeller.com/html/memory.html

I have processed datasets of about 40GB using "bigmemory" or "ff" packages.

I want to allocate this example and I am having problems:

side=790

my.mat

The problem here is the dist function. You have to create your own dist function that takes a bigmemory argument.

Many thanks for this advice. I am working with very large soil tomographic images and these procedures are the subject of my attention today.

is this problem same with the message that i got : "error: protect(): protection stack overflow"?