Hi Manual,

I would personally use a Apache Spark Big Data cluster for that kind of task and install R on the master node. 85Gb of RAM is difficult to reach if not on a High Performace Computer (HPC) or on larger, more expensive, server in the cloud such as Amazon EC2 instances.

You can set up big data cluster and rent the access time on Azure, Amazon, etc.

An example: https://blog.revolutionanalytics.com/2017/09/hdinsight-tutorial.html

Probably, your university has a Big Data cluster ready and might be able to help you.

Christophe

What is the output of memory.limit() in RStudio, and have you tried increasing this limit? Does your computer/server have enough memory to handle 80GBs?

To analyze the data remotely, I would suggest you Amazon web services. Its quite easy also to create a Rstudio server on AWS.

Though con is you have to spend some money to buy the services.

Also, if you know to work on cluster with HPC, it may also help you. Long shot but worth a try.

load subsets of the data if its a local machine, and analyze that first.

then, you can use AWS or google cloud once you figure out what the exact task(s) are.

Ask that you get access to the servers on campus. Have the IT dept install Microsoft R open on the server and process your data on the universities servers. All of this should be free.

To analyze such large data sets, you'd probably need 256gb of ram or more and a good multicore processor and software that uses a multithread calculation paradiagm. (Think Intel MKL.) Without any of that, your left not being able to analyze most of that data at the same time.

Hi Manual,

I would personally use a Apache Spark Big Data cluster for that kind of task and install R on the master node. 85Gb of RAM is difficult to reach if not on a High Performace Computer (HPC) or on larger, more expensive, server in the cloud such as Amazon EC2 instances.

You can set up big data cluster and rent the access time on Azure, Amazon, etc.

An example: https://blog.revolutionanalytics.com/2017/09/hdinsight-tutorial.html

Probably, your university has a Big Data cluster ready and might be able to help you.

Christophe

Manuel Alonso Ma use R and SAS with large datasets

-->R loads all data into memory (by default)

-->SAS allocates memory dynamically to keep data on disk (by default)

-->Result: by default, SAS handles very large datasets better

-->another thing you can do it follow the instruction given by Christophe Stevens or you can use The Map and Reduce Function of Hadoop Streaming http://www.bytemining.com/wp-content/uploads/2010/08/r_hpc_II.pdf

Thanks

Folk here would probably use Python, which integrates very nicely with R.

Best wishes,

Darren

I would take the direction Christophe Stevens suggested. Going Big Data, you finally need some horizontal scaling capabilities Python cannot provide. With Spark you can use PySpark if you cannot switch to Scala or Java. Sooner or later you may again have the same problem finding a server large enough, because 84 GB is not big, and you might suddenly have to deal with 840 GB or more.

Henning.

Have you tried MATLAB? It is ideal for large datasets, as it is cloud-based.