In handling memory there is not a large difference between C++ and Fortran. Both languages need to use dynamic allocation. The advantage of C++ is that you can use the STL's vector for this purpose. Together with RAII (Runtime Allocation Is Initialization) it will also automatically deallocate the memory for you. Modern Fortran is not much worse. Define you arrays as 'allocatable' (this is always faster than using 'pointer'). According to the 2003 standard assigning another arry to an allocatable array for the first time allocates the memory necessary for this if nothing has been allocated. (the Intel compiler needs an additional compiler flag for this). Allocatables also automatically deallocate when the variable goes out of scope. So, here there is no advantage or disadvantage of C++ over Fortran.

Since you are working on tables and not matrices/vectors Fortran also does not have any other advantages built into the language. You should be aware, though, that using most of the modern Fortran features will break all performance benefits of Fortran and make your code really slow. That is the only reason why I would prefer C++ over Fortran because it allows you to write cleaner code in general (by using better abstractions) without loosing performance.

Operating on two-dimensional tables is not very complicated in C++. Only that it is not possible to allocate a two-dimensional array directly. You should write a simple wrapper class for this like this:

template

class BigTable

{

size_t cols, rows;

T *data;

public:

BigTable(size_t num_rows, size_t num_cols)

: cols(num_cols), rows(num_rows), data(new T[rows*cols]) {}

~BigTable() { delete data }

T &operator()(size_t row, size_t col) { return data[row*cols+col]; }

const T&operator()(size_t row, size_t col) const { return data[row*cols+col]; }

};

This can be used in the following way:

BigTable t(20, 100);

t(12, 99) = 2.0;

double x = t(12,99);

...

t(0,3) = t(1,3)+t(2,3);

If you want to you can also shift indexing by 1 so that it is equivalent to Fortran. Putting the class into a header file all the functions will be inlined when used. With any reasonable C++ compiler there is no additional overhead for accessing individual elements of the table. It will also allow the compiler to do every possible optimization.

Just for completeness I'd like to show equivalent Fortran code. Here, you don't need to write your own class.

real(kind=8), allocatable, dimension(:,:) :: t

real(kind=8) :: x

allocate(t(20,100))

t(13,100) = 2.0

x = t(13,100)

...

t(1,4) = t(2,4)+t(3,4)

As I said, you don't need to deallocate the array explicitly. The advantage of 'allocatable' over 'pointer' besides automatic deallocation is that it is guaranteed that data is continuous in memory which allows for better optimization.

Hi,

I would recommend VB.net or C# for this job.

A related topic:

http://stackoverflow.com/questions/13078736/fortran-vs-c-does-fortran-still-hold-any-advantage-in-numerical-analysis-thes

hi marek.,

plz go through the link

http://brenocon.com/blog/2009/02/comparison-of-data-analysis-packages-r-matlab-scipy-excel-sas-spss-stata/

http://www.pythonlearn.com/html-008/cfbook015.html

thank you

Look at pytables and pydata projects - Python that will allow for a convenient scripting like in Scilab with Fortran/C backend and highly optimised data I/O is doing this job great.

In handling memory there is not a large difference between C++ and Fortran. Both languages need to use dynamic allocation. The advantage of C++ is that you can use the STL's vector for this purpose. Together with RAII (Runtime Allocation Is Initialization) it will also automatically deallocate the memory for you. Modern Fortran is not much worse. Define you arrays as 'allocatable' (this is always faster than using 'pointer'). According to the 2003 standard assigning another arry to an allocatable array for the first time allocates the memory necessary for this if nothing has been allocated. (the Intel compiler needs an additional compiler flag for this). Allocatables also automatically deallocate when the variable goes out of scope. So, here there is no advantage or disadvantage of C++ over Fortran.

Since you are working on tables and not matrices/vectors Fortran also does not have any other advantages built into the language. You should be aware, though, that using most of the modern Fortran features will break all performance benefits of Fortran and make your code really slow. That is the only reason why I would prefer C++ over Fortran because it allows you to write cleaner code in general (by using better abstractions) without loosing performance.

Operating on two-dimensional tables is not very complicated in C++. Only that it is not possible to allocate a two-dimensional array directly. You should write a simple wrapper class for this like this:

template

class BigTable

{

size_t cols, rows;

T *data;

public:

BigTable(size_t num_rows, size_t num_cols)

: cols(num_cols), rows(num_rows), data(new T[rows*cols]) {}

~BigTable() { delete data }

T &operator()(size_t row, size_t col) { return data[row*cols+col]; }

const T&operator()(size_t row, size_t col) const { return data[row*cols+col]; }

};

This can be used in the following way:

BigTable t(20, 100);

t(12, 99) = 2.0;

double x = t(12,99);

...

t(0,3) = t(1,3)+t(2,3);

If you want to you can also shift indexing by 1 so that it is equivalent to Fortran. Putting the class into a header file all the functions will be inlined when used. With any reasonable C++ compiler there is no additional overhead for accessing individual elements of the table. It will also allow the compiler to do every possible optimization.

Just for completeness I'd like to show equivalent Fortran code. Here, you don't need to write your own class.

real(kind=8), allocatable, dimension(:,:) :: t

real(kind=8) :: x

allocate(t(20,100))

t(13,100) = 2.0

x = t(13,100)

...

t(1,4) = t(2,4)+t(3,4)

As I said, you don't need to deallocate the array explicitly. The advantage of 'allocatable' over 'pointer' besides automatic deallocation is that it is guaranteed that data is continuous in memory which allows for better optimization.

Thank You Simon for this comprehencive answer. It cemented my conviction that I should use Fortran.

Hm, I 'smell' the start of another 'language battle' between Fortran and C++... :)

Personally I use Fortran and since I cannot find any kind of extreme differences, I deny to learn C...

You see, if we learn all languages, then our time will vanish.

Fortunatelly there is no any battle here (yet). But I feel exactly the same Demetris, in matter of time of course.

I personally use Java along with JExcelAPI for my data table handling. It allows you to operate XLS format sheets as both input and output.

One other consideration, you said that the table/array is 20 x 1 000 000.

Will you be accessing it by row slices (ie, row 1, cols 1-1000000, C++) or column slices (ie column 1, rows 1-20, Fortran)?  This may be a trivial question, but the tables are stored in memory differently in C++ v Fortran, and accessing consecutive cells could be more processing intensive if you choose the wrong language.

Jerrold, I wrote dimension of my table in the oposite way than it used to be. So my table is 20 columns and 10^6 rows. I will access to data by column slices.

Marek, then the same holds true.  Fortran stores arrays in memory in column major order, this means that cell 1,1 is immediately followed in memory by cells 2,1 ... 1000000,1.  C++ uses row major order, this means cell 1,1 is immediately followed by cells 1,2 ... 1,20.  It has to perform an additional calculation to get to 2,1

http://en.wikipedia.org/wiki/Row-major_order

You've got really good answers about low-level programming solutions. As a complement, I would suggest you take a look at high-level (scripting) languages such as R, more preferably a boosted version of the R environment called Revolution R, from Revolution Analytics. I did not use it for Big Data problems yet, but it's made for handling such problems efficiently. And maybe the calculations you want to run are already programmed in packaged functions. And if you not, you still can call external routines programmed in Fortran, C, or C++ from a DLL dynamically loaded into R within your script. I love R, as the code is easy to understand. If you're interested, take a look at the web site linked below. Cheers.

http://www.revolutionanalytics.com/

It depends on how big your "big data" is and what type of data it contains, but if you are prepared to use C++ or Java and to learn Gigaspaces (www.gigaspaces.com), you can do very fast processing on a dataset of up to several GB of data.

Gigaspaces has much the same functionality as a database, but it is not a "real" database, supports its own subset of SQL-like functionality and is accessed through an API rather than true SQL, so even if you have prior experience with databases there is a steep learning curve. The advantage over traditional databases is that all data is simultaneously kept in RAM (with continuous backup to disk running in the background) and accessed at very high speed directly through the API, instead of being processed as a disk-stored database and though an SQL interface.

Dear Marek, we have similar problems to deal with, synchrotron radiation data from time resolved experiments that amount to several gigabyte binary data typically, and we made quite good experience with the "newer" C# language. You may find some information in J. Synchrotron Rad. 19 (2012) 920. (doi:10.1107/S0909049512038599)

Dirk

Hi, I have tried C#, C++, Excel/VBA, Matlab, R with Big-data. R is the popular choice among Big-data community; however, I found Matlab to be very convenient with the additional tool-boxes. You can concentrate more on the analysis than the tedious coding. Unless you are a student/staff, Matlab option would be costly compared to others where C#, C++ could be found in free Microsoft Visual Studio Express versions, and Excel/VBA option can be found in cheaper MS office academic versions for under $300.

For Big Data - Best to use scripting languages such as R  or Python easy to learn and easy to use.