The maximum size of an array is determined by the amount of memory that a program can access. On a 32-bit system, the maximum amount of memory that can be addressed by a pointer is 2^32 bytes which is 4 gigabytes. The actual limit may be less, depending on operating system implementation details. I believe that on most 32-bit Linux systems, 1 GB is reserved for the kernel, so the virtual address space of any given application for user data is 3 GB. All the application's code and data are mapped into this 3 GB address space, so the amount of contiguous address space available for a single array is even less. Your array amounts to about 2.6 GB, so I am not the least bit surprised that you ran into trouble.

Note that this has nothing to do with the amount of physical memory you have available. Even on a machine with substantially less than 1 GB of RAM, you can allocate a 2 GB array... it's just going to be slow, as most of the array will be in virtual memory, swapped out to disk. The limitation you are up against is the 32-bit address space. Therefore, even if your machine has more than 4 GB of RAM that can be accessed through PAE, individual applications won't be able to access more than 4 (3) GB directly, only through using additional libraries that allow user-level access to to PAE.

Given your question, however, you may be running into another issue. Your question indicates that you get an error while compiling your code. This suggests that your monster array is declared as local to a function. Local variables are allocated on the stack, and the stack size is usually much less than gigabytes, and while it is possible to increase the stack size, it's not necessarily a good idea. (Sorry, I don't know off the top of my head how GCC, which I presume is the compiler you use, determines the stack size and what command lines influence it.) So if you need a very large array, you are better off allocating it on the heap, either as a global variable, or allocating it dynamically (using malloc in C, or the new operator in C++).

The maximum size of an array is determined by the amount of memory that a program can access. On a 32-bit system, the maximum amount of memory that can be addressed by a pointer is 2^32 bytes which is 4 gigabytes. The actual limit may be less, depending on operating system implementation details. I believe that on most 32-bit Linux systems, 1 GB is reserved for the kernel, so the virtual address space of any given application for user data is 3 GB. All the application's code and data are mapped into this 3 GB address space, so the amount of contiguous address space available for a single array is even less. Your array amounts to about 2.6 GB, so I am not the least bit surprised that you ran into trouble.

Note that this has nothing to do with the amount of physical memory you have available. Even on a machine with substantially less than 1 GB of RAM, you can allocate a 2 GB array... it's just going to be slow, as most of the array will be in virtual memory, swapped out to disk. The limitation you are up against is the 32-bit address space. Therefore, even if your machine has more than 4 GB of RAM that can be accessed through PAE, individual applications won't be able to access more than 4 (3) GB directly, only through using additional libraries that allow user-level access to to PAE.

Given your question, however, you may be running into another issue. Your question indicates that you get an error while compiling your code. This suggests that your monster array is declared as local to a function. Local variables are allocated on the stack, and the stack size is usually much less than gigabytes, and while it is possible to increase the stack size, it's not necessarily a good idea. (Sorry, I don't know off the top of my head how GCC, which I presume is the compiler you use, determines the stack size and what command lines influence it.) So if you need a very large array, you are better off allocating it on the heap, either as a global variable, or allocating it dynamically (using malloc in C, or the new operator in C++).

Does it make a difference to 'malloc' the memory? Also, Python numpy has this awesome feature of memory maps, that basically create a file accessible as an array. Maybe one can create such structure in C, too.

Justus: In most C implementations, using malloc() allocates memory on the heap. Static and global variables are also allocated on the heap. In contrast, local variables are allocated on the stack, and the size of the stack can be a limitation. As to memory mapped files, that is not so much a language feature but an operating system feature (e.g., mmap() POSIX function).

If an int is 4 bytes, your int table is about 2.6 GB.

One problem is that a 32 bit operating system cannot really make use of 16 GB of RAM as it cannot address more than about 4 GB of addresses.

You would be well advised to switch to an OS with 64 bit.

So if the OS and any running programs consume 1.4 GB which is likely, you will run out of memory.

However, I think the real problem is using the wrong data structure and algorithm.

As others have mentioned with a 32 bit OS one can only address up to 4 GB. Due to some processor extensions the OS can use more than just 4 GB in total. But, this memory has to be distributed to different process. Each process has only 32 bit pointers and is thus restricted to 4 GB. This space is distributed between the kernel (someone already mentioned 1 GB for Linux), the code, the stack and the heap. I am not exactly sure, but I think that Linux divides the address space into 2+2GB -- 2 GB for code and 2 GB for data. The code is again separated to 1 GB for the kernel and the rest for your own program. In your data area there will be some constants and the stack already. At the point where you try to allocate your array, there are also already some other variable. So, in the end you will not even be able to allocate a 2 GB array.

Using a memory mapped file is certainly a solution to this. But, maybe you should rethink your algorithm. Is it really necessary to have all the data available all the time? If it has to be stored on disk, maybe recomputing the values every time is faster?

@Peter: It is true that you cannot map the entire array into the address space at once. But, you can map chunks of the file containing the array to the memory. This also means you'll need a smart algorithm to remap the file when necessary. I wouldn't want to write that kind of code in C. C++ with an overloaded index operator would be much easier to use.

But still, any solution we can offer is a really ugly hack. So, the best solutions are, as mentioned before, to either revise the algorithm or use a 64 bit OS.

32bit system, 4G is the limit of array, maybe smaller since program needs space, 64bit system, no limitation except your physical memory size

Muyi: You must mean _virtual_ memory size. You can allocate an array larger than your physical memory, it's just that parts of it will be swapped to disk. And by the way, even on a 64-bit system, the address space is large but finite: it is 16 exabytes. Most of us mere mortals don't (yet) have to worry about this limitation, but if you are planning to place the NSA's data holdings in an array, you might run into this limit. And the limit may be a lot less in practice. For instance, on Windows (except for Windows 8.1), the user-mode address space for applications in 8 TB. On Linux, the limit is usually 128 TB. The processor hardware may also introduce a limitation (e.g., restricting addresses to 48 bits, or 256 TB). For some details, see https://en.wikipedia.org/wiki/X86-64#Virtual_address_space_details

First of all 16 GB of RAM with 32-bit operating systems is a bad combination as you are not utilizing RAM fully.

Coming to the question as the OS is 32 bits maximum size that can be accessible by the OS is 2 power 32 i.e. 4 GB . Actual size may be less as at the time when the program is run, other programs are also running and hence using RAM.

Thus you have lesser memory available (which is accessible by OS). But as due to the OS limit you are not able to use rest 12 GB ram. Because of that you are finding this error. 

So best solution  could be, change operating system to 64-bit type.

Hope it helps :)

You are not indicating the type for your array, or the compiler; assuming 32-bit integer, or single-precision float (alto 32-bit in size), that is 2.6GB; it might probably be that your compiler does not support more than 2GB in a single variable, if they use the sign bit for anything else.

Also, the point others have made about the difference between heap memory and stack memory is valid: you will never be able to get 2.6GB of stack memory 😉 

You're probably better off by using some kind of malloc pointer, and using pointer arithmetic, or some of the C99/C11 variable length arrays.

I got the same trouble, using matrix 4000x4000

In VS 2015, and then tried to use vector like this:

typedef vector Row;

typedef vector Matrix;

it good!

In C, Linux Debian, using malloc() I allocated 3300 MB unsigned short table.

All worked correctly.

Have you tried using 'static' storage qualifier? It works for me using static:

#include

#include

int main (int argc, char **argv)

{

static int tmparray[700000000];

printf ("size: %zu, nelems: %zu\n", sizeof tmparray,

sizeof tmparray / sizeof tmparray[0]);

return 0;

}

---------output-----------

size: 2800000000, nelems: 700000000