Hello, I have some action in Java (8) which is performed in parallel threads over a uniform list of data. On one hand I need a single thread for each piece of data, on the other hand, I don't know how many of them I will have, and how much memory each of them needs, so I get OutOfMemoryException for large samples.
So the question is the following: how to easily estimate in the runtime should the thread be created immediately, or should it wait for some other thread to finish and for garbage to collected.
There are some obvious solutions to this problem like forbidding the action to use more than, say, 3/4 of total JVM memory, but perhaps somebody have a better solution.
Peter, thank you for your advice, but the problem is that I indeed don't know how much memory do I need per thread (for example it depends on file data file size). Something like that you proposed was the first thing on my mind.
Nane, your recommendation looked very appealing, but sadly it does not work. I just made an artificial test for it and it still throws OutOfMemoryExceptions.
I think the problem is that JVM can't probe how much memory the thread needs without lunching it. And after lunching it throws an exception. For now the only two ways I see is to either keep some memory reserve, or estimate memory consumption and count threads like peter proposed. Also one can just catch OutOfMemoryException, but I don't think that is a good way.
You should setup a pool of worker threads which is limited to some configurable value which you set from command line (e.g. -Dthread.pool.size=100) and only start new processing when your pool has a free one. This way you reuse threads instead of create new objects all the time.
Also play around with java memory options, e.g. decrease used memory per thread with java options (use minimum -Xss64k) . And increase memory of your JVM to e.g. 4GB (-Xmx4G).
And use the Server JRE for better performance (be sure you start with java -server).
Peter, I received a few pieces of valuable advice. Let me summarize what I found the most interesting from my personal point of view.
1)It is not possible to control how much memory the thread will need when one is starting it, so one fully can not protect oneself from OutOfMemoryException. The only option is to catch it and evaluate properly.
2) One can limit the number of threads (either by dispatcher or built-in methods of java.util.stream.Stream).
3) Also one could limit the memory usage per thread to avoid too large data fragments to be worked with in parallel.
Also I found that in my precise task it is much easier to just totally forbid parallel execution for the most memory consuming tasks (since the do not require much processor time).
I thank everybody for valuable advice.
P. S. Of course I knew about -Xmx option, but honestly overlooked the -Xss.
Alexander, i think memory is not the most important factor here. In a parallel task, speed is the main concern. You can limit the number of threads based on the number of system's cores AND memory limits. Having a large number of thread even without OutOfMemoryException is not efficient.
Alexander knows better the circumstances, But my conclusion was based on the part "I need a single thread for each piece of data, on the other hand, I don't know how many of them I will have". Based on this sentence, the number of threads can be large.
Mohsen, java internal thread manager is good enough for user not to think about processor parallelism. It is a good idea to limit the number of processes to the number of system cores (via Runtime.availableProcessors()), but in my case it is the memory one should think about, not the processor performance.
As already mentioned there is no reason to create more threads than the number of available CPU cores, However if processing of different chunks of equal size can take different time it can make sense to create more smaller chunks and use a thread pool to process them. Another option is to use ForkJoin framework which was created for exactly this kind of problems.
Hi Alexander,
this is usually done with the help of a ThreadGroup in which you can group threads (as the name tells you :) ) and can control how many threads are currently a member of the group.
Here comes the documentation:
http://docs.oracle.com/javase/7/docs/api/java/lang/ThreadGroup.html
Cheers,
Marc
There is nothing in the ThreadGroup that can be used to _control_ anything. In fact it is virtually a useless thing. And it is not a surprise that JConsole knows nothing about thread groups.
Hi Igor,
well that's the most annoying thing i ever read here. Anyway, it seems like Alexanders problem is solved!
Cheers,
Marc
- Badges
- Science topic