I am trying to do a topic modeling study of a dataset of about 4 million tweets using Mallet and running into issues with working memory, or "heap space." My computer does have around 15 GB of working memory, but Mallet, by default, utilizes only 1 GB. So I was getting the following error:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
I expanded the Mallet heap space allocation in the manner prescribed on https://programminghistorian.org/lessons/topic-modeling-and-mallet#issues-with-big-data. But it didn't help. So I was wondering if anyone had a solution.
Thanks.
Pl see the link for possible answer.
https://arxiv.org/ftp/arxiv/papers/1711/1711.04305.pdf
Ajit, thanks. This is a good introduction to topic modeling but doesn't seem to address my specific concern about big data analysis in Mallet.
May be slice the data set into time frames (but that depends on your objective of the study)?
Thanks, Malmi. I did slide the data into time frames, but I still want to run the analysis on all the times concurrently so that I get a common set of topics to compare the time frames.
Have you ever looked at the ISO Topic Maps standard？
http://www.isotopicmaps.org/sam/
Open the mallet file you can find in the bin directory. Look for MEMORY=1g and then increase it to 2g or 4g or more.
Source: https://uwspace.uwaterloo.ca/bitstream/handle/10012/11751/Getting%20Started%20with%20Topic%20Modeling%20and%20MALLET%20_%20Programming%20Historian.pdf?sequence=1&isAllowed=y
- Badges
- Science topic