Hi Qasim,

You should look at PAM (Pachinko Allocatiom Model), and I'm avaiable to participate in our paper.

Tiago Pereira

I just started working on a similar project. I will apply LDA to a quite big (~1300) corpus of research papers published by my University Department (DIA, University of Trieste, in which I am a member of the research board). The department is by nature characterized by inter-disciplinarity (Department of Engineering and Architecture, encompassing civil, computer, ICT, industrial, naval engineering, and architecture in general) and one of the aims of my activity is to verify if and to which degree research papers exhibit this interdisiplinarity. My plan is to use LDA, some form of dimensionality reduction (such as MDS or PCA) in order to easy the visualization of the data, and some form of clustering.

Hi Eric, thanks for your message. In my initial exploratory work, I have done a couple of LCA examples: in example one I applied LCA 1000 papers on Sustainability and Well-being; and in the second example, I used a corpus of 3500 full length papers from JSTOR dfr:

Topic Modeling Tutorial: http://mqasim.me/?page_id=77

Example 1: http://mqasim.me/sw1000/

Example 2: http://mqasim.me/sw3500/

The models are beautifully capturing topics in the corpus. However, I am still confused how can I establish Sustainability and Well-being are actually one discipline.

I would probably use a different set of documents as a validation set. But I am just guessing. I agree that generating a brief answer to the question "are corpora A and B related to the same topic" is not straightforward. In some sense, you could see it as an "author verification" problem, but, usually, in that case style matters more than content.