I have very large dataset and its affinity matrix is dense if I propose to use pagerank therefore the matrix cannot be fitted into memory. I am looking for other variants of successful instance ranking algorithms working scalable at big data problems.
Ranking in information retrieval is normally based on relevance, which is a judgment. It's best to start from what a user is likely to consider relevant. Statistically, page rank is one approach to this, but simpler probabilistic approaches such as BM25 are also valid, possibly even more so if you don't have the kinds of community references between instances that page rank depends on. So it ought to depend on the collection and the kinds of queries people are asking.
The technical aspect of memory isn't really a big deal. Once you know what the ranking ought to be, there are always ways of scaling it up. Simple vector space and probabilistic approaches are far easier to scale than page rank, as the instances are effectively independent of one another. Naive sharding will do just fine in these cases.
PageRank is Global context Algorithm for link score computation. If your retrieval is done based on queries , you can exploit Local context Algorithms (for instance : Topical Pagerank, HITS, SALSA, ...) your link score computation will be based only on a subset (retrieved documents & link between them) of the dataset.
if it is necessary to use Pagerank there are many computation algorithms that do the job: approximative methods, or use distributed algorithms.
Good Luck.
- Badges
- Science topic