as i know inverted index consist of keywords and the document identifiers contain this keyword and the search on this index structure is sequential for single keyword search but if i want to search for multi keyword in this index . will divide this multi keyword for single keywords and search for it sequentially . iam thinking to use map-reduce algorithm to make this search in parallel.
Read the IR book: http://nlp.stanford.edu/IR-book/
You use MapReduce to build the index and then efficiently query it.Querying the index is commonly NOT implemented in the MapReduce Paradigm since it's too slow and not suited for that.
You can also use ElasticSearch https://www.elastic.co/ to take care of all of this.
For a collection of 1500 documents, every decent index will work, it's not a lot of data.
Read the IR book: http://nlp.stanford.edu/IR-book/
You use MapReduce to build the index and then efficiently query it.Querying the index is commonly NOT implemented in the MapReduce Paradigm since it's too slow and not suited for that.
You can also use ElasticSearch https://www.elastic.co/ to take care of all of this.
For a collection of 1500 documents, every decent index will work, it's not a lot of data.
First of all MapReduce is not an algorithm in the sense like QucikSort or similar once. It's approach that helps to solve heavy and computation intensive data processing problems.
Chris is right, implementing "querying index" in MapReduce way is not a good idea. However, build index itself can be done using MapReduce approach. For instance, here is detailed description how to build tf-idf - http://henning.kropponline.de/2014/06/08/map-reduce-tf-idf/
At the end, if you have only 1500 documents it makes no sense to use any kind of distributed computations. Even a single threaded application/utility can do it quite fast.
Good luck!
The challange in the task is "share" the data between the tasks. Read the paper of google and you understand, but depends of data structure that you use to read the index file from the disk (or from the cluster node). You must share the lexicon between the resources that you use. I really think that for ranked search or "words in the bag" search method you do not need a mapreduce strategy. If you need, I created a simple project for study. For a 322 documents collection (600MB size) my implementation of a simple ranked search spends 0 ms for a seach (10 words). I only use parallel resources in the index method and I had some problems for share the lexicon.
(Sorry about my english :)
http://infolab.stanford.edu/~backrub/google.html
https://github.com/zeferreira/eDoc
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Analysis
- Data Analysis
- Dataset
- Big Data