If one reducer is handling more than one key, how are the key-value pairs (received by the reducer) differentiated for different keys?
I am not familiar with hadoop technologies but this might help you:
The mechanism sending specific key-value pairs to specific reducers is called partitioning . In Hadoop, the default partitioner is HashPartitioner, which hashes a record’s key to determine which partition (and thus which reducer) the record belongs in. The number of partition is then equal to the number of reduce tasks for the job.
http://www.philippeadjiman.com/blog/2009/12/20/hadoop-tutorial-series-issue-2-getting-started-with-customized-partitioning/
Partition pairs are sorted by their keys and all values sharing the same key are clustered into the same set.
From the book "The Semantic Web: Joint International Semantic Technology "
They are first of grouped by key and that sorted. While processing key/value pairs in your reducer (data is passed into it in sequence fashion) you have to implement by your own 'splitting" of incoming data sequence. Here is example for you (it's in Python, but could be use with Hadoop Streaming) - https://gist.github.com/anonymous/f28f051bb2a7b1532d03
- Badges
- Science topic
- Similar topics
- Automotive Engineering
- Machines