In driver class I have added Mapper, Combiner and Reducer classes and executing on Hadoop 1.2.1. I have added NumReduceTasks as 2. The output of my mapreduce code is generated in a single file part-r-00000. I need to generate output in more than one file. Does it need to be add our own Partitioner class?
https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/Partitioner.html
HashPartitioner is the default Partitioner in hadoop. you may use other partitioner provided by hadoop or create custom partitioner class too as per your requirements.
following link may be useful to you. kindly go through all the content because it will answer almost all your questions related to hadoop map-reduce.
http://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Partitioner
Each Reducer should write the output to 1 output file. I think you get only 1 output file because all your input records fail into one partition by default, and then for one partition you have one Reducer.
public int getPartition(K key, V value, int numReduceTasks) { return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks; }
You can write your own partition() method to send the result of map() to more than one partition. Also, you may want to have a look at the API, MultipleOutputs facility.
https://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html
The default partitioner in Hadoop will create one Reduce task for each unique "key" as output by context.write(key, value). All values with the same key will go to the same instance of your reducer, in a single call to the reduce function.
The thing you want to ask when partitioning is "which values need to be processed together in a single reducer?"
If it is ok in your program to have the values distributed randomly, then you can set your key as a random integer in the range of (0..reducer_count). When you do this, be sure to set mapreduce.job.reduces to the number of reducers that you want created.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering