1. I don't believe running with IDE is faster than running jar with command line for serious hadoop applications with considerable data size (not toy examples);

2. When there may be exceptions in your runnings, your comparsions would not make sense anymore, i.e. your so-called 'fast/slow" are meaningless!

JAR file usually needs to unpack set a JRE on JVM and convert them as byte code, each time you call a module. On the other hand eclipse IDE has JRE is always active and byte code compilation is pretty much faster. hope this answers.

Also its better to make an executable JAR file and run it without an IDE, for preliminary coding and testing rely on IDE but not always.

Thanks Aravindhan, I agree with your answer. But I have also some exception when running jar file but not when running from IDE. Exceptions are like machines may not sync., unknown host slavenode, nulll pointer exception etc. I am newbie to hadoop.

1. I don't believe running with IDE is faster than running jar with command line for serious hadoop applications with considerable data size (not toy examples);

2. When there may be exceptions in your runnings, your comparsions would not make sense anymore, i.e. your so-called 'fast/slow" are meaningless!

I am not so expert in playing with hadoop. I realize this inference that we can not run mapreduce applications from Eclipse in fully distributed mode cluster. Its only for localhost testing. Am I right ? I request comments from both of you.

You are running it in single mode when using eclipse and pseudo mode when running jar file. You can inforce  pseudo mode in your code if you want to get the same results.

Dear Mr M Alwarsh, Finally i have to run of fully distributed mode which is only possible by jar file. Jar file take more time for small input but its effect can be seen for large files. I just want to sure that we can not run in cluster mode from eclipse IDE.

You can run it in fully distributed mode from eclipse.

Does it need additional configuration in Eclipse? I am using Distributed Cache/Cache file in mapreduce program. In localhost its cool but in cluster mode it could not get cache file.

no but it depends on how you wrote your code. 

Lets the example of WordCount2 on new MR api of Apache Hadoop. What should be added or modified in this existing code? 

there are many versions of wordcount2. Post your code here

The url of the code is 

http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html

It is Example: WordCount v2.0 at the bottom of the page.

import java.io.BufferedReader;

import java.io.FileReader;

import java.io.IOException;

import java.net.URI;

import java.util.ArrayList;

import java.util.HashSet;

import java.util.List;

import java.util.Set;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.Counter;

import org.apache.hadoop.util.GenericOptionsParser;

import org.apache.hadoop.util.StringUtils;

public class WordCount2 {

public static class TokenizerMapper

extends Mapper{

static enum CountersEnum { INPUT_WORDS }

private final static IntWritable one = new IntWritable(1);

private Text word = new Text();

private boolean caseSensitive;

private Set patternsToSkip = new HashSet();

private Configuration conf;

private BufferedReader fis;

@Override

public void setup(Context context) throws IOException,

InterruptedException {

conf = context.getConfiguration();

caseSensitive = conf.getBoolean("wordcount.case.sensitive", true);

if (conf.getBoolean("wordcount.skip.patterns", true)) {

URI[] patternsURIs = Job.getInstance(conf).getCacheFiles();

for (URI patternsURI : patternsURIs) {

Path patternsPath = new Path(patternsURI.getPath());

String patternsFileName = patternsPath.getName().toString();

parseSkipFile(patternsFileName);

}

}

}

private void parseSkipFile(String fileName) {

try {

fis = new BufferedReader(new FileReader(fileName));

String pattern = null;

while ((pattern = fis.readLine()) != null) {

patternsToSkip.add(pattern);

}

} catch (IOException ioe) {

System.err.println("Caught exception while parsing the cached file '"

+ StringUtils.stringifyException(ioe));

}

}

@Override

public void map(Object key, Text value, Context context

) throws IOException, InterruptedException {

String line = (caseSensitive) ?

value.toString() : value.toString().toLowerCase();

for (String pattern : patternsToSkip) {

line = line.replaceAll(pattern, "");

}

StringTokenizer itr = new StringTokenizer(line);

while (itr.hasMoreTokens()) {

word.set(itr.nextToken());

context.write(word, one);

Counter counter = context.getCounter(CountersEnum.class.getName(),

CountersEnum.INPUT_WORDS.toString());

counter.increment(1);

}

}

}

public static class IntSumReducer

extends Reducer {

private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable values,

Context context

) throws IOException, InterruptedException {

int sum = 0;

for (IntWritable val : values) {

sum += val.get();

}

result.set(sum);

context.write(key, result);

}

}

public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();

GenericOptionsParser optionParser = new GenericOptionsParser(conf, args);

String[] remainingArgs = optionParser.getRemainingArgs();

if (!(remainingArgs.length != 2 || remainingArgs.length != 4)) {

System.err.println("Usage: wordcount [-skip skipPatternFile]");

System.exit(2);

}

Job job = Job.getInstance(conf, "word count");

job.setJarByClass(WordCount2.class);

job.setMapperClass(TokenizerMapper.class);

job.setCombinerClass(IntSumReducer.class);

job.setReducerClass(IntSumReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

List otherArgs = new ArrayList();

for (int i=0; i < remainingArgs.length; ++i) {

if ("-skip".equals(remainingArgs[i])) {

job.addCacheFile(new Path(remainingArgs[++i]).toUri());

job.getConfiguration().setBoolean("wordcount.skip.patterns", true);

} else {

otherArgs.add(remainingArgs[i]);

}

}

FileInputFormat.addInputPath(job, new Path(otherArgs.get(0)));

FileOutputFormat.setOutputPath(job, new Path(otherArgs.get(1)));

System.exit(job.waitForCompletion(true) ? 0 : 1);

}

}

http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html

change the input and output path to hdfs url instead of local files

    FileInputFormat.addInputPath(job, new Path("hdfs://localhost:9000/" + otherArgs[0]));

    FileOutputFormat.setOutputPath(job, new Path("hdfs://localhost:9000/" + otherArgs[1]));

Thanks a lot. Now it works. But I could not get the running status and information about the jobs from web ui 8088 and jobhistoryserver 19888 when running the job from eclipse, while these url shows the the jobs running from terminal using jar file. Can you please explain? 

@Adnan I am running Apriori algorithms on Hadoop cluster. I configured a 5 node cluster in which one is NameNode purely and others are DataNodes. My input data size is about 5MB to 10 MB. Data mining is cpu intensive tasks so a small amount of data may take significant time for smaller support count value. 

Can you help me to learn and use Spark that process data in memory ?