How to customize hadoop configuration for a job?

Question

According to the book "Hadoop; The Definitive Guide", it is possible to use "-D property=value" to
override any default or site property in the configuration.

I gave it shot and it is true. The property specified with "-D" is ignored.

Then I put the property in an xml file and use "-conf xml_name" on the command line. But still I cannot
override the property.

The only way to override the default property is to get a Configuration reference in the code and set the property via the reference. But that is not convenient as I need to recompile the code each time I change the property.

Now the question is what is the right way to customize the configuration for a job?

Answer 1

You can implement your driver code using ToolRunner. So that you can pass your extra configuration through command line instead of editing your code all the time.

Driver code

public class WordCount extends Configured implements Tool {
    public static void main(String[] args) throws Exception 
    {
        int exitCode = ToolRunner.run(new Configuration(), new WordCount(), args);
        System.exit(exitCode);
    }
    public int run(String[] args) throws Exception 
    {
        if (args.length != 2) {
            System.out.printf("Usage: %s [generic options]  n", getClass().getSimpleName());
            return -1;
        }
        Job job = new Job(getConf());
        job.setJarByClass(WordCount.class);
        job.setJobName("Word Count");
        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        job.setMapperClass(WordMapper.class);
        job.setReducerClass(SumReducer.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        boolean success = job.waitForCompletion(true);
        return success ? 0 : 1;
    }
}

command line
$ hadoop jar myjar.jar MyDriver -D mapred.reduce.tasks=10 myinputdir myoutputdir

Comments

That is exactly what I did. But the command line parameters are ignored. The hadoop I am using is 2.0.0-cdh4.1.2. The property I tried is io.sort.mb.

---

could you try to remove the space between -D and the "property=value", i.e. use -Dproperty=value instead of "-D property=value". That should work :)