top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Hadoop: mapreduce.jobtracker.expire.trackers.interval no effect

+2 votes
630 views

we want to set the heartbeat timout for a tasktracker.

If the tasktracker does not send heartbeats for 60 seconds he should be marked as lost. I found the parameter mapreduce.jobtracker.expire.trackers.interval which sounds right to me.

I defined 

mapreduce.jobtracker.expire.trackers.interval 60000

in the mapred-site.xml on all servers and restarted the jobtracker and all tasktrackers.

I started a benchmark "hadoop jar hadoop-examples.jar randomwriter rand" and every tasktracker gets 2 jobs. It is a small test environment.

On one tasktracker i stopped the network. On the jobtracker i could see the "Seconds since heartbeat"
increasing. But after 60 seconds the tasktracker was still in the overview. Even in the log of the jobtracker I found nothing.

After over 600 seconds i found the message

org.apache.hadoop.mapred.JobTracker: Lost tracker .....

And the tasktracker wasn't shown any more on the jobtracker. Isn't this the right setting?

posted Dec 3, 2013 by Jagan Mishra

Share this question
Facebook Share Button Twitter Share Button LinkedIn Share Button
We use Cloudera 2.0.0-mr1-cdh4.2.0
commented Dec 3, 2013 by anonymous

1 Answer

+2 votes

I did a small test, and I a setting mapred.tasktracker.expiry.interval=60000 worked for me (TT became considered as lost after around 66 seconds).

May be the formula: mapred.tasktracker.expiry.interval + 2 * some-heartbeat-interval-that-is-3-sec-by-default? Otherwise, is the 6 sec some kind of time needed to make a decision to consider TT as lost?

answer Dec 3, 2013 by Ahmed Patel
I think I have found the reason. I looked at the job.xml and found the parameter
mapred.tasktracker.expiry.interval 6000000
and
mapreduce.jobtracker.expire.trackers.interval 30000

So I tried the deprecated parameter mapred.tasktracker.expiry.interval in my configuration and it works!

Why they write that the parameter is deprecated when the new one is not working and will be overwritten by the old one with the default value?
commented Dec 4, 2013 by anonymous
I tried the deprecated parameter mapred.tasktracker.expiry.interval in my configuration and it worked...
commented Dec 5, 2013 by anonymous
Similar Questions
+1 vote
Hadoop: jobtracker.jsp doesnot work?

I use Hadoop 2.2 and I want to run MapReduce web UI. So I visit following url: http://x.x.x.x:50030/jobtracker.jsp

Unable to connect Firefox can't establish a connection to the server at x.x.x.x:50030. Where am I wrong?

+1 vote
How to learn hadoop mapReduce on mongodb in java

I would like to know if you have any examples or tutorials where I can learn hadoop mapReduce on mongodb in java?

+2 votes
How to find min, max and mean of wordcount from text file in hadoop mapreduce? 
public class MaxMinReducer extends Reducer {
int max_sum=0; 
int mean=0;
int count=0;
Text max_occured_key=new Text();
Text mean_key=new Text("Mean : ");
Text count_key=new Text("Count : ");
int min_sum=Integer.MAX_VALUE; 
Text min_occured_key=new Text();

 public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
       int sum = 0;           

       for (IntWritable value : values) {
             sum += value.get();
             count++;
       }

       if(sum < min_sum)
          {
              min_sum= sum;
              min_occured_key.set(key);        
          }     


       if(sum > max_sum) {
           max_sum = sum;
           max_occured_key.set(key);
       }          

       mean=max_sum+min_sum/count;
  }

 @Override
 protected void cleanup(Context context) throws IOException, InterruptedException {
       context.write(max_occured_key, new IntWritable(max_sum));   
       context.write(min_occured_key, new IntWritable(min_sum));   
       context.write(mean_key , new IntWritable(mean));   
       context.write(count_key , new IntWritable(count));   
 }
}

Here I am writing minimum,maximum and mean of wordcount.

My input file :

high low medium high low high low large small medium

Actual output is :

high - 3------maximum

low - 3--------maximum

large - 1------minimum

small - 1------minimum

but i am not getting above output ...can anyone please help me?

...