top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Hadoop logs and mapred.local.dir

+2 votes
482 views

I have following queries with hadoop, please help me?
1. The size of mapred.local.dir is big(30 GB), how many methods could clean it correctly?
2. For logs of NameNode/DataNode/JobTracker/TaskTracker, are they all rolling type log? Whats their max size? I can not find the specific settings for them in log4j.properties.
3. I find the size of dfs.name.dir and dfs.data.dir is very big now, are there any files under them could be removed actually? Or all files under the two folders could not be removed at all?

posted May 11, 2014 by Kumar Mitrasen

Looking for an answer?  Promote on:
Facebook Share Button Twitter Share Button LinkedIn Share Button

Similar Questions
+3 votes
Hadoop-2.2.0 "mapred.child.java.opts"

Is it correct to say that in hadoop-2.2.0, the mapred-site.xml node "mapred.child.java.opts" is replaced by two new node "mapreduce.map.java.opts" and "mapreduce.reduce.java.opts"?

+1 vote
How to set hadoop.tmp.dir if I have multiple disks per node?

I have ten disks per node,and I don't know what value I should set to "hadoop.tmp.dir". Some said this property refers to a location in local disk while some other said it refers to a directory in HDFS. I am confused, who can explain it ?

I want to spread I/O since I have ten disks per node, so should I set a comma-separated list of directories (which are on different disks) to "hadoop.tmp.dir" ?

+1 vote
Hadoop: Unable to view logs through Web UI + Amazon EC2

I have Hadoop cluster set up on Amazon EC2. When I am trying to access the application logs through Web UI I am getting page cant be displayed.

Configuration of Cluster: My Namenode is mapped with elastic IP(static) of EC2. Other datanodes public IP changing everyday as we are stopping the clustering during non working hours.

Observation :When I try to view the logs ,Its picking one of the datanode private IP and I am getting Page cant be displayed.

...