top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Hadoop:Name node and data node replacement

+3 votes
887 views

I am trying to understand,how to recover a Name node or Data node failure in case if those nodes are unable to come up due to unexpected issues.

Is there a way to replace Name node with new name node without loosing data.I mean new Name node should have all the data from original Name node. Similarly for Data node.

posted Dec 9, 2013 by Satish Mishra

Share this question
Facebook Share Button Twitter Share Button LinkedIn Share Button

1 Answer

+2 votes

To recover a NN, you probably want to use the HA feature. If not, try writing your edits to a nfs volume in additional to the local fs.

There is no need to recover a DN. The NN compensates for a lost DN by using the other remaining replicas to create a new replica on another DN.

answer Dec 9, 2013 by Majula Joshi
Thanks for your response.
You mean NN will always try to maintain number of replicas as configured even if one DN goes down and unable to come up?
 
commented Dec 11, 2013 by anonymous
Yes you are right. It will periodically checks the under replicated blocks information and place those blocks on available data nodes if required.
commented Dec 11, 2013 by anonymous
Similar Questions
+2 votes
Hadoop: got Java.lang.NullPointerException on two node when I restart cluster

Did any one got these error before, please help

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: xxxxx.com:50010:DataXceiver error processing WRITE_BLOCK operation  src: /xxxxxxxx:39000 dst: /xxxxxx:50010

java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:167)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:604)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:126)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:72)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:225)
at java.lang.Thread.run(Thread.java:745)
2015-01-11 04:13:21,846 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 657ms (threshold=300ms)
+1 vote
How to set hadoop.tmp.dir if I have multiple disks per node?

I have ten disks per node,and I don't know what value I should set to "hadoop.tmp.dir". Some said this property refers to a location in local disk while some other said it refers to a directory in HDFS. I am confused, who can explain it ?

I want to spread I/O since I have ten disks per node, so should I set a comma-separated list of directories (which are on different disks) to "hadoop.tmp.dir" ?

+3 votes
Can we control data distribution and load balancing in Hadoop Cluster?

As I studied that data distribution, load balancing, fault tolerance are implicit in Hadoop. But I need to customize it, can we do that?

0 votes
Is it possible to turn on data node encryption without kerberos?

I've been trying to secure block data transferred by HDFS. I added below to hdfs-site.xml and core-site xml to the data node and name node and restart both. 

 dfs.encrypt.data.transfer
 true

 hadoop.rpc.protection
 privacy

When I try to put a file from the hdfs command line shell, the operation fails with "connection is reset" and I see following from the datanode log:

"org.apache.hadoop.hdfs.server.datanode.DataNode: Failed to read expected encryption handshake from client a /172.31.36.56:48271. Perhaps the client is running an older version of Hadoop which does not support encryption"

I am able to reproduce this on two different deployments. I was following https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SecureMode.html#Authentication , but didn't turn on kerberos authentication. No authentication works in my environment. Can this be the reason the handshake fails?

...