Is it possible to turn on data node encryption without kerberos?

Question

I've been trying to secure block data transferred by HDFS. I added below to hdfs-site.xml and core-site xml to the data node and name node and restart both.

 dfs.encrypt.data.transfer
 true

 hadoop.rpc.protection
 privacy

When I try to put a file from the hdfs command line shell, the operation fails with "connection is reset" and I see following from the datanode log:

"org.apache.hadoop.hdfs.server.datanode.DataNode: Failed to read expected encryption handshake from client a /172.31.36.56:48271. Perhaps the client is running an older version of Hadoop which does not support encryption"

I am able to reproduce this on two different deployments. I was following https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SecureMode.html#Authentication , but didn't turn on kerberos authentication. No authentication works in my environment. Can this be the reason the handshake fails?

Comments

Kerberos is used to authenticate user or service principal to grant access to cluster. It doesn't encrypt data blocks coming in and out of cluster.

Answer 1

It is possible to turn on data transfer protocol encryption without enabling Kerberos authentication. We have a test suite in the Hadoop codebase named TestEncryptedTransfer that configures data transfer encryption, but not Kerberos, and those tests are passing.

The hadoop.rpc.protection setting is unrelated to data transfer protocol. Instead, it controls the SASL quality of protection for the RPC connections used by many Hadoop client/server interactions. This won't really be active unless Kerberos authentication is enabled though.

Please note that even though it's possible to enable data transfer protocol encryption without using Kerberos authentication in the cluster, the benefit of that is questionable in a production deployment. Without Kerberos authentication, it's very easy for an unauthenticated user to spoof another user and access their HDFS files. Whether or not the data is encrypted in transit becomes irrelevant at that point.