top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Streaming data access in HDFS: Design Feature

+1 vote
1,334 views

Can anyone please explain what we mean by STREAMING DATA ACCESS IN HDFS.

Data is usually copied to HDFS and in HDFS the data is splitted across DataNodes in blocks.
Say for example, I have an input file of 10240 MB(10 GB) in size and a block size of 64 MB. Then there will be 160 blocks.

These blocks will be distributed across DataNodes in blocks. Now the Mappers will read data from these DataNodes keeping the DATA LOCALITY FEATURE in mind(i.e. blocks local to a DataNode will be read by the map tasks running in that DataNode).

Can you please point me where is the "Streaming data access in HDFS" is coming into picture here?

posted Mar 5, 2014 by Abhay Kulkarni

Share this question
Facebook Share Button Twitter Share Button LinkedIn Share Button

1 Answer

+1 vote

Streaming means process it as its coming to HDFS, like where in hadoop this hadoop streaming enable hadoop to receive data using executable of different types.

I hope you have already read this:
http://hadoop.apache.org/docs/r0.18.1/streaming.html#Hadoop+Streaming

answer Mar 5, 2014 by anonymous
Similar Questions
0 votes
Can anyone provide me steps to grant a particular user access equivalnent to that of "hdfs" user in hadoop.

The reason behind this is I want to have my custom user who can create anything on the entire hdfs file system (/).
I tried couple of links however, none of them were useful. Is there any way by adding/modifying some property tags I can do that ?

0 votes
How to get info about which data in hdfs or file system that a MapReduce job visits?

I was trying to implement a Hadoop/Spark audit tool, but l met a problem that I can't get the input file location and file name. I can get username, IP address, time, user command, all of these info from hdfs-audit.log. But When I submit a MapReduce job, I can't see input file location neither in Hadoop logs or Hadoop ResourceManager.

Does hadoop have API or log that contains these info through some configuration ?If it have, what should I configure?

+1 vote
HDFS Data Integrity in copyToLocal

How can I verify the integrity of files copied to local from HDFS? Does HDFS store MD5s of full files anywhere? From what I can find, FileSystem.getFileChecksum() is relevant to replication and not comparison across filesystems.

The Data Integrity section in HDFS Architecture (http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) does not make it clear if, or how, copyToLocal verifies the integrity of the copied file.

+1 vote
HDFS data transfer is faster than SCP based transfer?

I have a use case that requires transfer of input files from remote storage using SCP protocol (using jSCH jar). To optimize this use case, I have pre-loaded all my input files into HDFS and modified my use case so that it copies required files from HDFS. So, when tasktrackers works, it copies required number of input files to its local directory from HDFS.

All my tasktrackers are also datanodes. I could see my use case has run faster. The only modification in my application is that file copy from HDFS instead of transfer using SCP. Also, my use case involves parallel operations (run in tasktrackers) and they do lot of file transfer. Now all these transfers are replaced with HDFS copy.

Can anyone tell me HDFS transfer is faster as I witnessed? Is it because, it uses TCP/IP? Can anyone give me reasonable reasons to support the decrease of time?

...