top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Hadoop 2.2.0 YARN isolation and Windows

+3 votes
437 views

Few questions about the new Hadoop release regarding YARN:

  1. Does YARN need to run on the same machines that are hosting the HDFS services or can HDFS be remote of a YARN cluster? If this done by placing the remote HDFS clusters configuration files (core-site.xml and hdfs-site.xml) on the YARN clusters machines?

  2. According to http://www.i-programmer.info/news/197-data-mining/6518-hadoop-2-introduces-yarn.html, Hadoop 2.2.0 supports Microsoft Windows. How do/Can you configure YARN for secure container isolation in Windows? It seems that the ContainerExecutor and DefaultContainerExecutor can detect and run on Windows, but the secure LinuxContainerExecutor are for *nix systems, so is there anything in place for maximum security like LCE is?

  3. If 1 is yes, then is it possible to have a cluster mixed with both Linux and Windows machines running YARN and working together?

posted Jan 9, 2014 by Sheetal Chauhan

Looking for an answer?  Promote on:
Facebook Share Button Twitter Share Button LinkedIn Share Button

Similar Questions
+1 vote
Hadoop YARN 2.2.0 Streaming Memory Limitation?

We are currently facing a frustrating hadoop streaming memory problem. our setup:

  • our compute nodes have about 7 GB OF RAM
  • hadoop streaming starts a bash script wich uses about 4 GB OF RAM
  • therefore it is only possible to start one and only ONE TASK PER NODE

out of the box each hadoop instance starts about 7 hadoop containers with default hadoop settings. each hadoop task forks a bash script that need about 4 GB of RAM, the first fork works, all following fail because THEY RUN OUT OF MEMORY. so what we are looking for is to LIMIT the number of containers TO ONLY ONE. so what we found on the internet:

  • yarn.scheduler.maximum-allocation-mb and mapreduce.map.memory.mb is set to values such that there is at most one container. this means, mapreduce.map.memory.mb must be MORE THAN HALF of the maximum memory (otherwise there will be multiple containers).

done right, this gives us one container per node. but it produces a new problem: since our java process is now using at least half of the max memory, our child (bash) process we fork will INHERIT THE PARENT MEMORY FOOTPRINT and since the memory used by our parent was more than half of total memory, WE RUN OUT OF MEMORY AGAIN. if we lower the map memory, hadoop will allocate 2 containers per node, which will run out of memory too.

since this problem is a blocker in our current project we are evaluating adapting the source code to solve this issue. as a last resort. any ideas on this are very much welcome.

0 votes
Windows and Linux hadoop cluster

Please let me know if it's feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).

If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.

+3 votes
Problems installing Hadoop on Windows 7 enterprise N

Currently I'm trying to install Hadoop 2.6.0 on a Windows 7 enterprise computer. I used the following guide

http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os

I already fixed some previous problems by installing a correct version of Protocol Buffer. But now ran into a new problem. 

[ERROR] FAILED TO EXECUTE GOAL ORG.CODEHAUS.MOJO:EXEC-MAVEN-PLUGIN:1.2:
        EXEC (COMPILE-MS-WINUTILS) ON PROJECT HADOOP-COMMON: 
        COMMAND EXECUTION FAILED. PROCESS EXITED WITH AN ERROR: 1(EXIT VALUE: 1) -> 
        [HELP 1]

I searched the internet and found that perhaps my MSBuild was causing the problem, but I check my PATH and it contained the MSBuild path to the correct location. I'm using .Net Framework 4.0

I tried running "mvn clean install" and from the tests I got the following errors. 

-------------------------------------------------------------------------------
Test set: org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator
-------------------------------------------------------------------------------
Tests run: 14, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 33.35 sec

Any suggestion on how to proceed.

0 votes
Hadoop on Windows 8 with Java 8

I can't find any information on how possible or difficult it is to install Hadoop as a single node on Windows 8 running Oracle Java 8. The tutorial on Hadoop 2 on Windows mentions neither Windows 8 nor Java 8.

Is there anything known about this?

+2 votes
How to run hive on windows 7?

I'm a freshman in hadoop world. After some struggling, i've successfully make hadoop 2.6 running on my windows 7 laptop.

However when I want to run hive 1.0.0 on my win 7 system, I found there is no cmd line script as provided for linux. It's also hard to find any useful message in google.

Anyone can provide me any clue on how to run hive on window 7?

...