In the paper "Apache Hadoop YARN: Yet Another Resource Negotiator", it claims that Yarn can extend node number from 4000 in Hadoop 1.x to over 7000. My question is that what is the key principle of improving scalability behind YARN.
In Hadoop 1.x, there are centralized NameNode and centralized JobTracker. It seems that things haven't changed much in YARN since it also has centralized ResourceManager. The bottleneck in Hadoop is still the design of centralized control.
Yes, you are correct that the Resource Manager has cluster-wide responsibilities just as the Job Tracker had. However, the job-related responsibilities of the Job Tracker are largely handled by a new component - the Application Master. There is one Application Master per job - this is why YARN scales better than the previous Hadoop architecture. The Application Master for a given job can run on an arbitrary cluster node, and it runs until the job reaches termination.
Yarn enhanced Hadoop’s ecosystem by separating the Data processing engine (MR programming model) and resource management capabilities. MR was originally designed as a batch-oriented system, though; it becomes often used for other data analysis and processing types, which raised an emergent need for separating these two major functionalities. As a direct impact of loose coupling between cluster resource management and the application framework, the framework complexity was reduced, the system flexibility, scalability and performance were enhanced as well. Yarn become the paradigm backbone and responsible for scheduling users application tasks, implementing security controls and managing the computing resources, besides providing the high availability features of Hadoop
Both Suzanne McIntosh and Feras Awaysheh mentions that the master node is freed from the job management, and these freed resources could be used for management of more node. That does perfectly explain the improvements of the scalability.
I am now trying to understand why the number of nodes could be the factor limiting its scalability. Could it be the resource limitation of a single process since it keeps the state information of all slave nodes? If this is the case, will allocate the ResourceManager to a more powerful server help improve the scalability?
- Badges
- Science topic
- Similar topics
- Biological Science
- Physiology
- Physiological Processes
- Stress