University:
University of Arizona
Course:
CHEM 141 | General Chemistry Lecture I
Academic year:

2022
Views:

283

Pages:

10
Author:

Alma Stephenson

Hadoop Installation Hadoop can be run in one of three modes: Standalone (or local) mode • All Hadoop services run in a single JVM. • Local file system is used not HDFS • MapReduce jobs run with a single mapper and single reducer. Pseudo-distributed mode • • • • Real multimode cluster simulation All services are configured as fully distributed, but all daemon run on a single server. Cannot configure data replication or high availability –single server. Other projects such as Spark, Hive, Pig are needed to be installed separately. Fully distributed mode The Hadoop daemons run on a cluster of machines. Hadoop Installation– Your Task To get started, you'll need to create a virtual machine (VM) running Ubuntu OS and follow the provided steps to install and configure your Hadoop cluster; Hadoop_installation_Pseudo-distributed.txt Start up / Shutdown you cluster Start up / Shutdown you cluster http://localhost:9870 Start up / Shutdown you cluster http://localhost:9870 HDFS Hadoop Distributed File System (HDFS) HDFS • Very Large Distributed File System • 10K nodes, 100 million files, 10PB • Assumes Commodity Hardware • Files are replicated to handle hardware failure • Detect failures and recover from them • Optimized for Batch Processing • Data locations exposed so that computations can move to where data resides • Provides very high aggregate bandwidth HDFS Architecture NameNode • Manages File System Namespace • Maps a file name to a set of blocks • Maps a block to the DataNodes where it resides • Cluster Configuration Management • Replication Engine for Blocks • Metadata in Memory • The entire metadata is in main memory • No demand paging of metadata • Types of metadata • • • • List of files List of Blocks for each file List of DataNodes for each block File attributes, e.g. creation time, replication factor • A Transaction Log Records file creations, file deletions etc DataNode • A Block Server • Stores data in the local file system • Stores metadata of a block. • Serves data and metadata to Clients • Block Report • Periodically sends a report of all existing blocks to the NameNode • Facilitates Pipelining of Data • Forwards data to other specified DataNodes