Hadoop famous interview questions and answers ? (Part 1)

Commonly asked Hadoop famous interview questions and answers.

1. What is Hadoop?
   • Hadoop is an open-source framework used for distributed storage and processing of large datasets.
2. What are the different components of Hadoop?
   • The main components of Hadoop are Hadoop Distributed File System (HDFS), Yet Another Resource Negotiator (YARN), and MapReduce.
3. What is HDFS?
   • HDFS is a distributed file system used for storing large datasets across multiple nodes in a Hadoop cluster.
4. What is YARN?
   • YARN is a resource manager that manages resources in a Hadoop cluster and schedules tasks.
5. What is MapReduce?
   • MapReduce is a programming model used for processing large datasets in parallel across multiple nodes in a Hadoop cluster.
6. What are the advantages of using Hadoop?
   • Hadoop provides advantages like scalability, fault-tolerance, and cost-effectiveness in handling large volumes of structured and unstructured data.
7. What is the difference between Hadoop and Spark?
   • Hadoop is a distributed computing system used for storing and processing large datasets, whereas Spark is an open-source distributed computing system used for processing large datasets in real-time.
   • Hadoop is a framework that is designed to store and process large datasets using a distributed file system and the MapReduce programming model.
   • Spark, on the other hand, is a fast and general-purpose cluster computing system that is designed for data processing and analytics.
   • Spark provides an alternative to Hadoop’s MapReduce model and provides higher performance and more flexible programming options.
8. What is a NameNode in Hadoop?
   • NameNode is the master node in Hadoop’s distributed file system (HDFS) that manages the metadata of all the data stored in the HDFS.
   • The NameNode is the central component of HDFS and is responsible for managing the file system metadata, such as the location of data blocks and the permissions of files and directories.
   • It stores the metadata in memory and on disk and provides a high-availability solution through the use of multiple NameNodes in a cluster.
   • It is responsible for maintaining the metadata about the files stored in HDFS.
9. What is a DataNode in Hadoop?
   • DataNode is the slave node in Hadoop’s distributed file system (HDFS) that stores the actual data blocks in the HDFS.
   • The DataNode is a node in the Hadoop cluster that stores the actual data blocks that make up the files stored in HDFS.
   • It receives instructions from the NameNode on where to store data and communicates with the NameNode to report the status of the data blocks.
   • A DataNode in Hadoop is a node in the Hadoop cluster that stores data.
   • It is responsible for storing and retrieving data blocks as requested by clients.
10. How does Hadoop ensure fault-tolerance?
    • Hadoop ensures fault-tolerance by replicating data across multiple nodes in the cluster, and in case a node fails, the data can be retrieved from other nodes where it is stored.
11. What is the default port number for Namenode?
    • The default port number for Namenode is 8020.
12. What is a Hadoop cluster?
    • A Hadoop cluster is a group of computers that work together to store and process large datasets.
13. What is a block in HDFS?
    • A block is the smallest unit of data that can be stored in HDFS. By default, the block size is 128 MB.
    • A block in HDFS is the basic unit of storage. Each block is typically 128 MB or 256 MB in size, and each file is broken up into multiple blocks that are distributed across the cluster.
14. What is a TaskTracker in Hadoop?
    • TaskTracker is the node that executes tasks assigned by the JobTracker in Hadoop.
    • The TaskTracker is a node in the Hadoop cluster that is responsible for executing tasks assigned by the JobTracker.
    • It manages the execution of MapReduce tasks and provides status updates to the JobTracker.
    • A TaskTracker in Hadoop is responsible for executing the Map and Reduce tasks assigned to it by the JobTracker.
    • It reports its progress back to the JobTracker and handles any failures that occur during execution.
    • It is responsible for executing Map and Reduce tasks, reporting progress to the JobTracker, and handling task failures.
15. What is a JobTracker in Hadoop?
    • JobTracker is the master node that manages the execution of jobs in a Hadoop cluster.
    • The JobTracker is the central component of the MapReduce framework in Hadoop and is responsible for scheduling and monitoring MapReduce jobs across the cluster.
    • It tracks the progress of jobs and communicates with the TaskTrackers to manage the execution of tasks.
    • A JobTracker in Hadoop is the central node in a Hadoop cluster that manages job scheduling, resource allocation, and task tracking.
    •  It is responsible for dividing a MapReduce job into tasks and assigning those tasks to TaskTrackers.
16. What is the difference between Hadoop and traditional relational databases?
    • Hadoop is designed to handle large volumes of unstructured and semi-structured data, while traditional relational databases are designed for structured data.
    • Hadoop is also more scalable and cost-effective for handling large datasets.
    • Hadoop is optimized for batch processing and can handle data in various formats, while traditional relational databases are optimized for real-time transactional processing.
17. What is a combiner in Hadoop?
    • A combiner is a mini-reduce phase that runs on the output of the map phase in Hadoop, and it is used to reduce the amount of data transferred between the map and reduce phases.
    • The Combiner can help reduce the amount of data that needs to be shuffled and sorted during the Reduce phase, which can improve the overall performance of the MapReduce job.
    • Its purpose is to reduce the amount of data sent over the network by combining the intermediate key-value pairs produced by the Mapper.
    • A Combiner is a component of the MapReduce framework in Hadoop that performs local aggregation of the output of Map tasks before sending it to the Reduce tasks.
    • It reduces the amount of data that needs to be transferred across the network, which can improve the performance of the job.
18. What is a partitioner in Hadoop?
    • A partitioner is used to divide the output of the map phase into partitions before sending it to the reduce phase in Hadoop.
    • A Partitioner is a component of the MapReduce framework in Hadoop that is responsible for assigning the output of Map tasks to Reduce tasks.
    • It ensures that all data with the same key is processed by the same Reduce task.
19. What is the difference between Hadoop 1 and Hadoop 2?
    • Hadoop 1 consists of HDFS, MapReduce, and JobTracker, while Hadoop 2 includes HDFS, YARN, and MapReduce. YARN is a resource manager that manages resources in a Hadoop cluster and schedules tasks.
20. What is a rack in Hadoop?
    • A rack is a group of nodes in a Hadoop cluster that are located physically close to each other and are connected to a single network switch.
21. What is the difference between block and input split in Hadoop?
    • A block is the smallest unit of data that can be stored in HDFS, while an input split is the portion of a file that is processed by a single map task in Hadoop.
22. What is speculative execution in Hadoop?
    • Speculative execution is a mechanism in Hadoop that runs multiple instances of the same task on different nodes in a Hadoop cluster to ensure that the task is completed as quickly as possible.
23. What is the purpose of the Hadoop Streaming API?
    • The Hadoop Streaming API is used to allow non-Java programs to work with Hadoop’s MapReduce framework. It enables MapReduce programs to be written in languages other than Java.
24. What is a Pig in Hadoop?
    • Pig is a high-level scripting language used for querying and analyzing large datasets in Hadoop. It provides a simpler and more concise way of writing MapReduce programs.
25. What is a Hive in Hadoop?
    • Hive is a data warehousing framework used for querying and analyzing large datasets in Hadoop. It provides a SQL-like interface for querying data stored in Hadoop.
26. What is a ZooKeeper in Hadoop?
    • ZooKeeper is a distributed coordination service used for managing configuration information and providing distributed synchronization in a Hadoop cluster.
27. What is a Flume in Hadoop?
    • Flume is a distributed data collection and processing system used for transferring large amounts of data from various sources to Hadoop.
28. What is a Sqoop in Hadoop?
    • Sqoop is a tool used for transferring data between Hadoop and relational databases.
29. What is the purpose of the Hadoop Distributed Cache?
    • The Hadoop Distributed Cache is used for distributing files, archives, and other resources to the nodes in a Hadoop cluster. It is used to improve the performance of MapReduce programs by making necessary files available to all nodes in the cluster.
30. What is a checkpoint in Hadoop?
    • A checkpoint is a backup copy of the NameNode’s namespace, which is stored on the HDFS.
31. What is a block report in Hadoop?
    • A block report is a report sent by the DataNode to the NameNode that contains information about the data blocks it is storing.
32. What is the difference between a mapper and a reducer in Hadoop?
    • A mapper is a function that processes input data and produces intermediate key-value pairs, while a reducer is a function that processes the intermediate key-value pairs and produces final output.
33. What is a container in YARN?
    • A container is a runtime environment in YARN that provides resources such as CPU, memory, and disk to run a task.
34. What is a heartbeat in Hadoop?
    • A heartbeat is a signal sent by a node in a Hadoop cluster to indicate that it is still alive and functioning properly.
35. What is the purpose of speculative execution in Hadoop?
    • Speculative execution is used to prevent slow nodes or tasks from slowing down the entire job by running multiple instances of the same task on different nodes in the cluster.
36. What is a NameNode checkpoint in Hadoop?
    • A NameNode checkpoint is a process that creates a backup of the NameNode’s metadata, which is stored on the HDFS.
37. What is the difference between Hadoop and Spark streaming?
    • Hadoop streaming is used to process batch data, while Spark streaming is used to process real-time data.
38. What is Hadoop Common?
    • Hadoop Common is a set of common utilities and libraries used by all the Hadoop components.
39. What is the difference between a local file system and HDFS?
    • A local file system is designed to store and access files on a single machine, while HDFS is designed to store and access large datasets across a distributed network of machines. HDFS is optimized for handling big data, while a local file system is optimized for handling small data.
40. What is the role of a secondary NameNode in Hadoop?
    • The secondary NameNode in Hadoop is used to take periodic snapshots of the NameNode’s metadata and store it in a separate location. This is done to prevent data loss in case of NameNode failure.