Prerequisites and WSL2 Check
Before diving into the Hadoop installation, ensure your system meets the requirements. You'll need Windows 10 (build 19041+) or Windows 11, and WSL2 must be enabled with an Ubuntu distribution installed. The process typically takes 20-30 minutes and requires approximately 3GB of free disk space. It is crucial to verify your WSL version. Open PowerShell or Command Prompt and run wsl -l -v. If your Ubuntu distribution is not listed as VERSION 2, you must upgrade it. Use the command wsl --set-version Ubuntu 2, replacing "Ubuntu" with your distribution's name if different. This step is vital as many performance benefits and compatibility features of Hadoop on WSL are tied to WSL2.
Downloading and Installing Hadoop 3.5.x
Generic Linux guides often point to older Hadoop versions. This guide focuses on the current 3.5.x line. First, update your package lists and install necessary dependencies:
sudo apt update
sudo apt upgrade -y
sudo apt install -y openjdk-11-jdk wget tar ssh
Next, download the latest Hadoop 3.5.x distribution. Visit the official Apache Hadoop releases page to find the most recent stable version. Use wget to download the tarball directly to your Ubuntu environment. For example:
wget https://dlcdn.apache.org/hadoop/common/hadoop-3.5.1/hadoop-3.5.1.tar.gz
After downloading, extract the tarball to a suitable location, typically /usr/local or your home directory. Then, create a symbolic link for easier management.
tar -xvzf hadoop-3.5.1.tar.gz
sudo mv hadoop-3.5.1 /usr/local/hadoop
ln -s /usr/local/hadoop /usr/local/hadoop-current
This setup ensures you're working with the latest stable release, avoiding compatibility issues found in older versions.
Configuring Environment Variables
Hadoop relies on several environment variables being correctly set. The most critical is JAVA_HOME. Open your shell profile file (e.g., ~/.bashrc or ~/.zshrc) with a text editor:
nano ~/.bashrc
Add the following lines to the end of the file, adjusting the Java path if your JDK installation differs:
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop-current
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
Save the file and source it to apply the changes:
source ~/.bashrc
Verify the settings by running echo $JAVA_HOME and hadoop version. Incorrect JAVA_HOME is a common pitfall, especially on WSL where Java installations can be non-standard.
Hadoop Single Node Configuration
For initial setup and testing, configuring Hadoop in standalone (single-node) mode is sufficient. This mode does not require a distributed cluster. The default configuration often runs in this mode, but it's good practice to verify. Edit the core-site.xml and hdfs-site.xml files located in $HADOOP_CONF_DIR.
In core-site.xml, specify the default filesystem URI:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
In hdfs-site.xml, define the DFS data directory. A common practice is to create a hadoop-data directory within your home folder. This avoids permission issues often encountered when writing directly to the Windows filesystem from WSL.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file://${HOME}/hadoop-data/hdfs/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file://${HOME}/hadoop-data/hdfs/datanode</value>
</property>
</configuration>
Create the necessary directories:
mkdir -p $HOME/hadoop-data/hdfs/namenode
mkdir -p $HOME/hadoop-data/hdfs/datanode
The decision to use the user's home directory within WSL for HDFS data is critical. Writing directly to the mounted Windows filesystem (e.g., /mnt/c/Users/...) often results in severe I/O performance degradation and can cause Hadoop processes to hang or fail. Keeping data within the WSL2 Linux filesystem provides near-native performance.
SSH Configuration
Hadoop daemons (NameNode, DataNode, ResourceManager, NodeManager) communicate using SSH. You need to ensure passwordless SSH is set up between localhost and your user. Start by generating an SSH key pair if you haven't already:
ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
sudo chmod 0600 ~/.ssh/authorized_keys
Test passwordless SSH by running ssh localhost. It should connect without prompting for a password. If SSH fails to start or prompts for a password, Hadoop daemons will not launch correctly. Common issues include incorrect permissions on ~/.ssh or ~/.ssh/authorized_keys, or SSH service not running properly within WSL.
Initializing and Starting Hadoop
With the configuration complete, you can initialize the HDFS NameNode. This formats the NameNode's storage directory:
hdfs namenode -format
Once formatted, start the HDFS daemons:
start-dfs.sh
Then, start the YARN ResourceManager:
start-yarn.sh
Verify that the daemons are running using jps. You should see NameNode, DataNode, ResourceManager, and NodeManager processes listed. If any of these fail to start, check the Hadoop logs in $HADOOP_HOME/logs for specific error messages. SSH issues and incorrect environment variables are frequent culprits.
Testing the Installation
To confirm the installation, create a directory in HDFS, copy a local file into it, and then list the HDFS directory contents.
hdfs dfs -mkdir /user/$USER
hdfs dfs -put $HADOOP_HOME/README.txt /
hdfs dfs -ls /
This sequence verifies basic HDFS read/write operations. You can also access the Hadoop web UIs by navigating to http://localhost:9870 for HDFS and http://localhost:8088 for YARN in your Windows browser. These interfaces provide valuable insights into the cluster's status and running applications.
Troubleshooting Common WSL2 Issues
WSL2 introduces unique challenges compared to a bare-metal Linux installation. The primary issues encountered are:
- SSH Not Starting: Ensure the SSH daemon is running and configured for passwordless login. Sometimes, restarting WSL (`wsl --shutdown` in PowerShell) can resolve transient SSH issues.
- Slow HDFS I/O: As mentioned, avoid writing HDFS data directly to the Windows filesystem. Keep all Hadoop data and logs within the WSL2 Linux filesystem (e.g., in your home directory) for optimal performance.
- JAVA_HOME Misconfiguration: Double-check the
JAVA_HOMEpath in your shell profile. Ensure it points to a valid JDK installation within WSL. - Port Conflicts: While less common on a fresh WSL install, ensure no other Windows applications are using the default Hadoop ports (e.g., 9000, 8088).
By addressing these points, you can establish a robust Hadoop environment on WSL2 Ubuntu, ready for development and testing.