Introduction to Installing WSL and Ubuntu
This tutorial guides you through the process of installing Windows Subsystem for Linux (WSL), Ubuntu, and other necessary components for a privately installed conversational Large Language Model (LLM). The goal is to have this LLM insulated from the internet.
Enabling WSL in Control Panel
To start, navigate to Control Panel > Turn Windows features on or off. Ensure that:
Virtual Machine Platform
Windows Subsystem for Linux
are both turned on. Once these features are enabled, Windows will prompt you to restart your computer.
Post-Restart Steps
After restarting, open an elevated PowerShell terminal. Note that some installations may take a considerable amount of time; patience is key throughout this process.
Step #1—Install WSL and Ubuntu
An LLM’s size increases with use. Instead of the standard installation on the *C:* drive, consider dedicating a separate partition (e.g., 400GB) for Linux. To install Linux on a different drive:
In the PowerShell terminal, type set-location X: (where X is your preferred drive letter). The prompt will change to reflect this drive.
Then, execute wsl –install (that’s two dashes-it looks like one in this environment). This command likely includes the download and installation of Microsoft-tuned Ubuntu.
If WSL downloads and installs Ubuntu successfully, it might open a terminal and prompt you for a username and password. If not, we’ll address this shortly. Ensure that your PowerShell terminal remains open throughout this process.
Updating WSL (for Non-Windows 11 Users)
In Windows 11, WSL Version 2 is the default. For other versions of Windows:
Visit Microsoft’s documentation and follow the instructions to update manually.
Specifically, scroll down to Step 4—Download the Linux kernel update package, download the .msi package, and install it in Windows.
Then, back in PowerShell:
wsl –set-default-version 2
This sets the WSL default version to 2. You can either return to your Ubuntu terminal if it’s already activated or run Ubuntu from your elevated PowerShell using:
wsl -d Ubuntu
Step #2—Install the CUDA Toolkit
Note: This step is only required if you have an nVidia graphics card and wish to use the CUDA toolkit. The CUDA toolkit must be installed before any other packages in Ubuntu for Ollama to utilize it.
To install the CUDA toolkit, follow these steps in your Ubuntu terminal:
Download the CUDA repository configuration file:
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
This command downloads a configuration file necessary for the CUDA repository.
Move the configuration file to the correct location:
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
This step ensures that the CUDA repository is properly configured.
Download the CUDA repository package:
This command retrieves the package containing the CUDA repository.
Install the CUDA repository package:
sudo dpkg -i cuda-repo-wsl-ubuntu-12-8-local_12.8.1-1_amd64.deb
This step installs the CUDA repository, making its packages available for installation.
Copy the GPG key to the correct location:
sudo cp /var/cuda-repo-wsl-ubuntu-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
This command ensures that the package manager can verify the authenticity of CUDA packages.
Update the package list:
sudo apt-get update
This step updates the package list to include the newly added CUDA repository.
Install the CUDA toolkit:
sudo apt-get -y install cuda-toolkit-12-8
This final command installs the CUDA toolkit, which is necessary for Ollama to utilize GPU acceleration.
Step #3—Complete the Installation of the Ubuntu App and Update Ubuntu
Downloading the Linux App Package
To proceed, you’ll need to download the Linux app package. Open PowerShell and run the following command to pull the package from the Microsoft Store:
curl.exe -L -o Linux.appx https://wslstorestorage.blob.core.windows.net/wslblob/Ubuntu2404-240425.AppxBundle
Next, install the package using the following command:
Add-AppxPackage .\Linux.appx
Find and double-click the .appx file in Windows File Explorer to initiate the installation.
Important Note: When double-clicking on the .appx file, you might encounter an encrypted file/permissions error message. If this happens, try again after a few moments, as it may resolve itself.
Installing Linux
Once the installation process begins, a Linux terminal will appear on your screen. You’ll be prompted to create a username and password (if you haven’t already—otherwise just repeat the process with the username/password you’ve already established). After setting up your credentials, you’ll enter the Linux environment.
Updating Linux
To update Linux after installation, run the following commands:
sudo apt update
sudo apt full-upgrade -y
You’ll be asked for your password during this process.
Moving the Linux App
By default, the Linux app is installed on the C: drive. To avoid consuming space on your primary drive, you can move it to a different location. Here’s how:
Exit the Linux terminal.
Open an elevated PowerShell and shut down WSL using the command:
wsl –shutdown (again, two dashes)
Go to Settings > Apps > Installed Apps and find the Linux app.
Click the options icon (three stacked dots) on the right side, then select “Move”.
Choose your different drive you set in the beginning from the dropdown menu.
Alternative Method: If the move fails or you prefer a more manual approach, you can relocate the ext4.vhdx virtual drive (usually located in C:\Users\Admin\AppData\Local\wsl\{GUID}) to your preferred location (my preferrence is to copy/paste the VHD, then rename the original to 0ext4.vhdx) and update the registry keys:
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss\{GUID} (change the BasePath to your drive)
HKEY_USERS\S-1-5-21-{GUID}\Software\Microsoft\Windows\CurrentVersion\Lxss\{GUID} (change the BasePath to your drive)
Restoring Previous Versions: Moving Linux allows for easier cleanup and restoration of previous versions. You can create drive images of the installation partition at each successful step, making it simpler to recover from mistakes.
Launching Linux
Your Linux installation will appear in the Apps menu. However, if moving ext4.vhdx breaks the shortcut, you can create a desktop shortcut using the following command:
“C:\Windows\System32\wsl.exe -d Ubuntu su – (username)”
Replace <username> with your actual Linux username.
Step #4—Install Docker
To install Docker on Ubuntu in the Linux terminal, follow these steps:
Add Docker’s Official GPG Key and Repository
First, update the package list and install the necessary packages:
sudo apt-get update
sudo apt-get install ca-certificates curl
Next, create a directory for the GPG key and download the key:
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
Now, add the Docker repository to the APT sources running this batch file:
echo \
“deb [arch=$(dpkg –print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo “${UBUNTU_CODENAME:-$VERSION_CODENAME}”) stable” | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Update the package list again:
sudo apt-get update
Install Docker
With the repository added, install Docker using the following command:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
This will install the necessary packages for Docker.
Start Docker
Once the installation is complete, start the Docker service:
sudo service docker start
You’ll likely be prompted to enter your password, but might not.
Step #5—Install Ollama
Now that Docker is installed and running, use it to install Ollama. Run the following command in the Linux terminal:
curl -fsSL https://ollama.com/install.sh | sh
This script will download and install Ollama.
Installation Process
During the installation process, Ollama will automatically detect your system’s configuration, including your GPU and the CUDA Toolkit. You’ll see messages indicating that it has found these components.
Note: The installation script may take a few minutes to complete, depending on your internet connection and system specifications. Be patient and let the script run its course.
Step #6—Pull an LLM Model
With Ollama installed, use it to pull a Large Language Model (LLM) from the Ollama library. The models vary in size, so choose one that suits your needs.
Pulling the Model
To pull an LLM model using Ollama (I chose Llama3.3:70B), run the following command:
ollama pull llama3.3:70B
Replace llama3.3:70B with the name of the model you want to pull. In this example, we’re pulling the Llama3.3 model with a size of 70 billion parameters.
Note: The model download process may take some time, depending on your internet connection and the size of the model.
Running a Trial Chat
Once the model is pulled, you can try a trial chat using Ollama. Run the following command to start a conversation:
ollama run llama3.3:70B
Again, replace llama3.3:70B with the name of the model you pulled.
This will launch an interactive chat session with the LLM model, allowing you to test its capabilities and explore its features.
Tips:
Make sure you have enough disk space available for the model, as some models can be quite large.
Be patient during the download process, as it may take several minutes or even hours depending on your internet connection.
Experiment with different models and sizes to find the one that best suits your needs.
Step #7—Install OpenWebUI
To access your AI model through a browser interface, install OpenWebUI. This will provide a completely local and private experience.
Shut Down Ollama
Before installing OpenWebUI, shut down Ollama using the following command:
ollama stop llama3.3:70B
(substitute your model for llama3.3:70B.)
This ensures that Ollama is not running while you install OpenWebUI.
Install OpenWebUI
To install OpenWebUI, run the following Docker command:
sudo docker run -d –network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 –name open-webui –restart always ghcr.io/open-webui/open-webui:main
This command:
Runs OpenWebUI in detached mode (-d)
Uses the host network (–network=host) for communication between containers
Maps a volume (-v) to persist data
Sets an environment variable (-e) to specify the Ollama base URL
Names the container (–name) and sets it to restart automatically (–restart always)
Access OpenWebUI
Once installed, access OpenWebUI by navigating to http://localhost:8080 in your web browser.
Initial Setup
When you first open OpenWebUI, you’ll be prompted to create an account. This will serve as the administrator account for your local AI setup. Your chats will be saved in a chat log, which can be accessed by opening the column on the left side of the page.
Tips:
Make sure you have Docker installed and running before attempting to install OpenWebUI.
If you encounter any issues during installation, check the Docker logs for errors or try restarting the container.
Explore the OpenWebUI interface to familiarize yourself with its features and settings.
And that’s it. I’ve followed this tutorial twice without hiccups.
-
This topic was modified 1 month, 1 week ago by
.
-
This topic was modified 1 month, 1 week ago by
-
This topic was modified 1 month, 1 week ago by
-
This topic was modified 1 month, 1 week ago by
