Python 3 with pip

This section will demonstrate how to set up a virtual environment using python3 with the pip package manager.

Step 1: Get Home

First change into your home directory by typing the following command:

cd

You can always use the following command to check the current directory you are in.

pwd

Step 2: Set the Python Environment

Before creating the virtual environment, check which modules are loaded using module list. Load any additional modules needed for your workflow with module load, and unload any unneccessary modules with module unload.

For this example, we use default (latest) version of Python …

module load python

Most versions of Python available on HPC will have the virtualenv feature available for use globally, but it’s probably still a good idea to create an instance of the feature in our home directory. The following pip command uses the --user option, which will install virtualenv to a location in our home directory.

pip3 install --user virtualenv

Console: /bin/bash

$ pip3 install –user virtualenv

Requirement already satisfied: virtualenv in /mmfs1/tools/python-3.9.2/lib/python3.9/site-packages (20.4.3)
Requirement already satisfied: six<2,>=1.9.0 in /mmfs1/tools/python-3.9.2/lib/python3.9/site-packages (from virtualenv) (1.15.0)
Requirement already satisfied: filelock<4,>=3.0.0 in /mmfs1/tools/python-3.9.2/lib/python3.9/site-packages (from virtualenv) (3.0.12)
Requirement already satisfied: appdirs<2,>=1.4.3 in /mmfs1/tools/python-3.9.2/lib/python3.9/site-packages (from virtualenv) (1.4.4)
Requirement already satisfied: distlib<1,>=0.3.1 in /mmfs1/tools/python-3.9.2/lib/python3.9/site-packages (from virtualenv) (0.3.1)

If the command is successful, you should be able to see what files were installed by looking into a special hidden path (“.local”) in your home directory. The pip command, when used with --user, creates this path to hold user specific python files and configuration items …

ls -al ~/.local/lib/python-<version>/site-packages

Caution

pip3 install --user <package_name> is similar to python virtual environments, in that it creates a location in your home directory where you can install modules. However, it’s important to note that simply using pip --user does not generate all of the necessary changes needed to work in a fully isolated virtual environment.

Step 3: Create the Virtual Environment

With virtualenv installed, we need to create a new location in our home directory to hold the virtual environment files. You may end up with multiple virtual environments for various projects, so it’s a good idea give the new path a descriptive name (e.g. pytorch_project1). Here we just call our environment “env1” as a generic identifier, but you can name the path whatever you like.

mkdir ~/env1
cd ~/env1/

From within the new directory, we can now create a new virtual environment by telling python to copy all of the essential files to a location within our new project folder (“env1”). We use “env1_python” here, but you can name it whatever you want …

python3 -m virtualenv env1_python

Because Python has to perform a number of file copies in order to create the virtual environment, this command may take a little while to complete. We can see what has been copied by looking into the newly created “env1_python” directory …

ls -al env1_python
ls -al env1_python/bin
ls -al env1_python/lib/python<version>

Step 4: Activate the Virtual Environment

You may notice that several “activate” scripts have been created for us in env1_python/bin. These files perform all of the necessary tasks to ensure that your Linux shell is configured with all of the appropriate paths, libraries, etc. needed to run in the isolated virtual environment. In order to actually switch to this specific execution context, we use the source command along with the location of the activate script …

source env1_python/bin/activate

If we take a look inside env1_python/bin/activate we can see that the script manipulates a few environment variables to ensure that we only reference the python locations specific to virtual environment …

VIRTUAL_ENV

set

~/env1/env1_python

PATH

set

$VIRTUAL_ENV/bin:$PATH

PYTHONHOME

unset

NULL

With the environment now set after calling source, you should see that your prompt has changed to indicate the name of the virtual environment …

(env1_python) hpcuser@easley01:env1_python>

Step 5: Install Packages

From within this new Python execution context, we can now perform operations like we normally would for a locally installed python. Most importantly, you can now install packages using pip …

(env1_python) hpcuser@easley01:env1_python > pip3 install <package_name>

To confirm, we can take a look in the virtual environments dedicated site-packages directory, and we should see that any modules we have installed while activated are placed in that location …

(env1_python) hpcuser@easley01:env1_python > ls env/lib/python3.9/site-packages/

And, we can run interactively in our customized shell to test some code …

(env1_python) hpcuser@easley01:env1_python > python3

Before creating the virtual environment, check which modules are loaded, and load any additional modules needed. For this example use the latest version of Python 3. Move to the next section if you plan on using conda as the package manager.