.. _python_env_pip: Python 3 with pip ================= This section will demonstrate how to set up a virtual environment using python3 with the pip package manager. Step 1: Get Home ++++++++++++++++ First change into your home directory by typing the following command: .. hpc-prompt:: hpcterm > cd You can always use the following command to check the current directory you are in. .. hpc-prompt:: hpcterm > pwd Step 2: Set the Python Environment ++++++++++++++++++++++++++++++++++ Before creating the virtual environment, check which modules are loaded using ``module list``. Load any additional modules needed for your workflow with ``module load``, and unload any unneccessary modules with ``module unload``. For this example, we use default (latest) version of Python ... .. hpc-prompt:: hpcterm > module load python Most versions of Python available on HPC will have the ``virtualenv`` feature available for use globally, but it's probably still a good idea to create an instance of the feature in our home directory. The following ``pip`` command uses the ``--user`` option, which will install ``virtualenv`` to a location in our home directory. .. hpc-prompt:: hpcterm > pip3 install --user virtualenv .. admonition:: Console: /bin/bash :class: terminal_small | $ pip3 install --user virtualenv | | Requirement already satisfied: virtualenv in /mmfs1/tools/python-3.9.2/lib/python3.9/site-packages (20.4.3) | Requirement already satisfied: six<2,>=1.9.0 in /mmfs1/tools/python-3.9.2/lib/python3.9/site-packages (from virtualenv) (1.15.0) | Requirement already satisfied: filelock<4,>=3.0.0 in /mmfs1/tools/python-3.9.2/lib/python3.9/site-packages (from virtualenv) (3.0.12) | Requirement already satisfied: appdirs<2,>=1.4.3 in /mmfs1/tools/python-3.9.2/lib/python3.9/site-packages (from virtualenv) (1.4.4) | Requirement already satisfied: distlib<1,>=0.3.1 in /mmfs1/tools/python-3.9.2/lib/python3.9/site-packages (from virtualenv) (0.3.1) If the command is successful, you should be able to see what files were installed by looking into a special hidden path (".local") in your home directory. The ``pip`` command, when used with ``--user``, creates this path to hold user specific python files and configuration items ... .. hpc-prompt:: hpcterm > ls -al ~/.local/lib/python-/site-packages .. Caution :: ``pip3 install --user `` is similar to python virtual environments, in that it creates a location in your home directory where you can install modules. However, it's important to note that simply using ``pip --user`` does not generate all of the necessary changes needed to work in a fully isolated virtual environment. Step 3: Create the Virtual Environment ++++++++++++++++++++++++++++++++++++++ With ``virtualenv`` installed, we need to create a new location in our home directory to hold the virtual environment files. You may end up with multiple virtual environments for various projects, so it's a good idea give the new path a descriptive name (e.g. pytorch_project1). Here we just call our environment "env1" as a generic identifier, but you can name the path whatever you like. .. hpc-prompt:: hpcterm > mkdir ~/env1 cd ~/env1/ From within the new directory, we can now create a new virtual environment by telling python to copy all of the essential files to a location within our new project folder ("env1"). We use "env1_python" here, but you can name it whatever you want ... .. hpc-prompt:: hpcterm > python3 -m virtualenv env1_python Because Python has to perform a number of file copies in order to create the virtual environment, this command may take a little while to complete. We can see what has been copied by looking into the newly created "env1_python" directory ... .. hpc-prompt:: hpcterm > ls -al env1_python ls -al env1_python/bin ls -al env1_python/lib/python Step 4: Activate the Virtual Environment ++++++++++++++++++++++++++++++++++++++++ You may notice that several "activate" scripts have been created for us in ``env1_python/bin``. These files perform all of the necessary tasks to ensure that your Linux shell is configured with all of the appropriate paths, libraries, etc. needed to run in the isolated virtual environment. In order to actually switch to this specific execution context, we use the ``source`` command along with the location of the ``activate`` script ... .. hpc-prompt:: hpcterm > source env1_python/bin/activate If we take a look inside ``env1_python/bin/activate`` we can see that the script manipulates a few environment variables to ensure that we only reference the python locations specific to virtual environment ... +---------------+---------+--------------------------------+ | VIRTUAL_ENV | set | ~/env1/env1_python | +---------------+---------+--------------------------------+ | PATH | set | $VIRTUAL_ENV/bin:$PATH | +---------------+---------+--------------------------------+ | PYTHONHOME | unset | NULL | +---------------+---------+--------------------------------+ With the environment now set after calling ``source``, you should see that your prompt has changed to indicate the name of the virtual environment ... .. hpc-prompt:: hpcterm >,(env1_python)...> auto (env1_python) hpcuser@easley01:env1_python> Step 5: Install Packages ++++++++++++++++++++++++ From within this new Python execution context, we can now perform operations like we normally would for a locally installed python. Most importantly, you can now install packages using pip ... .. hpc-prompt:: hpcterm >,(env1_python)...> auto (env1_python) hpcuser@easley01:env1_python > pip3 install To confirm, we can take a look in the virtual environments dedicated ``site-packages`` directory, and we should see that any modules we have installed while activated are placed in that location ... .. hpc-prompt:: hpcterm >,(env1_python)...> auto (env1_python) hpcuser@easley01:env1_python > ls env/lib/python3.9/site-packages/ And, we can run interactively in our customized shell to test some code ... .. hpc-prompt:: hpcterm >,(env1_python)...> auto (env1_python) hpcuser@easley01:env1_python > python3 Before creating the virtual environment, check which modules are loaded, and load any additional modules needed. For this example use the latest version of Python 3. Move to the next section if you plan on using conda as the package manager.