Using Venv

Using Python & Venv¶

If you have a different use case or need/want to use conda you can also use so called "virtual environments" to install python packages on VSC.

There are other tools out there that utilize virtual environments (e.g. poetry, virtualenv, ...) but since python already comes with the venv module we will focus on this already available module.

Create a virtual env¶

To create a virtual environment first select a python version of your choice. You can either load one of the available VSC modules (module avail python/3) or install a different python version using conda.

For this example we will assume that the zen package tree is loaded.

After python is available we can then use the venv module to create a new virtual environment in the myvenv folder.

zen $ module load python/3.11.3-gcc-12.2.0-hn7p65z
zen $ which python
/gpfs/opt/sw/zen/spack-0.19.0/opt/spack/linux-almalinux8-zen3/gcc-12.2.0/python-3.11.3-hn7p65zs2brgmbpfme3f26s3k65tz56y/bin/python
zen $ python -m venv --upgrade-deps myvenv

Info

Specifying --upgrade-deps is optional but ensures that the contained pip installer has the most recent version.

Activate the virtual env¶

To activate the created virtual environment simply use source to execute the generated activate script in the current bash environment.

zen $ source myvenv/bin/activate
(myvenv) zen $ which python
/home/fs12345/vscuser/myvenv/bin/python

Checking the location of the used python binary should show that the python binary from the virtual environment is now used.

Install packages into the virtual env¶

Once the environment has been activated packages can be installed into it. The recommended way to do this is to create a requirements.txt file and then use pip to install all packages at once.

tensorflow==2.15

The packages can then be installed using pip:

(myvenv) zen $ pip install -r requirements.txt

...

Submit to slurm using a python virtual environment¶

Warning

Make sure that you submit using a clean environment. Environment variables of active environments might get taken into account when submitting slurm jobs and may produce unwanted side effects!

Create a slurm batch script that load the virtual environment and then executes a python script

#!/bin/bash
#SBATCH --job-name=slurm_venv_example
#SBATCH --time=00-00:05:00
#SBATCH --ntasks=2
#SBATCH --mem=2GB

source ~/myvenv/bin/activate

echo "Using python: $( python --version ) from $( which python )"

# now run your program using python from the virtual environment
python my_program.py

Virtual environment management¶

Deactivate the environment¶

To deactivate a venv use the deactivate command

(myvenv) zen $ deactivate
zen $

Removing virtual environments¶

To delete the virtual environment simply remove the folder:

zen $ rm -rf myvenv/