Using Conda
Using Python via Conda¶
Conda & Anaconda & Mamba¶
-
conda
is the package manager core. -
miniconda
is the official minimal installer that provides the conda toolset without any conda packages. -
anaconda
is a proprietary (unfree!) summary of packages. avoid unless you know how the licensing works. -
mamba
is an alternative implementation of the conda tool. mamba was the faster choice in the past but the mamba solver (libdnf) is now the default solver in theconda
tool. -
micromamba
is a reimplementation ofmamba
in C++ with minimal dependencies.
For our purposes we will focus on conda
via miniconda
.
Start Conda¶
First load the miniconda package with module
:
zen $ module load miniconda3
Execute the conda bash hook to fully setup the conda tool:
zen $ eval "$(conda shell.bash hook)"
(base) zen $ conda
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda is a tool for managing and deploying applications,
environments and packages.
...
Note how the prompt now shows the base env. This signifies that the default base environment is loaded.
Note
The VSC's base env is write protected since it is the conda
tools environment for it to run.
In general conda's base env should never be used for user installations.
Define your environment¶
It is convenient to create a yaml
file defining your environment.
name: myenv
channels:
- conda-forge
dependencies:
- python=3.10
- tensorflow=2.15.0
Create your environment¶
Create your conda env:
(base) zen $ conda env create --file myenv.yaml
Channels:
- conda-forge
- bioconda
Platform: linux-64
Collecting package metadata (repodata.json): /
...
Depending on the number of packages, solving and installing the environment might take some time (~ a couple of minutes).
Warning
Conda creates many small files and might fill up your $HOME
file system quickly.
You might want to store your environments in your project storage.
Use the --prefix <directory>
argument with conda env create
to use a different directory.
Activate your environment¶
Activate your conda env, note how the prompt changes:
(base) zen $ conda activate myenv
...
(myenv) zen $ python --version
Python 3.10.11
The python version shown is the installation from conda.
Note
Instead of using the environment's name you can also specify a path.
(Optional) Adding pip
packages¶
Warning
In general the manual installation of additional packages using pip should be avoided since pip packages have their own dependencies and can even pull in unwanted binaries. It is preferrable to either use a corresponding conda package or at least specify the pip packages directly in the conda env file.
name: myenv
channels:
- conda-forge
dependencies:
- python=3.10
- tensorflow=2.15.0
- pip
- pip:
- memory-profiler
Manually installing pip
packages¶
When installing a package using pip, make sure that you have an active conda environment (e.g. myenv
).
If you do not have an active conda environment, the packages will be installed into the default folder
of your user (~/.local/python
) and could interfere with other python environments.
(myenv) zen $ python -m pip install mypackage
Use python -m pip -v
to display the installed packages and their installations paths to make sure they end up in the right place.
Submit to slurm using a Conda environment¶
Warning
Make sure that you submit using a clean environment. Environment variables of active environments might get taken into account when submitting slurm jobs and may produce unwanted side effects!
Create a slurm batch script activating your conda env and running python:
#!/bin/bash
#SBATCH --job-name=slurm_conda_example
#SBATCH --time=00-00:05:00
#SBATCH --ntasks=2
#SBATCH --mem=2GB
module load miniconda3
eval "$(conda shell.bash hook)"
conda activate myenv
echo "Using python: $( python --version ) from $( which python )"
# now run your program using python from the conda environment
python my_program.py
Conda environment management¶
List all environments¶
(base) zen $ conda env list
# conda environments:
#
myenv /home/fs70824/trainee00/.conda/envs/myenv
base * /opt/sw/conda
py310 /opt/sw/conda/envs/py310
py311 /opt/sw/conda/envs/py311
py312 /opt/sw/conda/envs/py312
py38 /opt/sw/conda/envs/py38
py39 /opt/sw/conda/envs/py39
The star *
shows the currently active environment.
Environments that are located in /opt/sw/conda
are not writeable by users.
Deactivate your environment¶
Deactivate your conda environment to get back to (base)
:
(myenv) zen $ conda deactivate
conda deactivate
again to get out of conda completely.
(base) zen $ conda deactivate
zen trainee00@l55:~$
Remove your environment¶
If you don't need them anymore please remove any old environments:
(base) zen trainee00@l55:~$ conda env remove -n myenv
Remove all packages in environment /home/fs70824/trainee00/.conda/envs/myenv:
Everything found within the environment (/home/fs70824/trainee00/.conda/envs/myenv),
including any conda environment configurations and any non-conda files, will be deleted.
Do you wish to continue? (y/[n])?