Python Package Installation

Description

Python packages on the cluster are primarily managed with the Mamba python package manager. Direct use of pip, outside of a virtual environment, is discouraged on the FASRC clusters.

Mamba is a package manager that is a drop-in replacement for Conda, and is generally faster and better at resolving dependencies:

Speed: Mamba is written in C++, which makes it faster than Conda. Mamba uses parallel processing and efficient code to install packages faster.
Compatibility: Mamba is fully compatible with Conda, so it can use the same commands, packages, and environment configurations.
Cross-platform support: Mamba works on Mac, Linux and Windows.
Dependency resolution: Mamba is better at resolving dependencies than Conda.
Environment creation: Mamba is faster at creating environments, especially large ones.
Package repository: Mamba uses Mambaforge (aka conda-forge), the most up to date packages available.

Important:
repo.anaconda.com, which hosts the Anaconda Distribution / Miniconda default channels, is blocked at the network level on the FASRC cluster due to licensing restrictions. Use the FASRC python environment module, which is built with Miniforge3, and has conda-forge set as its default channel.

Mamba uses the same commands and configuration options as conda. You can swap almost all commands between conda and mamba. By default, mamba uses the conda-forge, the free Mambaforge package repository. (In this doc, we will generally only refer to mamba.)

Mamba usage

mamba is available on the FASRC cluster as a software module python. You can access mamba by loading either of the following modules:

$ module load python

To see Python’s version

$ python --version

On the FASRC clusters, we recommend creating your own environment and installing the python packages that you need inside that environment. See the Environments section for how to create an environment.

Mamba commands

For additional features, please refer to the Mamba documentation.

To create an environment:

$ mamba create -n <ENV_NAME>

Alternatively, you can create an environment and install packages at the same time. This ensures package dependencies are met and could also speed up your setup time significantly. The general syntax to create an environment and install packages is:

$ mamba create -n <ENV_NAME>

To activate an environment:

$ source activate python_env1

To deactivate the current active environment

$ source deactivate

To list packages inside the environment:

$ mamba list

To install new packages in the environment (optional: -y is to proceed with installation):

$ mamba install -y <PACKAGE>

To install a package from a specific channel, add --channel (or -c) argument:

$ mamba install --channel <CHANNEL-NAME> <PACKAGE>

To uninstall packages:

$ mamba uninstall <PACKAGE>

To delete an environment:

$ conda remove -n <ENV_NAME> --all

Environments

You can create a virtual environment with mamba in the same way as with conda. However, it is important to start an interactive session prior to creating an environment and installing desired packages in the following manner:

$ salloc --partition test --nodes=1 --cpus-per-task=2 --mem=8GB --time=0-02:00:00
$ module load python
$ mamba create -n <ENV_NAME>

Environments in a lab share

By default, Python packages are installed in your home directory, in ~/.conda/envs. If you would like to locate the packages elsewhere, like a Lab shared directory, then specify the absolute file path.

export CONDA_PKGS_DIRS=/<FILEPATH>/conda/pkgs
export CONDA_ENVS_PATH=/<FILEPATH>/conda/envs

Example: Environment creation from start to finish

Below, we show step-by-step how to:

request an interactive job
create a mamba environment in holylabs
install packages in the environment

Note: lines that start with # are comments, not commands to execute

# request interactive job
[jharvard@boslogin06 ~]$ salloc --partition test --nodes=1 --cpus-per-task=2 --mem=8GB --time=0-02:00:00
salloc: Granted job allocation 47054029
salloc: Waiting for resource configuration
salloc: Nodes holy8a24502 are ready for job

# load python module
[jharvard@holy8a24502 ~]$ module load python

# create directories
[jharvard@holy8a24502 ~]$ mkdir -p /n/holylabs/jharvard_lab/Lab/conda/envs
[jharvard@holy8a24502 ~]$ mkdir -p /n/holylabs/jharvard_lab/Lab/conda/pkgs

# set environmental variables
#   CONDA_ENVS_PATH: where the environment will be saved
#   CONDA_PKGS_DIRS: where temporary files are downloaded to 
[jharvard@holy8a24502 ~]$ export CONDA_ENVS_PATH=/n/holylabs/jharvard_lab/Lab/conda/envs
[jharvard@holy8a24502 ~]$ export CONDA_PKGS_DIRS=/n/holylabs/jharvard_lab/Lab/conda/pkgs

# create the environment
[jharvard@holy8a24502 ~]$ mamba create -n python_env pip wheel python=3.14 pandas
Using Cached Shard Index for conda-forge/linux-64 ✔ Done

... omitted output ...

warning libmamba Security Warning: This transaction includes executing package scripts (pre/post-link/unlink) if present. These scripts can contain arbitrary code. Please ensure you trust the package sources.
Confirm changes: [Y/n] Y

# activate environment
[jharvard@holy8a24502 ~]$ mamba activate python_env

# install additional packages (optional: -y is to proceed with installation)
(python_env) [jharvard@holy8a24502 ~]$ mamba install matplotlib -y

# install ipykernel to use environment on Jupyter (optional: -y is to proceed with installation)
(python_env) [jharvard@holy8a24502 ~]$ mamba install ipykernel -y

# deactivate environment
(python_env) [jharvard@holy8a24502 ~]$ source deactivate
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.

# end/exit interactive job (you get back to a login node)
[jharvard@holy8a24502 ~]$ exit
exit
salloc: Relinquishing job allocation 47054029
[jharvard@boslogin06 ~]$

Batch job with mamba environment

Now that you have an environment, you can use it in batch jobs by adding these two commands in your slurm batch script before you execute the python code:

module load python
source activate <ENV_NAME>

Below is the example script run_python.sbatch that runs numpy_pandas_ex.py, a Python code that uses numpy and pandas packages:

#!/bin/bash
#SBATCH -J np_pandas # job name
#SBATCH -o np_pandas.out # standard output file
#SBATCH -e np_pandas.err # standard error file
#SBATCH --cpus-per-task=1 # number of cores 
#SBATCH --partition=test # partition
#SBATCH --time=0-01:00 # time in D-HH:MM
#SBATCH --mem=10G # memory in GB

# Load required modules
module load python

# Activate the environment 
export CONDA_ENVS_PATH=/n/holylabs/jharvard_lab/Lab/conda/envs
mamba activate python_env

# Run program
python numpy_pandas_ex.py

Ensure that you submit a batch job from outside an environment (i.e., an environment is not activate)

[jharvard@boslogin06 ~]$ sbatch run_python.sbatch

Best Practices

Pip Installs

Avoid using pip outside (before you source an environment) of a mamba environment on any FASRC cluster. If you run pip install outside of a mamba environment, the installed packages will be placed in your $HOME/.local directory, which can lead to package conflicts and may cause some packages to fail to install or load correctly via mamba.

For example, if your environment name is python_env1, the correct way to use pip is:

$ module load python
$ source activate python_env1
$ pip install <package_name>

Use `mamba` environment in Jupyter Notebooks

If you would like to use a mamba environment as a kernel in a Jupyter Notebook on Open OnDemand (Cannon OOD or FASSE OOD), you have to install packages, ipykernel (for Python; for other languages, install the corresponding conda package for the supported Jupyter kernel). This packages will allow Jupyter to detect mamba environments that you created from the command line.

For example, if your environment name is python_env1:

$ module load python
$ source activate python_env1
$ mamba install ipykernel

After these packages are installed, launch a new Jupyter Notebook job (existing Jupyter Notebook jobs will fail to “see” this environment). Then:

Open a Jupyter Notebook (a .ipynb file)
On the top menu, click Kernel -> Change kernel -> select the conda environment

Mamba environments in a desired location

With mamba, use the -p or --prefix option to specify writing environment files to a desired location, such as the holylabs location. Don’t use your home directory as it has very low performance due to filesystem latency. Using a lab share location, you can also share your conda environment with other people on the cluster. Keep in mind, you will need to make the destination directory, and specify the python version to use. For example:

$ mamba create --prefix /n/holylabs/{YOUR_LAB}/Lab/<envname> python={PYTHON_VERS}

$ source activate /n/holylabs/{YOUR_LAB}/Lab/<envname>

To delete an environment at that desired location: $ conda remove -p /n/holylabs/{YOUR_LAB}/Lab/<envname> --all -y

Troubleshooting

Interactive vs. batch jobs

If your code works in an interactive job, but fails in a slurm batch job,

You are submitting your jobs from within a mamba environment.
Solution 1: Deactivate your environment with the command mamba deactivate and submit the job or
Solution 2: Open another terminal and submit the job from outside the environment.
Check if your ~/.bashrc or ~/.bash_profile files have a section of conda initialize or a source activate command. The conda initialize section is known to create issues on the FASRC clusters.
Solution: Delete the section between the two conda initialize statements. If you have source activate in those files, delete it or comment it out.
For more information on ~/.bashrc files, see https://docs.rc.fas.harvard.edu/kb/editing-your-bashrc/

Jupyter Notebook or JupyterLab on Open OnDemand/VDI problems

See Jupyter Notebook or JupyterLab on Open OnDemand/VDI troubleshooting section.

Unable to install packages

If you are not able to install packages or the package installation is taking a significantly long time, check your ~/.condarc file. As stated in Conda docs, this is an optional runtime configuration file. One can use this file to configure conda/mamba to search from specific channels for package installation.

We recommend users not have this file or keep it empty. This allows users to install packages in their conda/mamba environments using the defaults provided by the open-source distribution, Miniforge , that we have made available to our users via our newer Python modules.

If, for any reason, ~/.condarc exists in your cluster profile then check its contents. If the default channel is showing up as conda , edit it to conda-forge so that your ~/.condarc uses this open-source channel for package installation.

Similarly, if you had created an environment a long time ago using the Anaconda distribution and it is no longer working, then it is best to create a new environment using the open-source distribution as described above while ensuring that ~/.condarc, if exists, is pointing to conda-forge as its default channel.

For example, if you created a conda environment using one of our older Python modules, say Anaconda2/2019.10-fasrc01, you can see that conda is configured to use repo.anaconda.com for package installation.

$ module load Anaconda2 
$ conda info 
... 
channel URLs : 
https://repo.anaconda.com/pkgs/main/linux-64 
https://repo.anaconda.com/pkgs/main/noarch 
https://repo.anaconda.com/pkgs/r/linux-64 
https://repo.anaconda.com/pkgs/r/noarch 
...

In order to change this configuration, you can execute the conda config command to ensure that conda now points to conda-forge. This would also create a .condarc file in your $HOME, if it already doesn’t exist:

$ conda config --add default_channels https://conda.anaconda.org/conda-forge/ 

$ cat ~/.condarc 
default_channels: - https://conda.anaconda.org/conda-forge/ 

$ conda info 
... 
channel URLs : 
https://conda.anaconda.org/conda-forge/linux-64 
https://conda.anaconda.org/conda-forge/noarch
...

Last UpdatedSeptember 18, 2026

Search Docs by Keyword