Ada Project Environment Setup

The python environment that I shared with you for your homework assignments can certainly be used for your final project, but you may find it to be insufficient if you need to use models or libraries that we have not used in class.

Background

I gave you the following bash commands to run for your homework assignments:

module use /home/lbiester/modules
module load cs457/s24-hw

When you run these commands to load the s24-hw module, two important things happen:

  1. The HF_HOME variable is set. What this does is it sets the directory from which Hugging Face models will be read (and written). The intention of this was to prevent everyone from needing to download their own copies of DistilBERT and GPT-2 weights for the homework. This directory was initially set to be read-only. I’ve noticed some students running into issues with these read-only directories while working on projects, and I changed some settings such that you can now write to them.1 You won’t do this directly, it will only be through the Hugging Face transformers library.
  2. The path for the CS457 python environment (/home/lbiester/.conda/envs/CS457/bin) is added to the PATH environment variable. This gives you access to libraries that I installed for homework assignments and projects, including transformers, gensim, datasets, sklearn, numpy, pandas, nltk, matplotlib, and more.2

This environment may be insufficient for your project either because you need additional model weights or because you need additional python libarries. The rest of this page highlights workarounds for each issue.

Your Own Copies of Model Weights

If you are downloading additional models or you are running into permissions errors within the Hugging Face libraries, you might find it easier to download your own model weights. Doing so is actually pretty trivial! Just don’t run the module commands at all, and your HF_HOME variable will be set to the default. If you still want to use my python environment, check the instructions below.

Access to Additional Python Libraries

If you are using ada for your project, you might find that there is a library you’d like to use that isn’t available in the environment that we used for homework assignments. If that happens, you have two options:

Option 1: Ask Me to Install the Library

Generally, I’d be happy to install a library for you. If you email me, I’ll install the library and confirm that it’s been installed within 24 hours during the work week (I won’t be installing libraries on weekends).

Option 2: Set Up Your Own Python Environment

If you can’t wait for me to install a library, you can set up your own python environment on ada. I recommend using conda, which is already installed on the cluster.3

Step 1: Creating a Conda Environment

You can create your own conda environment using the following command:4

conda create -n CustomCS457 python=3.10.4

You will be prompted to enter y to proceed during the process, which might take a few minutes.

Step 2: Activate Your Environment

Activate your conda environment with the following command:

conda activate CustomCS457

You’ll need to do this each time you log in to ada!

You will also need to update the python interpreter that you have set through vscode (see our original ada setup instructions). You can get the path to your interpreter by running which python.

Step 3: Install Libraries

Now you need to install your libararies. Do so using pip, e.g.,

pip install seaborn

If you’d like to start by installing all of the libraries that are installed in the shared environment, you can run

pip install -r /home/lbiester/CS457HW/conda/requirements.txt

Using only the Python Environment or Hugging Face Weights

If you want to, you could use just my downloaded model weights or my python environment.

To use just the model weights, run

export HF_HOME=/storage/lbiester/.cache/

To use just the python environment, run

conda activate /storage/homes/lbiester/.conda/envs/CS457

You would need to run these each time you log in, just like you would with the module commands if you are using them. If you do this, you shouldn’t use the module commands, since those

  1. Please be a good classmate and do not directly edit anything in the HF_HOME directory. ↩

  2. If you want a full list of the installed libraries and versions, make sure that the python environment is activated, then run pip freeze. ↩

  3. Generally, these instructions should also work on your personal computer, but you’ll need to install conda first. ↩

  4. The listed python version matches the version of the class environment that you used for homework. ↩