Don’t Let Conda Eat Your Hard Drive

For those who’re an Anaconda person, you understand that conda environments allow you to handle package deal dependencies, keep away from compatibility conflicts, and share your initiatives with others. Sadly, they’ll additionally take over your pc’s onerous drive.

I write numerous pc tutorials and to maintain them organized, every has a devoted folder construction full with a Conda Environment. This labored nice at first, however quickly my pc’s efficiency degraded, and I observed that my SSD was filling up. At one level I had solely 13 GB free.

Conda helps handle this drawback by storing downloaded package deal recordsdata in a single “cache” (pkgs_dirs). Once you set up a package deal, conda checks for it within the package deal cache earlier than downloading. If not discovered, conda will obtain and extract the package deal and hyperlink the recordsdata to the lively setting. As a result of the cache is “shared,” totally different environments can use the identical downloaded recordsdata with out duplication.

As a result of conda caches each downloaded package deal, pkgs_dirs can develop to many gigabytes. And whereas conda hyperlinks to shared packages within the cache, there’s nonetheless a must retailer some packages within the setting folder. That is primarily to keep away from model conflicts, the place totally different environments want totally different variations of the identical dependency (a package deal required to run one other package deal).

As well as, massive, compiled binaries like OpenCV could require full copies within the setting’s listing, and every setting requires a duplicate of the Python interpreter (at 100–200 MB). All these points can bloat conda environments to a number of gigabytes.

On this Fast Success Knowledge Science challenge, we’ll take a look at some methods for lowering the storage necessities for conda environments, together with these saved in default places and devoted folders.

---

Reminiscence Administration Methods

Under are some Memory Management methods that can allow you to scale back conda’s storage footprint in your machine. We’ll focus on every in flip.

1. Cache cleansing
2. Sharing task-based environments
3. Archiving with setting and specs recordsdata
4. Archiving environments with conda-pack
5. Storing environments on an exterior drive
6. Relocating the package deal cache
7. Utilizing digital environments (venv)

1. Cleansing the Bundle Cache

Cleansing the package deal cache is the primary and best step for liberating up reminiscence. Even after deleting environments, conda retains the associated package deal recordsdata within the cache. You may release house by eradicating these unused packages and their related tarballs (compressed package deal recordsdata), logs, index caches (metadata saved in conda), and short-term recordsdata.

Conda permits an optionally available “dry run” to see how a lot reminiscence can be reclaimed. You’ll need to run this from both the terminal or Anaconda Immediate in your base setting:

conda clear --all --dry-run

To commit, run:

conda clear --all

Right here’s how this seems on my machine:

This course of trimmed a wholesome 6.28 GB and took a number of minutes to run.

2. Sharing Job-based Environments

Creating a couple of environments for specialised duties — like pc imaginative and prescient or geospatial work — is extra reminiscence environment friendly than utilizing devoted environments for every challenge. These environments would come with fundamental packages plus ones for the particular activity (equivalent to OpenCV, scikit-image, and PIL for pc imaginative and prescient).

A bonus of this strategy is that you would be able to simply hold all of the packages updated and hyperlink the environments to a number of initiatives. Nonetheless, this gained’t work if some initiatives require totally different variations of the shared packages.

3. Archiving with Setting and Specs Recordsdata

For those who don’t have sufficient storage websites or need to protect legacy initiatives effectively, think about using setting or specs recordsdata. These small recordsdata document an setting’s contents, permitting you to rebuild it later.

Saving conda environments on this method reduces their measurement on disk from gigabytes to some kilobytes. In fact, you’ll must recreate the setting to make use of it. So, you’ll need to keep away from this method in the event you ceaselessly revisit initiatives that hyperlink to the archived environments.

NOTE: Think about using Mamba, a drop-in alternative for conda, for quicker rebuilds. Because the docs say, “If you understand conda, you understand Mamba!”

Utilizing Setting Recordsdata: An environmental file is a small file that lists all of the packages and variations put in in an setting, together with these put in utilizing Python’s package deal installer (pip). This helps you each restore an setting and share it with others.

The setting file is written in YAML (.yml), a human-readable data-serialization format for information storage. To generate an setting file, you will need to activate after which export the setting. Right here’s how one can make a file for an setting named my_env:

 conda activate my_env
 conda env export > my_env.yml

You may title the file any legitimate filename however watch out as an current file with the identical title can be overwritten.

By default, the setting file is written to the person listing. Right here’s a truncated instance of the file’s contents:

title: C:Usershannaquick_successfed_hikesfed_env
channels:
  - defaults
  - conda-forge
dependencies:
  - asttokens=2.0.5=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - blas=1.0=mkl
  - bottleneck=1.3.4=py310h9128911_0
  - brotli=1.0.9=ha925a31_2
  - bzip2=1.0.8=he774522_0
  - ca-certificates=2022.4.26=haa95532_0
  - certifi=2022.5.18.1=py310haa95532_0
  - colorama=0.4.4=pyhd3eb1b0_0
  - cycler=0.11.0=pyhd3eb1b0_0
  - debugpy=1.5.1=py310hd77b12b_0
  - decorator=5.1.1=pyhd3eb1b0_0
  - entrypoints=0.4=py310haa95532_0

  ------SNIP------

Now you can take away your conda setting and reproduce it once more with this file. To take away an setting, first deactivate it after which run the take away command (the place ENVNAME is the title of your setting):

conda deactivate
conda take away -n ENVNAME --all

If the conda setting exists outdoors of Anaconda’s default envs folder, then embody the listing path to the setting, as so:

conda take away -p PATHENVNAME --all

Notice that this archiving method will solely work completely in the event you proceed to make use of the identical working system, equivalent to Home windows or macOS. It’s because fixing for dependencies can introduce packages that may not be suitable throughout platforms.

To revive a conda setting utilizing a file, run the next, the place my_env represents your conda setting title and setting.yml represents your setting file:

 conda env create -n my_env -f directorypathtoenvironment.yml

You too can use the setting file to recreate the setting in your D: drive. Simply present the brand new path when utilizing the file. Right here’s an instance:

conda create --prefix D:my_envsmy_new_env --file setting.yml

For extra on setting recordsdata, together with how one can manually produce them, go to the docs.

Utilizing Specs Recordsdata: For those who haven’t put in any packages utilizing pip, you should utilize a specs file to breed a conda setting on the identical working system. To create a specification file, activate an setting, equivalent to my_env, and enter the next command:

 conda listing --explicit > exp_spec_list.txt

This produces the next output, truncated for brevity:

 # This file could also be used to create an setting utilizing:
 # $ conda create --name  --file 
 # platform: win-64
 @EXPLICIT
 https://conda.anaconda.org/conda-forge/win-64/ca-certificates-202x.xx.x-h5b45459_0.tar.bz2
 https://conda.anaconda.org/conda-forge/noarch/tzdata-202xx-he74cb21_0.tar.bz2

------snip------

Notice that the --explicit flag ensures that the focused platform is annotated within the file, on this case, # platform: win-64 within the third line.

Now you can take away the setting as described within the earlier part.

To re-create my_env utilizing this textual content file, run the next with a correct listing path:

conda create -n my_env -f directorypathtoexp_spec_list.txt

4. Archiving Environments with conda-pack

The conda-pack command allows you to archive a conda setting earlier than eradicating it. It packs the whole setting right into a compressed archive with the extension: .tar.gz. It’s useful for backing up, sharing, and shifting environments with out the necessity to reinstall packages.

The next command will protect an setting however take away it out of your system (the place my_env represents the title of your setting):

conda set up -c conda-forge conda-pack
conda pack -n my_env -o my_env.tar.gz

To revive the setting later run this command:

mkdir my_env && tar -xzf my_env.tar.gz -C my_env

This system gained’t save as a lot reminiscence because the textual content file choice. Nonetheless, you gained’t must re-download packages when restoring an setting, which suggests it may be used with out web entry.

5. Storing Environments on an Exterior Drive

By default, conda shops all environments in a default location. For Home windows, that is beneath the …anaconda3envs folder. You may see these environments by working the command conda information --envs in a immediate window or terminal. Right here’s the way it seems on my C: drive (it is a truncated view):

Utilizing a Single Environments Folder: In case your system helps an exterior or secondary drive, you possibly can configure conda to retailer environments there to release house in your major disk. Right here’s the command; you’ll must substitute your particular path:

conda config --set envs_dirs /path/to/exterior/drive

For those who enter a path to your D drive, equivalent to D:conda_envs, conda will create new environments at this location.

This system works effectively when your exterior drive is a quick SSD and once you’re storing packages with massive dependencies, like TensorFlow. The draw back is slower efficiency. In case your OS and notebooks stay on the first drive, it’s possible you’ll expertise some learn/write latency when working Python.

As well as, some OS settings could energy down idle exterior drives, including a delay once they spin again up. Instruments like Jupyter could battle to find conda environments if the drive letter modifications, so that you’ll need to use a set drive letter and be certain that the proper kernel paths are set.

Utilizing A number of Setting Folders: As a substitute of utilizing a single envs_dirs listing for all environments, you possibly can retailer every setting inside its respective challenge folder. This allows you to retailer every thing associated to a challenge in a single place.

For instance, suppose you’ve a challenge in your Home windows D: drive in a folder referred to as D:projectsgeospatial. To put the challenge’s conda setting on this folder, loaded with ipykernel for JupyterLab, you’d run:

conda create -p D:projectsgeospatialenv ipykernel

In fact, you possibly can name env one thing extra descriptive, like geospatial_env.

As with the earlier instance, environments saved on a unique disk could cause efficiency points.

Particular Notice on JupyterLab: Relying on the way you launch JupyterLab, its default conduct could also be to open in your person listing (equivalent to, C:Usersyour_user_name). Since its file browser is restricted to the listing from which it’s launched, you gained’t see directories on different drives like D:. There are a lot of methods to deal with this, however one of many easiest is to launch JupyterLab from the D: drive.

For instance, in Anaconda Immediate, kind:

D:

adopted by:

jupyter lab

Now, it is possible for you to to choose from kernels on the D: drive.

For extra choices on altering JupyterLab’s working listing, ask an AI about “how one can change Jupyter’s default working listing” or “how one can create a Symlink to D: in your person folder.”

Shifting Present Environments: It’s best to by no means manually transfer a conda setting, equivalent to by chopping and pasting to a brand new location. It’s because conda depends on inside paths and metadata that may turn into invalid with location modifications.

As a substitute, you need to clone current environments to a different drive. It will duplicate the setting, so that you’ll must manually take away it from its authentic location.

Within the following instance, we use the --clone flag to provide a precise copy of a C: drive setting (referred to as my_env) on the D: drive:

conda create -p D:new_envsmy_env --clone C:pathtooldenv

NOTE: Take into account exporting your setting to a YAML file (as described in Part 3 above) earlier than cloning. This lets you recreate the setting if one thing goes improper with the clone process.

Now, once you run conda env listing, you’ll see the setting listed in each the C: and D: drives. You may take away the outdated setting by working the next command within the base setting:

conda take away --name my_env --all -y

Once more, latency points could have an effect on these setups in the event you’re working throughout two disks.

It’s possible you’ll be questioning, is it higher to maneuver a conda setting utilizing an setting (YAML) file or to make use of--clone? The brief reply is that --clone is the perfect and quickest choice for shifting an setting to a unique drive on the identical machine. An setting file is finest for recreating the identical setting on a totally different machine. Whereas the file ensures a constant setting throughout totally different techniques, it will possibly take for much longer to run, particularly with massive environments.

6. Relocating the Bundle Cache

In case your major drive is low on house, you possibly can transfer the package deal cache to a bigger exterior or secondary drive utilizing this command:

conda config --set pkgs_dirs D:conda_pkgs

On this instance, packages are actually saved on the D drive (D:conda_pkgs) as a substitute of the default location.

For those who’re working in your major drive and each drives are SSD, then latency points shouldn’t be important. Nonetheless, if one of many drives is a slower HDD, you possibly can expertise slowdowns when creating or updating environments. If D: is an exterior drive related by USB, you might even see important slowdowns for big environments.

You may mitigate a few of these points by preserving the package deal cache (pkgs_dirs) and ceaselessly used environments on the quicker SSD, and different environments on the slower HDD.

One very last thing to think about is backups. Major drives could have routine backups scheduled however secondary or exterior drives could not. This places you liable to shedding all of your environments.

7. Utilizing Digital Environments

In case your challenge doesn’t require conda’s intensive package deal administration system for dealing with heavy dependencies (like TensorFlow or GDAL), you possibly can considerably scale back disk utilization with a Python digital setting (venv). This represents a light-weight different to a conda setting.

To create a venv named my_env, run the next command:

The sort of setting has a small base set up. A minimal conda setting takes up about 200 MB and contains a number of utilities, equivalent to conda, pip, setuptools, and so forth. A venv is far lighter, with a minimal set up measurement of solely 5–10 MB.

Conda additionally caches package deal tarballs in pkgs_dirs. These tarballs can develop to a number of GBs over time. As a result of venv installs packages straight into the setting, no additional copies are preserved.

On the whole, you’ll need to think about venv once you solely want fundamental Python packages like NumPy, pandas, or Scikit-learn. Packages for which conda is strongly advisable, like Geopandas, ought to nonetheless be positioned in a conda setting. For those who use numerous environments, you’ll in all probability need to follow conda and profit from its package deal linking.

You’ll find particulars on how one can activate and use Python digital environments within the venv docs.

---

Recap

Excessive influence/low disruption reminiscence administration methods for conda environments embody cleansing the package deal cache and storing little-used environments as YAML or textual content recordsdata. These strategies can save many gigabytes of reminiscence whereas retaining Anaconda’s default listing construction.

Different excessive influence strategies embody shifting the package deal cache and/or conda environments to a secondary or exterior drive. It will resolve reminiscence issues however could introduce latency points, particularly if the brand new drive is a gradual HDD or makes use of a USB connection.

For easy environments, you should utilize a Python digital setting (venv) as a light-weight different to conda.