If you’re using Anaconda to create a new virtual environment and have encountered an issue where the environment creation process stalls at the stage of or this stage runs for an extremely long time (e.g., 10+ hours), the following steps outline the journey toward a solution. Solving Environment What Are the Reasons that “Solving Environment” Takes Excessively Long? To comprehend the factors contributing to the extended duration of the solving stage, I initially referred to a document on . The document provides a set of questions to consider when experiencing a slowdown: Conda Performance Are you creating a new environment or installing into an existing one? Does your environment have pip-installed dependencies in it? What channels are you using? What packages are you installing? Is the channel metadata sane? Are channels interacting in bad ways? To answer the questions, I was in the process of that contained dependencies. I utilized both the and channels and the packages are sourced from different channels. creating a new virtual environment pip-installed anaconda conda-forge To validate whether the channel metadata is sane, I executed the following commands. conda search --override-channels --channel=anaconda
conda search --override-channels --channel=conda-forge Thankfully, no errors were encountered, indicating that the channel metadata appeared to be in order. I remained uncertain about whether the channels were interacting in undesirable ways. Trying to Improve Conda’s Performance The document referenced earlier, together with the blog post titled , provides some suggested approaches to tackle the issue. Understanding and Improving Conda’s Performance Reduce Conda’s problem size (probably refers to the SAT problem) using “ ” conda-metachannel Configure channel priority Reduce the index — specify more specific package specs (e.g., version, build string) Using “conda-metachannel” Because of the large number of packages involved and the complexity of channel interactions, this particular method had not been tested at the time. Configure Channel Priority I initially attempted to resolve the issue by adjusting the channel priority in the file. I placed smaller channels like and before larger channels like , utilizing the mode. However, this approach did not effectively address the problem and sometimes even resulted in failure during the solving stage. Subsequently, I experimented with the mode, but the solving time remained excessively long. Here is an example of the configuration in the file. .condarc defaults anaconda conda-forge strict flexible .condarc channel_priority: flexible
channels:
  - defaults
  - anaconda
  - conda-forge Reduce the Index To optimize the indexing process, I opted to specify the version for each package explicitly. For instance, I used the format instead of . Theoretically, this approach should expedite the solving stage by allowing Conda to narrow down the candidate options more efficiently. numpy==1.15.4 numpy After implementing this modification, I observed that the solving stage took approximately to complete. To assess the reliability of this solution, I conducted tests on other machines. Surprisingly, the process concluded within approximately on certain machines, while on others, it seemed to . Consequently, it became evident that at hand. 70 minutes 15 minutes run indefinitely this method does not provide a definitive resolution to the issue Using mamba to Create the Environment Despite following the aforementioned suggestions without achieving success, I decided to explore an alternative approach to address the issue. Based on a recommendation from a colleague, I delved into , a C++ implementation of the Conda package manager. After experimenting with this tool, . Now, the environment creation process takes only a couple of minutes. mamba I successfully resolved the problem Install mamba Consequently, I started with by running (for Linux): install mamba curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh"
bash Mambaforge-$(uname)-$(uname -m).sh During the installation process, you may encounter a prompt asking whether to . In such a case, it is recommended to select “yes” for this option. This ensures that mamba can function properly alongside conda and avoids any potential conflicts between two package managers. initialize conda Once the installation is completed successfully, the terminal will display the information as depicted in the screenshot below. After restarting the terminal, you can verify the installation by typing in the command prompt. mamba A Potential Issue with the Installation In case you encounter an error stating even after restarting the terminal, it is advisable to check the conda section in your file. Ensure that the path to points to the , as illustrated in the second screenshot below. Command ‘mamba’ not found .bashrc conda.sh mamba installation directory Create A Virtual Environment using mamba Following the installation, I proceeded to create a new virtual environment using . I executed the command to create the environment using the specifications provided in the file, with the environment located in the current directory. mamba mamba env create -f environment.yaml --prefix $(pwd) environmental.yaml Compatibility with conda Command Although the issue was ultimately resolved, a new concern emerged due to the need to use instead of for creating the environment. This raised a potential challenge since there are numerous instances in our project where is utilized for configuring environment setup and deployment. mamba conda conda However, there is good news: and commands are interchangeable to some extent. This means that we can still use to interact with the environment created by , as illustrated in the screenshot below. mamba conda conda mamba Hence, it is great that our team does not have to refactor any of the existing processes, except for the specific step involving environment creation. This means that we can seamlessly integrate the use of mamba for creating environments without disrupting the rest of our workflow. Also published . here

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Train ML Models in Docker Container and Configure Access to on-premise GPU device

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Improve Productivity by Using mamba to Speed up Creating Python Virtual Environment

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Train ML Models in Docker Container and Configure Access to on-premise GPU device

2-minute recipe: How to rollback your conda environment

Learning Python the Hard Way - The Why, Hows, and Whats

How to Create Virtual Environments with Python [Tutorial]

Set up a Virtual Environment with Virtual Environment Wrapper on Linux

Train ML Models in Docker Container and Configure Access to on-premise GPU device

2-minute recipe: How to rollback your conda environment

Learning Python the Hard Way - The Why, Hows, and Whats

How to Create Virtual Environments with Python [Tutorial]

Set up a Virtual Environment with Virtual Environment Wrapper on Linux

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps