Creating requirements.txt in pip compatible format in a conda virtual environment

I have created a conda virtual environment on a Windows 10 PC to work on a project. To install the required packages and dependencies, I am using conda install <package> instead of pip install <package> as per the best practices mentioned in https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#using-pip-in-an-environment

In order to distribute my software, I choose to create an environment.yml and a requirements.txt file targeting the conda and non-conda users respectively. I am able to export the current virtual environment into a yml file, so the conda users are taken care of. But, for the non-conda users to be able to replicate the same environment, I need to create and share the requirements.txt file. This file can be created using conda list --export > requirements.txt but this format is not compatible with pip and other users can’t use pip install -r requirements.txt on their systems.

Using pip freeze > requiremens.txt is a solution that is mentioned here and here. This means that non-conda users can simply execute pip install -r requirements.txt inside a virtual environment which they may create using virtualenv in the absence of conda.

However, if you generate a requiremets.txt file in the above style, you will end up with a requirements.txt file that has symbolic links. This is because we tried to create a requirements.txt file for packages that are installed using conda install and not pip install. For example, the requirements.txt file that I generated in a similar fashion looks like this.

certifi==2020.6.20
cycler==0.10.0
kiwisolver==1.2.0
matplotlib @ file:///C:/ci/matplotlib-base_1603355780617/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
olefile==0.46
pandas @ file:///C:/ci/pandas_1602083338010/work
Pillow @ file:///C:/ci/pillow_1602770972588/work
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
sip==4.19.13
six==1.15.0
tornado==6.0.4
wincertstore==0.2

These symbolic links will lead to errors when this file is used to install the dependencies.

Steps I took that landed me to the above requirements.txt file:

1. Created a new conda virtual environment using conda create -n myenv python=3.8
2. Activated the newly created conda virtual environment using conda activate myenv
3. Installed pip using conda install pip
4. Installed pandas using conda intall pandas
5. Installed matplotlib using conda install matplotlib
6. generated a pip compatible requrements.txt file using pip freeze > requirements.txt

So, my question is how do you stick to the best practice of using conda install instead of pip install while still being able to distribute your software package to both conda and non-conda users?

Answer

The best solution I’ve found for the above is the combination I will describe below. For conda, I would first export the environment list as environment.yml and omit the package build numbers, which is often what makes it hard to reproduce the environment on another OS:

conda env export > environment.yml --no-builds

Output:

name: myenv
channels:
  - defaults
  - conda-forge
dependencies:
  - blas=1.0
  - ca-certificates=2020.10.14
  - certifi=2020.6.20
...

For pip, what you describe above is apparently a well-known issue in more recent versions of pip. The workaround to get a “clean” requirements.txt file, is to export as such:

pip list --format=freeze > requirements.txt

Output:

certifi==2020.6.20
cycler==0.10.0
kiwisolver==1.2.0
matplotlib==3.3.2
mkl-fft==1.2.0
...

Notice that the above are different between pip and conda and that is most likely because conda is more generic than pip and includes not only Python packages.

Personally, I have found that for distributing a package, it is perhaps more concise to determine the minimum set of packages required and their versions by inspecting your code (what imports do you make?), instead of blindly exporting the full pip or conda lists, which might end up (accidentally or not) including packages that are not really necessary to use the package.