Snakemake doesn’t activate conda environment correctly

I have a Python module modulename installed in a conda environment called myenvname.

My snakemake file consists of one simple rule:

rule checker2:
    output:
        "tata.txt"
    conda:
        "myenvname"
    script:
        "scripts/test2.py"

The contents of the test2.py are the following:

import modulename
with open("tata.txt","w") as _f:
    _f.write(modulename.__version__)

When I run the above snakemake file with the command snakemake -j 1 --use-conda --conda-frontend conda I get ModuleNotFoundError, which would imply that there is no modulename in my specified environment. However, when I do the following :

conda activate myenvname
python workflow/scripts/test2.py

… everything works perfectly. I have no idea what’s going on.

The full error is pasted below, with some info omitted for privacy.

Traceback (most recent call last):
  File "/OMITTED/.snakemake/scripts/tmpheaxuqjn.test2.py", line 13, in <module>
    import cnvpytor as cnv
ModuleNotFoundError: No module named 'MODULENAME'
[Thu Nov 17 18:27:22 2022]
Error in rule checker2:
    jobid: 0
    output: tata.txt
    conda-env: MYENVNAME

RuleException:
CalledProcessError in line 12 of /OMITTED/workflow/snakefile:
Command 'source /apps/qiime2/miniconda3/bin/activate 'MYENVNAME'; set -euo pipefail;  /OMITTED/.conda/envs/snakemake/bin/python3.1 /OMITTED/.snakemake/scripts/tmpheaxuqjn.test2.py' returned non-zero exit status 1.
  File "/OMITTED/workflow/snakefile", line 12, in __rule_checker2
  File "/OMITTED/.conda/envs/snakemake/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /OMITTED/.snakemake/log/2022-11-17T182715.495739.snakemake.log

EDIT: Typo in script fixed, the typo isn’t in the script I’m running so it’s not the issue here.

EDIT2: I’ve tried two different attempts from comments. All three attempts are run with the same CLI command snakemake -j 1 --use-conda --conda-frontend conda

Attempt 1

Rule in the snakemake:

rule checker3:
    output:
        "tata.txt"
    conda:
        "myenvname"
    shell:
        """
        conda env list >> {output}
        conda list >> {output}
        """

In the output file I had the following (I have lots of environs and packages I’ve cut out):

# conda environments:
#
...
myenvname              *  /OMITTED/.conda/envs/myenvname
...
# packages in environment at /OMITTED/.conda/envs/myenvname:
#
# Name                    Version                   Build  Channel
...
modulename                  1.2                       dev_0    <develop>
...

This attempt proves that the conda environment is activated and that this environment has modulename.

Attempt 2

Same as running the script, but I’ve modified the script to include

import time; time.sleep(30); import modulename

So I can snag a runnable script before it’s deleted. The script has the following inserted at the start:

######## snakemake preamble start (automatically inserted, do not edit) ########
import sys; sys.path.extend(['/OMITTED/.conda/envs/snakemake/lib/python3.10/site-packages', '/OMITTED/MYWORKINGDIRECTORY/workflow/scripts']); import pickle; snakemake = pickle.loads(####a lot of stuff here###); from snakemake.logging import logger; logger.printshellcmds = False; __real_file__ = __file__; __file__ = '/OMITTED/MYWORKINGDIRECTORY/workflow/scripts/test3.py';
######## snakemake preamble end #########

I have no idea what to do with this information.

Attempt 3

Instead of running script, I’ve ran a shell command that runs a python script.

rule checker4:
    output:
        "tata.txt"
    conda:
        "myenvname"
    shell:
        "python workflow/scripts/test3.py"

It worked (showed no errors), and when I open “tata.txt” I find “1.2” which is the version of of my module.

Conclusions

The snakemake actually activates proper environment, but the problem is in script part. I have no idea why this is.

There is a similar question here, so this is a duplicate question.

Answer

Question is answered. Snakemake actually activates correct environment, but running a python script with the script conflicts with this directive. I don’t know if this is a bug in snakemake (version is 6.14.0) or an intentional thing. I’ve solved the problem by running the python script via shell command with python workflow/scripts/MyScript.py – it’s a bit of a problem because I had to include a CLI wrapper that would normally be solved by a snakemake object.