Issues with PATH when Docker is enabled

[havranej]

Hi @lukfor, thanks for the work put into this!

I am running into an issue when I am trying to test processes that execute custom scripts within Docker – it seems that they are not added to PATH correctly. I have seen #31 and, indeed, when I run the test with docker.enabled = false, everything works as expected. I have also verified that executing the nextflow pipeline with the same parameters works as expected.

Here is a minimal example:

Command:

nf-test test tests/modules/hello.nf.test

Error:

  Command error:
    WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
    .command.sh: line 2: hello.py: command not found

Directory structure:

.
├── bin
│   └── hello.py
├── main.nf
├── modules
│   └── hello.nf
├── nextflow.config
├── nf-test.config
├── tests
│   ├── modules
│   │   └── hello.nf.test
│   └── nextflow.config

bin/hello.py

#!/usr/bin/env python3

print("Hello world")

modules/hello.nf

process HELLO {
    cpus 1
    memory '1 GB'
    container "<ECR-hosted image>"

    script:
    """
    hello.py
    """
}

tests/nextflow.config

docker {
    enabled = true
    runOptions = '-u $(id -u):$(id -g)'
}

tests/modules/hello.nf.test

nextflow_process {

    name "Test Process HELLO"
    script "modules/hello.nf"
    process "HELLO"

    test("Should run without failures") {

        when {
            params {
                // outdir = "tests/results"
            }
            process {
                """
                // input[0] = file("test-file.txt")
                """
            }
        }

        then {
            assert process.success
        }
    }
}

I am using nf-test 0.8.1 and nextflow 22.10.6

Cheers,
Jan

[antunderwood]

I am experiencing the same problem. And it seems to be a problem with mounting the meta directory correctly into the work

When I do some debugging all the meta subdirs are empty

    PARENT_DIR: /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta
    PATH: bin
    /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/bin
    ls -l /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/bin
    lrwxrwxrwx    1 root     root            71 Nov 14 13:16 /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/bin -> /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/bin
    
    
    PATH: assets
    /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/assets
    ls -l /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/assets
    lrwxrwxrwx    1 root     root            74 Nov 14 13:16 /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/assets -> /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/assets
    
    
    PATH: test_input
    /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/test_input
    ls -l /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/test_input
    lrwxrwxrwx    1 root     root            78 Nov 14 13:16 /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/test_input -> /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/test_input
    
    
    .command.sh: line 18: combine_plus_minus_counts.py: command not found

[antunderwood]

Further debugging. The symbolic links are broken

    PARENT_DIR: /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta
    PATH: bin
    ls -l /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/bin
    lrwxrwxrwx    1 root     root            71 Nov 14 13:44 /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/work/tests/1ea644dd7ee77edb703ab7e9de46492c/meta/bin -> /builds/brokenstringbio/induce-seq/induce-seq-post-pipeline-summary/bin
    The symbolic link is broken.

[antunderwood]

This seems to be exclusively an nf-test issue in that I can run the same dataset using nextflow run main.nf .... with docker and it completes successfully

[antunderwood]

As far as my investigations take me docker is not mounting the directory with code into the container.

[lukfor]

Thanks, Anthony, for the investigation. I will have a look at it tomorrow.

[antunderwood]

@lukfor

I have put together a small example of the problem
https://gitlab.com/brokenstringbio/analysis/nextflow/nf-test-ci-testing

An example of where it is failing is
https://gitlab.com/brokenstringbio/analysis/nextflow/nf-test-ci-testing/-/jobs/5654505015

The same files and pipeline run using plain nextflow run main.nf -profile test,local_test,docker -c tests/nextflow.config -resume is successful: https://gitlab.com/brokenstringbio/analysis/nextflow/nf-test-ci-testing/-/jobs/5654505017

The error is due to the bin directory not being available in meta

ls: /builds/brokenstringbio/analysis/nextflow/nf-test-ci-testing/work/tests/db8c8e52de564e363dacfc221b6ec15b/meta/bin: No such file or directory

This is despite it being available for an earlier process and a python script in the bin directory being accessed there

[antunderwood]

@lukfor. Happy 2024
Just wondering if you have been able to look at this and ascertain why the bin directory is not always available in meta?

[drpatelh]

We are seeing similar issues in nf-core/fetchngs too:
https://github.com/nf-core/fetchngs/actions/runs/7421745775/job/20196269395?pr=243

I am able to reproduce locally now but wasn't before for some reason. Weirdly if I run env in the script section of the process I see something like:

Command error:
    R_PROFILE_USER=/.Rprofile
    PYTHONNOUSERSITE=1
    HOSTNAME=be152dc66255
    NXF_TASK_WORKDIR=/tmp/tests/6520daeb4e2aea67c38b02762a58562a/work/ad/298ce95d57e77803b92061313d22fc
    PWD=/tmp/tests/6520daeb4e2aea67c38b02762a58562a/work/ad/298ce95d57e77803b92061313d22fc
    _=/usr/bin/env
    HOME=/root
    LANG=C.UTF-8
    SHLVL=2
    JULIA_DEPOT_PATH=/usr/local/share/julia
    R_ENVIRON_USER=/.Renviron
    NXF_DEBUG=0
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tmp/tests/6520daeb4e2aea67c38b02762a58562a/meta/bin

    .command.sh: line 4: sra_runinfo_to_ftp.py: command not found

The bin directory used by nf-test exists and has the binary in it too but for some reason isn't available in the container:

$ ls /tmp/tests/6520daeb4e2aea67c38b02762a58562a/meta/bin

multiqc_mappings_config.py  sra_ids_to_runinfo.py  sra_runinfo_to_ftp.py

[lukfor]

Thanks for all the details! I will try to find some time on the weekend to work on this.

[drpatelh]

PS: An additional mounting of the home directory is what fixed it but probably needs a more nf-test native solution:
nf-core/fetchngs#243 (comment)

[lukfor]

Thanks! Makes sense. It seems the issue may introduced creating a symlink to the bin folder since version 0.8.x.

It seems also that the problem depends on the Docker version and the operating system, since everything working as expected on macOS.

[antunderwood]

I can confirm that when I compile the code with the staging mode set to copy, use the compiled jar file in the code on gitlab CI it now works and the tests pass. Thanks!

@lukfor when are you targeting a 0.9.0 release?

[lukfor]

Thanks Anthony for testing the branch 🙏🏼 I will merge the PR and create a new release tomorrow.

[lukfor]

I just released version 0.8.3.

Harshil, could you give it a try? Thanks!

[drpatelh]

Looks good and seems to be working as expected. Thanks for the quick turnaround!

PS: https://www.nf-test.com/ seems to be down. Getting a 404.

[lukfor]

Thanks!

PS: Thanks, fixed!

[antunderwood]

Thanks for the quick turnaround @lukfor. 0.8.3 works for me :)