Add test-execution optimization and sharding options (#216)

* Add graph and resolver to support dependencies

* Add coverage flag and improve dry mode

* Add ignore patterns and restructure code

* Use dependency resolver to find tests by path

* Fix coverage when all tests are executed

* Refactore test resolver strategy

* Implement git support, `--only-changed` and `--changed-since`

* Fix bug in resolving includes in Nextflow script

* Update website in header

* Fix issue with tests path in config file

* Add optimized mode for firewall calculation

* Add `--changed-until` flag

* Fix issue in file pattern

* Implement sharding with round-robin

* Add snapshots to dependency graph and improve dot renderer

* Fix issue with test-hash after filename

* Add all new parameters to documentation

* Add `ignore` to documentation

* Add triggers

* Add testcases for dependency resolver

* Add testcases for dependency resolver

* Add `--ci` flag (#217)

* Add `--ci` flag

* Fix wrong commited files

* Fix wrong warning message

* Fix CI mode and add testcase

* Add tutorial to setup GitHub actions

* Update tutorial

* Add github token to env

.github/workflows/create-release.yml

@@ -25,6 +25,8 @@ jobs:
 
       - name: Build
         run: mvn install
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - uses: ncipollo/release-action@v1
         with:

.github/workflows/run-tests.yml

@@ -25,3 +25,5 @@ jobs:
 
       - name: Test
         run: mvn install
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

docs/docs/cli/test.md

@@ -12,6 +12,10 @@ nf-test test [<NEXTFLOW_FILES>|<SCRIPT_FOLDERS>]
 
 To run your test using a specific Nextflow profile, you can use the `--profile` argument. [Learn more](/docs/docs/configuration#managing-profiles).
 
+#### `--dry-run`
+
+This flag allows users to simulate the execution of tests.
+
 #### `--verbose`
 
 Prints out the Nextflow output during test runs.
@@ -24,6 +28,12 @@ The Linux tool `procps` is required to run Nextflow tracing. In case your contai
 
 Execute only tests with the provided tag. Multiple tags can be used and have to be separated by commas (e.g. `tag1,tag2`).
 
+#### `--debug`
+
+The debug parameter prints out debugging messages and all available output channels which can be accessed in the `then` clause.
+
+### Output Reports
+
 #### `--tap <filename>`
 
 Writes test results in [TAP format](https://testanything.org) to file.
@@ -32,9 +42,61 @@ Writes test results in [TAP format](https://testanything.org) to file.
 
 Writes test results in [JUnit XML format](https://junit.org/) to file, which conforms to [the standard schema](https://github.com/junit-team/junit5/blob/242f3b3ef84cfd96c9de26992588812a68cdef8b/platform-tests/src/test/resources/jenkins-junit.xsd).
 
-#### `--debug`
+#### `--csv <filename>`
 
-The debug parameter prints out debugging messages and all available output channels which can be accessed in the `then` clause.
+Writes test results in csv file.
+
+#### `--ci`
+
+By default,nf-test automatically stores a new snapshot. When CI mode is activated, nf-test will fail the test instead of storing the snapshot automatically.
+
+
+### Optimizing Test Execution
+
+#### `--related-tests <files>`
+
+Finds and executes all related tests for the provided .nf or nf.test files. Multiple files can be provided space separated.
+
+#### `--follow-dependencies`
+
+When this flag is set, nf-test will traverse all dependencies when the related-tests flag is set.
+This option is particularly useful when you need to ensure that
+all dependent tests are executed, bypassing the firewall calculation process.
+
+#### `--only-changed`
+
+When enabled, this parameter instructs nf-test to execute tests only for files that have been modified within the
+current git working tree.
+
+#### `--changed-since <commit_hash|branch_name>`
+
+This parameter triggers the execution of tests related to changes made since the specifie commit.
+e.g. `--changed-since HEAD^` for all changes between the HEAD and HEAD - 1.
+
+#### `--changed-until <commit_hash|branch_name>`
+
+This parameter initiates the execution of tests related to changes made until the specified commit hash.
+
+#### `--graph <filename>`
+
+Enables the export of the dependency graph as a dot file.
+The dot file format is commonly used for representing graphs in graphviz and other related software.
+
+### Sharding 
+
+This parameter allows users to divide the execution workload into manageable chunks, which can be useful for
+parallel or distributed processing.
+
+#### `--shard <shard>`
+Splits the execution into arbitrary chunks defined by the format `i/n`, where `i` denotes the index of the current
+chunk and `n` represents the total number of chunks. For instance, `2/5` executes the second chunk out of five.
+
+#### `--shard-strategy <strategy>`
+Description: Specifies the strategy used to build shards when the `--shard` parameter is utilized.
+Accepted values are `round-robin` or `none.`. This parameter determines the method employed to distribute workload
+chunks among available resources. With the round-robin strategy, shards are distributed evenly among resources in
+a cyclic manner. The none strategy implies that shards won't be distributed automatically, and it's up to the
+user to manage the assignment of shards. Default value is `round-robin`.
 
 ## Examples
 
@@ -63,3 +125,29 @@ The debug parameter prints out debugging messages and all available output chann
   ```
   nf-test test --tap report.tap
   ```
+
+- Run all tests (and possible integration tests) for module `modules/module_a.nf` and `modules/module_b.nf`;
+
+  ```
+  nf-test test --related-tests modules/module_a.nf modules/module_b.nf
+  ```
+
+- If your project is a Git directory and you have modified files, you can run tests only for these changed files by
+using the following command:
+
+  ```
+  nf-test test --only-changed
+  ```
+
+- If you want to test all changes made between the current state of the repository and the last commit,
+you can use the following command:
+
+  ```
+  nf-test test --changed-since HEAD^
+  ```
+
+- Run only the second of four shards:
+
+  ```
+  nf-test test --shard 2/4 
+  ```

docs/docs/configuration.md

@@ -14,8 +14,9 @@ The `nf-test.config` file is a configuration file used to customize settings and
 | `withTrace`  | Enable or disable tracing options during testing. Disable tracing if your containers don't include the `procps` tool.                                              | `true`                    |
 | `autoSort`   | Enable or disable sorted channels by default when running tests.                                                                                                   | `true`                    |
 | `options`    | Custom Nextflow command-line options to be applied when running tests. For example `"-dump-channels -stub-run"`                                                    |                           |
-| `requires`   | Can be used to specify the minimum required version of nf-test. Requires nf-test > 0.9.0                                                                           |                           |
-
+| `ignore`     | List of filenames or patterns that should be ignored when building the dependency graph. For example: `ignore 'folder/**/*.nf', 'modules/module.nf'`               | ``                        |
+| `triggers`   | List of filenames or patterns that should be trigger a full test run. For example: `triggers 'nextflow.config', 'test-data/**/*'`                                  | ``                        |
+| `requires`   | Can be used to specify the minimum required version of nf-test. Requires nf-test > 0.9.0                                                                           | ``                        |
 
 Here's an example of what an `nf-test.config` file could look like:
 

docs/tutorials.md

@@ -0,0 +1,11 @@
+# Tutorials
+
+---
+
+### [:material-file:{ style="color: #337ab7" } Setup nf-test on GitHub Actions](tutorials/github-actions.md)
+
+In this tutorial, we will guide you through setting up and running `nf-test` on GitHub Actions.
+
+[Read tutorial](tutorials/github-actions.md)
+
+---

docs/tutorials/github-actions.md

@@ -0,0 +1,177 @@
+# Setup nf-test on GitHub Actions
+
+In this tutorial, we will guide you through setting up and running `nf-test` on GitHub Actions. We will start with a simple example where all tests run in a single job, then extend it to demonstrate how you can use sharding to distribute tests across multiple jobs for improved efficiency. Finally, we will show you how to run only the tests affected by the changed files using the `--changes-since` option.
+
+By the end of this tutorial, you will have a clear understanding of how to:
+
+1. Set up a basic CI workflow for running `nf-test` on GitHub Actions.
+2. Extend the workflow to use sharding, allowing tests to run in parallel, which can significantly reduce the overall execution time.
+3. Configure the workflow to run only the test cases affected by the changed files, optimizing the CI process further.
+
+Whether you are maintaining a complex bioinformatics pipeline or a simple data analysis workflow, integrating `nf-test` with GitHub Actions will help ensure the robustness and reliability of your code. Let's get started!
+
+## Step 1: Running nf-test
+
+Create a file named `.github/workflows/ci-tests.yml` in your repository with the following content:
+
+```yaml
+name: CI Tests
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up JDK 11
+        uses: actions/setup-java@v2
+        with:
+          java-version: '11'
+          distribution: 'adopt'
+
+      - name: Setup Nextflow latest-edge
+        uses: nf-core/setup-nextflow@v1
+        with:
+          version: "latest-edge"
+
+      - name: Install nf-test
+        run: |
+          wget -qO- https://code.askimed.com/install/nf-test | bash
+          sudo mv nf-test /usr/local/bin/
+
+      - name: Run Tests
+        run: nf-test test
+```
+
+### Explanation:
+
+1. **Checkout**: Uses the `actions/checkout@v2` action to check out the repository.
+2. **Set up JDK 11**: Uses the `actions/setup-java@v2` action to set up Java Development Kit version 11.
+3. **Setup Nextflow**: Uses the `nf-core/setup-nextflow@v1` action to install the latest-edge version of Nextflow.
+4. **Install nf-test**: Downloads and installs nf-test.
+5. **Run Tests**: Runs nf-test without sharding.
+
+## Step 2: Extending to Use Sharding
+
+To distribute the tests across multiple jobs, you can set up sharding. Update your workflow file as follows:
+
+```yaml
+name: CI Tests
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        shard: [1, 2, 3, 4]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up JDK 11
+        uses: actions/setup-java@v2
+        with:
+          java-version: '11'
+          distribution: 'adopt'
+
+      - name: Setup Nextflow latest-edge
+        uses: nf-core/setup-nextflow@v1
+        with:
+          version: "latest-edge"
+
+      - name: Install nf-test
+        run: |
+          wget -qO- https://code.askimed.com/install/nf-test | bash
+          sudo mv nf-test /usr/local/bin/
+
+      - name: Run Tests (Shard ${{ matrix.shard }}/${{ strategy.job-total }})
+        run: nf-test test --shard ${{ matrix.shard }}/${{ strategy.job-total }}
+```
+
+### Explanation of Sharding:
+
+1. **Matrix Strategy**: The `strategy` section defines a matrix with a `shard` parameter that has four values: `[1, 2, 3, 4]`. This will create four parallel jobs, one for each shard.
+2. **Run Tests with Sharding**: The `run` command for running tests is updated to `nf-test test --shard ${{ matrix.shard }}/${{ strategy.job-total }}`. This command will run the tests for the specific shard. `${{ matrix.shard }}` represents the current shard number, and `${{ strategy.job-total }}` represents the total number of shards.
+
+## Step 3: Running Only Tests Affected by Changed Files
+
+To optimize the workflow further, you can run only the tests that are affected by the changed files since the last commit. Update your workflow file as follows:
+
+```yaml
+name: CI Tests
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        shard: [1, 2, 3, 4]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up JDK 11
+        uses: actions/setup-java@v2
+        with:
+          java-version: '11'
+          distribution: 'adopt'
+
+      - name: Setup Nextflow latest-edge
+        uses: nf-core/setup-nextflow@v1
+        with:
+          version: "latest-edge"
+
+      - name: Install nf-test
+        run: |
+          wget -qO- https://code.askimed.com/install/nf-test | bash
+          sudo mv nf-test /usr/local/bin/
+
+      - name: Run Tests (Shard ${{ matrix.shard }}/${{ strategy.job-total }})
+        run: nf-test test --shard ${{ matrix.shard }}/${{ strategy.job-total }} --changed-since HEAD^
+```
+
+### Explanation of Changes:
+
+1. **Checkout with Full History**: The `actions/checkout@v2` action is updated with `fetch-depth: 0` to fetch the full history of the repository. This is necessary for accurately determining the changes since the last commit.
+2. **Run Tests with Changed Files**: The `run` command is further updated to include the `--changed-since HEAD^` option. This option ensures that only the tests affected by the changes since the previous commit are run.
+
+
+## Step 4: Adapting nf-test.config to Trigger Full Test Runs
+
+In some cases, changes to specific critical files should trigger a full test run, regardless of other changes. To configure this, you need to adapt your `nf-test.config` file.
+
+Add the following lines to your `nf-test.config`:
+
+```groovy
+config {
+    ...
+    triggers 'nextflow.config', 'nf-test.config', 'test-data/**/*'
+    ...
+}
+```
+
+ The `triggers` directive in `nf-test.config` specifies a list of filenames or patterns that should trigger a full test run. For example:
+
+    - `'nextflow.config'`: Changes to the main Nextflow configuration file will trigger a full test run.
+    - `'nf-test.config'`: Changes to the nf-test configuration file itself will trigger a full test run.
+    - `'test-data/**/*'`: Changes to any files within the `test-data` directory will trigger a full test run.
+
+This configuration ensures that critical changes always result in a comprehensive validation of the pipeline, providing additional confidence in your CI process.
+
+## Summary
+
+1. **Without Sharding**: A straightforward setup where all tests run in a single job.
+2. **With Sharding**: Distributes tests across multiple jobs, allowing them to run in parallel.
+3. **With Sharding and Changed Files**: Optimizes the CI process by running only the tests affected by the changed files since the last commit, in parallel jobs.
+
+Choose the configuration that best suits your project's needs. Start with the simpler setup and extend it as needed to improve efficiency and reduce test execution time.

mkdocs.yml

@@ -65,6 +65,7 @@ nav:
           - Available Plugins: https://code.askimed.com/nf-test-plugins
           - Using Plugins: docs/plugins/using-plugins.md
           - Developing Plugins: docs/plugins/developing-plugins.md
+  - Tutorials: tutorials.md
   - Resources: resources.md
   - About: about.md
 

src/main/java/com/askimed/nf/test/commands/AbstractCommand.java

@@ -49,7 +49,7 @@ private void printHeader() {
 
 		System.out.println();
 		System.out.println(Emoji.ROCKET + AnsiText.bold(" " + App.NAME + " " + App.VERSION));
-		System.out.println("https://code.askimed.com/nf-test");
+		System.out.println("https://www.nf-test.com");
 		System.out.println("(c) 2021 - 2024 Lukas Forer and Sebastian Schoenherr");
 		System.out.println();