Importing Custom Pipelines
The user who imports a particular pipeline into Cirro is referred to as its 'author' (even if they did not write the actual workflow code). When importing custom pipelines, keep in mind that:
- The pipeline is only visible within those Cirro projects which are specified by the pipeline author
- The pipeline can only be added to projects for which the author is a Project Admin
Viewing Custom Pipelines
To access the list of custom pipelines that are available to you, and to add new pipelines, navigate to the Pipelines page and click on "Custom Pipelines".
The Custom Pipelines page displays the list of all pipelines which have been added to any of the projects that you have access to. Selecting any of those pipelines will open a menu for editing or syncing the pipeline.
Adding a Custom Pipeline
Clicking on the "Add Pipeline" button opens a new page and fill out the following information to add a custom pipeline.
- Process ID: A unique name for the process that contains only lowercase letters, numbers, and dashes. This is internal information and will never be shown to the user.
- Process Name: The name of the process the user will see.
- Description: The description of the process the user will see.
- Executor: If the process should be run will Nextflow or Cromwell (WDL) or upon upload of a new dataset, i.e. "Ingest".
- Documentation URL: (Optional) A link to the documentation of the process. This could be an external website, a public repository's README, etc.
- Definition Repository*: The GitHub repository which contains the workflow configuration files (e.g.
- Branch: The branch of the GitHub repository which contains the version of the configuration to be used.
- Folder:The folder which contains the configuration files for the workflow.
- Pipeline Repository*: The repository which contains the code needed to run the pipeline. This could be the same as the process definition repository (e.g.
- Pipeline Version: The branch of the GitHub repository which contains the version of the pipeline to be used.
- Pipeline Entrypoint: The file which will be run by the chosen workflow executor.
- Repository Type: If the repository is private or public.
- Parent Processes: The processes that output files which can be used as inputs for this new process.
- Child Processes: The processes that accept input files that come from this new process.
- Enabled Projects: Select the Cirro Project(s) in which this workflow can be run. Any user with Contributor-level permissions for that project will be able to execute the workflow.
*Note: When adding repository paths, do not include the
Updating a Custom Pipeline
After a pipeline has been added, it can be modified in terms of both user access and configuration.
By clicking on the entry for a single custom pipeline, a menu will open which displays information including the repository it was loaded from (with the exact code commit used).
To update the pipeline in Cirro to reflect the newest code available in that GitHub repository, click on the "Sync Pipeline" button.
Note: Any changes made to the GitHub repository containing the pipeline configuration files will not be applied in Cirro until the "Sync Pipeline" button is clicked. Only the pipeline author can sync with the source repository.
In addition to updating the pipeline, the author may also modify the list of projects in which the pipeline can be run.
Technical Note: Pipeline Configurations vs. Workflow Code
Keep in mind the difference between the pipeline configuration and the workflow code which is run using that pipeline.
If a pipeline configuration specifies that the latest version of a repository should be used, then the most up-to-date version of the repository will be executed every single time. However, if the configuration files for the pipeline are altered in any way (even in the same repository), that change will only take effect when "Sync Pipeline" is run again.