Whilst the default requirements set within the pipeline will hopefully work for most people and with most input data, you may find that you want to customise the compute resources that the pipeline requests. Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with any of the error codes specified here it will automatically be resubmitted with higher requests (2 x original, then 3 x original). If it still fails after the third attempt then the pipeline execution is stopped.
For example, if the nf-core/rnaseq pipeline is failing after multiple re-submissions of the STAR_ALIGN process due to an exit code of 137 this would indicate that there is an out of memory issue:
[62/149eb0] NOTE: Process `RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) -- Execution is retried (1)
Error executing process > 'RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)'
Caused by:
Process `RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137)
Command executed:
STAR \
--genomeDir star \
--readFilesIn WT_REP1_trimmed.fq.gz \
--runThreadN 2 \
--outFileNamePrefix WT_REP1. \
<TRUNCATED>
Command exit status:
137
Command output:
(empty)
Command error:
.command.sh: line 9: 30 Killed STAR --genomeDir star --readFilesIn WT_REP1_trimmed.fq.gz --runThreadN 2 --outFileNamePrefix WT_REP1. <TRUNCATED>
Work dir:
/home/pipelinetest/work/9d/172ca5881234073e8d76f2a19c88fb
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
To bypass this error you would need to find exactly which resources are set by the STAR_ALIGN process. The quickest way is to search for process STAR_ALIGN in the nf-core/rnaseq Github repo. We have standardised the structure of Nextflow DSL2 pipelines such that all module files will be present in the modules/ directory and so based on the search results the file we want is modules/nf-core/modules/azimuth/main.nf. If you click on the link to that file you will notice that there is a label directive at the top of the module that is set to label 'process_high_memory'. The Nextflow label directive allows us to organise workflow processes in separate groups which can be referenced in a configuration file to select and configure a subset of processes having similar computing requirements. The default values for the process_high_memory label are set in the pipeline’s base.config which in this case is defined as 50GB. Providing you haven’t set any other standard nf-core parameters to cap the maximum resources. Depending on your setup in your institute you may want to overite these default parameters as per below: https://github.com/wtsi-hgi/yascp/blob/73aed20e2c1b3ae693026704f2ccd3bb5d59deae/conf/base.conf#L215-L218
process {
withName: AZIMUTH{
maxRetries = 3
errorStrategy = { task.attempt > 2 ? 'ignore' : 'retry' }
memory = { check_max( 50.GB * task.attempt, 'memory' ) }
}
}
For the ultimate flexibility, we have implemented and are using Nextflow DSL2 modules in a way where it is possible for both developers and users to change tool-specific command-line arguments (e.g. providing an additional command-line argument to the AZIMUTH process) as well as publishing options (e.g. saving files produced by the AZIMUTH process that aren’t saved by default by the pipeline). In the majority of instances, as a user you won’t have to change the default options set by the pipeline developer(s), however, there may be edge cases where creating a simple custom config file can improve the behaviour of the pipeline.
As mentioned at the beginning of this section it may also be necessary for users to overwrite the options passed to modules to be able to customise specific aspects of the way in which a particular tool is executed by the pipeline. Given that all of the default module options are stored in the pipeline’s modules.config as a params variable it is also possible to overwrite any of these options via a custom config file.
Say for example we want to append an additional, non-mandatory parameter (i.e. --outFilterMismatchNmax 16) to the arguments passed to the STAR_ALIGN module. Firstly, we need to copy across the default args specified in the modules.config and create a custom config file that is a composite of the default args as well as the additional options you would like to provide. This is very important because Nextflow will overwrite the default value of args that you provide via the custom config.
The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the process name and override the Nextflow container definition for that process using the withName declaration. For example, in the nf-core/viralrecon pipeline, a tool called Pangolin has been used during the COVID-19 pandemic to assign lineages to SARS-CoV-2 genome sequenced samples. Given that the lineage assignments change quite frequently it doesn’t make sense to re-release the nf-core/viralrecon every time a new version of Pangolin has been released. However, you can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via -c custom.config.
Create the custom config accordingly:
For Docker:
process {
withName: PANGOLIN {
container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0'
}
}
For Singularity:
process {
withName: NORMALISE_AND_PCA {
container = '/software/hgi/containers/wtsihgi_nf_scrna_qc_6bb6af5-2021-12-23-3270149cf265.sif'
}
}
For Conda:
process {
withName: PANGOLIN {
conda = 'bioconda::pangolin=3.0.5'
}
}
NB: If you wish to periodically update individual tool-specific results (e.g. Pangolin) generated by the pipeline then you must ensure to keep the
work/directory otherwise the-resumeability of the pipeline will be compromised and it will restart from scratch.
In most cases, you will only need to create a custom config as a one-off but if you and others within your organisation are likely to be running nf-core pipelines regularly and need to use the same settings regularly it may be a good idea to request that your custom config file is uploaded to the nf-core/configs git repository. Before you do this please can you test that the config file works with your pipeline of choice using the -c parameter. You can then create a pull request to the nf-core/configs repository with the addition of your config file, associated documentation file (see examples in nf-core/configs/docs), and amending nfcore_custom.config to include your custom profile.
See the main Nextflow documentation for more information about creating your own configuration files.
If you have any questions or issues please send a message on Slack on the #configs channel.