Channels - the nextflow podcast Seqera

A small but intrepid team of Seqerans recently attended the ISMB 2024 conference in Montreal, a great mix of computational biology and bioinformatics with a largely academic / research-focused audience. Join Geraldine Van der Auwera, Rob Syme and Florian Wuennemann for a lively discussion about the scientific themes of the conference, their experience running the booth, meeting with the ISMB and BOSC community, and resources for job seekers.

Join us in exploring the latest Nextflow release, 24.04. Phil Ewels and Ben Sherman dive deep into the new updates in Nextflow’s most recent version. We discuss advanced retry strategies, job arrays, resourceLimits, Singularity’s OCI mode, and the game-changing Workflow Output Definition.



We’re actively seeking feedback from the community on these new features. We would love for you to experiment with the new syntax and let us know your thoughts.

Please let us know what you think either via Nextflow GitHub issues or via the community forum.



In this episode we refer to a the recent blog post about the release, which you can find here: Nextflow 24.04 - Release highlights.



Intro - tweaks, fixes and new support in 24.04

Performance and Stability Improvements

Ben kicks off the discussion with key performance and stability improvements. The latest stable release, 24.04, includes numerous bug fixes and performance enhancements. There’s a significant focus on closing gaps and adding retry strategies for better stability, especially concerning cloud providers. Notably, if Nextflow’s API calls against cloud providers fail for any server-side reason, it will retry automatically, avoiding pipeline failures caused by temporary server issues.

Publishing Changes

Previously, if a file failed to publish, Nextflow would only issue a warning. It was possible for a pipeline to complete successfully without noticing missing output files. Now, by default, Nextflow will fail the pipeline if the publishing fails, though there’s an option to revert to the old behavior. Additionally, retry strategies for publishing have been introduced to ensure retry attempts if an issue arises.

Job Arrays

One highly requested feature is the introduction of job arrays. This allows users to submit many jobs as a single submission, alleviating strain on schedulers. The submission happens in a batch, and then the scheduler can process and plan effectively. Once a job is submitted as part of a job array, the jobs run independently. If any child job fails, it is resubmitted without affecting the rest of the array.

Singularity OCI mode and GA4GH TES

In the past, Singularity supported Docker images by converting them into SIF files, consuming storage and time. Now, both Singularity and Apptainer can run OCI images directly, saving valuable resources.

Additionally, the TES executor has seen significant improvements, now supporting TES 1.1, which brings broader compatibility and integration with existing workflows.

Major new syntax features

Topic Channels

A new feature called topic channels offers a more straightforward approach to collecting channel outputs across pipelines. Channels can emit data to a named topic, simplifying the collection and use of version information from various processes within Nextflow pipelines.

Eval Outputs

Eval outputs simplify the addition of shell commands to tasks. With eval outputs, necessary post-task commands can be defined neatly, avoiding repetitive code within process definitions.

Workflow Output Definition

The concept of workflow output definitions has been introduced. This new syntax streamlines the publication of files by defining publish targets within workflows. Instead of defining publication behaviors within process definitions, users can now manage them at the workflow level, ensuring better clarity and fewer repetitions in the code.

Join us to hear Phil Ewels chat with Maxime Garcia
and Franziska Bonath about how nf-core started at the
National Genomics Infrastructure (NGI) at SciLifeLab in Sweden.We chat about how the nf-core/sarek pipeline grew alongside and eventually joined nf-core,as well as the early days of nf-core bytesize talks and outreach organisation.

Links:

- SciLifeLab
- National Genomics Infrastructure
- Maxime's Nextflow Summit Boston 2023 presentation about Sarek
- nf-core bytesize talks

Tune in to hear all about Pixelgen Technologies - creators of a novel technology
for doing single-cell spatial proteomics.



We chat about how this fascinating technology works, what you can do with the data
and why they chose to develop their analysis suite as an open-source nf-core pipeline.



Links:


Software:

nf-core/pixelator pipeline
Pixelator Python package


Science:

Nature Methods publication: “Molecular pixelation: spatial proteomics of single cells by sequencing”


Presentations:

Nextflow Summit 2022 talk: “Pixelgen Technologies ❤︎ Nextflow”
Nextflow Summit 2024 Boston talk: “Nextflow allows Pixelgen innovation engine to reach the market quicker”
Nextflow Summit 2024 Barcelona


Pixelgen:

Pixelgen Technologies homepage
@PixelgenTech on twitter / X and LinkedIn


BioLizard:

BioLizard homepage
@BioLizard_nv on twitter / X and LinkedIn





Chapters:


Start (00:00)
Welcome (00:09)
Introductions (00:50)
Intro to Pixelgen Technologies (01:41)
Lab methods (03:08)
Nature Methods publication (06:15)
Data visualisation (09:35)
Working with single cell data (12:29)
Cell interactions (15:06)
Why Pixelgen chose open source (16:05)
Pixelator Python package (20:10)
Building within nf-core (21:40)
Florian - Biolizard (23:01)
Pipeline overview (25:19)
Using nf-core from a commercial setting (26:50)
Coming out of stealth (28:53)
Extending Pixelator (32:24)
Dealing with customers (34:58)
Looking to the future (39:38)
Interactive analysis (41:32)
Advice for others (43:12)
Pixelgen at the Nextflow Summit (45:16)

Evan Floden (CEO and co-founder of Seqera) joins Phil Ewels (Product Manager for OSS at Seqera) to talk about two of the big announcements at the recent Nextflow Summit Boston 2024 - Seqera Pipelines and Seqera Containers.

We dive into what these two new community offerings are, why we built them and how they work.

Links:


https://seqera.io/pipelines/
https://seqera.io/containers/
Announcement blog post
Nextflow Summit Boston 2024 agenda
Summit YouTube playlist
Nextflow 24.04 blog post



Timeline:

00:00 - Introduction

00:37 - Nextflow Summit Boston 2024

01:14 - Summit talks available online

01:57 - Summit announcements



04:32 - Seqera Pipelines - motivations

05:59 - Seqera Pipelines - first look

06:48 - Testing requirements for community pipelines

10:05 - How we tested pipelines

11:53 - SRA-Explorer tangent



12:34 - Seqera Pipelines - live demo

16:14 - Getting Nextflow launch commands

19:12 - Seqera Containers - intro

24:03 - Seqera Containers - live demo

27:13 - Containers - ARM cpu arch builds

28:33 - Singularity containers

29:09 - Wave CLI

31:19 - Nextflow with Seqera Containers

33:22 - Seqera Containers - future usage



35:09 - Where to read more

36:53 - Thank you to the community

Join us as we discuss the brand new @nf-core / EuroFaang pre-print! 🔥 We chat to some of the key authors, covering the background to the paper, the main messages and how EuroFAANG and nf-core have collaborated. 🧬 👩🏻‍🔬