41 episodes

Nextflow is a leading open-source workflow orchestrator that simplifies writing and deploying compute and data-intensive pipelines at scale on any infrastructure. In this podcast we discuss news in the Nextflow ecosystem and speak with pioneers in the field.

Channels - the nextflow podcast Seqera

    • Science

Nextflow is a leading open-source workflow orchestrator that simplifies writing and deploying compute and data-intensive pipelines at scale on any infrastructure. In this podcast we discuss news in the Nextflow ecosystem and speak with pioneers in the field.

    Episode 41: New syntax in Nextflow 24.04.

    Episode 41: New syntax in Nextflow 24.04.

    Join us in exploring the latest Nextflow release, 24.04. Phil Ewels and Ben Sherman dive deep into the new updates in Nextflow’s most recent version. We discuss advanced retry strategies, job arrays, resourceLimits, Singularity’s OCI mode, and the game-changing Workflow Output Definition.



    We’re actively seeking feedback from the community on these new features. We would love for you to experiment with the new syntax and let us know your thoughts.

    Please let us know what you think either via Nextflow GitHub issues or via the community forum.



    In this episode we refer to a the recent blog post about the release, which you can find here: Nextflow 24.04 - Release highlights.



    Intro - tweaks, fixes and new support in 24.04

    Performance and Stability Improvements

    Ben kicks off the discussion with key performance and stability improvements. The latest stable release, 24.04, includes numerous bug fixes and performance enhancements. There’s a significant focus on closing gaps and adding retry strategies for better stability, especially concerning cloud providers. Notably, if Nextflow’s API calls against cloud providers fail for any server-side reason, it will retry automatically, avoiding pipeline failures caused by temporary server issues.

    Publishing Changes

    Previously, if a file failed to publish, Nextflow would only issue a warning. It was possible for a pipeline to complete successfully without noticing missing output files. Now, by default, Nextflow will fail the pipeline if the publishing fails, though there’s an option to revert to the old behavior. Additionally, retry strategies for publishing have been introduced to ensure retry attempts if an issue arises.

    Job Arrays

    One highly requested feature is the introduction of job arrays. This allows users to submit many jobs as a single submission, alleviating strain on schedulers. The submission happens in a batch, and then the scheduler can process and plan effectively. Once a job is submitted as part of a job array, the jobs run independently. If any child job fails, it is resubmitted without affecting the rest of the array.

    Singularity OCI mode and GA4GH TES

    In the past, Singularity supported Docker images by converting them into SIF files, consuming storage and time. Now, both Singularity and Apptainer can run OCI images directly, saving valuable resources.

    Additionally, the TES executor has seen significant improvements, now supporting TES 1.1, which brings broader compatibility and integration with existing workflows.

    Major new syntax features

    Topic Channels

    A new feature called topic channels offers a more straightforward approach to collecting channel outputs across pipelines. Channels can emit data to a named topic, simplifying the collection and use of version information from various processes within Nextflow pipelines.

    Eval Outputs

    Eval outputs simplify the addition of shell commands to tasks. With eval outputs, necessary post-task commands can be defined neatly, avoiding repetitive code within process definitions.

    Workflow Output Definition

    The concept of workflow output definitions has been introduced. This new syntax streamlines the publication of files by defining publish targets within workflows. Instead of defining publication behaviors within process definitions, users can now manage them at the workflow level, ensuring better clarity and fewer repetitions in the code.

    • 46 min
    Episode 40: Origins of nf-core - SciLifeLab NGI

    Episode 40: Origins of nf-core - SciLifeLab NGI

    Join us to hear Phil Ewels chat with Maxime Garcia
    and Franziska Bonath about how nf-core started at the
    National Genomics Infrastructure (NGI) at SciLifeLab in Sweden.We chat about how the nf-core/sarek pipeline grew alongside and eventually joined nf-core,as well as the early days of nf-core bytesize talks and outreach organisation.

    Links:

    - SciLifeLab
    - National Genomics Infrastructure
    - Maxime's Nextflow Summit Boston 2023 presentation about Sarek
    - nf-core bytesize talks

    • 34 min
    Episode 39: Spatial proteomics at Pixelgen

    Episode 39: Spatial proteomics at Pixelgen

    Tune in to hear all about Pixelgen Technologies - creators of a novel technology
    for doing single-cell spatial proteomics.



    We chat about how this fascinating technology works, what you can do with the data
    and why they chose to develop their analysis suite as an open-source nf-core pipeline.



    Links:


    Software:

    nf-core/pixelator pipeline
    Pixelator Python package


    Science:

    Nature Methods publication: “Molecular pixelation: spatial proteomics of single cells by sequencing”


    Presentations:

    Nextflow Summit 2022 talk: “Pixelgen Technologies ❤︎ Nextflow”
    Nextflow Summit 2024 Boston talk: “Nextflow allows Pixelgen innovation engine to reach the market quicker”
    Nextflow Summit 2024 Barcelona


    Pixelgen:

    Pixelgen Technologies homepage
    @PixelgenTech on twitter / X and LinkedIn


    BioLizard:

    BioLizard homepage
    @BioLizard_nv on twitter / X and LinkedIn





    Chapters:


    Start (00:00)
    Welcome (00:09)
    Introductions (00:50)
    Intro to Pixelgen Technologies (01:41)
    Lab methods (03:08)
    Nature Methods publication (06:15)
    Data visualisation (09:35)
    Working with single cell data (12:29)
    Cell interactions (15:06)
    Why Pixelgen chose open source (16:05)
    Pixelator Python package (20:10)
    Building within nf-core (21:40)
    Florian - Biolizard (23:01)
    Pipeline overview (25:19)
    Using nf-core from a commercial setting (26:50)
    Coming out of stealth (28:53)
    Extending Pixelator (32:24)
    Dealing with customers (34:58)
    Looking to the future (39:38)
    Interactive analysis (41:32)
    Advice for others (43:12)
    Pixelgen at the Nextflow Summit (45:16)

    • 46 min
    Episode 38: Seqera Pipelines & Seqera Containers

    Episode 38: Seqera Pipelines & Seqera Containers

    Evan Floden (CEO and co-founder of Seqera) joins Phil Ewels (Product Manager for OSS at Seqera) to talk about two of the big announcements at the recent Nextflow Summit Boston 2024 - Seqera Pipelines and Seqera Containers.

    We dive into what these two new community offerings are, why we built them and how they work.

    Links:


    https://seqera.io/pipelines/
    https://seqera.io/containers/
    Announcement blog post
    Nextflow Summit Boston 2024 agenda
    Summit YouTube playlist
    Nextflow 24.04 blog post



    Timeline:

    00:00 - Introduction

    00:37 - Nextflow Summit Boston 2024

    01:14 - Summit talks available online

    01:57 - Summit announcements



    04:32 - Seqera Pipelines - motivations

    05:59 - Seqera Pipelines - first look

    06:48 - Testing requirements for community pipelines

    10:05 - How we tested pipelines

    11:53 - SRA-Explorer tangent



    12:34 - Seqera Pipelines - live demo

    16:14 - Getting Nextflow launch commands

    19:12 - Seqera Containers - intro

    24:03 - Seqera Containers - live demo

    27:13 - Containers - ARM cpu arch builds

    28:33 - Singularity containers

    29:09 - Wave CLI

    31:19 - Nextflow with Seqera Containers

    33:22 - Seqera Containers - future usage



    35:09 - Where to read more

    36:53 - Thank you to the community

    • 37 min
    Episode 37: The nf-core / EuroFAANG pre-print

    Episode 37: The nf-core / EuroFAANG pre-print

    Join us as we discuss the brand new @nf-core / EuroFaang pre-print! 🔥 We chat to some of the key authors, covering the background to the paper, the main messages and how EuroFAANG and nf-core have collaborated. 🧬 👩🏻‍🔬

    • 54 min
    Episode 36: The Nextflow Ambassadors Program

    Episode 36: The Nextflow Ambassadors Program

    Join us in this episode of the Channels podcast for a closer look at the Nextflow Ambassadors Program. Marcel Ribeiro-Dantas and Geraldine Van der Auwera discuss the origins of the program, recent highlights and next steps, including the open call for new Ambassadors, which will close June 14.

    Relevant links:


    Call for applications: https://www.nextflow.io/ambassadors.html
    Original announcement: https://www.nextflow.io/blog/2023/introducing-nextflow-ambassador-program.html
    Marcel’s favorite Ambassador-contributed Nextflow blog post: https://www.nextflow.io/blog/2024/reflections-on-nextflow-mentorship.html

    • 24 min

Top Podcasts In Science

Hidden Brain
Hidden Brain, Shankar Vedantam
Ologies with Alie Ward
Alie Ward
Radiolab
WNYC Studios
Making Sense with Sam Harris
Sam Harris
The Infinite Monkey Cage
BBC Radio 4
StarTalk Radio
Neil deGrasse Tyson

You Might Also Like

The Ezra Klein Show
New York Times Opinion
Planet Money
NPR
Up First
NPR
Throughline
NPR
The Daily
The New York Times
Why Is This Happening? The Chris Hayes Podcast
Chris Hayes, MSNBC & NBCNews THINK