awesome-pipeline

A workflow system for simple linear success/failure workflows.

Small package to describe workflows that are not completely known at definition time.

workflow manager with a strong focus on provenance, performance and extensibility.

Python-based workflow system created by AirBnb.

Component-based workflow framework for scientific data analysis.

High-level language for biology.

Computational science made reproducible and publishable.

Polyglot workflows without leaving the comfort of your technology stack.

A community and framework centered around metagenomics, designed to facilitate reproducible exploration and visualization of data.

Framework for executing and managing computational workflows on distributed computing resources.

Event-driven automation for sequencing centers. Initiates workflows based on events.

A container based workflow platform.

A tool for the automated exploration of possible computational workflows based on semantic annotations.

Get stuff done with container-native workflows for Kubernetes.

A pipeline based Continuous Delivery Service written in Golang.

Curated list of notable ETL (extract, transform, load) frameworks, libraries and software.

Curated list of awesome streaming frameworks, applications.

Curated list of awesome open source workflow engines.

Build software just as engineers do at Google.

Highly generalized task-management and automation in Python.

Unified cross platforms builds.

Command and recipe runner similar to Make, built in Rust.

The GNU Make build system.

Build system for data science pipelines.

Turn a GitHub repo into a collection of interactive notebooks powered by Jupyter and Kubernetes

Interactive data workflows built on Python.

Data pipeline framework supporting SQL and Python in the same DAG. Built-in data quality assertions, cross-database lineage, and incremental processing. Targets data warehouses (BigQuery, Snowflake, Postgres, etc.).

Dataform is a framework for managing SQL based operations in your data warehouse.

Managed cloud object storage transfers for ingestion workflows.

Hevo is a Fully Automated, No-code Data Pipeline Platform that supports 150+ ready-to-use integrations across Databases, SaaS Applications, Cloud Storage, SDKs, and Streaming Services.

A data processing & ETL framework for Ruby.