awesome-etl

"Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases."

"combines data preparation, data blending, and analytics — predictive, statistical, and spatial — in a visual workflow designer."

"enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS."

"a serverless data integration service that makes it easy for analytics users to discover, prepare, move, and integrate data from multiple sources."

"Fully managed, cloud-native data integration platform."

"automates data movement from disparate sources into your destination."

"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed."

"an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes."

"Dagster is a data orchestrator for machine learning, analytics, and ETL. It lets you define pipelines in terms of the data flow between reusable, logical components, then test locally and run anywhere. With a unified view of pipelines and the assets they produce, Dagster can schedule and orchestrate Pandas, Spark, SQL, or anything else that Python can invoke."

"a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in."

"a workflow orchestration framework for building resilient data pipelines in Python."

"a scalable and reliable runtime for durable function executions called Temporal Workflow Executions."

"a unified programming model for Batch and Streaming data processing."

"a framework and distributed processing engine for stateful computations over unbounded and bounded data streams."

"Change data capture for a variety of databases."

"a tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka."

"a fast and general-purpose cluster computing system. It provides high-level APIs in Scala, Java, and Python that make parallel jobs easy to write, and an optimized engine that supports general computation graphs. It also supports a rich set of higher-level tools including Shark (Hive on Spark), MLlib for machine learning, GraphX for graph processing, and Spark Streaming."

"an open source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data."

"A lightweight, comprehensive batch framework designed to enable the development of robust batch applications that are vital for the daily operations of enterprise systems."

"a rich, web-based interface for designing, controlling, and monitoring a dataflow."

"Use Cask Data Application Platform to visually build and manage data applications in hybrid and multi-cloud environments."

An ETL tool for extracting data from source systems, transforming it, and loading it into target systems using a visual mapping and workflow designer.

"a component of the Microsoft SQL Server database software that can be used to perform a broad range of data migration tasks."

"Free and open fair-code licensed node based Workflow Automation Tool. Easily automate tasks across different services."

"a graphical ETL tool for designing data integration workflows using a drag-and-drop interface, also known as Kettle."

A curated list of awesome pipeline toolkits.

"a Python library for pulling data out of HTML and XML files."

"an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well."

"a flexible parallel computing library for analytics."

A wrapper around SQLAlchemy that simplifies database operations (including upserting).

"enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications."

"an open-source Python library that loads data from various, often messy data sources into well-structured datasets."

"a cloud asset inventory built for platform teams. Sync your cloud infrastructure metadata into your data warehouse, powering insights and automation."

"provides parallelized processing of multi-stage, language-agnostic pipelines with data versioning and data lineage tracking."

"a declarative data streaming and integration tool with 300+ pre-built connectors, configured via YAML."