2 Apr 2025, Wed

Workflow Orchestration

Workflow Management

  • Apache Airflow: Platform to programmatically author, schedule, and monitor workflows
  • Luigi: Python package for building complex pipelines
  • Prefect: Workflow management system for data engineering
  • Dagster: Data orchestrator for machine learning, analytics, and ETL
  • Apache Oozie: Workflow scheduler system for Hadoop
  • Keboola: Data operations platform

Scheduling Tools

  • Cron: Time-based job scheduler in Unix-like systems
  • Jenkins Job Builder: System for configuring Jenkins jobs
  • Azkaban: Batch workflow job scheduler for Hadoop
  • Rundeck: Job scheduler and runbook automation
  • Temporal: Microservice orchestration platform