핵심 개념

핵심 개념 (Core Concepts)

Apache Airflow®의 핵심 개념들 하나하나를 자세히 설명하는 문서 모음의 인덱스 페이지예요. Dag, Dag Run, Task, Operator, Sensor, Executor, XCom 등 Airflow를 이해하는 데 필요한 개념들을 주제별로 정리해 두었어요. 각 개념의 상세 문서로 바로 이동할 수 있는 목차 역할을 해요.

출처: 문서

본문

여기서 Apache Airflow®의 핵심 개념 각각에 대한 상세 문서와 사용법, 그리고 높은 수준의 아키텍처 개요를 확인할 수 있어요.

아키텍처 (Architecture)

• Architecture Overview
  • Airflow components
  • Deploying Airflow components
  • Architecture Diagrams
  • Task execution architecture
  • Workloads
  • Control Flow
  • User interface

워크로드 (Workloads)

• Dags

  • Declaring a Dag
  • Loading Dags
  • Running Dags
  • Dag Assignment
  • Default Arguments
  • The Dag decorator
  • Control Flow
  • Dynamic Dags
  • Dag Visualization
  • Dag, Task Group & Task Documentation
  • Packaging Dags
  • .airflowignore
  • Dag Dependencies
  • Dag pausing, deactivation and deletion
  • Dag Auto-pausing (Experimental)
  • Deadline Alerts
  • Testing a Dag

• Dag Runs

  • Dag Run Status
  • Manual Triggering and Data Intervals
  • Re-run Dag
  • UI
  • CLI
  • External Triggers
  • To Keep in Mind

• Tasks

  • Relationships
  • Task Instances
  • Timeouts
  • SLAs
  • Special Exceptions
  • Retry Policies
  • Task Instance Heartbeat Timeout
  • Executor Configuration

• Operators

  • Jinja Templating
  • Reserved params keyword
  • Pre- and post-execute methods
  • Difference between pre-/post-execute and setup/teardown

• Sensors

  • BaseSensorOperator parameters
  • Common sensors

• TaskFlow

  • Context
  • Logging
  • Passing Arbitrary Objects As Arguments
  • Sensors and the TaskFlow API
  • History

• Resumable Tasks

  • Durable execution
  • Deferrable Operators
  • Resumable Tasks
  • Asynchronous Tasks
  • Comparison

• Executor

  • Executor Types
  • Using Multiple Executors Concurrently
  • Writing Your Own Executor

• Auth manager

  • Available auth managers to use
  • Why pluggable auth managers?
  • Writing your own auth manager
  • Dag and Dag Sub-Component Authorization
  • Next Steps

• Multi-Team

  • When to Use Multi-Team Mode
  • Core Concepts
  • Enabling Multi-Team Mode
  • Creating and Managing Teams
  • Configuring Team Resources
  • Team-based Executor Configuration
  • Dag Bundle to Team Association
  • How Scheduling Works
  • Team-scoped Triggerer
  • Team-Based Asset Event Filtering
  • Team-based Metrics
  • Important Considerations
  • Architecture

• Object Storage

  • Cloud Object Stores are not real file systems
  • Basic Use
  • Configuration
  • Path API
  • Extensions
  • Copying and Moving
  • External Integrations

• Backfill

  • Control over data reprocessing
  • Concurrency control
  • Run ordering
  • Dry run
  • Example
  • Partitioned Dag backfills

• Message Queues

통신 (Communication)

• XComs

  • Object Storage XCom Backend
  • Custom XCom Backends
  • Verifying Custom XCom Backend usage in Containers

• Variables

• Params

  • Dag-level Params
  • Task-level Params
  • Referencing Params in a Task
  • JSON Schema Validation
  • Use Params to Provide a Trigger UI Form
  • Disabling Runtime Param Modification

Task 및 Asset State Store

• Task and Asset State Store Overview

  • Task and Asset State Store
  • When to use Task and Asset State Store
  • Further reading

• Task State Store

  • Accessing task state store
  • Reference
  • Some Example Use Cases
  • Sync vs. deferrable tasks
  • Mapped tasks
  • Automatic cleanup ( clear_on_success )

• Asset State Store

  • When is asset_state_store available?
  • Accessing asset state store using context
  • API reference
  • Using asset_state_store inside a Watcher Trigger
  • Some Example Use cases
  • Lifetime and garbage collection

디버깅 (Debugging)

• Debugging Airflow Dags
• Debugging Airflow Dags on the command line

더 알아보기 (Learn more)

• 아키텍처 개요 (Architecture Overview)
• Dag 문서
• Operators 문서
• Airflow 최신 버전 문서 홈