Overviews

Airflow provides operators for many common tasks, including:

  • BashOperator - executes a bash command
  • PythonOperator - calls an arbitrary Python function
  • EmailOperator - sends an email
  • HTTPOperator - sends an HTTP request
  • MySqlOperator, SqliteOperator, PostgresOperator, MsSqlOperator, OracleOperator, JdbcOperator, etc. - executes a SQL command Sensor - waits for a certain time, file, database row, S3 key, etc…

(https://airflow.apache.org/concepts.html)

Setup

Setting up a local (postgres) database:

psql# create database airflow 
CREATE ROLE airflow;
GRANT ALL PRIVILEGES on database airflow to airflow
ALTER ROLE airflow SUPERUSER;
ALTER ROLE airflow CREATEDB;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO airflow;

then in airflow update the airflow.cfg

result_backend = db+postgresql://airflow:airflow@postgres/airflow

and then run the database install

airflow initdb

Integrations

Testing

 
apache_airflow.txt · Last modified: 2019/02/26 00:03 by root
 
RSS - 200 © CrosswireDigitialMedia Ltd