Scale Your Data Pipelines with Airflow and Kubernetes

Written by tal-peretz | Published 2020/03/15
Tech Story Tags: airflow | data-engineering | kubernetes | software-architecture | big-data | machine-learning | scaling | python

TLDR Tal Peretz is CTO & Chief Data Scientist @Supertools. He developed a workflow engine using Flask, Celery, and Kubernetes. Airflow is both scalable and cost-efficient. We use Git-Sync containers to update the workflows using git alone. We can destroy and re-deploy the entire infrastructure easily easily. Decoupling of orchestration and execution is a great advantage for Airflow. We are using a template template to set up scalable airflow workflows.via the TL;DR App
no story
Written by tal-peretz | CTO & Chief Data Scientist @Supertools
Published by HackerNoon on 2020/03/15
ABOUTHow hackers start their afternoons. We publish tech stories and build publishing software.

READEntirely free library read by hundreds of millions to learn anything about any technology.

WRITEHackerNoon elevates quality technology writing far and wide across the interwebs.

PARTNERHackerNoon works directly with tech companies to place ads by content relevancy