Scale Your Data Pipelines with Airflow and Kubernetesby@tal-peretz
1,433 reads
1,433 reads

Scale Your Data Pipelines with Airflow and Kubernetes

by Tal Peretz7mMarch 15th, 2020
Read on Terminal Reader
Read this story w/o Javascript
tldt arrow

Too Long; Didn't Read

Tal Peretz is CTO & Chief Data Scientist @Supertools. He developed a workflow engine using Flask, Celery, and Kubernetes. Airflow is both scalable and cost-efficient. We use Git-Sync containers to update the workflows using git alone. We can destroy and re-deploy the entire infrastructure easily easily. Decoupling of orchestration and execution is a great advantage for Airflow. We are using a template template to set up scalable airflow workflows.

Companies Mentioned

Mention Thumbnail
Mention Thumbnail
featured image - Scale Your Data Pipelines with Airflow and Kubernetes
Tal Peretz HackerNoon profile picture
Tal Peretz

Tal Peretz

@tal-peretz

CTO & Chief Data Scientist @Supertools

About @tal-peretz
LEARN MORE ABOUT @TAL-PERETZ'S
EXPERTISE AND PLACE ON THE INTERNET.
L O A D I N G
. . . comments & more!

About Author

Tal Peretz HackerNoon profile picture
Tal Peretz@tal-peretz
CTO & Chief Data Scientist @Supertools

TOPICS

THIS ARTICLE WAS FEATURED IN...

Permanent on Arweave
Read on Terminal Reader
Read this story in a terminal
 Terminal
Read this story w/o Javascript
Read this story w/o Javascript
 Lite