In this article, we will learn how to develop ETL(Extract Transform Load) pipeline using Apache Airflow. Here are list of things that we will do in this article: Call an API Setup database Setup airflow Call an API We will create a module , and inside it we will create a function which will call the API. getWeather.py get_weather() We will then create a directory where we will save daily data obtained from API. We do this under function as shown below. data/ createDirectory() Setup Database We will create a module , and inside it we will create a function which will create database. createTable.py make_database() Setup Airflow In order to use Airflow, you will have to set up Airflow first. You can see Airflow installation on how to setup Airflow. documentation Once Airflow has been set up, we will define our dag. Now we can run our DAG from Apache Airflow. Complete code for this article can be found in this Github Repository. Special thanks to Michael Harmon. This article is developed using his publication. You can find it over . here
![featured image - Data Pipelines: OpenWeatherMap-Airflow [A How-To Guide]](https://hackernoon.imgix.net/drafts/lf5ez3x4n.png?auto=format&fit=max&w=1200)
