This is a self-contained demo of FastAPI and . Materialize This demo project contains the following components: : A fast, modern, and feature-rich framework for building APIs with Python. FastAPI : Kafka® compatible event streaming platform written in C++. Redpanda : A streaming database for real-time analytics. Materialize A mock service written in . The mock service simulates data of air quality readings of IoT devices. BASH for producing records to a Redpanda topic Diagram Running the demo Clone the repository: git clone https://github.com/bobbyiliev/materialize-tutorials.git Access the FastAPI demo project directory: cd mz-fastapi-demo Pull all Docker images: docker-compose pull Build the project: docker-compose build Finally, run all containers: docker-compose up Create the Materialize sources and views Once the demo is running, you can create the Materialize sources and views. Let's start by creating a Redpanda/Kafka : source CREATE SOURCE sensors
FROM KAFKA BROKER 'redpanda:9092' TOPIC 'sensors'
FORMAT BYTES; Then create a which you can think of essentially an alias that we will use to create our materialized views. The non-materialized views do not store the results of the query: non-materialized view CREATE VIEW sensors_data AS
    SELECT
        *
    FROM (
        SELECT
            (data->>'id')::int AS id,
            (data->>'pm25')::double AS pm25,
            (data->>'pm10')::double AS pm10,
            (data->>'geo_lat')::double AS geo_lat,
            (data->>'geo_lon')::double AS geo_lon,
            (data->>'timestamp')::double AS timestamp
        FROM (
            SELECT CAST(data AS jsonb) AS data
            FROM (
                SELECT convert_from(data, 'utf8') AS data
                FROM sensors
            )
        )
    ); After that, create a materialized view that will hold all records in the last 10 minutes: CREATE MATERIALIZED VIEW sensors_view AS
    SELECT
        *
    FROM sensors_data
    WHERE
        mz_logical_timestamp() < (timestamp*1000 + 100000)::numeric; Note that we are using the function rather than the function. This is because in Materialize doesn’t represent the system time, as it does in most systems; it represents the time with timezone when the query was executed. It cannot be used when creating views. For more information, see the documentation . mz_logical_timestamp() now() now() here Next, let's create materialized view that will only include data from the last second so we can see the dataflow and use it for our Server-Sent Events (SSE) demo later on: CREATE MATERIALIZED VIEW sensors_view_1s AS
    SELECT
        *
    FROM sensors_data
    WHERE
        mz_logical_timestamp() < (timestamp*1000 + 6000)::numeric; With that our materialized views are ready and we can visit the FastAPI demo project in the browser! FastAPI Demo Finally, visit the FastAPI demo app via your browser: Endpoint for all records in the last 10 minutes: http://localhost/sensors SSE Endpoint streaming the latest records as they are generated using : TAIL http://localhost/stream Example response: Materialize Cloud If you want to run the demo on the cloud, you would need the following: A publicly accessible Redpanda/Kafka instance so that you can connect to it. A Materialize Cloud account. You can sign up for a free account to get started with Materialize Cloud. Materialize Cloud If you already have that setup, you would need to make the following changes to the demo project: When creating the source, change the to your Redpanda/Kafka instance: redpanda:9092 CREATE SOURCE sensors
FROM KAFKA BROKER 'your_redpanda_instance:9092' TOPIC 'sensors'
FORMAT BYTES; Change the environment variable to your Materialize Cloud database URL and uncomment the certificate-specific environment variables in the file.
in the file. DATABASE_URL docker-compose.yml docker-compose.yml Download the Materialize instance certificate files from your Materialize Cloud dashboard. Stop the demo To stop the demo, run the following command: docker-compose down -v You can also stop only the data generation container: docker-compose stop datagen Helpful Links Materialize Materialize Cloud Redpanda CREATE SOURCE CREATE VIEW CREATE MATERIALIZED VIEW

How to Use Server-Sent Events (SSE) With FastAPI

How to Build a Decoupled Microservice Using Materialize

Nominated for 2022 - HackerNoon Contributor of the Year - Open Source

Nominated for 2022 - HackerNoon Contributor of the Year - Sql

Nominated for 2022 - HackerNoon Contributor of the Year - Docker

Nominated for 2022 - HackerNoon Contributor of the Year - Kubernetes

Nominated for 2022 - HackerNoon Contributor of the Year - Database

Nominated for 2022 - HackerNoon Contributor of the Year - Books

Too Long; Didn't Read

Make resilience your competitive advantage

FastAPI with Materialize for Real-Time Data Processing: An Essential Guide

FastAPI with Materialize for Real-Time Data Processing: An Essential Guide

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Build your Own Live Chart With Deno, WebSockets, Chart.js and Materialize

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Threats to an Open API Ecosystem

10 Indications That You Should Invest in Automation Via APIs

10 Best Practices for Securing Your API

Build your Own Live Chart With Deno, WebSockets, Chart.js and Materialize

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Threats to an Open API Ecosystem

10 Indications That You Should Invest in Automation Via APIs

10 Best Practices for Securing Your API

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps