Introduction This is a self-contained example of a real-time chart powered by Deno, Web Sockets, Chart.js, and . Materialize is a simple and secure runtime for JavaScript and TypeScript that uses V8. Deno, just like Materialize, is also written in Rust. Deno In this demo, we will build a simple live dashboard app that displays real-time data from a Deno Web Socket server. Deno will then connect to Materialize and our live materialized view to get the latest data and display it in a real-time chart using Chart.js. TAIL Overview Here is a quick overview of the project: A mock service to continually generate user score events. Redpanda instance to store the user score events in a topic. Materialize instance that is connected to the Redpanda instance and ingests the data from the topic in a live materialized view which we can query in real-time using just SQL. A Deno backend service that connects to Materialize and TAIL the live materialized view to get the latest data and display it in a real-time chart. Frontend service that connects to the Deno app via a web socket and displays the data in a real-time chart using Chart.js. Here is a diagram of the project: Prerequisites To run this demo, you need to have the following installed. . Install Docker . Install Docker Compose Running the demo To get started, clone the repository: git clone git clone https://github.com/bobbyiliev/materialize-tutorials.git Then you can access the directory: cd materialize-tutorials/mz-deno-live-dashboard With that you can then build the images: docker-compose build And finally, you can run all the containers: docker-compose up -d It might take a couple of minutes to start the containers and generate the demo data. After that, you can visit in your browser to see the demo: http://localhost Next, let's review the Materialize setup and the Deno backend setup. Materialize setup The Deno service will execute the following DDL statements on boot so that we don't have to run them manually: Create a Kafka source: Creating a source in Materialize does not actually start the data ingestion. You can think of a non-materialized source as just the metadata needed for Materialize to connect to your source but not process any data: CREATE SOURCE score
FROM KAFKA BROKER 'redpanda:9092' TOPIC 'score_topic'
FORMAT BYTES; Create a , that essentially only provides us with an alias for the statements they include: non-materialized view SELECT CREATE VIEW score_view AS
    SELECT
        *
    FROM (
        SELECT
            (data->>'user_id')::int AS user_id,
            (data->>'score')::int AS score,
            (data->>'created_at')::double AS created_at
        FROM (
            SELECT CAST(data AS jsonb) AS data
            FROM (
                SELECT convert_from(data, 'utf8') AS data
                FROM score
            )
        )
    ); Create a materialized view: CREATE MATERIALIZED VIEW score_view_mz AS
    SELECT
        (SUM(score))::int AS user_score,
        user_id
    FROM score_view GROUP BY user_id; To check if the views and the sources were created, launch the Materialize CLI: docker-compose run mzcli This is just a shortcut to a docker container with postgres-client pre-installed, if you already have you could run . psql psql -U materialize -h localhost -p 6875 materialize Then check the views and the sources: SHOW VIEWS;
-- Output:
-- +-----------------+
-- | score_view      |
-- | score_view_mz   |
-- +-----------------+

SHOW sources;
-- Output:
-- +-----------------+
-- | score           |
-- +-----------------+ Using TAIL Next, to see the results in real-time we can use : TAIL COPY ( TAIL score_view_mz ) TO STDOUT; You will see a flow of the new user score that was generated in real-time. We can also start a without a snapshot, which means that you will only see the latest records after the query is run: TAIL COPY ( TAIL score_view_mz WITH (SNAPSHOT = false) ) TO STDOUT; This is what we will use in our Deno application to get the top user scores and display them in a real-time chart. For more information on how the function works, see the . TAIL Materialize documentation Deno Now that we have Materialize ready, let's review the Deno setup. We would use two Deno modules: The Postgres module to connect to Materialize. The Web Sockets module to create a Web Socket connection to our Frontend service. You can find the code in the directory. backend import { WebSocketClient, WebSocketServer } from "https://deno.land/x/websocket@v0.1.4/mod.ts";
import { Client } from "https://deno.land/x/postgres/mod.ts";

// Specify your Materialize connection details
const client = new Client({
  user: "materialize",
  database: "materialize",
  hostname: "materialized",
  port: 6875,
});

await client.connect();
console.log("Connected to Postgres");

// Start a transaction
await client.queryObject('BEGIN');
// Declare a cursor without a snapshot
await client.queryObject(`DECLARE c CURSOR FOR TAIL score_view_mz WITH (SNAPSHOT = false)`);

const wss = new WebSocketServer(8080);

wss.on("connection", async function (ws: WebSocketClient) {
  console.log("Client connected");
  setInterval(async () => {
    const result = await client.queryObject<{ mz_timestamp: string; mz_diff: number, user_id: number, user_score: number}>(`FETCH ALL c`);
    for (const row of result.rows) {
      let message = { user_id: row.user_id, user_score: row.user_score };
      broadcastEvent(message);
    }
  } , 1000);

});

// Broadcast a message to all clients
const broadcastEvent = (message: any) => {
  wss.clients.forEach((ws: WebSocketClient) => {
    ws.send(JSON.stringify(message));
  });
} Rundown of the code: As Materialize is Postgres wire compatible, first we import the class from the module. This is the class that we will use to connect to the Materialize instance. Client https://deno.land/x/postgres/mod.ts Next, we create a new instance and pass it the credentials for Materialize. Client Then we call the method on the client instance to connect to Materialize. connect() Next, we call the method on the client instance to start a transaction and also call the method on the client instance to declare a cursor without a snapshot. queryObject() queryObject() Finally, we create a new instance and pass it the port to listen on. WebSocketServer We then define a event handler on the instance, which is called when a client connects. connection WebSocketServer We then set an interval to fetch the latest data from Materialize and broadcast it to all clients. Frontend Setup For the frontend, we will not be using any JavaScript framework, but just the . Chart.js library Thanks to the web sockets connection, we can now receive the latest data from Materialize and display it in a real-time chart. <!DOCTYPE html>
<html lang="en">
    <head>
        <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
    </head>
    <body>
        <div class="w-full mt-10">
            <canvas id="myChart"></canvas>
        </div>
    <script>
      const ctx = document.getElementById("myChart");
      const myChart = new Chart(ctx, {
        type: "bar",
        data: {
          labels: [ "Player 1", "Player 2", "Player 3", "Player 4", "Player 5", "Player 6" ],
          datasets: [
            {
              label: "# of points",
              data: [0, 0, 0, 0, 0, 0],
              backgroundColor: [
                "rgba(255, 99, 132, 0.2)",
                "rgba(54, 162, 235, 0.2)",
                "rgba(255, 206, 86, 0.2)",
                "rgba(75, 192, 192, 0.2)",
                "rgba(153, 102, 255, 0.2)",
                "rgba(255, 159, 64, 0.2)",
              ],
              borderColor: [
                "rgba(255, 99, 132, 1)",
                "rgba(54, 162, 235, 1)",
                "rgba(255, 206, 86, 1)",
                "rgba(75, 192, 192, 1)",
                "rgba(153, 102, 255, 1)",
                "rgba(255, 159, 64, 1)",
              ],
              borderWidth: 1,
            },
          ],
        },
        options: {
          scales: {
            y: {
              beginAtZero: true,
            },
          },
        },
      });

      webSocket = new WebSocket("ws://127.0.0.1:8080");
      webSocket.onmessage = function (message) {
        const data = message.data;
        const dataObj = JSON.parse(data);
        const dataArray = Object.values(dataObj);
        console.log(dataArray);
        index = dataArray[0] - 1;
        myChart.data.datasets[0].data[index] = dataArray[1];
        myChart.update();
      };
    </script>
  </body>
</html> Rundown of the code: We first define the new chart using the library: and pass the different configuration options. Chart.js new Chart() Then we create a new instance and pass it the URL of the Web Socket server with WebSocket webSocket = new WebSocket("ws://backend:8080"); Finally, we define an event handler on the instance, which is called when a message is received and updates the chart. onmessage WebSocket You can find the code in the directory. frontend Conclusion You can leave the Deno application running so that it would be subscribed to the Materialize instance and update the chart in real time. As a next step you can check out the Materialize + dbt + Redpanda demo which is based on the same user reviews mock data: Materialize + dbt + Redpanda demo Helpful resources: TAIL CREATE SOURCE CREATE VIEWS SELECT Community If you have any questions or comments, please join the ! Materialize Slack Community Also published here

Flow

Fetch

Slack

Build a Live Dashboard with Materialize, Airbyte, MySQL and Redpanda/Kafka

Nominated for 2022 - HackerNoon Contributor of the Year - Open Source

Nominated for 2022 - HackerNoon Contributor of the Year - Sql

Nominated for 2022 - HackerNoon Contributor of the Year - Docker

Nominated for 2022 - HackerNoon Contributor of the Year - Kubernetes

Nominated for 2022 - HackerNoon Contributor of the Year - Database

Nominated for 2022 - HackerNoon Contributor of the Year - Books

Too Long; Didn't Read

Build your Own Live Chart With Deno, WebSockets, Chart.js and Materialize

Build your Own Live Chart With Deno, WebSockets, Chart.js and Materialize

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

An In-Depth Guide to Blue-Green Deployments with Materialize

10 Steps To Digital Transformation While Simultaneously Cutting Costs

10 Minute Guide to Fixing Damaged SQL Databases - No Recovery Required!

10 Cool CI/CD Tools For Your Project

121 Stories To Learn About Databases

14 Open Source SQL Parsers

An In-Depth Guide to Blue-Green Deployments with Materialize

10 Steps To Digital Transformation While Simultaneously Cutting Costs

10 Minute Guide to Fixing Damaged SQL Databases - No Recovery Required!

10 Cool CI/CD Tools For Your Project

121 Stories To Learn About Databases

14 Open Source SQL Parsers

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps