We created a new video streaming app to help you learn about ScyllaDB. This blog covers its features, tech stack, and data modeling. We created a new video streaming app to help you learn about ScyllaDB. This blog covers its features, tech stack, and data modeling. We just published a new ScyllaDB sample application, a video streaming app. The project is available on GitHub. This blog covers the video streaming application’s features, tech stack, and breaks down the data modeling process. available on GitHub Video streaming app features The app has a minimal design with the most essential video streaming application features: List all videos, sorted by creation date (home page)List videos that you started watchingWatch videoContinue watching a video where you left offDisplay a progress bar under each video thumbnail List all videos, sorted by creation date (home page) List videos that you started watching Watch video Continue watching a video where you left off Display a progress bar under each video thumbnail Technology stack Programming language: TypeScriptDatabase: ScyllaDBFramework: NextJS (pages router)Component library: Material_UI Programming language: TypeScript Programming language: TypeScript Database: ScyllaDB Database: ScyllaDB Framework: NextJS (pages router) Framework: NextJS Component library: Material_UI Component library: Material_UI Using ScyllaDB for low-latency video streaming applications ScyllaDB is a low-latency and high-performance NoSQL database compatible with Apache Cassandra and DynamoDB. It is well-suited to handle the large-scale data storage and retrieval requirements of video streaming applications. ScyllaDB has drivers in all the popular programming languages and, as this sample application demonstrates, it integrates well with modern web development frameworks like NextJS. Low latency in the context of video streaming services is crucial for delivering a seamless user experience. To lay the groundwork for high performance, you need to design a data model that fits your needs. Let’s continue with an example data modeling process to see what that looks like. Video streaming app data modeling In the ScyllaDB University Data Modeling course, we teach that NoSQL data modeling should always start with your application and queries first. Then you work backwards and create the schema based on the queries you want to run in your app. This process ensures that you create a data model that fits your queries and meets your requirements. ScyllaDB University Data Modeling course With that in mind, let’s go over the queries that our video streaming app needs to run on each page load! Page: Continue Watching On this page, you can list all the videos that they’ve started to watch. This view includes the video thumbnails and the progress bar under the thumbnail. Query – get watch progress: Query – get watch progress: SELECT video_id, progress FROM watch_history WHERE user_id = ? LIMIT 9; SELECT video_id, progress FROM watch_history WHERE user_id = ? LIMIT 9; Schema – watch history table: Schema – watch history table: CREATE TABLE watch_history ( user_id text, video_id text, progress int, watched_at timestamp, PRIMARY KEY (user_id) ); CREATE TABLE watch_history ( user_id text, video_id text, progress int, watched_at timestamp, PRIMARY KEY (user_id) ); For this query, it makes sense to define `user_id` as the partition key because that is the filter we use to query the watch history table. Keep in mind that this schema might need to be updated later if there is a query that requires filtering on other columns beyond the `user_id`. For now, though, this schema is correct for the defined query. Besides the progress value, the app also needs to fetch the actual metadata of each video (for example, the title and the thumbnail image). For this, the `video` table has to be queried. Query – get video metadata: Query – get video metadata: SELECT * FROM video WHERE id IN ?; SELECT * FROM video WHERE id IN ?; Notice how we use the “IN” operator and not “=” because we need to fetch a list of videos not just a single video. Schema – video table: Schema – video table: CREATE TABLE video ( id text, content_type text, title text, url text, thumbnail text, created_at timestamp, duration int, PRIMARY KEY (id) ); CREATE TABLE video ( id text, content_type text, title text, url text, thumbnail text, created_at timestamp, duration int, PRIMARY KEY (id) ); For the video table, let’s define the `id` as the partition key because that’s the only filter we use in the query. Page: Watch Video If you click on any of the “Watch” buttons, they will be redirected to a page with a video player where they can start and pause the video. Query – get video content: Query – get video content: SELECT * FROM video WHERE id = ?; SELECT * FROM video WHERE id = ?; This is a very similar query to the one that runs on the Continue Watching page. Thus, the same schema will work just fine for this query as well. Schema – video table: Schema – video table: CREATE TABLE video ( id text, content_type text, title text, url text, thumbnail text, created_at timestamp, duration int, PRIMARY KEY (id) ); CREATE TABLE video ( id text, content_type text, title text, url text, thumbnail text, created_at timestamp, duration int, PRIMARY KEY (id) ); Page: Most Recent Videos Finally, let’s break down the Most Recent Videos page, which is the home page of the application. We analyze this page last because it is the most complex one from a data modeling perspective. This page lists ten of the most recently uploaded videos that are available in the database ordered by the video creation date. We will have to fetch these videos in two steps: first get the timestamps, then get the actual video content. Query – get the most recent ten videos’ timestamp: Query – get the most recent ten videos’ timestamp: SELECT id, top10(created_at) AS date FROM recent_videos; SELECT id, top10(created_at) AS date FROM recent_videos; You might notice that we use a custom function called `top10()`. This is not a standard function in ScyllaDB. It’s a UDF (user-defined function) that we created to solve this data modeling problem. This function returns an array of the most recent `created_at` timestamps in the table. Creating a new UDF in ScyllaDB can be a great way to solve your unique data modeling challenges. user-defined function Creating a new UDF in ScyllaDB These timestamp values can then be used to query the actual video content that we want to show on the page. Query – get metadata for those videos: Query – get metadata for those videos: SELECT * FROM recent_videos WHERE created_at IN ? LIMIT 10; SELECT * FROM recent_videos WHERE created_at IN ? LIMIT 10; Schema – recent videos: Schema – recent videos: CREATE MATERIALIZED VIEW recent_videos_view AS SELECT * FROM streaming.video WHERE created_at IS NOT NULL PRIMARY KEY (created_at, id); CREATE MATERIALIZED VIEW recent_videos_view AS SELECT * FROM streaming.video WHERE created_at IS NOT NULL PRIMARY KEY (created_at, id); In the recent videos materialized view, the `created_at` column is the primary key because we filter by that column in our first query to get the most recent timestamp values. Be aware that in some cases, this can cause a hot partition. hot partition Furthermore, the UI also shows a small progress bar under each video’s thumbnail which indicates the progress you made watching that video. To fetch this value for each video, the app has to query the `watch history` table. Query – get watch progress for each video: Query – get watch progress for each video: SELECT progress FROM watch_history WHERE user_id = ? AND video_id = ?; SELECT progress FROM watch_history WHERE user_id = ? AND video_id = ?; Schema – watch history: Schema – watch history: CREATE TABLE watch_history ( CREATE TABLE watch_history ( user_id text, user_id text, video_id text, video_id text, progress int, progress int, watched_at timestamp, watched_at timestamp, PRIMARY KEY (user_id, video_id) PRIMARY KEY (user_id, video_id) ); ); You might have noticed that the watch history table was already used in a previous query to fetch data. Now this time, the schema has to be modified slightly to fit this query. Let’s add `video_id` as a clustering key. This way, the query to fetch watch progress will work correctly. That’s it. Now let’s see the final database schema! Final database schema CREATE KEYSPACE IF NOT EXISTS streaming WITH replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': '3' }; CREATE TABLE streaming.video ( id text, content_type text, title text, url text, thumbnail text, created_at timestamp, duration int, PRIMARY KEY (id) ); CREATE TABLE streaming.watch_history ( user_id text, video_id text, progress int, watched_at timestamp, PRIMARY KEY (user_id, video_id) ); CREATE TABLE streaming.recent_videos ( id text, content_type text, title text, url text, thumbnail text, created_at timestamp, duration int, PRIMARY KEY (created_at) ); CREATE KEYSPACE IF NOT EXISTS streaming WITH replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': '3' }; CREATE TABLE streaming.video ( id text, content_type text, title text, url text, thumbnail text, created_at timestamp, duration int, PRIMARY KEY (id) ); CREATE TABLE streaming.watch_history ( user_id text, video_id text, progress int, watched_at timestamp, PRIMARY KEY (user_id, video_id) ); CREATE TABLE streaming.recent_videos ( id text, content_type text, title text, url text, thumbnail text, created_at timestamp, duration int, PRIMARY KEY (created_at) ); User-defined function for the Most Recent Videos page -- Create a UDF for recent videos CREATE OR REPLACE FUNCTION state_f(acc list<timestamp>, val timestamp) CALLED ON NULL INPUT RETURNS list<timestamp> LANGUAGE lua AS $$ if val == nil then return acc end if acc == nil then acc = {} end table.insert(acc, val) table.sort(acc, function(a, b) return a > b end) if #acc > 10 then table.remove(acc, 11) end return acc $$; CREATE OR REPLACE FUNCTION reduce_f(acc1 list<timestamp>, acc2 list<timestamp>) CALLED ON NULL INPUT RETURNS list<timestamp> LANGUAGE lua AS $$ result = {} i = 1 j = 1 while #result < 10 do if acc1[i] > acc2[j] then table.insert(result, acc1[i]) i = i + 1 else table.insert(result, acc2[j]) j = j + 1 end end return result $$; CREATE OR REPLACE AGGREGATE top10(timestamp) SFUNC state_f STYPE list<timestamp> REDUCEFUNC reduce_f; -- Create a UDF for recent videos CREATE OR REPLACE FUNCTION state_f(acc list<timestamp>, val timestamp) CALLED ON NULL INPUT RETURNS list<timestamp> LANGUAGE lua AS $$ if val == nil then return acc end if acc == nil then acc = {} end table.insert(acc, val) table.sort(acc, function(a, b) return a > b end) if #acc > 10 then table.remove(acc, 11) end return acc $$; CREATE OR REPLACE FUNCTION reduce_f(acc1 list<timestamp>, acc2 list<timestamp>) CALLED ON NULL INPUT RETURNS list<timestamp> LANGUAGE lua AS $$ result = {} i = 1 j = 1 while #result < 10 do if acc1[i] > acc2[j] then table.insert(result, acc1[i]) i = i + 1 else table.insert(result, acc2[j]) j = j + 1 end end return result $$; CREATE OR REPLACE AGGREGATE top10(timestamp) SFUNC state_f STYPE list<timestamp> REDUCEFUNC reduce_f; This UDF uses Lua, but you could also use Wasm to create UDFs in ScyllaDB. Creating the function make sure to enable UDFs in the scylla.yaml configuration file (location: /etc/scylla/scylla.yaml): UDF uses Lua use Wasm to create UDFs in ScyllaDB Clone the repo and get started! To get started… Clone the repository: git clone https://github.com/scylladb/video-streaming git clone https://github.com/scylladb/video-streaming https://github.com/scylladb/video-streaming Install the dependencies: npm install npm install Modify the configuration file: APP_BASE_URL="http://localhost:8000" SCYLLA_HOSTS="172.17.0.2" SCYLLA_USER="scylla" SCYLLA_PASSWD="xxxxx" SCYLLA_KEYSPACE="streaming" SCYLLA_DATACENTER="datacenter1" APP_BASE_URL="http://localhost:8000" SCYLLA_HOSTS="172.17.0.2" SCYLLA_USER="scylla" SCYLLA_PASSWD="xxxxx" SCYLLA_KEYSPACE="streaming" SCYLLA_DATACENTER="datacenter1" Migrate the database and insert sample data: npm run migrate npm run migrate Run the server: npm run dev npm run dev Wrapping up We hope you enjoy our video streaming app and it helps you build low-latency and high-performance applications with ScyllaDB. If you want to keep on learning, check out ScyllaDB University where we have free courses on data modeling, ScyllaDB drivers, and much more! If you have questions about the video streaming sample app or ScyllaDB, go to our forum and let’s discuss! ScyllaDB University data modeling ScyllaDB drivers go to our forum More ScyllaDB sample applications: CarePet – IoTCloud getting started guideFeature Store CarePet – IoT CarePet – IoT Cloud getting started guide Cloud getting started guide Feature Store Feature Store Relevant resources: Video streaming app GitHub repositoryUDFs in ScyllaDBHow ScyllaDB Distributed Aggregates Reduce Query Execution Time up to 20XWasmtime: Supporting UDFs in ScyllaDB with WebAssemblyScyllaDB documentation Video streaming app GitHub repository Video streaming app GitHub repository UDFs in ScyllaDB UDFs in ScyllaDB How ScyllaDB Distributed Aggregates Reduce Query Execution Time up to 20X How ScyllaDB Distributed Aggregates Reduce Query Execution Time up to 20X Wasmtime: Supporting UDFs in ScyllaDB with WebAssembly Wasmtime: Supporting UDFs in ScyllaDB with WebAssembly ScyllaDB documentation ScyllaDB documentation ScyllaDB Getting Started ScyllaDB Getting Started ScyllaDB Getting Started About Attila Tóth Attila Tóth is a developer advocate at ScyllaDB. He writes tutorials and blog posts, speaks at events, creates demos and sample applications to help developers build high-performance applications. Attila Tóth
![Tensor Flow - [Technical Documentation]](https://hackernoon.imgix.net/images/undefined-q683u5r.jpeg?auto=format&fit=max&w=96)
