Brevio is a UK social enterprise on a mission to simplify grant research, matching and application, reducing admin and increasing impact across the third sector. 🤝 A grants research, matching & application platform for non-profits 📊 Data-powered solutions for grant givers We automate the initial steps in grant research and application by matching non-profits with grant givers, to free up hundreds of millions of pounds in admin each year.

Brevio

Data and Its Color

Performance Analysis of Diverse Hateful Meme Detection Datasets

Analyzing Audience Responses to Ukrainian Memetic Warfare Tactics

How to Create a Blog using Golang

After getting fed up with React, SPAs, and Javascript around 2021 I decided to re-write my webpage in Rust, after which on how to build a simple blog entirely with Rust. It ended up becoming one of my most popular articles and for good reason; Rust is exciting, fun to write, and blazingly fast. After a while though, I started to feel frustrated with the development process for adding new features to my site: the feedback loop was simply too long. I wrote an article I've always been interested in solo entrepreneurship and technology. But, as I'm getting older, I realize that I might have been more interested in trying out new technologies. Great for learning and growing as an engineer, bad for shipping projects, and starting to see that sweet MRR grow. I decided to re-re-write my site once again, this time in Go for multiple reasons: I write Go for a living Simple language, with a decent type system and fast performance Blazingly fast compile times In this post, I will show how you can create your blog using . I'll assume you're familiar with Go, and know how to configure a router/database/server, and so on. Should you not, feel free to grab a clone of my Go starter template that has lots of things configured for you out of the box. Go Grafto Foundations I write all my stuff in markdown; if you're a developer who also wants to start blogging, chances are you also are quite familiar with it. After Go 1.16 where we got the package included in the standard library most of our work is already done. We basically only need to have a method for storing some filenames, associating them with an ID or a slug, grabbing the file, and serving it to the user. Pretty simple. embed Whether you've created your own setup or grabbed a copy of Grafto, create a new directory in the root of your project called and in there, create a file called . posts posts.go Open it and add the following: package posts

// imports omitted

//go:embed *.md
var Assets embed.FS Now, any file with the extension will get included in the binary that ultimately gets built once we run . We can simply grab the files from our global variable using to handle any error that might occur or return the file as a string, e.g: .md go run build Assets Assets.ReadFile(name-of-file) file, err := Assets.ReadFile("my-post.md")
if err != nil {
	return err
}

return string(file) It won't be pretty (we'll fix that later), but it gets the basic idea out, which we can build on. We have a way to include our blog posts in the binary; let's add a way to associate them with a slug. You could use an ID here, but it just looks better to have URLs like: " " compared to " ". https://acme.com/my-first-blog-post http://acme.com/44a2530b-567d-472f-9495-e2ee64e7ae6d So, assuming you have a database up and running, add this table: create table posts (
    id uuid primary key,
    created_at timestamp not null,
    updated_at timestamp not null,
    title varchar(255) not null,
    filename varchar(255) not null,
    slug varchar(255) not null
); Not much exciting going on here. Basically, for the unaware readers, the slug column above will be the title but URL-friendly. So if you have a post with the title "My First Blog Post" the slug equivalent would be "my-first-blog-post" easy. Lastly, we need to be able to serve this to readers. That flow would typically involve them hitting a landing page showing a list of articles they can choose from, which links to the article. The implementation of this will depend a bit on your setup, but let us implement the handler to deal with grabbing a specific article using Echo as our router. Assuming you have a route like , create the following: /posts/:slug type ArticleStorage interface {
	GetPostBySlug(slug string) (Post, error)
}

func ArticleHandler(ctx echo.Context, storage ArticleStorage) error {
	postSlug := ctx.Param("postSlug")
	
	postModel, err := storage.GetPostBySlug(postSlug)
	if err != nil {
		return err
	}
	
	postContent, err := posts.Assets.ReadFile(postModel.Filename)
	if err != nil {
		return err
	}

	return ctx.String(http.StatusOK, string(postContent))
} I'm putting some decisions on your here in terms of implementing the ArticleStorage. We just need that grabs the data on the post from the DB, based on the slug. something This is the foundation of what we need...but it's not pretty let's fix that by letting the server do what it was always supposed to do: return HTML. Enter templ If you've spent time in the Go ecosystem, chances are you've probably heard about . It lets you write templates as Go packages and it's just such a pleasant way of building out a UI. Add some HTMX and alpine.js and you've at least 95% of what you get with SPAs, with the added complexity. templ HTML It's good practice to have a base template that wraps around your other templates, so we have a single point for adding things like stylesheets, javascript, metadata, etc. Create a directory in root called views and add the following to a file called . base.templ package views

templ base() {
	<!DOCTYPE html>
	<html lang="en">
		<nav>
			<a href="/">MBV Labs</a>
		</nav>
		<body>
			{ children... }
			<footer>
				<aside>
					<p>Copyright ©2024 </p>
					<p>All right reserved by MBV Labs</p>
				</aside>
			</footer>
		</body>
	</html>
} For this to work, you'll need to install templ and run which will produce a file called that we can then import into other templates to wrap around them. For the sake of brevity, we'll only create the template to show the actual article. templ generate base_templ.go Create a file called , and add the following: article.templ type ArticlePageData struct {
	Title             string
	Content           string
}

templ ArticlePage(data ArticlePageData) {
	@layouts.Base() {
		<div>
			<div>
				<h1>{ data.Title }</h1>
			</div>
			<article>
				@unsafe(data.Content)
			</article>
		</div>
	}
} and run once again. templ generate We can now go back and update our ArticleHandler handler: func ArticleHandler(ctx echo.Context, storage ArticleStorage) error {
	postSlug := ctx.Param("postSlug")
	
	postModel, err := storage.GetPostBySlug(postSlug)
	if err != nil {
		return err
	}
	
	postContent, err := posts.Assets.ReadFile(postModel.Filename)
	if err != nil {
		return err
	}

	return views.ArticlePage(
		views.ArticlePagedata{
			Title: postModel.Title,
			Content: postContent,
		},
	).Render(
		ctx.Request().Context(), 
		ctx.Response().Writer,
	)
} If you run your application now and visit a valid URL, you should see a (rather ugly) page showing the markdown of your article but this time, with some sweet hypertext markup. Making things (slightly) less ugly In terms of styling things, throwing some tailwind or vanilla CSS at what we have now will get you a long way. But, we still show raw markdown to the user when they visit our articles. Additionally, we might want to show some nicely formatted code snippets in our articles. Let's fix this now. For this, we need something that can transform the markdown into HTML components e.g ## Some sub header into <h2>Some sub header</h2> Luckily, a create library already exists for this: Goldmark. So let's refactor the file to parse the content we store using embed. posts/posts.go //go:embed *.md
var assets embed.FS // unexport assets

type Manager struct {
	posts embed.FS
	markdownParser goldmark.Markdown
}

func NewManager() Manager {
	md := goldmark.New(
		goldmark.WithParserOptions(
			parser.WithAutoHeadingID(),
			parser.WithAttribute(),
		),
		goldmark.WithRendererOptions(
			html.WithHardWraps(),
			html.WithXHTML(),
			html.WithUnsafe(),
		),
	)

	return Manager{
		posts:           assets,
		markdownHandler: md,
	}
}

func (m *Manager) Parse(name string) (string, error) {
	source, err := m.posts.ReadFile(name)
	if err != nil {
		return "", err
	}

	// Parse Markdown content
	var htmlOutput bytes.Buffer
	if err := m.markdownHandler.Convert(source, &htmlOutput); err != nil {
		return "", err
	}

	return htmlOutput.String(), nil
} Lastly, update the ArticleHandler to use the Manager: func ArticleHandler(
	ctx echo.Context, 
	storage ArticleStorage,
	postManager posts.Manager
) error {
	postSlug := ctx.Param("postSlug")
	
	postModel, err := storage.GetPostBySlug(postSlug)
	if err != nil {
		return err
	}
	
	postContent, err := postManager.Parse(postModel.Filename)
	if err != nil {
		return err
	}
	

	return views.ArticlePage(
		views.ArticlePagedata{
			Title: postModel.Title,
			Content: postContent,
		},
	).Render(
		ctx.Request().Context(), 
		ctx.Response().Writer,
	)
} Try and edit your article by adding some code blocks and they will now get nicely formatted. You can add custom themes to the parser so your code snippets will be shown with your favorite theme.

How to Build a Blog with Go

Introduction The article demonstrates the use of null-state analysis to remove compiler warnings of “check code for null safety”. Learning Objectives Discover how to set up the nullable context in your C# project or codebase. Prerequisites for Developers Familiarity with introductory-level C# programming Know how to use Visual Studio Code or Visual Studio .NET SDK version 6.0 or newer Getting Started .NET developers often face the System.NullReferenceException, which happens when a null is referenced at runtime and results in the most common exception in the .NET application. As the creator of null, Sir Tony Hoare, refers to null as the "billion-dollar mistake." Example The variable records is set to null and then immediately referenced which results in System.NullReferenceException TestRecord records = null;
    
    _ = records.ToString();
    
    record TestRecord(int Id, string Name); As the applications grow in the number of lines of code and become more complex, spotting such issues as a developer can be challenging. This is where C# compiler steps in. Define null safety In the previous example, a developer can avoid the System.NullReferenceExceptionby checking if the records variable was null as shown below. TestRecord records = null;
    
    // Check for null
    if (records is not null)
    {
        _ = records.ToString();
    }
    
    record TestRecord(int Id, string Name); Nullable Types The default value for all reference types is null. string first;                  // first is null
    string second = string.Empty   // second is not null, instead it's an empty string ""
    int third;                     // third is 0 because int is a value type
    DateTime date;                 // date is DateTime.MinValue In the example mentioned earlier: The variable “first” is null because a reference type “string” was declared but not assigned any value. The variable “second” is assigned the value “string.Empty” during declaration. The variable “third” has a value of 0 even though it was not explicitly assigned. The variable “date” is uninitialized, but its default value is “System.DateTime.MinValue.” Post C# 2.0 version, we can define nullable values using Nullable<T>. This allowed value types to be assigned with a value of null. int? first;            // first is implicitly null (uninitialized)
    int? second = null;    // second is explicitly null
    int? third = default;  // third is null as the default value for Nullable<Int32> is null
    int? fourth = new();    // fourth is 0, since new calls the nullable constructor Nullable context As per my experience, this is a must-have feature that should be enabled in every .Net application as it enables control for how the compiler understands reference type variables. There are four types of nullable contexts disable enable warnings annotations Enable nullable context it can be enabled by adding <Nullable> item to the <PropertyGroup> inside the application .csproj file as shown below <Project Sdk="Microsoft.NET.Sdk">
    
        <PropertyGroup>
            <OutputType>Exe</OutputType>
            <TargetFramework>net6.0</TargetFramework>
            <Nullable>enable</Nullable>
        </PropertyGroup>
    
        
    
    </Project> Alternatively, developers can also add scope nullable context which means the nullable context will be applicable only in the defined scope. #nullable enable Complete Code on GitHub GitHub — ssukhpinder/30DayChallenge.Net C# Programming🚀 Thank you for being a part of the C# community! Before you leave: If you’ve made it this far, please show your appreciation with a clap and follow the author! 👏️️ Follow us: | | | | | X LinkedIn Dev.to Hashnode Newsletter Tumblr Visit our other platforms: | | | | GitHub Instagram Tiktok Quora Daily.dev More content at C# Programming Also published . here

The 30-Day .NET Challenge, Day 9: Null Safety

I blogged about in the past, but I skipped an important method that's worthy of a post of its own: peek. This blog post delves into the practicalities of using to debug Java streams, complete with samples and common pitfalls. Java Stream debugging peek() code Understanding Java Streams Java Streams represent a significant shift in how Java developers work with collections and data processing, introducing a functional approach to handling sequences of elements. Streams facilitate declarative processing of collections, enabling operations such as filter, map, reduce, and more in a fluent style. This not only makes the code more readable but also more concise compared to traditional iterative approaches. A Simple Stream Example To illustrate, consider the task of filtering a list of names to only include those that start with the letter "J" and then transform each name into uppercase. Using the traditional approach, this might involve a loop and some if statements. However, with streams, this can be accomplished in a few lines: List<String> names = Arrays.asList("John", "Jacob", "Edward", "Emily");
// Convert list to stream
List<String> filteredNames = names.stream()       
                  // Filter names that start with "J"
                  .filter(name -> name.startsWith("J"))  
                  // Convert each name to uppercase
                  .map(String::toUpperCase)              
                  // Collect results into a new list
                  .collect(Collectors.toList());         
System.out.println(filteredNames); Output: [JOHN, JACOB] This example demonstrates the power of Java streams: by chaining operations together, we can achieve complex data transformations and filtering with minimal, readable code. It showcases the declarative nature of streams, where we describe what we want to achieve rather than detailing the steps to get there. What is the Method? peek() At its core, is a method provided by the interface, allowing developers a glance into the elements of a stream without disrupting the flow of its operations. The signature of is as follows: peek() Stream peek() Stream<T> peek(Consumer<? super T> action) It accepts a functional interface, which means it performs an action on each element of the stream without altering them. The most common use case for is logging the elements of a stream to understand the state of data at various points in the stream pipeline. To understand peek, let’s look at a sample similar to the previous one: Consumer peek() List<String> collected = Stream.of("apple", "banana", "cherry")
                               .filter(s -> s.startsWith("a"))
                               .collect(Collectors.toList());
System.out.println(collected); This code filters a list of strings, keeping only the ones that start with "a.” While it's straightforward, understanding what happens during the filter operation is not visible. Debugging with peek() Now, let's incorporate to gain visibility into the stream: peek() List<String> collected = Stream.of("apple", "banana", "cherry")
                               .peek(System.out::println) // Logs all elements
                               .filter(s -> s.startsWith("a"))
                               .peek(System.out::println) // Logs filtered elements
                               .collect(Collectors.toList());
System.out.println(collected); By adding both before and after the operation, we can see which elements are processed and how the filter impacts the stream. This visibility is invaluable for debugging, especially when the logic within the stream operations becomes complex. peek() filter We can't step over stream operations with the debugger, but provides a glance into the code that is normally obscured from us. peek() Uncovering Common Bugs with peek() Filtering Issues Consider a scenario where a filter condition is not working as expected: List<String> collected = Stream.of("apple", "banana", "cherry", "Avocado")
                               .filter(s -> s.startsWith("a"))
                               .collect(Collectors.toList());
System.out.println(collected); Expected output might be , but let's say we also wanted "Avocado" due to a misunderstanding of the method's behavior. Since "Avocado" is spelled with an upper case "A" this code will return false: . Using , we can observe the elements that pass the filter: ["apple"] startsWith Avocado".startsWith("a") peek() List<String> debugged = Stream.of("apple", "banana", "cherry", "Avocado")
                              .peek(System.out::println)
                              .filter(s -> s.startsWith("a"))
                              .peek(System.out::println)
                              .collect(Collectors.toList());
System.out.println(debugged); Large Data Sets In scenarios involving large datasets, directly printing every element in the stream to the console for debugging can quickly become impractical. It can clutter the console and make it hard to spot the relevant information. Instead, we can use in a more sophisticated way to selectively collect and analyze data without causing side effects that could alter the behavior of the stream. peek() Consider a scenario where we're processing a large dataset of transactions, and we want to debug issues related to transactions exceeding a certain threshold: class Transaction {
    private String id;
    private double amount;

    // Constructor, getters, and setters omitted for brevity
}

List<Transaction> transactions = // Imagine a large list of transactions

// A placeholder for debugging information
List<Transaction> highValueTransactions = new ArrayList<>();

List<Transaction> processedTransactions = transactions.stream()
    // Filter transactions above a threshold
    .filter(t -> t.getAmount() > 5000) 
    .peek(t -> {
        if (t.getAmount() > 10000) {
            // Collect only high-value transactions for debugging
            highValueTransactions.add(t);
        }
     })
     .collect(Collectors.toList());

// Now, we can analyze high-value transactions separately, without overloading the console
System.out.println("High-value transactions count: " + 
       highValueTransactions.size()); In this approach, is used to inspect elements within the stream conditionally. High-value transactions that meet a specific criterion (e.g., amount > 10,000) are collected into a separate list for further analysis. This technique allows for targeted debugging without printing every element to the console, thereby avoiding performance degradation and clutter. peek() Addressing Side Effects While streams shouldn't have side effects. In fact, such side effects would break the stream debugger in IntelliJ, which I discussed in the past. It's crucial to note that while collecting data for debugging within avoids cluttering the console, it does introduce a side effect to the stream operation, which goes against the recommended use of streams. Streams are designed to be side-effect-free to ensure predictability and reliability, especially in parallel operations. peek() Therefore, while the above example demonstrates a practical use of for debugging, it's important to use such techniques judiciously. Ideally, this debugging strategy should be temporary and removed once the debugging session is completed to maintain the integrity of the stream's functional paradigm. peek() Limitations and Pitfalls While is undeniably a useful tool for debugging Java streams, it comes with its own set of limitations and pitfalls that developers should be aware of. Understanding these can help avoid common traps and ensure that is used effectively and appropriately. peek() peek() Potential for Misuse in Production Code One of the primary risks associated with is its potential for misuse in production code. Because is intended for debugging purposes, using it to alter state or perform operations that affect the outcome of the stream can lead to unpredictable behavior. This is especially true in parallel stream operations, where the order of element processing is not guaranteed. Misusing in such contexts can introduce hard-to-find bugs and undermine the declarative nature of stream processing. peek() peek() peek() Performance Overhead Another consideration is the performance impact of using . While it might seem innocuous, can introduce a significant overhead, particularly in large or complex streams. This is because every action within is executed for each element in the stream, potentially slowing down the entire pipeline. When used excessively or with complex operations, can degrade performance, making it crucial to use this method judiciously and remove any calls from production code after debugging is complete. peek() peek() peek() peek() peek() Side Effects and Functional Purity As highlighted in the enhanced debugging example, can be used to collect data for debugging purposes, but this introduces side effects to what should ideally be a side-effect-free operation. The functional programming paradigm, which streams are a part of, emphasizes purity and immutability. Operations should not alter state outside their scope. By using to modify external state (even for debugging), you're temporarily stepping away from these principles. While this can be acceptable for short-term debugging, it's important to ensure that such uses of do not find their way into production code, as they can compromise the predictability and reliability of your application. peek() peek() peek() The Right Tool for the Job Finally, it's essential to recognize that is not always the right tool for every debugging scenario. In some cases, other techniques, such as logging within the operations themselves, using breakpoints and inspecting variables in an IDE, or writing unit tests to assert the behavior of stream operations, might be more appropriate and effective. Developers should consider as one tool in a broader debugging toolkit, employing it when it makes sense and opting for other strategies when they offer a clearer or more efficient path to identifying and resolving issues. peek() peek() Navigating the Pitfalls To navigate these pitfalls effectively: Reserve strictly for temporary debugging purposes. If you have a linter as part of your CI tools it might make sense to add a rule that blocks code from invoking . peek() peek() Always remove calls from your code before committing it to your codebase, especially for production deployments. peek() Be mindful of performance implications and the potential introduction of side effects. Consider alternative debugging techniques that might be more suited to your specific needs or the particular issue you're investigating. By understanding and respecting these limitations and pitfalls, developers can leverage to enhance their debugging practices without falling into common traps or inadvertently introducing problems into their codebases. peek() Final Thoughts The method offers a simple yet effective way to gain insights into Java stream operations, making it a valuable tool for debugging complex stream pipelines. By understanding how to use effectively, developers can avoid common pitfalls and ensure their stream operations perform as intended. As with any powerful tool, the key is to use it wisely and in moderation. peek() peek() The true value of is in debugging massive data sets, these elements are very hard to analyze even with dedicated tools. By using we can dig into said data set and understand the source of the issue programmatically. peek() peek() Also published . here

Debugging Java Streams With The Peek Method

Recommendation systems have become an integral and indispensable part of our lives. These intelligent algorithms are pivotal in shaping our online experiences, influencing the content we consume, the products we buy, and the services we explore. Whether we are streaming content on platforms like , discovering new music on , or shopping online, recommendation systems are quietly working behind the scenes to personalize and enhance our interactions. Netflix Spotify The unique element of these recommendation systems is their ability to understand and predict our preferences based on historical behavior and user patterns. By analyzing our past choices, these systems curate tailored suggestions, saving us time and effort while introducing us to content/products that align with our interests. This enhances user satisfaction and fosters discovery, introducing us to new and relevant offerings that we might not have encountered otherwise. At a high level, developers understand that these algorithms are powered by machine learning and deep learning systems (interchangeably called neural networks), but what if I tell you there is a way to build a recommendation engine without going through the pain of deploying your neural net or machine learning model? This question is specifically relevant in the context of early and mid-stage startups because they don't have tons of structured data to train their models. And as we already know, most machine learning models will not give accurate predictions without proper training data. I recently built and deployed a basic recommendation engine for a , which led to a 40% jump in our key metrics. At the time of writing this blog, the system is generating more than 30 million recommendations per month. Even though this recommendation system was built for a social network, you can apply the basic architecture to any use case, such as product recommendations, music recommendations, content recommendations on text and video platforms, or anything else. Let me start by describing the problem statement. voice-first social network Problem statement from the engineering perspective I had an extensive and because we were building the recommendation system for a product that is already used by thousands of users daily. But to keep this blog short and on point, I will write only the high-level requirements and then discuss the solution of the same. If you are building a recommendation system for your product (simple or neural net-based) and are stuck somewhere, please feel free to contact me on or and I will be more than happy to answer your questions. product requirement document subsequent engineering requirements document Twitter , Linkedin At a high level, we had the following requirements from an engineering perspective - The system should be able to capture a user's interests in the form of keywords. The system should also be able to classify the level of interest a user has with specific keywords. The system should be able to capture a user's interest in other users. It should be able to classify the level of interest a user has in content created by another user. The system should be able to generate high-quality recommendations based on a user's interests. The system should be able to ensure that the recommendations already viewed/rejected by the user shouldn't re-appear again for X number of days. The to ensure that the posts from the same creators aren't grouped on the same page. The system should try its best to ensure that if a user consumes ten posts (our page size), all of those should be from different creators. system should have logic The system should be fast. Less than 150 milliseconds of P99 latency. All the other non-functional requirements, such as high availability, scalability, security, reliability, maintainability, etc, should be fulfilled. Again, this is a highly oversimplified list of problem statements. In reality, the documents were 3000+ words long as they also covered a lot of edge cases and corner cases that can arise while integrating this recommendation engine into our existing systems. Let's move on to the solution. Solution - High-level working of the recommendation engine I will discuss the solutions to the problem one by one and then will describe the overall working of the entire system. Our first problem is capturing the user's interests and defining their interest level with a specific interest For this, we created something called a . To put it simply, a social graph stores the relationships and connections between different entities in a social network. These entities can be different users or a relationship of users with a specific interest. Social graphs are a powerful way to understand and structure the relationships within a particular system. For the sake of brevity, I will not explain the social graph in detail, but I will recommend you google it and learn more about it. Following is a simplified version of the social graph I built for our recommendation engine. social graph As you can see from the above image, we are storing a lot of information, such as the number of interactions (likes, comments, shares) and recency of these interactions (when they happened last) as relationship data between two users as well between a user and an interest. We are even storing the relationship between two different interest keywords. I used , a managed graph database by AWS, to store this social graph. You can use any other graph database, such as , etc. Amazon Neptune Neo4j, JanusGraph, ArrangoDB These interest keywords are predominantly nouns. There is a system in place that breaks down the contents of a post into these keywords(nouns). It's powered by a natural-language processing (NLP) service that uses machine learning to break text into entities, key phrases, etc. Again, you can use any managed NLP services (several available) to accomplish the same. You don't need to learn or deploy your machine-learning models! If you already understand machine learning, then you can go check . AWS comprehend; open-source NLP models as well on Huggingface Our second problem is generating high-quality recommendations based on a user's interest The following diagram is a simplified high-level representation of how the system works. While the above looks easy, there is a lot more going on at each step, and those things have to be carefully thought through and then programmed to ensure that the system is performing optimally. Let me explain step by step: Step 1 - Converting post content into vector embeddings To generate these recommendations, first, we have to convert the contents of a post into something called - . With the recent uptick in the branding of LLMs, OpenAI( the makers of ChatGPT) and , Vector embeddings are becoming an everyday term. I will not go into the details of what they are and how they work, but I highly recommend reading more about them. But generating viable candidates for a feed also has to account for things like content privacy and moderation (removing profane words, abuses, sexual content, harassment, filtering blocked users, etc). Vector embeddings Vector databases For generating the vector embeddings, you can use any prominent embedding model such as the , or any text embedding model, depending on your use case. We went with because of its friendly pricing, performance and operational ease. OpenAI embedding model Amazon titan open-source Amazon Titan Step 2 - Query the user's interest Now, this is where things get interesting. You would want to design the queries based on your specific business needs. For example, we give more weightage to the recency of engagement while querying interests than the number of engagements with a specific keyword or user. We also run multiple parallel queries to find different types of interest of the user - keyword or other user. Since we generate multiple feeds for a single user, we also run some queries promoting a specific topic according to the trend (for example, you will see many Christmas-related posts near Christmas or earthquake-related posts if some earthquake has happened). Needless to say, this topic will only come up in the query results if the user has expressed some interest in them in their journey. So, choose the logic that suits your business use case and the behavior that you want to drive and run multiple queries to get a big enough list of all the user's interests. Step 3 - Do an ANN search based on the interests found Vector databases are predominantly used for performing a particular type of search called (ANN). Again, the way you categorize various interests and whether you are doing one big ANN search or parallel difference searches should entirely be based on your use case and business requirements. I recommend doing more than cohort-based searches and then ordering the results (we will discuss this later in this blog) for the best end-user experience. What ANN search does, in this case, is find other posts on the platform that are similar (closer) to the interests of the user. Approximate nearest neighbor search Step 4 - Store the results in a cache database with ordering. Cache database because one of the problems that we need to solve is speed. We used for storing the unique IDs of the posts for a specific user. We used redis sorted sets because the order of posts in a user's feed is critical. Also, another problem that you have to solve is that the" to ensure that the posts from the same creators aren't grouped on the same page". To avoid repetition of content from the same creator, we have written a simple algorithm which ensures that if a specific creator's post is inserted at any position in a particular user's feed (sorted set), we don't insert another post from the same creator for successive ten positions (we have a page size of 10 while serving the feed to the end user, so we kept it static to avoid complexity). sorted sets redis system should have logic For deciding the order of a specific recommendation of the user, we factored in the following things - : It's determined by an arithmetic formula that takes various data points from the social graph. All of this is engagement data like the timestamp of the last likes created, number of likes created, last comment, etc. User engagement behavior is the indicator of their interest in something. The strength of the relationship with a specific interest (or another user) for this user To determine this, we have created an algorithm that takes various factors such as engagement, engagement-to-impression ratios, number of unique users who engaged, etc., to generate an engagement score of that post at a platform level. The popularity of the post on the platform: In some feeds, we prioritize popularity; in others, we prioritize the social graph. But mostly, all of them are a healthy mix of the two. How the System Works As you can see from the diagram above, the system has been intentionally kept very simple. Following is how the system works - When user A creates a post, the post service, after saving that post, triggers a pub/sub event to a queue, which is received by a background service meant for candidate generation. We use for the pub/sub functionality. Google Pub/Sub This background service receives this asynchronously and performs functionalities discussed earlier - Privacy checks, moderation checks, and keyword generation and then generates the vector embeddings and stores them in the vector database. We are using (discussed later). AstraDB as our vector database Whenever a user engages (like/comment/share, etc.) after updating our main NoSQL database, the post-service triggers a pub/sub event to the recommendation engine service. This recommendation engine service updates the graph database and then updates the recommended feed of the user in near real-time by performing the ANN search and updating the Redis database. So, the more users interact, the better the feed keeps getting. . Those checks are performed while we query the Graph database. This service also updates the engagement score asynchronously. Engagement scores are re-calculated on users viewing the post as well. There are checks to ensure that the recommendations are not biased towards a specific list of keywords Since all of the above steps are performed asynchronously behind the scenes, these computations have no impact on the end-user experience. The feed is finally served to the end user through a feed service. Since this service just performs a lookup on redis and our main NoSQL database ( ), its P99 latency is less than 110 milliseconds. Both these databases return query results in single-digit millisecond latency irrespective of scale. DyanmoDB Tools and technologies used Some services have been written in , while others have been written in (with typescript). Go programming language NodeJS We are using as our vector database. We arrived at this decision after evaluating multiple other databases, such as and . Apart from its excellent query and indexing capabilities on vector and other data types, it offers a pocket-friendly serverless pricing plan. It runs on top of a Cassandra engine, which we use as a database in several other features on our platform, and it gives a CQL query interface, which is very developer-friendly. I highly recommend trying it for your vector use cases. AstraDB by Datastax pinecone, milvus weaviate We use for our asynchronous communication because, at our current scale (few lakh total users, few thousand daily active users), it's highly cost-effective. I have run it at a scale of a few lakh users with thousands of events per second. It works well, and it's effortless to use and extend. Google pub/sub - Speed, simplicity and powerful data structure. I don't think I need to discuss why redis in 2024. Redis - Again, it is highly scalable and easy to use, and we run it in the serverless mode where, despite hundreds of thousands of queries per minute, our total bill is quite low. It also offers very powerful indexing capabilities and single-digit millisecond latency in reads and writes. DynamoDB Problems to be solved in the future As you can imagine, this same setup can be tweaked to build a basic recommendation engine for any use case. But, since ours is a social network, we will require some tweaks down the line to make this system more efficient. Machine learning/ Deep learning algorithms will be needed at the social graph level to predict the keywords and users most relevant for the user. Currently, the data set is too small to predict anything accurately as it is a very new product. However, as the data grows, we will need to replace the current simple queries and formulas with the output of machine learning algorithms. Relationships between various keywords and users must be fine-tuned and made more granular. They are at a very high level right now. But they will need to be deeper. We will need to explore the second and third-degree relationships in our graph to refine the recommendations first. We are not doing any fine-tuning in our embedding models right now. We will need to do that in the near future. End note I hope you found this blog helpful. If you have any questions, doubts or suggestions, please feel free to contact me on , or . Do share this article with your friends and colleagues. Twitter Linkedin Instagram Also published . here

How to Build a Basic Recommendation Engine without Machine Learning 

কিভাবে মেশিন লার্নিং ছাড়া একটি বেসিক রেকমেন্ডেশন ইঞ্জিন তৈরি করবেন

So erstellen Sie eine einfache Empfehlungs-Engine ohne maschinelles Lernen

Cómo construir un motor de recomendación básico sin aprendizaje automático

Comment créer un moteur de recommandation de base sans apprentissage automatique

मशीन लर्निंग के बिना एक बुनियादी अनुशंसा इंजन कैसे बनाएं

機械学習を使用せずに基本的なレコメンデーション エンジンを構築する方法

머신러닝 없이 기본 추천 엔진을 구축하는 방법

Como construir um mecanismo de recomendação básico sem aprendizado de máquina

Как создать базовую систему рекомендаций без машинного обучения

Makine Öğrenimi Olmadan Temel Öneri Motoru Nasıl Oluşturulur

Cách xây dựng công cụ đề xuất cơ bản mà không cần học máy

如何在不使用机器学习的情况下构建基本的推荐引擎

BREVIO

brevio.org

11-50 employees

Since 2019

COMPANY RANKING

BREVIO

EVERGREEN INDEX #2189

Representing this company? Click here to claim and customize this page!

Brevio WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

BREVIO

brevio.org

11-50 employees

Since 2019

COMPANY RANKING

BREVIO

EVERGREEN INDEX #2189

RELATED COMPANIESRANK

STORIES ABOUT BREVIO

Representing this company? Click here to claim and customize this page!

Brevio WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!