Budgetree builds world-class financial solutions and products to manage the business expenses of Corporates, SMEs, \u0026 Startups through automated and innovative workflows

Budgetree

With less than a semester left of college, and a need to build something I realized I needed a budget app. I knew of course about Mint, and later learned of Clarity Money far into development, but I wanted something smaller and more focused without all the extra heft that Mint provides, or the advertising and recommendations that Clarity Money includes. Budgeteer was born (My sister came up with the name at dinner when I was home on vacation, I am not that creative). This is the story of how I built it, lessons learned, and experience gained. It’s not finished, and still requires a fair amount of work left, but not so much that I couldn’t finish in a couple of weeks if I really wanted to, or couple of months if I want to relax with it. Please feel free to and leave any comments you have here (Please just avoid adding or removing too many accounts in the settings page as it does update the DB and I don’t want to check one day and see 50 linked accounts that I have to remove). Design advice is particularly welcome. check it out Why another budgeting app you ask. While banks and credit cards provide a lot of the same statistical information, few companies aggregate all of it together. Most people I figure have at least 2 credit cards, a savings account, a checking account, and a retirement account. Thats a lot of accounts to keep track of and the data from each individual account is useless without being combined with the others. This is the why for Budgeteer. The Stack MERN. MongoDB, ExpressJS, ReactJS and NodeJS. Its awesome and powerful. In addition to these, I use Mongoose, PassportJS, Webpack , date-fns (awesome date library — very modular = tiny size), Plaid, ReactRouter, Recharts, and a few others. Why am I sharing all the tech I used? Cause I doubt someone else wants to go through it and rebuild exactly the same product. And frankly if they do and can do it better, then go for it. I’m a free market kind of guy. I may try and bring this to market and sell it as a subscription (Plaid is the only big cost if I were to sell it as a service) but with a developer account, I am happy to be the only user and can do so for free. Scope Defining the scope of this project was an evolutionary process. First I focused on a desktop site. Then I wanted it to also be mobile friendly and look great on a phone with a screen as small as 4". Then I decided to build it as a Progressive Web App both as a means to learn what PWAs entail and because I think PWAs will be far more popular in the near future. With one codebase I can hit every device and get native app functionality. The core of this product revolved around getting data from banks. Was I to resign to picking a few banks (aka the ones I use) and go through their API documentation (if it even existed) which was bound to be terrible or was there a better way. Enter , an awesome API that lets developers let users link bank accounts and access transactional information among many other data points. But for me, I got TRANSACTION DATA. The only data I needed, beautifully formatted and organized. If you ever need to work with bank accounts for whatever reason, check them out. Plaid Core Functionality Above all I wanted simplicity, an elegant design, and a tool that would in the end be helpful and not just a shallow clone. No advertising, no weird partnerships, no financial products recommendation, no nothing. Just an honest, quality product that I would want to use. I hope you will too. I’ve listed below the core functionality. View and search transaction data across all accounts Provide meaningful feedback that helps users understand their spending patterns — visualized statistics with charts Easily view your Net Worth Track monthly recurring expenses like Netflix, Spotify, HBO NOW, etc Easily add or remove accounts without any fuss or hoop jumping Transactions Being able to view all your transactions across all your accounts gives users the power to see in one place where they spend their money, where they can cut back, and make tackling a tough subject easier. Being able to search transactions and filter by category, date, and account type makes tracking various expenses easy. Knowing what you’ve spent in the past week in the chart keeps you honest about how you’re doing in the short term while the statistics sections help focus your attention on the medium and long term. (NOTE: The calendar button doesn’t do anything yet. Still trying to find a good calendar drop in that I can use — suggestions welcome). Statistics This required a charting library. I first started with ChartJS and after sticking with it for a while, I eventually switched to Recharts. ChartJS styling and option configuration is weird. Mentally picturing how something should look after an edit frequently failed to match the reality. Recharts, though more complex at first, was far easier to configure and style. The five charts I create are explained below Categorical breakdown of spending as a doughnut chart Annual spending by month for the past year with a line for the average amount per month Monthly Budget chart showing amount spent so far in the current month and the amount remaining Week vs Weekend spending to better understand if your spending changes drastically during one or the other Heat map showing transactions in various locations with hotter regions indicative of more spent (IN PROGRESS — Not in the demo). Each of these charts answers valuable financial questions. Where is my money going? How am I spending it? Where can I cut back? Am I on track with my Budget? Is my spending consistent month to month? Where physically am I spending the most? Each of these charts and graphs offers a quick snapshot of a different perspective on a user’s financial health without overwhelming them with extraneous information. It crystallizes the rough idea people keep mentally remember by showing them the hard numbers of their finances. Above all, each chart is insightful and simple. Net Worth and Recurring Payments Net Worth quickly allows people to view their total account balances and know how much cash they have on hand in their various accounts whether it be an emergency fund they set up or their primary checking account. Showing each account gives users the transparency they need to ensure they can allocate their money effectively instead of roughly tracking amounts in their head. Recurring transactions give users a good sense of the products and services they pay for monthly. Maybe that Pandora subscription you forgot about isn’t necessary anymore now that you started using Spotify or since you stopped using it entirely. Settings Page The settings page is all about bringing even more transparency to the user. They can easily remove any accounts they would rather not have linked and see if they are missing anything. Once they click remove the access tokens stored in the database are deleted and more importantly invalidated so even if someone got them they wouldn’t work. Design Designing Budgeteer was the most interesting part of this project. I have gone through so many iterations of finding the right way to display transactions, show the charts, style the nav bar etc. I’ve checked-out different commits across the history so you can see the progress below. Transactions Page 3 Different Versions of the Transactions Page The difference is stark. Transactions are far more elegant and take up much less space while communicating nearly double the information. The graph adds a visual perspective which breaks up the UI and provides the user with another quick summary of their spending trends. All the sorting options were compacted into smaller expandable parts reducing clutter and adding much needed visual breaks. It makes the design mobile friendly and easy to navigate. Statistics Page At first I started with a tab based design. I thought that looked cool on desktop but knew it wouldn’t scale on mobile. Then I decided to have a slider like option with next and previous buttons which the user could press to change graphs. While it doesn’t look awesome on desktop yet, it does on mobile. Maybe next I’ll switch to having a little menu bar on the bottom with an icon for each graph. That would take up the extra visual space on a computer while still also fitting on mobile. Desktop could contain words if they fit, switch to icons with a breakpoint when they don’t, and stay that way for mobile. The Nav Bar Oh boy. The Nav Bar. Easy on desktop but a lot of iterations on mobile. The three main ones are below. The biggest difference in the final version is in the nice little bump at the end which makes it feel more dynamic and interactive. Whether this remains the final version or I go with a more native app feel of leaving the left side attached is yet to be determined. — UPDATE — I went with a native mobile navbar. on your phone or make the window smaller on desktop to see how it looks. View the site Broader Comments on Design In some of the images you might notice the font is a little thicker. I switched at some point from a 400 weight (normal) to a 300 weight (just a little thinner) and the difference was astounding. It created so much space I was amazed. Everything felt modern, cleaner and just looked better. Another idea I had was to make the color of the price in a transaction red or green based on if it was a payment or a return. Further, using larger font sizes for the transaction name and a smaller one for the account name below it, focuses the users attention to the more important information. These and many other tricks I learned by spending time on other websites and apps just looking. Taking a moment to notice just how an animation of a menu bar in an app looks, seeing where your eye is drawn to on a page and why, or by looking at how color is used to convey information gave me a lot of inspiration for what good design and UX entails. Lastly, absolutely check out for awesome designs and trends. This was my first go to when I felt stuck or needed quick inspiration. The designers behind their submissions are seriously talented. https://medium.muz.li/ Developing as a Progressive Web App One of the most interesting parts of this project has been working to build it as a Progressive Web App. This allows it to be installed and function like a native app. The primary benefits are push notifications, setting up offline functionality and cacheing resources locally to reduce network requests. This means that with one codebase I get a site for desktop and mobile as well as native app like functionality. PWAs are kind of weird though. I have had some problems cacheing resources from a CDN with a different domain, a CORS issue I know, properly configuring a service worker, setting up push notifications, and figuring out how to get the install pop up to appear automatically. Also testing it is always difficult since it caches everything but updates to the code may not always invalidate the now stale cache. While React Native would have let me actually build a native app, this lets users choose exactly how they want to use this product whether it be infrequently on their computers, a few times a week on their phone’s browser, or as often they use other apps. The PWA version isn’t pushed as I’m still working out the kinks of cacheing and push notifications but I hope to finish this part next and then tackle final design issues followed by any functionality I’ve put off *ahem, sorting transactions *cough, by date *cough, ahem, clears throat. Conclusion I’ve learned a tremendous amount about design and building a product. From tooling, to API integrations this has been one of the most rewarding projects I’ve taken up. When bored at school or when I wanted to procrastinate HW I would work on this and working on it was so much fun. As a final thought I’ll ask you all to please leave a comment if you think a light theme would be better than the current dark theme. This has been an ongoing debate with my sister.

Building Budgeteer - Lessons and Experiences

The tech workforce in the US is not growing at an optimal pace. The number of schools offering computer science is not enough which is impacting the access of computer science to young women and students from marginalized communities. The truth is that there are over in the US but only graduated into the tech workforce last year. 410,020 open computing jobs 71,226 computer science students This is because only in the US offer computer science which deeply impacts its access to young women and students from marginalized communities. 47% of public high schools To address this pressing issue, we have launched #MyStartInTech, a campaign dedicated to drawing attention to this cause and in turn, help widen access to computer science in schools. Learn more about the campaign by visiting www.startin.tech As part of this campaign, we present the #MyStartInTech interview series where some of the esteemed tech entrepreneurs and professionals in the industry share details about their journey in tech. With this, we hope to draw attention to the infinite opportunities that will lay open if young women and students are given the opportunity to study computer science. In this interview, Jordan Silverman, Co-Founder of , talks about how he got his start in the tech world. Budge Technology Jordan is currently building a software product for the restaurant/hospitality space. He grew up in a family of food distributors and now he hopes to solve focal issues for restaurants with tech. Let’s find out what he has to say! When was your first interaction with computer science? The first real interaction I had with computer science was not until I joined the Budge Technology team. I had no prior experience before then. Just using the base product and working close to our dev team motivated me to learn more about programming. I started teaching myself a lot about web design and eventually started doing challenges on for some practice. freecodecamp.org I regularly do those challenges as I continue to learn more. What nudged you into taking up a career in technology? I'm almost 2 full years out of college. I was a student-athlete on the baseball team where I studied Communications and Marketing. An above-average student with a strong work ethic but no real direction. My initial job search wasn't the best as I was more focused on baseball in the Spring of 2019 rather than finding a job out of college. Once I graduated, I landed a sales job that just did not pan out to what I wanted it to be. But, I was networking and trying to keep my eye out for another opportunity. That's when I was approached by Max Weiner, the founder of Budge Technology Inc. A few weeks later, I left my sales position and joined Budge. Since then, I've found a passion for technology and startups. My family has always been involved in the restaurant industry and I've worked in a variety of them, so the product was fitting to my prior experience. Now, I hope to solve focal issues in restaurants with tech. What are your earliest memories of using a computer in school? If we go all the way back to grade school, I remember playing Oregon Trail and the Mavis Beacon typing games to improve my typing speed and accuracy. What is the importance of technology to you now and how does it impact your day-to-day life? I feel as though my life is technologically driven. I'm always on either a computer or using my iPhone. I resort to technology to be more productive and simplify my work. Whenever a problem arises, I will initially search for software to solve said problem. How important is it to increase access to computer science and technology to underrepresented communities and young women today? I think having any sort of computer science or technology experience goes a long way no matter where or who you are. Programming has become completely democratized. Anyone can learn to code at the pace they'd like. The barrier of entry is fairly small. You can get a Rasberry Pi 400 computer that costs $70. The information to learn is scattered at your disposal throughout the Internet. This unleashes the great potential for the future of tech and the world we live in. More access = more opportunity for innovation. Why do you think access to computer literacy in school is important? Having access to computer literacy at an early age can open so many new doors for young individuals. Schools should tie in more computer science and programming into their curriculum. I'm a huge advocate of contributing to the marketplace of ideas. More computer literacy means more contributions to the marketplace of ideas. And more people that have access to learning more about programming and computer science will be more inclined to execute their ideas. What do you think about .Tech Domains taking up this cause? I think it's completely relevant to the times we're in currently. We are seeing more innovation than ever. It's great that you're covering how technology is shaping the world we live in and how important it is to provide access to underrepresented groups of people. Learn more about #MyStartInTech campaign and lend your support at . https://startin.tech/

#MyStartInTech: Interview With Jordan Silverman, Co-Founder of Budge.Tech

Let’s say you have a job interview in a few days. How should you prepare for it so that you can make an informed decision about joining the company, as well as make sure that your interests are taken care of? Prepare your questions An interview is a two-way street: it’s not only the company investigating you as a potential employee; in addition, you are investigating the company as a potential employer. Hopefully, you have some other options available—such as staying at your current place or trying out other companies interested in you. In the best case, you’ll have the flexibility to decide whether you want to follow through with the company or not. To learn about the potential employer, you need to ask questions. The question will depend on what’s important for you at the current stage of your career. Some things I would like to know before deciding to join a new company are as follows: What is their development process? If they use some standardized methodology such as Scrum, I would like to hear how they implement it. Especially whether they have Scrum Master and Product Owner roles covered—and whether those people are easily reachable. I would also ask if they do retrospective meetings. How do they do quality automatization: things such as unit tests, e2e, CI & code-style enforcing? For things that they don't have, I would ask if they would like to see it in the project. I would ask about their product and how they do product design/discovery. Life–work balance questions inspired by : How do they deal with emergencies? How often is there a need to receive a call or respond to messages outside working hours? Healthy Software Developer’s video how long do employees usually work? Salary negotiation The point of going to the interview is to get to the salary negotiation. The result of the negotiation will have a great impact on your life for the whole time you spend at the company—it’s a smart move to prepare well. The earlier you start, the better—as you will have more time to investigate, ask questions, and plan your strategy. The best resource for preparing for an interview in IT is . You can buy the complete e-book, read the chapters online, or get the key advice from the bite-size articles on the website. Fearless Salary Negotiation Research compensation Before you start negotiation, figure out the current situation in the market. A little bit of time spent on systematic investigation can give you a much more precise picture than anecdotes heard here or there. Check out this . chapter on market value estimation Decide your minimum Great advice is to decide on the minimum pay that you’ll accept you start speaking with the company. It could happen that during the various rounds of interviews, they will manage to the company very well or that the effort you have already invested into the company will make you more flexible than you should be. In those cases, having the minimum salary decided ahead of time can be very helpful. . before sell Relevant chapter from the book here Think openly about your options. As you negotiate with the company, it’s good to be flexible about the options you consider. If you are asking for more money than they have budgeted for, you can propose reduced hours with the same hourly rate that you are happy with. Similarly, commuting to the office can easily take 1–2 hours of your day, so a lower salary could be worth it if it comes with guaranteed remote work. Practice interview Try role-playing an interview with someone. If you can handle it, you can ask them to be tough on you—stricter than what you would expect from a real interview. Ideally, you’ll have a tough, low-stakes interview that would make you relaxed during the real thing. Investigate the company Learn about the company—all you can do before the meeting, that is. Check their website, their LinkedIn page, and their key employees. Play around with their product if you can. In short, spend an hour or two trying to learn as much as you can about the company. It will help you ask important questions and make a good impression. Summary A bit of preparation can go a long way. As developers, we often think about the technical part of the interview, but learning about the company and preparing for salary negotiation can help us even more. We can filter out companies we shouldn’t join, and we can make sure that we get as good a deal as possible. Also published . here

Nailing Your Job Interview: Preparation, Research, and Looking Out For Your Best Interests

Somebody's Trying to Tell Me Something

341 Stories To Learn About Tech Careers

How This Man Fueled Election Fraud Conspiracy Theories in Wisconsin

In your deployment phase, a crucial question to address is the cost estimation of the resources required for deploying your amazing application within the allocated or recommended budget. Failing to address this question could result in exceeding the allocated or budgeted amount due to resource creation costs. One tool that was developed to help with putting a check on the cost of cloud resources is Infracost. Infracost is a valuable tool designed to monitor and control the cost of cloud resources. Unlike tools like the AWS calculator, Infracost stands out with its automated approach. By simply parsing the format of your Terraform-created resources as an argument to the Infracost CLI command, it automatically provides you with the monthly cost estimation for the resources to be deployed, whereas the AWS calculator requires manual input of each resource to be created. JSON This article aims to guide you, the reader, on obtaining cost estimates for your cloud resources using Infracost. It covers creating an Open Policy Agent (OPA) policy to enforce monthly budget constraints and integrating the infracost command into your Jenkins pipeline. To set up Infracost, refer to the for detailed instructions on getting started. official documentation The terraform files for the resources to be created are stored in a folder which is in the same location as the This is the file location format used in this article. terrafom Jenkinsfile. As we execute the Infracost command within a Jenkins pipeline, we utilize the official Infracost docker image as the Jenkins agent, alongside configuring essential environment variables. Here is an example of how the configuration should be set up: agent {
  docker {
    image 'infracost/infracost:ci-latest'
    args "--user=root --entrypoint=''"
          }
       }
environment {
   INFRACOST_API_KEY = credentials("INFRACOST_API_KEY")
   INFRACOST_VCS_PROVIDER = 'github'
   INFRACOST_VCS_REPOSITORY_URL = 'https://github.com/Okeybukks/devops-automation'
    } To get your you can run this command in your cli. INFRACOST_API_KEY infracost configure get api_key Alternatively, you can access the Infracost dashboard, login with your credentials, navigate to " ," and click on the " " button next to the API Key. This key needs to be securely stored as secret text in Jenkins credentials, with an ID of . Considering that our Git repository is hosted on GitHub, set the value of to "GitHub." Additionally, specify the as the URL of the repository containing your Jenkinsfile and Terraform folder. Org settings Copy INFRACOST_API_KEY INFRACOST_VCS_PROVIDER INFRACOST_VCS_REPOSITORY_URL After configuring Infracost for Jenkins, the subsequent step involves executing the command. This command analyzes the Terraform plan and provides a cost estimate for the cloud resources intended for deployment. The resulting output is saved in a JSON file format, which will be utilized by our OPA policy. infracost breakdown stage("Check Financial Expense of Infrastructures Job with Infracost"){
   agent {
     docker {
       image 'infracost/infracost:ci-latest'
       args "--user=root --entrypoint=''"
             }
         }
   environment {
       INFRACOST_API_KEY = credentials("INFRACOST_API_KEY")
       INFRACOST_VCS_PROVIDER = 'github'
       INFRACOST_VCS_REPOSITORY_URL = 'https://github.com/Okeybukks/devops-automation'
            }
   steps{
       dir("./terraform") {
          sh 'echo "This is the financial check job"'     
          sh 'infracost breakdown --path . --format json --out-file infracost.json'
          archiveArtifacts artifacts: 'infracost.json'
           } 
          }
   } The terraform files are located in the terraform folder, so to change into the terraform folder, we will be using the block. dir The command takes a couple of arguments with values passed to it. infracost breakdown takes the path where the terraform files are located. Since we are already in the terraform folder, we use to reference the path. --path . takes the format of the output of the command. --format infracost breakdown takes the name of the file you wish to save the output. --out-file infracost breakdown To enable the utilization of the output file generated by the command in subsequent stages, the file is parsed to the "archiveArtifacts" command, converting it into an artifact. For a more comprehensive understanding of artifact creation and copying, I invite you to explore the I have authored on Jenkins artifacts. infracost breakdown article To establish a cost constraint on the creation of resources, an OPA policy is employed. In this article, we are imposing a limit of $100 on the allocated resources. OPA serves as a comprehensive toolset and framework for policy implementation throughout the cloud-native ecosystem. To delve deeper into OPA, you can find more information by clicking here OPA policies are written in the rego language, and as such, the policy file is suffixed with .rego. The for rego serves as a comprehensive resource to comprehend policy formulation. In this article, a straightforward policy will be presented, which verifies whether the cumulative monthly estimate of your cloud resources surpasses the predefined maximum monthly estimate that must not be exceeded. documentation package infracost

deny[out] {

    # define a variable
	maxMonthlyCost = 100.0

	msg := sprintf(
		"Total monthly cost must be less than $%.2f (Current monthly cost is $%.2f)",
		[maxMonthlyCost, to_number(input.totalMonthlyCost)],
	)

  	out := {
    	"msg": msg,
    	"failed": to_number(input.totalMonthlyCost) >= maxMonthlyCost
  	}
} The line is the infracost package or module utilized by OPA to understand the content of the output file from the command. OPA policies are written in a block. Any code written outside this block will return a error. package infracost infracost breakdown deny[out] rego_parse_error is the defined variable which is the maximum amount budgeted for our resources. maxMonthlyCost The message to be output in the Jenkins pipeline and GitHub commit message if the set condition fails is defined in the block. msg The condition logic is set using the block. The total estimate our resources will cost us is saved in variable. This variable is a default variable in the rego language. The is the total monthly cost estimated by infracost. out input.totalMonthlyCost input.totalMonthlyCost From our policy, if is true it stops the pipeline and prevents moving to the next stage of the pipeline. The message in the block explains to us that we cant go forward. The content of this policy is stored with file name. This filename can be changed though, but it must have the format appended to it. to_number(input.totalMonthlyCost) >= maxMonthlyCost msg infracost-policy.rego .rego With the policy written, it is then parsed to the command. This is what the stage looks like. infracost comment github stage("Post Infracost comment"){
  agent {
     docker {
        image 'infracost/infracost:ci-latest'
        args "--user=root --entrypoint=''"
             }
        }
  environment {
     INFRACOST_API_KEY = credentials("INFRACOST_API_KEY")
     INFRACOST_VCS_PROVIDER = 'github'
     INFRACOST_VCS_REPOSITORY_URL = 'https://github.com/Okeybukks/devops-automation'
     INFRACOST_VCS_BASE_BRANCH = 'main'
     GITHUB_TOKEN = credentials("GITHUB_TOKEN")
     GITHUB_REPO = "Okeybukks/devops-automation"
         }
  steps{
     dir('./terraform'){
         sh 'echo "This is the financial check job"'
         copyArtifacts filter: 'infracost.json', fingerprintArtifacts: true, projectName: 'test', selector: specific ('${BUILD_NUMBER}')    

         sh 'infracost comment github --path infracost.json --policy-path infracost-policy.rego \
                    --github-token $GITHUB_TOKEN --repo $GITHUB_REPO --commit $GIT_COMMIT'
         }
     }
 } Upon observation, you will notice the addition of new environment variables required for this stage. Since the infracost command will be responsible for posting the commit message to our repository, it will necessitate specific GitHub access permissions to fulfil this task effectively. Generate a classic token with repo permission ticked. If you can read how to generate a classic GitHub token using this . Once generated, save it in your Jenkins credential using secret text and save it with the ID . article GITHUB_TOKEN The GitHub repo name of this project is also needed. Store this value in the ID. GITHUB_REPO The output of the command is saved in the infracost.json file. To access the file at this stage, the is utilized. infracost breakdown copyArtifacts Now we have all the values to the arguments needed by the command to utilize our policy in creating a check for resource creation. infracost comment github value is the file created with the command, i.e infracost.json --path infracost breakdown value is the policy file we created i.e . --policy infracost-policy.rego value is our classic GitHub token stored in . --github-token GITHUB_TOKEN is the project repo name stored in --repo GITHUB_REPO is required to post on a pull request's commit. Its value is stored in variable, one of Jenkins default variables. --commit $GIT_COMMIT When you run your pipeline, you should have something similar to this if your resources estimate exceeds the set mark. + infracost comment github --path infracost.json --policy-path infracost-policy.rego --github-token **** --repo Okeybukks/devops-automation --commit a0e8fa201d81ba9d2a508e4adc55e2b8f1ffca6d
time="2023-06-12T04:29:15Z" level=info msg="Finding matching comments for tag infracost-comment"
time="2023-06-12T04:29:21Z" level=info msg="Found 0 matching comments"
time="2023-06-12T04:29:21Z" level=info msg="Creating new comment"
time="2023-06-12T04:29:21Z" level=info msg="Created new comment https://github.com/Okeybukks/devops-automation/commit/a0e8fa201d81ba9d2a508e4adc55e2b8f1ffca6d#commitcomment-117572438"
Comment posted to GitHub

Error: Policy check failed:

 - Total monthly cost must be less than $100.00 (Current monthly cost is $168.72) This concludes the article. Implementing this cost monitoring approach will not only lead to cost savings in resource usage but also prevent unintended deployments of unplanned cloud resources. Thank you for reading. Also published here. The lead image for this article was generated by HackerNoon's AI Image Generator via the prompt "A mount counting coins."

Efficient Cloud Resource Cost Management with Infracost and Open Policy Agent in Jenkins

This is a practical guide for using Apache Cassandra as a real-time feature store. We explore real-time AI and the unique performance and cost attributes of Cassandra that make it an excellent database for a feature store and then dive into the basics of feature stores and their role in real-time applications. Cassandra is used as a feature store by large companies, including and Netflix; under real-world conditions, it can serve features for real-time inference with a tp99 < 23ms. Uber The guide is divided into several key sections. We start by introducing Cassandra and its features that make it an ideal choice for a feature store. Then, we explain the basics of feature stores, including what they are and how they can be used in real-time applications. After that, we explore the implementation details of creating a feature store using Cassandra. This includes data modeling, ingestion and retrieval of features, and handling data updates. Finally, we offer up best practices and tips for using Cassandra as a feature store to ensure optimal performance and scalability - from latency requirements to estimated performance metric requirements to reference architectures and ecosystem compatibility. This guide doesn’t discuss the or the in a . The best practices we’ll cover are based on technical conversations with practitioners at large technology firms such as Google, Facebook, , , and on how they deliver real-time AI experiences to their customers on their cloud-native infrastructures. Although we’ll specifically focus on how to implement real-time feature storage with Cassandra, the architecture guidelines really apply to any database technology, including Redis, MongoDB, and Postgres. data science aspects of real-time machine learning lifecycle management aspects of features feature store Uber AirBnB Netflix What is real-time AI? Real-time AI . Traditionally, training models and inferences (predictions) based on models have been done in batch – typically overnight or periodically through the day. Today, modern machine learning systems perform inferences of the most recent data in order to provide the most accurate prediction possible. A small set of companies like TikTok and Google has pushed the real-time paradigm further by including on-the-fly training of models as new data comes in. makes inferences or training models based on recent events Because of these changes in inference and changes that will likely happen to model training, the persistence of feature data – data that is used to train and perform inferences for a ML model – needs to also adapt. When you’re done reading this guide, you’ll have a clearer picture of how Cassandra and DataStax Astra DB, a managed service built on Cassandra, meets real-time AI needs, and how they can be used in conjunction with other database technologies for model inference and training. What’s a feature store? A feature store is a data system specific to machine learning (ML) that: Runs data pipelines that transform raw data into feature values Stores and manages the feature data itself, and Serves feature data consistently for training and inference purposes Real-time AI places specific demands on a feature store that Cassandra is uniquely qualified to fulfill, specifically when it comes to the and of features for model serving and model Training. storage serving Best Practices **Implement low-latency queries for feature serving For real-time inference, features need to be returned to applications with low latency at scale. Typical models involve ~200 features spread across ~10 entities. Real-time inferences require for collecting features, lightweight data transformations, and performing an inference. According to the (also confirmed by our conversations with practitioners), feature stores need to return the features to an application performing inference in under 50ms. time to be budgeted following survey Typically, models require “inner joins” across multiple logical entities – combining rows values from multiple tables that share a common value; this presents a significant challenge to low-latency feature serving. Take the case of Uber Eats, which predicts the time to deliver a meal. Data needs to be joined from order information, which is joined with restaurant information, which is further joined by traffic information in the region of the restaurant. In this case, two inner joins are necessary (see the illustration below). To achieve an inner join in Cassandra, one can either denormalize the data upon insertion or make two sequential queries to Cassandra + perform the join on the client side. Although it’s possible to perform all inner joins upon inserting data into the database through denormalization, having a 1:1 ratio between model and table is impractical because it means maintaining an inordinate number of denormalized tables. Best practices suggest that the feature store needs to allow for 1-2 sequential queries for inner joins, combined with denormalization. Here’s a summary of the performance metrics that can be used to estimate requirements for real-time ML pipelines: Testing Conditions: features = 200 number of tables (entities) = 3 number of inner joins = 2 Query TPS : 5000 queries / second Write TPS : 500 records / second Cluster Size : 3 nodes on AstraDB* Latency Performance summary (uncertainties here are standard deviations): tp95 = 13.2(+/-0.6) ms tp99 = 23.0(+/-3.5) ms tp99.9 = 63(+/- 5) ms Effect of compaction: tp95 = negligible tp99, tp999 = negligible, captured by the sigmas quoted above Effect of Change Data Capture (CDC): tp50, tp95 ~ 3-5 ms tp99 ~ 3 ms tp999 ~ negligible *The following tests were done on DataStax’s , which is a serverless environment for Cassandra. Users should expect similar latency performance when deployed on three notes using the . Astra DB’s free tier following recommended settings The most significant impact on latency is the number of inner joins. If only one table is queried instead of three, the tp99 drops by 58%; for two tables, it is 29% less. The tp95 drops by 56% and 21%, respectively. Because Cassandra is horizontally scalable, querying for more features does not significantly increase the average latency, either. Lastly, if the latency requirements cannot be met out of the box, Cassandra has two additional features: the ability to support denormalized data (and thus reduce inner joins) due to high write-throughput capabilities and the ability to selectively replicate data to in-memory caches (e.g. Redis) through Change Data Capture. You can find more tips to reduce latency here. Implement fault tolerant and low latency writes for feature transformations A key component of real-time AI is the ability to use the most recent data for doing a model inference, so it is important that new data is available for inference as soon as possible. At the same time, for enterprise use cases, it is important that the writes are durable because data loss can cause significant production challenges. *Object stores (e.g. S3 or HIVE) can be replaced with other types of batch-oriented systems, such as data warehouses. There is a trade-off between low-latency durable writes and low-latency feature serving. For example, it is possible to only store the data in a non-durable location (e.g., Redis), but production failures can make it difficult to recover the most up-to-date features because it would require a large recomputation from raw events. A common architecture suggests writing features to an offline store (e.g., Hive / S3) and replication of the features to an online store (e.g., in-memory cache). Even though this provides durability and low latency for feature serving, it comes at the cost of introducing latency for feature writes, which invariably causes poorer prediction performance. Cassandra provides a good trade-off between low-latency feature serving and low-latency “durable” feature writes. Data written to Cassandra has typically been replicated a minimum of three times, and it supports multi-region replication. The latency from writing to availability to read is typically sub-millisecond. As a result, by persisting features directly to the online store (Cassandra) and bypassing the offline store, the application has faster access to recent data to make more accurate predictions. At the same time, CDC, from the online store to the offline store, allows for batch training or data exploration with existing tools. Implement low latency and writes for prediction caching and performance monitoring In addition to storing feature transformation, there is also the need to store predictions and other tracking data for performance monitoring. There are several use cases for storing predictions: – In this scenario, a database used to cache predictions made by either a . The streaming architecture is particularly useful when the time it takes to inference is beyond what is acceptable in a request-response system. Prediction store batch system or a streaming system It is often necessary to monitor the prediction output of a real-time inference and compare to the final results. This means having a database to log the results of the prediction and the final result. Prediction performance monitoring Cassandra is a suitable store for both use cases because of its high-write throughput capabilities. Plan for elastic read and write workloads The level of query and write transactions per second usually depends on the number of users simultaneously using the system. As a result, workloads may change based on the time of day or time of year. Having the ability to quickly scale up and scale down the cluster to support increased workloads is important. Cassandra and Astra DB have features that enable dynamic cluster scaling. The second aspect that could affect write workloads is if there are changes in the feature transformation logic. With a large spike in write workloads, Cassandra automatically prioritizes maintaining low-latency queries and writing TPS over data consistency, which is typically acceptable for performing real-time inference. Implement low-latency, multi-region support As real-time AI becomes ubiquitous across all apps, it’s important to make sure that feature data is available as close as possible to where inference occurs. This means having the feature store in the same region as the application doing inference. Replicating the data in the feature store across regions helps ensure that feature. Furthermore, replicating just the features rather than the raw data used to generate the features significantly cuts down on cloud egress fees. Astra DB supports multi-region replication out of the box, with a replication latency in milliseconds. Our recommendation is to stream all the raw event data to a single region, perform the feature generation, and store and replicate the features to all other regions. Although theoretically, one can achieve some latency advantage by generating features in each region, event data often needs to be joined with raw event data from other regions;. from a correctness and efficiency standpoint, it is easier to ship all events to one region for processing for most use-cases. On the other hand, if the model usage makes the most sense in a regional context, and most events are associated with region-specific entities, then it makes sense to treat features as region specific. Any events that do need to be replicated across regions can be placed in keyspaces with global replication strategies, but ideally, this should be a small subset of events. At a certain point, replicating event tables globally will be less efficient than simply shipping all events to a single region for feature computations. Plan for cost-effective and low-latency multi-cloud support Multi-cloud support increases the resilience of applications and allows customers to negotiate lower prices. Single-cloud online stores such as DynamoDB result in both increased latency for retrieving features and significant data egress costs but also creates lock-in to a single cloud vendor. Open-source databases that support replication across clouds provide the best balance of performance cost. To minimize the cost of egress, events and feature generation should be consolidated into one cloud, and feature data should be replicated to open-source databases across the other clouds. This minimizes egress costs. Plan for both batch and real-time training of production models Batch processing infrastructure for building models is used for two use cases: building and testing new models, and building models for production. Therefore it was typically sufficient for feature data to be stored in slower object stores for the purpose of training. However, newer model training paradigms include updating the models in real-time or near real-time (real-time training); this is known as “online learning” (e.g. ). The access pattern for real-time training sits somewhere between inference and traditional batch training. The throughput data requirements are higher than inference (because it is not usually accessing a single-row lookup), but not as high as batch processing that would involve full table scans. TikTok’s Monolith Cassandra can support a TPS rating in the hundreds of thousands per second (with an appropriate data model), which can provide enough throughput for most real-time training use cases. However, in the case the user wants to keep real-time training from an object store, Cassandra achieves this through CDC to object storage. For batch training, CDC should replicate data to object storage. It’s worth noting that machine learning frameworks like and PyTorch are particularly optimized for parallel training of ML models from object storage. Tensorflow For a more detailed explanation of “online learning”, see Chip Huyuen’s explanation on , or this . Continual Learning technical paper from Gomes et. al Support for Kappa architecture is gradually replacing Lambda architecture due to costs and data quality issues due to online/offline skew. Although lots of articles discuss the advantages of moving from separate batch and real-time computation layers to a single real-time layer, articles don’t often describe how to architect the serving layer. Kappa architecture Using Kappa architecture for generating features brings up some new considerations: Updating features are being updated en masse and can result in a significant number of writes to the database. It’s important to ensure that query latency does not suffer during these large updates. The serving layer still needs to support different types of queries, including low-latency queries for inference, and highTPS queries for batch training of models. Cassandra supports Kappa architecture in the following ways: Cassandra is designed for writes; an increased influx of writes does not significantly reduce the latency of queries. Cassandra opts for processing the writes with eventual consistency instead of strong consistency, which is typically acceptable for making predictions. Using CDC, data can be replicated to object storage for training and in-memory storage for inference. CDC has little impact on the latency of queries to Cassandra. Support for Lambda architecture Most companies have a Lambda architecture, with a batch layer pipeline that’s separate from the real-time pipeline. There are several categories of features in this scenario: Features that are only computed in real-time, and replicated to the batch feature store for training Features that are only computed in batch, and are replicated to the real-time feature store Features are computed in real-time first, then recomputed in the batch. Discrepancies are then updated in both the real-time and object store. In this scenario, however, DataStax recommends the architecture as described in this illustration: The reasons are the following: Cassandra is designed to take batch uploads of data with little impact on read latency By having a single system of record, data management becomes significantly easier than if the data is split between the feature store and object store. This is especially important for features that are first computed realtime, then recomputed in batch. When exporting data from Cassandra via CDC to the object feature store, the data export can be optimized for batch training (a ), which significantly cuts down on training infrastructure costs. common pattern used at companies like Facebook If it is not possible to update existing pipelines, or there are specific reasons that the features need to be in the object store first, our recommendation is to go with a two-way CDC path between the Cassandra feature store and the object store, as illustrated below. Ensure compatibility with existing ML software ecosystem To use Cassandra as a feature store, it should be integrated with two portions of the ecosystem: machine learning libraries that perform inference and training, and data processing libraries that perform feature transformation. The two most popular frameworks for machine learning are TensorFlow and PyTorch. Cassandra has Python drivers that enable easy retrieval of features from the Cassandra database; in other words, multiple features can be fetched in parallel ( ). The two most popular frameworks for performing feature transformation are Flink and . Connectors for both and are available for Cassandra. Practitioners can use tutorials for and , and Cassandra. see this example code Spark Structured Streaming Flink Spark Flink Spark Structured Streaming Open Source feature stores such as FEAST also have a and for Cassandra as well. connector tutorial Understand query patterns and throughput to determine costs The number of read queries for Cassandra as a feature store is dependent on the number of incoming inference requests. Assuming the feature data is split across multiple tables, or if the data can be loaded in parallel, this should give an estimate of the fanout between real-time inference that can be made. For example, 200 features across 10 entities in 10 separate tables give you about a 1:10 ratio between real-time inference and queries to Cassandra. Calculating the number of inferences being performed will depend on the inference traffic pattern. For example, in the case of an inference will be performed whenever a relevant feature changes, so the total number of inferences is dependent on how often the feature data changes. When inference is performed in a “request-reply” setting, it is only being performed when a user requests it. “streaming inference,” Understand batch and real-time write patterns to determine costs The write throughput is primarily dominated by how frequently the features change. If denormalization occurs, this too could impact the number of features that are written. Other write throughput considerations include caching inferences for either batch or streaming inference scenarios. Conclusion When designing a real-time ML pipeline, special attention needs to be paid to the performance and scalability of the feature store. The requirements are particularly well satisfied by NoSQL databases such as Cassandra. Stand up your own feature store with Cassandra or and try out the with the . AstraDB Feast.dev Cassandra connector

BUDGETREE

budgetree.in

-- employees

Since n.d.

COMPANY RANKING

BUDGETREE

EVERGREEN INDEX #2165

Representing this company? Click here to claim and customize this page!

Budgetree WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

BUDGETREE

budgetree.in

-- employees

Since n.d.

COMPANY RANKING

BUDGETREE

EVERGREEN INDEX #2165

RELATED COMPANIESRANK

STORIES ABOUT BUDGETREE

Representing this company? Click here to claim and customize this page!

Budgetree WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!