We’re an exciting new EdTech platform in stealth mode focused on employee learning and development

5Mins.ai

Dive into how AI is transforming the investment landscape Remember the early days of robo-advisors? They were like the first smartphones—revolutionary but limited. Now, generative AI is stepping in, offering more nuanced and personalized financial services. For instance, Q.ai offers customers investment kits picked by an AI that considers various data points, from market metrics to social media sentiment. 📊 Performance Metrics While AI's promise is high, its performance is still under scrutiny. ETF Managers Group's AIEQ, which uses IBM's Watson, has returned 4.9% over the last five years, trailing the S&P 500 index fund's 11.78%. It's like a new athlete who shows promise but has yet to win significant titles. 🛠️ Guardrails and Accountability AI isn't given free rein. Companies like Q.ai and AIEQ (see examples below) have human oversight to monitor AI output for biases and errors. Think of it as a self-driving car with a human driver ready to take over in complex situations. 🌐 Accessibility and Inclusion AI is democratizing financial advice. Firms like Fisdom in India use automated tools to offer financial guidance to a broader base of customers. It's like opening up an exclusive club to the general public, making high-quality advice accessible to all. 🤖 The Future: AI in the Back Office Even if AI isn't directly advising clients, it's finding a place in back-office systems. For example, it summarizes meeting transcripts, monitors risky transactions, and generates reports. Examples of startups / smaller companies using AI for investment management: 🤖 Q.ai Location: Jersey City, N.J. About: Q.ai offers no-fee "investment kits" consisting of 4 to 20 securities in a market sector. These kits are selected by an AI that evaluates market metrics, news, Google search trends, and social media sentiment. 📊 ETF Managers Group’s AIEQ Location: Summit, N.J. About: AIEQ uses IBM’s Watson AI platform to analyze millions of data points from various sources like news, social media, and financial statements of over 6,000 U.S. companies. 🌐 Nemo Location: Abu Dhabi About: Nemo uses a ChatGPT model to allow investors to ask questions they might typically ask a human broker. The company emphasizes that they don't let their AI go "off-leash." 📈 Toggle AI Location: Not specified About: Toggle AI aims to implement ChatGPT that will be programmatically constrained to provide reality-grounded answers to investors' questions. 🌟 Fisdom Location: India About: Fisdom uses automated tools to provide financial guidance to a broader base of customers, including individuals with low net-worth. 💼 Betterment Location: New York About: Betterment, one of the first mass-market robo-investing firms, remains cautious about using AI for financial advice. However, they do use AI in their back-office systems for tasks like summarizing meeting transcripts and diagnosing software issues. Examples of incumbent banks using AI for investment management: JPMorgan Chase: This financial giant is leveraging AI to optimize asset management and offer personalized investment solutions. Morgan Stanley is using AI to enhance its asset management strategies. UBS: Known for its wealth management services, UBS is adopting AI to improve its investment strategies and customer engagement. HSBC: This global bank is using AI to bolster compliance and risk management functions and augment and automate data analysis in the investment sector. 🔮 Summary The integration of generative AI into investment management is a significant development, offering more personalized and nuanced services. Performance metrics are crucial for assessing the efficacy of AI-driven investment platforms, and they currently show a mixed picture. The human oversight in AI-driven systems is a necessary safeguard, ensuring that the technology does not go "off-leash." Whether it's automating data analysis, enhancing customer engagement, or optimizing asset management, AI is playing a pivotal role in shaping the future of investment management. Excerpt from One More Thing in AI Newsletter. Get smarter on AI in 5 mins. Carefully curated from hundreds of sources for business leaders and founders. ) Join 1000+ subscribers for free.

How AI is Transforming the Investment Landscape

A lot of companies struggle to bring their data science projects to production. It is mainly because there is a huge knowledge gap, as data scientists understand model building well but lack skills. The simple reason is that these skills are not taught on Youtube videos and are hardly being touched by data science courses or the Kaggle learning method. productionization Objective of this newsletter is to share my learning from those various deployments, I have done. Tech Rule to deployment: Lesser dependency ∝ Faster deployment A friend of mine called me sometime back requesting that I help him with his use case and deploy the model to production. Let’s understand data science model deployment with a real problem. We discussed the problem for an hour or two to understand the constraints. Discussed Constraints Summary: Data Source is Elastic Search (ES very frequently updates with new entries) Real-time or Near Real-Time inference (with a delay acceptable till 10mins) Low on Budget Minimum Failure Rate with Fallbacks Alerting system in case of any failure occurred Post understanding the constraints and the problem, he was trying to solve. I proposed an architecture (check diagram below) that is near-realtime (batch inference every 5mins), fallbacks to the last model update (backend fetches previous update results from s3), and shared a slack webhook simple alert route. After about two weeks, he called up sharing that the solution worked well 🥳 (emotion super happy). The above is tried and tested: Low on Budget and Low on Maintenance — Model Productionisation Design. Let’s reason! The architecture design on how the above solves for the constraints Alternative Sagemaker Batch Inference: This could have been a good option but we did not go for it because he already had an EC2 instance running 24x7 and it was underutilized. Apart from the above for 5mins inference (near-real-time inference architecture), it is safer not to use Sagemaker batch inference while the real-time inference option is costlier. Dev familiarity is another factor that is super important when building an architecture design, EC2 has always been a playground for him. Another learning point: if you are running a model for with an update frequency of 1 day, and you want to run task compute on a new EC2 machine, on every run one can still use above architecture. One can follow this blog to learn “ ” This will help save cost of EC2 machine by running it only till compute window. How do I stop and start Amazon EC2 instances at regular intervals using Lambda? Why Airflow? is an open-source scheduler to manage your regular jobs. It is an excellent tool to organize, execute, and monitor your workflows so that they work seamlessly. Apache Airflow Airflow would trigger the data fetch from ES and prediction task in a defined interval for the above case - every 5mins. To write a scheduler expression one can use (this expression writing tool is excellent, I often use it when writing an Airflow task). Crontab.Guru Other Reasoning Pointers on the Architecture: Model is loaded from S3 in memory for inferencing so that local disk storage is not utilized by model weights. Advice: I have seen many data scientists keep the model file on machine storage. And when something goes wrong with EC2 all files get deleted, and with it all their efforts. Always keep the model on S3 as a backup. Output is overwritten on S3 and the backend picks model results from S3; S3 storage is reliable and cheapest on AWS. For alerts, slack is the best option given it is always on during office hours and when you are unavailable your team members can have visibility on alerts, one can even add airflow failure and retry emails but I prefer alerts with webhooks on the office communication tools: slack/flock/teams. I know there are better options to Airflow component like Dagster or Perfect; similarity for other components in the architecture there are new/comparative alternatives. But never forgot the factor of Dev familiarity when choosing tools for your model pipelines. Older the tool better the support and one cannot really downplay it. “WHAT IF, WE WANT TO DEPLOY A REAL-TIME MODEL IN PRODUCTION?” — my friend asked, even readers here must be thinking the same. We discussed the constraints, summarized below: A pool of Items is fetched from ES Real-Time Model Output with Timeout of 100ms required On Time-Out, Model Fallbacks to Cached version Logging is required Below is the real-time model productionisation architecture design 👇 Reasoning and Other takeaways on real-time Architecture I have shown dockerized model deployed on EC2. It can be deployed on ECS/SageMaker as well, would leave that choice to you. For caching — personal preference . Redis is for logging response and info logs within model service Kibana For model serving one can use , , , , etc. I prefer BentoML as in under 10 minutes, you’ll be able to serve your ML model over an HTTP API endpoint, and build a docker image that is ready to be deployed in production. MLFlow BentoML FastAPI Cortex Conclusion I hope that data science model productionization architecture design is not an out-of-syllabus question anymore. There is so much more to it than we can cover but do spend some time racking your Brain on it..! Also published here. I hope you learned something new from this post. If you liked it, hit 👍 or ❤️ and share this with others. Stay tuned for the next one! My is now read by more than 4500+ subscribers. If you are building an AI or a data product or service, you are invited to become a sponsor of one of the future newsletter issues. Feel free to reach out to shauryauppal97@gmail.com for more details on sponsorships. Newsletter on LinkedIn I am nominated for the HackerNoon 2022 Noonies, Vote for me: || HackerNoon Contributor of the Year - Data Data Science Demon Data Science Book Recommendations: [1] The Book of Why [2] Naked Statistics

The Design of Model Productionization Architecture in Data Science

Hello and welcome back. This is part 2 of the tutorial for building a transcriber app using OpenAI’s Whisper and the GPT-3 API. , we started the journey by looking up what elements are necessary for the transcriber app and how to get them. In part 1 We chose to use podcast audios for the transcription and uncovered the 2 ways to develop the app, either locally or from the Colab environment. We saw how to use OpenAI Whisper and the GPT-3 API to get the transcription from a podcast episode. And also, how to get the summary of an episode with working Python code that we can use locally or upload to a cloud service. In this second part, we’ll pick up from where we left off and go through the remaining steps on the road ahead. Alright, let’s go. Act 3: The Deployment (Whisper transcriber) In this final act, we want to take the Python function that creates the transcription and host it online. This is so that others can interact with the transcriber app and see the results. You can stay with your working prototype locally but you won’t be able to show it to others and no one will know that it works or how it works. That’s why we’ll see how to upload it to the cloud so that we can use it from anywhere, even if our main computer goes offline. The standard process here is to take the function and upload it to the cloud so that it can accept network requests (like from the front end of an app). For this, there are several options. You can use any of the traditional cloud providers like AWS, or GCP. There’s also an alternative to use services that make cloud deployments easier like Modal or . Begin Here’s where it can get tricky because of the many alternatives. And for the next step, which is building and deploying a frontend, there are alternatives. even more But fear not, tech adventurer, we’ll bring this journey to a conclusion with the help of a service called . Streamlit Streamlit is an open-source service that lets you build web apps from Python scripts and is mostly used for apps that involve data visualization and manipulation. If you’re developing locally, you can use it to build graphics and charts from data that you have on your PC. It’s as simple to use as running the following commands in a terminal. pip install streamlit
streamlit hello That’s it. But that’s not the reason I’m telling you about it. What’s interesting about Streamlit for our use case, is that it has a feature called “Streamlit Cloud” which is used for… you guessed it… uploading scripts to the cloud. Oneq of the cool things about it is that you can use it to turn your Python code into a fully functioning app without having to worry about: Encapsulating the code in a Docker container. Uploading it to a traditional cloud provider. Provisioning a virtual machine. Updating package dependencies. Installing the necessary software on the VM. Creating an instance for the backend and for the frontend. Managing the cloud infrastructure. Going through logs to figure out what went wrong. And many more…. We’re not trying to get fancy here. We just want to get this transcriber function running online so others can interact with it and see the results. Instead of turning it into a case of “it works in my machine”. And for that reason, we are simplifying the process as much as possible. Here’s how to get that Python script uploaded to the Streamlit Cloud. There are 2 ways you can go about that. Create a repository in GitHub to upload the code along with its dependencies. And then create the Streamlit account to deploy an app from that repo. Create the Streamlit account by connecting with GitHub and then adding the code using GitHub Codespaces (a dev environment online). Choose the one you think is more convenient. And if you need help creating the account, you can . check out the docs Now, here are the files we need to have in the repository for the app to work. First, create a simple text file called “requirements.txt” which will have these 2 entries. streamlit
openai-whisper Next, another text file called “packages.txt”. This one is for a dependency that Whisper has ‘ffmpeg’. ffmpeg And lastly, the actual code. You can call this file whatever you want, I called it “audio-transcriber.py” import streamlit as st
import whisper 

st.title("Podcast Transcriber")

st.sidebar.header("Transcription section")

audio_file = st.file_uploader("Upload audio", type=["wav", "mp3", "m4a"])

process_button = st.sidebar.button("Transcribe Episode")
st.sidebar.markdown("**Note**: Processing can take up to 5 mins, please be patient.")

model = whisper.load_model("base")
st.text("Whisper Model Loaded")

if process_button:
    if audio_file is not None:
        st.sidebar.info("Transcribing audio...")        
        result = model.transcribe(audio_file.name)
        
        st.sidebar.success("Transcription Complete")
        st.markdown(result['text'])
    else:
        st.sidebar.error("Please upload an episode.") With that code, we’ll now have a web app that looks like this: Let’s look at the code more closely. After the imports, we use the Streamlit syntax to create a title and header for the web app. The next line is an important one audio_file = st.file_uploader("Upload audio", type=["wav", "mp3", "m4a"]) With this line, we create a file uploader component that it’s going to receive the audio files for the transcription. And if you’re thinking, “an entire file uploader with one line of code?” only Yeah, building that with HTML, CSS, and JavaScript is a whole tutorial in itself. That’s the power of Streamlit. Now, let’s keep going. The following 2 lines of code allow us to create a button that will trigger the transcription process. And then a note for the user to be patient. After that is a familiar piece of code. model = whisper.load_model(“base”) And then another message letting the user know the model is now loaded and ready to use. If we keep going we’ll find some conditional logic. What that does is to check if the “process_button” was clicked. And if it was then it checks if there is a file uploaded. If the check passes, we now have the main part of the code. st.sidebar.info("Transcribing audio...")        
  result = model.transcribe(audio_file.name)
        
  st.sidebar.success("Transcription Complete")
  st.markdown(result['text']) What that does is print out a message that the transcription is in progress. Then, the transcription process takes place. And finally, we get a completion message with the results of the transcription. The rest of the code will only run in the case that the user did not upload a file when clicking the transcription button. And there you have it. Both the backend and frontend of the transcriber app have been created thanks to Streamlit. It wasn’t that difficult, was it? As I said before, this is the easiest and most straightforward route to get it working online in a way that we can share with friends and family. But if you do share it, make sure to tell them not to use it with lengthy audio files. They might keep waiting forever or break the app. Conclusion Now you have your own transcriber app working online, you can keep improving it further if you want. A good next step would be to use a Whisper replacement so that longer audio files can get transcribed without taking too long. There are a couple of options out there like the “ ” project or “ ”. Faster Whisper Whisper Jax You can also import the ‘openai’ module and save your API key as part of the “secrets” of the application. And with that, you can make calls to the GPT-3.5 API for summarization, highlight extraction, and more. This might be the end of our journey together but it can be the beginning of you using all these tools to create your own AI-powered apps like this podcast transcriber. That’s it for now. Thanks for coming along for the ride. Don’t forget to so that you don’t miss the upcoming articles. subscribe to me on Hackernoon

Using OpenAI's Whisper and GPT-3 API to Build and Deploy a Transcriber App – Part 2

Let's learn about via these 74 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Observability /Learn Repo 1. So, How are Observability and Monitoring Different, Actually? The similarities and differences between monitoring and observability and how to pair the two strategies 2. Observability Tips and Tricks For Using Grafana and Prometheus Rapid prototyping tips when using Grafana and Prometheus observability and monitoring stack. 3. Observability and Monitoring in a nutshell A guide to understanding the concepts and differences between monitoring and observability. 4. Debugging AWS Lambda Runtime Errors Debugging AWS Lambda can get tricky in Amazon CloudWatch. KloudMate lets you get to Lambda errors quickly and effectively using Lambda error logs. 5. An Intro to Dynamic CDK Dashboards How to create a construct based on other constructs in the same stack without forcing the developer to call a method for each? That's exactly what aspects do! 6. N/A please delete this 7. Developer Trends from Thoughtworks’ Technology Radar 27 Thoughtworks’ Technology Radar is a regular time to take a look at what trends are changing our industry. 8. How Observability and Monitoring Produces Better Software An article focused on deep diving into observability and its significance in software. Its history, goals, the importance of observability, and the issues that 9. How to Build Your Own Observability Platform on Kubernettes End-to-end full observability using Odigos and other back-end observability tools. 10. Maintaining Quality When Transitioning from Monolith to Microservices Piece by piece, legacy monolith applications are being broken down and replaced by microservices. 11. How to Monitor Apache Flink with OpenTelemetry Apache Flink monitoring support is now available in the open source OpenTelemetry collector. 12. Observability vs Monitoring: What's the Difference? Monitoring has been a basic system to track the health of servers for years. Now it is not enough. 13. Observability and Monitoring Have a Symbiotic Relationship, but They Are Different Observability vs. monitoring, what is the difference? Monitoring is the what to observability’s why. Here we dig into the differences. 14. Microservice.add(observability) != Microservice.add(monitoring) You are reading this content, which means that you are not novice to the microservices field. So let me just  scratch the surface of it before moving to Observable Microservices.  Once upon a time Monolith Application was now transformed into Microservices based application. 15. How Do You Make Cloud Migration Easier? Engage in intelligent change. 16. Application Monitoring: Closing Observability Gaps with Custom Metrics Which application metrics should you collect for your microservices? 17. The Challenges of Modern Observability It’s tough, impossible, to get a holistic view of your entire application when it’s running on 200 pods deployed to 12 nodes distributed across the world. 18. Why OpenTelemetry Should Matter to Network and Systems Admins OpenTelemetry is young, even by internet standards. Born out of the merger of OpenTracing and OpenCensus projects at the Cloud Native Computing Foundation CNCF 19. What is the ideal memory size to lower costs of running a task on Lambda? Should you increase Lambda memory? It might sound crazy, but increasing your AWS Lambda memory could actually lower your bills. Find out how. 20. What "Shifting Left" in Software Really Means for Blameless DevOps A truly "blameless culture" in software must to evolve from incident reporting, to telemetry aimed at Proactive Observability in DevOps 21. CI/CD/CF — The "Missing-Link" in the DevOps Toolchain? Everyone is familiar with CI and CD processes, but whatever happened to Continuous Feedback? Leveraging observability in dev creates a new type of dev process 22. Why Your Monitoring Dashboard May Be Feeding You Phantom Metrics We trust our metrics to show us the status of our system and where it misbehaves. But do our metrics show us what really happened? 23. Cloud Native Applications: 4 Building Blocks You Need in Your Platform To create amazing cloud native applications your platform needs these components: Kubernetes, Continuous Delivery Pipelines, Edge Stack, and Observability. This article will explain the importance of each component. 24. Comparing Different Serverless Monitoring Platforms Technology touches almost every corner of the world economy. Even when it’s an indirect relation, in many cases tech is an essential, vital part of our societies. It just can’t fail without causing too much distress and losses. Not only financially, but especially to the human aspect. 25. Get Started With Sidekick Open Source Live Debugger in 5 Mins Sidekick is a live application debugger that lets you troubleshoot your applications while they keep on running. Here is how you can start using it in 5 minutes 26. Logging in Observability - Part 1 rsyslog vs journald vs filebeat 27. Managing Collectors on K8 – Why We Chose the OpenTelemetry Collector for Odigos Without proper implementation and continuous configuration of your collectors, observability tools will be limited at best at best and many times ineffectual. 28. On Dynamic Observability and Team Culture with Liran Haimovitch, Rookout CTO Rookout Co-Founder and CTO, Liran Haimovitch, shares the origin story of their debugging tool, what excites him about the startup life, PLG, and more. 29. 9 Data Trends You’ll See in 2023 2022 saw the data space grow by leaps and bounds. Here are the top 9 things our team of data experts expects to see in 2023. 30. Observability and AI Partner Well to Tackle All of That Data Gone are the days when operations and development were a back-office affair. 31. Logging in Observability - Part 2 In this article, I want to describe how to work with logs, analyze them in the command line, and we will consider new modern tools to visualize logs. 32. Tracing: An Essential Component of Observability In this article, I am going to shed light on one important aspect of observability - tracing. 33. Microservice Observability Patterns [Part 1] Logging is one of the most important parts of software systems. Whether you have just started working on a new piece of software, or your system is running in a large scale production environment, you’ll always find yourself seeking help from log files. Logs are the first thing people look for when something goes wrong, or something doesn’t work as expected. 34. What is Observability? Is it Cultural? In order to leverage observability we need a significant shift in our corporate culture that encapsulates the entire company & goes beyond the tools. 35. Murre - the lightweight K8s metrics monitoring tool 36. Introductory Guide to Debugging Lambda Configuration Errors The “Lambda configuration error” is as generic as it gets but at the end of the day, it's a pathing issue. Let's go over how to resolve this. 37. Contextual Intelligence and Observability: Without the Former, You Really Don’t Have the Latter Observability is a hot term in the industry, but don’t let it fool you: having visibility into your organization's apps and services only gives you a piece. 38. The Everything Guide to Data Collection in DevSecOps Collecting wide swaths of observability and security data is key to a high-quality digital experience. Find out what you need to know to get started. 39. Better Performance and Security by Monitoring Logs, Metrics, and More Monitoring is a crucial part of observability. Learn how monitoring can specifically improve security, performance, and reliability. 40. Goodbye, Cost-depth Trade Off: Cloud-Native Observability, Redefined It turns out that, by rethinking the fundamentals of your approach to monitoring, you can have it all 41. Prometheus — Easy In, Difficult Out Setting up a Prometheus server can be easy, but getting actionable information out is a very long and never-ending journey. 42. Choosing a Computing Method: a Serverless SWOT Analysis If you ever find yourself deciding for or against serverless the following tries to make the decision easier for you. 43. Setting Up AWS CloudWatch Alerts (vs Dashbird Alerts) To Monitor Your Applications Learn about the best practices for AWS Cloudwatch Alerts and Dashbird Alarms, to not miss out on critical info about your serverless app. 44. Exploring Differences Between Monitoring And Observability Monitoring vs Observability: in this article, we're explaining what is observability exactly and how does it differ from monitoring. 45. Kuma and Prometheus for Observability in Kubernetes Microservices Clusters A year ago, Harry Bagdi wrote an amazingly helpful blog post (link at bottom of article) on observability for microservices. And by comparing titles, it becomes obvious that my blog post draws inspiration from his work. 46. To Log Or Not To Log, Is The (Eternal) Question An alternative logging strategy to make loggers your friends, not enemies 47. 2023 Observability Predictions: Open Source is the Key to Explosive Growth The observability industry is projected to grow about 8% in 2023, a continuation of a strong growth trend since the 90’s. 48. How Goji Investments Enhances Developer Experience via Observability This post was written by Dean Record, Engineer at Goji Investments. 49. What is API Observability? APIs are the blood vessels of digital business. 50. A Guide to Deploying Jaeger on Kubernetes in Production Logs, metrics, and traces are the three pillars of the Observability world. The distributed tracing world, in particular, has seen a lot of innovation in recent months, with OpenTelemetry standardization and with Jaeger open source project graduating from the CNCF incubation. 51. How To Use a FastAPI Abstraction To Query AWS Data Lake Our guide in creating FastAPI abstraction that allows us to query the AWS Glue metadata catalog - a Data Discovery API for AWS Data Lake. 52. Sizing Your Cloud Resources: 10 Common Mistakes to Avoid Learn the common pitfalls of moving to the cloud and how you can avoid them to truly benefit from the cloud’s elasticity. 53. CNCF Tools Overview: Are You Cloud-Native? Cloud computing is becoming more and more of a household name, with even the most conservative fields of business figuring out how to make the best use of it. Cloud computing usually starts with running a private cloud solution on premises before venturing onto the public cloud. Of course, the cloud is not a single uniform being. It may come from different providers, Amazon Web Services, Google Cloud Platform, and Microsoft Azure being the biggest players here. Or it may come with different visibility and hosting, that is, public (resides with the provider), private (self-hosted), or hybrid (which uses a bit of both). And the cloud can use different tools and APIs for management as well. 54. Microservice Observability Patterns [Part 2] In my previous article, I talked about the importance of logs and the differences between structured and unstructured logging. Logs are easy to integrate into your application and provide the ability to represent any type of data in the form of strings. 55. Improving Observability Through Service Level Metrics 56. Observability on Heroku: How to Monitor Apps on a Managed Infrastructure Cloud platforms like Heroku make it easier than ever to host applications: just upload your code, and they’ll deploy it for you. But a common misconception is that, because you don’t own the infrastructure, you can’t really monitor your applications or see under the hood. 57. Setting Up Notification Systems to Observe and Analyze Your Application By building observability and analytics into your notification system, you can identify and quickly resolve issues by monitoring how your product is performing. 58. Better Failure Detection in Serverless Apps Having reliable failure detection in place is a must for any application in production. Here's how to best achieve it for AWS Lambda. 59. 4 Key Observability Metrics for Distributed Applications A common architectural design pattern these days is to break up an application monolith into smaller microservices. Each microservice is then responsible for a specific aspect or feature of your app. For example, one microservice might be responsible for serving external API requests, while another might handle data fetching for your frontend. 60. Lua-Land CPU Flame Graphs in OpenResty XRay This post will introduce the idea of Lua-land CPU flame graphs and use OpenResty XRay to produce real flame graphs for several small and standalone Lua examples 61. 5 Key Takeaways from New Relic’s 2022 State of Logs Report The 2022 State of Logs Report provides a detailed view of how this practice is shaping engineering and the technologies of the future. 62. Introduction to Observability in ITOM and AIOps Observability is a best practice implemented by AIOps, enabling automation and expanding visibility into the entire organizational ecosystem. 63. Internal AWS Monitoring is Hard to Grasp Learm how a serverless monitoring solution can catch problems for you without the painful learning curve connected to serverless failure detection. 64. Jaeger Persistent Storage with Elasticsearch, Cassandra and Kafka Running systems in production involve requirements for high availability, resilience and recovery from failure. When running cloud-native applications this becomes even more critical, as the base assumption in such environments is that compute nodes will suffer outages, Kubernetes nodes will go down and microservices instances are likely to fail, yet the service is expected to remain up and running. 65. How to Monitor Serverless Applications With AWS CloudWatch Alarms Running any application in production assumes reliable monitoring to be in place and serverless applications are no exception. 66. How to Optimize Large Scale Serverless Applications for Operational Excellence Managing applications at scale often comes up as one of the biggest concerns for businesses; How can it work smoothly? How do we monitor so many resources? How do we maintain best practices with constantly evolving infrastructure? In this article, we run through the best approach for operational excellence looking at serverless monitoring strategy, serverless alerting strategy, and security and compliance best practices. 67. Fixing The ClickHouse Node Failure On Distributed Systems - A How-To Guide Part One: ClickHouse Failures, by Marcel Birkner 68. Introduction To Distributed Tracing Pattern A distributed architecture brings in several challenges when it comes to operability and monitoring. Here, one may be dealing with tens if not hundreds of microservices, each of which may or may not have been built by the same team. 69. 10 Threats to an Open API Ecosystem Despite tight economic situations worldwide, the API economy continues to grow. 70. Data Access for Microservices If you want to access data in a distributed environment such as in a microservice architecture, then data services are the way to go. The idea is to create a data abstraction layer (DAL) that the rest of the system’s applications and services can share. Thus, a data service gives you a generalized interface to the data you’re exposing and provides access to it in a standard manner. This would be in a well-understood protocol and a known data format. For example, a popular approach is to use JSON via HTTP/S. 71. Rethinking Programming: Automated Observability Introduction 72. Watching the Requests Go By: Reconstructing an API Spec with APIClarity In this post, we’re going to look at what API reconstruction is and how APIClarity solves the API observability problem. 73. Navigate Your Way to Production Bliss with Caretta The easiest way to map our cluster: Caretta - a standalone OSS tool, leveraging eBPF technology to be lightweight and frictionless 74. The Importance of Monitoring Big Data Analytics Pipelines In this article, we first explain the requirements for monitoring your big data analytics pipeline and then we go into the key aspects that you need to consider to build a system that provides holistic observability. Thank you for checking out the 74 most read stories about Observability on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

74 Stories To Learn About Observability

171 Stories To Learn About Monitoring

“Everything artificial is designed.” - Dan Norman, author of The Design of Everyday Things What is Canva? Canva is a software product that helps designers of any skill level to create media 10x faster. Canva simplifies designing: Presentations Blog post covers Social media posts of different formats (regular posts, stories & WhatsApp Status, etc.,) Ads across every digital ad network (Facebook Ad, Twitter Ad, LinkedIn Video Ad, Leaderboard Ad, Story Ad, Feed Ad) Canva makes it simpler and faster to design media assets embedded across the internet. Product: The product is designed with the perspective of getting a particular job done for the user rather than aggregating a bunch of tools that allow a designer to edit an image. Not everyone who wants to create a LinkedIn cover photo has the skill, time & patience to wrestle with the complex tooling available in Adobe Photoshop. Instead with 1 click, Canva allows anyone to get started, and within 5 mins of intuitive exploration of Canva’s predefined templates the user will be able to come up with a new custom LinkedIn Cover image. Example: The effort to do the job of creating a LinkedIn Cover post is reduced by 10x with Canva. The same holds for any type of job that Canva supports currently. Time Saver as a Mental Model: Great products save great amounts of time to their consumers. Cornelius Vanderbilt's railways saved time to travel across the country for people and goods. Amazon saved time for consumers to go to the big box stores. Canva saves time in designing media that is routinely used everywhere on the internet. Business Model: Canva uses the popular freemium model commonly used by most of the Software as Service products. They give away the product for free and charge a premium price for more advanced features. Pricing options include Free ($0), Canva Pro(12.95/Month $119.40/Year) & Canva for Enterprise (Pricing is unavailable). Canva's freemium version is generous and for a majority of use cases, the premium subscription is not required. This strategy helps Canva in building a large satisfied customer base that will become strong advocates for their product and even drive their enterprise subscriptions. The reciprocation effect is at its peak effect. It is also a bottom-up strategy that has been successful in other SaaS products like Slack. As someone who has created a ton of media assets on Canva, I would bet Canva has one of the highest NPS scores in the industry. Positives: Product: Canva nailed the in-browser product experience for creating media assets while the industry started moving from desktop to browser. They also nailed down the use cases consumers are using the product for, which was a step away from what Adobe was building at the time. Instead of competing with other design products like Figma, Canva's approach has been about integrating with competition and making them partners. A designer could still design things on multiple products like Figma, , or Adobe and in the end, create the final product in Canva. Facet.ai For every investor dollar that goes into a startup, $0.5 goes into two types of taxes. Cloud tax: Paid to AWS, Azure, or GCP Marketing tax: Paid to Google or Facebook Canvas adoption is also the result of nailing the use case of creating Ads across Google and Facebook ad networks. With a significant customer base, Canva has the opportunity to expand its product into the arena of companies like HubSpot in case of marketing and also compete with Shopify in case of Click to Print category services. Product & Market Expansion: Negatives: This is a personal bias as a long-time Canva user, but there is hardly any negative that I can see as an outsider in Canva's product, strategy & execution. Competition: Adobe has a Canva equivalent (Adobe Spark) which doesn't address the customer job to be done as well as Canva. There are new startups like , & (in India) which are creating products to make it easier to create media assets. Simplified Facet.ai Invideo The previous iteration of companies made it simpler to create & edit media in browser, the next generation of products will be an attempt to make them smarter using AI & ML. The Big Question: For a long time, the prediction was that Adobe would acquire Canva. This scenario is now more unlikely. Canva is estimated to be making a cool $1B revenue/year in revenue. This is higher than a lot of publicly traded SaaS companies. Adding to that equation is the recent fundraising announcement of $200M at a $40B valuation. It looks like a Canva IPO is more likely in the next couple of years than a Canva acquisition. P.S: The cover page for BMB newsletter is designed in Canva. P.P.S. If you are interested in getting an edge in early-stage startup business models, subscribe to the BMB newsletter here.

Canva Business Model Breakdown: Product, Revenue, Competition, and The Big Question 

If our approach to cyber security is working, why in 2019, are we reading about hacking, social engineering and data breaches every day? Every day? I know it’s , but we all know in the industry that many of the successful attacks can be prevented. Unfortunately, “PEOPLE” are the weakest link in the chain of our security solutions. And unfortunately, telling PEOPLE not to open emails from people they don’t recognize doesn’t stop them from… you know,… opening emails from people they don’t recognize. And telling them to “check the URL” is the single worst piece of advice you can give to anyone. easier to attack than it is to defend Below are domains that are available right now for purchase. Buy one and then get yourself a free SSL cert from Let’s Encrypt. That will fool 99.9% of everyone you test, including security professionals. Can you see what’s wrong with the domains below? If like me, you get 99% in ‘feature detection’ aptitude tests, you’ll be able to spot the difference, eventually. Unfortunately you can probably tell from my poor writing skills, that I tested average to below average in English. 🥴 Did you spot what’s different about the domains? Below is a funny thread I found on Reddit that summarizes the problem with today’s anti-phishing awareness training techniques. Awareness training is important, but you get my point. 🤣 If existing email security solutions that use massive threat intelligence systems work well, why are the five bullet points below true in 2019? When speaking to people at RSA, it was apparent that most companies are focused on blocking threats. And many of the companies on display, collaborate beautifully by sharing knowledge, technology and techniques, so together, we can tackle these challenges. I absolutely love how collaborative the cyber security world is. But is this enough? Is it working? Do we need to stop and think differently? known It’s 2019 and we are only now starting to see the Web being encrypted — but there’s still a very long way to go. It’s only now that Let’s Encrypt, a non-profit initiative, is helping to create “a more privacy-respecting Web”. But as . I’ve discussed in previous posts, that introduces other challenges Even organizations that use five solutions from five different security vendors say that Phishing remains a serious concern for them. And while these systems may block over 95% of spam and threats, it only takes one person to open one link that goes to the wrong site or downloads the wrong software. If you want to go down a rabbit hole, here’s a post I wrote in April 2015 “ ” — most people at the time didn’t even know what a WebView was and today, almost every mobile app opens links inside the WebView instead of the native browser. How are we detecting and preventing malicious links from being opened or shared inside Facebook, Twitter, LinkedIn, Telegram, Slack, WeChat, WhatsApp…? How WebView has weakened the TCB of the Web infrastructure We read about a hack or data breach everyday 90% of data breaches start with Phishing 59% of ransomware attacks originate with phishing emails 91% of all malware is delivered by email 93% of Phishing sites have a padlock thanks to automatically issued DV certs Are we proud of these stats? How I came to learn that chasing after the Phishers is like chasing Moby-Dick In summary, I learned that we were doing it wrong. In this instance I’m referring to MetaCert, and not us cyber geeks as a collective. In June 2017 MetaCert was approached by a number of crypto currency companies to help bring a stop to the outrageous Phishing scams on Slack — they were happening every 5mins, literally. MetaCert was the only company with an anti-phishing security integration, for which I own foundational patents. More specifically, I own patents for anti-phishing and anti-malware detection and prevention inside mobile app WebView using an API for URI detection techniques. It’s impossible to detect a phishing or malware link inside a mobile app without using this technique. I digress, but I’m simply highlighting some personal insights to show that this stuff is all I’ve been thinking about for a while — even though our website is terrible and doesn’t tell you anything. So I made a risky call and decided not to execute a go-to-market strategy for our enterprise customers to see if we had a good product/market fit, in favor of addressing the immediate need of people who were losing their money every 5 minutes. Again, literally because the Slack bot reminder was set to repeat attacks every 5 minutes in many instances. This was a big risk because our customers included IBM, Sage, SAP, NTT Security, AppDirect, Blackhawk Network and about 900 other companies. I had no idea at that time, if they appreciated the utility and whether they were willing to pay for the solution or not. But what can I say? I didn’t know if there was utility, because I had no way of knowing if we were actually preventing attacks from happening. Crypto was rife with phishing problems, and in public communities, so I knew we could achieve demonstrable, measurable success. It took three months of further product iteration to get it right, as threat actors were using techniques that we couldn’t have predicted when building our solution for enterprise customers — they were using Slackbot Reminders, DMs and the incoming webhook API. Our integration ended up automatically deleting messages that contained a malicious link with a DM sent to an admin so they knew what had happened. Finally, we nailed product/market fit. MetaCert was installed across almost every crypto company worldwide and like turning off a tap of water, Phishing was completely eradicated, with zero attacks today. And naturally, soon after the attacks stopped, crypto customers decided that they no longer needed our service. But that’s another pain point in the life of a startup trying to find its way in the world of big amazing competitors. 🤦‍♂️ Along the way, we built a very passionate community of over 7,000 members who would immediately (and still do today) report new attacks, which MetaCert would classify within minutes, and then blocked by our products across every company in real time — all while the Google Safe Browser API is fast asleep. I sometimes wonder if Google’s API is addicted to sleeping pills it’s that slow — they’re too big for comfort so I don’t mind naming them. Don’t get me wrong, I don’t think we built a solution that could detect and prevent every new attack. That’s the problem I came to learn the hard way. We didn’t make it impossible for threat actors to scam people. We just made it less attractive to attack our customers and by doing that, we made Slack a less attractive platform, as it became more difficult for them to gain momentum before being caught and blocked. Imagine if you could achieve the same success for every customer using email. 🤔 I predicted that threat actors would migrate their efforts to a less secure platform. I recommended to the crypto ecosystem that they remain on Slack as by then, it had become the most secure platform for communities and companies who want to connect with other companies, as Phishing was now a thing of the past. I specifically warned them that if they moved to Telegram, threat actors would move there with their scams. Guess what happened? Crypto companies moved to Telegram to build their communities as it became the new cool tool in town. And of course, the threat actors migrated with them and started their shenanigans yet again. $1.6bn was stollen in 2018 alone. So, MetaCert built a security bot that’s now protecting over 1.3 million crypto people across 1,300 groups on Telegram. And I’m pleased to say that phishing has stopped there too. But what about email? All of this work and more, taught us that no matter what we do to detect and protect people from new threats, it’s technically impossible to stay ahead of the bad guys — they’re just too smart and most people are easy prey. It’s impossible to protect 99% of the people 99% of the time. At that time, we didn’t have a security solution for native email clients (but we do now, and they are mind blowing!), so we asked ourselves; “if we were to invent how we do things at our core today, what would that look like?” — similar to what Apple did when they re-invented how we listen to music with the AirPods. Apple didn’t design tangle-free wires. They removed the problem by changing the narrative. The AirPods is arguable one of the best products launched since the first iPhone, if not the best. We answered that question with a social experiment that started in December 2017, involving a new way to present Website identity with new visual indicators. I will explain what happened next in follow up posts. But suffice to say, we learned the hard way over a period of 2 years, while protecting the most widely targeted people on the planet, by an order of magnitude, that trying to detect threats alone, isn’t the answer. It helps, and it’s very important, but it’s not the answer. new Dear cyber colleagues, we need to rethink this stuff. While we opted for Google’s search algorithms over Yahoo!’s carefully curated listings back in the day, perhaps we need to consider going back to URL curation. Perhaps we need to curate what is verified as “safe”, and assume everything else is either hostile, or part of the — content we shouldn’t care about. long tail Perhaps we should ignore the long tail and focus on the head. What do you think?

A rhetorical question about the effectiveness of our security solutions

Hi guys, i work as a programmer more than 15 years and was using many different languages, paradigms, frameworks and other shit. And i want to share with you my rules of writing good code. Always write code that simple to read and which will be understandable for developers. Because time and resources that will be spent on hard readable code will be much higher than what you get from optimization.If you need to make optimization, then make it like independent module with DI, with 100% test coverage and which will not be touched for at least one year. Optimization VS Readability. Fuck the optimization. I saw many people who was saying “We need doing things fast we have no time on making architecture”. And about 99% of them got big problems because of such thinking.Writing code without thinking of its architecture is useless in the same way as dreaming about your desires without a plan of achieving them.Before writing the first line of the code, you should understand what it will be doing, how, what it will use, how modules, services will work with each other, what structure will it have, how it will be tested and debugged, and how it will be updated. Architecture first. Tests are good thing but they are not always affordable and make sense for the project.When you need tests:- When you are writing modules, micro-services which will be not touched for at least one month.- When you are writing open source code.- When you are writing code that touches financial channels.- When you have resources for updating tests at the same time as code was updated.When you don’t need test:- When you are a startup.- When you have small team and code changing is fast.- When you write scripts that can be simply tested manually by their output.Remember that code with badly written tests can be more harmful then code without tests. Test coverage. Don’t write complex code. More it simpler then less bugs it may have and less time needed to debug them. Code should do only what it need without tons of abstraction and other OOP shit (especially it concerns java developers) + 20% of things that may be needed in future to update it in simple way. Keep It Simple, Stupid. **Comments.**Comments showing bad code. Good code should be understandable without a line of comments. But what to do to save time for new developers? — Write simple inline documentation describing what and how method work. This will save much time for understanding and even more — it will give people more chances to come up with better implementation of this method. And also it will be good start for global code documentation. **Hard coupled VS Less Coupled.**Always try to use micro-service architecture. Monolithic software can run faster than micro-service software, but only in the context of one server.Micro-services give you possibility to distribute your soft efficiently not only on many servers but sometimes even on one machine(i mean process distribution). **Code reviews.**Code review can be as good as it can be bad.You can organize code review only if you have developer who understand 95% of the code and who can monitor all updates without wasting to much time. In other situation it will be just time consuming and everyone will hate this.On this part got to many questions so describe this more deeply.Many people think that code review it’s a good way of teaching new guys, or teammates who works on different part of code. But the main target of code review it’s maintaining code quality, and not teaching. Let’s imagine that your team making code for controlling cooling system for nuclear reactor, or space rocket engine. And you made huge mistake in very hard logic, and then you are giving this for code review to the new guy. How to you think what would be the risk of accident? — On my practice more than 70%.Good team is where each person has own role and responsible for exact piece of work. If someone want to understand another piece of code then he goes to a person responsible for it ant ask her. Impossible to know everything and better excellent understand small piece of code than all but on 30%. **Refactoring does not work.**During my career i heard many times “Don’t worry we will refactor it in future”. And in future this results in big technology debt or in deleting all the code and writing from scratch. So don’t get a debt unless you have money to develop you software few times from scratch. When developers tired they are making x2–5 more bugs and mistakes then when they are full of energy. So working more is very bad practice. That’s why more and more countries thinking about 6 hours work day, and some of them already have it. Mental work is not the same as working with your biceps . Don’t write code when you are tired or in a bad mood. **Don’t write all at once — make developing iterative.**Before writing code analyze and predict, what your customers/clients really need, then select MVF(Most Valuable Features) that you can develop with good quality in a short term. Use such iterations to deploy quality updates, and not waist time and resources on unreasonable desires and sacrifice with quality. **Automation VS manual.**Automation is a 100% success in a long term. So if you have resources to automate something right now it should be done. You may think “it takes only 5 minute, why i should automate this?”. But lets calculate this. For example it’s an everyday task for team of 5 developers. 5mins * 5devs * 21days * 12 month = 6 300mins = 105hours = 13.125days ~ 5250$.and how much this will cost if you have 40 000 employees? **Go out, get hobbies.**Work differentiation increases mental abilities and gives new fresh ideas. So make pauses and go out on fresh air, talk with friends, play the guitar, etc. **Learn new things as you get free time.**When people stop learning they start to degrade. I’ll be very grateful if you can share in comments your ideas and practices about writing good code. My new story: _Today I want to share with you few things that helped me become a better person than I was. It’s not some fast super…_medium.com How to forge a man out of yourself. Story of my life.

5MINS.AI

5mins.ai

-- employees

Since n.d.

COMPANY RANKING

5MINS.AI

EVERGREEN INDEX #1633

Representing this company? Click here to claim and customize this page!

5Mins.ai WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

5MINS.AI

5mins.ai

-- employees

Since n.d.

COMPANY RANKING

5MINS.AI

EVERGREEN INDEX #1633

RELATED COMPANIESRANK

STORIES ABOUT 5MINS.AI

Representing this company? Click here to claim and customize this page!

5Mins.ai WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!