Empower your business team to explore transaction data in your CDW, combine with business data from your CRM to help to diagnose and solve revenue problems, without SQL or escalating to Engineering

Facet Data

Realistic prototype data is a cornerstone of successful , offering invaluable insights and support across various stages of the development process: software development Mock data allows developers to simulate real-world scenarios and test the functionality, performance, and usability of their applications under different conditions. By providing representative data sets, developers can identify and fix bugs, validate features, and ensure the reliability of their software before deployment. Testing: During the development phase, realistic data enables developers to work efficiently by providing them with data that closely resembles what the application will encounter in production. This allows developers to design and implement features more accurately, as they can see how the application interacts with actual data. Development: When onboarding new users, especially in the case of complex applications or platforms, realistic mock data can facilitate a smooth and intuitive user experience. By presenting users with data that reflects typical usage scenarios, they can familiarize themselves with the application's features and functionality more quickly, leading to higher user satisfaction and adoption rates. Onboarding New Users: In essence, while ‘lorem ipsum’ has its place, it's the authenticity and relevance of data that truly drives software development forward. Open Source Synergy: Classroomio's Revolution with Supabase and Sveltekit As an active contributor to open-source projects, I recently had the opportunity to collaborate on , an innovative educational platform reshaping virtual classrooms. Given the nature of the project, where educators and learners interact dynamically, the need for realistic prototype data was paramount. Classroomio Built with Supabase and Sveltekit, Classroomio combines the power of a robust database management system with the flexibility and efficiency of a lightweight front-end framework, ensuring scalability, performance, and good user experience. Database Management with Supabase Supabase is a feature-rich software built on top of . It abstracts away many of the complexities involved in working directly with PostgreSQL, providing simplified APIs and integrations for common tasks, and adding additional features like real-time updates, authentication, and storage. PostgreSQL By providing a friendly interface to interact with the Postgres data server, developers can build and manage web and mobile applications while leveraging the power and flexibility of PostgreSQL as the underlying database engine. Unraveling Database Dilemmas: A Case Study While tasked with the essential process of seeding the database for our Supabase application, I came across an interesting hurdle that shed light on the intricate nuances of database management. Initially, the objective appeared straightforward: generate an SQL dump file containing mock data reflective of real-world scenarios to populate our Supabase app's database. However, what ensued was an illuminating journey through debugging, ultimately leading to a deeper comprehension of database intricacies. In this article, I’m delighted to share the invaluable insights garnered from troubleshooting this database conflict. Despite grappling with a limited understanding of the schema at the onset, a common occurrence when immersing oneself in a new codebase, I've distilled these experiences into actionable lessons that might resonate with any developer navigating similar challenges. By reading this article, you'll find: A practical understanding of database management with Supabase Troubleshooting techniques that can empower you to confidently navigate similar challenges in your own Supabase projects. Seeding the Database As I delved into the process of seeding the database, I leveraged the user interface provided by the application. This task extended beyond merely populating the database; it involved meticulously curating demo course content and configuring various attributes such as course titles, descriptions, and associated resources. One crucial aspect to note is the importance of realism and relevance in our data curation efforts. This means that for our use case, each facet of data had to be carefully curated to mimic real-world scenarios as closely as possible, ensuring that our mock data accurately reflects the anticipated user experience. Generating Seed Data with Supabase The next critical step in our database seeding process is generating the SQL seed file. This file serves as a comprehensive snapshot of both the database schema and its data, facilitating seamless replication and setup of the Supabase app for new users. To accomplish this, we utilize the command, adding specific parameters to indicate that we want to capture only the necessary data without including any structural information. supabase db dump Executing the following command: supabase db dump -f supabase/seed.sql --data-only This command instructs the Supabase tool to create a backup file named `seed.sql`, containing exclusively the actual data stored within our database. Therefore, it omits any details regarding the database structure, focusing solely on the prototype data essential for seeding the database. Task Evaluation Now, upon completion of this step, one might expect the aforementioned task to conclude successfully. However, the subsequent action which is resetting the database and repopulating it with the contents of seed.sql yielded unexpected results. Let’s just say things took another turn. Encountering Database Integrity Challenges: Obstacles in Data Population When attempting to populate the database, I encountered an unexpected setback—a series of database integrity violations (I share a screenshot showing one of many below.) Essentially, this meant that default data already existed in our database, conflicting with attempts to add new data with the same IDs. Two-Step Debugging Process: Database Examination and Data Exclusion In addressing this issue, my approach was twofold: Comprehensively examine the database. Exclude tables with existing data from the SQL dump file. This process demanded a better understanding of the Supabase backend. My first point of call was definitely the docs, there it says that . every Supabase project comes with a full Postgres database Great, now the next action then was to engage the PostgreSQL server via Docker to enable me work directly with the database. Harnessing Terminal Efficiency: Docker and PostgreSQL Utilizing the terminal as my primary debugging tool, I executed the following commands: Cmd 1: docker ps Cmd 2: docker exec -it [container_id] psql -U postgres The first command displayed the running containers, enabling me to identify the specific container image for PostgreSQL. Subsequently, logging into the PostgreSQL server granted access to the database, allowing me to view its contents. By examining the tables, I identified those with default data—a crucial step in rectifying the primary key violation error. Database Resolution: Achieving Successful Database Seeding This debugging process thus provided necessary insights into the database structure of the project, informing subsequent decisions regarding the contents of the SQL dump file. Armed with this knowledge, I resolved the primary key violation error, ensuring the successful seeding of the database. Personal Reflections and Key Takeaways While fine-tuning the seeding process, I encountered an opportunity to optimize data dependencies and validate the SQL dump file to make for a smoother workflow overall and an error-free database population. This was done by organizing the order of table seeding to prevent key violations. However, I came to appreciate that this cleanup operation could also be effectively handled with a custom script using Python, Perl, or Bash, or a third party tool like . Snaplet I’d like to think it all depends on the use case(s) too. Best Practices for Working with Seed Data In this section, we'll discuss some important considerations to keep in mind when navigating similar tasks on a project. Schedule regular backups of your database to ensure you have up-to-date copies of your data in case of unexpected issues or data loss. This ensures that you always have a recent version of your database available for seeding or restoration purposes. Regular Backups: Consider storing SQL dump files in a version control system. This allows you to track changes to the database schema and data over time and revert to previous versions if needed. It also makes it easier to collaborate with team members and maintain a history of database changes. Version Control: Document the structure and contents of SQL dump files to provide context for future users or collaborators. Include information about the database schema, table relationships, and any custom configurations or constraints applied during the seeding process. This helps ensure consistency and understanding across different environments. Documentation: Before deploying SQL dump files to production or sharing them with others, validate the integrity of the data and test the seeding process in a controlled environment. This helps identify any potential issues or discrepancies early on and allows you to make adjustments as needed before impacting live systems or users. Validation and Testing: Consider implementing incremental updates or differential backups instead of full database dumps whenever possible. This approach only captures changes made since the last backup, reducing the size and complexity of the dump files and speeding up the seeding process. Incremental Updates: Take appropriate security measures to protect SQL dump files from unauthorized access or tampering. Encrypt sensitive data, restrict access to authorized users or systems, and implement secure transfer protocols (e.g., HTTPS) when sharing or transferring dump files over the network. Security Measures: Conclusion This article has taken us through the process of seeding a Supabase application using a real scenario. From resolving database integrity conflicts to optimizing the seeding process, each step along the way provided valuable insights and lessons that can be applied beyond the confines of our project. The importance of attention to detail cannot be overstated, as even the smallest oversight can have significant implications for the integrity and performance of our applications. Further Reading - A great comic by Julia Evans that can help you become more effective at debugging “How I got better at debugging” The Supabase database docs

How to Seed a Supabase App: A Hands-On Guide

Now more than ever before, personalization in marketing and consumer messaging matters. Customers nowadays expect the brands they interact with to provide them with a personalized experience, as this allows them to feel seen and heard. Hyper-digitalization of has only further led to many brands increasing their online presence, allowing them to leverage new opportunities in new markets and helping to boost their engagement among loyal customers. marketing In fact, shows that personalization has become one of the most important facets of the customer journey as it helps to drive performance and further enhance the customer experience, with around 71 percent of consumers expecting brands to deliver personalized interactions, and the same percentage feeling frustrated when companies don’t provide a personalized experience. research Countless studies, surveys, and reports have shown that personalization not only leads to improved customer engagement but also helps to drive influence, boost revenue, and increase the customer conversion rate. Yet, with all this research and countless proof that indicates personalization works, some businesses are still lagging behind, leaving them in the dust, while competitors scoop a growing portion of the market share. Boosting Marketing Personalization Already in 2023, and the years before that, we’ve realized that marketing personalization can play a pivotal part in a company's success in engaging and retaining clients. Now, in 2024, two entrepreneurs, Dusty and Peter McMullin, are hoping to shed some more light on the matter of personalization, encouraging others to follow a framework that could further boost their customer retention and grow revenue. Collect Data and Analysis “Data has become the name of the game, and without it, your business is losing out on so many more valuable opportunities,” says Peter McMullin, President of Sibu Sea Berry Therapy, a natural specialty supplement and functional food brand. “There’s more to data than just numbers; in fact, we tend to think of data as our roadmap or blueprint that’s helping to guide us towards building a more personalized customer experience.” In the last several years, companies of all shapes and sizes have boosted their investment in data tools, including data harvesting and analysis tools. Using these channels, companies can gain an increased overview of who their customers are, where they’re situated, and what type of questions they’re asking or answers they’re looking for. Data collection, along with the use of Artificial Intelligence (AI) now gives a more in-depth view of what customers want from the brands they are shopping at. Data allows companies to ultimately understand customers’ pain points and the problems they’re looking to resolve. Establish a buyer persona. Now that you have the tools to collect consumer data, the second step can be to develop a buyer persona. The buyer persona should represent or be a representation of the type of people you would want to see in your target audience. requires an in-depth look at social and financial demographics. As part of this process, you’ll need to identify the touch points through which you can reach your buyers, as these can include an omnichannel of diverse areas, including social media, email marketing, in-store experiences, or other forms of messaging. Developing a buyer persona Dusty McMullin, Vice President of Operations at Sibu shares, “Developing a buyer persona gives you an understanding of the type of customers you’ll most likely be marketing for, and what are some of their likes and dislikes.” The more layers you begin to add to your buyer persona, the easier it will become for you to build a well-detailed action plan for your marketing efforts. On top of this, it would allow you the opportunity to better understand how you can craft your messaging or even further improve your personalization efforts. Segment your audiences “As a company that sells a range of niche consumer-focused products, segmentation allowed us to know what products we should be marketing to which groups of our target audience,” says Dusty. With segmentation, you begin to take another step further down the road of personalization, allowing you to further divide your audience into different groups based on various demographics, including age, gender, income, and location. By having a segmented approach, you will begin to cancel out some consumers who don’t fit your target message and further focus on those groups that provide you with more upside potential. Segmentation allows you to get behind your clients and know exactly who they are and what they’re looking for. More importantly, segmentation makes it easier for you to craft messages based on these groups’ needs and where you will need to provide customers with a point of interaction i.e., social media, email marketing, print media, etc. Get to know what your customers want By asking your customers what they want, you begin to collect more real-world data that can form part of your wider personalization marketing plan. This is simply yet another facet of data collection and is often considered a more practical approach for smaller businesses that don’t necessarily have the resources or tools to harvest and analyze vast amounts of consumer data. By asking customers what they want, you further paint a more detailed picture of your customer persona, helping to answer their most burning questions and see which areas of the business strategy and marketing campaign can be improved to help generate more engagement. Take, for example, some Search Engine Optimization (SEO) or email marketing report tools that can become valuable resources for determining what your clients are looking for, but also what are current marketplace trends. Peter shares, “In the beginning years of our business, around the time the internet was picking up steam, we had to craft emails in such a way that it deliberately asked a few simple questions, with a call to action for each.” He continues, “This allowed us to get an idea of our clients, where they may be in the customer journey, and what we can do to improve the road ahead.” Peter and Dusty recently launched a natural Omega-7 supplement that contains the proprietary T7 Sea Buckthorn Oil, created by Sibu. Sea Buckthorn has been found to hold a variety of natural properties that alleviate inflammation and boost cell production. In this scenario, we see that providing a highly specialized and niche product would require brands to develop a strategy that would enable them to better understand where clients are at in the customer journey and whether a product like this is something that will help answer their needs. Leverage social media tools If you didn’t already know, social media has a plethora of business marketing tools that can help you build a more robust marketing strategy through data collection, segmentation, and targeted ads. As a business owner, you can use social media to create a plan that focuses on meeting certain consumers at specific touch points. Although you may already have a social media marketing strategy - if you don’t already have one, start planning - taking into account which consumer segments make use of which platforms can help you further craft content messages according to their needs. People use social media for different things. Research by the found that nearly half of American social media users currently obtain their news from social media platforms, including Facebook, YouTube, Instagram, and X (Twitter). Then there are things such as gender, where research shows that more women tend to use social media platforms compared to men as a whole of total active social media users. Pew Research Center Using these platforms gives you a bird's eye view of what customers are doing on different platforms, how they perceive your messages, and what you can do to craft more engaging content, including multimedia and written content. “When our father, Bruce McMullin, founded his business back in 2004, social media wasn’t even around yet, well at least not in the capacity it is today,” says Dusty. “However, in recent years, we’ve managed to grow our brand representation and continue building new relationships with existing and potential clients through the use of social media. I think it’s mostly about our approach, but also about how we craft our messaging in such a way that we can inform customers about new product ranges or trends taking place in the industry.” Integrate Artificial Intelligence (AI) The future of online business, more specifically e-commerce, will be powered by Artificial Intelligence. Already, we’re seeing businesses across a wide range of industries and sectors bringing onboard AI tools to automate specific tasks, such as generating new social media content, analyzing customer trends, developing more advanced automated responses, and even building robust marketing strategies. Without artificial intelligence, it’s becoming increasingly difficult to navigate new challenges and obstacles in the market. More than this, there’s an entire ecosystem now being developed to help business owners find more meaning from the data they have on their customers and create personalized experiences with foolproof insights. However, while AI integration is important and valuable, the integration process thereof is just as important. You will need to that allows you to understand the basic fundamentals of artificial intelligence and what this technology can do for your business. have an AI strategy Without a clear plan about how you can implement AI in your marketing campaigns, it will leave you falling out of the pot and into the pan. Make sure that you conduct a thorough investigation of the different AI tools that are available, how they’re used, and what can benefit your business the most. Be human in your delivery Customers want to have an authentic experience more than anything, perhaps. While it’s important to build a marketing strategy around the facts and figures, letting that take charge of your business will only lead to customers feeling disengaged from your brand or your messaging. One study by revealed that around 57% of customers will likely increase their spending with a brand that they feel more connected with. Even better than this, 76% of those customers are more likely to buy from a brand, which in this case may be you, over a competitor if they feel connected with that brand. Sprout Social “People want to have real connections with a brand,” says Peter. “The limitation in this scenario is only how well you can craft a message in such a way that it makes people feel seen, heard, and noticed. Nowadays, customers want to know who’s the people behind the brand, their faces and voices. Make that part of your personalization statement.” Perhaps the takeaway here is that it’s important to be human first and then everything else. Creating a more authentic experience carries value and helps to drive customer engagement. By making this an important part of your business, you will help customers better understand the people behind the brand and the personas that are helping them find workable solutions to their problems. Wrapping Up In 2024, we can expect to see a lot more brands pushing to create authentic and personalized experiences for customers in a hyper-digital ecosystem of online shopping and social media. However, this creates a bit of a challenge for younger, less experienced entrepreneurs who are only starting their business journey. Although countless tried and trusted methods work, finding the one that perfectly fits your business niche and consumer market can help you drive the conversation and foster more meaningful relationships with your clients. “Trust the process; that's perhaps the one thing we learned when it comes to personalization marketing. Not everything will always go to plan. However, knowing that you can deliver an authentic experience that resonates with your brand and tone of voice will see you build a more meaningful connection with loyal and potential customers,” says Peter.

7 Tips To Actively Boost Your Marketing Personalization In 2024 According To Entrepreneurs  

In the midst of the Fourth Industrial Revolution, a fundamental transformation is unfolding, driven not by traditional resources but by data. This shift towards ‘data capitalism’ is taking root, as the economic value of information escalates. In this world, data, when leveraged appropriately, becomes a linchpin for growth and innovation. The Emergence of Data Capitalism In today’s landscape, the market is dominated by large entities that collect, analyze, and apply data across diverse functions. These companies have strategically claimed their market positions by absorbing potential competitors and capitalizing on their vast data portfolios. For example, Russian multinational corporation Yandex leverages its collected customer data from its taxi service to bolster its competitive edge against competitors like Uber. The Power of Data Monetization Data monetization, a vital facet of data capitalism, focuses on deriving profit from the collection, analysis, and application of data. Businesses comprehend the enormous potential of data and exploit it to their advantage. For instance, they accumulate vast datasets, seldom shared or sold externally, and utilize these for their internal benefits. This proprietary approach can establish a significant competitive advantage, as it allows a business to collect, categorize, and use data effectively. From Data Collection to Effective Application The intricate web of data collection and application can be seen in Yandex’s unique film rating system on KinoPoisk. This system, designed for Russian speakers globally, is superior to any other. The system’s success is not just dependent on the collection of data, but also its strategic analysis. However, it’s worth noting that the focus is not merely on data collection; it’s about the practical application of this data. Even with vast data collected by platforms like KinoPoisk and Netflix, recommendations are often unsatisfactory because they lack context. Challenges in Data Management and Privacy As the technology landscape evolves, the challenges for data management and privacy are escalating. Governments find it increasingly challenging to combat data privacy breaches. These challenges are majorly due to the volume of unstructured and non-standardized data and the pace at which digital evolution is outstripping legislative developments. AI and AGI: Transforming Data Capitalism As we look to the future, we can’t overlook the vital role of Artificial Intelligence (AI) and Artificial General Intelligence (AGI) in this scenario. AI and AGI, the latter being a technology with potential to equal or surpass human intelligence, have the potential to enhance our capacity to process and analyze large datasets exponentially, thereby reshaping the dynamics of data capitalism. AI and AGI integration could revolutionize data processing and analysis, accelerating the processes from weeks to mere minutes. Consequently, this will increase the economic value of data, further cementing the shift towards data capitalism. More importantly, with the growth and adoption of AGI, we could witness data analysis reaching levels of sophistication and efficiency that are currently unimaginable. However, with every significant technological advancement comes the risk of misuse and potential exacerbation of existing challenges. As AI and AGI technologies become more powerful and pervasive, the need to ensure ethical handling of data becomes paramount. As technology grows in its capabilities, so must our commitment to data privacy and regulatory adherence. The Future of Data Capitalism In conclusion, the narrative of data capitalism emphasizes the importance of data as an economic driver in the modern world. This shift brings immense opportunities for growth and development but also poses significant challenges, particularly in terms of privacy and data protection. As we navigate the terrain of data capitalism, it’s crucial that we continually re-evaluate and reinforce our approach to data privacy and protection. The integration of AI and AGI technologies presents promising Appendix: Additional Perspectives on Data Capitalism According to John Deere, a noted data analyst from Silicon Valley, the emergence of data capitalism signifies a pivotal shift in our understanding of resources. Traditionally, resources were tangible — such as oil, gold, or land. However, the digital age has spotlighted data, an intangible asset, as a resource with immense economic value. Data as a Resource: : Emma Schultz, CEO of European tech firm Blue Chip Technologies, highlights the increasing importance of data sovereignty. As data capitalism grows, the task of protecting data and controlling access becomes increasingly complex. Schultz emphasizes that legal frameworks must progress to safeguard the data rights of individuals and businesses. Data Sovereignty : Joshua Greenbaum, a prominent technology consultant based in San Francisco, discusses the concept of the data economy. He elaborates that in this digital era, corporations can monetize their data by creating innovative products and services. This data economy, as he terms it, is fostering a new category of goods that rely on data as their primary ingredient. The Data Economy : Francois Gossieaux, a Boston-based economist, underscores the vital role that regulatory bodies play in ensuring fair competition in the age of data capitalism. He cautions that regulatory policies must adapt to ensure that data giants do not stifle competition and innovation. The Role of Regulations : Dr. Maya Ackerman, CEO of Wave AI, discusses the importance of ethics in data handling. As AI and AGI become increasingly prevalent in data processing, Ackerman insists that ethical considerations in data handling and analysis are paramount. She suggests that a comprehensive AI governance policy will be critical in guiding the ethical use of AI and AGI. Data Ethics and AI Governance : Tom Wheeler, a former Chairman of the Federal Communications Commission (FCC), draws attention to the growing concerns around data privacy. He asserts that while data capitalism offers immense benefits, it also presents significant risks to privacy. Wheeler calls for a global collaborative effort to address these pressing data privacy issues. Data Privacy In the contemporary digital ecosystem, the market is increasingly dominated by large entities like Facebook, Google, Apple, and Netflix. These companies collect, analyze, and apply data across diverse sectors, often shaping the very industries they operate in. They have strategically positioned themselves in the market by offering innovative products and services, while concurrently leveraging their vast data portfolios to fuel growth and maintain their competitive edge. Also published . here

Data Capitalism: Exploring the Information Ownership Landscape in the Age of AI

Our 25 Favorite Data Science Courses From Harvard To Udemy

As this fracas unraveled, my first reaction was ! People on were losing their mind over the enormity of the data loot. Even my tech-leaning acquaintances were at-least mildly impressed that that data belonging to as many as 30 to 50 users was siphoned off in the guise of academic research and fed into sinister statistical pipelines drenched in malaise to wreck political mischief, put mildly. Cambridge Analytica HOLY-EFFING-SH*T social media million Facebook legally Now, here comes the plot-twist. This was a whole kind of a moment for me. As someone whose was in the field of , I knew for a fact that 50 million was . Academics who worked on the big data facet of the field had routinely amassed and touted far bigger numbers in their publications during my doctoral studies. different HOLY-EFFING-SH*T PhD thesis literally Network Science puny small Given the inherently romantic nature of the field, I still devote some of my Sundays towards keeping myself abreast of the exciting developments in the field. One such rather I recall having read early this year (published in Dec 2017) came to my mind when I saw the . This paper had bragged about figures that will make all those worried about the enormity of the Cambridge Analytica breach dive deep into depression. impressive paper Channel 4 news expose Are you ready for the number? Are ya sure? OK, here it goes: 368 MILLION USERS. In a paper smartly titled ‘ ’, a Swedish-Polish team of merely 4 researchers from 2 humble schools ( Blekinge Institute of Technology and Wrocław University of Science and Technology) had amassed a dataset of interacting in little over ! Do We Really Need to Catch Them All? A New User-Guided Social Media Crawling Method 368 million unique users 1.3 billion social interactions The abstract of the paper But how did a team of 4 pull this off? Combination of 2 ideas: a) Smart ‘attack/seed points’: Facebook pages. b) A nifty crawling engine. The team had built this rather impressive crawler — SINCE- Social Interaction Network Crawling Engine that they used to collect publicly available Facebook data. Now, which 160 pages you ask? It’s all openly available neatly curated and hosted on Harvard’s . dataverse The numbers on their descriptive stats table is strictly orgasmic. Take a look. Now, what if a reader/journalist wants to give this a shot? Sans sinister intentions, purely for intellectual curiosity? Can’t code? No worries. GUI tools to the rescue! If you can spare , you can get access to like the one above that will allow you to indirectly target users based on one rather interesting backdoor: ! $75 tools Facebook fan pages Example: Now, bear in mind that this is not an expose or a witch-hunt against the authors of the fine paper above or their excellent piece of engineering research. For what it’s worth, the authors do vividly state that ‘ ’, and I for one take their word for that. This work is from mainstream reputed academics. We adhere to Facebook’s data privacy policy by anonymizing all data to an extent where it is only possible to backtrack the particular public page that is analyzed do The main point here is that if you are considering state-of-the-art publications in areas such as Graph exploration or even recommendation engines, data belonging to a user-set spanning a few hundred million users is par for the course. Proof? This: So, now what? Does Facebook go through the entire list of academics who were given academic access one by one and flies one their Data engineer and a lawyer to make sure that the data is strongly anonymized? This where I squirm a little. Online social networks by their volition are tricky beasts to anonymize. This is not an unfounded fear. During my Ph.D, I knew of researchers who were actively working on de-anonymization strategies and had met with plenty of successes. Example: Also, believe it or not, we have been here before! In my first year at CMU, I had read Mike Zimmer’s paper on Facebook’s infamous breach of 2008. Because we millennials co-exist co-temporally on several social media platforms and carry forward our social allegiances elsewhere, all it takes is one or a few breaches (See figure below) and we will face cataclysmic ripple effect breakdown of anonymity. Harvard freshman from Wyoming So, perhaps, the only solution is forced deletion from every single academic repository that there is out there? Possibly so.

‘Academic’ honeypot datasets with 368 million users. Why is nobody talking about these!?

For an enterprise to succeed, it’s no longer enough to launch a product that’s “good enough.” Businesses today must provide that are not only performant and highly available but also private and secure. high-quality digital experiences But how do you achieve all of this? One way is for DevSecOps teams to adopt an observability practice across your user experience and threat environments. By logging and analyzing both security and observability data, you can better detect and remediate a host of problems, such as performance issues, vulnerabilities, and security breaches, resulting in a higher-quality experience. that uses logs (and other means) to collect wide swaths of data In this article, we’ll focus on the ins and outs of wide-reaching data collection, and . We’ll look at the difference between observability data and security data, and how best to collect all this data. Then we’ll explore how to use this data to improve your app and finally see how to implement a centralized, one-stop shop for your data collection. how logs in particular can help you What is Observability Data? What is Security Data? Observability is a property of systems that defines what an operator can tell about the state of a system, both currently and historically. Leading from that, is the representation of that system state — think percentage of errors, memory usage, and flow of traffic. Observability data can be raw data recorded from various sources or down-sampled, aggregated, and processed data. observability data How is that different from security data? is the subset of your observability data that is used to discover security threats. If this seems fuzzy, that’s because it is! Security data For example, if an application exhibits slow response times, is it security data? It depends. Maybe the application is simply misconfigured for the current load, or maybe an attacker has compromised the system and is running a workload that has slowed everything down. Even what is typically not considered security data could be. For example, if a new CVE is disclosed for a particular library, you’ll need to know which services depend on that version of the library. And sometimes clear-cut cases of security data — for example, the identity of a caller to some API endpoint — also end up being useful for tracking non-security bugs. In the end, we can say that a piece of observability data might be security data…and it might not. It depends on the specifics of the use case, which often isn’t known until after the fact. The Three Pillars of Observability Data When we look at observability data, we often break it down into three pillars: logs, metrics, and traces. Let’s look at those and how they might apply to both observability and security data. Logs Logs capture time-stamped events and provide a timeline of what happened to the system. For example, your application may emit a log message if it fails to connect to a database, and may write who performed a certain action to an audit log. Any non-trivial system will have many different logs. Recording and organizing these logs in a centralized logging system where they can be sliced and diced are a primary facet of observability data collection. Log analytics data is critical for monitoring, troubleshooting and investigating reliability and security issues, to get down to the root cause of “why” that issue occurred. Log data is often the most detailed information available about a company’s systems, so it makes sense to put that data to work, pulling log files across the organization for end-to-end visibility and faster troubleshooting. Metrics While logs generally contain textual information, metrics are numerical time series — for example, the number of pending pods in your Kubernetes cluster over time. Metrics are typically recorded periodically and can help identify trends and anomalies. Metrics are much more economical than logs. They don’t require parsing and transformation, and they cost less to transmit and store. Metrics are the vital signs of your system. Alerting is primarily built on top of metrics. Traces Distributed tracing associates an identifier with each request. A trace of a request is a collection of spans and references. You can think of a trace as a directed acyclic graph that represents the request’s traversal through the components of a distributed system. Each span records the time the request spent in a given component, and references are the edges of the graph that connect one span to the following spans. Traces are a very important tool for modern systems built on microservices, serverless functions, and queues. It provides a “chain of custody” to the system operators that is crucial for analyzing both performance and security issues. So how do these pillars cross with security data? . Nearly all security information comes from logs. While metrics and traces can help you understand when something is wrong (for example, the CPU is high), you most likely won’t know what is actually happening (in this case, the CPU is high because of a crypto-miner) until you examine the logs How to Collect Data Now that we understand observability and security data, why they are important, and where they come from, let’s look at some more practical aspects: how to collect data and how to use it effectively. Let’s look at each pillar, some best practices, and a few tools that can help. These steps apply to all your data — observability and security. Logs You probably have many logs created by your OS, Kubernetes, and your cloud provider. Your applications and third-party software also generate log files. However, standard logs bring several challenges: Accessing a wide variety of log files across many hosts is inefficient and not user-friendly. Log files will eventually fill the disk (unless you’re using cloud persistent storage). If you’re rotating log files to address the disk space issue, then you lose historical data. Your servers may crash from time to time. Disks can get corrupted. To address these issues, a best practice is to use a log collector agent on every host that ships the logs to a centralized observability platform. The centralized log management system will store the logs in durable storage, provide a query interface, provide aggregations and other transforms, and will retain the log backups as needed for compliance and finding historical trends. To alleviate some of the pain, it’s a good idea to use industry standards and tooling like OpenTelemetry ( ). For data collection specific to logs, open-source tools like and are also popular. https://opentelemetry.io LogStash Fluentd Metrics Metrics are somewhat easier to handle than logs since all metrics from any source can be represented uniformly as a single time series with a set of tags associated with it. Similar to the log collector, a metrics collector can run on every host and periodically read relevant data, such as CPU and memory usage of every process or the number of requests in flight, and send it to the observability platform. Application-level metrics can be collected in two ways: the application may expose a standard interface that the metrics agent can scrape or, alternatively, the application can be instrumented (typically using a library) to send its metrics directly to the observability platform. Data points are captured at fixed intervals and sent to an observability platform for analysis. For metrics, Prometheus is the undisputed king, designed to scrape application endpoints that expose application metrics. Grafana works very well to visualize your metrics. Tracing Traces are collected by instrumenting applications and services to intercept requests and attach an id that follows the request when making calls to other services. The traces are sent to the observability platform for analysis. For distributed tracing, Jaeger (originally developed by Uber) leads the pack. Other projects include Zipkin and Signoz. How is data used for improving application security? How do you use all this data to specifically improve your application security? Let’s consider the role that these pieces of observability data might play in the different aspects of incident detection and response. We begin with , which hinges on capturing metrics to detect abnormal activity. Monitored by an observability platform, these metrics will trigger an alert to the DevSecOps team. incident detection At this point, the team only knows that something abnormal is happening, but it doesn’t know if this is a security-related incident or not. To the incident, the inspection of logs becomes critical. If the incident is security-related, the team can use logs to determine how the malicious actors are accessing the system and further access by revoking access for compromised users or services. identify block Next, the team can use the combination of logs, metrics, and traces to , determining which parts of the system have been impacted and whether data was exfiltrated. By knowing the breadth of impact, the team can by making those components unavailable until they can be fixed. assess breach impact reduce the blast radius Finally, the team can use logs and metrics, specifically when performing a , to help in root cause analysis for all incidents, security or otherwise. The resulting actions may include , , or further. post-mortem tightening access control patching vulnerable systems fine-tuning the observability strategy What does this look like in practice? The elite level of observability comes when you integrate all the various data sources and technologies into a cohesive platform that allows you to gain insights into your system. You can build this yourself using many of the open-source projects already mentioned. While this offers high customization, it requires a lot of effort to build, maintain, and operate, especially if you want a system that is robust, scalable, and always available. Additionally, you also need to observe your observability platform. Another option is to choose a managed platform to handle the observability concerns. This allows you to focus on your core competencies. An example from Black Friday Let’s look at an example around Black Friday. With many sites — especially in retail — experiencing an uptick in user activity, how might leveraging observability data and a platform help mitigate security incidents? I found this about Ulta Beauty using metrics like invalid password attempts and login attempts per IP address and per country, all possible indicators of fraudulent activity. timely post Ulta pumps these metrics into an observability platform from Sumo Logic, which provides a Brute Force Attack dashboard to aid in incident detection. From there, an incident response team can begin analyzing logs to determine and address the root cause of the security incident, just as we walked through above. The platform from Sumo Logic ingests, aggregates, enriches, and retains all of the observability data. Sumo Logic lets you decide if you want to install data collectors locally or have hosted collectors running on AWS. It also integrates with observability staples like Prometheus and OpenTelemetry, which can ease migration efforts and alleviate concerns about vendor lock-in. Conclusion Collecting observability and security data in wide swaths is key to a high-quality digital experience for your users. To succeed, you’ll need to rely on all three pillars of observability: logs, metrics, and distributed tracing. You’ll need to use this data to detect, isolate, and mitigate attacks. In the next post in the series, we will take a deeper look at monitoring, which is the next logical step. Once you have collected all this data, what do you do with it? Have a really great day!

The Everything Guide to Data Collection in DevSecOps

Today’s business climate is one where you only need one opportunity to turn things around. We’ve all seen it repeatedly - several companies never really believed that they had what it takes, but they got one killer idea that changed things for them. In an increasingly opportunistic environment, being able to spot the right chances to take your business to the next level is rare. But, the companies that can recognize opportunities and seize upon them easily become the next big thing. At the moment, one of the most impressive opportunities facing companies today is tech transformation. Organizations are constantly in search of the next wave that can help them become better and improve their operations. And as the past few years have shown, optimizing your tech stack as much as possible can easily be the key to your expansion as a company. This is the reason why Devart is here: the firm has been providing a variety of productivity solutions to Fortune 500 companies, providing them with assistance in enhancing their operations and the communication that occurs within their technology teams. We spoke to Dmytro Nikolaenko, Founder and CEO of Devart, to understand what factors helped the company stay competitive in the exponentially developing technology market for over two decades and how they are preparing for the digital transformation, connected to the arrival of AI and other newest technologies. 1. What is the vision behind Devart? How did you choose the niche for your business? Data is one of the keys to the modern world, no doubt about that. It's a cornerstone of sorts that will never lose any of its relevance. As long as there is software, it has to interact with data. And this is where our solutions come into play. They empower users to manage data with maximum convenience and minimum effort. They provide users with multiple tools that effectively augment their capabilities in daily work. They help reduce manual routine, automate it where possible, and focus on things that really matter. To achieve that, the whole team has to work a lot. We have to evolve our tools non-stop. We have to keep them powerful and intuitive for users of different backgrounds and skill sets. To top it off, we're committed to providing our clients with exceptional, highly responsive support. More or less, this sums up the vision behind Devart. And I was lucky to gather together a team of skilled and talented people who proved capable of making it all work. 2. What is the role of database management in today's fast-paced software development landscape? It's a vital part of that landscape. A database's main functions are information storage, retrieval, and updating. Data about every facet of company operations may be kept in a database, and database administration is essential for creating software that is dependable, strong, and efficient. Imagine that you're in charge of a software product, and you release updates under a standard development cycle. The release of each update is a chance to keep up-to-date with the demands of your customers, to prove the relevance of your product. Basically, it's a chance to sell. Naturally, you want to make the road to each release as short and painless as possible. And when it comes to database development and management, database management tools can offer a true shortcut. For example, dbForge tools for SQL Server can help you unify database development, management, testing, and deployment into a single DevOps cycle, perfectly automated and integrated into your general workflow. And if your workflow changes, these tools are flexible enough to catch up with it. 3. What are the best strategies for the software project teams to adopt to break the cycle of misaligned goals, fragmented tech stack, workflows, or output that come with a siloed team? Proper clarification of goals and well-aligned teamwork is a matter of good management. To avoid teams becoming fixated on increasingly specific, narrow objectives that often lead to misalignment between departments and duplication of efforts, tech leaders should provide a broad, holistic vision for the entire organization to aspire to and set clear priorities. The subject of fragmented tech stacks and workflows is one of the most common problems faced by our clients. In many use cases, a rapidly growing company needed to reshape its database development and management workflows to keep up with the pace of changes. And in each case, our tools helped them make it possible. When performing everyday routine work and coding, users of Devart database administration and development tools consistently report a considerable gain in productivity of 20–30%. We're really proud to be part of their success. 4. How do you plan to accommodate the growing trend for AI, decentralized data, or decentralized database management solutions? The primary incentive for us is the market demand which is why we are focused on the traditional relational databases. This market blooms and doesn't show any signs of decline. In addition, we are very interested in AI utilization in the forthcoming perspective and keep an eye on the rapid developments in this particular area and use cases. As soon as there are proven cases for AI-enabled tools for software or database development, we will expand our offering - our team has enough talent and perseverance to make it happen. Conclusion Tech transformation is one of the most critical parts of business operations in our world. At the end of the day, tech is everywhere - being able to manipulate and control it, while also gaining the right insights from your use of technology, will be one of the areas to focus on as a business looking to grow.

FACET DATA

facetdata.com

-- employees

Since n.d.

COMPANY RANKING

FACET DATA

EVERGREEN INDEX #676

Representing this company? Click here to claim and customize this page!

Facet Data WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

FACET DATA

facetdata.com

-- employees

Since n.d.

COMPANY RANKING

FACET DATA

EVERGREEN INDEX #676

RELATED COMPANIESRANK

STORIES ABOUT FACET DATA

Representing this company? Click here to claim and customize this page!

Facet Data WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!