Clics Store

clicsstore

Why Embedded Analytics Matters: Unleashing Data Insights within Applications is becoming an indispensable capability for modern SaaS applications across industries. By embedding analytics directly into applications, insights can guide internal application users and external customers to enable better and faster decision-making. A strong embedded analytics solution that SaaS companies can benefit from starts with the data layer. Many SaaS companies try to determine the best database for their SaaS solution and quite often it becomes an AWS Redshift vs Snowflake comparison. Embedded analytics Exporting data to external business intelligence tools for analysis is becoming less common. Leading organizations are realizing the competitive advantage and of using live data within their apps, so choosing the right database matters. monetization opportunities Data Warehousing: The Engine Powering Embedded Analytics To enable real-time and/or multi-tenant embedded analytics, applications need a high-performance data warehousing layer that can efficiently process queries and serve up data analysis. The data warehouse organizes and stores data from various sources specifically for use cases that span reporting, data visualization, dashboards, and analytics applications. Choosing the right data warehouse is therefore critical. Choosing the Right Tool: Redshift vs Snowflake Two leading cloud data warehouse contenders that show great promise for embedded use cases are AWS Redshift and Snowflake. Both platforms offer advantages such as scalability and flexibility which suit them well for embedded analytics. We compare the two options across crucial criteria to determine which choice best meets embedded needs. Redshift vs Snowflake: Comparing Strengths and Weaknesses AWS Redshift AWS Redshift is a fully managed, petabyte-scale data warehousing service provided by (AWS). It is a cloud-based, massively parallel processing (MPP) database optimized for analytical and reporting workloads. This makes it useful for powering dashboards, ad-hoc queries, and data warehousing. Amazon Web Services Redshift provides fast query performance by using columnar storage and parallel processing to quickly analyze large datasets using multiple nodes. Many enterprises rely on Redshift given its ability to handle heavy analytics workloads. To manage those larger workloads, Redshift can scale storage and compute capacity independently. This offers you the flexibility to pay only for what you need. Scalability and Performance: Brute Force Meets Efficiency with Redshift A pioneer in cloud data warehousing, Redshift offers fast query performance leveraging a massively parallel processing (MPP) architecture optimized for high throughput analytics workloads. Redshift allows scaling compute and storage separately on demand, automatically distributing data across nodes. Performance remains high even with ultra-large datasets and complex queries. Users have reported 50-100x faster queries near the petabyte scale. Cost-Effectiveness: Pay-as-You-Go vs Predictability As part of AWS, Redshift offers pay-as-you-go pricing allowing optimization of costs based on current needs. However, costs can vary significantly based on changing query volumes, underlying data sizes, and other factors – making longer-term budgets and forecasts difficult. Cost optimization requires continual fine-tuning of Redshift clusters and workload monitoring. For embedded analytics specifically, this cost model requires careful management as SaaS usage is meant to grow over time. Deployment and Management: The AWS Ecosystem Advantage Being natively part of AWS, Redshift enables deployment leveraging other AWS services for storage, ETL, monitoring, and more. Companies already using AWS experience less management overhead as a result. But reliance on AWS also leads to vendor lock-in – migrating to other platforms would require significant re-architecture. User-friendliness: Is Redshift Beginner-Friendly? Redshift exposes a standard SQL interface for executing queries. However optimal configuration and cost management require deeper expertise in areas like cluster sizing, workload management, and query optimization. The platform may present a learning curve for beginners. Snowflake Snowflake is a cloud-based data warehousing service that offers a unique architecture optimized for scalability, flexibility, and performance in the cloud. It utilizes a multi-cluster, shared data architecture to efficiently separate storage and computing. This allows independent scaling of resources to match workload demands. Snowflake also has native support for public clouds AWS, Azure, and GCP cloud platforms. The decoupled storage/compute architecture can auto-scale clusters and warehouse capacity based on query volumes and data sizes. This provides high concurrency and performance, similar to Redshift. Snowflake uses a SQL database engine optimized for data warehousing workloads such as analytics, dashboards, reporting, etc. Elastic Power: Scale on Demand, Pay for What You Use with Snowflake Snowflake pioneered a unique cloud-native architecture optimized for flexibility and scalability. The decoupled storage and compute allow auto-scaling to handle extreme workloads without overload. Snowflake also offers per-second pricing – pay only for the capacity used per query without paying for idle clusters. This has similar concerns to Redshift for embedded analytics use cases. As SaaS usage increases, companies realize that usage remains consistent throughout the day, contrary to their initial expectations. These cost increases present challenges for using Snowflake with embedded analytics. Cloud Agnostic Freedom: Beyond the AWS Walls A multi-cloud and hybrid cloud option, Snowflake avoids vendor lock-in by deploying across AWS, Azure, and GCP. Snowflake offers easy migration between clouds with push-button cloud failover capabilities. Snowflake also offers flexibility to query data in external stores without copying across the warehouse. Rich Data Ecosystem: Seamless Integration and Collaboration Snowflake is a strong hub for sharing and exchanging data. It helps teams, partners, and other stakeholders access and collaborate on data easily. Snowflake also offers extensive compatibility with third-party tools. Future-Proof Innovation: Embracing the Evolution of Analytics With rapid innovation across query processing, security, compliance, and machine learning capabilities, Snowflake is leading the way in cutting-edge features for modern internal analytics. Their unique architecture choices make it easy to evolve the platform over time. Organizations can benefit from new capabilities without migrations. Embedded Analytics: Where Redshift and Snowflake Shine (and Stumble) Real-Time Insights: Delivering Data at the Speed of Thought to SaaS Users Embedded analytics requires querying and aggregating live, real-time data with minimal latency to drive contextual insights and guided action within apps. Both Redshift and Snowflake leverage MPP architectures to enable speedy analysis across large datasets. Slight advantages go to Snowflake for its adaptive elastic scaling and per-second pricing which optimizes costs for spiky query workloads common in real-time dashboards and applications. Simplicity and Integration: Seamless Embedding for User Delight For delightful embedded experiences, analytics components need easy integration and simple configuration within applications built using various programming languages, frameworks and platforms. Both data warehouses offer standard JDBC/ODBC connectivity for executing SQL queries from within apps. Redshift may have quicker learning curves for current AWS application teams. But Snowflake offers SDKs for more turnkey embedding across diverse tech stacks. Security and Compliance: Building Trust with Embedded Data Embedded analytics puts live data directly into apps, so security and controls are paramount. Both Snowflake and Redshift enable enterprise-grade user access controls, encryption and data governance capabilities leveraging the underlying cloud infrastructures. For highly regulated industries, Snowflake offers additional native capabilities to track data usage, mask sensitive data and implement fine-grained access policies. Big Data Challenges of Redshift vs Snowflake: When Volume and Variety Demand More As use cases expand to big data sources like , clickstreams or genomics data, the volume, velocity and variety of data can push conventional systems over the edge. Ingesting semi-structured data like JSON events get tricky. (Although ) IoT analytics Qrvey handles all data natively Serverless options on Snowflake like Snowpark handle varied data with less friction. Handling data volumes above 100s of TB can stretch Redshift capabilities. At massive scales, Snowflake better absorbs extreme spikes in storage and concurrent users. Picking the Champion for Your Use Case in This Redshift vs Snowflake Decision Cost Considerations: Balancing Budget and Performance AWS Redshift follows typical cloud pay-as-you-go pricing with node-based commitments. Cost efficiencies kick in at higher scales above a few TB. Snowflake’s per-second pricing and adaptive scaling remove overhead for idle clusters. But per-second billing can also lead to unexpected spikes on shared systems with uneven workloads. Cross-cloud deployment, data sharing and BYOL options on Snowflake provide more levers for optimization. Read more about or try our . Snowflake cost optimization Snowflake Cost Optimization Calculator Technical Requirements: Matching Capabilities to Needs Redshift provides a tightly coupled solution with quick time-to-value for simpler analytics integrated into AWS-centric application environments. More complex use cases like large-scale machine learning, and hybrid transactional/analytical processing may benefit from Snowflake’s more advanced architecture. Snowflake better fulfills needs for multi-cloud flexibility or rich data-sharing ecosystems. Choosing a Platform to Grow With: Redshift vs Snowflake Snowflake’s platform is cloud-based offering fast innovation in security, compliance, data science, and governance. This makes it an ideal solution for the long term…assuming costs are kept in check. The underlying separation of storage and computing eases future migrations. Future-proofing for unforeseen changes favors Snowflake, but Redshift is still likely a good option. Redshift vs Snowflake: Collaboration and Hybrid Solutions The data warehousing landscape continues to evolve rapidly, with the boundaries between Redshift, Snowflake and other platforms becoming more porous over time. Rather than a winner-take-all dynamic, we see increasing convergence and collaboration between platforms. Many organizations leverage hybrid solutions with Redshift for high-intensity operational workloads integrated with Snowflake for larger-scale data science experiments. Connectors like the recently launched make interoperation easier. AWS Redshift integration for Snowflake As analytics use cases grow more sophisticated, matching the ideal platform to each specific embedded scenario will unlock more value than a one-size-fits-all choice. Takeaway: Embracing the Right Data Warehouse for Your Embedded Analytics Journey The data warehousing engine powering embedded analytics should align with technical requirements, cost constraints, and future ambitions. Both AWS Redshift and Snowflake bring unique strengths as the foundation for real-time data applications. How Qrvey is Different At Qrvey, we know that a strong data layer is the foundation that makes any embedded analytics solution successful. We are the only solution with a built-in data warehouse layer made for multi-tenant, security-first embedded analytics. However, did you know that while we connect with Redshift, Snowflake, PostGres and more, we know don’t use any of these for our native data warehouse? Discover to power our embedded analytics for SaaS applications solution. why we chose AWS OpenSearch Also published here.

AWS Redshift vs Snowflake: A Comprehensive Guide to Embedded Analytics Solutions

AWS রেডশিফ্ট বনাম স্নোফ্লেক: এম্বেডেড অ্যানালিটিক্স সলিউশনের জন্য একটি ব্যাপক গাইড

AWS Redshift vs. Snowflake: Ein umfassender Leitfaden für eingebettete Analyselösungen

AWS Redshift vs Snowflake: una guía completa para soluciones de análisis integradas

AWS Redshift vs Snowflake : un guide complet des solutions d'analyse embarquées

एडब्ल्यूएस रेडशिफ्ट बनाम स्नोफ्लेक: एंबेडेड एनालिटिक्स सॉल्यूशंस के लिए एक व्यापक गाइड

AWS Redshift と Snowflake: 組み込み分析ソリューションの包括的なガイド

AWS Redshift와 Snowflake 비교: 내장형 분석 솔루션에 대한 종합 가이드

AWS Redshift vs Snowflake: um guia abrangente para soluções analíticas incorporadas

AWS Redshift против Snowflake: комплексное руководство по решениям для встроенной аналитики

AWS Redshift ve Snowflake Karşılaştırması: Yerleşik Analitik Çözümlerine Yönelik Kapsamlı Bir Kılavuz

AWS Redshift vs Snowflake: Hướng dẫn toàn diện về giải pháp phân tích nhúng

AWS Redshift 与 Snowflake：嵌入式分析解决方案综合指南

Some of you might remember the article I wrote last time: , it was extremely productive, and even though the like vs dislike proportion favors the latter many of you provided valuable insight into how to improve berrynews.org SEO ranking and Domain Authority, thank you for that. SEO is DEAD ❌ Based on your replies I've compiled a list of how to optimize your SEO and although many had valid points, a platform that aggregates news articles from various sources would be close to impossible to improve, I want to consider it as a challenge🦾 Answers to my 'What actions should I take?' question This brings me to the first actionable improvement point: Understand that SEO takes time. Two weeks is typically not long enough to see significant results, especially for keywords with high search volumes. I'll post an update once more time has passed and I've had the time to develop the SEO. Patience is key: Target relevant search queries. Being on the first page for low-volume searches won't drive much traffic. Aim for higher-volume, relevant keywords. Keyword optimization is an important factor and you have built-in tools in Google Search and Bing Search where you can find out what would work best for your website and update the metadata accordingly. Focus on relevant keywords: Optimize beyond just page speed. Focus on creating great content, building domain authority, acquiring quality backlinks, and setting up technical aspects like metadata and sitemaps properly. On the topic of backlinks, I will explain more later in this article. SEO is a comprehensive strategy: SEO in highly competitive sectors, like news, demands more work and strategic planning. Truer words have never been spoken, as I said challenge ACCEPTED🫡 Competitive niches require more effort: Dedicate time to understanding and implementing SEO best practices, including meta tags, content writing, and backlink strategies. Results will show over time. I've just finished reading "The Beginner's Guide to SEO" by MOZ and now moved to "Search Engine Optimization (SEO) Starter Guide" from Google Search. Invest time in SEO: Adjust your expectations for SEO outcomes. Initial traffic might be low, and growth is typically incremental. Consistent effort is required to see improvements and we shall see this in a couple of months when I post the update Set realistic expectations: Plan for SEO as a long-term strategy, giving it a few months to start showing results. Early expectations for quick gains are often unrealistic. This was one extremely valid but hard-to-achieve bullet point that I am working on. Long-term strategy: These points emphasize the importance of patience, relevant content targeting, comprehensive SEO strategies, and setting realistic expectations in the competitive landscape of online content. tl;dr: How exactly do you improve SEO? 🧘 if you want the full-length version you can find it . I will try as best as possible to answer these questions for myself, but they are a blueprint for any SEO improvements. Keyword research: here Ask what your users want. Ad-free news Time efficiency: A single platform to host them all Multiple sources and languages Unfiltered content What are your users searching for? TBD How often are your users searching the terms? The top 10 is unachievable but the top 100 is more than better What is your competition doing? Keywords by competitor Keywords by season Is it Easter already? 🐇🥚 Keywords by region (Bonus) What type of search types do we have? Informational queries - how fall is Brad Pitt Navigational queries - just Google search "berry news" if you don't know the full link Transactional queries: buy a concert ticket Commercial research Local queries: best oliebollen in Den Haag? Tools: - (PAID) The tool extracts accurate search volume data by using live clickstream data. Moz Keyword Explorer - (FREE) Google's AdWords Keyword Planner has historically been the most common starting point for SEO keyword research. Google Keyword Planner - Google’s keyword trend tool is great for finding seasonal keyword fluctuations. Google Trends (PAID) - This free tool populates commonly searched-for questions around a specific keyword. AnswerThePublic (PAID) - Provides some really neat competitive keyword data. SpyFu Keyword Research Tool as backlinks (links from other websites to yours) are a significant ranking factor for search engines. Developing an effective backlink strategy: Backlink strategy is an essential component of SEO, Understanding backlinks Quality over quantity: One high-quality backlink from a reputable site is worth more than many low-quality links from dubious sources. Relevance: Backlinks from sites within your niche or industry are more beneficial as they are seen as more relevant. Diverse Link Profile: A natural backlink profile has links from various sources and types (e.g., blogs, news sites, educational sites) Content creation Valuable content: Create high-quality, informative, and engaging content that others naturally want to link to. Shareable assets: Infographics, research studies, comprehensive guides, and thought leadership articles are examples of content that typically attract backlinks. As recently "announced" berrynews Europe and Spain "article share feature" Outreach and networking Blogger outreach: Reach out to bloggers or influencers in your industry for guest posting opportunities or to share your content. Relationship building: Develop relationships with industry leaders and content creators. Networking can lead to organic backlinks. Utilize social media and forums Share your content on social media platforms and relevant online communities. While most social media links are nofollow, they can lead to exposure and natural backlinking. Broken link building Identify broken links on other websites and suggest your content as a replacement. To be honest, this one I didn't properly understand. Guest blogging Write articles for other websites in your industry. Include a link back to your website within the content or in your author bio. If anyone wants to connect you can find me at @email Leverage PR and publicity Utilize press releases or reach out to news outlets for coverage of significant events, studies, or announcements from your company. Monitor your backlink profile Use tools like , , or to monitor your backlinks. Keep an eye out for spammy links that might harm your site’s reputation. Ahrefs SEMrush Moz Competitor analysis Analyze the backlink profiles of your competitors to understand where they are getting their links from. This can provide ideas for your own strategy Be Active in industry forums and blogs Participate in discussions on industry forums and comment on relevant blogs. Make sure your contributions are meaningful; this can lead to natural backlink opportunities. Directories and listings Get listed in reputable directories relevant to your industry. Avoid low-quality directories as they can be harmful. Best practices and caution Avoid black-hat techniques: Buying links, participating in link schemes, or excessive link exchanges can lead to penalties from search engines. Natural growth: Build your backlinks gradually and naturally. A sudden spike in backlinks can appear manipulative to search engines. Focus on value: Always focus on providing value through your content and interactions. High-quality content that serves the user's needs is more likely to earn backlinks. Developing a strong backlink profile takes time and effort, but it's a vital part of a comprehensive SEO strategy. Remember, the goal is to build a natural-looking backlink profile that grows organically over time. This brings my article to an end. If you are not familiar with some of these terms, I suggest that you check out the . And this funny message Glossary As always, thank you stranger for your time and if you want to "make acquaintance" you can always subscribe to my Hackernoon account. Until next time much love from Den Haag 🫶

SEO is not DEAD ✅

SEO মৃত নয় ✅

SEO ist nicht TOT ✅

El SEO no está MUERTO ✅

Le référencement n'est pas MORT ✅

SEO ख़त्म नहीं हुआ है ✅

SEOは死んでいない ✅

SEO는 죽지 않았습니다 ✅

SEO não está MORTO ✅

SEO не МЕРТВО ✅

SEO ÖLÜ değil ✅

SEO không CHẾT ✅

SEO 还没有死 ✅

Today’s business requires accurate and timely data in order to make decisions. Data-driven applications and dashboards built on time series data are found in every enterprise. A great deal of business data is time series - from system and application metrics to weather, stock prices, network telemetry, IoT sensors, all rely on complex time series analysis and visualization to provide business insight. There are more than a few databases that are built for this kind of analysis, such as Clickhouse (for more on Clickhouse and MinIO see ), Snowflake, Amazon’s TimeStream and RedShift, and Apache Druid. A typical workflow involves gathering data, likely being streamed in real time or pulled from a data lake, transforming and ingesting it into a database, and then building applications and dashboards using SQL queries running against the database. Integrating ClickHouse with MinIO This blog post focuses on , specifically how to run Druid with MinIO as deep storage. We’ll go into more detail later, but in short Druid saves data to S3 deep storage and swaps it in and out of local memory as needed when running queries. We’re going to install MinIO, install Druid, edit some configuration files to tell Druid to use MinIO, and then import demo data and visualize it using . Apache Druid Apache Superset Prepare to be amazed at how quickly you can build a high-performance analytics and visualization stack with open source products. Druid Overview Druid is one of the fastest data stores around - it was built to combine the advantages of data warehouses, time series databases and search systems. Druid achieves tremendously speedy query responses by combining the key characteristics of these three systems into its ingestion layer, storage format, query layer and core architecture. Druid is typically deployed with other open source technologies such as Apache Kafka, Apache Flink and sits between a storage or processing layer and the end user, serving as a query layer for analytic workloads. To learn more about how Druid works, please see . Apache Druid 101 Druid is cloud-native and relies on a microservice-based architecture. Each core service in Druid (ingestion, query, coordination) can be deployed and scaled independently in containers or on commodity hardware. In addition, Druid services can independently fail and restart or be updated without impacting other services. Druid has several process types: processes manage data availability on the cluster. Coordinator processes control the assignment of data ingestion workloads. Overlord processes handle queries from external clients. Broker processes are optional; they route requests to Brokers, Coordinators, and Overlords. Router processes store queryable data. Historical processes ingest data. MiddleManager These processes are typically deployed in groups onto three different server or node types: : Runs Coordinator and Overlord processes, manages data availability and ingestion. Master : Runs Broker and optional Router processes, handles queries from external clients. Query : Runs Historical and MiddleManager processes, executes ingestion workloads and stores all queryable data. Data Druid has three major external dependencies: deep storage, metadata storage and ZooKeeper. Briefly, metadata is used to track segment usage and task information, and is usually handled by PostgreSQL or MySQL. Zookeeper is used for internal service discovery and coordination. As this is the MinIO blog, we’re going to focus on deep storage. Deep storage is a shared file system accessible by every Druid server where data is stored after it is ingested. In a single-server deployment this is simply a local drive. In clustered deployments, deep storage would take the form of a distributed object store like S3 or MinIO, HDFS or a network mounted filesystem. is how Druid transfers data between processes and backs up data. Druid stores data in files called segments. There’s an internal mapping that pulls segments from deep storage to cache them on local drives and in memory. In order to minimize latency, queries run against segments locally, not against segments stored in deep storage. Deep storage enables Druid’s elastic, fault tolerant design. Druid provides durability by bootstrapping from deep storage in the event that a data node goes down and comes back up. One caveat is that you’ll need enough free space on data nodes and in deep storage for the data you want to work with in Druid. Deep storage Druid uses the S3 API for object storage for deep storage. Using S3 API compatible object storage such as MinIO gives you the freedom to run Druid anywhere, which is something you might want to do to improve performance and security. MinIO is a superb deep storage for Druid because it has excellent S3 compatibility, the performance to meet requirements for real time OLAP and, perhaps most of all, the ability to deliver performance at scale. MinIO also provides necessary and . data protection lifecycle management You can run Druid, MinIO and Superset directly on Linux or in containers on Docker or Kubernetes. There are already some great resources out there that show you how to get Druid up and running, so if for some reason this blog post doesn’t satisfy you, check out for Linux, for Docker and for Kubernetes. Apache Druid Quickstart Docker · Apache Druid Clustered Apache Druid® on your Laptop Installing MinIO If you’re not already running MinIO, here are . The process is described thoroughly in . installation instructions The MinIO Quickstart Guide On Linux, for example, download MinIO and run it (feel free to change the root password - and remember it): wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
MINIO_ROOT_USER=admin MINIO_ROOT_PASSWORD=password ./minio server /mnt/data --console-address ":9001" Launch a browser and connect to MinIO Server at using the root user and password you configured earlier. http://<your-IP-address>:9000 The first time you log in, you’ll be taken to the Buckets page. Create the bucket . druidbucket In the real world, you would probably want to secure access to your Druid data, but we don’t have to do that in our lab. You would create a service account (under the Identity menu), edit the user policy to only allow access to and in the Druid configuration below use the service account’s access key and secret key. druidbucket Installing Druid The easiest way to test drive Druid is via Docker. Follow the instructions  and use this . This creates a container for each Druid service, as well as Zookeeper and PostgresSQL. This method uses an to specify the Druid configuration. Download to where you’ve saved Druid’s containers. For the instructions below (Configure Druid for MinIO), you can edit the environment file to pass the variables to standard files or edit the configuration files directly within the container. Docker · Apache Druid docker-compose.yaml environment file this file Druid configuration Start Druid with . Once the cluster comes up, open a browser and navigate to to use the Druid console. docker-compose up http://localhost:8888 Configure Druid for MinIO Now it’s time to configure Druid to use MinIO for deep storage. In this step, you will need to edit Druid config files (or the Docker environment files as described above) to reference MinIO and then restart Druid. The process is fully described in and we’re including an abbreviated version here for quick reference. How to Configure Druid to Use Minio as Deep Storage - DZone Big Data Depending on how you’ve installed Druid, In the file, add to . Add extensions by name, in quotes and separated by a comma, for example: conf/druid/_common/common.runtime.properties "druid-s3-extensions" druid.extensions.loadList druid.extensions.loadList=["druid-parser-route", "druid-s3-extensions"] According to the above referenced article, the S3 extension for deep storage relies on jets3t under the hood. You need to create a file on the class path. For this example, we’ll create a new inside the directory with the following: jets3t.properties jets3t.properties conf/druid/_common s3service.s3-endpoint=localhost
s3service.s3-endpoint-http-port=9000
s3service.disable-dns-buckets=true
s3service.https-only=false Now, comment out the configurations for local storage under the “Deep Storage” section and add appropriate values for Minio. In my case, I’m running MinIO on the same machine as Druid so the S3 Endpoint URL includes localhost and a port. If you’re running on another machine or in a container, you’ll need to change this line to address MinIO in your environment. After this, the “Deep Storage” section should look like: #
# Deep storage
#
# For local disk (only viable in a cluster if this is a network mount):
# druid.storage.type=local
# druid.storage.storageDirectory=var/druid/segments
# For HDFS:
# druid.storage.type=hdfs
# druid.storage.storageDirectory=/druid/segments
# For S3:
druid.storage.type=s3
druid.storage.bucket=druidbucket
druid.storage.baseKey=druid/segments
druid.s3.accessKey= <insert your MinIO accessKey>
druid.s3.secretKey= <insert your MinIO secretKey>
druid.s3.protocol=http
druid.s3.enabePathStyleAccess=true
druid.s3.endpoint.signingRegion=us-east-1
druid.s3.endpoint.url=http://localhost:9000/ We’re also going to configure Druid to store indexing service logs in Minio. It is a good practice to store everything in one place to make it easier to find - Druid is complicated enough without having to search multiple locations for logs. Update the “Indexing service logs” section with appropriate values in . conf/druid/_common/common.runtime.properties After this, the “Indexing service logs” section should look like: # Indexing service logs
# For local disk (only viable in a cluster if this is a network mount):
# druid.indexer.logs.type=file
# druid.indexer.logs.directory=var/druid/indexing-logs
# For HDFS:
# druid.indexer.logs.type=hdfs
# druid.indexer.logs.directory=/druid/indexing-logs
# For S3:
druid.indexer.logs.type=s3
druid.indexer.logs.s3Bucket=druidbucket
druid.indexer.logs.s3Prefix=druid/indexing-logs OK! That covers all of the configuration changes to make to Druid. Save the file and restart your cluster. As you work with Druid, you will see segments and indices written to . druidbucket Installing Apache Superset Apache Superset is a powerful visualization tool that is frequently used with Druid. Superset is a cloud-native business intelligence application that is fast, lightweight, feature-packed, and, best of all, it is free! Superset’s intuitive interface makes crafting interactive dashboards relatively easy. Superset provides out-of-the-box support for most SQL databases. Superset integrates with authentication backends such as the database, OpenID, LDAP and OAuth to secure visualizations. As cloud-native software, Superset is highly scalable and highly available. It was designed to scale out to large, distributed environments. We’re more interested in getting started with Superset than we are with scaling it in production, so we’re going to do the simplest installation we can using . This is the fastest way to try out Superset on Linux or Mac OSX. Docker Compose If you don’t already have Docker installed, please follow . Once you’ve installed Docker, please . Docker’s instructions install Docker Compose Open a terminal window and clone the Superset repo: $ git clone https://github.com/apache/superset.git This will create a superset folder in your current directory. Navigate to that directory and run the following commands (which may take several minutes to complete): $ docker-compose -f docker-compose-non-dev.yml pull
$ docker-compose -f docker-compose-non-dev.yml up You’ll see a whole lot of initialization text pass by on your terminal. Superset is ready once you see something like this: superset_app            | Loaded your LOCAL configuration at [/app/docker/pythonpath_dev/superset_config.py]
superset_app            | 127.0.0.1 - - [12/May/2022:23:59:35 +0000] "GET /health HTTP/1.1" 200 2 "-" "curl/7.74.0" Open a browser and navigate to to access Superset. Log in with the default username “admin” and password “admin”. http://localhost:8088 After logging in to Superset, you must connect it to Druid. On the top right, click the “+” to add a database. From the dropdown list select Apache Druid. Then name the connection and enter the following under SQLALCHEMY URI, making sure to update the IP address or replace it with localhost as necessary. If we had protected the Druid API by requiring authentication, we would also enter a username and password. druid://10.0.0.10:8888/druid/v2/sql Test the connection to make sure it works, and then click Finish. Woo-hoo! You now have Druid, MinIO and Superset running in your environment. With the triumvirate of Druid, SuperSet and MinIO, we can have a complete and timely view of real-time data. There are numerous organizations that have built web-scale data-drive applications on top of this combination for clickstream analytics (web and mobile analytics), risk/fraud analysis, network telemetry analytics (network performance monitoring), supply chain analytics (manufacturing metrics) and application performance metrics. This cloud-native analytics and visualization stack is flexible and extensible, and is scalable and performant so you can build applications that delight your users. Give them the power to monitor, analyze and visualize real-time data. Let them slice and dice, filter and group by any combination of columns so they can make conclusive action-oriented data-driven decisions at the pace business demands. Putting it All Together In order to demonstrate a bit of the functionality we’ve built, we’ll load a tutorial data file of Wikipedia edits over time into Druid and then visualize the edits in SuperSet. Remember, this is backed by MinIO as deep storage. The Wikipedia edits are covered in more detail in . Quickstart · Apache Druid Ingesting into Druid Druid loads data by submitting an ingestion task spec in JSON to the Overlord server. You can build an ingestion spec using the web GUI or write it yourself and schedule it. Once you build and tune an ingestion spec, you can reuse the JSON so you don't have to repeat the task. In production, teams frequently schedule and automate ingestion. Open a browser and load the Druid web interface at . http://localhost:8888 At the top of the window, click Load Data. From here, select Example Data, then select Wikipedia Edits from the drop-down on the right and click the blue Load example button. The web GUI walks you through building an ingestion spec that will read data from a source and store it in segments on deep storage. Ingestion is based on a time stamp that is used to partition and sort data, while data is stored as columns of dimensions and metrics. For more information about how to optimize for performance on ingestion, please see . Data Modeling and Query Performance in Apache Druid With the tutorial data, we’ll simply click through the ingestion workflow accepting the defaults, or click Edit JSON spec and then click Submit. If you wanted (or needed) to, you could look at the raw file as JSON and parse it. Druid creates its own time column called to organize and segment data. Because time is so important to Druid, it’s very good at figuring out how to handle timestamps regardless of their format. In addition, we can transform data, and optimize for query performance by partitioning data and sizing segments. Finally, you can tune the ingestion job in terms of threads and memory utilization, then schedule it and run it. __time The GUI sends you to view Tasks: This is a very small ingestion job so it finishes almost immediately. From here, you could run queries against the data in Druid and get lightning-fast responses. You can run queries via the web GUI or the API. Druid finds itself behind many a dashboard because of its ability to provide excellent query performance over massive amounts of data at scale. Visualizing in Superset Return to the browser where you’ve logged into Superset, and once again click the “+” on the top right of the home screen. Then click Chart. We could run individual SQL queries at the bottom of the screen. Dashboards are groups of individual queries. Responses to repetitive queries are cached in Superset to improve performance. Select the dataset and then select a visualization. Superset includes a number of time-series visualizations. We’re going to choose Time-series Chart and then click New Chart. druid.wikipedia Initially, you won’t see any results, but don’t let that fool you. First, remove the Time Range by setting it to “no filter”. The Time Grain is set too high at “day”, so change that to “5 minute”. We’re now viewing a chart of the number of Wikipedia edits happening in 5 minute windows in our sample data. You can filter and group results using Superset. To save the chart, click Save at the top of the screen, then give the chart a name and save it. By now you’ve learned the basic steps to install Druid and MinIO, ingest your first data set and generate your first Superset chart. A helpful next step could be to learn more about . Exploring Data in Superset Massively Scalable Analytics In this blog post, we introduced a group of cloud-native applications: Apache Druid for analytics, Apache Superset for visualization and MinIO for S3 deep storage. This simple demo is the start of building a cloud-native analytics and visualization stack, and can be extended with Apache Spark for ML or Jupyter for data science. Thanks for reading through this demo. If you have any questions, please send us an email at hello@min.io, or join the and ask away. MinIO slack channel Also published . here

Integrating Apache Druid and Apache Superset with MinIO

Integración de Apache Druid y Apache Superset con MinIO

Intégration d'Apache Druid et d'Apache Superset avec MinIO

अपाचे ड्र्यूड और अपाचे सुपरसेट को मिनिओ के साथ एकीकृत करना

Apache Druid および Apache Superset と MinIO の統合

Integrando Apache Druid e Apache Superset com MinIO

Tích hợp Apache Druid và Apache Superset với MinIO

将 Apache Druid 和 Apache Superset 与 MinIO 集成

11 Real-World Applications of Machine Learning in Ecommerce 

11 aplicaciones del mundo real del aprendizaje automático en el comercio electrónico

11 applications réelles de l'apprentissage automatique dans le commerce électronique

ईकॉमर्स में मशीन लर्निंग के 11 वास्तविक-विश्व अनुप्रयोग

電子商取引における機械学習の 11 の実世界の応用

11 aplicações do mundo real de aprendizado de máquina no comércio eletrônico

11 ứng dụng thực tế của Machine Learning trong thương mại điện tử

机器学习在电子商务中的 11 个实际应用

Enterprise data solutions can quickly become expensive. reports that organizations see 30% data growth every 12 months and risk seeing their solutions fail if this growth isn’t managed. NetApp echoes the concerns, stating that data solutions costs tend to be an afterthought, not addressed until they’re already a problem. Gartner Being a data-driven organization, you already know this and understand the critical need for cost optimization in your data solutions. So what can you do? In this article, I’ll look at the factors that increase costs, what you can do to contain them, and how stream processing can be a major source of savings. Through it all, I’ll help you create an effective plan that leads to operational efficiencies in data solutions. Why Are Data Solutions Expensive? First, let’s look at the factors that contribute to the increasing costs of data solutions. You probably already face many, if not all, of these situations: : Data is being generated at an accelerated rate. And the cost of storing and managing that data has also increased. Increasing volume of data : Disparate data sources make data integration a herculean task that leads to increased data quality checks and management costs. A myriad of data sources and their associated complexity : Inevitable technological advancements lead to frequent upgrades in hardware and software, resulting in higher costs. Frequently updated technology : Not only do we have to worry about how to store and manage data, but also how to protect it from bad actors. Data privacy is critical, demanding hefty investments in encryption, security, and maintenance. Data security and privacy concerns : Many organizations hoard a tremendous amount of data before consulting their data practitioners and experts. This leads to unnecessary storage costs, incompatible systems, and inefficient and unscalable solutions. Data consolidation : According to , the big data market will grow to $229.4 billion by 2025. With that growth comes the need for specialized—and expensive—skills, such as Apache Kafka, Apache Flink, Hadoop, Docker, containerization, DevOps, and others. Lack of skilled data professionals Bloomberg How Can Enterprises Reduce Their Data Solution Costs? What can enterprises do to keep these costs down? There are many strategies, including: Data compression and archiving (reduce data size and archive unused data) Data partitioning (make it easier and faster to process data) Data caching (keep frequently used data in high-speed storage) On-demand autoscaling Data governance measures (ensure accurate, complete, and well-managed data) Efficient data movement (improve routing from cloud and on-prem in order to speed insights) Cloud-agnostic deployment (reduce vendor lock-in, optimize costs) Using a multi-cloud approach Let’s add one more to this list, the solution that we’ll look at in more detail in this article: . stream processing What Is Stream Processing? Stream processing, as the name suggests, is a data management strategy that involves ingesting (and acting on) continuous data streams—such as the user’s clickstream journey, sensor data, or sentiment from a social media feed—in real time. By using stream processing, organizations can and see an over common frameworks for managing data, such as Splunk, Apache Spark, and others. greatly reduce their data solutions costs increased ROI Stream processing solves many of the issues encountered with other data solutions, including rising costs, by giving you: : You handle the data in real time. Reduced storage costs : Stream processing systems can handle large-scale data using a distributed computing architecture. Real-time scalability : Stream processing enables real-time data validation, transformation, and filtering at the source. Preventing the downstream flow of bad data : Real-time analysis gives you real-time analytics—and real-time insights. Faster decision-making to identify high-impact opportunities : Automated workflows in stream processing platforms reduce complexity by abstracting the low-level details of data processing. This enables your data team to focus on data analytics, visualization, and reporting. Team efficiency through automation Manage Stream Processing Effectively Stream processing is powerful and opens many use cases, but managing the implementation is no easy task! Let’s look at some challenges associated with implementing stream processing from scratch. Storage and Computing Resources Anomalies in streaming data can be difficult and time-consuming to detect and address without specialized tools. Continuous Delivery Deploying updates to streaming applications can be challenging without disrupting the production version of the application. Database Integration Integrating data from different databases can be complex and time-consuming for developers. Handling schema management and scalability frequently becomes cumbersome for developers when integrating databases. Migrating data from one database to another can be a costly and time-consuming exercise, requiring additional infrastructure. Developer Productivity Developing streaming applications requires your developer to invest a significant amount of time in managing the infrastructure and configuration. Such efforts increase with the complex data processing logic, taking more time away from actual application development. We can solve these issues by using a stream processing framework, such as . A stream processing framework connects to an upstream resource to stream data in real-time, receive the data, process it, then send it on to a destination. Meroxa’s Turbine Meroxa has a on what this looks like at a high level. If we were to look more closely at how we might use Go to interact with Turbine, we could look at : good write-up code from this example application func (a App) Run(v turbine.Turbine) error {
	source, err := v.Resources("demopg")
	if err != nil {
		return err
	}
	// a collection of records, which can't be inspected directly
	records, err := source.Records("user_activity", nil)
	if err != nil {
		return err
	}
	// second return is dead-letter queue
	result := v.Process(records, Anonymize{})

	dest, err := v.Resources("s3")
	if err != nil {
		return err
	}
	err = dest.Write(result, "data-app-archive")
	if err != nil {
		return err
	}

	return nil
}

func (f Anonymize) Process(records []turbine.Record) []turbine.Record {
	for i, r := range records {
		hashedEmail := consistentHash(r.Payload.Get("email").(string))
		err := r.Payload.Set("email", hashedEmail)
		if err != nil {
			log.Println("error setting value: ", err)
			break
		}
		records[i] = r
	}
	return records
}

func consistentHash(s string) string {
	h := md5.Sum([]byte(s))
	return hex.EncodeToString(h[:])
} In the above code, we see the following steps in action: Create an upstream source (named ), from a PostgreSQL database (named ). source demopg Fetch records (from the table) from that upstream source. user_activity Call , which performs the stream processing. This process iterates through the list of records and overwrites the email of each record with an encoded hash. v.Process Create a downstream destination (named ), using AWS S3. dest Write the resulting stream-processed records to the destination. As we can see, you only need a little bit of code for Turbine to process new records and stream them to the destination. Using a framework such as Turbine for stream processing brings several benefits, including: It reduces the need for storage and computing resources by using stream processing instead of batch processing. It provides a powerful mechanism to from large volumes of streaming data, assisting with a quick fix using Novelty Detector, a highly specialized real-time anomaly detection tool. identify anomalous behavior Because for streaming applications, Turbine streaming data apps offer Feature Branch Deploys, allowing you to deploy the branch whenever it is ready—without impacting the production version of the application. continuous delivery is a big challenge It integrates any source database to any destination database by leveraging that receives real-time streams and publishes them downstream. The Meroxa platform handles data transformation or processing logic and orchestration through the Turbine app, sparing the developers from worrisome schema management or scalability issues. Change Data Capture (CDC) Ultimately, this allows developers to focus their time and effort on your core business needs. It also reduces the cost of migration associated with additional infrastructure. Its code-first integration is aimed at helping developers to focus on building applications, leading to faster development while also reducing the infrastructure typically needed to support stream processing. Conclusion Data-driven organizations are betting high on data assets to fuel their transformation journeys. But they need better—and more cost-efficient—ways to manage the fast-growing datasets. Stream processing provides the necessary benefits: improved data quality, faster decision-making, and more. But it requires the right plan and the right stream processing platform to handle the data efficiently. By following the tips in this article, you should be well on your way!

Efficient Enterprise Data Solutions With Stream Processing

In this AMA, we host the CEO of Pool Data, Shiv Malik. Shiv walks us through Pool Data and how it supports data unions in bringing value to the data economy. We also chat about the wider data economy. slogging This Slogging thread by Jack Boreham, Char736, Shiv Malik, Mónica Freitas, Melissa Brown, Sara Pinto, Cris Silva, Blocklawd, Jimmy L, Sonic3625, james Burrows and Gilbert Hill occurred in slogging's official #amas channel, and has been edited for readability. Jack Boreham Jul 13, 2022, 11:00 AM Hi, @channel, please join me in welcoming the CEO of Pool Data, Shiv Malik. Pool Data is a platform that seeks to redistribute power, value, and control in the data economy. Pool supports data unions by providing them with a marketplace and infrastructure to scale their organizations, meaning everyone can benefit from data sharing. You can ask the Shiv anything about: What is Pool Data? How did it come about? Who are the team members behind Pool? What’s their background? Why should we care about our personal data? What’s wrong with the data economy at the moment? What are data unions and how is Pool supporting them? How does this fit in with data buyers, brands, and what they’ve been doing with cookies etc? How are data unions related to Web3? What is your roadmap for the coming year? What is your vision for the future of data? Jack Boreham Jul 13, 2022, 11:01 AM Hi Shiv Malik a pleasure to have you with us. I hope you are prepared for these Qs 😁. To start off, can you tell our audience about your background and the birth of Pool Data? Shiv Malik Jul 13, 2022, 11:03 AM Sure thing! So I’ve got had a bit of a peripatetic career. I was an investigative journalist for many years for the Guardian working on economics issues and also terrorism. I am the author of a couple of books and the co-founder of a think tank. I started in Web3 in early 2017. First as a day trader and then I worked for Golem and then Streamr. Shiv Malik Jul 13, 2022, 11:05 AM Since 2018 I’ve spent my time working to incubate and foster data unions, and building frameworks for them to utilise. I started Pool back last year to specifically concentrate on helping data unions get off the ground. and now Pool has a team of 14 people. Char736 Jul 13, 2022, 11:03 AM Shiv Malik. I'm interested by the idea of a data union what is it and how does pool worth with them? Shiv Malik Jul 13, 2022, 11:05 AM Great question. So a data union is basically a way of allowing people to share and monetize their data. They can be structured in a good many ways. Some of them are DAOs others are more like traditional cooperatives. Pool helps them in three ways. 1. A marketplace to sell the data. 2. An analytics solutions so all the data unions can work cooperatively to create amazing analytics for the world. 3. A data wallet to allow individuals to store their own data and also join other data unions. Mónica Freitas Jul 13, 2022, 11:05 AM Hi Shiv Malik! Great to have you with us. How does Pool work? And where did the idea for Pool come from? Shiv Malik Jul 13, 2022, 11:09 AM Mónica Freitas - the idea has long intellectual roots. Jaron Lanier, the author of Who Own the Future (great book by the way) was talking about them years ago. Glen Weyl is another great proponent of the idea. I started working on them in 2017 to make them a reality and now we count at least 90 data unions around the world monetizing all sorts of data from mobility to clickstream data to mobile sensor data and financial transaction data. Sara Pinto Jul 13, 2022, 11:05 AM Hey Shiv Malik, it's great to have you here with us! We see a lot of news about shared data. What can be benefits of data sharing? Shiv Malik Jul 13, 2022, 11:12 AM Sara Pinto I think the main benefit of data sharing is 1. the income you can derive from doing so as a collective (individually your data is worth very little, combined it is worth a fortune). We expect that to be around $150-200 a year. 2. People will get so much in terms of convenience and control by storing their own data in a vault. Imagine signing in with your wallet (i.e. signing in with ethereum) and being able to share your full data story with anyone, on your terms and under your full control. Cris Silva Jul 13, 2022, 11:06 AM Hello, Shiv Malik. Why did you decide to have your own token? What are the perks of that token for individuals and businesses? Shiv Malik Jul 13, 2022, 11:15 AM Cris Silva - Great question! So it’s turns out when you work as a cooperative you really need utility tokens to sort out certain common problems. And it’s not just governance we’re talking about her. For example, the revenue we expect to garner from our combined analytics will pull upon raw data from all the data unions in our ecosystem. Who is to say which data sets are the most valuable or not in a given query. The Token in this instance will really help us sort out who should get more revenue or not. Our token is also a discount token to allow DUs who really want to be invested in common success to get full value out of the platform. Blocklawd Jul 13, 2022, 11:10 AM Shiv Malik we here about data privacy issues all the time in the news, Facebook, etc. Where does Pool Data fit into all of this? What benefits does pool offer the average consumers? Shiv Malik Jul 13, 2022, 11:18 AM - privacy is massively important. But I think people can now see that Google, apple and even facebook are starting to co-opt the privacy space. They’re basically saying, ‘only we can protect user privacy’. That has meant that everyone else is locked out of the data economy and they will garner even more control over advertising and analytics and basically the world’s data. That has to stop. Especially since the economic future of everyone depends on ordinary people having control and leverage over the data they produce if they are going to have any control over AI. Also I think privacy tools have been great at stopping people from sharing information but not so great at allowing us to share on our terms, when we do want to. And humans are very pro-social beings. We want to share information, we just don’t want to be taken for a ride and we want to be able to say no when we chose. Melissa Brown Jul 13, 2022, 11:12 AM Shiv Malik what do you mean by storing data in a wallet. Is it like crypto Shiv Malik Jul 13, 2022, 11:20 AM Melissa Brown - yes! That’s pretty much correct. Imagine a regular wallet but now you’ve also appended a data vault to it (we’re using Ceramic as the protocol). And now if you join a data union, you will also be able to copy and paste the same data you share with them into your own data vault. The read permission of the vault will be controlled by the same private key you use to send money. It’s pretty revolutionary. Sara Pinto Jul 13, 2022, 11:19 AM Shiv Malik storing our own data in a vault makes total sense. What are the issues revolving data? Regarding the economy and storing it Shiv Malik Jul 13, 2022, 11:23 AM Sara Pinto so the idea of a personal data vault is pretty old. There’s an amazing paper written in Harvard Biz Review from 1999 for example. It’s the dream idea: everyone gets to port their data and store their own digital story. It solves a lot of KYC issues for example. The hard part has always been about where does the individual get that data from? If everyone stores random bits of data, that are basically an unstructured mess, how will anyone else be able to read that data in a programatic fashion?    That’s where data unions come in. They already produce well structured data. So it’s easy to store that stuff on IPFS (using Ceramic). Does that answer your question? Cris Silva Jul 13, 2022, 11:20 AM Shiv Malik, interesting. So, do you operate solely in a blockchain/web3-centric strategy? Or do you have options for the not-so-up-to-date web2 companies? 😂 👍 1 Shiv Malik Jul 13, 2022, 11:25 AM Cris Silva - yeah - short version of this is that the data buyers are al in Web2 and all the data unions are already in Web3 or are moving (because they need scalable micropayments and only blockchain delivers this). So Pool has to act as a bridge between Web2 and Web3. Cris Silva Jul 13, 2022, 11:26 AM Shiv Malik, I bet there are very specific issues that arise when you have to bridge that web2 - web3 gap. What would you say are the biggest issues Pool faces? Shiv Malik Jul 13, 2022, 11:29 AM Cris Silva great question again! So I’d say, the monetary bridge is tricky but that’s why Data Union data products are priced in dollars, we accept payments in dollars, and then we pay out using Gnosis Chain in xDAI which is (as you guys I’m sure know) a dollar based stable coin. The other main issue will be getting major websites to allow people to sign in with Ethereum in the near future. But that’s why we are partnered with a very serious Web2 advertiser who totally believes in data walletts replacing the cookie. Of course greater adoption of the “sign in with Ethereum” protocol will also massively increase web3 adoption. Jack Boreham Jul 13, 2022, 11:22 AM Shiv Malik my goodness that's amazing. So how are you trying to educate people on the value your wallet brings them! It must be a challenge to make this mainstream, given all the Blockchain jargon haha. Shiv Malik Jul 13, 2022, 11:27 AM Jack Boreham - great question! So the data wallet will be out next year for people to test. But for us the big moment is in March 2024 when everyone in Europe (450m people) will get a new legal right to port data from Google, Facebook, Apple, Amazon and Microsoft (linkedin, Github etc), in real time, at the touch of the button. All you have to do is sign in and connect to a third party. That’s when we expect Data Unions to explode. And that’s when we will market our data Wallet - called Pocket - on a vast scale. Sonic3625 Jul 13, 2022, 11:29 AM Hi Shiv Malik! Would you say the current data economy is broken? How will pool compete with giants like meta who take our data for their own use? Shiv Malik Jul 13, 2022, 11:36 AM - totally, yes. It’s actually undergoing a HUGE crises. First there was GDPR, then data scandals shuttered companies like Cambridge Analytica and Jumpshot. Then Apple shut down 3rd party tracking of apps. And now cookies are being blocked by Chrome in 2023. So the old Surveillance data economy is finally defunct. Which is great news. BUT the big battle now is to find a replacement. Google basically is using privacy protection as a way to leverage even more economic control of advertising. Ordinary people will get almost nothing if they succeed. (And perhaps not even better privacy). Data Unions are a way that really fixes this. In fact the EU believes in the idea as well and is putting in €2bn to support them. Sonic3625 Jul 13, 2022, 11:29 AM Especially as they want to be THE Dominent meta player Shiv Malik Jul 13, 2022, 11:38 AM To answer the second half  - people who store data in a data vault will actually have far more info about their digital story than Facebook would have on them. Combine clickstream data, or data from your car with fianncial transaction data and you’ve already got way more data than Facebook does. That’s how we outcompete everyone - working together to create much, much richer data. Sara Pinto Jul 13, 2022, 11:30 AM Shiv Malik yes it does! Thank you. So what having said that, what do you think it will be the future of data? Shiv Malik Jul 13, 2022, 11:39 AM Sara Pinto - the future of data - that’s easy! Data Unions! A world in which everyone has a controlling stake in the data economy. Where people have data dignity and agents representing them for all the hard stuff. That’s what I want! Melissa Brown Jul 13, 2022, 11:32 AM Shiv Malik what about the rest of the world, how will you market it to them? Mónica Freitas Jul 13, 2022, 11:34 AM Shiv Malik, I'm new to data unions. I assume they reassemble any other union/syndicate. So, in essence, a group of people working together to protect their data and how it's being used. Is this correct? What are the particularities that make data unions unique? Shiv Malik Jul 13, 2022, 11:43 AM Mónica Freitas - that’s right Monica! and my we should have called them “data syndicates”! They have functions of both a union (in that they negotiate better prices for you on your behalf and better terms of conditions for using your data). But they are also like cooperatives in that they share out revenue. There’s a basic explainer here btw: https://www.pooldata.io/blog/what-is-a-data-union 💚 1 Cris Silva Jul 13, 2022, 11:33 AM Shiv Malik, that's a great choice of a crypto coin - not as subject to fluctuations as others. Have you noticed an increase in crypto adoption from your web2 clients? Especially after collaborating with Pool? (I'm always curious to know about the impact companies have on the transformation of the web3 space) james Burrows Jul 13, 2022, 11:40 AM Shiv Malik, you've mentioned the EU putting two bn towards it, but how do you ensure that kind of money isn't wasted? How will you work with data unions to ensure their offerings are substantial and beneficial for users? Jack Boreham Jul 13, 2022, 11:43 AM Shiv Malik just out of curiosity do guys lobby governments for funding data unions? Cris Silva Jul 13, 2022, 11:45 AM Shiv Malik, that's very true. You're a relatively recent company. What has the growth curve been like? How many unions do you currently have? Shiv Malik Jul 13, 2022, 11:41 AM Cris Silva We think so too!Data buyers are still very reluctant to think about crypto. but anyone running a data union has realised that paying millions of people small amounts each day can only be done through a blockchain. “that’s a great choice of a crypto coin” - 🔥 1 Shiv Malik Jul 13, 2022, 11:48 AM and Jack Boreham - I’ll take your questions toghether. So the funds right now are administered by the EU through the NGI programme, Horizon 2020 and others. (I don’t have all the links to hand). We work hard to chose good teams, great ideas and products with clear traction. In the end though it is the data buyers who tell us which of the products they find most interesting. We ran a catalyst programme for Data unions in the first half of the year to really tease all of this out with our major data buyer partners. Shiv Malik Jul 13, 2022, 11:50 AM Cris Silva So it’s a small enough space at the moment that we pretty much have contacts with all the big data uninos. (Some have 250k-1m users now). We will work with the six best to get them onboarded to our data marketplace by the end of the year. The next stages are then working on the combined analytics platform and the data wallet, Pocket. The hard part is bringing them together to start with but I think we’ve overcome that hurdle! Mónica Freitas Jul 13, 2022, 11:46 AM Shiv Malik, oh, that's a great model. Do you work with already existing data unions, or do you help to create new ones too? Shiv Malik Jul 13, 2022, 11:51 AM Mónica Freitas - the ☝ should answer your question too I hope! Shiv Malik Jul 13, 2022, 11:54 AM Jack Boreham - sorry I don’t think I quite answered your question - we haven’t yet applied for grants , but we did lobby for the ecosystem to get proper regulation, that wasn’t too stringent, and yes, the ecosystem funding to get it all going. Back at the start of the pandemic, in 2020 that’s mainly what I was doing - lobbying people in Brussels (along with a good amny others0 to help push through the Data Governance Act which means what they call “data intermediaries” (i.e. data unions) can be officially recognised and supported. That also means the data they sell will also have much more standing in a world where most data brokers are operating under the table because they don’t have real user consent to sell it. Gilbert Hill Jul 13, 2022, 11:53 AM EU Horizon program Data trust & sovereignty call for funding https://ec.europa.eu/info/research-and-innovation/funding/funding-opportunities/funding-programmes-and-open-calls/horizon-europe_en https://www.ngi.eu/opencalls/#ngizeroentrust_opencall 🙏 2 james Burrows Jul 13, 2022, 11:53 AM Shiv Malik, can you talk about what data union platforms you at excited about? Shiv Malik Jul 13, 2022, 11:58 AM - So here are a list (I can’t pick faves!) of some great data unions. Some are Web2 others are Web3 as mentioned before. , , , , , and https://gener8ads.com/ https://cake.app/ https://unbanx.me/ https://dimo.zone/ https://www.ozone.ai/account https://swashapp.io/ https://play.google.com/store/apps/details?id=tech.solipay.consumer&hl=en_GB&gl=US 🙌 1 Sara Pinto Jul 13, 2022, 11:54 AM Shiv Malik you mentioned that data unions can help monetize data, and you mentioned some of the ways you can do it. Does Pool offers those 3 methods? Shiv Malik Jul 13, 2022, 11:54 AM Sara Pinto yes - we offer all three! Cris Silva Jul 13, 2022, 11:54 AM Shiv Malik, those are giant unions already. That gives you a pretty big pool of data to share. How do you manage to navigate the different data privacy regulations? Melissa Brown Jul 13, 2022, 11:55 AM Shiv Malik I would love to know what you guys think about the metaverse as a separate entity from web3, how will Pool fit in this? Shiv Malik Jul 13, 2022, 12:00 PM Melissa Brown - great question on the metaverse - in short you are not going to be able to traverse the metaverse without being able to store your digital identity and story. Pocket, our data wallet will be an absolutely central part of all of this. Mónica Freitas Jul 13, 2022, 11:56 AM Shiv Malik that does answer my question 😊 One of your highlights in the landmarks achieved as a company was establishing a company culture. What's your company culture? Shiv Malik Jul 13, 2022, 12:00 PM Mónica Freitas - We’ve run out of time but I’ll try and answer this briefly! 💚 1 Shiv Malik Jul 13, 2022, 12:01 PM I’d say it’s the people. We have an AMAZING team of real experts who are such a joy to work with. 🔥 1 Shiv Malik Jul 13, 2022, 12:01 PM That’s the biggest landmark. If you can bring great people together - ANYTHING IS POSSIBLE! Jack Boreham Jul 13, 2022, 12:01 PM We certainly have!! Thank you Shiv Malik for partaking in this AMA, fantastic answers and best of luck with pool!! Mónica Freitas Jul 13, 2022, 12:01 PM Shiv Malik Thank you! Gilbert Hill Jul 13, 2022, 12:01 PM Shiv Malik thanks! Shiv Malik Jul 13, 2022, 12:02 PM Thank you everyone for your brilliant questions and bearing with me as I typed as fast as possible! 💚 5 james Burrows Jul 13, 2022, 12:03 PM Thanks all!

Putting Value back in the Data Economy with Pool Data CEO Shiv Malik

Devolviendo valor a la economía de datos con el CEO de Pool Data, Shiv Malik

Remettre de la valeur dans l'économie des données avec Shiv Malik, PDG de Pool Data

पूल डेटा के सीईओ शिव मलिक के साथ डेटा अर्थव्यवस्था में मूल्य वापस लाना

Pool Data の CEO である Shiv Malik がデータ エコノミーに価値を取り戻す

Recuperando valor na economia de dados com o CEO da Pool Data, Shiv Malik

Đưa giá trị trở lại nền kinh tế dữ liệu với Giám đốc điều hành của Pool Data Shiv Malik

与 Pool Data 首席执行官 Shiv Malik 一起为数据经济带来价值

Ads are a crucial part of the modern internet, but it is also a very complex and ever-changing market with trends that affect us all. Are there any players that can challenge the power of Google, what are the current data security developments in this space, and what is the next big ad thing? I've discussed these and some other questions with Vitaliy Gerko, co-founder and Chief Product Officer of OTM, acquired in 2021 by Veon global digital operator to elaborate on the current trends in 2022. Please share some background on OTM and your story in AdTech and MarTech I have more than 15 years of experience in the tech industry, it all started back in 2005 after I graduated from university. At that time, back in Belarus, I opened a web development and programming company, and we worked in the international market. After the release of the first iPhone, I became very interested in apps and AppStore, and in 2011, I was invited to join a new project team. It was a mobile application that offered motivated traffic, where users do certain things for bonuses. I was responsible for the creation of the product and its commercial success. After selling this project in 2014, I joined the team of my current partner Dmitry: the company OTM, which at the time was engaged in SMS marketing. We began building a full-stack programmatic platform, and since then, I have been responsible for the product and its technological development. We provide a fully automated stack of buying and selling ad inventory, we work directly with publishers, aggregating the traffic and selling it to our own DSP platform, where we have a lot of direct clients and agencies. We also trade that traffic to BigTech (Google, Yandex, VK) and international programmatic platforms. We're building OTM based on the experience of American companies, covering the full value chain: from publishers to advertisers. That way, you have the best control over the whole money chain between them. Our platform is full-stack, we have full-service and self-service DSP, DMP, several SSP, and ad exchanges, a lot of predictive and ML models which allow you to optimize traffic for KPIs and benchmarks of clients. In addition, my previous experience helps a lot when it comes to numbers. I have a good understanding of the unit economy and user acquisition, I know how third-party platforms work: DV360 by Google, and Meta, and use it for the development of OTM. We also use examples of such American platforms as Index Exchange and The Trade Desk. This year we will finish the realization of our deal with VEON, a global digital operator that provides mobile connectivity and services to 217 million customers all over the world. For us, this means that we will fully integrate the operator's capabilities into our own. At the same time, we are not implementing a full merger of the company into an operator, as we are trying to take into account the mistakes in the cases of Verizon after acquiring AOL and Yahoo, as well as AT&T after acquiring AppNexus. We are keeping the OTM brand, and we will use the flexibility of a small company to quickly get a slice of a big advertisement pie. We are moving all the operator's capabilities to us: data, communication channels, and we will build a holistic MarTech product based on OTM. What’s your view on the strained position that marketers find themselves in today, with all privacy-focused trends? Over the past few years, BigTech and regulators have become concerned about privacy and the protection of user data. Of course, this has affected the tech market. For example, starting from iOS 14, the system came out with the ability for the user to hide personal data from targeting advertising opportunities, By year-end 2023, Google plans to officially stop supporting third-party cookies on its Chrome browser, the most popular browser in the world. Google already has a monopoly position because of its browser, and if it prohibits using third-party cookies for targeted advertising, advertisers will be forced to carry money in closed ecosystems  (walled garden), which Google is. There's also Meta, Amazon Advertisement, but still now, Google's action could be seen by the regulator as a concession to itself. But the reality is that in the last year we've seen that others started working on technical solutions to get around the ban on third-party cookies. So, operators can issue a stable ID. In fact, for a telecom operator, the ID is a phone number, and it knows IDFA / GAID and user’s cookies, which allows using stable ID as an alternative replacement of third party cookies. Criteo and The Trade Desk and some other companies are joined the Unified ID 2.0 initiative to keep working in the cookieless world. is a new approach to identity that preserves the value exchange of advertising on the open internet, while providing improved consumer controls. It will be open source, interoperable, non-commercial and administered by an independent organization. This initiative Data privacy regulations also put analytics through hard times. Firms complying with GDPR in Europe and following an opt-in model (not tracking analytics unless a visitor grants them permission), recorded as much as 90% traffic declines. Nevertheless,  a cookieless version of Google Analytics allows you to capture some traffic with the help of Client IDs. So, even though restrictions are getting tighter, on the other hand, we shall also see more opportunities. How does your company tackle user data protection? We fully comply with local and global user data regulations, operate only with IDs and no personal data. We use them for further matching and audience segmentation. Our clients can use this information of impersonal user cohorts, without specific user information for targeting and ad campaign setup. Do you think that due to the recent changes and buzz around ads people generally have become more educated on advertisement mechanisms? I don't think users have come to understand the advertising ecosystem very deeply, rather, their understanding has somewhat progressed. But I do know that, in general, we are all very tired of advertising as it is now. Certainly, users have understood that all devices, platforms, and services are following us to using information about our behavior, purchases and clickstream for their commercial purposes basically to show ads. I think the solution is not the choice between paid service/platform/content with no ads, and free access with ads. I think there is an option to make some kind of balance between these extremes. Of course, users data has a direct impact on the advertising ecosystem. But as Tim O'Reilly assessed in his article "Data is the New Sand," the money that a user can get from "selling" their data to Google and Facebook is negligible. Since users generally have now more advertisement literacy, for example, they know how retargeting works, it's likely that in the attention economy there will be some sort of equal exchange, and users will take their place in that value chain. So that for their attention and data, they could get some kind of value point. What it will be is a question, it is unlikely to be about money. Perhaps it will be resolved at the level of granting access to content and services. What  ‘privacy-first’ solutions do you see as most promising? After the deal with VEON, we solved the problem for ourselves, because telecom operators have unique IDs in the form of phone numbers. So you can match IDFA, GAID and cookies with the phone number, The operator has a lot of user data, and we can make models based on operator data involving stable IDs. The other issue is that any operator in the world doesn't take 100% of the population. Of course, it would be desirable that all operators create a single platform with stable IDs. I have not seen such a solution in the U.S., but it is generally possible. On any request to this platform on a commercial basis, this platform will give you stable IDs. No matter, Verizon, AT&T or Wi-Fi operator: the user goes to the site, the publisher sends this incoming request to the platform, and gets a unique unified ID. And you can use this ID to build knowledge about this user and use ID to collect the necessary segment for targeting ad campaigns In general, we use operator capabilities, but we also follow developments of other programmatic platforms and big tech: Google, Meta, Amazon, and the ecosystem trends in general. Final question, what are the most powerful current tech trends that will also affect online advertising? The transition to the creators' economy and everything that is happening now in the AdTech and MarTech markets foreshadows that the entire advertising ecosystem, BigTech, everything will change one way or another. Right now my interest is most attracted to the Web3 industry, Blockchain and Crypto-projects are sprouting numerous talented people, groups of serious and advanced specialists, even though there are a lot of scams and other disadvantages in these areas at the moment. The advertising system will change a lot, because, for example, blockchain technology is about transparency, including advertising money. There's a lot of fraud on the ad market, and a lot of money and budgets are wasted, so blockchain is more likely to have a positive effect. Only a small part of the money from the chain reaches the publisher, users are very little valued, and, most likely blockchain will correct this, it is only unknown how quickly there will be qualitative changes in the value chain.

Digital Ad Trends to Watch in 2022: Market Monopoly, Privacy Concerns, and New Disruptive Tech

Have you ever noticed your Amazon shopping cart getting a bit too full, then had to delete a few items in order to stay within budget? Is there a luxury item or neat new gadget you'd love to buy, but just can't justify the expense at the moment? Tired of pulling out your credit card for every little purchase? Buy Now Pay Later (BNPL) may just be the solution you've been looking for. BNPL is a reimagining of short-term financing for the e-commerce era. This new kind of personal loan, issued at the point-of-sale, is an innovative way that companies can take advantage of data-driven strategies in order to unlock value in a sales environment that is rapidly shifting online. Now, two years into the Covid pandemic, with virtually all of our everyday purchases being carried out on the internet, BNPL is certainly a trend to watch. From the customer's perspective, BNPL is a definite upgrade over legacy lending models. For one, the application process does not affect the customer's credit score. Second, BNPL offers more flexibility than do other payments plans, typically allowing customers to customize the number of installments and payment schedule. Third, BNPL plans usually do not charge interest. Revenue in this model is instead from referrals and from penalty fees that collected on late payments. There are numerous advantages for merchants as well. To begin, BNPL platforms have their own user bases and can drive traffic to merchants' online stores. This kind of referral marketing comes out as cheaper than other forms of customer acquisition. Second, statistics show that e-commerce users who pay in installments tend to purchase more items than those who pay in full. Finally, BNPL helps to eliminate chargebacks and other kinds of fraud risk for online stores. E-Commerce, Brought To You By AI The Buy Now Sell Later model has only really become viable over the last year or so because of technological innovation. Namely, BNPL has been made possible by the relatively recent maturity of artificial intelligence and machine learning. Short-term point of sale loans are by no means a new thing. Indeed, stores and restaurants have attempted to increase sales by allowing customers to open tabs for ages. Already decades ago, major retailers and big box stores would, in partnership with banks, offer credit cards and installment loans to customers right at the cash register. E-commerce companies have been trying to find ways to offer customers credit since the start of the information age. More recently, this has become an essential need for millions of consumers and the companies that serve them. Indeed, today, in the midst of the global COVID 19 pandemic, more and more people are finding themselves in situations where they can only make purchases online. Very often, these people need access to credit. BNPL, it seems, is the answer. Here are just a few reasons why: Better Customer Onboarding In the legacy credit system, the issuance of short-term, PoS, loans depends on the ability of customers to prove good credit. This is usually achieved by looking up customers' credit scores. This may sound straightforward, but actually it presents two key problems: First, the customer may have too low a score to be issued a loan. This customer, who many have been quite costly to acquire, will need to be rejected. Second — since frequent checks can negatively affect scores — many customers who do have good credit may not be willing to allow merchants to look up their scores. These customers end up only purchasing a lower cost item, or nothing at all. Buy Now Pay Later addresses both of these issues using, first, a procedural innovation, and second, a technical one. To begin, BNPL payment plans are designed in such a way so as to bypass credit checks. When a customer chooses to purchase an item, they never actually attempt to apply for a loan through the merchant. Instead, they create an account with the BNPL provider, who then carries out what is called a "soft inquiry" on their credit. In other words, the customer authorizes the installment plan provider to look up their credit score not as a financial institution evaluating a loan application, but instead as some other interested party. This kind of check should not have any positive or negative effect on the rating. Second, by taking advantage of AI/ML, BNPL platforms are empowered to collect and analyze customer data in order to make more sophisticated decisions. When a customer chooses to make a purchase via BNPL, the provider may prompt them to answer a series of questions or to submit certain documents. Natural language processing can then be applied to this information. NLP technology can provide valuable insights about the customer. Customers that are shown via data analytics to be low-risk could be offered better loan conditions, such as longer repayment periods. If the customer is considered  higher-risk, their payment plan would be configured differently. Ongoing Data Analytics and Risk Modeling As was discussed above, BNPL platforms regularly collect and analyze data in order to better evaluate their customers for future payment arrangements during the onboarding process. Increasingly, BNPL providers continue to collect this information from customers even following the initial purchase. By applying AI and data analytics strategies to this ongoing stream of data, it becomes possible to turn one-time customers into highly-valuable, long-term clients. Provided that this data has been collected with the user's consent, there are a couple of interesting ways this data can be used. One possibility is for BNPL companies to look at what the user is purchasing and make suggestions for additional purchases from other merchants. This not only increases overall turnover, but it also boosts revenue in the form of referral fees. Another application of data analytics, here, is reflected in BNPL providers' ability to use data from merchants' platform and websites to different kinds of scoring of customers, based on their online behavior. For example, AI can be used to analyze the links that a customer clicks (their "clickstream") and previous purchases made through merchants' stores. Moreover, BNPL providers can develop their own risk models based on their own interactions with customers. These models can be used to create value further down the line. For example, a user who always pays their installments on time could be offered exclusive products, better loan conditions, and even things like loyalty rewards and cashback. A user with a less-than-perfect record could be targeted with higher interest rates and worse conditions, such as lower installment allowances and higher late-payment fees. Fraud prevention The early detection and prevention of fraud has long been one of the primary applications of AI/ML technology in both fintech and e-commerce. Indeed, payment processors today use machine learning to detect suspicious transactions as a default, and AI-powered identity checks have become standard for many kinds of regulated services. For BNPL providers, AI/ML provides numerous benefits as well. Like in other forms of online lending, photo recognition and natural language processing can be utilized in order to verify the personal documents of borrowers. Machine learning can be used to analyze the transactions of users in real time and block suspicious activity before any money changes hands. Moreover, predictive algorithms can be applied to data collected by merchants in order to flag users who may attempt to repeat fraudulent transactions or schemes that had been attempted in the past. Collections One of the unfortunate truths about the lending industry is that not all borrowers will be willing to pay on time. This holds true in the BNPL model as well. Luckily for providers, great progress has been made of late in the introduction of AI into the collections process. Primarily, AI can be used to address problems relating to communication. These issues are significant when dealing with borrowers that were initially engaged online. In legacy lending, collectors tend to only attempt to reach debtors using a few channels, such as by phone or email. Many individuals who owe money have little issue avoiding these interventions. Using AI-powered data analytics strategies, lenders can monitor all the different channels that a borrower might use. If the customer has been shown to regularly post to Instagram, the collection team will be able to understand that a direct message may be the best way to approach the client. This can work for virtually any social network or messaging platform. Conclusion Buy Now Pay Later and the technology behind it is having a massive effect on e-commerce. Since we specialize in developing unique AI/ML solutions that unlock value for businesses operating in this sector, we at Daiger have been observing this process closely. Please feel to get in touch, we would be happy to discuss how we can create innovation in online retail together.

How Buy Now Pay Later (BNPL) Is Bringing E-Commerce into the AI Era

Embeddings have pervaded the data scientist’s toolkit, and dramatically changed how NLP, computer vision, and recommender systems work. However, many data scientists find them archaic and confusing. Many more use them blindly without understanding what they are. In this article, we’ll deep dive into what embeddings are, how they work, and how they are often operationalized in real-world systems. What’s an embedding? To understand embeddings, we must first understand the basic requirements of a machine learning model. Specifically, most machine learning algorithms can only take low-dimensional numerical data as inputs. In the neural network below each of the input features must be numeric. That means that in domains such as recommender systems, we must transform non-numeric variables (ex. items and users) into numbers and vectors. We could try to represent items by a product ID; however, neural networks treat numerical inputs as continuous variables. That means higher numbers are “greater than” lower numbers. It also sees numbers that are similar as being similar items. This makes perfect sense for a field like “age” but is nonsensical when the numbers represent a categorical variable. Prior to embeddings, one of the most common methods used was one-hot encoding. One-Hot Encoding One-hot encoding was a common method for representing categorical variables. This unsupervised technique maps a single category to a vector and generates a binary representation. The actual process is simple. We create a vector with a size equal to the number of categories, with all the values set to 0. We then set the row or rows associated with the given ID or IDs to 1. An illustration of One Hot Encoding This technically works in turning a category into a set of continuous variables, but we literally end up with a huge vector of 0s with a single or a handful of 1’s. This simplicity comes with drawbacks. For variables with many unique categories, it creates an unmanageable number of dimensions. Since each item is technically equidistant in vector space, it omits context around similarity. In vector space, categories with little variance are not any closer together than those with high variance. This means that the terms “Hotdog” and “Hamburger” are no closer together than “Hotdog” and “Pepsi”. As a result, we have no way of evaluating the relationship between two entities. We could generate more one-to-one mappings, or attempt to group them and look for similarities. This requires extensive work and manual labeling that’s typically infeasible. Intuitively, we want to be able to create a denser representation of the categories and maintain some of the implicit relationship information between items. We need a way to reduce the number of categorical variables so we can place items of similar categories closer together. That’s exactly what an embedding is. Embeddings solve the encoding problem Embeddings are dense numerical representations of real-world objects and relationships, expressed as a vector. The vector space quantifies the semantic similarity between categories. Embedding vectors that are close to each other are considered similar. Sometimes, they are used directly for the “Similar items to this” section in an e-commerce store. Other times, embeddings are passed to other models. In those cases, the model can share learnings across similar items rather than treating them as two completely unique categories, as is the case with one-hot encodings. For this reason, embeddings can be used to accurately represent sparse data like clickstreams, text, and e-commerce purchases as features to downstream models. On the other hand, embeddings are much more expensive to compute than one-hot encodings and are far less interpretable. How are Embeddings created? A common way to create an embedding requires us to first set up a supervised machine learning problem. As a side-effect, training that model encodes categories into embedding vectors. For example, we can set up a model that predicts the next movie a user will watch based on what they are watching now. An embedding model will factorize the input into a vector and that vector will be used to predict the next movie. This means that similar vectors are movies that are commonly watched after similar movies. This makes for a great representation to be used for personalization. So even though we are solving a supervised problem, often called the surrogate problem, the actual creation of embeddings is an unsupervised process. Defining a surrogate problem is an art, and dramatically affects the behavior of the embeddings. For example, YouTube’s recommender team realized that using the “predict the next video a user is going to click on” resulted in clickbait becoming rampantly recommended. They moved to “predict the next video and how long they are going to watch it” as a surrogate problem and achieved far better results. Common Embedding Models Below are some common embedding models to learn more about. Principal Component Analysis (PCA) One method for generating embeddings is called Principal Component Analysis (PCA). PCA reduces the dimensionality of an entity by compressing variables into a smaller subset. This allows the model to behave more effectively but makes variables more difficult to interpret, and generally leads to a loss of information. A popular implementation of PCA is a technique called SVD. SVD Singular Value Decomposition, also known as SVD, is a dimensionality reduction technique. SVD reduces the quantity of data set features from N-dimensions to K-dimensions via matrix factorization. For example, let’s represent a user’s video ratings as a matrix of size (Number of users) x (Number of Items) where the value of each cell is the rating that a user gave that item. We first pick a number, k, which is our embedding vector size, and use SVD to turn it into two matrices. One will be (Number of users) x k and the other will be k x (Number of items). In the resulting matrices, if we multiply a user vector by an item vector, we should get our predicted user rating. If we were to multiply both matrices, we’d end up with the original matrix, but densely filled with all of our predicted ratings. It follows that two items that have similar vectors would result in a similar rating from the same user. In this way, we end up creating user and item embeddings. Word2Vec Word2vec generates embeddings from words. Words are encoded into one-hot vectors and fed into a hidden layer that generates hidden weights. Those hidden weights are then used to predict other nearby words. Although these hidden weights are used for training, word2vec will not use them for the task it was trained on. Instead, the hidden weights are returned as embeddings and the model is tossed out. Words that are found in similar contexts will have similar embeddings. Beyond that, embeddings can be used to form analogies. For example, the vector from king to man is very similar to the one from queen to woman. One problem with Word2Vec is that single words have one vector mapping. This means that all semantic uses for a word are combined into one representation. For example, the word “play” in “I’m going to see a play” and “I want to play” will have the same embedding, without the ability to distinguish context. BERT Bidirectional Encoder Representations of Transformers, also known as BERT, is a pre-trained model that solves Word2Vec’s context problems. BERT is trained in two steps. First, it is trained across a huge corpus of data like Wikipedia to generate similar embeddings as Word2Vec. The end-user performs the second training step. They train on a corpus of data that fits their context well, for example, medical literature. BERT will be fine-tuned for that specific use case. Also, to create a word embedding, BERT takes into account the context of the word. That means that the word “play” in “I’m going to see a play” and “I want to play” will correctly have different embeddings. BERT has become the go-to transformer model for generating text embeddings. Embeddings in the Real World Embedding usage started in research labs and quickly became state of the art. Since then, embeddings have found themselves in production machine learning systems across a variety of different fields including NLP, recommender systems, and computer vision. Recommender Systems A recommender system predicts the preferences and ratings of users for a variety of entities/products. The two most common approaches are collaborative filtering and content-based. Collaborative filtering uses actions to train and form recommendations. Modern collaborative filtering systems almost all use embeddings. As an example, we can use the SVD method defined above to build a recommender system. In that system, multiplying a user embedding by an item embedding generates a rating prediction. This provides a clear relationship between users and products. Similar items beget similar ratings from similar users. This attribute can also be used in downstream models. For example, Youtube’s recommender uses embeddings as inputs to a neural network that predicts watch time. Semantic Search Users expect search bars to be smarter than a regex. Whether it’s a customer support page, a blog, or Google, a search bar should understand the intent and context of a query, not just look at words. Search engines used to be built around TF-IDF, which also creates an embedding from text. This kind of semantic search worked by finding a document embedding that’s closest to the query embedding using the nearest neighbor. Today, semantic search utilizes more sophisticated embeddings like BERT and may use them in downstream models. In fact, even Google uses BERT on a large percentage of their queries: . https://searchengineland.com/google-bert-used-on-almost-every-english-query-342193/ Computer Vision In computer vision, embeddings are often used as a way to translate between different contexts. For example, if training a self-driving car, we can transform the image from the car into an embedding and then decide what to do based on that embedded context. By doing so, we can perform transfer learning. We can take a generated image from a game like Grand Theft Auto, turn it into an embedding in the same vector space, and train the driving model without having to feed it tons of expensive, real-world images. Tesla is doing this in practice today. Another interesting example is the AI Art Machine: . It will generate an image based on user input text. For example, if we type in “Nostalgia”, we get the following image. https://colab.research.google.com/drive/1n_xrgKDlGQcCF6O-eL3NOd_x4NSqAUjK#scrollTo=TnMw4FrN6JeB It works by transforming the user’s text and an image into an embedding in the same latent space. It’s composed of four transformers: Image -> Embedding, Text -> Embedding, Embedding -> Text, Image -> Text. With all these, transformations we can translate text to image and visa-versa using an embedding as an intermediate representation. Embedding Operations In the above examples, we see that there are a few common operations applied to embeddings. Any production system that uses embeddings should be able to implement some or all of the below. Averaging Using something like word2vec, we can end up with an embedding for each word, but we often need an embedding for a full sentence. Similarly, in recommender systems, we may know the items a user clicked on recently, but their user embedding may not have been retrained in days. In these situations, we can average embeddings to create higher-level embeddings. In the sentence example, we can create a sentence embedding by averaging each of the word embeddings. In the recommender system, we can create a user embedding by averaging the last N items they clicked. Subtraction/Addition We mentioned earlier how word embeddings also encode analogies via vector differences. Adding and subtracting vectors can be used for a variety of tasks. For example, we can find the average difference between a coat from a cheap brand and a luxury brand. We can store that delta and use it whenever we want to recommend a luxury item that’s similar to the current item that a user is looking at. We can find the difference between a coke and a diet coke and apply it to other drinks, even those that don’t have diet equivalents, to find the closest thing to a diet version. Nearest Neighbor Nearest neighbor (NN) is often the most useful embedding operation. It finds things that are similar to the current embedding. In recommender systems, we can create a user embedding and find items that are most relevant to them. In a search engine, we can find a document that’s most similar to a search query. Nearest neighbor is a computationally expensive operation however. Performed naively it is O(N*K), where N is the number of items and K is the size of each embedding. However, in most cases when we need nearest neighbors, an approximation would suffice. If we recommend five items to a user, and one is technically the sixth closest item, the user probably won’t care. Approximate nearest neighbor (ANN) algorithms typically drop the complexity of a lookup to O(log(n)). Implementations of ANN There are many different algorithms to efficiently find approximate nearest neighbors, and many implementations of each of them. We’ll list a few of the most common algorithms and implementations below and break down how they work at a high level. Spotify’s Annoy In Spotify’s ANN implementation (Annoy), the embeddings are turned into a forest of trees. Each tree is built using . At every intermediate node in the tree, a random hyperplane is chosen, which divides the space into two subspaces. This hyperplane is chosen by sampling two points from the subset and taking the hyperplane equidistant from them. This is performed k times to generate a forest. Lookups are done via in-order traversal of the nearest tree. Annoy’s approach allows the index to be split into multiple static files, the index can be mapped in memory, and the number of trees can be tweaked to change speed and accuracy. random projections Here’s how Annoy breaks down the embeddings into an index of multiple trees via random projections. Locality Sensitive Hashing (LSH) Locality Sensitive Hashing (LSH) is another common approach. LSH employs a hash table and stores nearby points in their respective buckets. To create the index, LSH runs many hashing functions which place similar points in the same bucket. In doing so, LSH keeps points with large distances in separate buckets. To retrieve the nearest neighbor, the point in question is hashed, a lookup is done and the closest query point is returned. Some pros include a sub-linear run time, zero reliance on data distribution, and the ability to fine-tune accuracy with the existing data structure. Facebook’s FAISS and Hierarchical Navigable Small World Graphs (HNSW) Facebook’s implementation, FAISS, uses Hierarchical Navigable Small World Graphs (HNSW). HNSW typically performs well in accuracy and recall. It utilizes a hierarchical graph to create an average path towards certain areas. This graph has a hierarchical, transitive structure with a small average distance between nodes. HNSW traverses the graph and finds the closest adjacent node in each iteration and keeps track of the “best” neighbors seen so far. HNSW has a polylogarithmic time complexity (O(logN)). HNSW works by creating a hierarchical index to allow for faster nearest neighbor lookup. How embeddings are operationalized today Moving embeddings out of labs and into real-world systems, has surfaced real gaps in the current data infrastructure capabilities. For example, traditional databases and caches don’t support operations like nearest neighbor lookups. Specialized approximate nearest neighbor indices lack durable storage and other features required for full production use. MLOps systems lack dedicated methods to manage versioning, access, and training for embeddings. Modern ML systems need an embedding store: a database built from the ground up around the machine learning workflow with embeddings. Getting embeddings into production isn’t easy. The most common ways we’ve seen embeddings operationalized today are via Redis, Postgres, and S3 + Annoy/FAISS. We’ll cover how each of those implementations typically looks and the challenges with each. Redis Redis is a super-fast in-memory object-store. It makes storing and getting embeddings by key very fast. However, it doesn’t have any native embedding operations. It can’t do nearest-neighbor lookups and it can’t add or average vectors. All of this must be done on the model service. It also doesn’t fit cleanly in a typical MLOps workflow. It doesn’t support versioning, rolling back, or maintaining immutability. When training, the Redis client doesn’t automatically cache embeddings and this can cause unnecessary pressure and cost. It also doesn’t support partitioning embeddings and creating sub-indices. Postgres Postgres is far more versatile but far slower than Redis. Via plugins, it can be made to perform some of the vector operations manually. However, it does not have a performant nearest neighbor index. Also, having a Postgres lookup in the critical path of a model may add too much latency. Finally, Postgres doesn’t have a great way to cache embeddings on the client when training, causing very, very slow training. S3 files + Annoy/FAISS Annoy or FAISS are often used to operationalize embeddings when nearest-neighbor lookups are required. Both of these systems build indices of embeddings for approximate nearest neighbor operations, but they do not handle durable storage or other vector operations. Alone, they only solve one problem. Typically, companies will store their embeddings in S3 or a similar object storage service to fill in the rest of the gaps. They will load the embeddings and create the ANN index directly on the model when required. Updating embeddings becomes tough, and the system typically ends up with a lot of ad-hoc rules to manage versioning and rollbacks. FAISS and Annoy are great, but they need a full embedding store built around them. The Embedding Hub Machine learning systems that use an embedding need a type of data infrastructure that: - Stores their embeddings durably and with high availability - Allow for approximate nearest neighbor operations - Enable other operations like partitioning, sub-indices, and averaging - Manage versioning, access control, and rollbacks painlessly Embedding Hub sits atop your existing infrastructure. There are a few examples of companies like Pinterest who have built their own in-house, but the gap is still yet to be filled. Some startups have raised a few millions to build proprietary systems, and some database companies have tried to add nearest neighbors as a feature on top. There is a clear gap for an open-source embedding store, and we’ve decided to try to fill it. We’ve made our initial release of Embedding Hub on . Feel free to try it out, leave your comments and issues, and hand us a star! Github Conclusion Embeddings are a critical part of the data science toolkit, and continue to gain in popularity. Embeddings have allowed teams to break the state of the art in multiple disciplines from NLP to recommender systems. As they grow in popularity, a lot more focus will go into operationalizing them in real-world systems. We think that an embedding store will become a critical piece of machine learning infrastructure and that’s why we’ve open-sourced ours. GIve it a try, and if this sounds like an awesome project to work on full-time, we’re hiring! This article was first published on the featureform blog

Embeddings in Machine Learning: Everything You Need to Know

What Analyzing Google Search Results Revealed About Google

Whether you want to develop a content strategy or want to optimize your site content; good keywords research is one of the most efficient ways to attract consistent, quality traffic to your site and gain a better ranking on google. For that, you need a smart and advanced keyword research tool. A good Keyword research tool provides you with specific search data that can help you gain insights like: => What are people searching for? => How many people are searching for it? => What is CPC for that keyword? => What are the top ranking pages of that keyword? Since there are so many keyword research tools are available online, how to find the best keyword research tool? To help you, we’ve compiled some of the Top Keyword Research Tools to help you conquer your content strategy, with a list of their popular features and website links. List Of Best Keyword Research Tools 1.Google Keyword Planner 2.Google Trends 3.SEMrush 4.KWFinder 5.Ahrefs Keyword Explorer 6.Ubersuggest 7.SpyFu 8.Keyword Tool 9.Moz Keyword Explorer 10.Serpstat 1.Google Keyword Planner Google's Keyword Planner is an AdWord tool designed to help you research keywords to use in your Search Network campaigns. It is the most accurate and handy free keyword research tool that allows you to discover keywords related to your business. And give you stats to see the estimated monthly searches they receive as well as the costs to target them. => Notable features => You can locate commercial keywords using the “top of page” bid range => Give accurate search volume => Best choice for business websites => You can use Basic Filtering to Refine Your Audience & Budget. Pricing: Free 2.Google Trends Google Trends is a data exploration tool that helps users visualize and discover trends in people's search behavior and lets marketers better understand what audiences are interested in and curious about, in real-time. It is an extremely useful keyword research tool when it comes to reviewing successful content or SEO strategy. => Notable features => Help you get Keyword Search Volume. => Identify Seasonal Trends. => Help you understand user's search behavior => Avoid Temporarily Popular Keywords. => Help you with Trending Relevant Topics. => Utilize Trend Predictions. => Find Related Queries to help you gain a competitive edge. => Optimize Your SEO Strategy. Pricing: Free 3.SEMrush Semrush is an all-in-one keyword research tool suite for improving online visibility and gaining marketing insights. Semrush is one of the best keyword research tools for competitive research. Instead of producing a list of keywords, it suggests 5-10 very specific keywords that your competitors are using and you can make your SEO strategy accordingly. Its reports can help marketers for improving their SEO, PPC, SMM, Keyword Research, Competitive Research, PR, Content Marketing, Marketing Insights, Campaign Management. => Notable features => Uncover long-tail keywords => Give ad group ideas => Help you with yearly keyword trends => Give worldwide CPC distribution statistics => Access to the complete keyword research toolkit => Do keyword difficulty analysis => Help you with organic traffic insights => Allow hyper Targeting by ZIP Code => Help you with devices and Locations Report => Generate competitors discovery report => Generate cannibalization Report => Filter by SERP Feature => Generate Featured Snippets Report => Generate stunning PDF reports SEMrush hosts a 7-day free trial beyond that you can choose one of the three main plans: Pro, Guru, and Business. Pricing: (for limited budget sites and freelancers)- $119.95/ month Pro ( for SEO agencies and small to mid-sized businesses)- $229.95/ month Guru (for eCommerce stores and agencies)- $449.95/ month Business 4.KWFinder KWFinder is a keyword research tool designed by Mangools – a developer known for delivering the best keyword research tools across categories. The KWFinder platform covers both traditional and competitor keyword research to help you find long-tail keywords with low SEO difficulty. Allow bulk upload a list of keywords to analyze in the tool. => Notable features => Integrated with other Mangools tools. => Give Keyword look-up and domain look-up => Allow Bulk upload and review => Allow List creation => Easy to navigate user interface => Allow you to find keywords in any language target => You can choose a keyword from the list of powerful keywords => Give statistics for keyword competition => In-depth SERP overview alongside a list of keyword suggestions => You can use filters to refine the list of keyword suggestions Pricing : KWFinder is a freemium keyword research tool, You can have their 10-day FREE trial where you'll get 5 lookups per 24 hours, 25 related, and 10 competitor keywords per lookup. Then you can switch to any of the three plans Basic, Premium, Agency. (for freelancers and small businesses)- $30/ month Basic (for new startups and blogs) - $40/ month Premium (for eCommerce stores and heavy sites) - $80/ month Agency 5. Ahrefs Keyword Explorer Ahrefs is one of the best keyword research tools with some extra bonuses and a much more intuitive design. Ahrefs mainly analyze a website's link profile, keyword rankings, and SEO health and show the average monthly number of clicks for your target keyword. When it comes to the number of relevant keyword suggestions, Ahrefs goes the extra mile and the platform provides a broad keyword report to the user and relies on clickstream data to provide unique click metrics. => Notable features => Help you with an accurate keyword difficulty score => Provide 1,000+ keyword suggestions => Supported in over 171 countries => Give click metrics for improving CTR => Guide with extensive SERP overview => Help you with competitive research Pricing Ahrefs  has four pricing plans as follows: - $179/ month Standard $399/ month Advanced- $999/ month Agency- 6.Ubersuggest Ubersuggest is a free online keyword research tool that will help you generate keyword ideas to improve and expand your content marketing strategy. Whether you're writing blog posts or want to optimize your site content, UberSuggest helps you to find more than 100 long-tail keywords in a few minutes. It provides complete insights related to your website traffic and organic keywords. Notable features: Provides competitor's page ranking report for keywords. You can discover organic keywords from other relevant websites. Helps you to identify potential backlinks. Provide search volume, competition, and seasonal trends for every keyword. Offer keywords suggestions based on what other people are typing into the Google search engine. Help you in crafting a content strategy based on the topics that people are interested in. Pricing: Free 7.SpyFu SpyFu is a  cloud-based competitive intelligence and keyword research tool that is used by digital marketers to improve their performance in an online search. Using SpyFu with a few clicks, you can easily identify the most profitable search terms in your competitors' paid and organic search campaigns. => Notable features => Compare different websites competing for a single keyword => Discover ranking difficulty => In-depth domain analysis => Unlimited Backlink Results. => Data Exports. => In-Depth SEO Reports. => Backlinks Exports. => Backlinks Searches. => AdWords Advisor Reports. => Domain Searches. => Paid Keywords. Provide a free 30-day trial and comes with three plans that cost between $39 and $299 per month. Pricing: - $39/ month Basic $99/ month Professional- $299/ month Team- 8.Keyword Tool Keyword Tool is a free online keyword research tool that uses Google Autocomplete to generate hundreds of relevant long-tail keywords. The platform is regarded as one of the top keyword research tools on the market and offers detailed keyword data mining and analysis for multiple search engines including Google, YouTube, Bing, Amazon, and more. => Notable features => Find great keywords using Google Autocomplete => Supplement your social media marketing with keyword analysis => Export results to CSV => Simple UI for beginners => Provides comprehensive and accurate keyword suggestions Keyword Tool does not offer any free trial. Keyword tool pricing plans are as follows: Pricing: - $89 / month Pro Basic $199 / month Pro Business- 9.Moz Keyword Explorer Moz Keyword Explorer is a time-saving all-in-one keyword research tool that helps you find profitable keywords and organize your potential keywords into lists. The Moz Keyword Explorer helps businesses manage numerous aspects of their SEO campaign and serves up easy-to-use out-of-the-box keyword recommendations. => Notable features => Get related keyword suggestions => Organic CTR => Priority score for smart targeting => Analyze keywords by search volume => Generate and save keyword lists => Export your data => Find keywords in question format => Sort by predictive keyword metrics => Review SERP details by keyword => Provide Keyword Difficulty score Moz offers a one-month trial period. You can try  Moz Keyword Explorer Free for 30 Days. Pricing: - $99/ month Standard $149/ month Medium- $249/ month Large- $599/ month Premium- 10.Serpstat Serpstat is a smart and powerful tool that allows you to easily identify your niche competitors by topic or by clustered keywords. And while few other keyword research tools can only analyze a single keyword, they give you analysis for the whole cluster of keywords. It’s an all-in-one solution that combines website analytics, keyword research tool, SERP analysis, backlink analysis, a rank tracker, and website audit features. => Notable features => In-depth competitive analysis => Find keyword difficulty and  trends => Clustering. => Text Analytics. => Search Volume Crawling Service. => Team management. => API and integration. => Custom reports. => Keyword Analysis. => Rank Tracking tool Serpstat pricing ranges between $69 and $499/ month. Pricing: $69/ month,  Up to 1 user, & 15,000 keywords. Lite- $149/ month, Up to 4 users, & 60,000 keywords. Standard- - $299/ month,Up to 6 users, & 150,000 keywords. Advanced $499/ month, Up to 8 users, & 300,000 keywords. Enterprise- Conclusion: These are some of the top keyword research tools including both paid and free that you can leverage for optimum SEO performance. To find the one that works best for your business, start by analyzing your budget and then consider the primary focus of your business SEO strategy. You can go for free keyword research tools but the number of keyword ideas and data they give access to you will always pale in comparison to paid tools. While paid tools allow you to go deeper and do more advanced marketing research and offer more valuable keyword suggestions. If budget is your chief concern, You can narrow your search to Google Keyword Planner or Serpstat, they are free but still best at their places. If you need a more robust keyword research tool, you can go for KWFinder, SEMrush, or Moz Keyword Explorer. Do you have any questions about choosing the Businesstop-ranking best keyword research tool? Ask away in the comments!

10 Best Keyword Research Tools for Content Strategists

AWS has launched Amazon Kinesis - the service that is famous for real-time big data processing and ingestion. What is Amazon Kinesis? is a significant feature of Amazon Web Services (AWS) that easily gathers or collects, processes, and analyzes video and data streams in a real-time environment.  Key offerings: This enables to gain quick timely insights as well as reaction to new information instantly. There are a few key capabilities and functionalities offered by Amazon Kinesis, such as processing of streaming data cost-effectively at any scale as well as flexibility feature to opt for the tools best suit the requirements of the application. Amazon Kinesis Using Amazon Kinesis, real-time data can be ingested, such as audio, video, website clickstreams, application logs, and IoT telemetry data for artificial intelligence, machine learning, and other analytics applications. Amazon Kinesis also assists with processing and analyzing data as it reaches and responds instantly without having to wait for the entire collection of data so that the processing could begin. What are Amazon Kinesis Data Streams? Amazon Kinesis Streams are used to gather together and process huge streams of data records in real-time. Kinesis Data Stream Applications can be created, which are data-processing applications. These applications perform the reading from a data stream in the form of data records. They use Kinesis Client Library for these operations and can run on Amazon EC2 instances. Processed records can be sent to dashboards and can be used to generate alerts, send data to other AWS services, and dynamically change advertising and pricing strategies. Some scenarios for implementing Kinesis Data Streams are as follows: Real-time data analytics: Parallel Processing power is combined and coordinated with the value of real-time data here. For instance, website clickstreams can be processed in real-time and analysis of site usability management using various Kinesis Data Streams executing in parallel. Real-time metrics and reporting: Data collected or gathered into Kinesis Data Streams can be used for simple data analysis and reporting in real-time. For example, Data Processing Applications working on reporting and metrics for system and application logs while data streaming. Complex stream processing: Directed Acyclic Graphs (DAGs) can be developed out of data streams and Kinesis Data Stream Applications. This includes transferring data from various Kinesis Data Stream applications to another stream for downstream processing through different Kinesis Data Streams Applications. Merits of Data Streams We will be discussing here some of the . Those are as follows: Merits of Data Streams Amazon Kinesis is fully managed, and you do not need to maintain or manage any infrastructure for running your streaming applications. Amazon Kinesis can manage large amounts of streaming data and process data from hundreds of thousands of sources with low latencies. You can consume, process, and buffer data in real-time so that you can obtain insights within no time. Use Cases of Amazon Kinesis **Use Cases of Amazon Kinesis** A few of the real-life examples where Amazon Kinesis is being applied in various industries are: Building real-time applications: Amazon Kinesis comes into existence for various purposes such as fraud detection, live leaderboards, and application monitoring. Kinesis Data Streams can be used to ingest streaming data which can be processed further using Kinesis Data Analytics. These results are then radiated to any application or data store using Kinesis Data Streams. Analysis of IoT Device Data: Streaming data coming from IoT devices such as embedded sensors, consumer appliances, and TV set-top boxes can be processed using Amazon Kinesis. Data can be used to transmit real-time alerts or take actions when a sensor exceeds certain operating thresholds. Building Video Analytics Applications: Video can be streamed securely from camera-equipped devices at home or offices or public places to AWS using Amazon Kinesis. It will serve purposes such as security monitoring, machine learning, face detection, playback, and other various analytics. Evolving from Batch to Real-Time Analytics: Data that has been traditionally analyzed using batch processing can be performed on real-time analytics using Amazon Kinesis. Common streaming use cases involve streaming extract-transform-load, sharing data between different applications, and real-time analytics. Explore More About Amazon Kinesis by reading the following post. All You Need To Know About AWS Kinesis Conclusion Amazon Kinesis comes with extraordinary features and capabilities of supporting Kinesis Data Streams, Kinesis Video Streams, Kinesis Data Analytics, and Kinesis Data Firehose. Kinesis Data Streams are durable and scalable real-time data streaming services that can frequently abduct gigabytes and terabytes of data per second from a hundred and thousands of sources such as financial transactions, social media feeds, and operating logs. Kinesis Video Streams securely streams video from connected devices to Amazon Web Services (AWS) for artificial intelligence, machine learning, or other analytics processing applications. P.S. Amazon Kinesis Data Firehose captures, transforms, and loads the streams into Amazon Web Services (AWS) data stores. It performs real-time analytics with the help of existing Business Intelligence Tools.

Amazon Kinesis: The AWS Data Streaming Solution

The business impact companies are making with is driving investment in digital transformation across the board. Faced with multiple waves of disruption in a COVID-19 world, almost 92% of companies are reporting plans to spend the same or more on data/AI projects, according to a recent survey from . Small wonder. Data mature companies business-critical benefits from using big data, including: big data analytics NewVantage Partners are citing Informed decision-making Cost reduction Better understanding of customers New product development Data monetization Offensive drivers such as new competitive advantages, innovation, and transformation override defensive ones as change is becoming the only market constant nowadays. Let’s explore what business benefits exactly companies are achieving with to edge out the competition. big data But a few words about big data first Big data came into prominence a decade ago, with an explosive growth in data generated every second in the new world of digital everything. A quick fact: we created roughly 2.5 quintillion data bytes daily in 2020, with almost 90% of all current data generated in the last couple of years. Combining structured, semi-structured, and unstructured data (imaging, text, videos, social media posts, sensor data, etc.), big data is not just big but complex for traditional analytics systems. This is perhaps a reason that has never been analyzed for insights or larger trends (another quick fact). and new techniques, however, have made it possible to unearth hidden patterns and correlations in the massive volumes of data companies have been collecting for years. And big data has started talking, and making big business impacts too. 60% to 70% of all data within organizations Advanced AI solutions So, what is the impact of big data on business today? As all kinds of companies are trying to ride the big data wave for various business benefits, big data adoption doesn’t come easy. This is because just like anything else with big data, are big as well, from poor data quality to silos, lack of coordination, and business misalignment. There are a few success stories, though. Dive in. challenges Retail & Ecommerce With heightened customer expectations, the rise in online shopping and returns, and Amazon being a major competitor almost to everyone, increasingly more retailers are relying on big data analytics to capture a business impact. bring leading companies a lot of business benefits, including the ability to win customers with personal recommendations, forecast demand and revenue, improve day-to-day operations, and prevent fraud. Big data/AI technologies in retail Take Walmart, a leading retailer in the world that’s been on a transformational journey for a few years. Putting data democratization at its heart, Walmart has set up an analytics hub to provide employees with controlled access to data from hundreds of sources. Data-driven teams can now find quick solutions, like it was with for a particular item. Data manipulation helped them discover that the product price was uncompetitive due to simple miscalculations, and take prompt action. a grocery team puzzled by a drop in sales Major retail stores use video and data mining techniques to understand customer behavior patterns. This provides them with insights into questions about dead spots in the store ignored by customers, optimum staffing, say, on a Friday night, products that are in demand in the hot weather, and many, many others. Using market basket analysis to explore customers’ demographics and purchasing patterns, retailers are making their products look like a total steal with adjusted prices and layouts. analytics enabled by CCTV Ecommerce sites, on the other hand, can better target users with product recommendations by tracking clickstream data and other customer activity. The data gives them insights into channels driving the most traffic, pages or products attracting the most visits, who their most profitable customers are, and where the site might be losing users. By applying text mining to queries, they also help customers find the desired product much faster. Big data analytics empowers inventory management with end-to-end visibility. A 360-degree view of stock helps prevent missed opportunities that stem from product unavailability, non-synchronized stock levels across sales channels leading to overselling, or slow delivery. Amazon, for instance, has presented a patent for its ’ ‘ system that can predict what people want to buy before they even place their orders. anticipatory shipping Finding new customers has also been made easy with sophisticated big data/AI technologies when implemented in CRM systems. These platforms not only track unstructured email and calendar data but automate follow-up sequences and flag warm leads. Marketing “The aim of marketing is to know and understand the customer so well the product or service fits him and sells itself,” — Peter Drucker, a celebrated management consultant and educator. Marketing and sales teams are leading the way in big data analytics, making a great business impact with big data. The three types of data they are gathering — customer, financial, and operational data — enable a more accurate analysis of target audiences, financial performance, and business processes. Netflix is a great example of how data-powered advertising can fuel subscriber growth despite counterintuitive decisions like a VPN ban and a price hike in 2016. Those who are in the fan club know how they send you recommendations of the next show to watch. Netflix’s recommendation engines and new content production decisions rely on what titles people search or watch, time spent selecting movies, how often playback is stopped, ratings given, etc. Hollywood screenwriting star William Goldman said, “Nobody, nobody – not now, not ever – knows the least goddamn thing about what is or isn’t going to work at the box office.” Well, Netflix knows! Google and Facebook also know a few things about data-driven advertising, deriving huge business benefits from big data. By collecting and analyzing massive volumes of personal data we share, they serve better targeted and more personalized ads of products and services that we might be actually interested in buying. With the proliferation of smartphones, big data analytics has enabled hyper-localized advertising (which, let’s admit, is quite creepy). Capturing location and demographics data, analytics platforms can send shoppers near the store an ad offering discounts or other initiatives. Another good example of big data and its business impacts in marketing is the coupon story of Kroger. The US largest grocery network mails highly personalized coupons to shoppers, reporting within the next six weeks. Studying each customer’s DNA rather than segmenting them demographically, Kroger offers coupons for what people actually like, meaning that a Coke drinker is unlikely to be embarrassed with a Pepsi coupon offer ever. redemption rates at more than 70% Healthcare by storm, helping predict epidemics, cut down treatment costs, avoid preventable conditions, and improve the overall quality of our lives. Operational improvements are also on the list of sought-after big data business benefits. Insights are flowing in from multiple sources that include clinical data, electronic patient records, payer records, patient portals, results of medical examinations, wearables, smartphones, genetic databases, and other. Big data analytics is taking healthcare Electronic health records (EHR), perhaps the widest application of big data in healthcare, give health professionals access to complete information on a patient, including demographics, medical diagnoses, prescriptions, allergies, etc. EHR systems improve care through automation and better collaboration between providers, sending screening or vaccination reminders or alerts of contraindicating conditions, abnormal lab results, or drug interactions. They empower , with examples here featuring an ML-driven predictive analytics tool built by to accelerate antibiotic treatment and reduce sepsis mortality using EHR data. predictive analytics in healthcare North Oaks Health System Big data business benefits in healthcare include optimized staffing levels. A group of French hospitals, for instance, are using an . The platform can predict emergency department visits and admissions for the next 15 days, powered by a database on more than 470,000 patient encounters and additional data on external factors such as weather, holidays, and flu incidence that correlate with hospital visits. analytics platform to address the overcrowding problem Another important area where big data is making a great impact in healthcare is epidemic management. During the 2014-16 Ebola virus outbreak in West Africa, researchers turned to data on the location and frequency of calls to helplines to map the population’s movements in the region. A similar mobile phone data analysis was used after the Haiti earthquake in 2010 for targeting relief operations and predicting the spread of the subsequent cholera outbreak. Big data research is making big strides in improving the detection and treatment of certain conditions. Examples include technology company that has created a massive library of clinical and molecular data to enrich context on each patient’s cancer case for more personalized treatment options. In the dermatology field, modern AI algorithms trained to detect benign and malignant skin lesions of board-certified clinicians. The automatic classification of skin lesions based on images is deployable on smartphones, holding the potential to increase low-cost access to vital diagnostic care. Tempus have achieved the accuracy Finance The increasing use of data is that will be shaping the banking sector in the next decade. For banks and other financial institutions, the impact of big data on business centers around new levels of security and better customer loyalty driven by personalized offerings. named among the major trends Better customer service empowered by big data analytics is on the list of top priorities for the Royal Bank of Scotland (RBS). The RBS analyzes packaged bank accounts to detect whether a customer might be paying twice for the same service and uses analytics to spot customers who could save money from consolidating their loans. Their systems also send deal renewal reminders to mortgage holders when they approach an automatic transfer to a more expensive variable rate. Fraud prevention is another common application of big data in the banking sector. Lloyds Banking Group, for instance, creates what it describes as audio fingerprints of customer calls by analyzing unique call features (location, background noise, calling history, etc.). Meanwhile, American Express a 100% improvement in its resolution rate for digital fraud and a 21% reduction in POS disruption since adopting AI-powered data analytics in 2014. AmEx also uses big data for new customer acquisition via online channels and the development of apps linking cardholders with merchants on personalized offers. is reporting Big data delivers big impacts when it comes to risk management in banking. Success stories include Singapore-based that has reduced the time cycle of calculating the value at risk (the risk of loss for investments) from hours to minutes and that has achieved greater accuracy in financial risk prediction with the help of automated pattern recognition. United Overseas Bank (UOB) Morgan Stanley Education Big data/AI technologies are , making it possible to analyze the student’s performance and behavior in real time, improve teaching approaches, and reinvigorate entire education systems. revolutionizing education By capturing data such as test scores, response times to questions, or questions that students fail or answer successfully, big data analytics systems allow educators to personalize the learning process and improve students’ results. With real-time assessment tools and sophisticated applications, Ashford School in the UK, for instance, has not only created a more optimal learning environment but . reduced overall IT costs Big data technologies are also used to monitor students’ facial expressions and other biometric data via cameras or wearables to gauge their learning engagement or boost safety ( behind this big data application is delicate though). the ethics Assisting students with a career choice is another important application of big data in education. College planning platforms like help students understand fields they are interested in, track progress toward their educational goals, and follow schools of their liking. Overgrad Ending note Many companies across industries have woken up to big data technologies and business impacts they are delivering. The game is on, and more is coming. According to , the big data analytics market should reach $103 billion by 2023. Meanwhile, global spending on big data, , is projected to grow at a CAGR of 13.2% between 2020 and 2022. This means competition will get bigger. Don’t hesitate, go big too. Wikibon reported at $180 billion in 2019 If you don’t know how to get started with a big data journey in your industry or are stuck in the middle, big data experts. They will be happy to share their experience and expertise. contact ITRex

The Big Impact of Big Data on Businesses Today

An in-memory data grid (IMDG) is a set of networked/clustered computers that pool together their random access memory (RAM) to let applications share data with other applications running in the cluster. Though IMDGs are sometimes generically described as a distributed in-memory data store, IMDGs offer more than just storage. IMDGs are built for data processing at extremely high speeds. They are designed for building and running large-scale applications that need more RAM than is typically available in a single computer server. This enables the highest application performance by using RAM along with the processing power of multiple computers that run tasks in parallel. IMDGs are especially valuable for applications that do extensive parallel processing on large data sets. What Is an In-Memory Data Grid? Where would you draw the line between a data store and a cache? Persistence? Hazelcast allows you to write your in-memory data on disk. Derived data vs. source of truth? If the cost of creating the data is cheap, why would you persist them? Let’s agree that there’s no clear-cut defining property but a blurry continuum between those two concepts. The for caches is two-fold: performance, data is available faster, and availability, data is “always” available (at least the availability is higher than the underlying store). raison d’être Therefore, it stands to reason queries would benefit from the same properties. You may argue that caches are key-value stores: they shine when retrieving an entry by its key. However, if you execute a query, you’d need to iterate entry by entry, something akin to a full table scan in a SQL database. The performance would be abysmal, even more so if your cache is distributed like Hazelcast. And yet, sometimes, the underlying store offers no straightforward query mechanism either. Think about a Kafka topic or a web service. In this post, I’d like to describe how you can take advantage of Hazelcast to query your cached data in different ways and still be fast. Our Use Case: Collecting User Interactions Collecting user interactions with the application is the first step when you want to improve your application. Whether it’s documentation to make it more relevant or e-commerce to increase your sales, so-called “clickstreams” can be seen as the new oil. We will use a (significantly) simplified click-collecting application architecture: A web page with JavaScript code that sends click events to the server The server part is a Python Flask application. It enriches the events and stores them in a Hazelcast cluster. The Hazelcast cluster itself. It provides querying capabilities. A third-party component to query the cache data. In a real-world setup, we probably would want to store the event data in a persistent store, , a Kafka topic or a MongoDB instance. We would then capture the inserts via a Jet pipeline to load the Hazelcast cluster. In the context of this post, it would only make the architecture unnecessarily complex. e.g. We can tentatively model the structure of an event with the following attributes: The timestamp of the event The date on the client machine; it might differ from the previous data. The screen coordinates The source component ID The IP of the client machine The related session Anything else that might be relevant When users interact with the application, the front-end will send a JSON payload that contains client-related data to the server. The server will also enrich the JSON with server-related data, IP, session ID, and store the complete JSON in a Hazelcast IMap. The key is not that important; the value is the JSON. Now, our marketing team wants to query the data to understand how users interact with our application. The Predicate API The Predicate API is the one that predates all other ways presented in this post. The idea behind it is that it will serve as a filter before returning the values of an IMap. Before going further, we need to understand what it means to query a distributed system. From the : documentation The  requested predicate is sent to each member in the cluster. Each member looks at its own local entries and filters them according to the predicate. At this stage, key/value pairs of the entries are deserialized and then passed to the predicate. The predicate requester merges all the results coming from each member into a single set. That being said, it’s time to query! Here’s the code to select all values that match a component’s name: entry = Predicates.newPredicateBuilder().getEntryObject(); predicate = entry.get( ).equal(name); Collection<String> values = map.values(predicate); var var "component" // 1 // 2 name is the component's ID values contain all values that match the component's name The above is Java code, but do offer the Criteria API. all of our clients Unfortunately, if you run the code as-is, values will be empty even though some values match. The reason is that by default, Hazelcast stores values as bytes arrays. Thus, it has no understanding of the underlying format. The fastest way to fix the issue is to wrap the JSON value into a object. HazelcastJsonValue On the “put” side, we need to replace the first line with the second one: analytics.set(uuid.uuid4(), json.dumps(data)).result() analytics.set(uuid.uuid4(), HazelcastJsonValue(data)).result() # 1 On the query side, the only change is to update the generic type of the collection: Collection<String> values = map.values(predicate);
Collection<HazelcastJsonValue> values = map.values(predicate); The Shape of Stored Using means we are still storing data as JSON-formatted Strings. While some language ecosystems favor the usage of JSON, , JavaScript, some others favor the usage of dedicated data structures, , Java and Go. If developers on different language stacks use your Hazelcast cluster, you need to store the data in the most “consumable” shape. HazelcastJsonValue e.g. e.g. Enters Portable - another data serialization format that is query-friendly. With , you need to Portable Design a class or a struct (for Go) that models a unit of data Write down how to store and load an instance in an IMap Let's keep the storage part in Python and the query part in Java. Here's the Python code: hazelcast.serialization.api Portable dic: self.timestamp = dic[ ]
        self.event = dic[ ]
        self.instant = dic[ ]
        self.component = dic[ ]
        self.session = dic[ ]
        self.client = dic[ ]
        self.value = dic.get( , )
        self.type = dic.get( , )
        self.x = dic.get( , )
        self.y = dic.get( , ) writer.write_long( , self.timestamp)
        writer.write_long( , self.instant)
        writer.write_utf( , self.event)
        writer.write_utf( , self.component)
        writer.write_utf( , self.session)
        writer.write_int( , self.client[ ])
        writer.write_int( , self.client[ ])
        writer.write_int( , self.client[ ])
        writer.write_int( , self.client[ ]) self.value :
            writer.write_utf( , str(self.value)) self.type :
            writer.write_utf( , self.type) self.x :
            writer.write_int( , ) :
            writer.write_int( , self.x) self.y :
            writer.write_int( , ) :
            writer.write_int( , self.y)


factory = { : Analytics
} from import : class Analytics (Portable) : def __init__ (self, dic=None) if not return 'timestamp' 'event' 'instant' 'component' 'session' 'client' 'value' None 'type' None 'x' None 'y' None : def get_class_id (self) return 1 : def get_factory_id (self) return 1 : def write_portable (self, writer) 'timestamp' 'instant' 'event' 'component' 'session' 'client.left' 'left' 'client.height' 'height' 'client.top' 'top' 'client.width' 'width' if is not None 'value' if is not None 'type' if is None 'x' -1 else 'x' if is None 'y' -1 else 'y' 1 On the Java side, we can avoid creating the class by using GenericRecord. As its name implies, it’s generic. Hence, we can query the data without having the definition of a Java Analytics class on the classpath! Again, the change is straightforward: Collection<HazelcastJsonValue> values = map.values(predicate); 
Collection<GenericRecord> values = map.values(predicate); In both cases, you can access a field by its name: values.stream()
        .map(value -> value.getString( ))
        .forEach(System.out::println); "component" Indexing Data Indexing data is not (only!) related to databases and how one store's data on disk. If you already know about indexes, please feel free to skip this introduction. Key-Value stores, such as Hazelcast’s IMap, provide fast key-based access: that's their . However, to query the value - or part of it, the engine needs to: raison d'être Go through each of the entry in turn Check whether the predicate applies And eventually, add the entry to the collection of matching entries While it’s faster to execute the flow in-memory than reading from a disk, it’s not optimal. The solution is to create a dedicated data structure that stores additional information: an index. You can imagine an index as a sort of Key-Value store. The key is the value of the attribute, and the value the collection of keys from entries whose attribute’s value matches the value. Now, when we query for entries whose value match “foo”, the engine will look at the index, locate the key “foo”, read the value, and look for entries whose keys are “a”, “c” and “e”. Of course, this is a pretty simple example. Since Hazelcast allows you to store complex values such as JSON and Portable, you can set an index on any attribute (or nested attribute). Here, you’d be able to set an index on the “comp” attribute or the “t” attribute. You should carefully evaluate on which attribute to set an index, though, as indexes have two main downsides: An index slows down storing values as the engine needs to maintain the index. An index-only makes sense if a query is using it. With the above IMap, if the predicate is entry.get("component").equal("button"), but the index is on "t", it won't improve performance. The above explanation caters to exact matches. Imagine that we want now every entry whose “t” is above a certain threshold, , . For this usage, we need a index. A sorted index keeps its keys, well, sorted. The engine will go through the keys in order, and once it has reached a key that doesn't match the predicate, it will return the results. e.g. entry.get("t").lessThan(2) sorted Finally, for queries that involve multiple attributes, we need a compound index that contains all attributes. Here’s such a query: map = client.getMap( ); entry = Predicates.newPredicateBuilder().getEntryObject(); predicate = entry.get( ).equal(name)
                     .and(entry.get( ).lessThan(instant)); var "analytics" var var "component" "instant" The relevant index configuration would be: hazelcast: map: analytics: indexes: - type: HASH attributes: - "component" - type: SORTED attributes: - "instant" At this point, we know how to query an IMap with the Criteria API and make it fast with the proper index configuration. SQL and JDBC As a Java developer, you might already be familiar with SQL and JDBC. Learning a new API might feel like a roadblock. The good news is that Hazelcast now offers a JDBC driver. To use it: Add the driver to your classpath (here with Maven) com.hazelcast hazelcast-jdbc 4.2 < > dependency < > groupId </ > groupId < > artifactId </ > artifactId < > version </ > version </ > dependency Use the standard JDBC API: ( connection = DriverManager.getConnection( ); statement = connection.prepareStatement( )) {
    statement.setString( , name);
    statement.setLong( , instant); resultSet = statement.executeQuery(); modelBuilder = TableModelBuilder<>(); (resultSet.next()) { component = resultSet.getString( );
         System.out.println(component);
    }
} try var "jdbc:hazelcast://localhost:5701/" var "SELECT * FROM analytics where component = ? and instant < ?" 1 2 var var new while var "component" That’s all! The icing on the cake, you can use SQL in other language stacks (without JDBC, obviously). At the time of this writing, it still uses part of the Criteria API, but we intend to move it outside. The above code can be rewritten in Python like the following: analytics = client.get_map( )
select = .format(component = component, instant = instant)
results = analytics.values(sql(select)).result() result results:
    print(result.component) 'analytics' 'component = {component} AND instant < {instant}' for in Conclusion In this post, we went through several core concepts covering querying your data on Hazelcast: It makes sense to query your in-memory data because data access is fast By default, Hazelcast offers the Criteria API; you can use SQL as an alternative Use indexes judiciously to speed up your queries The complete code for this post is available on . GitHub Originally published at https://hazelcast.com on May 12, 2021.

How You Can Take Advantage of Hazelcast to Query Your IMDG Data

. SQL is great, but sometimes you may need something else By and large, the prevalent type of data that data engineers deal with on a regular basis is . Tables in a data warehouse, transactional data in (OLTP) databases — they can all be . But does it mean that NoSQL is irrelevant for data engineering? In this article, we’ll investigate for which . relational Online Transactional Processing queried and accessed using SQL use cases data engineers may need to interact with NoSQL data stores The reasons behind NoSQL These days, data is stored at such a ( ) that many relational database systems can’t keep up. Historically, this was the main reason why large tech companies developed their own NoSQL solutions to mitigate those issues. In 2006, Google published their paper which laid the foundations for the open-source NoSQL data store and GCP’s . In 2007, Amazon offered their alternative solution with paper. velocity, volume, and variety in short: Big Data Bigtable HBase Cloud Bigtable Dynamo From that time on, other NoSQL distributed database systems kept emerging. All of them have been mainly trying to mitigate the issue of scale ( ) that is hard to achieve with the traditional vertically-scalable RDBMS. Instead of scaling vertically , they operate on horizontally-scalable distributed clusters allowing them to increase the capacity by simply adding more nodes to the cluster. data volume (buying more RAM and CPU for your single server) Apart from scalability and addressing the sheer data volume, many distributed NoSQL database systems address the challenge of data coming in many , i.e., , which is difficult to achieve in relational databases. I’ve seen many creative solutions in my career when people tried to serialize and blob non-relational data to a relational database. Still, such self-built systems tend to be hard to operate and manage at scale. with such as MongoDB allow much more flexibility in that regard. different formats (highly nested JSON-like structure, XML, time-series, images, videos, audio…) variety Document databases schema-on-read Finally, with the of data, continuously streaming in real-time, many NoSQL databases help address that issue. For instance, handles data streaming from a variety of IoT devices in a very intuitive and easy way. velocity Amazon Timestream Why do I need to know NoSQL data stores as a data engineer? One of a data engineer responsibilities is and data from various sources and providing it to data consumers consistently and reliably so that this data can be used for analytics and building data products. This means building data pipelines that ( ) and store it in a data lake or data warehouse. Due to the above-mentioned data , data lakes are particularly useful for storing data from NoSQL data stores. integrating consolidating pull data from among others NoSQL databases variety NoSQL use cases for data engineering Now that we know why NoSQL is important for data engineering let’s look at the typical use cases that we may encounter. 1. Analysis of log streams These days, many applications are sending logs to Elasticsearch and can be analyzed and visualized using Kibana. Thanks to the dynamic schema and indexing, Elaticsearch is a handy NoSQL data store in your data engineering toolbox, especially for container monitoring and log analytics. 2. Extracting time-series data from IoT devices and real-time applications Relational database systems are usually not intended to be used with thousands of open connections used simultaneously. Imagine that you manage a large fleet of servers or IoT devices that keep streaming metrics such as CPU utilization to some centralized data store so that this information can be used for a real-time dashboard and for anomaly detection analysis. If you would use a relational database for that, you would likely encounter connection issues ( ) since RDBMS is not designed for thousands of short-lived connections. In contrast, NoSQL distributed data stores such as can handle that easily. (showing the health of the system) leading to some data loss AWS Timestream 3. Extracting data from NoSQL backend systems NoSQL databases are used as a backend data store for so many applications, ranging from e-commerce, content management platforms (blogs), mobile apps, websites, web analytics, clickstream, shopping carts, and many more. All of this data can be used for analytics once you established data pipelines that extract or stream this data into your data lake or data warehouse. 4. Caching data for fast retrieval If you’ve ever built a dashboard with some BI tools such as Tableau, you probably preferred working with extracts rather than with live data, provided that the dataset was fairly large. Making connections to relational databases and waiting till they process and retrieve the data is often too slow to build really performant dashboards ( ). For this purpose, in-memory NoSQL data stores such as are great. If you use those to that you need for your dashboards, you can make sure that you experience no lag and provide a good user experience. that don’t hang when you apply a filter Redis cache recent data Demo: extracting data from NoSQL Let’s demonstrate , one of the most popular ( ) NoSQL data stores on AWS. I created a table called demo with partition key index. This index will be equivalent to my Pandas dataframe’s index. how we can extract data from DynamoDB serverless Now we can use a combination of boto3 and awswrangler to load a sample dataset. Then, starting from line 34, we build a simple ETL to retrieve data that has been inserted since yesterday. Additionally, we could use the PartiQL query editor in the management console to validate that our ETL returns correct data. DynamoDB PartiQL query editor — image by author How to keep an eye on your DynamoDB resources? To monitor the read and write capacity units used by our table, it’s helpful to make use of an observability platform that provides you more details about your infrastructure and your applications’ health. For example, with you can track any that occurred when reading or writing your data, the average of your read and write operations, whether a continuous is enabled for a quick , and how many are consumed by your resources. Dashbird errors latency backup point-in-time recovery read (RCU) and write capacity units (WCU) The “demo” DynamoDB table in Dashbird — image by author The average latency in your read and write operations observed using Dashbird — image by author Drawbacks of NoSQL databases The previous sections demonstrated the use cases where NoSQL data stores shine in data engineering. But now to the drawbacks. The major drawback of NoSQL data stores is that , the developers need to some proprietary vocabulary, API, or interface to access data. Timescale DB put it nicely on their blog: without a full-fledged SQL interface learn ( again ) “SQL is back. Not just because writing glue code to kludge together NoSQL tools is annoying. Not just because retraining workforces to learn a myriad of new languages is hard. Not just because standards can be a good thing. But also because the world is filled with data. […] Either we can live in a world of brittle systems and a million interfaces. Or we can continue to embrace SQL.” — Source Even though many NoSQL data stores offer SQL interfaces on top, they will where you can create complex queries ( ). never be as powerful as a full-fledged relational database not that complex queries are always a good thing So purely from a , it’s best to those NoSQL systems . Use them when there is a use case that clearly justifies the need for NoSQL’s scale and flexibility coming from schema-on-read. data engineering perspective work with rather than against them If data comes in a format that is not suitable to be stored in a data warehouse, we can always extract it, store it in a data lake ( ), and . These days, many tools allow us to make sense of data stored in a data lake. When leveraging serverless SQL query engines such as Presto ( ) or BigQuery, you can analyze even semi-structured nested data. ELT-approach provide it for analytics in its native raw format or AWS version: Amazon Athena Conclusion To answer the question from the title: yes, NoSQL data stores are important for data engineering. Even though they often lack a friendly SQL-like interface to retrieve the information you need, there are still very beneficial in various use cases. Given that for data retrieval ( ), we should expect that the trend will continue. It’s possible that one day, we may be able to write SQL queries to retrieve data the same way regardless of the backend data store under the hood. In fact, if we look at federated queries with Presto, this future may not be that far away. more and more vendors realize the need for a SQL interface such as PartiQL from AWS, the “good old” Hive, Spark SQL, CQL in Cassandra, SQL queries in Timestream Previously published at https://dashbird.io/blog/nosql-database-data-engineering/

Are NoSQL databases relevant for data engineering?

The popularity of Apache Kafka is not backing down. If anything, its rate of adoption has been something to behold. To put things visually, here are Google search trends for Kafka globally, over the last six years: If you haven’t gotten your head around Kafka yet, you probably should. To get you going, I have put together a list of well-respected, proven sources of knowledge on Kafka — both free and paid—that should get you started and well on your way to becoming a Kafka aficionado. Books If you want to get properly good at something, you need to invest in training and preparation. Everyone’s experience will vary, but I found I learned the most by reading books that were written by experts. Effective Kafka: A Hands-On Guide to Building Robust and Scalable Event-Driven Applications Author: Emil Koutanov The use of the word “effective” in a title of a book is normally reserved for the most authoritative texts in their respective fields. For example, “Effective C++” by Scott Meyers or “Effective Java” by Josh Bloch. They are the absolute cream of the crop. So, does the book hold up to the title? Effective Kafka is the absolute bible on Apache Kafka. I don’t want to overstate how good this book is, but it would be hard to. To start with, is not a small book, at nearly 500 pages. But it reads quite well and is intelligently structured, with good grammatical style, a well-thought-out progression and lots of good analogies and illustrations. Effective Kafka The book is a complete guide on the core technology platform and is suitable for all levels of skill — from beginner to intermediate and advanced. It starts by assuming you know nothing and explains the core concepts with lots of diagrams. It then covers off architecture and design considerations, client and broker configuration, operational concerns, security and transactions. The book is a good blend of theory and hands-on practicality, with lots of snippets and examples in Java. I found it to be very up to date, covering off all Kafka features that I knew of and some that I didn’t. If you are only going to buy one book this year, this is it. Reasons to buy: In-depth coverage of all core concepts and supplementary areas Architectural insights and design best-practices Suitable for beginner to advanced levels Up-to-date coverage of all Kafka features Solid value for money and longevity Reasons to pass: None that come to mind Link to on the author’s site. Effective Kafka Kafka: The Definitive Guide — Real-Time Data and Stream Processing at Scale Authors: Neha Narkhede, Gwen Shapira, Todd Palino is one of the pioneering books from none other than the founding maintainers of the platform. Kafka: The Definitive Guide The book is great at explaining the key concepts behind Kafka in a way that is easy to follow and understand. It’s an excellent book for both absolute beginners and intermediate users alike, and is a fantastic complement to the official documentation. The writing style is simple and concise. The book has a ton of practical examples and technical illustrations. There are some negatives too. The book is around four years old now and is beginning to show its age. Some of the new features, such as transactions, are not covered. The book sticks to the core topics well but at the expense of coverage of the more advanced topics, such as security. Reasons to buy: Concise delivery and comprehensive coverage of core concepts Excellent for beginners and intermediate users Good value for money Reasons to pass: Somewhat dated, missing out on new features and tech Not suitable for advanced users Link to on O’Reilly Kafka: The Definitive Guide Streaming Architecture: New Designs Using Apache Kafka and MapR Streams Authors: Ted Dunning, Ellen Friedman is an alternate take on the Apache Kafka literature. Where other books present Kafka is a generic building block, Dunning and Friedman have decided to zero in on the stream processing use case, showcasing not only Apache Kafka but also MapR Streams, which is a complementary technology. Streaming Architecture This book is ideal if you have a very specific set of use cases in mind, rather than looking to get into Kafka. Perhaps your work is heavy on stream processing; for example, analytics, complex event processing, windowed aggregation, clickstream analysis, and so on. is an excellent opportunity to kill two birds with one stone. (No animals were harmed…) Streaming Architecture When considering this book, bear in mind that this is an introductory text. It is not a deep dive by any stretch and you will likely need to buy another book or take a course to fill in the gaps. It doesn’t make it a bad book because of the two-in-one (Kafka and MapR) proposition, but it is a niche book nonetheless, which may not be exactly what you are looking for. Reasons to buy: Focuses on map-reduce style stream processing Touches on the negatives of Kafka as well as the positives Good use of practical examples to convey the authors’ knowledge Reasons to pass: Light on content (only 120 pages) Not suitable beyond a beginner level There are better value alternatives Link to on O’Reilly. Streaming Architecture Tutorials While there are some terrific books on Kafka, it doesn’t mean that the free blogs and articles are necessarily bad. Nor does it mean that all books are great—there are certainly a fair number of appalling ones on Amazon. I’ve included three popular articles on Kafka that should get you started. They are purely introductory, so don’t expect to come out an expert. Kafka in a Nutshell is exactly what the name suggests. It is a super-concise breakdown of Apache Kafka. What’s more, it’s written by Emil Koutanov—the author of and the maintainer of . Kafka in a Nutshell Effective Kafka Kafdrop The article includes a bit of history and some typical use cases and patterns. It then looks at the architecture of the core components —brokers, ZooKeeper nodes, producers and consumers, and how they inter-relate. For its size, the quality of content and the delivery is excellent. galops through the core concepts, such as topics, partitions, offsets, consumer groups, order of records, parallelism and at-least once delivery. Sure, it does not have the depth of the book, but it gives you a basic understanding. Kafka in a Nutshell The article also covers tooling, which is something that is often overlooked. There are examples that use the command-line interface to publish and consume messages, so it is a practical “getting started” guide as much as a theoretical primer. also touches on the drawbacks of Kafka and some of the gotchas that practitioners should be aware of. It also talks about and how Kafka achieves its throughput. There is also mention of geographical replication, multi-tenancy and security. Kafka in a Nutshell performance Overall, has got to be the best all-rounder—a great way to get your feet wet without spending a penny. Kafka in a Nutshell Apache Kafka: Getting Started is a highly respected introductory guide to the platform, written by one of the authors of Kafka: The Definitive Guide — Gwen Shapira. Apache Kafka: Getting Started The article does the whole “getting started” thing in a slightly untraditional way — by starting with the examples first, then explaining what they do and how the various components fit into the broader picture. If there is one bit of criticism is that the article lacks a clear structure; Gwen jumps from one concept to another in an attempt to cover ground but the result may sound incohesive at times. The article does not attempt to be a complete introduction unlike, say, . It tries to be more hands-on. It gives you a bit of theoretical knowledge and follows with links to other useful resources. One such resource we will look at next. Kafka in a Nutshell The Official Documentation Why go far looking for blogs and other freebies when you have the at your disposal? Official Kafka Documentation If you’re a seasoned Kafka developer, you have probably read over the official docs countless times. It is an excellent reference guide. But if you are still finding your bearings, the documentation can be a very valuable resource. The “Getting Started” section in official docs gives a compelling overview of the platform. It starts with an overview of and some of its potential use cases. It then presents Kafka in light of event streaming, which is also a good differentiator from message queues. event streaming Having introduced Kafka, the guide then covers some of the main concepts and terminology, such as producers, consumers, topics and partitions. One advantage you get with the official docs is the assurance that they are correct and thorough. Beyond a short introductory section, the rest of the documentation reads like a user guide, which is basically what it was designed for. Kafka is a highly flexible event streaming platform that brings about its own set of concepts and principles. It is not the easiest piece of technology to conquer but the effort will be worth your while. Being a fairly mature platform, it is not short on documentation and other sources of knowledge, but there is also a lot of fluff out there that isn’t helpful for someone just starting out. I hope this list has given you some pointers in the right direction. Also published on Medium .

3 Top Resources To Learn About Apache Kafka

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka. With all of these stream processing and real-time data store options, though, also comes questions for when each should be used and what their pros and cons are. In this post, I’ll discuss some common real-time analytics use-cases that we have seen with our customers here at Rockset and how different real-time analytics architectures suit each of them. I hope by the end you find yourself better informed and less confused about the real-time analytics landscape and are ready to dive in to it for yourself. First, an obligatory aside on real-time analytics. Historically, analytics have been conducted in batch, with jobs that would run at some specified interval and process some well defined amount of data. Over the last decade however, the online nature of our world has led rise to a different paradigm of data generation in which there is no well defined start or end to the data. These unbounded “streams” of data are often comprised of customer events from an online application, sensor data from an IoT device, or events from an internal service. This shift in the way we think about our input data has necessitated a similar shift in how we process it. After all, what does it mean to compute the min or max of an unbounded stream? Hence the rise of real-time analytics, a discipline and methodology for how to run computation on data from real-time streams to produce useful results. And since streams also tend have a high data velocity, real-time analytics is generally concerned not only with the correctness of its results but also its freshness. Kafka fit itself nicely into this new movement because it is designed to bridge data producers and consumers by providing a scalable, fault-tolerant backbone for event-like data to be written to and read from. Over the years as they have added features like Kafka Streams, KSQL, joins, Kafka ksqlDB, and integrations with various data sources and sinks, the barrier to entry has decreased while the power of the platform has simultaneously increased. It’s important to also note that while Kafka is quite powerful, there are many things it self-admittedly is not. Namely, it is not a database, it is not transactional, it is not mutable, its query language KSQL is not fully SQL-compliant, and it is not trivial to setup and maintain. Now that we’ve settled that, let’s consider a few common use cases for Kafka and see where stream processing or a real-time database may work. We’ll discuss what a sample architecture might look like for each. Use Case 1: Simple Filtering and Aggregation A very common use case for stream processing is to provide basic filtering and predetermined aggregations on top of an event stream. Let’s suppose we have clickstream data coming from a consumer web application and we want to determine the number of homepage visits per hour. To accomplish this we can use Kafka streams and KSQL. Our web application writes events into a Kafka topic called clickstream. We can then create a Kafka stream based on this topic that filters out all events where and applies a sliding window with an interval of 1 hour over the stream and computes a . This resulting stream can then dump the emitted records into your sink of choice– S3/GCS, Elasticsearch, Redis, Postgres, etc. endpoint != '/' count(*) Finally your internal application/dashboard can pull the metrics from this sink and display them however you like. Note: Now with ksqlDB you can have a materialized view of a Kafka stream that is directly queryable, so you may not necessarily need to dump it into a third-party sink. This type of setup is kind of the “hello world” of Kafka streaming analytics. It’s so simple but gets the job done, and consequently it is extremely common in real-world implementations. Pros: Simple to setup Fast queries on the sinks for predetermined aggregations Cons: You have to define a Kafka stream’s schema at stream creation time, meaning future changes in the application’s event payload could lead to schema mismatches and runtime issues There’s no alternate way to slice the data after-the-fact (i.e. views/minute) Use Case 2: Enrichment The next use case we’ll consider is stream enrichment– the process of denormalizing stream data to make downstream analytics simpler. This is sometimes called a “poor man’s join” because you are effectively joining the stream with a small, static dimension table (from SQL parlance). For example, let’s say the same clickstream data from before contained a field called . countryId Enrichment might involve using the to look up the corresponding country name, national language, etc. and inject those additional fields into the event. This would then enable downstream applications that look at the data to compute, for example, the number of non-native English speakers who load the English version of the website. countryId To accomplish this, the first step is to get our dimension table mapping to name and language accessible in Kafka. Since everything in Kafka is a topic, even this data must be written to some new topic, let’s say called . Then we need to create a KSQL table on top of that topic using the KSQL DDL. countryId countries CREATE TABLE This requires the schema and primary key be specified at creation time and will materialize the topic as an in-memory table where the latest record for each unique primary key value is represented. If the topic is partitioned, KSQL can be smart here and partition this in-memory table as well, which will improve performance. Under the hood, these in-memory tables are actually instances of RocksDB, an incredibly powerful, embeddable key value store created at Facebook by the same engineers who have now built Rockset (small world!). Then, like before, we need to create a Kafka stream on top of the Kafka topic. Let’s call this stream . Then using some SQL-like semantics, we can define another stream, let’s call it which will be the output of the join between that Kafka stream and our Kafka table from above. clickstream S T For each record in our stream , it will lookup the in the Kafka table we defined and add the and fields to the record and emit that record to stream . S countryId countryName language T Pros: Downstream applications now have access to fields from multiple sources all in one place Cons: Kafka table is only keyed on one field, so joins for another field require creating another table on the same data that is keyed differently Kafka table being in-memory means dimension tables need to be small-ish Early materialization of the join can lead to stale data. For example if we had a userId field that we were trying to join on to enrich the record with the user’s total visits, the records in stream would not reflect the updated value of the user’s visits after the enrichment takes place T Use Case 3: Real-Time Databases The next step in the maturation of streaming analytics is to start running more intricate queries that bring together data from various sources. For example, let’s say we want to analyze our clickstream data as well as data about our advertising campaigns to determine how to most effectively spend our ad dollars to generate an increase in traffic. We need access to data from Kafka, our transactional store (i.e. Postgres), and maybe even data lake (i.e. S3) to tie together all the dimensions of our visits. To accomplish this we need to pick an end-system that can ingest, index, and query all these data. Since we want to react in real-time to trends, a data warehouse is out of question since it would take too long to ETL the data there and then try to run this analysis. A database like Postgres also wouldn’t work since it is optimized for point queries, transactions, and relatively small data sizes, none of which are relevant/ideal for us. You could argue that the approach in use case #2 may work here since we can set up one connector for each of our data sources, put everything in Kafka topics, create several ksqlDBs, and set up a cluster of Kafka streams applications. While you could make that work with enough brute force, if you want to support ad-hoc slicing of your data instead of just tracking metrics, if your dashboards and applications evolve with time, or if you want data to always be fresh and never stale, that approach won’t cut it. We effectively need a read-only replica of our data from its various sources that supports fast queries on large volumes of data; we need a real-time database. Pros: Support ad-hoc slicing of data Integrate data from a variety of sourcesAvoid stale data Cons: Another service in your infrastructure Another copy of your data Real-Time Databases Luckily we have a few good options for real-time database sinks that work with Kafka. The first option is Apache Druid, an open-source columnar database. Druid is great because it can scale to petabytes of data and is highly optimized for aggregations. Unfortunately though it does not support joins, which means to make this work we will have to perform the enrichment ahead of time in some other service before dumping the data into Druid. Also, its architecture is such that spikes in new data being written can negatively affect queries being served. The next option is Elasticsearch which has become immensely popular for log indexing and search, as well as other search-related applications. For point lookups on semi-structured or unstructured data, Elasticsearch may be the best option out there. Like Druid, you will still need to pre-join the data, and spikes in writes can negatively impact queries. Unlike Druid, Elasticsearch won’t be able to run aggregations as quickly, and it has its own visualization layer in Kibana, which is intuitive and great for exploratory point queries. The final option is Rockset, a serverless real-time database that supports fully featured SQL, including joins, on data from a variety of sources. With Rockset you can join a Kafka stream with a CSV file in S3 with a table in DynamoDB in real-time as if they were all just regular tables in the same SQL database. No more stale, pre-joined data! However Rockset isn’t open source and won’t scale to petabytes like Druid, and it’s not designed for unstructured text search like Elastic. Whichever option we pick, we will set up our Kafka topic as before and this time connect it using the appropriate sink connector to our real-time database. Other sources will also feed directly into the database, and we can point our dashboards and applications to this database instead of directly to Kafka. For example, with Rockset, we could use the web console to set up our other integrations with S3, DynamoDB, Redshift, etc. Then through Rockset’s online query editor, or through the SQL-over-REST protocol, we can start querying all of our data using familiar SQL. We can then go ahead and use a visualization tool like Tableau to create a dashboard on top of our Kafka stream and our other data sources to better view and share our findings. For a deeper dive comparing these three, check out this . blog Putting It Together In the previous sections, we looked at stream processing and real-time databases, and when best to use them in conjunction with Kafka. Stream processing, with KSQL and Kafka Streams, should be your choice when performing filtering, cleansing, and enrichment, while using a real-time database sink, like Rockset, Elasticsearch, or Druid, makes sense if you are building data applications that require more complex analytics and ad hoc queries. You could conceivably employ both in your analytics stack if your requirements involve both filtering/enrichment and complex analytic queries. For example, we could use KSQL to enrich our clickstreams with geospatial data and also use Rockset as a real-time database downstream, bringing in customer transaction and marketing data, to serve an application making recommendations to users on our site. Hopefully the use cases discussed above have resonated with a real problem you are trying to solve. Like any other technology, Kafka can be extremely powerful when used correctly and extremely clumsy when not. I hope you now have some more clarity on how to approach a real-time analytics architecture and will be empowered to move your organization into the data future. Previously published at: https://rockset.com/blog/ksql-stream-processing-real-time-databases-analyze-streaming-data-in-kafka/

Using KSQL Stream Processing & Real-Time Databases to Analyze Kafka Streaming Data [A How-To Guide]

What if your best customers come through a channel you overlook? Maybe it wasn't the right decision to turn off that hight traffic Google Ads campaign because of no conversions? Those visitors might get back to your website a few times more and eventually buy your stuff. Maybe it's just you didn't know it? So if you want to measure the real effectiveness of your marketing dollars spent, keep on reading. We'll break down what is marketing attribution modeling, what types are available, and which one might work for you. What is marketing attribution modeling Marketing attribution modeling is an organized set of rules that attribute credits for different touchpoints across the customer journey. In other words, a way for you to measure the real impact of your marketing activities. Why you're putting yourself at risk by not having marketing attribution modeling in place Some paid marketers I know are obsessed with optimizing their campaigns for higher CTR and lower CPC. Those who look a bit further are optimizing for conversions instead, which seems logical. But conversions don't always impact main targets in a way you think. Not all customers have equal , and not all campaigns are converting from the first touchpoint. I would even say, not every touchpoint is designed to convert. But you still need to measure its effectiveness. For B2B, these rules are especially true. LTV So you can continue optimizing your Facebook ad campaign for the best-in-class CTR or conversion rate. For sure, it may bring you hundreds of dollars plus. But on the scale of a business, you might be doing a tiny thing that doesn't make a difference at all. Your company's success is your success. So you should optimize marketing towards the primary company goal - revenue. And if you've never thought this way, you definitely should start. How digital marketing attribution modeling works You diversify your revenue streams to reduce risks, right? You have multiple channels and probably hundreds of campaigns running. And you need to understand their performance at scale. So attribution software pulls the data from the channels you use and combine it into one customer journey, assigning credits to each interaction between your marketing and a customer. Technically it happens with the use of IDs, cookies, clickstream data, etc. This data is usually siloed across the platforms, and you cannot analyze it holistically. So having it pulled up is a good thing anyway. Examples of marketing attribution modeling Imagine you have a customer journey with a Google ad as the first touchpoint. By clicking the ad, people start engaging with your brand on social, webinars, whatever. And eventually, after a month or so, they type in your website address and buy your product. Most likely, you'll attribute revenue to direct or organic traffic and forget about what happened during this journey. But it's like getting your salary and saying it was not essential to do the job to get it. So you can end up turning off a campaign that brings you customers just because it has a low conversion rate. But it can drive awareness like crazy. The thing is nobody cares. During our project with Betty Bossi ( case study here ), there were multiple channels and even website pages tracked as a part of the customer journey. So each touchpoint was considered. And one essential and surprising insight was gained from the analysis. The Google Ads branding campaigns turned out to be invaluable in attracting customers who then return over organic channels. This was not visible using the standard, simplistic last-click attribution models. Think about it. Is marketing attribution modeling only for paid channels? No, it is not. With marketing attribution modeling tools, you can pull the data from your ad accounts, social media accounts, email provider, website, programmatic ad network, influencer marketing campaigns, etc. It even works with offline channels and TV advertising - it is called "Mix Modeling." It's all about getting a full picture of your customer journeys and giving credits to marketing activities that deserve it. Why Google Analytics is not always the answer Many marketers use Google Analytics as a hub of their cross-channel marketing data. Google has some to help you measure the performance of your marketing channel. But with adding more channels, like programmatic advertising, it becomes painful to measure the impact. common attribution models Sometimes, it makes much more sense to have all costs and conversions easily accessible in a dashboard and different attribution models compared on the same screen. This way, you have much clearer and actionable data, save time, and even headcount. Of course, if you do it right. How to choose the right attribution model for your marketing To get all the benefits of marketing attribution modeling, you need to understand your goals and have a firm grip on the customer journey. Below you'll find 11 attribution models with our recommendations on when to use and to avoid each of them. 1. First Click Attribution Model The first click attribution model works on a simple formula. It attributes a conversion to the source that referred a buyer to your website for the first time. Example: A prospective visitor sees your Google ad and lands on your website for the first time. They might browse around, signup to your email newsletter, follow your Facebook page, and leave. After a few days, they come back and purchases your product. Despite no immediate purchase and interactions with other channels, the credit goes to the Google ad as it was the first touch of your brand with the client. Use first click attribution model: To identify channels that bring in the most leads or customers. To determine the best channel for creating awareness of your brand. If you have minimal marketing channels (i.e., focusing only on Facebook and Google Ads). Avoid first click attribution model: If you have multiple marketing channels and a sophisticated customer journey. This model gives no value to other touchpoints. If your customer buying cycle is longer than 30 to 90 days. Google Analytics cookies expire within that period, and you can lose the data. 2. Last Click Attribution Model In the last click attribution model, all credit goes to the marketing touchpoint that resulted in conversion. Example: You knew Apple for years from their ads and word of mouth. But you bought a new Mac during the Black Friday sale. According to last click attribution, the Black Friday campaign gets all the credits for the conversion. Use last click attribution model: To understand high-impact touchpoints at the bottom of your funnel. It helps you find channels that are driving the most conversions. Avoid last click attribution model: If you have a complicated customer journey that requires research before making a buying decision. With this model, you neglect other critical touchpoints and limit your understanding of marketing performance. 3. Last Non-Direct Click Attribution Model In this attribution model, the significance is given only to the last non-direct marketing channel. Google Analytics considers as Direct traffic when a user types in your website address into a search bar. Any traffic that has no referral folds into this bucket as well. Example: In other words, it's like saying that the user who typed in yourwebsite.com into search and purchased your product, made the decision beforehand. So you're giving all the credits to the touchpoint that came before the direct visit resulted in conversion. Use last non-direct click attribution model: To understand the effectiveness of your marketing tactics, without accounting for the direct traffic. Avoid last non-direct click attribution model: If you have offline campaigns that lead a user to the website to convert. In this case, "direct" could be the only touchpoint with your brand online. 4. Linear Attribution Model The linear attribution model gives the same credit to every channel across the whole customer journey from the first interaction to making a purchase. Example: Someone sees your ad on Facebook and clicks it to visit the website. (33.3%)A person comes back later, directly typing your website address to get more info and compare your products with the competition. (33.3%)A few days later, they click on your retargeting ad and buy. (33.3%) According to the linear attribution model, you'll have 33.3% attributed to each of the touchpoints. If the user directly visited your website 6 times, 75% of the value will go to direct traffic. So remember, EVERY touchpoint is credited. Important note: Use linear attribution model: To understand and find channels that result in brand awareness and conversion. To identify channels that are consistent across all customer journeys, especially the ones driving actual conversions. Avoid linear attribution model: If you already have a decent understanding of your customer journey and need to identify critical touchpoints. Equal value to each touchpoint is not what you're looking for in this case. You have a journey where most of the decision making happens within a particular touchpoint like having a 1-to-1 workshop alongside social media. The magic happens during a conversation. 5. Position-Based Attribution Model (U-shaped Attribution Model) The position-based or U-shaped attribution model gives a certain percentage to all the touchpoints contributing towards conversion. The first and last touchpoints get 40% of the credit each, and the rest of the touchpoints divide the remaining 20%. Example: Some guy decides to purchase an insurance policy online and starts researching it on Google. He lands your website from the SERP. (40%) He finds you on Facebook and clicks on the article from the feed. (10%) In a few days, he comes back to your website directly to read a few more articles and leaves an email for a brochure. (10%) He eventually converts after seeing your Facebook retargeting ad. (40%) In that case, organic search and a Facebook ad will get the 40% of credit each, while the rest of the touchpoints will share the remaining 20% among them. Use position-based attribution model: To understand which channel is best for acquiring an audience and which one is excellent for conversions. If you have the buying cycle within the expiration of the GA cookies (30 to 90 days). Avoid position-based attribution model: If you heavily rely on nurturing campaigns. If you have a long decision-making cycle. You can lose first touch data due to cookie expiration and end up acting on misleading data. If you're in the e-commerce business and have many seasonal campaigns. In this case, the first touchpoint will most likely get too much credit assigned compared to the influence it made. 6. W-Shaped Attribution Model In the W-shaped attribution model, 90% of the credit is equally divided among the three touchpoints: First Touch Lead Creation Conversion And the remaining 10% is distributed among other channels within the journey. The W shaped attribution model is an enhanced version of the position-based or U-shaped attribution model. Example: Let's take the U-shaped case above. 90% would be divided among steps 1, 3, and 4. And the rest will go to step 2. Some guy decides to purchase an insurance policy online and starts researching it on Google. He lands your website from the SERP. (30%) He finds you on Facebook and clicks on the article from the feed. (10%) In a few days, he comes back to your website directly to read a few more articles and leaves an email for a brochure. (30%) He eventually converts after seeing your Facebook retargeting ad. (30%) Use W-shaped attribution model: For identifying the touchpoints that result in an action. It removes focus from additional channels and helps you identify your audience builder, lead generator, and conversion creator channels. If you're in B2B marketing with clear funnel stages your customers go through. Avoid W-shaped attribution model: If you're in B2C marketing with short sales cycles and user journeys, not necessarily including a lead generation step. 7. Time Decay Attribution Model In this attribution model, the preference goes to the channels that are closer to the conversion point. Example: You're going to the kitchen from the living room. With the time decay attribution model, your first steps would be much less valuable than the last few before entering the kitchen. The same applies to marketing touchpoints. The first website visit has the least value, while the touchpoint that led to the conversion will be the most important one. Use time decay attribution model: To understand which channels are assisting and driving conversions in B2C space or B2B with short customer journeys. Avoid time decay attribution model: If you're focusing on driving awareness. If you have long B2B sales cycles where the decision is usually made before the last touchpoint, which becomes a formality. 8. Last Google Ads {any other platforms} Click Attribution Model This attribution model focuses only on the performance of Google Ads campaigns. This model gives all credit to the last Google Ads campaign on a journey or a Google ad that directly led to a conversion. The same applies to other channels like Facebook, Twitter, LinkedIn, etc. But be careful with this attribution model for evaluating multiple channels. If you have Google Ads and Facebook Ads within one journey, each of them will attribute conversion (the same one) to their channel. This means you'll meet a problem of duplicated conversions and having misleading data in your reports. Use last Google Ads click attribution model: To determine which ad campaign is driving the best results for your business.To identify top Google ad keywords driving the conversions. Avoid last Google Ads click attribution model: If you have a complicated customer journey with valuable touchpoints within. If you want to evaluate the performance of multiple paid channels at a time. 9. Lead Conversion Touch Attribution Model The lead conversion touch attribution model gives all the credit to the channel through which the lead was generated. Example: It is widely applicable for a B2B scenario when a visitor becomes a lead but doesn't convert into a customer. Let's say you landed in this article and found out the benefits of attribution modeling. You didn't become a paying customer right away but were eager to learn more, went to website and signed up for a trial. windsor.ai At this stage, you become a lead, and if you convert into a customer, all the credits will go to the channel resulted in a free trial - this article. Use Lead Conversion Touch Attribution Model: If you need to identify touchpoints or channels that are driving leads. Avoid Lead Conversion Touch Attribution Model: If you have a long nurturing process with multiple activities. However, if your nurturing funnel works nice and all you need is to bring more leads to the pipeline, the lead conversion touch model is a nice place to go. 10. Full-Path (Z-Shaped) Attribution Model In the full-path ( Z-shaped) attribution model, 90% of the credit is equally divided among the four primary channels: The First touch (22.5%) Lead Generation (22.5%) Opportunity Creation (22.5%) Customer Close (22.5%) While the remaining 10% is shared among the rest of the channels. Example: You are the owner of an online training course website. Someone Googles for a training course and lands on your website to find your available courses. First touch: After a few days, they come back again and downloads an E-book on how to learn efficiently. Lead generation: You email them and offer a free class for their preferred courses. The person accepts it. Opportunity Creation: After the free class, you send them an email with a 20% discount on that course. The person buys it and becomes your customer. Customer Close: In this scenario, the full-path attribution model will give equal credit (22.5%) to all channels that resulted in action till customer closing. The remaining 10% goes to all other channels within the journey. Use full-path (Z-shaped) attribution model: To evaluate which marketing channels are driving actions towards closing a deal. If you have a comprehensive customer journey and strong alignment between marketing and sales teams. Avoid mull-path (Z-shaped) attribution model: If you have a simple customer journey. If your sales team works on closing opportunities without marketing support. 11. Custom, Algorithmic, or Data-Driven Attribution Model Custom, Algorithmic, or Data-Driven attribution model performs an in-depth analysis of the customer journey. It identifies all marketing channels playing a significant role in bringing visitors to your website and converting them into customers. Compared to other models' focus on extremes, this model uses algorithms to give each channel the credit it deserves. It helps you better evaluate the performance of each click and interaction. Example: There is no defined process for the data-driven attribution model as it varies from organizations to the marketing processes. It means you don't have to choose but create a model that fits your needs. For this, you need to review your past marketing performances, identify your end goals, and evaluate each channel on par with effectiveness in accomplishing those goals. When to use and to avoid custom attribution model: Ideally, every business should use a data-driven attribution model as it gives the most credit to your best channels and eliminates underperforming ones. But to have a data-driven attribution model, you need the help of a pro data scientist and crunchers at your hand. If you have the resources, budget, and time to create a data-driven attribution model, you should ditch all other attribution models against it. And similarly, avoid a data-driven attribution model, when you can't bear the burden. Summary Each marketing attribution model solves its purpose. While the custom model gives the most advanced insights, it's not always the best solution to spend recourses on building it. Star with a clear goal, define your channels, and then choose one of the models that fit your case.

Hidden Benefits of Digital Marketing Attribution Modeling: Improve Your 2020 Numbers

Stories, News & Mentions about CLICSSTORE

Clicsstore’s Stories on HackerNoon