Snowplow was founded with the belief that data teams should spend their time innovating, not extracting and wrangling behavioral data from CDP’s or analytics platforms

Snowplow

It’s remarkable how so many things are made better with great searches. Google made it easy for normal folks to find whatever they needed online, no matter how obscure. IDEA’s fuzzy matching and symbol search helped programmers forget the directory structure of their code bases. AirTag added an advanced spatial location to my cat. A well-crafted discovery feature can help add that “wow” factor that iconic, habit-forming products have. In Search of Meaning is a search method for surfacing highly relevant results based on the of the query, context, and content. It goes beyond simple keyword indexing or filtering. Semantic Search meaning It allows users to find things more naturally and with better support nuance than highly sophisticated but rigid traditional relevancy methods. In practice, it feels like the difference between asking a real person or talking to a machine. Tech companies from all over the world are racing to add these capabilities to their existing products. Instacart on how they added semantic deduplication to their search experience. published an extensive article Examples of companies implementing some form of semantic search include eBay, Shopee, Ikea, Walmart, and many others. The reason for this rush towards semantic search is simple: more relevant results = happier customers = more money. Discovery, relevancy, and trustworthiness are some of the hardest problems to solve in e-commerce, and an entire ecosystem exists to help companies solve them. Vectors to the Rescue There is an emerging group of highly capable semantic search SaaS offerings. A prime example is Algolia’s NeuralSearch - if you want a top-notch, batteries-included system that will take care of most of the complications of implementing search right, this is a great place to look. Sadly, you are going to pay - a lot. This might be OK for a low to medium-traffic site or a POC, but do your math before you fully commit to them. Don’t worry though: you can still create an awesome semantic search experience even if you have a more down-to-earth budget. It will just require a bit of doing. Many companies working on semantic search today are using - a way of representing meaning as vectors. document embeddings Since semantic search alone may not be able to provide enough relevant hits, traditional full-text indexing is used as a backup method. A feedback loop is added to track user interactions, and use them to provide super relevant results through result re-ranking. This is what the architecture looks like: This system has three key processes: indexing, querying, and tracking. is done by converting a document’s content to an embeddings vector through a text-to-vector encoder (ex: OpenAI’s API) and inserting it into a Vector Database (ex; Qdrant, Milvus, Pinecone, etc.). Indexing Embeddings Documents are also indexed in a traditional full-text search engine (ex: Elasticsearch). This combination is usually referred to as “hybrid search.” relies on encoding incoming queries into vectors (preferably using the same encoder as the previous step) and querying the vector database using them. These results are then combined with traditional full-text results and re-ranked for relevancy. Querying Search re-ranking is usually a complex problem, and often relies on a mix of machine learning and heuristics. involves capturing important user interactions - ex: clicking on results, liking items, etc. - and using these events to update the machine learning models involved in re-ranking. Tracking This provides a feedback loop that uses user input to continuously improve relevancy. Snowplow is an example of a capable tracking system. What’s Next? If you have the budget for a SaaS solution, then congratulations: you are well on your way to impressing your users with a spanking new search function. If, like most of us, you are not made of money, then it’s time to roll up your sleeves. Implementation can be a daunting challenge. If you need any help, I to help you get started. In either case, you should seriously consider whether your users could benefit from semantic search. wrote about the subject It’s a hard problem to solve, but the upside is definitely there and users are getting more used to this raised bar every day. Happy searching!

Searching for Meaning: What Real World Semantic Search Looks Like

The data warehouse is the foundation of the modern data stack, so it caught my attention when I saw Convoy head of data declare, “ ” on LinkedIn. Chad Sanderson the data warehouse is broken Of course, Chad isn’t referring to the technology, but how it’s being used. As he sees it, data quality and usability issues arise from the conventional best practice of “dumping” data in the warehouse to be manipulated and transformed afterward to fit the needs of the business. This is not out of line with the general efforts of providers like Snowflake and Databricks to ensure their customers are being efficient (in other words, saving money and resources) in their storage and consumption. Whether or not you agree with Chad’s approach detailed below, what can’t be disputed is how his opinions have generated a tremendous amount of debate. “One camp is mad at me because they think this is nothing new and it requires long manual processes and data architects with 30 years of experience. The other camp is mad at me because their modern data stack is fundamentally not set up this way and it isn’t how they have been building out their data products,” said Chad. I’ll let you decide for yourself if the “immutable data warehouse” (or active vs passive ETL) is the right path for your data team. Either way, I’m a strong proponent that moving our industry forward will require more than overviews of technologies such as data warehouses and , but frank discussions and unique perspectives on how to deploy them. data observability platforms I’ll let Chad take it from here. How an Immutable Data Warehouse Combines Scale and Usability A perspective from Chad Sanderson The modern data stack has many permutations, but the data warehouse is a foundational component. To oversimplify: Data is extracted via passive pipelines (really just the “E” in ETL) and dumped into… A data warehouse where it is processed and stored before it is then… Transformed into the format needed by data consumers for… A specific use such as analytic dashboards, machine learning models, or activation in systems of records such as Salesforce or Google Analytics… With technologies or processes such as data observability, governance, discovery, and cataloging running across the stack. Before diving into the challenges of this approach, and a suggested alternative, it’s worth exploring how we arrived at what we define as “the modern data stack. How did we get here? In the early days of data, with pioneers such as , the original ETL ( ) process involved extracting from the source and transforming it before landing in the data warehouse. Bill Inmon extract, transform, load Many businesses still operate this way today. For large companies where data quality is paramount, this process involves a manual, intensive governance framework with a tight coupling between data engineers and the data architects embedded across different domains in order to leverage data quickly for operational insights. Tech giants like Google, Facebook, and others ditched this process and started dumping virtually everything in the data warehouse. The ROI of logically organizing the data just wasn’t as high for rapidly growing startups as this much faster, more scalable process. Not to mention, loading (the “L” in ELT) had become much easier to integrate in the cloud. Along the way, popular made transforming data in the warehouse easier than ever. Modular code and dramatically reduced runtimes made the ETL model radically less painful…so much so the use of popular transformation tools expanded downstream from data engineers to data consumers such as data scientists and analysts. transformation tools It seemed like we had found a new best practice and we were on our way to a de facto standardization. So much so, that suggesting an alternative would generate swift and strong reactions. The challenge with passive ETL or transformations in the warehouse There are several problems with an architecture and process that heavily relies on transforming data once it has entered the data warehouse. The first problem is the disconnect, really chasm, it creates between the data consumer (analysts/data scientists) and the data engineer. A project manager and a data engineer will build pipelines upstream from the analyst, who will be tasked with answering certain business questions from internal stakeholders. Inevitably, the analyst will discover that data will not answer all of their questions and that the program manager and data engineer have moved on. The second challenge arises when the analyst’s response is to go directly into the warehouse and write a brittle 600 line SQL query to get their answer. Or, a data scientist might find the only way they can build their model is to extract data from production tables which operate as the implementation details of services. The data in production tables are not intended for analytics or machine learning. In fact, service engineers often explicitly state NOT to take critical dependencies on this data considering it could change at any time. However, our data scientist needs to do their job so they do it anyway and when the table is modified everything breaks downstream. The third challenge is when your data warehouse is a dumping ground, it becomes a data junkyard. An from the Hadoop era found between 60% and 73% of all data within an enterprise goes unused for analytics. A more found 68% of data available to the enterprise goes unused. older Forrester study recent Seagate study As a result, data scientists and spend too much of their time searching for context in an overly processed production code haystack. As data engineers, we need to emphasize data in addition to data quality. analysts usability If  your users can’t find and leverage what they need reliably in your current data warehouse, what’s the point? Another approach: introducing the immutable data warehouse The immutable data warehouse concept (also referred to as active ETL) holds that the warehouse should be a representation of the real world through the data instead of a tangled mess of random queries, broken pipelines, and duplicated information. There are five core pillars: . For businesses to truly gain value from the massive amounts of data they possess, teams need to take a step back and model their business semantically before defining entities and events through code for the express purpose of analytics. This can be an iterative process starting with the most crucial elements of the business. The business is mapped and owners are assigned An entity relationship diagram (ERD) is a map of the business based on the REAL world, not what exists in the data warehouse or production databases today. It defines the critical entities, their relationships (cardinality, etc), and the real world actions that indicate they have interacted. An engineering owner is established for each entity and event. End-to-end automated lineage can help establish the ERD and make it actionable. . Perhaps the most controversial tenant is that data should bubble up from business needs instead of trickle down from unstructured pipelines. Instead of data analysts and scientists combing through the dusty shelves of your warehouse to see if there is a data set to what they need, no data enters the warehouse unless it is directly requested and defined by the data consumer first. Data consumers define their needs upfront and contracts are created close enough No data enters the warehouse without a business question, process, or problem driving it. Everything is purpose built for the task to be done. It’s essential this process is designed to be simple as data needs are always changing and increased friction will threaten adoption. At Convoy, implementing a new contract takes minutes to hours not days to weeks. Next, it’s time to draw up the data contract, which is an agreement between the business and engineering leads about what the schema of an event/entity should be and the data that is needed most for that asset to be most effective. For example, perhaps an existing inboundCall event is missing an OrderID which makes it difficult to tie phone calls to completed orders. SLAs, SLIs, and SLOs are one type of to this model of change management and stakeholder alignment. data contract you can apply . In the same way we need a peer review process for code (GitHub) or UX (Figma) that hits production, there should be an equivalent for data assets. However, the right level of abstraction for this review is not code – but semantics. Peer reviewed documentation within an active environment That review process should have the same outcome as a GitHub pull request – version control, sign-off of relevant parties, etc – all handled through the cloud. By applying modern, cloud based technologies we can speed up old processes making them far more viable for even the fastest growing internet businesses. There is a place for data catalogs as a pre-data warehouse definition surface, but the challenge is there is no carrot and no stick for data consumers to keep metadata current. What is the incentive for a data scientist that uses an ELT process and finishes their model to go back and document their work? . Engineers then implement the data contracts within their service. The data is piped into the data warehouse, and with the modeling metadata can ideally be automatically JOINed and categorized. Data is piped into the warehouse pre-modeled as defined in the contract. Transformations take place upstream from the consumption layer (ideally in the service) . A transactional outbox pattern is used to ensure the events in the production system match what’s in the data warehouse while a log and offset processing pattern ( ) protects against data loss. Together, these two systems ensure data is preserved with complete integrity so the immutable data warehouse is a direct representation and source of truth for what is occurring across the business. An emphasis is placed on preventing data loss as well as ensuring data observability, integrity, usability, and lifecycle management which we use extensively at Convoy Data quality and usability require . Data quality is a technical challenge for the most part. Think “back-end” engineering. Data usability on the other hand, is a “ ” engineering challenge that requires the same skill set used to create great customer experiences. Lastly, an immutable data warehouse does not lend itself to petabyte measuring contests and whipping out your big data stats. Deprecation and maintenance is just as important as provisioning. two different mindsets front-end This approach leverages the advantages of technology to achieve the best of both worlds. The governance and business driven approach of traditional approaches with the speed and scalability associated with the modern data stack. How an immutable data warehouse works. Treating data like an API. Let’s start by reviewing the full stack surrounding the immutable data warehouse. Unlike traditional warehouses, a descriptive layer moves the business logic above the services layer and puts the data consumer in the driver’s seat. The consumer is able to provide their requirements without the need for technical skills as the data engineer serves as a crucial requirement to code translator. These contracts can be held in a data catalog or even a general document repository. 1. Descriptive Layer: The warehouses function primarily as a ‘data showcase’ and the underlying compute layer. 2. Data Warehouse: Data consumers build data products that are validated and shared with the business. Assets in the semantic layer should be defined, versioned, reviewed, and then made available through an API for consumption in the application layer. 3. Semantic Layer: This is where data is used to accomplish some business function, such as experimentation, machine learning, or analytics. 4. Application Layer: Solutions that support data operations across the data stack such as data observability, catalogs, testing, governance, and more. The ideal is to have perfect, pre-modeled, highly reliable data once it hits the warehouse, but you still need to cover all the permutations the real world may throw at you (and have enforcement mechanisms when processes move out of bounds). 5. End-To-End Support: The immutable data warehouse itself is designed for streaming – it’s easier to go from streaming to batch data than vice versa–and therefore fed by three different types of APIs. : This API is for semantic real world service level events that are the core building blocks of the company, not events from front end applications. For example, in Convoy’s case this could be when a shipment is created or goes off hold. Events from the real world are built in the service code, not SQL queries. Semantic Events API Data consumers do not need to see all production tables, particularly when they’re simply implementation details of the data service they’re using to generate insights or power decision making. Instead, when properties of a data asset in a production table are updated, , an API wrapper or abstraction layer (for instance, dbt)  will expose the CRUD concepts that are meaningful to data consumers in the warehouse – for instance, whether or not the data is fresh or row volume is within expected thresholds. CRUD Abstraction API: : There are many tools that already handle front-end event definition and emission like Snowplow, Mixpanel, and Amplitude. That being said, some front-end events are important enough that teams need to be able to ensure their delivery and integrity using the long offset pipeline. In some cases, front-end events are critical for machine learning workflows and “close enough” systems just won’t cut it. Frontend API There also needs to be a mapping layer that sits outside of the warehouse as things change (perhaps one service needs to become many) or if a schema a data scientist has in mind does not fit with what is happening in the real world. Mapping should be handled either upstream of the warehouse through a streaming database or in the warehouse itself. This layer is where a BI engineer matches what is coming up from engineering to what a data consumer needs, which can be automated to produce . Kimball data marts Immutable data warehouses have challenges too. Here are some possible solutions. I am under no delusion that an immutable data warehouse is a silver bullet. Like any approach it has its pros and cons, and is certainly not for every organization. Like the data mesh and other lofty data architectural initiatives, the immutable data warehouse is an ideal state and rarely the reality. Achieving one – or attempting to achieve one – is a journey and not a destination. Challenges that should be considered and mitigated are: The upfront costs of defining the descriptive layer Handling entities without clear ownership in place Implementing new methods to enable rapid experimentation While there is a cost to defining the descriptive layer, it can be greatly accelerated through software and done iteratively by prioritizing the most important business components. This needs to be a collaborative design effort that includes data engineers to prevent the diffusion of data quality responsibility across distributed data consumers. It’s OK if you don’t get it right the first time, this is an iterative process. Handling entities without clear ownership can be a tricky governance problem (and one that is frequently bedeviling data mesh proponents). It’s not typically in the data team’s purview to sort these issues on the business side. If there is a core business concept that crosses multiple teams and is generated by a monolith rather than microservice, the best way forward is to have a strong review system in place and a dedicated team standing by to make changes. Data engineers can still be allowed to experiment and given flexibility without limiting workflow. One way to do this would be through a separate staging layer. However, API data from these staging areas should not be allowed to be consumed downstream or across external teams. The key is that when you move from experiment to production or make it accessible to the border team, it must go through the same review process. Just like in software engineering, you can’t make a code change without a review process just because you want to move faster. Wishing you luck on your data quality journey There are many permutations of the modern data stack, and as an industry, we’re still going through an experimentation phase to understand how to best lay our data infrastructure. What’s clear is that we are rapidly moving toward a future where more mission critical, external facing, and sophisticated products are” powered by” the data warehouse. Regardless of the chosen approach, this will require us as data professionals to raise our standards and redouble our efforts toward reliable, scalable, usable data. Data quality must be at the heart of all data warehouses, no matter the type. The bottom line from my perspective: when you build on a large, amorphous foundation, stuff breaks and it’s hard to find. And when you do find it, it can be hard to figure out exactly what that “thing” is. Immutable or not, maybe it’s time we try something new. Also Published Here

Is The Modern Data Warehouse Dead?

Introduction The worst nightmare of analytics managers is accidentally blowing up the data warehouse (DWH) cost. This could happen by accidentally running a query that scans the entire raw table that is huge. Even when the table is not at the petabyte scale, the accumulation of repeated queries could add up to. The above scenario is especially true when you are using BigQuery, AWS Athena, or Redshift Spectrum, whose cost is tied to the total data scanned by queries. When you are billed by the computational capacity purchase as in Snowplow and Redshift, you may be shielded from such incidents, but your DWH performance gets severely impacted by costly queries. How can we avoid receiving unexpectedly expensive bills from the BigQuery/Athena/Redshift Spectrum? How can we keep our queries efficient, so we don't need to increase the over cost from DWH and operational inefficiency? This article introduces basic ideas that help you avoid the blowup and lower the overall cost. Enforce partitions to the raw tables The first thing you can do is to enforce partitions on the raw tables. The big tables are almost always tied to the day-to-day ingestion of the raw data. The DWH can internally divide the big data into chunks, typically based on one of the timestamp columns in the table. The smaller chunks in the table are called partitions. When you scan the partition with a specific range of timestamps, DWH scans only the partitions that fall into the range. That way, your query finishes fast with smaller data scanned. In most DWHs, you can set up to make the timestamp range always required by query to avoid scanning the entire table by accident. If you are a BigQuery user, loader does this automatically. We can also help you set up partitions for other DWHs. Learn more at handoff.cloud, or just let us know by dropping an email to hello@handoff.cloud. Note: ANELEN's target-bigquery-partition Set up materialized tables The raw tables are cleaned and aggregated before they are put to use. You may be doing so by defining SQL views. Such views may be hooked to your dashboard. So, every time someone opens the dashboard, the DWH may be executing a query through the views. You should pay extra attention to this if you are using dbt which encourages the use of views by design. Unless the view is configured to be automatically "materialized", or to cache the query result for the repeated requests, the view is tapping on the raw data each time someone sends the query. This could also cause a higher cost and performance issue. One explicit way of avoiding the costly scanning of raw tables is to periodically replicate the view result into a separate table. Let's call it materialized table. If your dashboard just needs daily updates, you can run a job to refresh the materialized table. Some DWHs such as BigQuery have schedule features so you don't have to do it manually. Otherwise, you can define a job in Airflow or dbt cloud. Use delta view to get fresh data efficiently One problem with the above approach is data freshness. Say, your data extractor (StitchData, Fivetran, Airflow, etc) pulls data from the source every hour but materialization happens daily basis. By accessing the materialized table, you only get yesterday's metrics and of course, you want to peek at today's intra-day progress. The so-called delta view will help you achieve that. A delta view combines the raw data and the materialized table to synthesize the most recent data efficiently. First, it pulls out the pre-aggregated data from the materialized table. Then it checks the latest timestamp of the pulled data. Using the timestamp, it pulls the "delta" by scanning the raw table with the timestamp. This way, the delta view utilizes the best of both the raw table and the materialized table. Here is an example dbt macro of delta view. The script also de-duplicates the rows based on the primary keys and the replication timestamp: {% macro events_delta(
    source_table,
    primary_keys,
    event_tstamp_key,
    etl_tstamp_key,
    all_keys=["*"],
    historical_table=None,
    current_date='current_date()',
    lookback=30)
%}
/*
This query block is created by events_delta macro.

1. events_delta checks the (hopefuly partitioned-) destination {{ destination }}
  table within the time-window {{ lookback }} (days) since the last ingestion 
  {{ etl_tstamp_key }}.
2. The new data is identified in source {{ source_table }} table having
  {{ etl_tsamp_key }} > the last destination ETL timestamp.
3. If the destination has an entry with the primary key, it will honor the
  existing one. So, to replace the old one, you need to drop the partition
  (or just that record) before running this query.

Assumption: event time is always older than the etl time

Usage:

{% set primary_keys = ["pk-1", "pk-2", "pk-3",...] %}
{% set all_keys = ["pk-1", "pk-2", "pk-3",..., "non-pk-1", "non-pk-2", "non-pk-3", ...] %}
WITH
source AS (
    -- I'm inserting a CTE to demonstrate pre-casting a field before calling the macro 
    select cast(timestamp as date) as date,
    * except(timestamp, date)  -- This is a BigQuery dialect
    from {{ source('my_schema', 'my_raw_data_table') }}
),
{{ events_delta(
    'source',
    primary_keys,
    'date',
    'my_etl_timestamp',
    all_keys=all_keys,
    historical_table=source('my_schema', 'my_historical_table'),
    lookback=30)
}}
select * from events_delta
union all
select * from source('my_schema', 'my_historical_table')
*/

_events_delta_new AS (
  SELECT
{% for key in all_keys %}
    n.{{ key }}{% if not loop.last %},{% endif %}
{% endfor %}
  FROM {{ source_table }} AS n
{% if historical_table %}
  LEFT JOIN (
    SELECT 
{% for key in primary_keys %}
      {{ key }}{% if not loop.last %},{% endif %}
{% endfor %}
    FROM {{ historical_table }}
    WHERE DATE_ADD({{ current_date }}, INTERVAL -{{ lookback }} day) <= CAST({{ event_tstamp_key }} AS DATE)
  ) AS p
      ON 1=1
{% for key in primary_keys %}
            AND n.{{ key }} = p.{{ key }}
{% endfor %}
{% endif %}
  WHERE DATE_ADD({{ current_date }}, INTERVAL -{{ lookback }} day) <= CAST(n.{{ etl_tstamp_key }} AS DATE)
    AND DATETIME_ADD({{ current_date }}, INTERVAL -{{ lookback }} day) < CAST(n.{{ event_tstamp_key }} AS DATETIME)
{% if historical_table %}
{% for key in primary_keys %}
    AND p.{{ key }} IS NULL
{% endfor %}
{% endif %}
),
events_delta AS (
  SELECT o.*
  FROM _events_delta_new AS o
  INNER JOIN (
    SELECT
{% for key in primary_keys %}
      {{ key }},
{% endfor %}
      MAX({{ etl_tstamp_key }}) AS {{ etl_tstamp_key }},
    FROM _events_delta_new
    GROUP BY
    {% for key in primary_keys %}
    {{ key }}{% if not loop.last %},{% endif %}
    {% endfor %}
  ) AS oo
ON o.{{ etl_tstamp_key }} = oo.{{ etl_tstamp_key }}
{% for key in primary_keys %}
  AND o.{{ key }} = oo.{{ key }}
{% endfor %}
)
{% endmacro %}
/* This is the end of the query block generated by events_delta macro */ You can use this delta view definition to update the materialized table. Then the updated materialized table is referred to by the delta view next time someone wants the up-to-date data. Seamlessly run extraction and materialization Delta view works by coordinating data extraction and materialization. You may do so by: Schedule data extraction by StitchData, Fivetran, or create a task in Airflow. Define delta views. We recommend managing it with dbt with a macro like in the example above. Schedule the materialization by DWH's scheduling feature if available or set up a job by yourself. These steps may sound complex, and the efficient coordination between the schedules at Step 1 & 3 could be tricky. handoff.cloud can run all 3 steps together whenever the data is replicated. This seamless delta view process resolves the coordination issues and generates the most up-to-date data with the lowest query cost. Plus, our fully customizable service can integrate data from the applications not supported by existing connectors in the market. We can also support more complex ELT logic and custom transformations. Learn more at Note: handoff.cloud Conclusion As your data and team grow bigger, the query cost and efficiency become a headache to analytics managers. The use of a partitioned table and the materialized table save you DWH cost and increase the cluster efficiency. Delta views are an efficient way of pulling up-to-date metrics while keeping the query cost low. About the author Daigo Tanaka is CEO and data scientist at ANELEN, a boutique consulting firm focused on data engineering, analytics, and data science. ANELEN’s mission is to help innovative businesses make smarter decisions with data science. ANELEN is the provider of . handoff.could ETL/ELT service Also published on: https://blog.handoff.cloud/Save-data-warehouse-cost-with-delta-view-and-partitioned-raw-table/

Cost Effective Data Warehousing: Delta View and Partitioned Raw Table

I introduce the concept of storing and processing data focussing primarily on user privacy in my book, “Data is Like a Plate of Hummus”. I know many of you have read it – perhaps you're even thinking about it now ahead of the upcoming changes to Apple’s privacy settings which will block the attribution of users without consent. The reality – unfortunately – is that most of us are still in the discussion phase, rapidly losing valuable time. In fact, I don't think many of us really understand enough about user privacy, instead hoping that a third party will solve the problem for us. There, I said it. Sigh. Why oh why did Apple make this change to begin with? Whose interest do they have in mind? The user? You, the client? Or themselves and their profits? you'll find yourself without data, without users and without a functioning business. Companies need to start playing the privacy game. I'm not just talking about saying “we're GDPR/CCPA/LGDP compliant” – no, I mean processing user data in-house, sending only what you MUST share to third parties and keeping as much user data to yourself as you possibly can. It doesn’t matter if you're an e-commerce company, a dating app or even a health app, if you haven't developed a first-party tracking tool by now, in the long-run, you'll find yourself without data, without users and without a functioning business. How many of you are scratching your heads right now saying "but we have lawyers and DPOs that said we were doing fine!"? How many of you didn’t even think to consult someone (even though your competition has made all the right inquiries)? How many of you had a vendor tell you "this is industry-wide practice!"? I hope you've heard about the “privacy by design” methodology at least. You can read its seven key principles . Let's focus on the seventh point (the one I believe is the most important): “respect for user privacy”. This is where you treat user privacy as the most important part of the product design process. I completely agree with this principle, even though I know many have criticized it for being too strict. here Let me give you my three tips for how you can safeguard your user's privacy and still enjoy a great flow of data – practices that I apply in each company I work with! Build a first-party tracker for your product (App/Web…) There are a variety of tools out there that help you track users without paying a third party to store and process user data for you. Instead, you actually own the tracker and store data on your own server. Snowplow is such a tool, but there are others on the market that can also enable your business to advance and allow you to take back control over user data without sacrificing your tracking abilities. Another advantage to this solution is that you will only need the user's consent for the data that you forward, so there will be less need for onboarding screens. Create three data layers Collect it, store it and allow access only to the teams or individuals that need access. Yikes! We're all aware of this best practice, so why are we still not abiding by it? Sometimes it's down to a lack of engineers and other times it's the result of a lack of knowledge. It’s super important to have as much data on your server as possible so you can analyze the user and better understand them. In that respect, I suggest you create the following three data layers: First layer, raw data layer, data lake, S3 bucket... call it whatever you want, but only data engineers and specific individuals will have access to it. You will need to write everything about your APIs, tracking information and basically any user information you collect into this layer. Create the second layer using an automatic ETL process only using the data that you really need. You should ensure that any personal data is hashed: country, city, user ID and any other types of identifiable data. In this way, the data on the second layer is for analysis purposes only and just contains pseudonymized data. This will ensure the user that nobody can access their private data. Now for the third layer. This is where you keep the sharable dashboards and reports that are already accessible throughout the company. But remember, this is precisely why you need to ensure that the data is anonymized and aggregated! It's not that complicated when you think about it, but truthfully, how many of you do it? Share only what you need How many of you have more than one SDK on your apps or more than one tracking tool on your site? How many services do you share your user information with? Not only is allowing third parties to collect your user data foolish – a mistake that can be used against you at a later stage – but it’s also completely mindless to slow your site down each time a user creates an action. So what should you do? Well, the best solution is to fire as many events as possible from your server to the third party server, but you need to make sure the events are hashed to avoid a data breach. In other words, you should only fire user’s events with the right consent, and this is how you do it: Firstly, you need to explain to your user how you process their data. You should tell them what information you share with third parties and you have to be crystal clear about it – no legal games that leave your user asking “Huh?!”. You need to come clean about what you send and how – in fact, the more you tell your user, the better. If they don't opt in, maybe there's a reason for it. It might even encourage you to rethink the way you approach the whole concept of privacy. Whilst we're doing a lot of things right, we are still way too obsessed with our business strategy and goals – we're so distracted that we forget to think about the user we serve! We need to start thinking about our user's privacy first and foremost and considering what they would say if we do X or Y. How would they feel about our decisions? At the end of the day, you are offering a service and you need to be able to track and trace your users so that you can adapt your service to them. If you can't do that, well, you'll soon find yourself in big trouble. How will you know what features your user likes and which ones they don't? How will you know what drives your profits and where you should direct your marketing efforts to attract more customers?

The Difference Between Privacy Talkers and Privacy Doers

Making a wedding cake out of a cupcake. Illustration: Adrian Hogan When presented with a customer problem to solve through technology, the age-old question when it comes to & product and one that can exert a heavy influence on the success of a company is whether to build, buy or partner. engineering technology From one perspective, if done right it can enable internal teams to move with speed. From another perspective it can enable 3rd party platform costs to be managed. And from a third perspective it can bring valuable IP in-house, creating competitive advantages that can create a moat around the product you’re building and preventing competitors from breaching the castle too easily. For me (with the loving backing of our finance team), the decision comes quite easily when determining whether to build or partner (we’re not at a point where we can buy) — is the purpose and feature set to the product and platform or not? If it’s not then we look to best-of-breed platforms to partner with and if it is, we use the fantastic team we have in-house to build, iterate on and lovingly tend to from time to time to ensure technical debt doesn’t accrue to a point where it slows us. core Examples of functionality/platforms we partner with include; (security and personal information storage is too important to do half-heartedly) — we chose Auth0 (soundbite why is ) Identity here Anything to do with (we’re not telcos) — we chose the Seranova platform which has Twilio under the hood telephony or SMS Anything to do with (it’s too important to trust home-grown tooling) — we’ve gone with New Relic and Logz.io primarily logging, alerting and monitoring (there is no need to re-invent the wheel) — Snowplow and Google Analytics 360 being the primary tools of choice Analytics (because machines do a much better job at being binary than humans) — we like Codacy Code quality tooling (because on-prem hosting is so passé) — we run our workloads on AWS Cloud computing While on the other side of the fence, we trust the team internally with the likes of; (which we see as a long term competitive advantage) Data architecture (so we can be masters of our own destiny and move with speed) — we use a mix of CircleCI, Helm, Kubernetes CI/CD — I’m a big believer that you can have 2 out of cheap, fast or good when it comes to coding and I’d much rather not do it on the cheap if it’s something core to the experience. The medium-long term return on investment far outweighs any short term tactical wins from doing something cheap and/or fast Coding I touched on it earlier that we make decisions with the loving backing of our finance team. In that sense we’re lucky that they back our judgement, that they trust us to know when building and maintaining something is going to be detrimental in the long run. This piece from Intercom’s has one of my favourite phrases of all time: blog “ Features are like children…once they’re born you’re stuck with minding them and saving for their college fund.” And it’s SO true. The cost of building an Identity platform that stores a username/password and some basic data might sound trivial compared to a 3rd party platform. But that Capex cost isn’t comparing like for like. There’s a hidden Opex cost associated with iterating on it, “just” adding a new field that can be stored against a profile, regularly ensuring it meets regulatory standards for personal information storage and more importantly, ensuring that knowledge share within the engineering and product org is ongoing so that the platform doesn’t become a monolithic black-box over time that engineers are scared to touch out of a fear that something, somewhere will break and make as much sound as a tree falling in a forest when no one’s around. We’ve all either worked for or worked with companies that have built things they should have bought or partnered for with and we’ve also had the same experience with teams and companies that have bought or partnered when it was more important to retain control and be masters of their own destiny with it. Next time you’re at the crossroads, have a think about how core to the business the next problem you’re about to solve is and think long and hard about whether you want to build/buy or partner. It can set the business and the team up for success or it can just as easily also cause drag, inhibiting future success. See our stacks and . here here This story was originally posted on our . engineering blog

Build/Buy/Partner

Graph databases are the hottest thing around right now. Whether you are just getting started, or you are in one of the 51% of organizations already using them, this is the place to get your news and analysis. The popularity of graph databases has gone through the roof almost overnight it seems. Everything points this way: the , the , the response to my . trend lines from database engines reports from the Forresters of the world graph-related ZDNet posts Why is that, and should you care? Do you really need a graph database, and if yes, how do you choose one? That’s the million dollar question. Well, that’s more than one questions actually, and that’s more than a million in value there too. Either way, i can help you answer, starting today. This is the first edition of the monthly Year of the Graph newsletter. Every month i will collect, republish, and comment on the 10 most important Graph database related news items. Why me? Ever since i implemented my first graph database prototype in 2005, i have worked on award-winning research, consulted the (then) leading vendor on distributed query implementation, and lead teams of all sizes and shapes working with graph databases. I have also published a number of reports and articles with analysis on big data, distributed systems and analytics with and . Perhaps more importantly though: Gigaom ZDNet I work for nobody else but me, and by extension, you. No fluff, just stuff. No automatically harvested or sponsored posts. No vendor affiliations or hidden agendas. Just hand picked, curated content, and objective, concise analysis. Why the year of the graph? Glad you asked. In a nutshell, because graph databases are coming of age and getting attention. Some of it has to do with the infrastructure and the technology enabling graph to get mainstream. The use cases have always been there, and heavyweights like AWS and Microsoft are moving in this space. _It's official: graph databases are a thing. That's the consensus here on Big on Data among fellow contributors Andrew…_www.zdnet.com The year of the graph: Getting graphic, going native, reshaping the landscape | ZDNet 2. How do you model you graph? This is the question Dilyan Damyanov from Snowplow explores. Using graphs is part of what Snowplow does, with an emphasis on managing events. The analysis on different answers on how to model event data as a graph sheds some light on the fine art of modelling graphs. _In recent months we've been busy expanding the variety of storage targets available for Snowplow users to load Snowplow…_snowplowanalytics.com Building a model for event data as a graph - Snowplow 3. So, you have a Hadoop data lake. There’s graph data in there, and “graphy” queries you can do on that data. ArangoDB’s Max Neunhöffer gives examples of graph use cases, and provides an introduction to some graph algorithms you can use, as well as a walkthrough of how to get started with ArangoDB. _Graphs in the sense of linked data with vertices and edges are everywhere these days: social networks, computer…_www.oreilly.com Fishing for graphs in a Hadoop data lake 4. Graph is a much more natural and efficient paradigm for doing multiple joins (hops) than relational data. Furthermore, graphs also work well together with another mega-trend, machine learning. The inherent structure in graphs can be leveraged in your machine learning algorithms, as Graphistry’s Leo Meyerovich discusses with O’Reilly’s Ben Lorica. _The O'Reilly Data Show Podcast: Leo Meyerovich on building large-scale, interactive applications that enable visual…_www.oreilly.com Graphs as the front end for machine learning 5. In their original inception data lakes grant universal access to data in their native formats, yet lack the necessary metadata and semantic consistency for long term sustainability. What can give you that? The combination of enterprise-wide ontologies, taxonomies, and terminology, says Franz Inc’s Jan Aasman. _One of the most significant results of the big data era is the broadening diversity of data types required to solidify…_www.infoworld.com Harmonizing big data with an enterprise knowledge graph 6. Graph databases come in 2 main flavors. RDF is one of them, and Linked Data is a collection of related standards, including taxonomies, that enable graph-based navigation and querying. If you’re interested in the foundamendals of this approach, this account of the Network for Information and Knowledge Exchange’s workshop may be for you. _Conrad Taylor writes: The first NetIKX meeting of 2018, on 25 January, looked at new technologies and approaches to…_netikx.wordpress.com Making true connections in a complex world - Graph database technology and Linked Open Data 7. If you think a taxonomy sounds fancy, how about an ontology? As Teodora Petkova from Ontotext argues, this is not about a philosophical debate on the essence of being. In order to define what something “is” (to a computer program), information technology resorts to the use of ontologies. Some graph databases use those — this is how you build a knowledge graph. _A program that wants to compare or combine information across the two databases has to know that these two terms are…_ontotext.com Ontology, Meaning and Knowledge Management 8. The other graph database flavor is LPG — Labelled Property Graphs. One of the differences between RDF and LPG is support for taxonomies and ontologies, and as a consequence of this, inference: RDF has it, LPG does not. Or at least, that was the case up to now, says Thorsten Liebig from derivo GmbH. Their solution wants to support RDF-like reasoning on LPG. _Discover how the Neo4j graph database and the Cypher graph query language perform as an OWL reasoning engine when…_neo4j.com Neo4j: A Reasonable RDF Graph Database & Reasoning Engine [Community Post] 9. Titan, an open source LPG graph database, was an important piece of the graph database world. As Titan is no longer maintained, IBM and others have been supporting a group of people who stepped up and adopted its codebase, forking it as JanusGraph. Ted Wilmes from Expero is one of the architects of JanusGraph, and he recently spoke about the state of JanusGraph in 2018. 10. What is the current status quo in the graph database world? All the latest developments from AWS, Cambridge Semantics, Neo4j and Tigergraph, in the “ best overview of the graph database world(s) so far “, plus analysis on the options for querying graphs. _Fifty years ago, relational databases were neither ubiquitous nor standardized. The story of how we went from the…_www.zdnet.com Back to the future: Does graph database success hang on query language? | ZDNet Would you like to receive the latest Year of the Graph Newsletter in your inbox? Easy — just signup below. Have some news you think should be featured in an upcoming newsletter? Easy too — drop me a line here . Originally published at http://linkeddataorchestration.com/2018/04/03/yearofthegraph/

The Year of the Graph Newsletter: April 2018

24 Best Live Action Anime Movies Ranked by Review Scores

When you combine a great team, strategy, technology, product-to-market fit and track record of execution, you’ve got the ingredients for a great success. In reviewing a number of cryptocurrencies to include in our groups fund, part of my due diligence process involves making sure that cryptocurrency groups qualify in at least 4 out of 5 on this list: Team Strategy Technology Product-to-Market fit Track Record of Execution When it comes to meeting the criteria above, there should be few exceptions. Fundamental questions arise around each of these points such as: Team How long have the team members known one another? Have they worked together in the past? If so, for what period of time and through what kinds of circumstances? Strategy Do they have a well thought out strategy? Is their strategy too similar to that of their competitors. Do they know who their competitors are? Technology Is their technology unique or hard to duplicate? Is their approach scalable? Do they have any ‘ ’? technical debt Product-to-Market fit Since most early cryptocurrency groups are operating with a whitepaper alone, you’ll most likely have to do your own product-to-market fit for them. If they have a working product, ask them to see any customer feedback they can share. Track Record of Execution Does the team or any of its members have a track record that shows they’ve accomplished or executed anything to completion in the past? If so, how was it received? What was the projects impact, level of innovation and creativity? Here are a select group of cryptocurrencies that meet most of the criteria mentioned above and that might also be worth positioning in sooner than later before they list on larger exchanges: ( ) Current Price: Quant Exchange IDEX: QNT $1.24 Site: https://www.quant.network/our-technology/ From their site: “Quant Networks solutions help healthcare organisations by limiting the complexity of adopting blockchain technology.” That quote is already sets the tone for a valuable crypto. The most common problem with user adoption in crypto is the complexity. It really is just too complicated for the average non-technical person to jump through all the hoops to trade and invest in crypto. Quant is here to solve that! They also intend to advance science which is another significant indicator that QNT is a high quality crypto: “This in turn will improve their operational inefficiencies, so they can instead focus on what truly matters — helping patients and advancing science.” QNT has also recently joined the the (GLBC), composed of more than 100 large companies, law firms, software companies and universities focused on developing standards and compliance guidelines for blockchain and smart contract technologies in the business of law. More on this . Global Legal Blockchain Consortium here Given Quant’s reasonable total supply of 45M, our firm see’s Quant as one of the top 10 runners of this year. Time to load up! — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — - ( ) Current Price: Vectorspace Exchange IDEX-only: VXV $0.45c Site: https://vectorspace.ai These guys actually have a product called QuantBot, similar to the company above! What’s interesting about VXV here is that they are doing an AMA in front of 16 million readers of the science community at this Sunday. This is a real crypto with real technology folks. More on the AMA can be found here — WOW! reddit.com/r/askscience vectorspace.ai/reddit-ama.html They have an API here v that uses a “wallet-enabled” API key which means everyone will have to buy VXV on the open market! ectorspace.ai/api.html Their partners include , the US Dept of Energy (DOE), Google and a reddit data engineering company. Lawrence Berkeley Lab pushshift.io, From their site: “The datasets are correlation matrices engineered with ‘feature attributes’ or continuously valued distance calculations. These ‘feature vectors’ are related to today’s ‘word embedding’ methods used in Natural Language Processing (NLP). Resulting feature vectors consist of scored and ranked relationships between entities and surrounding human language that define their contexts. What we do is not magic but basic and fundamental to any machine learning effort, however, as Andrew Ng put it, “Coming up with features is difficult, time-consuming, requires expert knowledge. “Applied machine learning” is basically feature engineering.” — Machine Learning and AI via Brain simulations. We’ve had the opportunity to operate in different knowledge domains such as finance where we’ve developed datasets that allow for context-controlled correlations of sympathetic, parasitic and symbiotic known and hidden relationships between entities such as public companies correlated to the periodic table of elements, cryptocurrencies, human genes, diseases, phytochemicals, pharmaceuticals and other entities. So far we’ve received about 350 custom dataset requests in this area.” VXV can go beyond the moon folks. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — Aeternity (AE) Current Price: $1.07 Site: https://aeternity.com In very simple terms, æternity is a new blockchain platform, but it is also a vision for the future. This vision is being realized by a global community of believers, convinced in the disruptive potential of scalable, public blockchain technology. It is a movement that aims to propose alternatives to the existing governance, economic and financial intermediaries. From a technological point of view, æternity is a public, open-source, blockchain-based distributed computing and digital-asset platform that builds upon decentralized cryptographic P2P technology. The platform is open to anyone, anywhere in the world to use. For more detailed information, take a look at the dedicated to æternity or visit the . Wikipedia article GitHub page AE are the access tokens to the æternity blockchain and act as a unit of account for the resources spent on æternity. AE tokens are the energy (or “crypto-fuel”) used to power any application implemented on the platform. Every action (and even inaction) on the æternity platform costs something, even if only fractions of cents. AE tokens are the fuel for the new types of trustless apps (or æpps) that will be enabled by æternity. For more information regarding applications, visit the or visit the . æpps website æpps FAQ Aeternity is certainly a top performer! — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — - ) Current Price: Aurora DAO (Exchange IDEX: AURA $0.07c Site: https://auroradao.com/ What is Aurora? Aurora is a suite of DAPPs and protocols that make up a fully decentralized banking and financial network. The components of Aurora support the boreal, a decentralized stablecoin, and together enable global banking in the new currency. Revenue from the various components of Aurora will be used both to increase the asset reserves backing the boreal, allowing Aurora to expand boreal banking, and to provide economic incentives that ensure the system runs as designed. IDEX will be the first exchange to operate on the Snowglobe network, transitioning from a semidecentralized to a fully decentralized exchange. All fees from IDEX are remitted to the Aurora reserves. As the reserves grow Aurora can increase the value of outstanding loans, generating additional revenue for the Aurora network. Fees from the Aurora banking system will then flow back to those who stake the AURA token and provide the economic foundation that holds Aurora and Snowglobe together. Similar to other blockchain networks, Snowglobe requires a well-designed incentive structure to ensure that the integrity of the order book and transaction sequence are not compromised. Aurora has its own native network token, AURA, that aligns the interests of Aurora and the Snowglobe operators. All of the revenue from Aurora is used to compensate those who stake their AURA and provide security for the Snowglobe network. AURA staking aligns the economic interests of the operators with the health of the network, and makes it extremely costly for any would be attacker to disrupt operations. Prior to the launch of Snowglobe, all revenue from IDEX will be deposited into the boreal reserves. This revenue will be collected for future use in aligning economic incentives with AURA holders after the Snowglobe protocol is live. In the event that there is not enough revenue from Aurora to provide sufficient security for the platform, fees from IDEX may be used directly to enforce security in the system, bypassing the remittance to Aurora reserves. AURA definitely has the right stuff! This one is on our watchlist for double digits! — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — - ( ) Current Price: TomoChain Exchange IDEX: TOMO $0.29c Site: http://tomochain.com TomoChain is interesting because they are already up and running with a real blockchain product and getting great reviews similar to QNT and VXV. From their site: “The Tomochain testnet 2.0 launch is a great proof marking a major step toward our mission to be a leading force in building the Internet of Value. Tomochain is The Most Efficient Platform for Token Economy, designing its infrastructure and working to create an alternative financial system which is more secure, transparent, efficient, inclusive and equitable for everyone.” I like this valuable piece of information found in their whitepaper as it also validates their current product releases: “A consensus protocol is considered live if it can eventually propagate and make valid transactions onto the blockchain [12]. An occurrence of a liveness fault is when transaction omission, information withholding, or message reordering, among a number of violations are observed. This type of fault is unlikely to happen in TomoChain because the block creation masternodes list is ordered in a predetermined way for each epoch, thus if even an attacking masternode omits some transactions, the latter will be processed and validated by the next honest masternode in the next block.” Conclusion There’s a part of the due diligence process that’s more of an art than a science. It’s when you include another valuable but less tangible component, namely the Story. There was once a time when our fund was solely focused on traditional stocks and some of those were but that is of course, a another story. called ‘story stocks’, These plays have massive upside and positioning on the ground floor here before they list on larger exchanges can result in tremendous profit! Happy Trading!

SNOWPLOW

snowplow.io

-- employees

Since n.d.

COMPANY RANKING

SNOWPLOW

EVERGREEN INDEX #9230

Representing this company? Click here to claim and customize this page!

Snowplow WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

SNOWPLOW

snowplow.io

-- employees

Since n.d.

COMPANY RANKING

SNOWPLOW

EVERGREEN INDEX #9230

RELATED COMPANIESRANK

STORIES ABOUT SNOWPLOW

Representing this company? Click here to claim and customize this page!

Snowplow WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!