“I thought the Philippines was the friend of China,” Philippine President Ferdinand Marcos Jnr reportedly said during a spirited exchange earlier this year with China’s envoy in Manila amid ...

South China Sea: only diplomacy can avert open China-Philippines conflict

MANILA, Philippines – Pageant fans and supporters can now help their favorite Miss Philippines 2023 contestant advance in the competition. On Sunday, October 15, voting for The Miss Philippines ...

LOOK: Miss Philippines 2023 opens fan voting

HONOR Philippines, a leading provider of intelligent devices, achieved remarkable success at the Marketing Excellence Awards 2023 held on October 13, 2023. HONOR Philippines is Marketing Excellence Awards 2023's Marketing Team of the Year The company's ...

HONOR Philippines Wins "Marketing Team of The Year" at Marketing Excellence Awards 2023

according to Philippines armed forces chief Romeo Brawner. "These dangerous and offensive manoeuvres by China's PLAN not only risk collision but also directly endanger the lives of maritime personnel from both sides," Brawner said in a statement on Sunday.

Philippines demands China halt "dangerous and offensive" actions in S.C. Sea

MANILA (AP): Canada will help the Philippines detect illegal fishing with its satellite surveillance system under a new agreement, Philippine officials said Monday (Oct 16). The arrangement gives ...

Canada to help Philippines track illegal fishing vessels with satellite technology

MANILA, Philippines (AP) — Canada will help the Philippines detect illegal fishing with its satellite surveillance system under a new agreement, Philippine officials said Monday. The arrangement ...

Canada forges agreement to help Philippines track illegal fishing vessels using satellite technology

Hyundai introduces the crossover-inspired Stargazer X as an addition to its compact MPV family. First shown globally in Indonesia last August, the three-row, 7-seater Stargazer X blends elements of both Hyundai’s award-winning MPV and SUV range. The new ...

Hyundai Philippines Launches 2024 Stargazer X For P 1,348,000 (w/ Specs)

Mental health has long been considered a sensitive topic in the Philippines. As a country with a substantial Catholic population, depression, and anxiety can sometimes be associated with a lack of ...

Mental health in the Philippines - statistics & facts

CHANGAN AUTO PHILIPPINES is flush with hope, excitement, and promise. That’s what a new beginning usually brings. This fresh start is care of a partnership with London-based multinational automotive distribution and retail firm Inchcape (with the CATS ...

Changan Philippines changes hands

The Philippines has a lucrative food service industry that includes a variety of establishments such as restaurants, fast-food chains, cafes, and small stalls and kiosks. With numerous food trends ...

Food service industry in the Philippines - statistics & facts

Thumbtack Philippines

thumbtack

This chat between Casey Winters, growth advisor in residence at Greylock VC, and Katie Muto, growth at VentureApp. originally appeared on VentureApp is one of tech’s most preeminent experts with incredible experience on the front lines of growth and at Pinterest and Grubhub, and advisory roles at Airbnb, Reddit, Thumbtack, Eventbrite, and more. Now, Casey is putting all of that experience to perfect use at as their Growth Advisor in Residence. Casey Winters growth marketing Greylock Partners Our team was lucky to from his treasure trove of experience. Fun fact: In his spare time, Casey loves to listen to music and go bouldering. But we’ll get back to that… chat with Casey about growth secrets Even though our team was very excited to chat with Casey, we decided to kick the conversation off with a question from the audience — asked whether growth should be baked into the role of product managers rather than entirely new growth orgs. Craig Zevin So, in Casey’s advisory roles, here’s what he would tell teams considering whether to hire and build specific growth teams: As you can imagine, we had a lot of questions from the audience. Next up, Nikin Tharan asked Casey to explain the toughest challenge or failure he had to overcome and a lifelong principle he learned from it? Hint: it has everything to do with communication style and lessons learned at Grubhub about changing management style to work better with others… Now let’s get into the topic at hand: . Now, if you’ve read anything about growth marketing, you’ve heard a certain phrase tossed around — growth hack or growth-hacking. It’s actually become one of the most despised phrase in the marketing industry since it indicates success via a silver bullet. , the Sponsorship & Program Coordinator at MIT Enterprise Forum, decided to ask Casey his opinion… growth Ronnie Deaver Now to get into the nitty gritty of Casey’s growth experience, we asked if he could share a specific growth experiment from his days at Pinterest and to elaborate on whether the results surprised him. His experience touched on a topic that all startups grapple to understand and master with their own customer base — user onboarding. We also asked Casey to name some startups who truly nail user onboarding. Some of his favorites? , , LinkedIn, . And his least favorite? It’s a good one and an example you would probably agree with. Hint: it’s a social product that proves confusing to many outside of highschool/college. Slack Instagram Engineering Dropbox Read to find out the worst user onboarding experience according to Casey Winters. We continued our conversation with another growth focused question for Casey —cautionary advice for eager growth teams like ours: What are some of the biggest & most consistent barriers to growth you’ve seen at various companies? Venture here to read the rest of our chat with Casey Winters, growth advisor in residence at Greylock Partners and former growth lead at Pinterest and GrubHub. PS — feel free to ask Casey your own questions about growth and marketing – the chat lives on forever and he will receive an alert that you asked a question or left a comment. 👍

Growth Secrets from Pinterest, Airbnb, Reddit, Thumbtack and more

Google Search Ads in the U.S.: Why They Compose a Relevant Market

Meta recently conceded that in-app behavioral advertising in the EU can no longer be opt-in by default, marking the end of an era. The regulators won, at least for now. Every few decades, major shifts happen that change how an industry fundamentally operates and who the new winners and losers are. We are in the midst of one such shift in the digital advertising industry. This has been driven by three critical changes in the ecosystem, all privacy-related: Apple announced the in 2021 App Tracking Transparency framework Google is in 2024 deprecating third-party cookies A slew of privacy legislation is either already in place (like the , ) or is about to be in place (like the ) California Privacy Rights Act EU General Data Protection Regulation EU Digital Services Act While the effects of these changes have been felt by the advertising industry globally, the for a few years now. . In the past three weeks, there have been significant announcements from , , and about upcoming changes to their products in the EU. It’s probably fair to say these changes mark the end of an era for behavioral advertising in the EU, and the platforms have made peace with that. EU, in particular, has been at war with behavioral advertising And winning Meta Snap TikTok In this piece, we’ll dive into: How the advertising ecosystem runs (or used to run) Privacy interventions and impact on behavioral advertising EU’s war on behavioral advertising and why platforms are conceding Possible future paths How the advertising ecosystem runs (or used to run) Let’s start with some simple concepts. An ad shown to you can be contextual or behavioral (sometimes both). A is something that is shown alongside organic results (non-ads) in the context of whatever you are doing. For example, if you are on a food delivery app like DoorDash and you see an ad for a restaurant alongside organic results, that is a contextual ad. In that case, your identity doesn’t really matter. DoorDash knows your location and what you’re looking for and therefore shows you a relevant ad. Knowing your identity and interests does help improve the relevance of these ads, but it’s secondary to the context itself. contextual ad A is something that’s shown to you because of your past behavior. For example, you might have visited an office chair brand’s page on Instagram, and therefore you get served ads for another office chair brand; that’s exactly what happened to me with the ad on the right. This can either be based on direct behavior (like the office chair example) or inferred behavior — people like you who looked for chairs also looked for monitors. One step further, people who looked at chairs are also likely to work from home more and therefore be open to lunch service. All of those are potential ads that can be shown to you. behavioral ad A subset of products, most often search products, are perfect for contextual ads — Google Maps, Google Search, DoorDash, Yelp, Thumbtack, TripAdvisor, and Zocdoc, to name a few. The user has high intent to do something specific. However, most high-intent products are low-medium frequency. You are not looking for plumbers every day on Thumbtack, and you are not looking to order food delivery every hour of the day on DoorDash; therefore, the amount of contextual ad inventory is lower — highly effective but lower in volume. On the other hand, social media platforms where users spend hours doomscrolling have a large volume of non-contextual inventory. Random ads are annoying and ineffective, and therefore the most effective way to monetize this inventory is through behavioral ads. Unsurprisingly, . Before ~2018 (when EU’s GDPR kicked in), there was essentially a about you, collected through a long list of highly effective AdTech mechanisms. Some notable examples: behavioral ads require a lot of data, specifically data about you free flow of data Every iOS or Android phone had a unique “ ” assigned to it (and by association, to you). This ID was accessible to all apps on your device, i.e. your Uber app and your Google Maps app both knew you are mobile advertising ID 123 and could easily cross-identify you across apps. mobile advertising ID Most browsers (except Safari and Firefox) support “ ”, which are essentially a piece of data that one site could place on your browser and another site could access. There would be a “cookie ID” associated with you, and you could be easily cross-identified across websites. third party cookies Note that . You could go deep inside your settings and reset these IDs, but most people never do that. This resulted in a couple of second-order effects. these identifiers, for all practical purposes, were permanent First, companies that are “ ” started purchasing data from several different data providers and ; for example, app 1 could tell the broker that you bought an expensive piece of furniture, website 2 could tell them you have an account with a high-end bank, and the data broker could put that together and categorize you as “high propensity of spend” person; note that when I say “you,” that refers to your identifiers (mobile advertising ID or third party cookie ID) data brokers building “profiles” about you As more and more data became available, the profiles had more fidelity to them, and identification became more sophisticated; for example, if app 1 had your mobile advertising ID and website 2 had your Chrome third-party cookie ID, a data broker or an ad platform could compare your IP addresses and know it is the same person. Take this one step further. A data broker could also use a third data source and find an email associated with your IDs. One more step further, they could buy your address from a utility or telecom company and find your address. This might sound far-fetched, but it’s not — are a very commonly used data product in the AdTech stack today, and telecom companies are . If you need a more entertaining take, check out . device graphs notorious for selling your data John Oliver’s episode on data brokers Second, ; through a combination of technical mechanisms, an advertiser could say you first saw an ad on Facebook, then an ad on Google Search, then a display ad on NYTimes, and eventually bought an item from their website, so each of those three advertising platforms get shared credit. Attribution/measurement of advertising campaigns became more and more precise While this sounds privacy invasive (and it is), this resulted in a highly efficient advertising ecosystem. Advertisers knew exactly which users they were targeting, and since they had all these extra behavioral signals to know how likely a user was to engage, they were willing to pay higher cost per impression (CPM) for ad inventory, thereby generating more revenue for a media publisher. Precise attribution/measurement turbocharged this further. However, you can see how this was . This rightfully raised concerns about data consolidation in the hands of both data brokers and large technology companies, and an intervention was inevitable. It was less a matter of if and more a matter of when. becoming the wild west — highly effective advertising no doubt, but also an uncontrolled orgy of data acquired with non-existent or questionable user consent Privacy interventions and impact on behavioral advertising Let’s dive into each of the three privacy interventions that arose in an attempt to tame the Wild West. First, Apple introduced the . It sounds jargon-y, but the change is relatively simple. Prior to ATT, every app by default had access to your advertising ID, i.e. it was opt-in by default. This meant you could easily be tracked across apps and therefore shown effective behavioral ads. For example, you installed the Strava app to track your runs, you are now on Facebook, and you are shown an ad for Strava Premium. After ATT, the access to this , i.e. an app had to show you a pretty aggressive prompt to get access to your identifier, and you explicitly needed to say yes. The average opt-in rate ended up close to (with a lot of caveats).' App Tracking Transparency (ATT) framework identifier became opt-out by default ~34% We won’t go into much detail here, but this prompt was launched by Apple in the guise of embracing privacy — a smart chess move. The consensus opinion today is that this was an opportunistic move from Apple, which no doubt improves privacy but also heavily hurts Apple’s competitors as they . The impact was that user identity was available much less often. prop up their own ads business Note here that for behavioral advertising to happen in the Strava-Facebook example, there is not one but two apps that need to have received opt-in from you, i.e. the addressable market does not drop to 34%, it drops to 34% * 34% = ~12%. Therefore, . cross-app behavioral advertising on iOS is no longer effective at scale Second, Google announced that it will . The consensus opinion is that the change helps Google achieve a dual purpose: appease regulators who are breathing down their neck for , while taking control back from what’s now a . Google’s post third-party cookies will still allow cross-site retargeting, but in a more private way where all information is stored on-device within the browser, i.e. there are no more cross-site “cookie IDs” assigned to you. While the new mechanism preserves some of the status quo, . deprecate third-party cookies in 2024 potential anti-trust behavior in AdTech fairly bloated advertising tech stack new mechanisms cross-site behavioral advertising is going to have much less fidelity and, therefore, effectiveness Which brings us to the third intervention — . The most aggressive of these is the EU’s GDPR, which went into effect in 2018. The California Consumer Privacy Act (CCPA) went into effect in 2020. While the progression has been gradual, the reason these laws matter for advertising companies today more than ever is because the laws take aim at the (i.e. you do a bunch of different things inside the Facebook app and Facebook gets to use that data to show you behavioral ads within the app). privacy regulations only remaining and mission-critical advertising mechanism — behavioral advertising within companies’ own apps EU’s war on behavioral advertising and why platforms are conceding A primary feature that makes the California laws (arguably the strictest privacy law in the US) less aggressive than the EU’s GDPR is that it . For example, the 2020 California Consumer Privacy Act (CCPA) requires companies that are considered “data sellers” under the law to provide explicit opt-outs on web pages, but the . does not require explicit opt-ins and only requires platforms to provide opt-outs default is still opt-in EU’s GDPR takes this up another notch and requires for behavioral advertising. This consent needs to be . For example, Meta cannot gate content behind a behavioral advertising consent prompt. explicit opt-in/consent freely given, specific, informed, and unambiguous So, the simplistic inference from this is that Meta needs to get explicit consent for all behavioral understanding, including the last remaining mechanism — showing ads within their own platform. If Meta is forced to do this, the opt-in will likely be small (the Apple opt-in rates were ~34%), and this majorly shrinks Meta’s addressable advertising market in the EU. To not meet that fate, Meta made a creative legal argument: GDPR requires any company to have one of ** **if the company needs to process personal data. Paraphrasing, these are — consent, a necessity to fulfill a contract, a necessity for legal compliance, a necessity to protect the vital interests of the user, a necessity to perform a public interest task, and a necessity for legitimate interests pursued by the company. six legal bases Of these six, Meta held the stance that they were processing personal data using “ legitimate interests” (the last of the six) as the legal basis and therefore did not need to show an explicit opt-in prompt. Last month, the Norwegian Data Protection Authority that . Paraphrasing the : provided their enforcement decision Meta’s use of “legitimate interests” as the legal basis is not valid enforcement decision Based on several past court rulings, Meta must prove that legitimate interest “cannot reasonably be achieved just as effectively by other means less restrictive of the fundamental rights and freedoms of data subjects.” Based on another past court ruling, despite the fact that Meta provides a free service, the user of that network cannot reasonably expect that the operator of the social network will process that user’s personal data, without his or her consent, for the purposes of personalized advertising. Four days after the Norwegian Data Protection Authority’s ruling, Meta that they would be changing their legal basis from “legitimate interests” to “consent.” In practice, what this means is that . While Apple’s changes significantly cut down the ability to advertise cross-app, Meta was holding on to hope that it would be able to preserve all in-app behavioral advertising (including in the EU), and the writing is now on the wall. shortly after with their own announcements that non-personalized versions of their products will soon be available to users. A total of are likely to follow suit. announced Meta is conceding that behavioral advertising within their own app in the EU can no longer be opt-in by default and followed suit Snap TikTok 19 platforms that are in scope Possible future paths It is still to be seen what percent of users opt into behavioral advertising. If the Apple opt-in rates are any indication, it may get to the ~34% rate seen there, but it may also go higher if platforms are allowed to be creative about opt-in language and user interfaces. It is hard to say what exactly the long-term effects of ending the opt-in-by-default regime for behavioral advertising in the EU will be, but here are some educated guesses: — Meta has made towards letting users provide inputs to the feed algorithm. I would expect the creation of more explicit preferences/interests that users explicitly select. While it has worked to a smaller extent for some products like , it is likely that the quality of content provided by just preferences will be nowhere close to the relevance of a system that is constantly ingesting data and inferring interests. Explicit preferences instead of inferred interests some progress Reddit : TikTok has announced that they will create a that is based on what’s popular in your region. Most social feed-based platforms will also (re)introduce simplistic ranking options like chronological. I am not bullish that any of these will create an engaging user experience. Non-personalized feeds non-personalized feed : My (potentially contentious) take is that opted-out users will see a significantly degraded experience and eventually decide to provide platforms consent to personalize. The platforms will likely use this mechanism to receive content for all personalization (including behavioral advertising) — It’s unclear if there are any regulatory constraints that force them not to bundle consent for user experience personalization and behavioral advertising. I don’t expect the consent rate to go anywhere close to the status quo, but I think it will be a meaningful bump to what’s seen with Apple’s opt-in rates. Degraded user experience, leading to more opt-in : Twitter launched Twitter Verified a few months back, and Meta is with a similar concept. I am not bullish that a majority (or even a minority) of users will be willing to pay for social media products after years of being conditioned to get these products for free. Experiments with subscriptions experimenting If platforms are not able to effectively monetize the EU market, it is likely that they will start de-prioritizing investments for the EU market, which is a real risk of an aggressive regulatory regime. I don’t expect them to stop supporting EU markets, but every new feature built is an investment, and companies will start heavily questioning if they are willing to do any extra work for this market. For example, if there’s additional work to make a feature comply with GDPR, that might not be worth it anymore. We have seen other equivalent situations of this — several apps have a much better experience on iOS than Android because iOS apps monetize better. De-prioritization of EU markets: Conclusion The advertising market exploded over the last several decades due to the availability of essentially infinite user data, which brought both large efficiencies in advertising and major privacy risks. Regulation was necessary. Great regulation is about finding the balance between promoting innovation/letting new businesses emerge that move society forward and having guardrails so people are protected. Did the EU go too far? I personally think they did, and if that is true, the effects will start showing in a 5–10 year horizon, and course correction will follow. Until then, this is the new reality of behavioral advertising in the EU. The regulators and legislators came all guns blazing, and they won. 🚀 If you liked this piece, consider subscribing to Every week, I publish one in the form of a 10-minute read. Best, Viggy. . my weekly newsletter deep-dive analysis on a current tech topic/product strategy Also published . here

The New Reality of Behavioral Advertising in the EU

The Drive to Migrate Data to the Cloud With data being valued in recent years, many organizations feel the pressure to become innovative and cost-effective when it comes to consolidating, storing, and using data. Although most enterprises are aware of big data opportunities, their existing infrastructure isn’t always capable of handling massive amounts of data. more than oil By migrating to modern cloud data warehouses, organizations can benefit from improved scalability, better price elasticity, and enhanced security. But even with all these benefits, many businesses are still reluctant to make a move. This is because these benefits are great . However, before companies can get there, they must go through . once you get there the dreaded data migration process, and that process can be painful— especially if you don’t have the right tools To create an effective cloud data migration strategy, companies often struggle to find answers to key areas like: Gaps or anomalies in the source data Full or partial data migration Data owners and stakeholders Data lineage and dependencies Data consolidation process Automated regression testing for accuracy With so many daunting concerns, it’s no wonder why many businesses delay their migration. And as a result, they miss out on the benefits of cloud data warehousing. In this article, . While our focus in this article will be on Snowflake, a nearly identical approach can be used for migrating to Redshift, BigQuery, Databricks, or any other cloud data warehouse. we will show you how—with a tool like Datafold—you can avoid a lot of the stress and complexities of data migration to Snowflake Snowflake is a Software-as-a-Service (SaaS) data warehouse hosted entirely on cloud infrastructure. It has one of the and is available on all three major cloud platforms, offering unlimited scalability in both storage and computing. Snowflake biggest software IPOs in history Another reason for its popularity is its simple maintenance. Unlike most legacy data solutions, Snowflake takes care of the following: management upgrades tuning backups high availability indexing It keeps its storage, and computing layers separate, allowing significant cost-saving when not using computing power. Other attractive features include time travel, end-to-end encryption, and integration with popular identity providers. Snowflake also natively integrates with , creating a rich ecosystem of analytical capabilities. , a modern data reliability platform, can make data migration to Snowflake a breeze. For example, Datafold can: third-party applications Datafold Intelligently profile data sources to show exactly where a piece of data is coming from and what it depends on Show before and after migration differences between source and target data sets. Use built-in machine learning models to automatically find data anomalies and alert users. Integrate with all major data warehouses—including Snowflake Be automated with tools like or or integrated into a continuous integration (CI) process. dbt Apache Airflow The important question is: Let’s see. Can we prevent potentially costly mistakes when migrating to a Snowflake data warehouse? Typical Data Migration Mistakes and How to Avoid Them There are a few common mistakes DataOps teams make when migrating to a data warehouse. Ignoring Key Stakeholders Data engineering teams often rush through a migration project without consulting business stakeholders or including their feedback about data validation rules. This results in a flood of post-migration support tickets for erroneous or missing data. What’s worse is that some errors can go unnoticed. For example, let’s say the team is migrating billing data, but the migration team hasn’t properly consulted the finance team. The currency conversion rates in the legacy system might use a four decimal point rounding while the new system uses only three. The engineers may deem this acceptable, but for the finance team, a change like this can mean skewed numbers across all their reports and subsequent business impact. To address this issue, in Datafold can list all your data sources in one place. This allows you to search and filter to find a particular table, database, or column from thousands of databases across your environment. Data Catalog Once you find a data asset, you can easily add descriptions and tags and also set owners for the assets. This makes it easy to know who to contact if you need to make a change or what types of changes are acceptable. Taking a “Lift and Shift” Approach Many businesses take a “lift and shift” approach to their data migration. This is where all the data from old systems is migrated to a new one with little to no validation, cleansing, or transformation. One of the reasons businesses take this approach is because they deem it too time-consuming to comb through huge datasets to find out what’s relevant and what’s not. Lift and shift can only work when you’re 100% sure that the old system’s data is correct and portable to the new system—and this is seldom the case. Consequently, you risk migrating incorrect or duplicate data and paying for its storage. When working with TBs—or even PBs—of data, these storage costs quickly add up. as-is The Data Catalog summarizes the quality of the source data in terms of volume, completeness, and distribution of values. Using this, you can identify if important table columns like birthdate, email_address, or last_name are complete and if they need special business rules. In the image below, we can see a table’s data profile: The table has more than 19 million rows and 21 columns. There are no upstream dependencies. Part of the column’s data distribution looks like the following: As you can see, some fields—like pickup_latitude and pickup_longitude—have lower completeness ratios. The graph also shows data distribution in different fields. Data engineers can use such information to decide whether (or how) to cleanse and preprocess the data before migration. The feature can also help by finding the most frequently used (and, therefore, the most critical) tables and columns in the source. column-level lineage One of the major drawbacks of lift-and-shift is that you’re never sure what fields or tables are unnecessary and don’t need to be ported. Column-level lineage makes this task much easier as it can discover all the dependencies of a table or column. Using the Data Catalog and its lineage feature, let’s say you find a large source table that hasn’t been updated in the last five years, and this table has no dependencies. You can feel confident that it can be archived and doesn’t need to be migrated. As an example, in the image below, we can see that doesn’t have any dependencies: taxi_trips Migrating Old and Inefficient Queries Engineers should also be looking for inefficient queries powering existing analytic reports. Not surprisingly, this is also often overlooked in fear of breaking applications. For example, maybe you have a query with 10 joins, four of which are joining to obsolete data tables. The migration project can be the perfect time to optimize it by cutting out irrelevant joins. Showing side-by-side improvements can further showcase the value of planned migration to stakeholders. Once again, the column-level lineage feature can help refactor or remove inefficient queries. Column-level lineage can show how a data field relates to its upstream and downstream objects and what the effect will be when modifying queries containing this field. In the image below, we can see that a data field is related to two tables. This should tell the data engineer which tables may be affected when fine-tuning a query containing this field. Datafold’s feature can also show the potential impact of changing your old queries. Data Diff compares two datasets and can show if their schemas, primary keys, row counts, or column values are mismatched. Data Diff You can use such results for quality assurance of your migrated data. If the target Snowflake tables show no discrepancies, stakeholders can have confidence that the new queries are safe and effective. If the comparison shows differences, then the queries can be further fine-tuned. In the images below, we see one such Data Diff operation. Here, the target table shows non-matching columns with non-matching values: Manual or No Regression Testing For organizations dealing with terabytes of data, testing and validation is a guaranteed recipe for failure. Manually cross-checking millions of rows of query results or data is simply not an option. The result is that many engineers will simply skip validation altogether. manual According to the data engineers in , their manual validation process used to take up to two hours per pull request. In many cases, this validation just wasn’t realistic to complete, and mistakes can be costly. There are two ways to eliminate—or largely automate—manual data validations when migrating data to Snowflake. Thumbtack The first method is to use Datafold’s feature. Datafold uses machine learning to find the normal trend or seasonality in your data. You can configure alerts to send proactive messages to DataOps when anomalies are detected in that data. You can also adjust the alert thresholds. Alerting You can create alerts by writing your own SQL statements for data completeness checks or comparisons, as shown below: In both cases, proactive alerts can help data engineers fix any issues in the source data before migration. The other option is to use the Data Diff feature within the CI process. With this approach, after creating a data migration script, it’s committed to a Git repository. Once the commit is made to the development branch, a developer creates a pull request to merge the change to the main branch. At this stage, a senior team member can review the code to approve or reject it. Now, when integrated with the CI process, Datafold’s Data Diff feature can show the reviewer the effect of the ETL code—how data will change between the source and the target as the code cleanses, aggregates, and performs its operation. This is automatic regression testing for the data pipeline. If reviewers see the target table will have unwanted changes when the code runs, they can reject the change, preventing garbage data from entering the Snowflake database. When Thumbtack started using Data Diff, they reported savings of over 200 hours per month, increased productivity by at least 20%, and over 100 pull requests automatically tested every month. Conclusion There are many advantages to moving to a cloud data warehouse like Snowflake. However, migrating even a single database to a new system can have its challenges, especially when the source data hasn’t been profiled, or the ETL code hasn’t been tested. The key to successful data migration is leveraging tools that can automate most of the hard work. Datafold can help dramatically reduce the number of errors, failures, and silent issues by automating most of the testing. Although we’ve focused on Snowflake in this article, the same features of Datafold can be used for other cloud products like or . Redshift BigQuery Also Published Here

Migrating to Snowflake, Redshift, or BigQuery? Avoid these Common Pitfalls

Imagine a car factory where nameless workers in a sprawling complex make expensive cars all day and all night.  Thousands of shiny, new expensive cars move off the line every millisecond and are shipped instantly all over the world to wealthy buyers, generating $195 million of profit daily for the car company. Sounds impossible, right? Not if everyone on earth is a worker and does the work for free. This is the modern impossible system made possible by Big Tech. You are the free laborer. Big tech companies put great tools in our hands like News Feed and Search and tell us our lives are better for it. In reality, we are giving them free labor. Every time we browse a site, comment, like, subscribe, purchase, we are mining data gold for these companies, which in turn roll up our data into neat bullion bars behind our backs and serve to other big companies to make each other rich. This is the lopsided deal of tech that we all signed up for over a decade ago, and now we all find ourselves on the losing end of this deal. Data goes out, money comes in ($2 Trillion every year). Where is our piece?  Aren’t we the ones making the product? My company, , is here to change that.  Ozone is the first company to build a simple wealth-generating system on top of the internet. The Ozone layer gives users the ability to capture, clean, and monetize whatever data exhaust they spit into the atmosphere.  Before it gets into the hands of the advertisers, users get a say, and they get a piece of the upside from the end user (companies). Ozone This is the way the internet should have been from the start. The internet was originally formed as a communication system between universities on the US west coast (ARPANet) which allowed the people talking to each other to make money and create new things off the data/information they exchanged.  Somehow it morphed into the system today, where the owners of that piping make all of the money off of the people manning the microphones. Ozone’s HQ sits just a few blocks away from where the original internet, ARPANet, was conceived (The RAND Corporation) in Santa Monica, CA.  We are imagining our shared future as a new beginning of the original internet, where now if you are applying effort and attention in the system, you get rewarded and paid for the privilege. My inspiration for Ozone started in 2014, when I was working for the biggest and most successful drug dealer in the world: Facebook.  Facebook (like Google) dispenses dopamine hits every millisecond every day of the year, and because of its near $1 trillion market cap, can now fund social change. My job was to figure out ways to make it easier for advertisers to give all of their money to Facebook. We had some advertisers in Thailand who, because of a declined bank card, were jam-packing duffel bags full of cash and hand-delivering them to FB Advertiser events in Asia, begging us to turn their ads back on when we shut them off for delinquency. Every time I queried a user table for data, I saw what the frenzied desperation for Facebook Advertising product was all about: access to brands a user likes, his / her favorite sports, relationship preferences, relative place on the social hierarchy, and a bunch of other personal stuff that Mark Zuckerberg prob never dreamed of being able to access when he created a “Hot or Not” for Harvard dorms. After a couple of years at Facebook and then Google, I started my first company, called Leviathan Analytics.  This company was about spatial distribution of data and human behavior, i.e. where on a map do people do particular things in patterns? How much regularity can we find in those activities / patterns?  This company got snapped up by a much better-funded company called Findyr, where I became CEO. Findyr was a company with the 2010s-era misspelled name to denote tech chic, much like “Burb’n” before it was rebranded as Instagram.  The company’s premise was crowdsourced data collection: mobilize over 100,000 people all over the world with the app on their phone to get paid for uploading local data. Our main customer was Gallup, who worked with us to source the opinion of thousands of people in countries like Brazil and India.  My LinkedIn started getting bombarded with messages from mobile users who were asking me how else they could get paid for more data - they were willing to give it up! They would ask me, “what else do you want to know?  Do you want access to my Facebook posts?” That’s when I realized that this problem was real and affecting billions of people all over the world, who were asking: why does everyone else gets paid for my data except for me? I left Findyr and linked up with my co-founder Ben Colman, who, like me, spent time building startups trying to give people more power over their data.  Ben built a company that let users remote-detonate their private data from another location, basically like an IED for loose data.  Ben and I hit the road in Silicon Valley and New York to raise money around the idea alone: take from the rich (Google and Facebook), and give to the poor. Our idea caught the eye of one of Silicon Valley’s leading angel investors who had a reputation for finding outsized bets early and doubling down on companies that empower the everyman to be true owners of their time and assets: Jason Calacanis.  Jason had many notable successes in the “unlock hidden inventory” economy, namely Uber, Robinhood, Thumbtack, and Calm; he and our other investors saw us as the next thing on the right side of history After raising our seed round we set up in Santa Monica, where the internet age was born (via the RAND Corporation), and continues to live on with heavy hitter companies that emphasize privacy (Snapchat), and decentralization (Everipedia, founded by our chief evangelist, Mahbod Moghadam).  When I moved here, I noticed that all of the OG tech entrepreneurs were using CBD products, almost like the modern LA-version of mind-altering psychedelics from the San Francisco ‘60s, but more chill and fewer weirdos. In Santa Monica, I linked up with my homies and tech OGs, Mahbod (founder of Genius and Everpedia), and Joe Combs (founder of direct-to-consumer subscription company EarFleek and architect of our consumer platform), and Ozone started to accelerate in ways it couldn’t have if I stayed in enterprise tech rent-collector kingdom NYC. That’s when we started to figure out that CBD companies have the same problem as Ozone users: they get abused by the tech giants! CBD companies face daily censorship for posting ads for simple products like bath bombs. Olivia Alexander, the founder of our partner KushQueen, has been fighting the tech giants for a place at the advertising table for years, and despite repeated shutdown of her ads, she has grown a legit CBD empire.  KushQueen is our valued partner, and we’re finally giving them an advertising home they’ve been denied for years from FB and Google, despite having bags of cash to give them.  We’ll gladly take that from Zuck. As of the launch, Ozone is simple: you get $5 for signing up and then earn more for doing tasks, like learning about CBD bath bombs. If you are into CBD, it’s actually kind of fun! It’s not like a job where you’re getting paid, it’s more like a game where you get to keep the value you are creating. My aspirations, however, are for Ozone to eventually dominate the entire data sector - I want Ozone to eventually permeate into all facets of the internet, giving users the value their actions merit. Like the Iberian explorer expeditions that left Europe for the New World, many are claiming to be the first to try what we’re doing.  The true bearer of that claim is probably Brave (the browser).  Brave does some amazing stuff, and I admire the founder Brendan Eich for attempting to create a pure alternative internet.  The problem with Brave is twofold: (1) it limits users to earning cryptocurrency (BAT), which requires users to surmount a massive intellectual and technical hurdle to understand what earning BAT means in terms of everyday savings and disposable income and (2) it promises privacy freedom independent of Google, yet relies on anti-phishing and malware services provided by Google, making it a circular reference. Ozone is different in that we give users the option to get earnings in the same currency that Facebook and Google get their earnings: pure hard cash.  Unlike Brave, we are fundamentally unambitious in trying to create the latest Manhattan Project to nuclear detonate the Internet: like O3, we are a simply a compound layer on top of the existing internet that requires users to do very little to realize earnings and the power of true ownership.  We leverage existing tools that all users--not just crypto- and privacy-savvy users understand to empower them to be CEOs of their own data. We are wise to the times however and are already laying the foundation for a crypto architecture. The earliest crypto iteration of Ozone will simply be that you can choose whether you get paid in cash or bitcoin. The goal, however, is for Ozone to someday have a utility token where users are not only getting to own their data, but their activities on the site give them more control over the community we are building. In this new internet nation we are building, citizens get paid for what they put in. The way the internet should have been.

Everyone Works for Facebook and Google

Thread: We all know managed marketplaces (where the platform provides additional value-add in terms of intermediating the service delivery) can create a step-function improvement in user experience, but despite that, they can actually struggle to scale as businesses. Examples of the functions that managed marketplaces often take on include vetting providers, handling logistics, or authenticating goods. This translates into a much better user experience, but can also mean greater operational overhead, lower margins, and difficulty growing. The complexity of intermediating the transaction is not trivial. Consider Beepi–a managed marketplace for used cars–which created an amazing user experience (90+ NPS), but ultimately shut down due to challenges becoming unit economic profitable in many markets. For Beepi, the costs associated with each transaction included rigorous inspections, free returns, delivery, and a guarantee to buy the car if it didn’t sell within a certain time frame. In categories where services are viewed by users as commodities and/or users are especially price-sensitive, the open version of a marketplace often wins out over more managed versions. For example, Craigslist and Thumbtack are much bigger businesses than marketplaces that have attempted to employ home services pros themselves and own the end-to-end experience. But for more emotional categories that are more ongoing in nature or  have variable outcomes, where each provider is meaningfully different, and the stakes or $ values are higher, the managed version often does better than the unmanaged version. Here’s some examples where managed marketplaces have gained ground: Uber (more managed w/ centralized matching and pricing) vs. Sidecar (double opt in, no centralized pricing) Lambda School (managed) vs. open MOOC platforms Rover vs. finding a dog walker on CL/via WOM TheRealReal vs. unmanaged consignment marketplaces What do these managed marketplace companies have in common? Focus on a service category where users value the experience differential and are willing to pay a premium that offsets increased costs Building software to improve efficiency Focus on high AOV & high frequency Another factor to consider is that labor/service marketplaces typically have more overhead than product marketplaces in terms of management. That’s because every aspect of a service can be managed end-to-end, and there’s frequently a geo-specific aspect to the overhead, vs. a product marketplace can just manage specific aspects of the transaction (validation, logistics) – much of which can be outsourced / centralized. With any unmanaged marketplace, there’s going to be variance in quality. For categories where there’s less tolerance for uncertain quality/experience and desire for more consistency/certainty, a managed approach can win out. For founders, the key is finding a way to provide a more delightful, managed experience where it matters, but doing so scalably, leveraging software. For instance, designer goods marketplaces can reduce the # of items needing manual authentication by predicting which items are most likely to be fake. In addition, validate that you can charge higher prices or increase take rate to cover the costs of managing the marketplace. TL;DR: Managed marketplaces can create a step-function improvement is user experience, but they can be complicated to operate & challenging to scale. There’s a number of factors to consider in whether to go unmanaged vs. managed, and to what degree. For more on managed marketplaces, see our blog post: https://a16z.com/2018/11/27/services-marketplaces-service-economy-evolution-whats-next/ Originally published as “ Pros and Cons of Managed Marketplaces "

Pros and Cons of Running a Managed Marketplace

Startup Metrics: 16 Ways to Measure Network Effects

Power users drive some of the most successful companies — people who love their product, are highly engaged, and contribute a ton of value to the network. In ecommerce marketplaces it’s power sellers, in ridesharing platforms it’s power riders, and in social networks it’s influencers. The importance of power users All companies want more power users, but you need to measure them before you can find (and retain) them. While DAU/MAU — dividing daily active users (DAUs) by monthly active users (MAUs or monthly actives) — is a common metric for measuring engagement, . it has its shortcomings Since companies need a richer and more nuanced way to understand user engagement, we’re going to introduce what we’ll call the “Power User Curve” — also commonly called the activity histogram or the “L30” (coined by the Facebook growth team). It’s a histogram of users’ engagement by the total number of days they were active in a month, from 1 day out of the month to all 30 (or 28, or 31) days. While typically reflecting top-level activity like app opens or logins, it can be customized for whatever action you decide is important to measure for your product. The Power User Curve has a number of advantages over DAU/MAU: It shows if you have a hardcore, engaged segment that’s coming back every day. It shows the variability among your users: some are slightly engaged, whereas others are power users. Contrast this with DAU/MAU: it’s a single number and so blurs this variance. When mapped to cohorts, Power User Curves let you see if your engagement is getting better over time — which in turn helps assess product launches and performance of oher feature changes. Power User Curves can be shown for different user actions, not just app opens. This matters if the core activity that matters for your product is deeper in the funnel. In other words, while the DAU/MAU gives you a single number, the Power User Curve gives entrepreneurs several avenues of analysis to assess their product’s engagement to the most addicted users — in a single snapshot, over time, and also in relation to monetization. This is useful. So how does it work? The Power User Curve will “smile” when things are good The shape of the Power User Curve can be left-leaning or smile-like, all of which means different things. Here’s a smile: The Power User Curve above is for a social product, and shows the characteristic smile shape that indicates there’s a group of highly engaged users using the app daily or nearly daily. Social products with frequent user engagement like this lend themselves well to monetization via ads—there’s enough users returning frequently that the impressions can support an ad business. Remember that Facebook would have a very right-leaning smile, with 60%+ of its MAUs coming back daily. What matters is that, over time, the platform is able to retain and grow its power users: successive Power User Curves should ideally show users shifting over more to the right side of the smile. As the density of the network grow, and with stronger network effects, it’s expected that there’s more reason for users to return on a daily basis. Let’s look a different example, which doesn’t smile: The Power User Curve can show when strong monetization is needed This Power User Curve of a professional networking product looks quite different than that of a social product. It’s left-weighted with a mode of just 1 day of activity per month, and decays rapidly after those few days. There’s no power users. But this light engagement can be okay — not every company needs to have a smile-shaped Power User Curve, just as not every product category necessarily lends itself to an ultra-high DAU/MAU. When there’s low engagement, what matters is that the company has a way to extract enough value from users when they areengaged. Think about an investing product like Wealthfront or networks like LinkedIn — few users are likely to actively check it on a daily basis, but that’s ok, since they have business models that aren’t tied to daily usage. CEOs of such companies should therefore,think about: Is there a way to create revenue streams where the business can still monetize effectively despite users’ infrequent engagement? Or, who are the users using this product more frequently, and how can I get more of them? Is there something about the product — e.g. onboarding, the core experience, etc. — where a significant chunk of the user base isn’t experiencing the ‘aha moment’ that makes them “get” the product, and therefore not getting value from it right now (and if so how to get there)? Some products should be analyzed in a 7 day timeframe – like SaaS/productivity – and others on 30 days Another flavor of the Power User Curve is a histogram of users’ engagement for a 7-day period, also commonly called L7. The 7 day Power User Curve shows weekly actives, not monthly actives. Plotting this version can make sense if your product naturally follows a weekly cycle, for instance, if it’s a productivity/work-related product that users engage with Monday through Friday. B2B SaaS products will often find it useful to show this version, as they want to drive usage during the work week. Note that using DAU/MAU wouldn’t be the appropriate metric for this product as it’s not designed to be a daily use product. You can also see there’s actually a smile curve through 5 days, but fewer users are using it 6-7 days, which makes sense for the power users of a workweek product like this. CEOs of such product companies should therefore want to understand: Who are the users engaging just 1 or 2 days each week? Are there certain teams or functions within an organization that are getting more value, and how can I build out features to capture the teams with less engagement? Or, if the product is really driving a lot of value for specific departments — how can I understand their needs better and make sure we continue building in a direction that supports their daily workflow (and that we can upsell new features)? Plotting the Power User Curve for different WAU or MAU cohorts can also be very insightful. Over time, you can see if more of your user base are becoming power users, by seeing the shift towards higher-frequency engagement. The trend of over time can show if the product is getting more engaging over time Here’s an example: The Power User Curve for MAU cohorts from August through November shows a positive shift in user engagement, where a larger segment of the population is becoming active on a daily basis, and there’s more of a smile curve. You can see when the line starts to inflect in order to see when a critical product release or marketing effort might have started to bend the curve.  This might be a place to double down, to increase engagement. For a network effects product, you might expect to see newer cohorts gradually improve as you achieve network density/liquidity. On an ongoing basis, you can measure the success of product changes or new releases by looking at different cohorts’ Power User Curves. If a product unblocks a bunch of features for power users, you might see a gradual increase in power users. The frequency histogram can be keyed on actions beyond the visit — did someone show up or not — you can also go with deeper user actions. For instance, you may want to plot the core activity that maps closely to how your business is monetized… or that better represents whether users are getting value from your product. This is important because it forces you to think about what really matters to measure. The Power User Curve can be based on core activity, not just app opens or logins The above chart for a content publishing platform shows the total number of days in the month users posted content. A lot of products have smile-shaped core activity Power User Curves, because while most people tend to contribute lightly, there is a small contingent of users who are power users. Think of the distribution of Youtube creators, or Ebay sellers, or even how often you post on Facebook. As the CEO or product owner of a platform like this, it’s important to design the platform such that the everyone has a chance to succeed. On Facebook, the news feed algorithm makes sure that if you feel strong affinity to a person or organization, you’ll still see their posts even if the sheer volume of other content (for instance, from more prolific media companies) would otherwise drown it out. On OfferUp, even if I seldom sell items, when I do list something, their algorithm makes sure that it’s surfaced to the relevant potential buyers. Not everything is a daily use product, and that’s okay. Why does this all matter? Power user analysis allows you to get a better understanding of how users are engaging with your product, and make more informed decisions using that data. That might mean choosing an appropriate business model that works for your pattern of engagement, or designing better re-engagement loops for lower-engaged user segments, or doubling down on use cases that your high-engagement user base is already getting value out of. The beauty of the Power User Curve over DAU/MAU is that it shows heterogeneity among your user base, reflecting the nuances of different user segments (and therefore what drives each of those segments). Creating versions of Power User Curve by various user segments can also be particularly insightful. For instance, for a business with local network effects (like Uber or Thumbtack), showing Power User Curves by market can reveal which geographies are developing density and strong network effects. Power User Curves show if your product is hitting a nerve among a super engaged core group of users, even if perhaps the overall blended DAU/MAU is low. It also doesn’t have to just reflect app opens or logins — you can hone in on an action that maps closely to users getting specific value out of your specific product and plot the Power User Curve for that action. The key for founders is to know that there isn’t a single silver bullet to measure perfect engagement — rather, the goal is to find the set of metrics that are appropriate for their businesses. Comparing the Power User Curve of a social app vs. a work collaboration app doesn’t make sense, but looking at your own Power User Curve over time, or finding benchmarks for your product category, can tell you what’s working… and what’s not. Originally published as “ " The Power User Curve: The best way to understand your most engaged users

What to Learn from the Power User Curve

Advancements in technology have made it possible for businesses to operate as normal with significantly fewer in-house staff. The is growing rapidly, with more freelancers and companies alike possessing the tools needed to quickly collaborate, share work and seamlessly communicate. gig economy The driving force behind the growth of freelance work can be found in cloud computing, where just about anybody in possession of an internet connection across the world can effectively work for a business elsewhere. (Figures suggest that the majority of US citizens will be engaging in some form of freelance work by 2027. Image: Website Planet ) As the cloud continues to leverage remote work, it’s widely expected that the gig economy will continue to grow throughout the 2020s. This can potentially be seen as good news for small businesses operating within a smaller working environment. With this in mind, let’s take a deeper look at the pros and cons for the gig economy, and explore whether remote workers can be an asset for small business owners: Pros: Cost-effective employment Of course, the greatest asset freelancers bring to is the lower costs. Not only do you get to save on the price of a salary, but also company perks, health insurance and office usage are saved. small businesses This is particularly useful if you’re planning on scaling in the near future and need to keep on top of your budgets. Without the need for catering to fixed salaries, you can manage your company money much better. Less risk The lower costs associated with hiring freelancers will always mean less risk, but it’s especially pertinent when you’re and the short term future of your business is not assured. managing a startup Since you only need to hire and pay freelancers when you have specific jobs to be done, it saves lots of time and money when compared to hiring somebody who may not always have a discernible workload on hand. ‘Downtimes’ can be part and parcel of a business that’s still establishing itself, so by hiring freelancers, you can go some way in limiting the risks involved in bringing in-house staff into your organisation. Strong vetting processes There’s no shortage of places where businesses can find their ideal freelancers, and platforms like , , and are great solutions that offer strong vetting processes to ensure that you only recruit the right worker for the task at hand. Freelancer.com UpWork Fiverr Thumbtack While the process of finding a freelancer was formerly arduous, digital platforms have made it a fairly speedy and streamlined job. This will be particularly good news to entrepreneurs who will undoubtedly have lots on their plate as their company establishes itself in a new market. Many platforms not only have a strong vetting process for applicants, but also run a review-based system for workers, so potential employers will be able to see an Uber-style star rating for each potential freelancer they recruit - helping to limit the possibility of a poor performer or bad fit for your business. Cons: No chance of supervision You may be capable of getting in contact with your freelancer, but there’s little chance of you supervising them in a similar way to that of an employee. This means that you’ll never really know the time they invest in your work, or how well they’re sticking to your brief, or there are certain aspects of their task that they don’t understand. Furthermore, freelancers can’t be trained by in-house staff, meaning they may be unable to comply with your own house styles or standardised practices. Fundamentally, there’s always the danger that you’re getting billed for a freelance worker to spend 30% of their hourly rate sat on YouTube. As an entrepreneur who needs to give your company the best chance of survival as possible, this lack of clarity could prove costly. Quality won’t be assured Despite there being strong vetting processes in place, the quality of the work you receive from your freelancer won’t be assured. There could be a range of unforeseen limitations in the work your , or worse, it could’ve missed some significant parts of the brief that was sent. freelancer produces Because there’s little chance of supervision, there’s a higher risk that you’ll receive work that isn’t right for your business. This could cost more time and money in correcting underlying errors in-house. Lack of loyalty Naturally, freelancers will have less interest in your business. They’ll likely be working for a number of companies at the same time and their primary focus will be on making ends meet. While it’s perfectly reasonable for a worker to put their own needs first, freelancers will have less vested interest in the performance of your business and less reason to stick around than in-house workers. Virtually every freelancer you use will be juggling multiple jobs at the same time, so their loyalty to one specific brand will generally be lower than that of a full-time member of staff. For many businesses, this may not matter, but it could be important if, say, you’re looking to use a freelancer on a recurring basis to convey a consistent tone of voice within your on-site content, for example. The gig economy is expanding at a rapid rate, and as technology continues to evolve, the market will carry on its rate of growth. There are plenty of positives to take on board when utilising the help of a freelancer, but some negatives could prove to be a sticking point for certain small businesses. Be sure to explore your options, allocated budgets and cash flow projections before deciding between freelancers or in-house staff. Technology has offered us an unprecedented level of choice when it comes to dealing with our operations, it’s up to you to decide whether you’re ready to delve into the gig economy or not.

Pros and Cons of The Gig Economy: Can Remote Workers Be Assets in Small Business?

Exception Handling Decades ago when a program crashed you would see a dire error message This meant that something horrible had happened. Usually the problem was the software tried to read or write some memory outside its address space. It could mean some other problem; a full disk, a device error, but it was an unrecoverable condition, something that software developers call an . exception There was only one option: tap the spacebar or use a mouse, and invoke “OK.” Why? It’s not OK. You may have just lost two hours of work (that’s your fault. Hit Ctrl-S every few seconds or you deserve to lose it). But this still sucked. Exceptions aren’t new. Dealing with them in a more elegant way than inelegantly crashing goes all the way back to 1962 in a near-forgotten language called LISP. See the Resources section for an article on the history. Languages in modern programming pulled the better ideas from the diverse implementations of handling exceptions and formed a simple language syntax with a few new keywords, and Structured Exception Handling (SEH) was born. The idea was that software developers could trap exceptions and offer a more elegant shutdown than simply crashing and losing everything: {
    do_something_unsafe();
} (Exception exception)
{
    save_work();
    apologize(exception); ();
} try catch exit So far so good. Unless the application data was corrupted by the Horrible Event you had a cleaner shutdown and could restart the program and continue. And it got better. Not all exceptions meant shutting down; some were recoverable. If the program tried to access forbidden memory it could return to the previous state and continue. Happy days. Can’t Leave Well Enough Alone Then some little smarty decided that the operating system shouldn’t have all the fun and that developers should be able to exception conditions and use this new architecture themselves, and in what is likely one of the worst things to ever happen to software development until pair programming came along, extended the new architecture with create ; throw Calling this new keyword would make the instruction pointer jump to the nearest block, wherever that may be; exception types were defined so the jump could be to a selected handler. This may have been a well-meant idea, more likely it was just to be orthogonal and provide developers with both ends of the transaction. catch Wise developers; experienced developers; thoughtful developers. Probably nobody ever thought what would happen when they put this catastrophic operation in the hands of less experienced developers. And we can’t put the genie back in the bottle. What’s So Special About Exceptions? The point of SEH was EH. Exception handling. Gravely serious conditions. But just as the keyword isn’t restricted to the end of a function, wasn’t restricted to exceptions. It wasn’t long before developers were using it in error conditions where return codes were more appropriate, like parameter validation. return throw And then for conditions that weren’t errors at all. And, inevitably, as a way to save a few minutes of typing. And a lot of these were the same people who compulsively hyper-optimized even the most performance-uncritical code, having no clue that exacts an enormous penalty. throw Example. Our intrepid short-attention-span test-driven-developer is three levels deep in functions and suddenly discovers a potential error condition. He doesn’t want to “refactor” three functions to return error codes, he has a foosball game with the team in five minutes, so he issues a call to and wraps the call to the first of the functions in a block and runs off to his game. Facepalm. void void throw void try-catch Throwing exceptions becomes just another tool in the box, a glorified ; why should it be restricted to real exceptions? longjmp Why indeed. And Then Came Java And in Java is idiomatic. Java developers are expected to throw routinely. It’s even better than return! Get a bad argument? Throw an exception. Didn’t find a value in an array? Throw an exception. throw Whee. Before this, using this sledgehammer of a keyword on a thumbtack of an issue was just bad judgment, Java came along and validated that bad judgment. The Year Everything Crashed Then SEH was added to C++. If you were using computers in the years that followed you remember the time well even if you didn’t know the reason; a lot of software that had been stable and solid before started crashing and misbehaving. In one year iTunes went from unbreakable to almost dangerous to use and you had to save off a copy of your music database or lose it every few days. There was a massive loss of software reliability and if you watched memory in your Task Manager you could see it leaking in torrents. Everyone was using everywhere, and it wasn’t working right. Garbage Collection was far from perfect, far from reliable, and developers using just expected things to be magically cleaned up. They weren’t. throw throw The Performance Penalty is hugely expensive. Some mobile groups at Microsoft started looking at their profiling data and their faces went white, word went out to people coding on weaker processors to call a halt to the keyword. Throw throw Processors and RAM have had to stay well ahead of the software running on them because too many developers write inefficient code. New languages had to collect allocated memory, cleaning up after sloppy work because hardly anyone had the discipline to properly manage memory (I remember a Microsoft division meeting where a David Stockman clone sagaciously told us that memory leaks were a fact of life). Rather than cultivate discipline the languages had to add toilet paper functionality. With this problem was squared and cubed. throw The Real Problem With Throw Control Flow Going all the way back to the earliest high-level languages we have had approximately the same constructs: ; functions with entrances and exits. Code could be analyzed, its behavior clearly deterministic, every path could be enumerated and every condition examined. This is control flow. Good developers could find bugs without a debugger, just reading the code. if/else, while, for Not Anymore Yes is expensive in CPU cycles, but that’s not the real problem. added a second system logic of control flow, disconnected from the usual one, unpredictable, hybridized and intertwined. Code using both could not be reliably analyzed, not by human eyes. Not unless the scope of and were very tightly and carefully constructed and if you think most developers are going to do that then you haven’t worked with many average ones. throw Throw throw catch One early large Java project was subject to analysis with advanced tools and found to have an astounding 13,000 ambiguous code paths based on thrown exceptions; the behavior of the code was massively unpredictable. What we predict, and with certainty, is that developers are going to write with the rigor required to use this keyword safely. Software development is disintegrating before our eyes; focus and concentration are deprecated and just as inefficient code is mitigated by ever faster processors, increasingly sloppy work is mitigated by an obsessive shift of focus from development to testing. can not People can’t enter a flow state anymore, not when their day starts with a slog of a commute to attend a daily scrum and they are interrupted all day by meetings and open office conversations. It’s not possible to rise above mediocrity when you can’t concentrate and software companies are more concerned with team cohesion than with quality work. And it’s in this world that we have this horrible keyword with its wholly unpredictable behavior. What We Can Do The Impossible Dream If I had my way the keyword would be removed from all languages that use it only the language runtime and operating system would be allowed to raise exceptions That isn’t going to happen; most projects would break. They would have to be rewritten. Within Our Grasp Languages with SEH should have a compiler option of not recognizing so it cannot be used. Barring that, . The mobile groups did so. Some languages already designate API that throw, calls to these API should be rigorously wrapped in blocks, , with a global catch around everything, or perhaps nested ones, solely to find unhandled throw conditions and patch them. Software is ready when the outer handlers are never hit. Outside the small local blocks the control flow returns to deterministic predictability. throw development leads should forbid its use local try-catch at the same scope as the exception-throwing API This preserves the original intent; programs don’t crash so much and recovery is possible. Software development goes back to being a deterministic endeavor and we regain some stability, probably lot of stability. Don’t use . Just stop. Stop now. throw Resources History of Exception Handling

Stories, News & Mentions about THUMBTACK

Thumbtack’s Stories on HackerNoon

Thumbtack's News & Mentions