Second Order Effects (SOE) is a California-based engineering firm specializing in custom electronics and end-to-end electronic system design

Second Order Effects

Most of us have already experienced at least some of the first-order impacts of COVID-19, such as the rapid shift to working remotely at home, schools teaching online, panic buying at retail shops, and restaurants closing their doors. So far, the global impact has been like nothing else we’ve experienced since at least World War II. Now that we’ve had a chance to move on from the initial shock, we’re beginning to experience some of the pandemic’s second-order effects on healthcare, the economy, and humanity as a whole. The new normal beyond the horizon is just starting to come into focus. The crisis will not only change how we live but will also yield new opportunities for discerning entrepreneurs willing to think critically about it all. So with that in mind, below are some of the business ideas emerging as silver linings of the second-order effects of COVID-19. Updated Delivery Services The demand for online delivery services for meals and groceries are already at an all-time high since the pandemic hit. But now there’s also a growing market for services that deliver things like pharmaceuticals and other miscellaneous commodities not already available via existing mainstream channels. Instead of using third-party providers, existing small businesses may use this opportunity to to local areas as part of their own services. Otherwise, entrepreneurs can focus on devising new and innovative hyperlocal delivery systems that are both cheaper and more efficient than any of the more mainstream services. Working with local businesses also provides an opportunity for locals to support each other. start offering delivery Remote Work Innovation Not long after the COVID-19 lockdown, employees proved that working from home was not only possible but also effective. Now that people have into home offices, most experts now agree that working remotely is here to stay. And the knock-on effects will be far-reaching, from less traffic and fewer car accidents to the reduced value of office space. renovated their spare rooms Companies will have to adapt systems and processes to suit remote work environments, as they have different needs than traditional offices. They will require the development of a wide range of remote services such as upgraded cybersecurity, invoicing software, file-sharing systems, and much more. Plus, this new global workforce will see an increased demand for those who facilitate outsourcing employment of temps and contract workers. Social Distanced Fitness The entire fitness industry as a whole has been hit pretty hard by the pandemic. So, while remote fitness was already growing in popularity before 2020, the extended gym closures combined with social distancing rules ultimately helped to accelerate the industry’s growth dramatically. Struggling fitness instructors quickly found ways they could reduce the negative impact of COVID-19 by adopting digital technology to create online instruction modules. Instructors developed training programs that they delivered to clients either by streaming the workouts online or via specially made personal training apps. By providing them with these forms of digital content, clients were able to work out independently with little or no at-home exercise equipment or learn how to use whatever else is available. Not only have these created steep competition for traditional gym memberships, but the whole idea of having your own virtual personal trainer will most likely explode into other exercise modalities as well as other non-exercise industries. new online workouts Accessed eCommerce As most people have been avoiding public spaces and unnecessary travel throughout the pandemic, the entire eCommerce sector has experienced a significant increase in growth. The reduced movement caused by working from home has seen more and more customers embrace digital channels over shopping at physical brick and mortar stores. This has proven how critical it has become for businesses to be able to operate and make sales online. Businesses everywhere need to integrate eCommerce to ensure they will not be impacted significantly by any future pandemics. Entrepreneurs can find ways of partnering with existing businesses to ensure they have a solid online presence, as well as finding opportunities to increase their sales. Search for new ways of providing them innovative services in areas like data management, digital marketing, and inventory solutions. Pivoting Entrepreneurship There are many innovative entrepreneurs out there who have already found ways of addressing the most critical needs of people during this crisis and turned them into sustainable businesses. From low-cost equipment created by healthcare professionals to contact tracing applications developed by engineers, there are innovations being created for the world today as well as after the pandemic has been dealt with. Innovating entrepreneurs are finding ways to into viable businesses by studying markets and identifying alternative issues they can solve. The most important aspect when attempting to identify suitable markets is by understanding what people currently need while conducting a thorough analysis of what the market will most likely want in the future. pivot their ideas Final Thoughts While the whole world continues to combat the ongoing effects of this pandemic, it’s important that you remain as optimistic as possible. Because ultimately it could end up providing you with an exciting new idea for a business of the future.

Innovative Business Tactics To Help You Pivot During The Second-Order Effects Of COVID-19

The premise of this article is: Wallets and Keys are Not Identities
Collapsing Keys and Identities is Dangerous
Decentralized Key Management is Tough but Powerful Public/private key pairs have been around for a long time. Useful for encryption and addressing, they are a cryptographic way to ensure that data is flowing between the right entities. While this article touches on public key infrastructure (PKI) and decentralized public key infrastructure (DPKI), I want to focus on the user perspective more than the systems of authentication. This is an article about user safety, socio-technical systems, and new technology. Wallets and Keys are Not Identities! Again, for the people in the background, crypto wallets and public keys are not identities. Even Decentralized Identifiers (DIDs) are not identities but rather simply identifiers. Identifiers within the space of digital identity include DIDs, crypto wallets, and many other implementations of public/private keys. The public key acts as a unique identifier, an address specific to the holder of the private key. It provides no other information about that key holder. The key could be held by anything from an IoT device to a profile on a personal computer to a server. The nature of that device or who has access to it is not held as part of the key itself. Identity is a little more complicated. Identity can be a legal tool: who you are in the eyes of the state — e.g., passports and driver's licenses. This is what is needed for KYC. This is the form of identity needed for a bureaucratic system to function and recognize you as a unique, unchanging entity. Identity is also social. It’s our relationships, personalities, hobbies, and cultures. These things aren’t really definable as discrete metrics, and as much as marketing companies want to capture them as such, we probably don’t want that. Social identities are all about progressive trust as we reveal aspects of ourselves to people over time. Social identities are relational. Each person who knows us knows us a little differently. Thus, there is no one social identity that you hold; rather, social identity exists in the eye of the beholder. Social and legal identity are separate things, and using digital identity to bind them together will have unintended negative consequences; for instance, platforms that monitor your activities and build a social profile that they attach to your legal identity abuse that data in turning you into a product sold to advertisers. The digital flattens nuance. We don’t want that sort of data going to our peers, either. Just as texting makes it hard to read tone, digital identities can give both too much and too little information at the same time. Collapsing Keys and Identities is Dangerous Keys are, of course, used to address and attach identity data, using documents that point to the key, signed by others making the claim — with implementations and proposals such as Soulbound Tokens, Disco, and more under the general banner of decentralized digital identity. Being able to attach verifiable data to an identifier in a decentralized system is massively powerful. However, it is very dangerous if the system is designed without consideration for second-order effects and if identities and identifiers are not properly differentiated. “Identity data” is used broadly here because, depending on the implementation, anyone can make any sort of claim about you and attach it to a digital identifier you control. Everything from web tracking data, to legal documents, to criminal records, to your concert tickets are all potential data points depending on how a system is implemented. Molly White describes the potential abuses of publicly visible identity information attached to crypto wallets where users have no recourse to edit or remove data. Even Web3 identity provider Civic points to similar concerns, encouraging the use of more private data stores. And Philip Sheldrake extends the point to talk about the dangers of collapsing our social and legal identities into each other. At the end of the day, I don’t want to use a system designed to present my legal identity and other data collected about me first and foremost. We must think about these systems beyond individual interactions and rather consider the way users will interact across time in emergent patterns. Protecting users should be the primary concern of digital identity solutions. We need to remember that keys are not people and have much broader uses than simply addressing identity data. Decentralized Key Management is Tough but Powerful In a lot of centralized systems, users do not have direct access to the private keys on their devices. This makes UX easy — offloading key management actions like revocation and replacement to the central service eases the burden on the user. A simplified analogy is: if you have forgotten your password, you can reach out to whatever entity you were trying to log into and prove your identity through other means to get a new password set up. With decentralized systems, where your key is authorized by itself alone, there is no such recourse. If you forget the password that decrypts your locally held private key, there is no central entity you can approach to ask them to revoke and replace your keys. So, how do we do decentralized key management well? And what are some ways we can make it safer and more empowering for users? Seed Phrases and Private Key Storage Anyone who has used a crypto wallet will be familiar with seed phrases. They are a series of random words which are used to generate the key pair. This allows the key pair to be regenerated by using the same seed phrase. To maintain the same identifier or access the same wallet on different devices, you need the same key. This is why seed phrases have to be kept secure: anyone who has access to your seed phrase has access to your private key and can act with the same authority as you can. So keep your seed phrase private and secure, but also don’t lose or forget it. A whole industry of seed phrase storage devices has sprung up around this challenge. But let’s not call any of it user-friendly. Key Revocation & Replacement Key revocation might not be so familiar to those using crypto wallets. This is because, for the most part, wallets cannot be revoked. This is something like never being able to change your address. You can move, but you can’t forward your mail. Literally a nightmare if someone manages to compromise your wallet, leaving you no recourse. Revocation in centralized systems is usually managed by public lists held by centralized entities, showing which keys have been compromised or replaced and pointing to the alternative, updated keys to be used in their place. As long as the central entity can be trusted, you can track which keys are relevant through their revocation lists, just like a forwarding address. Social Recovery The other key management tool that is being considered by a number of protocols is the social recovery of keys. Vitalik has talked about this and the Holochain team has also been working on an implementation, but the basic idea is that you can designate other keys as a collective authority that can revoke and rotate your keys for you. In a public document (like a smart contract on a blockchain, or a public DHT entry in Holochain) you list a set of keys that can revoke and rotate your key in the case that it is compromised. This document is then signed with your original key, proving the right to assign those custodians. Usually, this will follow some sort of multi-sig pattern where M (3) of N (5) keys need to sign the update to the document by rotating the keys. Who controls these custodial keys is up to the user, but usually, they might assign friends, family, or trusted institutions, along with keys that they keep backed up separately. The thing that makes this different from central systems is choice. This begins to reach the ideal of good user experience and user empowerment. People need to be able to choose what data they share and who is involved in the management of their keys, and those need to be easy ongoing decisions rather than technical nightmares. Takeaways We are not our keys. We are not even our data. (Or should I say the data that talks about us?) Social and legal identity are separate things. Conflating them through digital identity tools and attaching them to identifiers in ways that remove user agency will have negative consequences. Key management isn’t currently user-friendly. It’s worth putting effort into making it user-friendly. This will be required for safe adoption. We can do some cool things with decentralized key management. Personally, I’m looking forward to my passwords no longer being leaked in data breaches. But as we move the technology out of the realm of crypto and geeks to the general user, we need to be thoughtful about how we build technical, legal, and social norms around its use. Special thanks to Philip Sheldrake, Camila Hanada, Jarod Holtz, and Paul d’Aoust for their enriching reviews and assistance with this article’s feature image. All mistakes and opinions remain my own. The premise of this article is: The premise of this article is: Wallets and Keys are Not Identities Collapsing Keys and Identities is Dangerous Decentralized Key Management is Tough but Powerful Wallets and Keys are Not Identities Wallets and Keys are Not Identities Collapsing Keys and Identities is Dangerous Collapsing Keys and Identities is Dangerous Decentralized Key Management is Tough but Powerful Decentralized Key Management is Tough but Powerful Public/private key pairs have been around for a long time. Useful for encryption and addressing, they are a cryptographic way to ensure that data is flowing between the right entities. While this article touches on public key infrastructure (PKI) and decentralized public key infrastructure (DPKI), I want to focus on the user perspective more than the systems of authentication. This is an article about user safety, socio-technical systems, and new technology. Wallets and Keys are Not Identities! Again, for the people in the background, crypto wallets and public keys are not identities. Even Decentralized Identifiers (DIDs) are not identities but rather simply identifiers. Decentralized Identifiers (DIDs) Decentralized Identifiers (DIDs) Identifiers within the space of digital identity include DIDs, crypto wallets, and many other implementations of public/private keys. The public key acts as a unique identifier, an address specific to the holder of the private key. It provides no other information about that key holder. The key could be held by anything from an IoT device to a profile on a personal computer to a server. The nature of that device or who has access to it is not held as part of the key itself. Identity is a little more complicated. Identity is a little more complicated. Identity can be a legal tool: who you are in the eyes of the state — e.g., passports and driver's licenses. This is what is needed for KYC. This is the form of identity needed for a bureaucratic system to function and recognize you as a unique, unchanging entity. Identity is also social. It’s our relationships, personalities, hobbies, and cultures. These things aren’t really definable as discrete metrics, and as much as marketing companies want to capture them as such, we probably don’t want that. Social identities are all about progressive trust as we reveal aspects of ourselves to people over time. Social identities are relational. Each person who knows us knows us a little differently. Thus, there is no one social identity that you hold; rather, social identity exists in the eye of the beholder. progressive trust progressive trust Social and legal identity are separate things, and using digital identity to bind them together will have unintended negative consequences; for instance, platforms that monitor your activities and build a social profile that they attach to your legal identity abuse that data in turning you into a product sold to advertisers. The digital flattens nuance. We don’t want that sort of data going to our peers, either. Just as texting makes it hard to read tone, digital identities can give both too much and too little information at the same time. Collapsing Keys and Identities is Dangerous Keys are, of course, used to address and attach identity data, using documents that point to the key, signed by others making the claim — with implementations and proposals such as Soulbound Tokens, Disco, and more under the general banner of decentralized digital identity. Being able to attach verifiable data to an identifier in a decentralized system is massively powerful. However, it is very dangerous if the system is designed without consideration for second-order effects and if identities and identifiers are not properly differentiated. “Identity data” is used broadly here because, depending on the implementation, anyone can make any sort of claim about you and attach it to a digital identifier you control. Everything from web tracking data, to legal documents, to criminal records, to your concert tickets are all potential data points depending on how a system is implemented. Molly White describes the potential abuses of publicly visible identity information attached to crypto wallets where users have no recourse to edit or remove data. Even Web3 identity provider Civic points to similar concerns, encouraging the use of more private data stores. And Philip Sheldrake extends the point to talk about the dangers of collapsing our social and legal identities into each other. Molly White describes Molly White describes Civic points to Civic points to Philip Sheldrake extends Philip Sheldrake extends At the end of the day, I don’t want to use a system designed to present my legal identity and other data collected about me first and foremost. We must think about these systems beyond individual interactions and rather consider the way users will interact across time in emergent patterns. Protecting users should be the primary concern of digital identity solutions. We need to remember that keys are not people and have much broader uses than simply addressing identity data. Decentralized Key Management is Tough but Powerful In a lot of centralized systems, users do not have direct access to the private keys on their devices. This makes UX easy — offloading key management actions like revocation and replacement to the central service eases the burden on the user. A simplified analogy is: if you have forgotten your password, you can reach out to whatever entity you were trying to log into and prove your identity through other means to get a new password set up. With decentralized systems, where your key is authorized by itself alone, there is no such recourse. If you forget the password that decrypts your locally held private key, there is no central entity you can approach to ask them to revoke and replace your keys. So, how do we do decentralized key management well? And what are some ways we can make it safer and more empowering for users? Seed Phrases and Private Key Storage Anyone who has used a crypto wallet will be familiar with seed phrases. They are a series of random words which are used to generate the key pair. This allows the key pair to be regenerated by using the same seed phrase. To maintain the same identifier or access the same wallet on different devices, you need the same key. This is why seed phrases have to be kept secure: anyone who has access to your seed phrase has access to your private key and can act with the same authority as you can. So keep your seed phrase private and secure, but also don’t lose or forget it. A whole industry of seed phrase storage devices has sprung up around this challenge. But let’s not call any of it user-friendly. seed phrase storage devices seed phrase storage devices Key Revocation & Replacement Key revocation might not be so familiar to those using crypto wallets. This is because, for the most part, wallets cannot be revoked. This is something like never being able to change your address. You can move, but you can’t forward your mail. Literally a nightmare if someone manages to compromise your wallet, leaving you no recourse. Revocation in centralized systems is usually managed by public lists held by centralized entities, showing which keys have been compromised or replaced and pointing to the alternative, updated keys to be used in their place. As long as the central entity can be trusted, you can track which keys are relevant through their revocation lists, just like a forwarding address. Social Recovery The other key management tool that is being considered by a number of protocols is the social recovery of keys. Vitalik has talked about this and the Holochain team has also been working on an implementation, but the basic idea is that you can designate other keys as a collective authority that can revoke and rotate your keys for you. talked about this talked about this Holochain In a public document (like a smart contract on a blockchain, or a public DHT entry in Holochain) you list a set of keys that can revoke and rotate your key in the case that it is compromised. This document is then signed with your original key, proving the right to assign those custodians. Usually, this will follow some sort of multi-sig pattern where M (3) of N (5) keys need to sign the update to the document by rotating the keys. Who controls these custodial keys is up to the user, but usually, they might assign friends, family, or trusted institutions, along with keys that they keep backed up separately. The thing that makes this different from central systems is choice. The thing that makes this different from central systems is choice. This begins to reach the ideal of good user experience and user empowerment. People need to be able to choose what data they share and who is involved in the management of their keys, and those need to be easy ongoing decisions rather than technical nightmares. People need to be able to choose what data they share and who is involved in the management of their keys, and those need to be easy ongoing decisions rather than technical nightmares. Takeaways We are not our keys. We are not even our data. (Or should I say the data that talks about us?) Social and legal identity are separate things. Conflating them through digital identity tools and attaching them to identifiers in ways that remove user agency will have negative consequences. Key management isn’t currently user-friendly. It’s worth putting effort into making it user-friendly. This will be required for safe adoption. This will be required for safe adoption. We can do some cool things with decentralized key management. Personally, I’m looking forward to my passwords no longer being leaked in data breaches. But as we move the technology out of the realm of crypto and geeks to the general user, we need to be thoughtful about how we build technical, legal, and social norms around its use. Special thanks to Philip Sheldrake, Camila Hanada, Jarod Holtz, and Paul d’Aoust for their enriching reviews and assistance with this article’s feature image. All mistakes and opinions remain my own. Special thanks to Philip Sheldrake, Camila Hanada, Jarod Holtz, and Paul d’Aoust for their enriching reviews and assistance with this article’s feature image. All mistakes and opinions remain my own.

Keys are Not Identities 

Economists Agree, Politicians Don't — So Guess Who Loses

অর্থনীতিবিদরা একমত, রাজনীতিবিদরা না - তাই আপনি হারান

Ökonomen sind sich einig, Politiker nicht – also verlieren Sie

Los economistas están de acuerdo, los políticos no, así que usted pierde

Les économistes sont d’accord, les politiciens non – donc vous perdez

अर्थशास्त्री सहमत हैं, राजनेता नहीं - इसलिए आप हारे

経済学者は同意するが、政治家は同意しない — だから負ける

경제학자들은 동의하지만 정치인들은 그렇지 않습니다. 그래서 당신은 패합니다.

Os economistas concordam, os políticos não - então você perde

Экономисты согласны, политики — нет, поэтому вы проигрываете

Ekonomistler Kabul Ediyor, Politikacılar Katılmıyor - Yani Kaybediyorsunuz

Các nhà kinh tế đồng ý, các chính trị gia thì không - Vậy là bạn thua

经济学家同意，政客不同意——所以你输了

Why PostgreSQL Is the Bedrock for the Future of Data

কেন PostgreSQL ডেটার ভবিষ্যতের ভিত্তি

Warum PostgreSQL das Fundament für die Zukunft der Daten ist

Por qué PostgreSQL es la base del futuro de los datos

Pourquoi PostgreSQL est le fondement de l'avenir des données

PostgreSQL डेटा के भविष्य का आधार क्यों है?

PostgreSQL がデータの未来の基盤となる理由

PostgreSQL이 데이터 미래의 기반인 이유

Por que o PostgreSQL é a base para o futuro dos dados

Почему PostgreSQL является основой будущего данных

PostgreSQL Neden Verinin Geleceğinin Temel Taşıdır?

Tại sao PostgreSQL là nền tảng cho tương lai của dữ liệu

为什么 PostgreSQL 是未来数据的基石

Meta recently conceded that in-app behavioral advertising in the EU can no longer be opt-in by default, marking the end of an era. The regulators won, at least for now. Every few decades, major shifts happen that change how an industry fundamentally operates and who the new winners and losers are. We are in the midst of one such shift in the digital advertising industry. This has been driven by three critical changes in the ecosystem, all privacy-related: Apple announced the in 2021 App Tracking Transparency framework Google is in 2024 deprecating third-party cookies A slew of privacy legislation is either already in place (like the , ) or is about to be in place (like the ) California Privacy Rights Act EU General Data Protection Regulation EU Digital Services Act While the effects of these changes have been felt by the advertising industry globally, the for a few years now. . In the past three weeks, there have been significant announcements from , , and about upcoming changes to their products in the EU. It’s probably fair to say these changes mark the end of an era for behavioral advertising in the EU, and the platforms have made peace with that. EU, in particular, has been at war with behavioral advertising And winning Meta Snap TikTok In this piece, we’ll dive into: How the advertising ecosystem runs (or used to run) Privacy interventions and impact on behavioral advertising EU’s war on behavioral advertising and why platforms are conceding Possible future paths How the advertising ecosystem runs (or used to run) Let’s start with some simple concepts. An ad shown to you can be contextual or behavioral (sometimes both). A is something that is shown alongside organic results (non-ads) in the context of whatever you are doing. For example, if you are on a food delivery app like DoorDash and you see an ad for a restaurant alongside organic results, that is a contextual ad. In that case, your identity doesn’t really matter. DoorDash knows your location and what you’re looking for and therefore shows you a relevant ad. Knowing your identity and interests does help improve the relevance of these ads, but it’s secondary to the context itself. contextual ad A is something that’s shown to you because of your past behavior. For example, you might have visited an office chair brand’s page on Instagram, and therefore you get served ads for another office chair brand; that’s exactly what happened to me with the ad on the right. This can either be based on direct behavior (like the office chair example) or inferred behavior — people like you who looked for chairs also looked for monitors. One step further, people who looked at chairs are also likely to work from home more and therefore be open to lunch service. All of those are potential ads that can be shown to you. behavioral ad A subset of products, most often search products, are perfect for contextual ads — Google Maps, Google Search, DoorDash, Yelp, Thumbtack, TripAdvisor, and Zocdoc, to name a few. The user has high intent to do something specific. However, most high-intent products are low-medium frequency. You are not looking for plumbers every day on Thumbtack, and you are not looking to order food delivery every hour of the day on DoorDash; therefore, the amount of contextual ad inventory is lower — highly effective but lower in volume. On the other hand, social media platforms where users spend hours doomscrolling have a large volume of non-contextual inventory. Random ads are annoying and ineffective, and therefore the most effective way to monetize this inventory is through behavioral ads. Unsurprisingly, . Before ~2018 (when EU’s GDPR kicked in), there was essentially a about you, collected through a long list of highly effective AdTech mechanisms. Some notable examples: behavioral ads require a lot of data, specifically data about you free flow of data Every iOS or Android phone had a unique “ ” assigned to it (and by association, to you). This ID was accessible to all apps on your device, i.e. your Uber app and your Google Maps app both knew you are mobile advertising ID 123 and could easily cross-identify you across apps. mobile advertising ID Most browsers (except Safari and Firefox) support “ ”, which are essentially a piece of data that one site could place on your browser and another site could access. There would be a “cookie ID” associated with you, and you could be easily cross-identified across websites. third party cookies Note that . You could go deep inside your settings and reset these IDs, but most people never do that. This resulted in a couple of second-order effects. these identifiers, for all practical purposes, were permanent First, companies that are “ ” started purchasing data from several different data providers and ; for example, app 1 could tell the broker that you bought an expensive piece of furniture, website 2 could tell them you have an account with a high-end bank, and the data broker could put that together and categorize you as “high propensity of spend” person; note that when I say “you,” that refers to your identifiers (mobile advertising ID or third party cookie ID) data brokers building “profiles” about you As more and more data became available, the profiles had more fidelity to them, and identification became more sophisticated; for example, if app 1 had your mobile advertising ID and website 2 had your Chrome third-party cookie ID, a data broker or an ad platform could compare your IP addresses and know it is the same person. Take this one step further. A data broker could also use a third data source and find an email associated with your IDs. One more step further, they could buy your address from a utility or telecom company and find your address. This might sound far-fetched, but it’s not — are a very commonly used data product in the AdTech stack today, and telecom companies are . If you need a more entertaining take, check out . device graphs notorious for selling your data John Oliver’s episode on data brokers Second, ; through a combination of technical mechanisms, an advertiser could say you first saw an ad on Facebook, then an ad on Google Search, then a display ad on NYTimes, and eventually bought an item from their website, so each of those three advertising platforms get shared credit. Attribution/measurement of advertising campaigns became more and more precise While this sounds privacy invasive (and it is), this resulted in a highly efficient advertising ecosystem. Advertisers knew exactly which users they were targeting, and since they had all these extra behavioral signals to know how likely a user was to engage, they were willing to pay higher cost per impression (CPM) for ad inventory, thereby generating more revenue for a media publisher. Precise attribution/measurement turbocharged this further. However, you can see how this was . This rightfully raised concerns about data consolidation in the hands of both data brokers and large technology companies, and an intervention was inevitable. It was less a matter of if and more a matter of when. becoming the wild west — highly effective advertising no doubt, but also an uncontrolled orgy of data acquired with non-existent or questionable user consent Privacy interventions and impact on behavioral advertising Let’s dive into each of the three privacy interventions that arose in an attempt to tame the Wild West. First, Apple introduced the . It sounds jargon-y, but the change is relatively simple. Prior to ATT, every app by default had access to your advertising ID, i.e. it was opt-in by default. This meant you could easily be tracked across apps and therefore shown effective behavioral ads. For example, you installed the Strava app to track your runs, you are now on Facebook, and you are shown an ad for Strava Premium. After ATT, the access to this , i.e. an app had to show you a pretty aggressive prompt to get access to your identifier, and you explicitly needed to say yes. The average opt-in rate ended up close to (with a lot of caveats).' App Tracking Transparency (ATT) framework identifier became opt-out by default ~34% We won’t go into much detail here, but this prompt was launched by Apple in the guise of embracing privacy — a smart chess move. The consensus opinion today is that this was an opportunistic move from Apple, which no doubt improves privacy but also heavily hurts Apple’s competitors as they . The impact was that user identity was available much less often. prop up their own ads business Note here that for behavioral advertising to happen in the Strava-Facebook example, there is not one but two apps that need to have received opt-in from you, i.e. the addressable market does not drop to 34%, it drops to 34% * 34% = ~12%. Therefore, . cross-app behavioral advertising on iOS is no longer effective at scale Second, Google announced that it will . The consensus opinion is that the change helps Google achieve a dual purpose: appease regulators who are breathing down their neck for , while taking control back from what’s now a . Google’s post third-party cookies will still allow cross-site retargeting, but in a more private way where all information is stored on-device within the browser, i.e. there are no more cross-site “cookie IDs” assigned to you. While the new mechanism preserves some of the status quo, . deprecate third-party cookies in 2024 potential anti-trust behavior in AdTech fairly bloated advertising tech stack new mechanisms cross-site behavioral advertising is going to have much less fidelity and, therefore, effectiveness Which brings us to the third intervention — . The most aggressive of these is the EU’s GDPR, which went into effect in 2018. The California Consumer Privacy Act (CCPA) went into effect in 2020. While the progression has been gradual, the reason these laws matter for advertising companies today more than ever is because the laws take aim at the (i.e. you do a bunch of different things inside the Facebook app and Facebook gets to use that data to show you behavioral ads within the app). privacy regulations only remaining and mission-critical advertising mechanism — behavioral advertising within companies’ own apps EU’s war on behavioral advertising and why platforms are conceding A primary feature that makes the California laws (arguably the strictest privacy law in the US) less aggressive than the EU’s GDPR is that it . For example, the 2020 California Consumer Privacy Act (CCPA) requires companies that are considered “data sellers” under the law to provide explicit opt-outs on web pages, but the . does not require explicit opt-ins and only requires platforms to provide opt-outs default is still opt-in EU’s GDPR takes this up another notch and requires for behavioral advertising. This consent needs to be . For example, Meta cannot gate content behind a behavioral advertising consent prompt. explicit opt-in/consent freely given, specific, informed, and unambiguous So, the simplistic inference from this is that Meta needs to get explicit consent for all behavioral understanding, including the last remaining mechanism — showing ads within their own platform. If Meta is forced to do this, the opt-in will likely be small (the Apple opt-in rates were ~34%), and this majorly shrinks Meta’s addressable advertising market in the EU. To not meet that fate, Meta made a creative legal argument: GDPR requires any company to have one of ** **if the company needs to process personal data. Paraphrasing, these are — consent, a necessity to fulfill a contract, a necessity for legal compliance, a necessity to protect the vital interests of the user, a necessity to perform a public interest task, and a necessity for legitimate interests pursued by the company. six legal bases Of these six, Meta held the stance that they were processing personal data using “ legitimate interests” (the last of the six) as the legal basis and therefore did not need to show an explicit opt-in prompt. Last month, the Norwegian Data Protection Authority that . Paraphrasing the : provided their enforcement decision Meta’s use of “legitimate interests” as the legal basis is not valid enforcement decision Based on several past court rulings, Meta must prove that legitimate interest “cannot reasonably be achieved just as effectively by other means less restrictive of the fundamental rights and freedoms of data subjects.” Based on another past court ruling, despite the fact that Meta provides a free service, the user of that network cannot reasonably expect that the operator of the social network will process that user’s personal data, without his or her consent, for the purposes of personalized advertising. Four days after the Norwegian Data Protection Authority’s ruling, Meta that they would be changing their legal basis from “legitimate interests” to “consent.” In practice, what this means is that . While Apple’s changes significantly cut down the ability to advertise cross-app, Meta was holding on to hope that it would be able to preserve all in-app behavioral advertising (including in the EU), and the writing is now on the wall. shortly after with their own announcements that non-personalized versions of their products will soon be available to users. A total of are likely to follow suit. announced Meta is conceding that behavioral advertising within their own app in the EU can no longer be opt-in by default and followed suit Snap TikTok 19 platforms that are in scope Possible future paths It is still to be seen what percent of users opt into behavioral advertising. If the Apple opt-in rates are any indication, it may get to the ~34% rate seen there, but it may also go higher if platforms are allowed to be creative about opt-in language and user interfaces. It is hard to say what exactly the long-term effects of ending the opt-in-by-default regime for behavioral advertising in the EU will be, but here are some educated guesses: — Meta has made towards letting users provide inputs to the feed algorithm. I would expect the creation of more explicit preferences/interests that users explicitly select. While it has worked to a smaller extent for some products like , it is likely that the quality of content provided by just preferences will be nowhere close to the relevance of a system that is constantly ingesting data and inferring interests. Explicit preferences instead of inferred interests some progress Reddit : TikTok has announced that they will create a that is based on what’s popular in your region. Most social feed-based platforms will also (re)introduce simplistic ranking options like chronological. I am not bullish that any of these will create an engaging user experience. Non-personalized feeds non-personalized feed : My (potentially contentious) take is that opted-out users will see a significantly degraded experience and eventually decide to provide platforms consent to personalize. The platforms will likely use this mechanism to receive content for all personalization (including behavioral advertising) — It’s unclear if there are any regulatory constraints that force them not to bundle consent for user experience personalization and behavioral advertising. I don’t expect the consent rate to go anywhere close to the status quo, but I think it will be a meaningful bump to what’s seen with Apple’s opt-in rates. Degraded user experience, leading to more opt-in : Twitter launched Twitter Verified a few months back, and Meta is with a similar concept. I am not bullish that a majority (or even a minority) of users will be willing to pay for social media products after years of being conditioned to get these products for free. Experiments with subscriptions experimenting If platforms are not able to effectively monetize the EU market, it is likely that they will start de-prioritizing investments for the EU market, which is a real risk of an aggressive regulatory regime. I don’t expect them to stop supporting EU markets, but every new feature built is an investment, and companies will start heavily questioning if they are willing to do any extra work for this market. For example, if there’s additional work to make a feature comply with GDPR, that might not be worth it anymore. We have seen other equivalent situations of this — several apps have a much better experience on iOS than Android because iOS apps monetize better. De-prioritization of EU markets: Conclusion The advertising market exploded over the last several decades due to the availability of essentially infinite user data, which brought both large efficiencies in advertising and major privacy risks. Regulation was necessary. Great regulation is about finding the balance between promoting innovation/letting new businesses emerge that move society forward and having guardrails so people are protected. Did the EU go too far? I personally think they did, and if that is true, the effects will start showing in a 5–10 year horizon, and course correction will follow. Until then, this is the new reality of behavioral advertising in the EU. The regulators and legislators came all guns blazing, and they won. 🚀 If you liked this piece, consider subscribing to Every week, I publish one in the form of a 10-minute read. Best, Viggy. . my weekly newsletter deep-dive analysis on a current tech topic/product strategy Also published . here

The New Reality of Behavioral Advertising in the EU

Early-stage companies are constantly evolving their product to fit the market they operate in. They reach customers to keep them engaged using a high magnitude vector that contributes to their success. Architecting the communications strategy for your product thus becomes an important problem to tackle, which in turn can cause second-order effects like having to trade off the speed of product development iterations. The decisions you would have to make along the path can be hard, confusing and prone to change, and the last thing you want to invest precious time and energy in is these non-core features of your product. In this blog post, we start by diving into Nodemailer, a module that helps send emails from your Node.js backend, and then we steer our way into writing a transport layer plugin that can help you switch downstream email service providers purely by configuration instead of tedious and sometimes massive code changes. Projects that have stood the test of time In the fast-growing tech industry, there’s a special place in the hall of fame for projects that have stood the test of time. is one such project that deserves a well-earned spot. Launched in 2010, not long after Node.js’ inception, Nodemailer has been a go-to dependency for sending email messages for Node.js users. With 14k+ stars and 438k+ usages on and ~1.48M weekly downloads on NPM as of this writing, the Node.js developer community has benefited immensely from the project. Nodemailer GitHub Apart from a wide variety of features provided by Nodemailer, it comes with zero dependencies and has been evolving with a security-first mindset. While some of us might think how hard it can be to have zero dependencies, let's remember we’re in a Node.js world where there is a package for every little thing you could possibly imagine. Don’t believe it? Brace yourself while looking at and . https://www.npmjs.com/package/is-odd https://www.npmjs.com/package/is-even Yep, that’s right - there’s a dependency that can help determine if a number is odd or even. No more complex arithmetic, right? On another note, Nodemailer’s security-first development also deserves a round of applause. Those of you who have had to frequently run audits and rack your brains figuring out how to resolve vulnerabilities would agree. Moreover, considering the scale at which projects like Nodemailer have been adopted, even a single vulnerability can have huge negative impacts on the software of the world. One recent example is a that could potentially play a hand in malware delivery. vulnerability in the netmask npm package Extensibility and elegant abstractions are the cruxes Nodemailer is highly extensible via plugins that can operate on the mail object (pre-processing step), the mail stream (processing step) or the transports (sending step). For the scope of this blog post, we’re mostly concerned about the sending step. is the main transport in Nodemailer for delivering messages - but can be extended using the transport plugin to a different transport mechanism. SMTP For example, if you already use , you can use the SES transport plugin instead of SMTP, which is built-in. Like SES, there are several email service providers out there like SendGrid, Mailgun, Postmark, and SparkPost to name a few. In order to cater to the Node.js (and Nodemailer) audience, most of these providers do have a transport plugin of its own should you need to integrate with your tech stack. Amazon SES You can explore through all such plugins with a simple search query on the NPM registry. “Nodemailer transport” A definite question that always springs to mind is why there are so many email service providers out there and how to choose the one that fits best with your communication strategy. Unsurprisingly, such analysis paralysis is fairly ubiquitous in today’s SaaS world. Calling it a SaaS paradox of choice would hardly be an overstatement. Unless you have theoretically infinite resources at your disposal (read: you’re at a giant like Google or Facebook), these choices can slow down the speed of development significantly and don’t bode well with the “move fast and break things” mantra if you have to go back and forth between email service providers and rewrite your backend stack to accommodate the changeset. Source: https://xkcd.com/2224/ Classic build vs buy mindset In one of our blog posts, we talked about the build vs buy mindset and how it makes a difference, especially if you are a startup - read about it if you’re curious. The mindset not only makes a difference for your speed of iteration but also keeps you potentially ahead in the game especially if and when your competitors are adopting buy-over build for non-core features of their product. here One of the best examples is hosting on AWS vs maintaining on-premises cloud infrastructure - outsourcing cloud hosting to AWS gives you the speed of iteration with no cost of infrastructure maintenance. Imagine all of your competitors are on AWS but you aren’t - it’d certainly make a tremendous negative difference in the long run. While this certainly does not necessarily apply to all the companies out there, it does hold true for the most. “Disruptive economic events like COVID have caused many people to step back and think about how they want to change strategically. And many have come to the conclusion that they do not want to own and run their own data centers.”, says AWS CFO Brian Olsavsky . https://www.theregister.com/2021/07/30/amazon_q2_2021/ An effective communication strategy plays a critical role in maximizing the odds of product success. It's a no-brainer that you have to have a way to reach out to users via email - but that’s not going to be enough. Customer engagement via text messaging, in-app messaging or direct messaging (think Slack, Whatsapp, MS Teams, etc.) has become a necessity in today’s hyper-competitive market. Sooner or later (definitely sooner than you think), having these multiple channels of communication is going to make a difference to your business. As you would scale your product or company, notifications infrastructure has to scale as well in order to handle queuing, scheduling, routing, rendering, and ensuring delivery of messages across multiple channels. Companies like LinkedIn have spent millions to build this infrastructure in-house - depicted quite well in blog post. LinkedIn’s Air Traffic Controller Continuing my mini-rant on the SaaS analysis paralysis, as you’d have probably seen it coming, the paralysis exists on text messaging too - Twilio, Sinch, Plivo, and Vonage to name a few providers in the arena. Managing multiple channels and providers to suit the communication strategy you’ve embraced is not a cakewalk. Let’s circle back to the never-ending list of email providers and their corresponding Nodemailer transport plugin implementations out there. An easy way to fight the paradox is with a layer of abstraction, such that you can change providers without having to rewrite the code to switch the transport plugins. That’s exactly where Courier comes into the picture with its API layer of abstraction and a configurable UI to switch email providers without having to change the code on your end. Needless to say, Courier should expose its own Nodemailer transport plugin which abstracts using a specific email provider plugin. Having this abstraction would imply no vendor lock-in as far as downstream delivery is concerned. Show me the code! Let’s dive right into writing the plugin! (specifically the transport section) and a would be our reference points. While the core documentation is exhaustive, we’ll keep it relatively simple for the first version of our plugin. Nodemailer plugin creation Mailgun implementation TL;DR: is the code we’ll end up writing. https://github.com/tk26/nodemailer-courier-transport Step 1 - Create a new NPM package and adding courier-node as a dependency Here's the for communicating with the Courier REST API. The source code behind the package is MIT licensed at the repo. Node.js module courier-node Step 2 - Configuration prerequisites You will need to get a Courier API key to get started. You can sign up and create one for free at . Courier being a middleware, you’d have to configure an email provider of your choice that’d do the actual delivery of the email to the recipient. Stuck in the paradox of choice? We got you! Here’s an we published earlier this year that could be helpful in figuring out. Thereafter, you’d have to create a notification template with email as the channel. how to do it using SendGrid. courier.com article Here’s Step 3 - Writing the transport core Transports have a method and and properties. Transport object gets passed to the method to create the object. send () name version createTransport () transporter const transport = (options) => {
  const courier = CourierClient({ authorizationToken: options.apiKey });

  const courierSend = input => courier.send({
	eventId: input.eventId,
	recipientId: input.userId,
	profile: {
  	email: input.to,
	},
  });

  return {
	name: "Courier",
	version: packageData.version,
	send: send(courierSend),
  };
}; That’s pretty much it for the basic version of the plugin. We’ve pushed the code to . feel free to play around with it and reach out with any questions/feedback. It’s open-sourced under MIT license so feel free to contribute to the open-source project. https://github.com/tk26/nodemailer-courier-transport Step 4 - Using the code in your tech stack Here’s how things would look in your Node.js backend - const nodemailer = require("nodemailer");
const courierTransport = require("nodemailer-courier-transport");

// This is your API key that you retrieve from courier.com account (free up to 10k monthly sends)
const auth = {
  apiKey: "ABCDEFGHIJKLMN",
};

const nodemailerCourier = nodemailer.createTransport(courierTransport(auth));

nodemailerCourier.sendMail(
  {
	eventId: "courier-event-id", // Configured in Courier UI
	to: "recipient@domain.com",
	userId: "unique-user-id",
  },
  (err, info) => {
	if (err) {
  	console.log(`Error: ${err}`);
	} else {
  	console.log(`Response: ${info}`);
	}
  }
); Note: We haven’t published the package yet as it is still in a nascent stage. Let us know if you are looking to contribute to the package or want us to publish it so you can use it in your backend. :) Key takeaways Let's summarize the key takeaways from the post. First and foremost - writing and maintaining open source projects that stand the test of time is hard, and projects like Nodemailer are great examples to learn from. Buying over building non-core product features is one of the most important trade-offs you can do to grow your company faster. The sooner you do the trade-off, the higher the returns (and in turn the competitive advantage) you gain. Last but not least, communication strategy is a key pillar in your company’s success - make sure you get it right without compromising on the speed of iteration. If you’re looking for an easy to integrate, scalable, and reliable communication strategy - sign up for a free account. You build the next big thing and let us handle the communications for you! 🚀 Courier

The SaaS Paradox of Choice: A Nodemailer Story

As we get down to work, our mind is flooded with thoughts - " ", " ", " "... should I respond to that email I got last night or a message from my boss how about a cup of coffee why not start the project I have been avoiding this entire week or maybe just a quick catch up on the meeting I missed yesterday A series of these small decisions scattered throughout our day may seem harmless in the moment as they seem to demand only a small fraction of our mental energy, but as the day goes on and we continue to expend from this seemingly reserved pool of energy, our mental capacity to make decisions starts depleting. Unlike physical fatigue which we can feel and instantly express, the mental fatigue that comes after making multiple decisions is not visible to us. Decision fatigue as psychologists like to call it leads to poor quality of decision making - we show reluctance to make trade-offs, fall back to easy choices and may even find it difficult to exercise self-control after making a series of decisions. "If your work requires you to make hard decisions all day long, at some point you’re going to be depleted and start looking for ways to conserve energy. You’ll look for excuses to avoid or postpone decisions. You’ll look for the easiest and safest option, which often is to stick with the status quo." - Roy Baumeister Ordering a pizza as opposed to a healthy meal in the evening after a long session of decision making at work is a sign that our self-control against impulses decreases in the face of decision fatigue. Social psychologist Roy Baumeister describes this inability to exercise self-control as in his study on decision fatigue. ego depletion Decision fatigue also explains why we are more reluctant to approve new ideas, find it difficult to choose from a number of presented options and end up making poor choices in the evening as opposed to making these decisions in the morning when our reserved bank of mental energy is at its best. Before we get down to reduce and maybe even avoid decision fatigue, it’s important to recognise its symptoms. Knowing when we are running on a low mental battery will enable us to avoid making bad decisions and find creative ways to save the battery for important decisions. How to Identify When You are Experiencing Decision Fatigue Recognise if you are experiencing decision fatigue by watching out for these symptoms - 1. Recklessness Decision fatigue will cause you to act impulsively instead of taking the time to think through the consequences of your decisions. It shows up when you fall back to your instincts or choose an option just because its first order consequence is positive without analysing the second order effects of your decision. Taking the time to think through the impact of your decision requires putting multiple options on your mental stage and comparing them. A tired mind isn’t equipped to provide the resources for weighing in on different options, challenging your assumptions and navigating the potential obstacles required to make a thoughtful decision. "Effort is required to maintain simultaneously in memory several ideas that require separate actions, or that need to be combined according to a rule. System 2 is the only one that can follow rules, compare objects on several attributes, and make deliberate choices between options. The automatic System 1 does not have these capabilities." - Daniel Kahneman With your system 2 deliberate thinking mode refusing to cooperate, your system 1 automatic thinking springs to action and system 1 is known to act on impulse or as Daniel Kahneman calls it: “ ” The law of least effort is operating here. He will think as least as possible. 2. Choosing the status quo It’s easy to stick to what you know while exploring the unknown requires crossing the mental boundary from comfort to discomfort. "Picturing something you have not yet seen is going to take a lot of energy and effort. This partly explains why people spend more time thinking about problems (things they have seen) than solutions (things they have never seen). It explains why setting goals feels so hard (it’s hard to envision the future)." - David Rock After making multiple decisions throughout the day, decision fatigue can cause you to stay inside your comfort bubble and resist the idea of a change since it’s uncomfortable and demands a lot of effort and energy. Decisions made under the effect of decision fatigue with a mentally tired state can have a huge impact on your future. Sticking with the status quo can cause you to - Invest in sunk costs while ignoring the multiple opportunities that lie ahead Ignore technological advancements and shift in market trends that require you to reevaluate your strategies Reject proposal for a new product that could be a game changer for the company Fail to see value in making small changes that could yield significant rewards 3. Stuck with analysis paralysis Making a good decision requires bringing knowledge from several domains, stitching distinct ideas together, maintaining the right balance of exploring breadth and depth across these ideas, making new connections and then zeroing in on a few candidates that look promising. When your mental machinery is tired, you will find it hard to process information to separate noise from the goodness of an idea. This may lead to overthinking, the tendency to think too much and move back and forth on ideas without the ability to give them a specific direction. Decision fatigue may cause you to obsess about the problems that don’t even exist, draw biased conclusions and be stuck with analysis paralysis with the possibility that there might be something better. It can cause you to chase a perfect solution instead of making the best decision under the given circumstances. 4. Too many options Decision fatigue is not necessarily caused by making too many decisions, it could also be a result of too many choices. We prefer having choices even when they don’t necessarily add value to our decision. At work, too many choices can cause designers to waste hours in finalising the right shade of blue, marketers may find it difficult to choose the channel that will give them the maximum return for their investment, hiring manager may be burdened with the task of finalising a recruiting agency amongst the many possibilities and engineers may argue over minor details in the strategic planning phase to lock down on the specific tactics to be employed. Choices can be overwhelming and too many choices can tax your mind by consuming resources in a short frame of time leaving less energy for making further decisions. "Learning to choose is hard. Learning to choose well is harder. And learning to choose well in a world of unlimited possibilities is harder still, perhaps too hard." - Barry Schwartz 5. React aggressively Under the effect of decision fatigue, your brain's regulatory power weakens causing you to lose control over your emotions. This makes everything around you feel more intense than normal. Small mistakes can make you furious, disagreements may cause irritation and you may react aggressively to things that are not in line with your expectation. Without your realisation, the switch in your brain may flip after many rounds of decision making causing you to react to people and things in your environment instead of taking the time to analyse and act appropriately. How to Overcome Decision Fatigue Decision fatigue is a real thing, but it’s also something completely under your control. Only you can design a personal system that maximises your chances of making the best decision as opposed to trying to squeeze out every bit of your attention. Employ these 5 practices to avoid decision fatigue: 1. Be an essentialist We make a large number of decisions from the time we wake up - some consciously, others unconsciously, some trivial, others significant, some reversible, others irreversible. Without controlling the number of decisions we make by choice and dedicating our peak mental energy to those significant decisions that impact our lives in a huge way, we may end up with poor judgment. Inspired by Greg McKeown from his book , we need to be essentialists when it comes to decision-making: Essentialism "The way of the Essentialist means living by design, not by default. Instead of making choices reactively, the Essentialist deliberately distinguishes the vital few from the trivial many, eliminates the nonessentials, and then removes obstacles so the essential things have clear, smooth passage. In other words, Essentialism is a disciplined, systematic approach for determining where our highest point of contribution lies, then making execution of those things almost effortless." The first step to combat decision fatigue is to organise your life in a way that you prioritise what needs prioritising and put all your energy into getting it right. So let’s jump to this step in the process, which is to make your commitments visible by planning upfront. 2. Get things out of your head Identify activities that align with your goals and values and say no to all inconsequential activities. Then spend dedicated time daily to plan what you wish to achieve each day. Planning upfront has two major advantages: You don’t have to think about what demands your attention as you start your day It serves as a commitment and point of reference when you find yourself distracted By not spending mental cycles in deciding every instant on what to do next, you can avoid decision fatigue and free up more resources to do the real work. It will also help you attend to those urgent situations which are not under your control with the right state of mind. Putting this practice into action requires what David Rock suggests - "Creating visuals for complex ideas is one way to maximize limited energy resources. Another way involves reducing the load on the prefrontal cortex whenever possible. The idea is to get the concepts out of your mind and into the world and to save the stage for the most important functions. Minimize energy usage to maximize performance." A good way to get the plan out of your head is to utilise to divide your work in 4 quadrants based on their urgency and importance. The next step is to order these priorities for optimal resource usage - eat that frog first. Eisenhower’s productivity matrix 3. Eat that frog Tackle your most challenging task first, one that demands your mental capacity to process information at its best. "The stage uses up power quickly, and as the lights dim, it gets harder to hold actors in the right place and stop others from getting on the stage. This tendency means scheduling the most attention-rich tasks when you have a fresh and alert mind. This could be early in the morning, or perhaps after a break or exercise. Making a tough decision might take thirty seconds when you are fresh and be impossible when you’re not." - David Rock By prioritising work that will have the greatest positive impact on your life and matching the mental demands of the work with your energy levels, you will be able to make the best decisions when you are not under the effect of decision fatigue. What do you do when things don’t pan out the way you wanted? 4. Believe you have more willpower As much as we crave an organised world and like to be in control, things don’t always go according to a plan. You may be forced to make a decision when you are tired. What should you do then - say no to making a decision? Well, sometimes that may not be a possibility. Stanford University Professor of Psychology Carol Dweck found: “ " She states that " " She notes that " " that while decision fatigue does occur, it primarily affects those who believe that willpower runs out quickly. people get fatigued or depleted after a taxing task only when they believe that willpower is a limited resource, but not when they believe it's not so limited. in some cases, the people who believe that willpower is not so limited actually perform better after a taxing task. This does not mean that you will never feel decision fatigue if you keep making one decision after another. Rather, it guides you to believe in a growth mindset and exercise your willpower when put under tough situations, when you have to make a decision even though your mental resources have depleted. By exercising your willpower in such moments, you may not make the best decision, but it will be much better than the one without it. 5. Know when it’s time to stop Roy Baumeister, a social psychologist, known for his work on willpower, self-control and self-esteem says: “T ” he best decision makers are the ones who know when not to trust themselves. Even the wisest people won’t make good choices when they’re not rested and their glucose is low. That’s why the truly wise don’t restructure the company at 4 p.m. They don’t make major commitments during the cocktail hour. And if a decision must be made late in the day, they know not to do it on an empty stomach. Know when it’s time for you to stop - take feedback from your past decisions, identify patterns based on your best and worst decisions and incorporate it into your planning. By knowing when you operate well and when you are physically present but mentally absent, you can make a choice to opt-out of the decision process and push those decisions to a later stage. Be the master of your own life. Don’t let it run on autopilot by learning to make a few decisions and make them well by being an essentialist, getting things out of your head, eating that frog, exercising your willpower and opting out when you aren’t adding value. Previously published here .

How to Identify and Stick it to Decision Fatigue

We recently published a series of tutorial videos and on the Apache Kafka® platform as we see it. After you hear that there’s a thing called Kafka but before you put hands to keyboard and start writing code, you need to form a mental model of what the thing is. tweets give you the basics you need to know to have the broad grasp on Kafka necessary to continue learning and eventually start coding. This post summarizes them. These videos Events Pretty much all of the programs you’ve ever written respond to events of some kind: the mouse moving, input becoming available, web forms being submitted, bits of JSON being posted to your endpoint, the sensor on the pear tree detecting that a partridge has landed on it. Kafka encourages you to see the world as sequences of events, which it models as key-value pairs. The key and the value have some kind of structure, usually represented in your language’s type system, but fundamentally they can be anything. Events are immutable, as it is (sometimes tragically) impossible to change the past. Kafka topics Because the world is filled with so many events, Kafka gives us a means to organize them and keep them in order: topics. A topic is an ordered log of events. When an external system writes an event to Kafka, it is appended to the end of a topic. By default, messages aren’t deleted from topics until a configurable amount of time has elapsed, even if they’ve been read. Topics are properly , not queues; they are durable, replicated, fault-tolerant records of the events stored in them. logs Logs are a very handy data structure that are efficient to store and maintain, but it’s worth noting that reading them is not too exciting. You can really only scan a log, not query it, so we’ll have to add functionality on a future day to make this more pleasant. Partitioning Topics are stored as log files on disk, and disks are notoriously finite in size. It would be no good if our ability to store events were limited to the disks on a single server, or if our ability to publish new events to a topic or subscribe to updates on that topic were limited to the I/O capabilities of a single server. To be able to scale out and not just up, Kafka gives us the option of breaking topics into partitions. are a systematic way of breaking the one topic log file into many logs, each of which can be hosted on a separate server. Partitions This gives us the ability in principle to scale topics out forever, although practical second-order effects and the finite amount of matter and energy available in the known universe to perform computation do place some upper bounds on scalability. Kafka brokers Kafka is distributed data infrastructure, which implies that there is some kind of node that can be duplicated across a network such that the collection of all of those nodes functions together as a single Kafka cluster. That node is called a . broker A broker can run on bare metal hardware, a cloud instance, in a container managed by Kubernetes, in Docker on your laptop, or wherever JVM processes can run. Kafka brokers are intentionally kept very simple, maintaining as little state as possible. They are responsible for writing new events to partitions, serving reads on existing partitions, and replicating partitions among themselves. They don’t do any computation over messages or routing of messages between topics. Replication As a responsible data infrastructure component, Kafka provides replicated storage of topic partitions. Each topic has a configurable replication factor that determines how many of these copies will exist in the cluster in total. One of the replicas is elected to be the leader, and it is to this replica that all writes are produced and from which all reads are probably consumed. (There are some advanced features that allow some reads to be done on non-leader partitions, but let’s not worry about those here on day five.) The other replicas are called , and it is their job to stay up to date with the leader and be eligible for election as the new leader if the broker hosting the current leader goes down. followers Kafka producers Once the Kafka cluster is up and running with its minimal feature set, we need to be able to talk to it from the outside world. A is an external application that writes messages to a Kafka cluster, communicating with the cluster using Kafka’s network protocol. is a publicly documented thing, but it would be an extraordinarily bad idea to write your own interface library when so many excellent ones are available. producer That network protocol Out of the box, Apache Kafka provides a Java library, and Confluent supports libraries in Python, C/C++, .NET languages, and Go. The producer library manages all of the non-trivial network plumbing between your client program and the cluster and also makes decisions like how to assign new messages to topic partitions. The producer library is surprisingly complex in its internals, but the API surface area for the basic task of writing a message to a topic is very simple indeed. Kafka consumers The is an external application that reads messages from Kafka topics and does some work with them, like filtering, aggregating, or enriching them with other information sources. Like the producer, it relies on a client library to handle the low-level network interface in addition to some other pretty sophisticated functionality. A consumer can be just a single instance, or it can be many instances of the same program: a consumer group. consumer Consumer groups are elastically scalable by default, but the library only manages some of the challenges associated with scale-out and fault tolerance. For example, if your consumer is stateful (and it probably is), then you’ll have some extra work to do to manage that state during failover or scaling operations. The Kafka ecosystem Let’s pause on this day and set the stage for the rest of the series. With basic pub/sub, partitioning, producing, and consuming work in hand, other needs are going to arise. These things consistently emerge from organizations making serious use of Kafka. You need data integration, schema management, and options for stream processing. The Kafka community and have solved these problems in standard ways and are likely to continue solving new common problems as they arise. Confluent community Kafka Connect is a system for connecting non-Kafka systems to Kafka in a declarative way, without requiring you to write a bunch of non-differentiated integration code to connect to the same exact systems that the rest of the world is connecting to. Kafka Connect Connect runs as a scalable, fault-tolerant cluster of machines external to the Kafka cluster. Rather than write bespoke code to read data from a database or write messages to Elasticsearch, you deploy pre-built connectors from the extensive connector ecosystem, and configure them with a little bit of JSON. Connect then reads data from source systems and writes it to sink systems automatically. Schema Registry Schema change is a constant fact of life. Any time you serialize data, put it somewhere, and hope to go get it from that place later on, changes in the format of the data are a perennial challenge. We feel this problem most acutely in database schemas, but message formats in Kafka are no exception. The exists to help manage schema change over time. When you release a new producer or a new consumer application with a modified message format, the Schema Registry will help the client application determine whether the new schema is compatible with the old one, given the expectations of other clients that have yet to be versioned. It’s an indispensable tool for a complex deployment. Confluent Schema Registry Kafka Streams Producing messages to Kafka is often fairly simple: Messages come from some source, either read from some input or computed from some prior state, and they go into a topic. But reading gets complicated very quickly, and the consumer API really doesn’t offer much more abstraction than the producer. The exists to provide this layer of abstraction on top of the vanilla consumer. It’s a Java API that provides a functional view of the typical stream processing primitives that emerge in complex consumers: filtering, grouping, aggregating, joining, and more. It provides an abstraction not just for streams, but for streams turned into tables, and a mechanism for querying those tables as well. It builds on the consumer library’s native horizontal scalability and fault tolerance, while addressing the consumer’s limited support for state management. Kafka Streams API ksqlDB Writing stream processing applications in Java is a nice thing to do if you’re using Kafka Streams, and if you’re using Java, and if it makes sense to marry stream processing functionality with the application itself. But what if you didn’t want to do those things? Or what if you wanted a simpler approach in which you just used SQL to get your stream processing done? This is precisely what is: an application-oriented stream processing database for Kafka. A small cluster of ksqlDB nodes runs continuous stream processing queries written in SQL, constantly consuming input events and producing results back into Kafka. It exposes the same stream and table abstractions as Kafka Streams and makes tables queryable through a lightweight JSON API. ksqlDB Coda I hope this video series has helped you see the breadth of the Kafka ecosystem. I want you to have a basic mental model of how Kafka works and what other components have grown up around it to help you solve the kinds of problems that will inevitably present themselves as you build event-driven systems. If you’ve got some time in the next couple of weeks, and if a bit of study helps you relax as it does for me, be sure to . check them out Tim Berglund is a teacher, author, and technology leader with Confluent, where he serves as the senior director of developer advocacy. He can frequently be found at speaking at conferences in the U.S. and all over the world. He is the co-presenter of various O’Reilly training videos on topics ranging from Git to distributed systems, and is the author of . He lives in Littleton, CO, U.S., with the wife of his youth, their three children having grown up. Gradle Beyond the Basics

Apache Kafka: How Does It Work?

SECOND ORDER EFFECTS

soeffects.com

-- employees

Since n.d.

COMPANY RANKING

SECOND ORDER EFFECTS

EVERGREEN INDEX #9548

Representing this company? Click here to claim and customize this page!

Second Order Effects WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

SECOND ORDER EFFECTS

soeffects.com

-- employees

Since n.d.

COMPANY RANKING

SECOND ORDER EFFECTS

EVERGREEN INDEX #9548

RELATED COMPANIESRANK

STORIES ABOUT SECOND ORDER EFFECTS

Representing this company? Click here to claim and customize this page!

Second Order Effects WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!