Primer enables you to deploy new user screens between releases, without writing code

Primer

primer

, connect with me on You can find me on Twitter @bhutanisanyam1 Linkedin here Subscribe to my Newsletter for a weekly curated reads of Deep Learning and Computer Vision Articles This is Part 0 of the Series. The series will be in a beginner friendly form. All the links to the GitHub repositories will be provided in the posts. The Posts will be in the form on a theory minima appraoch-where only just the bare amount of theory needed to get you started is required, with in depth discussion of the code. If you feel you need a quick refresh of , Python Checkout my Basic Tutorial Series. Feel Free to contribute and improve the code, I’ll update my posts and Credit you for the contributions as well! The topics that will be covered are (Links will be updated as the drafts become public): Basics (Numpy Bridge, getting started) Linear Regression Logistic Regression Neural Network Convolutional Neural Networks Recurrent Neural Networks LSTM Deep Dive Collaborative Filters Please drop a comment below if you’d like to read more tutorials and I will try my best to have them up and ready ASAP! If you want to chat, ping me in the comments or find me on twitter. I’d love to hear from you! If Self Driving Cars excite you, Here is my Path to Self Driving. If you want a Primer on Reinforcement Learning, click here for a quick read

PyTorch Primer Series #0

Designed for developers at the intersection of curiosity and professional growth, this primer aims to illuminate how asynchronous programming can elevate application performance, scalability, and responsiveness. By focusing on the universal principles of non-blocking operations and event loops, we venture beyond specific technologies like AsyncIO, Node.js, or Go. This exploration is crafted for software developers eager to grasp the efficiencies driving modern software without the complexity of technical jargon. Think of how a single thread handling requests is like a single-lane road in a busy city: each request is a car, and when there's only one lane, cars line up, causing delays. But what if our city — our service — could manage traffic more cleverly? That's where the magic of an asynchronous runtime comes into play. It's like adding an intelligent traffic system to our city, guiding cars through multiple lanes and intersections without waiting. This system keeps traffic flowing smoothly, ensuring no car (or task) waits too long, which is precisely how an asynchronous runtime keeps our software running efficiently. Let's consider an example of a service that completes I/O operations synchronously. For clarity, these operations are shown outside the main flow of execution in the diagram: Blocking I/O during your service's startup phase might be acceptable, but avoiding this approach when processing external requests is advisable. The following diagram illustrates how service efficiency can be enhanced through the adoption of non-blocking I/O operations: These examples highlight scenarios where the server's performance gains are maximized. Nevertheless, the advantage of non-blocking operations persists across any frequency of incoming requests. Even in less-than-ideal conditions, performance benefits from integrating dedicated I/O threads into the processing workflow. The total time required to process a request (from the client's initial request to the final response, as depicted by the blue number on the right) will invariably decrease, provided there are sufficient threads to handle all requests. In the worst-case scenario, this duration will not surpass that of synchronous processing methods. We then encounter a practice that might seem counterintuitive to many developers. When I/O operations constitute a significant portion of request processing time, optimizing other code segments may yield slight improvement. The duration to fetch data from a cache might align closely with the time dedicated to business logic and template rendering. Employing caching or an in-process database can reduce data retrieval times compared to other processing activities. To segment code effectively and facilitate callback execution, one can instruct the runtime to proceed to the next event loop cycle. Below is an illustrative example of how this concept is applied: // blocking callbacks
function func1_cb(str, cb) {
 var res = func1(str);
 cb(res);
}

function func2_cb(str, cb) {
 var res = func2(str);
 cb(res);
}

// non-blocking callbacks
function func1_cb(str, cb) {
 var res = func1(str);
 process.nextTick(function () {
   cb(res);
 });
}

function func2_cb(str, cb) {
 var res = func2(str);
 process.nextTick(function () {
   cb(res);
 });
}

// usage example
func1_cb(content, function (str) {
 func2_cb(str, function (result) {
   // work with result
 });
}); By adopting this methodology to split two parts of calculations, the overall processing time for scenarios with nearly concurrent request arrivals remains the same. However, the response to the first request is delayed: This scenario represents the least favorable outcome when employing the “next tick” strategy. As shown in the initial example, the “next tick” method proves benign for infrequent requests. Should requests arrive at a moderate pace, leveraging this technique enhances processing speed by allowing the initiation of new requests and the start of non-blocking operations to be interleaved during execution breaks. This approach effectively reduces both the total processing time and the average time per request: In conclusion, adopting non-blocking I/O is crucial for enhancing application performance and beneficial in environments with both sparse and heavy incoming request volumes. Additionally, effectively sequencing the execution flow — illustrated by concepts similar to the "next tick" technique — significantly improves server efficiency. Embracing these asynchronous programming practices offers clear advantages over traditional, synchronous methods.

Asynchronous Runtimes: A Primer for the Perplexed Developer

The ability to earn rather than buy your cryptocurrency has always been an important aspect of the crypto community. Unfortunately, for many who are new to the blockchain space, the days of being able to mine hundreds of with just your home computer are long gone. However, there are now more inclusive ways to earn cryptocurrency, and while the idea is still new, crypto-earning platforms could soon make central banks start to sweat. Bitcoin What is Crypto Earning? While most people are familiar with Bitcoin and Ethereum miners, for the high-tech, slightly mysterious subset of individuals working the blockchain for paydirt, using high-powered computers stuffed with GPUs, that's not the only way to earn your coins. Investors looking to earn coins or tokens without spending $20,000 on mining equipment can still participate in community initiatives by way of Proof of Stake. POS is an alternate consensus method used by certain blockchains to secure the network and validate transactions. While the first Proof of Stake coin, PeerCoin, was introduced as early as 2012, the rise of DeFi and Ethereum's announcement that it would be moving to Proof of Stake has revitalized this once controversial consensus method and sent it into the mainstream. Now, the only thing stopping you from earning hefty returns on your coins is knowing where exactly you should put your money. Your return will depend on the level of risk you're willing to stomach, the lock-up terms you choose for your coins or tokens, and whether you plan to go with a centralized or decentralized earning platform. Centralized vs. Decentralized platforms Like all things in cryptocurrency, or even just investing in general, there is always a risk. The world of cryptocurrency is still very much the wild west, mostly lacking regulation and consumer protection, and putting your trust in the wrong hands can often lead to disaster. Investors who previously lost their Bitcoin fortunes to shady platforms like Mt. Gox will be quick to warn you: "Not your keys, not your coins," a popular saying in the crypto community. This saying stems from the fact that when you are using a centralized platform, you do not control the private keys to your wallets. That means that, unlike your private wallet, which can't be taken from you, funds kept on platforms owned by someone else are always at risk to some degree of being lost to insolvency, have a higher risk of theft, or can even be seized should your country's flavor of cryptocurrency wane in some way. Part of the appeal of Decentralized Finance (DeFi) platforms is that you are in control of your private keys and recovery phrases, allowing you to participate without giving up wallet ownership. So why would you ever use a centralized platform, then? Well, in its current iteration, is not as user-friendly as it could be. There are a lot of ways to make a mistake, and many people simply are not interested in or prepared to be a custodian of their own finances. DeFi This is part of the reason why central banking and credit cards have become a consumer staple; they provide protection to the user. With a non-custodial wallet, this protection is gone. You're responsible for your own private keys and recovery phrase, and if you lose those, then you're certainly out of luck, but you won't be alone. It's estimated that between 2.78 million and 3.79 million Bitcoin have been lost in this manner. While if you use the custodial wallet, support can come to the rescue in most cases if you lose access to your account or if you were to perhaps send your coins to the wrong wallet, which is an easier mistake to make than you might think. Especially with cross-chain confusion now coming into play. Earning on custodial vs. Non-Custodial Platforms Custodial Platform Earning A custodial platform, such as Nexo or Celsius, simply pays out an advertised interest rate on held cryptocurrencies, just like your bank savings account. Earnings typically go up if you are willing to commit to a longer lock-up term, which is great if you are long on an investment. You will, of course, not earn as much as you would using DeFi, but your risk is mitigated as long as you trust the platform. This is because these platforms are taking on much of the risk of blockchain finance for you. In return, they take a cut of the profits for themselves. Interest is paid out daily or weekly, typically in the same currency that you are staking, making it a good way to grow your holdings while you wait. It's the simplest way to earn if you are not technologically inclined or are risk-averse. These platforms make money by doing their own staking or lending, at varying degrees of risk, and typically also provide other services that allow them to profit off of trade or transaction fees from the users of the platform. What are the Risks of Custodial Earning Platforms? Custodial platforms are not without risk. Besides the fact that you are typically not in control of your private keys, management of the exchange can also be an issue. While the platform's job is to absorb risk, if they get into trouble, things could go south quickly, and your investments could be at risk if the company becomes insolvent. Custodial platforms are not immune to bug exploits or the will of overbearing governments. Hackers exploiting smart contract vulnerabilities or damaging new regulations could put your holdings at risk. . This loss was not passed on to the consumer, but should a loss be large enough to put a platform out of business, it could be. Even FDIC-insured exchanges only offer protection to the fiat currencies in your account. Celsius is estimated to have lost $50 million in the Badger DAO exploit Non-Custodial Platform (DeFi) Earning While DeFi staking is much more complicated and more risks are involved, the potential for profit is also greater if you are willing to take those risks. There's also a greater variety of ways to earn going the DeFi route when compared to a centralized earning platform. You can participate in yield farming, which is essentially becoming a liquidity provider to a platform that provides lending services to traders. Or, you could stake your assets and become a validator for transactions, helping to secure the network, similar to what miners do in Proof of Work setups. You can also participate in liquidity mining, a process that rewards you with native tokens or governance rights rather than a standard APY yield like yield farming does. Each of these methods of earning has its own benefits and dangers, so it's imperative that you thoroughly research each method before deciding on one. There's a very real risk of losing your initial deposit if you aren't careful. What are the Risks of Non-Custodial Earning Platforms? In addition to worrying about potential vulnerabilities in the smart contracts you are utilizing and liquidation, higher-yield currencies are often highly volatile, and shady platform providers can quickly throw a wrench in your plan, like when Sushi Swap's developer decided to sell all of his tokens, , and leaving investors high and dry. plummeting the price of the currency by 50% It's important to realize that pretty much anybody can create a token or even an entire platform for something like yield farming overnight, often anonymously, and in a faraway country, outside the reach of your country's legal jurisdiction. This is why it's important to make sure to research not just the type of Defi activity you wish to pursue carefully but also to be sure to vet the platform you intend to do it on properly. Spreading your investments across multiple platforms can also help to avoid a total loss of assets. In short, both custodial and non-custodial platforms have their own unique benefits and dangers. It's important to know what you're getting into before putting your life savings into a risky situation in which there could be no way out if things go badly.

Better Than Your Bank: A Primer on the Benefits and Risks of Crypto Earning

Elastic Block Store (EBS for short) is a block-level storage service for . Essentially, it's a virtual SSD or HDD that you attach to EC2 instances, so they can have persistent storage. Honestly, EBS is pretty boring to talk about, but if you're storing a ton of data, knowing the fine details can save you a lot of money. EC2 instances Let's start with the basics. EBS Basic Concepts What is an EBS Volume? An EBS volume is a virtual block-level storage device that can be used by EC2 instances to store persistent data. An EBS volume acts like an HDD or SSD, but behind the scenes, they're actually an array of physical discs in a RAID configuration, in the same data center but physically distanced from each other to minimize the probability of simultaneous failures (e.g. due to fires). When you create an EBS volume you can define the size, the performance (only for some volume types), and the volume type, which is explained in the next section. Size and performance can be changed later, but not volume type. EBS volumes exist separately from EC2 instances and can be detached to one instance and attached to another one. When you create an EC2 instance, an EBS volume is created and attached as a root volume, and the default behavior is to delete it when the instance is terminated (this can be changed). But it's important to understand that they are actually a separate service from EC2. Types of EBS Volumes As is often the case in , there are different types of resources to serve different use cases and needs. These are the types of volumes you can create in EBS. Remember that you set this on creation and cannot change it later. AWS EBS GP3 Volumes This is the general-purpose volume type, which you should use for most stuff, and default to when in doubt. Size and performance (IOPS) can be configured separately (unlike the previous generation, GP2). Here are some details: Volume Size: 1 GB to 16 TB Durability: 99.8% to 99.9% Max IOPS/Volume: 16,000 (operations of 16K) Max Throughput/Volume: 1000 MB/s Latency: single-digit milliseconds Price: $0.08/GB-month over 3,000 (the first 3,000 are free) $0.005/provisioned IOPS-month EBS IO2 Volumes GP3 is great for most use cases, but if you need more performance, you go with IO2. If you're running a database on an EC2 instance, this is the one to pick. Size and performance (IOPS) can be configured separately, and the limits are higher than for GP3. Details: Volume Size: 4 GB to 16 TB Durability: 99.999% Max IOPS/Volume: 64,000 (operations of 16K). 256,000 IOPS with io2 Block Express. Max Throughput/Volume: 1,000 MB/s. 4,000 MB/s with io2 Block Express. Latency: single-digit millisecond Price: $0.125/GB-month $0.065/provisioned IOPS-month up to 32,000 IOPS (no free IOPS) $0.046/provisioned IOPS-month from 32,001 to 64,000 IOPS EBS ST1 Volumes This one is actually an HDD (virtual, but backed by real HDDs). Spinning disks! The use case is sequential access to data that sits contiguously in the physical disks. In general, that means data that is written as a long stream, and then read as a long stream, instead of having different parts accessed at random. It offers pretty good performance for that compared to GP3, at almost half the price. Here are the specs: Volume Size: 125 GB to 16 TB Durability: 99.8% to 99.9% durability Max IOPS/Volume: 500 (operations of 1 MB, not 16K) Max Throughput/Volume: 500 MB/s Price: $0.045/GB-month Maximum performance varies per size, at 40 MB/s per TB. Additionally, ST1 Volumes use a burst credits system to accumulate credits while usage is below peak throughput and consume them to achieve for a period of time a higher throughput than the soft limit. In short, a 12.5 TB volume always performs at max 500 MB/s, and any volume below that has a lower soft limit but can reach 500 MB/s for a short period of time. This is better explained . here Certification Exam tip: ST1 volumes can't be root volumes. EBS SC1 Volumes SC1 volumes are also HDDs, aimed at offering the lowest price per GB of all block storage options. They're recommended for infrequently accessed data that needs to be accessed from block storage (the C in SC1 stand for Cold). If you don't strictly need block storage access, S3 Infrequent Access or Glacier are also viable options. Volume Size: 125 GB to 16 TB Durability: 99.8% to 99.9% durability ← Much less than S3! Max IOPS/Volume: 250 (operations of 1 MB, not 16K) Max Throughput/Volume: 250 MB/s ← Slightly higher than S3 Infrequent Access, but with EBS SC1 you're not billed for reads and writes Price: $0.015/GB-month SC1 volumes use the same burst credit system as ST1, though their performance is lower. Certification Exam tip: SC1 volumes can't be root volumes. Older Volume Types IO1 (predecessor of IO2) and GP2 (predecessor of GP3) were the norm a few years ago, and you'll most likely find some still in production. IO1 works just like IO2, but with less durability, and a higher price at high IOPS. GP2 has the same use cases than GP3, but performance was tied to volume size and it used a burst credits system, like ST1. It's also 25% more expensive than GP3. Cost-savings tip: Migrate GP2 volumes to GP3. Characteristics of EBS Volumes EC2 Root Volume An EC2 instance comes with an EBS volume associated with it, called the root volume. This volume contains the OS, some libraries and programs, and some configurations. This is where the EC2 instance boots from when starting up. Multiple EBS Volumes You can attach up to 128 EBS volumes to an instance (depending on instance type), so long as they're in the same Availability Zone as the EC2 instance. Once a volume is attached, you'll see it in the OS just like if you were physically attaching a disk, and you can mount it on the OS or file system. You can detach an EBS volume from an instance and attach it to another instance as many times as you want, so long as both instances are in the same Availability Zone as the EC2 instance. Volumes of any type other than IO1 or IO2 can only be attached to once instance at a time. IO1 and IO2 volumes can be attached to multiple instances at the same time, in read-write mode. Availability of EBS Volumes EBS volumes are zonal resources. They exist in a single Availability Zone, so they are . This is also true for IO2 volumes, which offer durability of 99.999%. not highly available EBS volumes are redundant within that Availability Zone, so data loss is significantly less likely than with a single disk. They're backed by an array of physical disks in a RAID configuration. Lifecycle of an EBS Volume It's important to understand that the lifecycle of an EBS volume is separate from that of the EC2 instance. You can create them, attach them, detach them and delete them on their own. You can also configure them to be deleted when the EC2 instance they're attached to is terminated, which is the default for the root volume. Encryption of EBS Volumes EBS volumes can be , in a way that's entirely transparent to you. You don't need to manage or use any encryption keys, the EBS service automatically fetches them and decrypts the data when you initiate a read operation. encrypted using KMS You can enable Encryption by Default on your AWS account, which means every new EBS volume you create will be encrypted unless you explicitly configure it not to. This is a highly recommended best practice for AWS security. Backing up EBS Volumes: EBS Snapshots EBS snapshots are point-in-time copies of EBS volumes, used to back up and restore data. They're incremental, which means they only capture the data that has changed since the last snapshot. This makes EBS snapshots more efficient and cost-effective than full-volume backups. The size of an EBS snapshot is calculated based on the amount of data stored in the volume at the time the snapshot was taken. Snapshots are regional resources, meaning you can use them to hold a copy of an EBS volume and, if the volume's Availability Zone fails, restore it from the snapshot in a different Availability Zone. They can also be shared across regions and AWS accounts. Here's where you can read more about automating EBS snapshots for Disaster Recovery. EBS Best Practices Default to EBS GP3 Volumes You should default to GP3 volumes, and only use the other volume types if you have a specific use case, or you know you need more performance. Here's a . guide to benchmark EBS volumes Use EBS-optimized instances for higher performance EC2 Instance families have a limit on performance with EBS volumes, which is independent of the EBS volume itself. If you need high performance, it may not be enough to just use a better EBS volume such as IO2. You'll also need to look into whether your EC2 instances support that level of performance, and possibly use EC2 instances that are Storage Optimized. EBS performance is also limited by instance size. You can use the and metrics in CloudWatch to help you determine whether your instances are sized correctly. Instances with a consistently low balance percentage should be increased in size, and instances where the balance percentage never drops below 100% should be reduced in size. EBSIOBalance% EBSByteBalance% Use EC2 Instance Store for extreme performance If you need extreme performance, you'll need to use the EC2 Instance Store. It's ephemeral (that means non-permanent) block storage with a much higher performance than EBS. The main disadvantages are the pricing (you need an EC2 instance of a special family, which isn't cheap) and the fact that data is lost if the instance is stopped or terminated. Encrypt your EBS Volumes This comes at no cost and no performance hit to you, so it should be a no-brainer. First, you should enable Encryption by Defaul , so all future EBS volumes are created with encryption. Then you should encrypt existing EBS volumes by creating a snapshot of them, encrypting that snapshot and creating a new volume from the encrypted snapshot. t Migrate GP2 volumes to GP3 GP3 volumes can do anything that GP2 volumes can, and they're 20% cheaper. Here's a guide to . migrate your existing GP2 volumes to GP3 Back up important EBS volumes Remember that EBS volumes are zonal resources, meaning that if an Availability Zone goes offline you won't be able to access them. Furthermore, most EBS volume types offer 99.8% durability, making data loss or corruption entirely possible. To guard against that, create Snapshots of your EBS volumes. Snapshots are regional, so you're good if an AZ goes down. But if you want to do cross-region disaster recovery, you'll need to export the snapshots to another AWS region. If you're exporting encrypted snapshots, use a multi-region KMS key. You can . use Data Lifecycle Manager to automate creation and export of EBS snapshots Accessing a block for the first time on an EBS volume created from a snapshot has huge latency because the data is lazy loaded from S3. To avoid this, you can initialize (pre-warm) the volume before putting the volume in production, by accessing each block once. You can do this on Linux by attaching the volume to an EC2 instance, installing the utility, and running the following program (example for a volume called ) fio xvdf sudo fio --filename=/dev/xvdf --rw=read --bs=1M --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize Another option is to enable EBS fast snapshot restore on the snapshot. This will ensure AWS does the initialization for you, and is much faster. However, it costs per snapshot (regardless of size), so I prefer the manual option. $540/month Also published . here

Navigating Amazon EBS: A Primer on Basics and Best Practices

Are you struggling to solve complex problems? Do you feel like you're stuck in a never-ending cycle of solutions that just create more problems? If so, you may be in need of systems thinking!
In this article, we'll explore how systems thinking can help you become a better generalist and improve your critical thinking, creativity, collaboration, and communication skills.
Ready to learn more? Let's dive in! Systems Thinking 101: Key Concepts and Principles Systems thinking is a way of looking at the big picture and seeing how everything is connected. It’s a framework, a methodology, and a worldview all at once, depending on how acquainted with it we become. Since we’re just getting started, it’s best to look at it as a way to avoid getting lost in the details and missing the forest for the trees. I want to come out and say it’s not a quick cool hack that can be applied without thinking. We need to get familiar with the different concepts and tools before being somewhat comfortable putting them into practice, and that’s what the next sections are for! What’s a System? A system is anything that has parts, connections, and a goal. The parts can be anything you can think of: people, plants, machines, ideas…etc. The connections are how the parts affect each other: physically, mentally, socially…etc. The goal is the ultimate reason the system is working towards: it could be love, money, happiness, or world domination. Emergence The tricky thing about systems is what’s known as ‘emergence’. When the parts interact, they create something new: a whole that has its own properties and behaviors. You can't understand or predict these by looking at the parts alone. You have to look at the whole system. If you’ve ever heard of the expression ‘more than the sum of its parts’, you know where it comes from now. Examples of systems: A family: people who love and support each other (parts), communicate and cooperate (connections), and form a social unit (goal). A car: mechanical parts that move and work together (parts), transfer energy and force (connections) and transport people and goods (goal). A forest: living organisms that grow and feed on each other (parts), exchange nutrients and oxygen (connections) and form an ecosystem (goal). Feedback Loops: The Good, the Bad, and the Ugly Feedback loops make systems work. They are the chains of action/reaction that shape the behavior and outcomes of systems. Feedback loops can be balancing or reinforcing. Balancing feedback loops keep things in check and prevent them from going out of control. Reinforcing feedback loops make things go faster and faster, either up or down. : Some examples Population growth: a reinforcing feedback loop where more babies mean more people, who make more babies, who make more people, and so on. Thermostat: a balancing feedback loop where a hot room turns on the AC, which cools down the room, which turns off the AC, and keeps the room comfy. Leverage Points: Big Changes with Small Moves Leverage points are the sweet spots in a system where a little tweak can make a big difference. They can be numbers (how much or how many), structures (how things are arranged or connected), goals (what things are trying to achieve), or rules (how things are allowed or required). Leverage points are not always easy to spot; they often hide behind obvious or superficial problems or solutions and require digging deeper into the underlying causes or structures. : Some examples Interest rates: leverage point that changes how much money people borrow and spend, which affects how fast or slow the economy grows. Recycling: a structure leverage point that changes how materials move from a one-way to a two-way street, which reduces how much trash and pollution we create. Education: a goal leverage point that changes what people want and can do, which affects how well or poorly they do in life. Mental Models: How We See and Make Sense of Systems Mental models are the lenses that we use to look at ourselves, others, or the world. Mental models shape how we perceive and understand systems. The value of mental models depends mainly on how close they represent reality, how aware we are of them, and our ability to spot them. : Some examples of mental models are The Pareto Principle: 80% of the effects come from 20% of the causes. This mental model helps you focus on the most important and impactful aspects of any situation and eliminate the trivial ones. The Circle of Competence: This mental model helps you avoid overconfidence and recognize your limitations and blind spots. By identifying weaknesses, we’re in a much better position to act on them. The Confirmation Bias: We tend to seek and interpret information that confirms our existing beliefs and ignore or reject information that contradicts them. Being aware of our own biases and perspectives is a critical first step before expanding or changing them. System Archetypes: The Usual Suspects System archetypes are the common patterns that pop up in different contexts. They’re useful for spotting and solving problems that come from similar sources, and they’re valuable time-savers because we can reuse solutions that worked before! Limits to growth: How fast growth hits a hard limit, resulting in an S-curve. For example, a company's sales grow until they max out their market or customers.
Shifting the burden: Quick fixes hurt or delay a lasting fix, resulting in a dependence on the quick fix. For example, taking painkillers eases a headache but stops fixing the cause.
Tragedy of the commons: a pattern that shows how a shared resource gets used up by selfish users, resulting in a loss or damage of the resource. For example, fishermen catch more fish than the safe level, leading to no fish left. The definition of ‘this is why we can’t have nice things’. Some examples of system archetypes are: The Iceberg Model The iceberg model in systems thinking is a way of getting to the bottom of a problem - not just dealing with what you can see but digging deeper to find out what’s really going on. The goal of the Iceberg model is to go beyond the surface-level causes and symptoms. It’s a style of root cause analysis if you’re more familiar with that framework. The main difference is the gradual approach. Let’s see the following example: is divided into 4 levels, the deeper we go the closer we are to the truth behind the apparent behavior: The Iceberg model This is the symptom, and in most cases, solutions are targeted at this level. If we use the example from above, the event is a person overeating. The Event Level This is when you start seeing repetitions. Overeating is not a single event, we can see how this happens continuously. It’s a trend. The Pattern Level This is usually the external reason causing/facilitating the patterns observed in the previous level. Usually, this is not a straight answer. In our example, it can be the following:
Physical: Easy access to bad food. Or, there aren’t health stores around your neighborhood.
Environment: Stressful work; Your home is not prepared for exercise; Your closest friends and family members love to have a rich dinner….etc.
Rituals: Deeply seated habits. Such as every time you’re bored, you eat. The Structure Level The beliefs, principles, and personal values that drive the behavior. The Mental Model Level In our example, a person might be overeating because they grew up in an environment where food was a calming factor, and became an emotional crutch they resort to in similar stressful situations, leading to overeating and all the negative aspects that come with it. This simple example can also be leveraged to apply the feedback loops we introduced earlier: The more they’re stressed, the more they eat, and the more weight they gain, which causes even more stress, which leads to even more overeating…etc. This is a prime example of a (bad) reinforcing feedback loop. Hopefully this -convenient- example helped clarified how all these concepts can be combined to think through problems and come up with meaningful solutions. Systems Thinking in Action: Generalists Rock It So that was a pretty hefty section on mostly conceptual topics, good job on getting through in one piece. In the upcoming articles, we’ll go through how all of this can be put into practice by generalists as a way towards finding true purpose. In the meantime, I wanted to share a couple of examples of people I find inspiring, that apply system thinking (and a bunch of other approaches) in their works, even if not explicitly: Angela Duckworth: The Grit Guru Angela Duckworth is a famous psychologist and educator who studies and promotes grit and character. She is the founder and CEO of Character Lab, a non-profit that helps schools teach character skills, and the author of the hit book ‘Grit: The Power of Passion and Perseverance’. She wants to know why some people make it while others don't, and she also helps students, teachers, parents, and organizations learn the skills and mindsets that lead to success and happiness. She uses systems thinking to understand and address this problem and opportunity by using some of these systems thinking ideas: People and environments drive motivation, persistence, and success. She discovers that grit (sticking to a long-term goal and working hard for it) is the secret sauce of achievement. She cooks up evidence-based ways to boost grit and character in students, teachers, parents, and organizations. She uses feedback loops and data to spice up these ways. For example, she develops online courses, videos, podcasts, and books on grit and character. She also builds character report cards, growth mindset lessons, and character coaching programs for schools and districts. Some of her achievements and impacts are: She has won prestigious awards and honors for her research, such as the MacArthur Fellowship, the Grawemeyer Award in Education, and the Christopher H. Browne Distinguished Professorship. She has helped millions of students, teachers, parents, and organizations around the world do better and feel better. She has reached a large audience through her online courses, videos, podcasts, books, speeches, and media appearances that have taught and motivated people on how to grow grit and character. Leyla Acaroglu: The Systems Change Maker She’s an educator and designer who runs Disrupt Design, a creative agency that helps organizations go green. She also runs The UnSchool, a learning platform that teaches people how to think and design differently. She faced the challenge of changing the way we think and act to solve the messy problems of our world. She also pursued the goal of turning people into change-makers and systems thinkers. She attempts to do that in the following way: Looking at how everything is connected and affects each other. Busting the myths and beliefs that make us do stupid and harmful things. Showing us the hidden costs and consequences of our actions. Coming up with creative and holistic solutions that tackle the real causes of problems. Using design thinking as a way to change systems. Using games, stories, and jokes as ways to make people interested and informed about systems thinking. Looking at different points of view and scenarios by working with different kinds of people. Combining insights and findings from different fields. Tailoring her approaches and recommendations to different situations and cultures. Conclusion If you're a generalist who wants to make a difference in a complex world, it’s important to learn to see things in their entire complexity. That’s why systems thinking is so useful. It's a skill and mindset that helps you think and solve problems holistically. If you want to go deeper, check out these handy resources and references: The Systems Thinker: A website that has everything you need to know about systems thinking. https://thesystemsthinker.com/ The UnSchool: An experimental learning platform that offers online courses, workshops, and fellowship programs on systems thinking, design thinking, sustainability, and social innovation. The platform aims to equip people with the knowledge and skills to become change-makers and systems thinkers. https://www.unschools.co/ I hope you liked this article and found it helpful. Let me know what you think. Next time, we’ll see how to balance breadth and depth as a generalist. See ya! If you liked this article, please spread the word and let me know what you think! You can find me on or . Twitter LinkedIn For more content on generalists, subscribe to the ! Generalist Thinkbox newsletter This article is part of a series on the topic of generalists and how they can find their purpose. We’ll provide opinion pieces, but also practical guides as well as interviews with contemporary generalists who’ll share their journey with us. Stay tuned!

System Thinking 101: A Primer for Generalists

সিস্টেম থিঙ্কিং 101: জেনারেলিস্টদের জন্য একটি প্রাইমার

System Thinking 101: Eine Einführung für Generalisten

System Thinking 101: un manual para generalistas

Pensée systémique 101 : Une introduction pour les généralistes

सिस्टम थिंकिंग 101: जनरलिस्ट्स के लिए एक प्राइमर

システム思考 101: ゼネラリストのための入門書

시스템 사고 101: 일반주의자를 위한 입문서

System Thinking 101: Uma Cartilha para Generalistas

Sistem Düşüncesi 101: Genelciler için Bir Başlangıç

Tư duy hệ thống 101: Cẩm nang cho những người theo chủ nghĩa tổng quát

系统思考 101：通才入门

The Complete Asynchronous Programming Primer for Unity Development

ইউনিটি ডেভেলপমেন্টের জন্য সম্পূর্ণ অ্যাসিঙ্ক্রোনাস প্রোগ্রামিং প্রাইমার

Die vollständige Einführung in die asynchrone Programmierung für die Unity-Entwicklung

El manual completo de programación asíncrona para el desarrollo de Unity

L'introduction complète à la programmation asynchrone pour le développement Unity

एकता विकास के लिए पूर्ण अतुल्यकालिक प्रोग्रामिंग प्राइमर

Unity 開発のための完全な非同期プログラミング入門

Unity 개발을 위한 완벽한 비동기 프로그래밍 입문서

O Primer completo de programação assíncrona para desenvolvimento Unity

Unity Geliştirme için Eksiksiz Eşzamansız Programlama Başlangıç Kılavuzu

Công cụ lập trình không đồng bộ hoàn chỉnh để phát triển Unity

Unity 开发的完整异步编程入门

Each time a new business ecosystem forms, we have to ask a simple question: where's value created? And once we are able to classify the ecosystem based on where value is created we can ask: how's value captured? From the above, we understand the business models building on top of that ecosystem. Since the release of ChatGPT at the end of November, one thing is clear AI commercial viability is accelerating, giving us a glimpse into how the AI ecosystem is building up. Let me explain. The foundational layer That might be comprised of general-purpose engines like GPT-3, DALL-E, StableDiffusion, and so on. This layer might have the following key features: it will be built to provide more generalized solutions to any specific need. General Purpose: This layer might be mostly a B2B/Enterprise layer, on the one hand, powering up a plethora of businesses. Just like AWS in the 2010s, powered by the applications made of Web 2.0 (Netflix, Slack, Uber, and many others). The AI foundational layer (still based on centralized cloud infrastructures) might power up the next wave of consumer applications. That will be a commercial Cambrian explosion… : these general-purpose engines will be multi-modal. Multimodal Meaning they might be able to handle any sort of interaction, be it text-to-text, text-to-image, text-to-video, and vice-versa. Thus it might move in two directions. On the one hand, the UX might be primarily driven by natural language instructions. On the other hand, the built-in AI into the plethora of tools on the web will be able to read, classify and learn patterns from all formats available on the web. This two-way system might bring the next evolution of the foundational model, to become general-purpose engines able to do many things. the main interface for those general-purpose engines might be natural language. Natural Language Interface: Today, this is expressed in the form of a prompt (or a natural language instruction). Prompting though might remain a key feature of the foundational layer; it might instead disappear in the apps' layer, where those AI engines might primarily work as push-based discovery engines (the AI will serve what it thinks it's relevant to users). these engines might be able to adapt in real-time, with the ability to read patterns as we navigate the real world. Real-time: This - I argue - will be a key feature to enable these general-purpose interfaces to be integrated into ! augmented reality A middle layer That might be comprised of vertical engines (imagine here you find your AI Lawyer, Accountant, AI HR Assistant, or AI Marketer). This middle layer might be built on top of the foundational layer, combining other "middle layer" engines able to become great at very specific tasks. This middle layer might: thus, the first step in this direction might be an AI that might be able to replicate each of the relevant corporate functions. Replicate corporate functions: From accounting to HR, marketing and sales. This middle layer will enhance a company, making it possible to run departments that are a combination of humans and machines. here, differentiation might be built on top of data moats. Data moats: Meaning that by continuously fine-tuning foundational layer engines to be adapted to middle layer functions, these AI specialized will become relevant for specific tasks. these middle-layer players might also have the ability to add other engines on top of the existing foundational layers in the creation of specific data pipelines to train the models for specific tasks. AI engines: And the ability to have those models adapted to make them more and more relevant to the specialized functions. And the app's layer That might see the rise of a plethora of smaller and much more specialized applications built on top of the middle layer. These will evolve based on the following: here, scaling up the user base will be critical to building network effects. Network effects: Feedback loops: users' feedback loops might become critical to enforce network effects. What business models will we see? In my opinion, the Foundational Layer might be together with the new App Store and AWS! On the one hand, it will work as the underlying infrastructure to build new apps. On the other hand, it might be the marketplace where these apps are built! The Middle Layer might initially primarily work as an Enterprise Business Model. Thus, providing organizations with very customized solutions that will fit the company's goals. Companies might have those AI Engines on the paychecks, almost as if this is the new employer's force. The Apps' Layer might follow three main kinds of business models: Ad-based, Subscription-based, and Consumption-based. If we all build tools on top of ChatGPT or similar models, how can we actually build a competitive moat? In other words, how can we build a company on top of AI that has a long-term advantage and that can't get easily commoditized? That's a critical question to answer, and I've been thinking about it a lot! So let me answer a few points. In the three layers of AI, I explained what the AI business ecosystem might look like. Now, once you've understood that, let's see what - I argue - can create a competitive moat in AI. Remixing foundational models (GPT-3, DALL-E, Stable Diffusion, Midjourney, and so forth) Right now, there are still some arbitrage opportunities. Meaning foundational models (the general-purpose engines like GPT-3 or Stable Diffusion), are still good at handling specific modalities. For instance, GPT-3 is incredible with text-to-text, but it sucks with images. DALL-E is incredible with text-to-image, but it sucks when it comes to generating text that makes sense on top of these images. So, for instance, if you're building an AI product, you can re-mix these models too: Improve their ability to handle more modalities. So, for instance, you can combine the capabilities of GPT-3 and DALL-E to enable the tool to handle both text and images properly for the user. In addition, you can fine-tune your model based on more foundational models. For instance, imagine you're building an AI marketplace; rather than relying on just DALL-E or Stable Diffusion, or yet Midjourney, you can remix these three models to make the images much more interesting for the years. A third element is you can add, as we'll see, things on top of these foundational layers to make the final output much more polished. Of course, chances are that this market might become an oligopoly, where a few players might control most of it and thus capture a good chunk of the value, which creates for them an incredible moat. That's because, if you're OpenAI, you can handle a large generative model like GPT-3 forward. If you're a small startup, trying to do it from scratch might be much harder. And the more these foundational models evolve, the more the barriers to entry will be hard to break, thus generating a leap ahead for the foundational layer players like OpenAI and the rest. The way OpenAI and other foundational layer organizations might capture value might be in the form of open APIs, as it is today. Or those might really become a sort of App Store for AI applications, where they will be able to tax each of the AI tools developed on top of each ecosystem, thus capturing value from that, similar to what happens today with Apple's App Store. Data moats AI models right now might have become extremely good at many tasks. However, to make them relevant for companies, at an enterprise level, or for users, at specific applications, data becomes critical to enable the model to be customized based on the tech stack (for enterprise companies) and context (for users) they sit on. For instance, imagine the case of an organization that wants to leverage AI to deliver custom experiences to its users. In order to do that, it'll need to use its "first-party data" to be integrated within these models. For instance, let's say the company wants to build a very specialized chatbot for support. Of course, it can do that in many ways, Train it on the content the company has built over the years. Build a Q&A dataset based on the most frequent requests of users. Tackle with much more accurate questions tied to conversion based on CRM data. In short, the enterprise organization will use this first-party data to integrate it into an AI model to make it as relevant as possible. That is how that AI application becomes valuable. For that, it becomes critical: understand what data is really relevant for the AI to become way better at specific tasks. Data Integration: to understand how to clean the relevant first-party proprietary data which can be used to train the model. Data Curation: foundational models are very powerful. However, they have been trained to perform many tasks. You can fine-tune the foundational model (by feeding contextual data and by tweaking these models) to generate much better outputs. Fine-tuning becomes, therefore, critical to making sure you can build a valuable AI product on top of existing foundational layers. Fine-tuning: Middle-layer AI engines: another interesting element is the fact that as a middle-layer AI company, you can still build refined engines on top of existing foundational models. For instance, take the case of a company that builds an AI tool for resume generation. You can still add an AI engine on top of it, which does rephrasing, further grammar checking, plagiarism, and more, which will be a value-added layer on top of the foundational layer! That is how you can transform a standardized output from the foundational, general-purpose engine into something way more specific. Prompt hacking (the new coding paradigm hidden in the backend) For years neural nets had been stuck until they started to do incredible things. And the most interesting part? Most of these interesting things were the result of scaling these networks. In other words, once a new architecture has been employed (transformer-based architecture), the rest of the work has been achieved through scaling. Now, there is an unpredictable component to scaling. Just like, when you scale a company, after a certain threshold, you don't know how that company might change and what properties might emerge. When scaling neural networks based on the same architectures, various properties emerge. In biology, emergence (or how a complex system shows completely different behaviors from its parts, as the overall system depends upon the interactions between its parts) is extremely powerful. Indeed, even in a real world that often looks fractal (the smaller resemble the larger), in reality, the much larger shows properties that are completely different! This is one of the topics that fascinates me the most in business. And this is also what makes AI so interesting to me right now. By scaling AI systems, we get - unpredictable - emergent properties that, for better or for worse, might affect the evolution of AI. For instance, prompting, or the ability to change the output of AI models based on a natural language, has been an emergent property. None has coded it into the system; it just came out from scaling these AI models. And another exciting aspect - I argue - is that prompting does look more like coding than searching or querying. Indeed, those who compare prompting to search are getting it backward, in my opinion. Prompting is way more powerful, and over time it might become something hidden in the user interface rather than shown to final users. In this context, prompt hacking, or the process of tweaking the natural language instruction to have the AI model completely revamp its output, can be extremely powerful. That is why I'm adding prompt hacking within the key elements to build an AI moat. My main argument is that, in a codebase that becomes much more commoditized (today, ChatGPT can generate code and also fix bugs in the code), prompt hacking might be the core value of the software, as it will enable an AI model to slightly improve its output! Of course, for enterprise-level AI applications, prompting might still be part of the interface, as it gives the enterprise client a chance to customize the input highly. Yet, there will be a piece of prompting (prompt hacking) which might be hidden in the UI. While for consumer applications, prompt hacking might be hidden entirely in the interface, giving users standardized options to customize their outputs. The remaining part of the customization will happen based on the context and interests of the user. Network effects and fast iteration loops The Internet Business Playbook has taught us that the value of a web app might be built not only in its tech but in its ability to get better at scale. You might be using every day, apps like Netflix, Uber, Airbnb, LinkedIn, YouTube, TikTok, and so forth, which value stands in their ability to become better and better the more users join in. This is known as network effects. Would you still jump on YouTube if you had not such a vast library of content and a discovery engine that keeps recommending interesting and engaging stuff to you? So just like digital businesses can build their moats via network effects, AI companies can do the same. There is nothing new here, as companies like Meta, Google, Netflix, TikTok, and many others have been using human interactions combined with AI algorithms to improve their products at scale. For instance, in 2019, I argued that TikTok was so interesting, not because it was a new social media app. Quite the opposite. It was so interesting because it moved beyond the social network, employing AI algorithms to make users discover content beyond their connections! That is what made TikTok so sticky... Workflow How a company combines all the above to create fast iterative loops to develop, launch, iterate, maintain and grow AI applications will become the critical moat for the company! Each AI company that works at scale will have its own workflow that will work as a barrier to entry, as opposed to economies of scale. It will be the equivalent of the economies of scope (with a more effective workflow, AI companies will be able to build more and more features and bundle various products to create a more comprehensive experience). Thus making it harder and harder for other companies to replicate! Brand and Distribution Where technology might get commoditized, over time, branding creates a strong differentiation. This has been true for tech companies of the web era, and it'll be true for AI companies, which will be an amplified version of tech companies. In addition to that, just like distribution played a key role for early tech players (I covered at great lengths deals like or ), it will be so when it comes to AI players. Google-AOL Apple-AT&T Indeed, we've seen already how some key partnerships have developed already: OpenAI/Microsoft DeepMind/Google Stability AI/Apple Amazon AWS leveraging its own stack. And so on... The way those partnerships will form will not only be important from a technological standpoint. They will matter from a distribution standpoint. Indeed, the paradox of these AI models right now is they work extremely well as general-purpose engines. Yet, suppose we were to limit them by adding too many guardrails. In that case, they might well end up losing relevance also at specific tasks (for instance, limiting ChatGTP's ability to give answers on topics where it can be misleading and factually worn answers might actually hamper its capabilities!) Thus, this implies that with these AI companies, we might see a different kind of distribution model, where for those models to become great at specific tasks, they will need to be employed first at much broader tasks. This is a paradigm shift from a distribution standpoint. Where in the past, we've seen tech players start as niche, then scale from there (Amazon was an online bookstore, Facebook was a social network for Harvard undergraduates), we might see these AI companies go broad, right on, and then narrow down their spectrum of applications! For instance, right now, ChatGPT might be a general-purpose engine. Yet, over time, once they figure out which applications are well-suited, they might also be released for specific verticals. This sort of broad distribution requires strong partnerships with other large tech players, which can take that burden! Capital Deployment The whole new field might require substantial capital to scale, not to kick off, at a foundational level. It will make it much cheaper to build basic (initially) and more advanced (later on) applications. Thus lowering substantially the cost of doing business. Yet, building powerful foundational AI engines might require massive capital.

A Primer On The AI Economy

Suppose you want to get large amounts of information from a website as quickly as possible. How can this be done? In this article, we will talk about data scraping and how to scrape the web. Additionally, we'll get into what data scraping is, why you would want to do it, how data scrapers work, and lastly, we'll go over different processes for scraping the web. I'll also include a quick example to reference. What is data scraping? If you've ever copied and pasted content from a website into a different location, you are doing a very manual version of data scraping. In this article, we will be using software applications to do the data scraping for us. Data scraping is the process of using an application to extract valuable information from a website. This will allow us to obtain large amounts of data from websites in a short amount of time. Many of the larger websites like Google, Facebook, and GitHub have APIs that allow you to access their data. This is super convenient because the data will be given to you in a structured format that is easy to consume. Unfortunately, that isn't always the case. There will be other times when you will have to collect content and data for your specific use case. This is where web and data scraping applications come in handy. You can program these scraping applications to visit websites and extract the content/data that you want. The obvious benefit of this is being able to get the precise data that you want easily and efficiently. Data scraping is comprised of two parts, the crawler, and the scraper. The crawler is the algorithm that we can create to browse the web and find the exact data that we want. An example of this would be navigating to a specific website and clicking on the page where the content you want exists. Once you have found that data, we will utilize the scraper. The scraper is used to "scrape" the data from the website. With the scraper, you can detect the data points you want and export them to a format that would work best for you. Once the data has been exported the fun can begin. You are free to use that data however you need. Use cases for data scraping While reading this article you've probably wondered, "what are some good use cases for web/data scraping?" Let's go over a couple of these use cases. The first one that we will be talking about is my favorite, price monitoring. You can use price monitoring to keep track of prices and make sure you are finding the best deal. Who doesn't want to save money? I have written a previous about scraping to monitor the prices of specific products. Companies can also use price scraping to see what competitors are pricing similar products. This provides them with the advantage of being able to provide optimal pricing for their products so they can obtain maximum revenue. blog post Amazon.com Contact scraping is another way data scraping is used. Many companies and individuals can scrape the web for contact information to use for e-mail marketing. A good example of this would be scraping locations like an online employee directory or a bulk mailing list. This use case is very controversial and often requires permission to collect this kind of data. If you have ever visited a website and given them access to your contact information in exchange for using their software, you have permitted them to collect personal data like your e-mail address and phone number. The last use case we'll go over is news monitoring. Many individuals and companies can scrape news sites to stay current on stories and issues relevant to them. This could be especially useful if you are trying to create a feed of some type, or if you just need to keep up with day-to-day reports. How do data scrapers work? Next, let's go over how data scrapers work. Scrapers can take all the content on web pages or just the specific data that you want. In many situations, it is best to pinpoint the specific data you want so that the data scraper can quickly extract it. For example, in the that I mentioned earlier, we look at pricing for office chairs. In that instance, we are only trying to identify the price of the chairs and the title of the item. That allowed the data scraper to swiftly filter out any unneeded clutter resulting in the script being run relatively fast. Amazon web scraping blog post Now when the process of scraping a site first begins there needs to be a URL that is provided to the script or application software. Based on the provided URL the scraper will navigate to that web page. Next, it will load the HTML code of that site. After that code has been loaded, the scraper can then begin to collect the data that is wanted/needed. Lastly, the collected data is outputted in a predefined format determined by the user. Usually, this is a JSON file, but it can also be saved in other formats like an excel spreadsheet or a CSV file. Stepping through the scraping process Now that we know how a data scraper functions let's identify some preliminary steps that are needed before you try to scrape a website yourself. There are many cool tools and software applications out there that help with scraping websites. Because of this we will stay at a high level and focus on the basics. First, you want to find the URLs you want to scrape. This might seem obvious, but it is a key factor and how well your data scrape will work. If the URL you give the scraper is even slightly incorrect the data you get back will not be what you want, or even worse, your scraper won't work at all. For instance, if you're trying to do a you want to make sure that your URL goes to a relevant site like Amazon or Google shopping. price monitoring data scrape Secondly, you want to inspect the webpage (f12 on most keyboards) to identify what your scraper needs to scrape. If we use the same Amazon price monitoring example, you could go to a search result page on Amazon, inspect that page, and locate where the price is in the HTML code. Once you have that, you want to identify the unique tags that are around the price so you can use that in your data scraper. Some good tags would be div tags with IDs or very specific class names. <div id="price">$1.99</div> After finding the code you want to collect and use, you'll want to incorporate this into your data scrape. This could be writing a script with the IDs or class names that you found in the previous step or simply inputting the tags into scraping software. You'll also probably want to add supporting information to help when displaying your data. Sticking with our Amazon example, if you are collecting the price of office chairs on Amazon it would be nice to also have the title of the item to accompany the price. Once you've specified the tags in your script or scraping application, you'll want to execute the code. This is where all the magic happens. Everything that we talked about in the above section about how data scrapers work comes into play here. Hopefully, you now have the data you need to start building your application. Whether that be a dashboard of charts, a cool table, or a sweet content feed the data is yours to do with it what you like. Lots of times you may get data back that you don't expect. That is completely normal. Just like anything else in the engineering world, if one tiny thing is off it can often lead to things being incorrect. Don't get discouraged! Practice makes perfect and you will catch on. Data scraping with Browserless All the basics have been covered for scraping the web. Before we end, I want to mention a cool tool that allows you to do data scraping. is a headless Chrome browser as a service. You can use Browserless with libraries like or to automate web-based tasks like data collection. To learn more, make sure to visit the where you can find blog posts, documentation, debuggers, and other resources. Browserless puppeteer Selenium Browserless website Data Scraping Example with Browserless In this example, we are going to do a simple data scrape of the Y Combinator Hacker News feed. You can also run this example in the . For this, we will use two main tools, Puppeteer and Browserless. In the above paragraph, I mentioned these tools with corresponding links. I highly recommend you check them out before diving into the example. Browserless debugger tool Initial setup Alrighty, let's get to it! We are going to start with the initial setup. Luckily for us, there aren't many dependencies we need to install. There's only one... Puppeteer. npm i puppeteer Once you've run that command you are good to go! 👍 Browserless setup Let's look at how to set up Browserless real quick. For starters, you'll need to set up an account. Once you get your account set up, you'll be directed to your Browserless dashboard. Here you can find your session usage chart, prepaid balance, API key, and all of the other account goodies. Keep your API key handy as we'll use that once we start writing our script. Other than that we are ready to start coding! The code Ok now for the exciting part! Here is the code example: const puppeteer = require("puppeteer");

const scrape = async () => {
    const browser = await puppeteer.launch({ headless: false });
    // const browser = await puppeteer.connect({ browserWSEndpoint: 'wss://chrome.browserless.io?token=[ADD BROWSERLESS API TOKEN HERE]' })

    const page = await browser.newPage();
    await page.goto('https://news.ycombinator.com');

    // Here, we inject some JavaScript into the page to build a list of results
    const items = await page.evaluate(() => {
        const elements = [...document.querySelectorAll('.athing .titleline a')];
        const results = elements.map((link) => ({
            title: link.textContent,
            href: link.href,
        }));
        return JSON.stringify(results);
    });

    // Finally, we return an object, which triggers a JSON file download
    return JSON.parse(items);

};
scrape(); Lines 4 and 5 are very important in this example. Line 4 uses Puppeteer when running your script. This is good for testing so you can view how the browser is interacting with your script. Line 5 is the Browserless connection. This is where you can add your API key which will link up to your Browserless account and allow you to run your script with Browserless. Make sure one of these two lines is commented out. You only need one. Final product Alrighty, that is all you need for this example. Once things are installed and the code is implemented, you can open up your preferred command-line interface in your project and run . node [insert name of js file here] The output should be a JSON file with real-time titles and links from the Hacker News feed! I hoped this article on data scraping was intriguing and exciting. There are endless possibilities as to what you can accomplish with web and data scraping. I hope this sparks some fantastic projects for some of you. Happy coding! ❤️ Also Published Here

A Quick Primer on Data Scraping

Properly securing sensitive customer data is more important than ever. Consumers are increasingly insisting that their data be secured and managed properly. The regulatory environment is also becoming tougher, and business requirements are becoming increasingly complex. The burden is placed on the company and its development teams to meet these requirements while still delighting users. If that leaves you in a bind, we’re here to help with a quick data privacy primer! First, we’ll help you to understand the various kinds of sensitive customer data and the regulations that apply to it. Next, we’ll guide you in assessing your current handling of that data. Finally, we’ll provide direction on how to properly govern that data. Learn The first task is to understand what kind of sensitive customer data you are already handling and what regulations apply to it. Three factors determine which regulations apply to a given set of data: The data itself Your company’s industry vertical The location(s) where your business operates First, consider the data itself. Depending on the type of data that is being stored in your system, different regulations apply. Let’s begin with (PI). This is perhaps the broadest category of regulated data, referring to nearly anything that is or can be associated with a person. PI is regulated by the (GDPR), the (CRPA), , and others. Examples of this data include: Personal Information General Data Protection Regulation California Privacy Rights Act NY SHIELD IP addresses Geolocation information Internal ID numbers Ethnic or racial origin (PII) is a subcategory of PI and refers to any data which could be used to distinguish or otherwise determine a person’s identity. Generally speaking, the same regulations which apply to PI also apply to PII, although with differing levels of sensitivity. Examples of PII include: Personally Identifiable Information Names Phone numbers Driver’s license or other ID numbers Social Security Numbers (SSNs are more highly regulated than other PII) Biometric records Other categories of private consumer data tend to be industry-specific. For example, (PHI) refers to all “individually identifiable health information.” This is regulated in the US by , and is defined as any information which relates to any of the following: Protected Health Information HIPAA The individual’s past, present, or future physical or mental health or condition The provision of healthcare to the individual The past, present, or future payment for the provision of healthcare to the individual Similarly, (NPI) refers to the personally identifiable financial information that is provided by a consumer to a financial institution, and as such is specific to financial service organizations. In the US, NPI is regulated by the (GLBA). Nonpublic Personal Information Gramm-Leach Bliley Act Finally, some types of personal information are governed by location-specific regulations. (SPI) is defined by the . This refers to data that does not directly identify an individual but may cause harm if made public. Examples include ID numbers, geolocation, account login information, and genetic information. Sensitive Personal Information CPRA is defined by the and applies to any data related to a resident of New York. Additionally, nearly every privacy regulation has some kind of location scope (such as GDPR for the EU, HIPAA for the US, and so on). Private Information NY SHIELD Act This is not an exhaustive list of regulations or categories of sensitive customer data. With this in mind, however, we can see several trends. First, many of these categories overlap with one another. The same data can fall under the scope of multiple regulations and must be handled according to the requirements of all regulations that are relevant to your business. Second, the relevant regulations will heavily depend on your organization’s industry vertical and the locations in which it operates. Finally, even within a given regulation, different data can have differing levels of sensitivity. With an understanding of the various types of sensitive customer data in mind, we are now ready to begin an assessment process. This assessment will answer the question, “How is my sensitive customer data being handled now?” Assess Before determining how sensitive customer data be handled, it is helpful to take some time to assess how it handled. To begin, compile a list of customer data that is already being handled and stored. Next, you should carry out two assessments: a technical assessment and a policy assessment should is currently being A tells you where the data flows through the system. Begin with the moment that data enters the system, and trace it through the various pieces of your application. Make sure to consider the application, service, and data layers. Additionally, note any places where that data the system, and make note of where that data goes, such as another application or a business reporting tool. Do this for each piece of sensitive customer data that is currently in the system. As you continue to do this, make a note of any inconsistencies or mishandling to address later. technical assessment exits Example: Tracing the flow of user PII through a system With a , you review current policies and controls around data governance. Begin by compiling your organization’s internal documentation around sensitive data and reviewing the written policies and procedures. From there, consider whether these policies adequately cover the types and amount of sensitive data that your company currently handles. Another critical part of this assessment is determining whether these policies are being effectively executed, or if actual governance has diverged from the original intent. policy assessment With both assessments in hand, you should be able to confidently describe your organization’s current handling of sensitive customer data. Additionally, you will have identified areas for improvement or areas that require further research. Now, we are ready to define what be done about your sensitive customer data. should Govern To properly govern your sensitive customer data, we recommend a three-step process. First, determine the regulations which apply to you, and which subsets of data they apply to. Second, determine which policies need to apply to each subset of sensitive data. Lastly, based on the required policies, design or find a technical solution to implement and reinforce those policies. Proper technical governance of sensitive customer data protects sensitive data without making it unnecessarily difficult to use. Rather than simply storing it side-by-side with the rest of your data, consider isolating your sensitive data in a zero trust data privacy vault. This approach allows for your sensitive data to be properly governed with its own separate controls. Additionally, your data privacy vault can grow along with the scope of sensitive data your organization handles, allowing for scalability. If you’re looking to achieve compliance with data privacy regulations in days instead of weeks, one of your options for a dedicated data privacy vault is . Skyflow uses a zero trust approach to storing sensitive data of all types. The data is isolated, encrypted, and secure, yet still usable in your business workflows. Granular access policies are available for all data in the vault and are easily defined with the Skyflow Data Governance Engine. All actions are logged and auditable. Skyflow has solutions for FinTech, Healthcare, PII, and more, and it is compliant with privacy regulations worldwide. Skyflow Wrap-up We’ve gone through a three-step approach (Learn. Assess. Govern.) for handling sensitive customer data in your organization. First, we went through the various types of sensitive data and the regulations which apply to them. Next, we described how you can assess how your organization handles sensitive customer data. Finally, we sketched out a flow for properly governing your sensitive customer data. We also considered how an isolated data privacy vault such as Skyflow can help with that. The landscape of sensitive data governance is continually changing, as new regulations are created and existing ones are strengthened. As such, this process is not something that can just be done once; it must be continually updated as your organization grows and as regulations change.

Handling Sensitive Data: A Primer

My intention with this primer is to give you just enough information to inspire appreciation for Open Banking as an initiative that is changing our financial landscape. This primer will show you the immediate application of Open Banking in your everyday life, and also encourage you to think of other use cases for it. Key Terms Here are some key terms you’ll need to understand Open Banking: : The core systems of a bank which hold information about its customers, e.g. bank balance, personal information, and transaction history. It is this core infrastructure that Third-Party Providers interact with in order to provide open-banking products and services. Core Infrastructure Any external party outside the host bank which is accessing the bank’s core infrastructure in order to use the information accessed to provide open-banking products and services. Third-Party Provider: Software intermediaries that dictate how two pieces of software can communicate or interact with each other. In Open Banking, the API dictates how the Third-Party Provider’s software interacts with the bank’s core infrastructure API: What is Open Banking? Open Banking is an initiative which has led to the enablement of Third-Party Providers, with the prior consent of the customer/account holder, to interact with the customer’s financial information which is being held in the bank’s core infrastructure. These interactions enable the creation of new financial offerings and services, which enhance the financial experience of the account holders. Third-Party Providers are able to access the bank’s core infrastructure solely through the use of APIs which helps the banks limit the information these providers have access to. Up until a few years ago, banks’ core infrastructure was locked away from external access, so what brought about this shift towards a more collaborative, “open” system? The intention of Open Banking is to bring about more competition and innovation to the financial services industry, with the ultimate goal of enriching the financial experience of the consumers. “Open Banking was designed to increase innovation and competition in banking and payment services.[…] It introduced a secure environment that enables customers to consent to third parties accessing their payment account information or making payments on their behalf” Source: The UK Financial Conduct Authority’s “Open finance - feedback statement” How is this useful to me?: Source: https://www.moneydashboard.com/ The immediate benefits of Open Banking can be evidenced by its application to bank accounts. The proliferation of easy-to-set-up digital banks and financial apps means that these days people can easily have multiple financial accounts. Let’s say you’ve got 3 different accounts: your daily spending account, your secondary one (which could be used to receive and sort your salary), and your credit card. This could easily go up in number if you have other savings or investment accounts. Ideally, you would want to be able to manage them all without having to log onto their various apps. You want one app that would have access to these accounts and would be able to present this information to you at once. The benefit of this would be especially visible in areas of personal finance management e.g. budgeting. This compilation of your various financial information in one place is called and is a type of in Open Banking. A good example of this is Money Dashboard in the UK. Account Aggregation Account Information Service Account aggregating apps can also go a step further from just showing you information to enabling you to make payments from within the app. Imagine you wanted to move some of your salary from your salary account to your primary spending account, make a payment from your primary account to a friend and also send some money to your investment account. Instead of having to log onto the various apps to make these transfers, you would be able to carry them out in the account aggregator app. This capability falls under which is not limited to just transferring funds between bank accounts; but is applicable wherever a payment can be made. E.g. Businesses can utilize Open Banking APIs to enable their businesses to receive payments directly from a customer’s bank, eliminating the need to input card details. Payment Initiation Services, Additional Use cases Since there exists the opportunity for innovation wherever any financial activity is taking place, an exhaustive list of use cases for Open Banking does not exist, neither would it be helpful. The fact that Open Banking is predicated on the interaction between a third-party and the financial information being accessed means that the limit of its use cases would be the extent of innovation that the third-party can bring. That being said, an additional use case that is gaining popularity is VRPs are a way of using Open Banking APIs to make recurring payments straight from a bank account. VRPs replace the need to have card information on file, and you would be able to set rules regarding variables such as how often, how much, and for how long money can be taken from your account. Variable Recurring Payments (VRPs). On-boarding and KYC diligence is also made much faster and easier with Open Banking. For any onboarding procedures that would require some sort of financial verification (e.g. applying for a loan), Open Banking APIs would enable this information to be accessed directly. This is especially useful in situations where you would need to verify information such as financial income and proof of address. Imagine all the time saved that would have been spent inputting most of this information manually or waiting for a bank statement in order to verify an address. Conclusion While it may take a few years for us to experience the full extent of Open Banking’s development and innovation, we can already acknowledge that it will change the legacy systems that have lagged behind the new age we live in. Data and technology are playing a greater hand in improving the financial industry and we can see that this leads to new business models and products and services, creating innovative ways for consumers to interact with their finances.

'Open Banking' for Noobs: A Primer

Business Silos Will Be Destroyed Forever… Before the pandemic, the last organizational change within the business structure came about 15 years ago with the addition of a team that would oversee in-house digital marketing efforts. Along with this Digital Marketing team came a new challenge in how to pay and manage these new team members. At the time, businesses viewed the job of digital marketing as something more than a fad but more akin to being a “necessary evil” business activity. Thus, their Digital Marketing teams became a highly fractured, generalized cost-center for their organization - where they were not quite responsible for generating revenue but were also doing activities and services that were not quite able to be billed to their customers. Had businesses realized that in less than 20 years, sales would have been touched by the digital world, then this department would not have been seen as an expense but rather as something that was responsible for generating revenue, and thus would have been more closely aligned with the sales departmental structure. all This is just a single example of how the traditional way of aligning your organization under a departmental structure led to implementing policies that did would not hold up under the digital “New Normal.” Digital Changes, Real-World Implications Now, if we look at how the business will be structured in 2022 and beyond, let’s assume that the rapid digital adoption that was necessitated along with the 2020 pandemic led to new policies and ways to operate a business. The next big challenge will come from these internal changes being implemented in a real-world setting when the roles and connections between sales, marketing, product, and finance have become blurred in the digital world to such a degree that middle management cannot possibly manage their team members, their deliverables, their clients, and the overall direction of their departments period. Naturally, the only way to compensate for this loose departmental structure will be to flatten the organization. How Will This Look In A Real-World Setting? In a middle-management-free organization, Data Operations Specialists will take over the supervision of their teams. Once these specialists manage their various data sources, the company will need to group employees based on their team’s shared responsibilities. Think, one team is responsible for “ ,” and another is responsible for “ ” generating qualified leads for an organization aligning the product features to the demands of the market. The first team would probably be composed of new business salespeople, marketing leadership, SEO specialists, Content-based writers, and people tasked with tracking how marketing generates leads and ties the data together to show a connection between marketing and sales efforts. On the second team, you will have product managers, Account Management Leadership, the CS/CX Team Leadership, product marketers, copywriters, and even the software engineering team. Clearly, you can see how businesses in 2022 and beyond will need to evolve to become results-oriented, leaving the need for the traditional business silos to become unnecessary. Hence, “The Great Restructuring” will inevitably change how businesses are operated in 2022 and in the future.

The Great Restructuring of 2022: A Primer on Business Operations

Blockchain is a term utilized to represent distributed ledger technology. Blockchain is used to build a storage system for data in a distributed and immutable mode. So there are key features we need to mention. – this means that once data is written to a blockchain data storage or ledger, it cannot be changed – so it’s there forever. For instance, in a relational database, no matter how much security you have, the data can be accessed and modified on the system. Immutability A blockchain system guarantees that if one bit of data is altered, it says it is in an invalid state, and since the data is distributed on various systems, the verified data with a justified state can be retrieved. – is the key to trust. As long as the data you see is in the blockchain and it’s in a valid state, you can trust the data to be accurate. This trust is the key, and this trust is achieved in a blockchain system by replicating the datastore on several hosts on the internet. For instance, as long as the peers accept that bit of data is valid, so you can completely trust that data that is stored on that system. Distribution In blockchain the trust is requisite. – Trust Immutability – valid/invalid Distribution How can Trust Issues be Resolved Using Blockchain? For many industries, trust is a problem. The automobile industry is one of those where trust, or lack of it, causes problems. People order used cars from other countries which do not easily share tracking information, and this can mean that faking data about the car’s provenance and history is easier. For instance, odometer fraud is one of the problems that can seriously impact the value of a vehicle. Representing a vehicle as having been used 5,000 miles instead of 50,000 miles can mean a huge difference in perceived value and potentially usable life span of the car or truck. In many countries, car service records are largely kept on paper which can, of course, be very easy to be fake. So, let’s imagine that pushing all service providers to a blockchain and all the readings kept on the Blockchain. The integrity and immutability of the blockchain data mean that a modification of the records would be detectable. Moreover, agencies such as insurance companies can also contribute by appending accident details of a car, giving a much more complete picture of the vehicle’s history in a way, which is very difficult or even impossible for dishonest vendors to fake. Why Does the Financial Industry have Trust Issues with Blockchain? Transferring money from one country to another takes your time and can be expensive because of the need to comply with multiple different regulations and policies, but eventually, another person receives the money. Your bank updates its records and sends them to the main clearing bank in your country, where they are passed through a number of systems where it analyzes them to prevent money laundering and to comply with various trade embargoes. Assuming your transaction passes the checks, the funds are then sent to another country’s clearing bank. They again update the record and send it to the receiver’s bank account. Even in these highly automated times, it can take several days for the money to arrive in the destination account. This is where Bitcoin can offer an alternative to the traditional brick and mortar financial industry. All you need to do is to give recipients’ Bitcoin wallet addresses and use your computer system to send funds. There are no middle parties or long waiting times. To summarize Blockchain: The data store that keeps all blockchain data is called a ledger Each record in the ledger is called a . transaction These transactions are timestamped and saved in groups of blocks. You update or delete anything. It’s written once and then read-only. can not The ledger is built to be using cryptographic algorithms. immutable And the main part is ledger is replicated by multiple nodes, which is why it is called Distributed Ledger Technology How Does Blockchain Achieve and Maintain Integrity? The system takes a huge data file and computes its unique “digest” using a hash function. Your system then sends that huge data file to your friend/receiving party along with its hashed value. The receiver the hash to ensure that the content of that file is not altered while in the transmission. recomputes Multiple standardized hash function implementations are utilized, such as SHA256, which produces a 256-bit hash value. It gives 2^256 combinations. Furthermore, Blockchain utilizes Public Key cryptography which is a cryptographic system used for the encryption and decryption of data. You start by generating a special, related pair of keys – called a “public” and “private” key – based on a mathematical equation that uses two large prime numbers. Once encrypted, that can be decrypted if you have the required keys. The public key of both ends of the transmission is used to encrypt the data. At your end, your private key (which is not ever sent or made available to anyone else) is used in conjunction with the receiving party’s public key to encrypt the data. The receiving end uses your key and their key to decrypt the data. Combined with the hash value, transactions like this are incredibly secure. public private Public Key Cryptography is also utilized in communication to create a secure connection between hosts and for things like . code signing What is Blockchain’s Data Structure? In a blockchain system, data is stored in blocks of transactions. In effect is a huge timestamped log of transactions, which is replicated on multiple peers. For a blockchain network to be trusted and valid, most of the nodes have to be uncompromised. If 51 percent of the nodes are compromised, the network is hacked and has lost integrity. Note that the Blockchain expects things to go wrong on nodes, and yet it still remains reliable! Each block in a blockchain represents transactions If a user tries to add a block containing an incorrect hash value, the block will be rejected by multiple peers and hence will not be made part of the blockchain. Where Can I Learn More About Blockchain? You can find many interesting and useful articles related to: Blockchain.com API Cryptography Bitcoin Delphi and Ethereum Working with smart contracts and more . For more articles about Blockchain and cryptocurrency, click on this dynamic search link: https://blogs.embarcadero.com/?s=blockchain

A Quick Primer on Everything You Need to Know About Blockchain

Are you using SQL engines, such as Presto, to query existing Hive data warehouse and are experiencing challenges including overloaded Hive Metastore with slow and unpredictable access, unoptimized data formats and layouts such as too many small files, or lack of influence over the existing Hive system and other Hive applications? To address these challenges, we launched in the 2.1 open source release, which includes the . Alluxio Structured Data Management Alluxio Catalog Service Here is the FAQ based on the top questions from the community: How Does the Alluxio Catalog Service Work With Hive Metastore? The manages metadata for structured data, such as schema and table information. To do so, The Alluxio catalog manages connections to external metadata services (like Hive Metastore or AWS Glue) via an abstraction called Under Database (UDB). With the UDB abstraction, the Alluxio catalog can interact with other external metadata services for database and table information. Alluxio catalog For the Alluxio catalog to manage and maintain interactions with external metadata services, the external service must be “attached” to the Alluxio catalog. This attaching process adds the relationship between the external service and the Alluxio catalog. Attaching an external metadata service is initiated via the . “attachdb” CLI command In the Alluxio 2.2 release, there is an implementation of the UDB for the Hive Metastore. This means databases in a Hive Metastore can be attached to the Alluxio catalog so that the Alluxio catalog can manage and serve this metadata. In future versions, support for the AWS Glue UDB will be added. After an external database is attached, the Alluxio catalog syncs all the information from the external database and imports the table definitions into the catalog. The Alluxio catalog can then serve the information to the client, which in this case is in the Alluxio connector in Presto. The triggers the Alluxio catalog to sync with the UDB again. “sync” CLI command Once a UDB is attached, the Alluxio catalog manages the table metadata. This means the Alluxio catalog will transparently use Alluxio file locations instead of the locations found in the Hive Metastore without modifying the Hive Metastore. This allows simple deployment of the Alluxio catalog service with existing Hive Metastores. How Does the Alluxio Catalog Service Work With Presto? Presto interacts with the Alluxio catalog via the Alluxio connector. The Alluxio connector in Presto is new functionality for the existing Hive connector. The new functionality has already been merged into the codebases of PrestoSQL and PrestoDB. The Presto catalog needs to be instead of the Hive Metastore. Once Presto is configured in this way, all of the existing Hive connector operations will interact with the Alluxio catalog for the metadata of tables and not the Hive Metastore. configured to interact with the Alluxio catalog In the Alluxio 2.2 release, only read-only queries are supported with the Alluxio connector and Alluxio catalog. In future versions, write queries like DDL and DML statements will be supported. How do Alluxio Transformations Work? are performed by an , which executes distributed, asynchronous tasks. When a transformation is initiated, this is converted into a distributed job and executed across the various Alluxio job workers. Once the job is complete, the Alluxio catalog is updated with the newest transformed locations. When the Alluxio catalog serves the table metadata, the new transformed locations are used transparently, without any modification to the UDB. Alluxio transformations Alluxio-internal job service When transformations are generated, they are written through to the underlying persistent store and not cached in Alluxio. Once the transformed data is written, it can be read from the new transformed locations. The first read of the transformed data will cache the data in Alluxio on-demand. In Alluxio 2.2, the output file format is the parquet format since it is a more compute-optimized file format. In future versions, additional file formats and parquet output options will be available. For coalescing many small files together, the default behavior is to generate files of about 2G, but that is configurable. What is the Architecture of Alluxio Structured Data Services? Alluxio Structured Data Services is a part of the and has 3 main components: Alluxio connector in Presto, Alluxio Catalog Service, and Alluxio Transformation Service. Alluxio system The Alluxio connector is part of the Presto SQL engine. It is a connector based on the existing Hive connector and is the client to the Alluxio catalog. Alluxio connector: The catalog service manages the metadata for structured data and is a part of the Alluxio master processes. Alluxio Catalog Service: The transformation service is managed by the Alluxio catalog, and the transformations are performed by the Alluxio job service, which is an internal service for executing distributed, asynchronous tasks. The jobs are executed and managed by the Alluxio job masters and Alluxio job workers. Alluxio Transformation Service: Also Published At: https://www.alluxio.io/blog/everything-you-want-to-know-about-how-to-decouple-sql-engines-from-hive-data-warehouse/

A Primer on Decoupling SQL Engines from Hive Data Warehouse

I’m a Concept Artist/Singer/Sound-Designer with a deep interest in World Building, Experience Design, and Games. Think open-world, AR/VR/XR, and all things Metaverse/NFTs. When I started to go down this rabbit hole in the crypto space, I discovered many hidden corners, but the question that seemed to continue to interest the ‘market’ was: “What will be the next Axie Infinity?” This is because the proof of adoption/ $$ is so far the greatest in Axie’s gaming metaverse, particularly in the Philippines, with total earnings at times becoming bigger than whole industries, such as total remittances back to the Philippines from abroad. This is remarkable. Now, here’s a neutral statement: In the crypto space, the two opposite ends of the spectrum are a speculator or a community builder and a lot of very creative room in between. That is to say, you can see it purely as a wild west space (hence the question: where do I put my money next, given it might become as big as Axie), or an opportunity to create a sustainable way for you and others to monetize their skill sets while developing fruitful and mutually beneficial relationships along the way, and at the same time cultivating an engaged community that continues to draw value. I want to highlight the latter ethos, and in doing so, I think a better question to ask is, “what are some platforms and spaces that have the potential for creating real, engaged, vibrant experiences and communities in the play-to-earn and metaverse space.” Here’s a list of platforms and ways to track them (randomly thrown together): (Sandbox.Game): One I’m most excited about (full disclosure: I own a parcel in it), simply because in developing relationships with AR/VR/Voxel artists and experience designers, I’ve seen them being blown away by the potential and adoption of The Sandbox with the VoxEdit and Game Maker capabilities. The work that has gone into building it out really shows. I can’t wait to complete the Sandbox Bootcamp Certification and envision audio-centric and visually mesmerizing experiences and games. The Sandbox Some others to research are Decentraland, Cryptovoxels, Somnium Space, and Substrata. ( ): Dubbed as “The RuneScape of Blockchain Games,” is in private alpha at the moment. The team behind it appears to be solid and the support undeniable, and if crypto Twitter is any indication, it has a bright future. (Full Disclosure: I own a Founders Plot in Treeverse) Treeverse Treeverse.net On that note, I think crypto Twitter is great for developing an intuition around the Metaverse. Everyone should get on that. You can also contact me if you’re interested in consulting on specifics but starting with some basic hashtags related to metaverses, gaming, NFTs, and play-to-earn is a good idea. Also, Discord and Clubhouse, for the same reason because that’s where people gather. If you’re interested in a specific platform, always get on their Discord first. For general connecting/discussion, Twitter (inc. Twitter Spaces) and Clubhouse are the way. This isn’t on the chain, but they’ll have native NFT minting and marketplace capabilities. It’s a high-fidelity spatial audio XR platform that aims to be device agnostic for communal experiences. I did a demo with them and try to attend training sessions. I think it’s simply excellent as it emphasizes expertise and fundamentals of world-building and custom plugins over flashy templates to customize and curate your own experiences and find ways to gamify and monetize (whether using NFTs or not). : SpatialWeb.net ( ): On their website: “YGG IS A PLAY-TO-EARN GAMING GUILD, BRINGING PLAYERS TOGETHER TO EARN VIA BLOCKCHAIN-BASED ECONOMIES.” Definitely one to track, and they do some great work. A use case for how to be profitable while also being one of the good guys. They have their own YGG token (full disclosure: I own YGG tokens).  A while back, I minted a YGG Badge on their website (for free except gas, you should too), and I’m excited about what utility it would bring. YGG is also how I became familiar with a whole universe of games in the play-to-earn space. This rabbit hole also leads you down to Sushi Swap, where currently you can stake your YGG token to earn 2x return on yield farming (see this: ) Yield Guild Games yieldguild.io/ https://www.youtube.com/watch?v=YGOb7n3CU9A YGG would be amazing just for their Discord which is an aggregator of all the buzz in this space. The “GAME DISCUSSIONS” space above has literally all the games you want to keep track of, including a channel for The Sandbox. (Notice Axie Infinity has its own dedicated space). So you can keep scrolling down, and titles keep getting added, and each one is its own universe, of course — a very rich ecosystem. Speaking of aggregators here’s another great way to see what’s happening in this space: https://dappradar.com/rankings ranks top Blockchain Dapps (decentralized apps) based on usage and $$ (sliced by duration 24h/7d/30d), and as you can see, many of them are gaming metaverses. Alien Worlds on WAX blockchain and CryptoBlades on Binance are of particular interest: DappRadar Needless to say, there’s a lot to absorb here, but it helps to start with a general overview using the resources above and then focus on areas that naturally connect with you and speak to your strengths. That could look like an informed investor, a skilled gamer, or an NFT/Metaverse creator inspired by these “wins in the wild” and replicating them in your own project. A request. If you found this useful, would like to consult re: specifics, have any feedback, or would like to send a note of appreciation, you can reach me here: e-mail: eth wallet address: 0x4BdAA7fCE6A3640142afCD52115fBdb6Edc4D6a3 falsa@falsamusic.com Thank you for reading! This article is part of The hosted by HackerNoon in partnership with Gaming Metaverse Writing Contest . The Sandbox Submit your #gaming-metaverse story today for your chance to win up to $2000 in SAND tokens.

A Quick Primer on Play-to-Earn Gaming/NFTs/Metaverses

In recent months, a Vincent Van Gogh painting, 45 drawings of a renowned Marvel Comic artist, and a rare digital sword from an online gaming world were stolen. In the traditional criminal network, these items would eventually be sold on the black market. But a painting called “Ethboy,” a Batman comic by Marvel Comics and DC comic book artist , and any digital gaming assets created in The Sandbox gaming world will have no value to thieves in the black market. These owners have tokenized their assets with non-fungible tokens (NFTs) registered on the transparent and immutable blockchain. José Delbo NFTs allow buyers and sellers to cryptographically prove authenticity, rarity, originality, and other unique attributes of digital and non-digital goods through unique digital identifiers. Based on the ERC-721 Ethereum token standard — the blockchain co-founded by the subject of “Ethboy,” boy genius Vitalik Buterin — these indivisible tokens both store and create value. The first Ethereum NFT CryptoPunk has no inherent value other than to demonstrate the NFT use case. Rare Punk 6487 recently sold for 100 ETH ($20,857). Global supply chains, gaming platforms, the art market, and mineral developers are already tokenizing their assets. In 2021, NFTs will become a standard for buying, selling, and trading goods from digital wallets. Collectibles, Fakes, and Forgeries NFTs are quickly tokenizing the . Old Monopoly board games, barbie dolls, and sports tickets are selling for over $1,000 on e-commerce sites. Collectors are willing to pay top dollar for that coveted Pokemon card, Hall of Fame card, or silver spoon to complete their collection. $370 billion collectibles market The rise in fakes and forgeries, however, is increasing the risk and lowering the value of the collectibles markets. Non-fungible tokens provide a way to trace the ownership and authenticity of any good. The Multi-millionaire Dollar Market for Tokenized Sports Shirts and Shoes Blockchain platforms like and The Sandbox are and their memorabilia and collectibles — the former the Paris Saint-Germain team with David Beckham and his soccer mates, and the latter Manchester City.  But now the game players are the asset owners who can create value from brands. Sorare tokenizing entire football teams In The Sandbox, you will also find the Smurfs, Care Bears, and dozens of other major entertainment brands. Beyond entertainment, Sandbox gamers can derive value from, for example, rare or original collectibles based on popular characters. Cross-chain Functionality NFT owners are gaining more monetization opportunities through blockchain interoperability. The Sandbox virtual gaming lands is one of the early cross-gaming worlds. In the traditional gaming world, large gaming houses own assets and games. The games and assets gamers create are perishable goods. This gaming monopoly is being toppled by NFT-backed gaming avatars, swords, shields, and other digital assets.  With The Sandbox digital game asset maker, gamers can instantly turn game items into NFTs. of The Sandbox will open soon. Through their utility token called SAND, game developers, gamers, and investors can take any digital games and assets they create to other gaming lands. From The Sandbox Metaverse, games and digital goods will be fungible across Atari, Ubisoft, and other popular gaming platforms. Pre-season 0 The rise of gaming genres like social gaming and esports is feeding the cross-gaming platform frenzy. Cross-chain functionality is not confined to the gaming world. New blockchain games like MixMarvel’s Force For Fast car racing and AnRKeY X™ ($ANRX) are combining cross-platform gaming and DeFi investing, through NFT-backed yield farming, asset trading, and lending. Moving Towards a Tokenized World The games have begun tokenization and many industries are following close behind. Limitless use cases for NFTs are emerging. The NFTs of virtual gaming and entertainment lands like Decentraland and The Sandbox currently are among the top-selling NFTs. Users buy and tokenize land using the Decentraland token or LAND token in The Sandbox. NFTs are quickly taking off in the real world. Global trade businesses, artists and comic book artists, mining companies, and casinos are signing up in masses to blockchain platforms that enable NFT tokenization. Tokenization platforms like Rarible, SuperRare for art, and Sorare for sports memorabilia are among the top 10 selling NFTs. Expect a lot of assets to be tokenized this year and a growing share of sales in the global economy to be done via NFTs from digital wallets.

Non-fungible tokens (NFTs) are Here to Stay: A Primer

I wrote this article for Stable Report back in September 2019. I am reposting now since it is a good primer for where we are, and I will be posting a update that will reference this article soon. Please note, that the Digital Euro was not live when I first write this article, but China's CBDC is already functional, and the Digital Euro is expected to go live in the coming few years. In a quest to decentralize the globe, entrepreneurs have strived to create a new status quo for the world. Money, and the functions money serve, have been the main target of innovation. In the half decade past, the industry has blossomed tremendously, with projects like Ethereum, R3’s Corda, Hyperledger and more, becoming platforms for a new generation of high tech solutions facilitated by Blockchain and cryptography. These innovations, although initially supported by small teams working on relatively fringe projects, has now reached highs that seems impossible a few years back. Market behemoths, like JP Morgan Chase, Facebook, Walmart, and more, have all become involved in the space with their own blockchain projects. Although Stable Report focusing on stablecoins, and we see the introduction of large corporates as validation of the potential of this market, we recognize that private corporation are not the only players entering the space. Central Banks and their Digital Currency projects are very real, and will enter the market sooner rather than later. The introduction on Facebook’s Libra (read Stable Report’s article on ) has been a wakeup call for Central Banks around the world. The People’s Bank of China has been leading the way, and the industry media are buzzing with updates on their upcoming plans. But what are Central Bank Digital Currencies (CBDCs), and how do they function? Who are the key players in the market, which Central Banks will act first, and what is the expected timeline of CBDCs? Is Bitcoin dead again? Libra here Types and Functions of CBDCs: Through a Bank of International Settlements’(BIS) paper on , a breakdown of money types was presented. The “Money Tree”, a Venn-like diagram the divided money characteristics into four separate categories, is presented below. These characteristics are a) digital, b) central bank issued, c) token/value-based, and d) widely accessible. Central Bank Digital Currency The Money Tree, which should be familiar to researchers of CBDCs, is a good tool for differentiating cryptocurrencies, traditional bank deposits and CBDCs from each other. In this diagram, three separate CBDC categories emerge: a) Central bank, account-based CBDCs, b) Token-based CBDCs for commercial banks, and c) Token-based CBDCs for general use. The differences between the two token-based models is slight but important, since the proposed use of the two models focuses on different users. The commercial bank token model is focused on maintaining the Bank’s reserve targets and alleviating the settlement delays between the central bank and commercial banks. This solves pain points like overnight loans, cross-back payments, and more. The general purpose token-based CBDC model, however, would expand these uses to everyday transactions within the market – something like a central bank-sponsored Venmo, PayPal or CashApp using CBDC tokens. The benefit of this model would be the continuation of existing banking practices with little deviation from today’s banking state of affairs. The third option, Account-based CBDCs, however, would expand the functions of central banks in way not seen in the past. This CBDC model would allow citizens and CBDC users to hold their funds in central bank-based accounts, potentially cutting into the market share of commercial banks. As presented in the BIS survey on CBDCs, the potential for users to shift funds from commercial bank accounts and into CBDC accounts can be quite high in certain occasions. There are a lot of other factors a Central Bank needs to be mindful of before issuing a CBDC, but in many ways, selecting which of the above models a central bank will develop is the most important determinant to how significant a CBDC will be to the banking sector and the future of monetary policy. Key Projects and Players in CBDCs: So which Central Banks are exploring CBDCs, and how close are they to being officially issued? As mentioned above, the interest towards CBDCs skyrocketed when the People’s Bank of China announced that it is expediting the launch of their own project; a decision which in turn was fueled by Facebook’s Libra stablecoin. Media coverage seems to suggest that the CBDC will be launching in late 2019, namely around November, and specifically before the launch of Libra in 2020. Official designs for The People’s Bank of China’s CBDC are still not public, but reports dictate that it will be a two-tier system, with tokens being issued to banks and large telecommunication companies directly from the central bank, and through a private blockchain. The second tier system, from banks and telecom business to users, is still no known, although the PBoC aims for a transaction rate of around 300.000 per second. A more in depth analysis can be found in . Binance Research’s dedicated segment on the topic Although PBoC’s CBDC is the world’s leading CBDC project, with research starting all the way back in 2014, you will find, unsurprisingly, multiple other banks extensively researching CBDCs. In the aforementioned BIS survey indicates that out of the 63 central banks which partook into their research, a whopping 70% (or 44 central banks) are currently, or soon to be, engaged in serious CBDC research. In fact, the first consumer-based pilot CBDC program was launched by the Bank of Uruguay in late 2017. The program, otherwise known as the e-Peso, was an effort to increase financial inclusion in the country, and 20m e-Pesos were minted and distributed to businesses and individuals. The fund transfers were instant and peer-to-peer, either through text messages or through a dedicated e-Peso mobile app. The pilot was a success and was concluded in April of 2018. No further trials or plans are public as of yet. An additional CBDC pilot project is underway in Sweden, however no official date is set as of yet. With the ever falling use of cash throughout Sweden, Sveriges Riksbank is exploring a e-Krona CBDC as a digital cash alternative. For Sveriges Riksbank, the rise of private payment providers, namely Swish - and app that has achieved near 60% market penetration by 2018 – brings with it unwanted third party risk, and the e-Krona could be a more secure solution as money enters the digital age. However, the number of projects being researched and tested are plentiful and quite diverse. Accenture’s Global Blockchain Technology division has been particularly invested in CBDC research and implementation. Their reports and do a great job exploring the technical setup and social need for smart CBDC development by central banks. (R)evolution of Money (R)evolution of Money II (R)evolution of Money II lists a myriad of projects Accenture has been a part of. Often times in partnership with R3, Accenture has been instrumental in leading Project Jasper, Project Jasper-Ubin, Project Stella and more, paving the way for CBDC experimentation and proving that the technology is mature enough to handle the volumes of transactions expected by CBDCs. This report mentions the exploratory work on CBDCs by the Bank of England, the Bank of Canada, the European Central Bank, the Riksbank, the Bank of Japan, the Monetary Authority of Singapore, and the Eastern Caribbean Central Bank; and is highly recommended reading for anyone that wants to learn more about CBDCs. This piece would be amiss if it didn’t mention the work being conducted by the Bank of Russia, the Bank of Iran, and Venezuela’s Petro. Although the aforementioned projects are under development, little is known about the structure of Iran’s and Russia’s CBDCs, except that both have been touted as responses to US-imposed financial sanctions. Venezuela’s Petro, although state-sponsored, has more characteristics similar to a traditional asset-backed stable cryptocurrency than a CBDC, and has been highly controversial and mistrusted. On a final note, CBDC exploration does not end there, with multiple other Central Banks reviewing the technology and gradually moving forward with it. The Hackernoon article “ ” is a good resource for anyone seeking to explore country-specific projects. CBCD: 19 Countries Creating or Researching the Issuance of a Digital Decentralized Currency The expected timeline for globe-wide CBDC development is yet unknown, but the space in heating up. China, which is leading the way, is expected to launch before 2020, but the timeline for CBDC issuance by western central banks still seems far away. Expected Timeline of CBDC Launch: In conference, the topic of CBDCs was discussed. Michael Warner, strategist for the San Francisco Fed, argued that a FedCoin is still at least a decade way, and that the Fed will be the last institution to explore CBDCs since the US Dollar is already leading the traditional global financial market. IBM’s Think 2019 In the Europe front, the ECB has mentioned that Wholesale CBDCs are a viable option for the future. In most cases, wholesale CBDCs between commercial banks and central banks seem to be the technology that central banks are most excited to explore. All signs show that these will lead the way. Therefore, it seems safe to say that the 2020s might be the decade of financial innovation on money, cryptocurrencies, stablecoins and CBDCs. Everything seems to align with this trajectory, and money is due to change forever. Get ready. Conclusions and Other Thoughts So, where do we stand? CBDCs seem to be becoming a reality sooner rather than later; at least in the East and in emerging markets. What is holding their development back, and are CBDCs a threat to cryptocurrencies? What about stablecoins? The answers are not yet known, but with the information available, let’s try to speculate. For most cases, CBDCs will start through wholesale, with Central Banks issuing token to commercial banks long before you, the end user, gets a hold of them. Why is that? Well, mostly because the technology can help elevate pain points in the settlement process between banks and central banks. Will general purpose, token-based CBDCs enter the retail market? Yes, but probably after wholesale CBDCs become the norm in the back-end of intra-bank and cross-bank transaction. Commercial bank account (or an alternative) will be needed to access these CBDCs when they come. What about the third option? CBDC accounts in the Central Banks for general purposes? The future for such project is unknown, but it seems to bring an unnecessary risk with their implementation. It will be a financial innovation that commercial banks won’t be ready for, since Central Bank accounts will inherently be less risky; and of course less profitable, with no interest returns offered to account holders. But will a meager interest rate counter the uncertainty of keeping your money in commercial bank accounts? Not in times of economic downturn, nor when interest rates are in near historic lows, like today. In most cases, account-based CBDCs will interfere with fractional reserve banking, and I expect them to be the last CBDC version to be explored, and only in countries where financial uncertainty is at its peak. What about Stablecoin competition? CBDCs might just be the main competitor to fiat-backed stablecoins as long as high trust toward the Central Banks exist. However, the cryptocurrency industry was built as a hedge to the whole industry, and decentralized stablecoins will provide exposure to an asset that is theoretically uncorrelated to CBDCs. Would you trust your money with the Central Bank of Greece during the early 2010s financial crisis if they had a CBDC, for example? What about Bitcoin and other pure cryptocurrencies? CBDCs are simply a different beast altogether. No one can offer the privacy protection of Monero, Z-Cash and Dash, and Bitcoin’s value is proven to be uncorrelated to traditional markets and will lead the price of the rest of the blockchain industry. Virtual Machine platforms, like Ethereum, EOS and Cardano, are also significantly different to non-smart contract capable CBDCs, and it looks unlikely that any of the CBDCs researched as the moment will aim to create the smart contract functionality Ethereum or stablecoins like Libra and Terra are focusing on. Lastly, non-backed, algorithmic stablecoins are multi-coin systems that aim to stabilize their price by expanding and contracting the supply of their token. Isn’t that similar to what central banks do since the end of Bretton Woods? It seems that highly innovative projects, like said algorithmic stablecoin, will only flourish when trust and ever expanding usage of their system is established. For that to be the case, it is very likely that a central bank will have to support it – which does beat the purpose initially, but these systems can then be tested accurately, iterated upon securely, and decentralized later on. Does this seem like an impossibility? Well, surprisingly no! Outgoing head of the IMF, Christine Lagarde, has brought up the possibility of as they enter the mainstream financial markets. Of course, existing regulations will need to apply to these companies before any such formal action takes place. But as Ms. Lagarde takes the helm of the European Central Bank, the world will follow closely and monitor the way she handles an industry created to leave her out of a job. Until November 1st, when she takes over the ECB, the IMF paper on is a must-read for anyone interested in the topic. Central Banks acting as a lender of last resort to blockchain businesses the Rise of Digital Money As a final word, the world of money is changing, and the efforts of cryptocurrency entrepreneurs seems to have been successful in pushing the status quo in ways that seemed impossible a few years back. CBDCs, in retrospect, seem like an inevitable response to Bitcoin and Stablecoins. Now is the time. Let’s see what these entrepreneurs can accomplish, and how CBDCs will affect their projects’ future. Editorial note: Since the creation of this article, a number of key steps have been taken. First of all, the Digital Euro seems well on its way in Europe. Libra has intentionally reissued it's whitepaper and rebranded into Diem, and Algorithmic Stablecoins has started to remerge as potentially viable money alternatives. This was Xenofon Kontouris. To follow up on my work on Stabelcoins, CBDCs and Crypto, please follow me on Twitter at @XenofonOfAthens

A Primer on Central Bank Digital Currency – What’s going on? 

DevOps, the method of , is one of the essential factors of successfully implementing a cloud-native approach. automating processes between development and operations Since cloud-native has a purpose to , DevOps is what streamlines individuals, tools, and systems, contributing to the overall success of the enterprise. This is what makes cloud-native DevOps a logical step towards improved productivity. reduce go-to-market time and bring more efficiency to companies Both cloud-native and DevOps movements are on the rise. predict that . Reports and Data the global DevOps market will grow from $4.49 billion in 2019 to $17.27 billion in 2027 What is more, . say that by the end of 2021, . This is mainly due to the shift to a remote-first workforce. the COVID-19 pandemic significantly influenced cloud adoption Predictions 80% of companies will double the adoption of cloud-centric infrastructure and apps A cloud-native DevOps approach maximizes agility with continuous deployment. Moreover, it scales up systems and applications without the need for drastic changes. Let’s dive in and see what this approach really means. What is the cloud-native approach to software development? Adopting the cloud-native approach means embracing many new options for leveraging innovations. The primary tools cloud-native teams use are offered by the cloud instead of on-premise infrastructure. Cloud-native applications are This means that teams build them with frequent and quick deployments and iterations. Instead of preventing failure, they embrace it and have . agile, adaptable, robust, and easy to monitor. the ability to improve in real-time Furthermore, they don’t rely on external factors when it comes to monitoring. Instead, the application can be controlled from the inside. The cloud-native beginnings date from around ten years ago, when a group of tech leaders introduced it to the public. by Paul Fremantle, VP of Technical Product Strategy at Weaveworks, is among the first related articles. In the post, he claims that applications should be written FOR the cloud to behave well ON the cloud. This blog post Then, in 2015, Matt Stine, Executive Director at JPMorgan Chase, cleared the cloud-native concept in his book Migrating to Cloud-Native Application Architectures . Now, , etc. provides a great overview of most of the tools used in the cloud-native approach.  To help and provide development communities with good alternatives for building cloud-native applications, the CNCF cultivates this landscape of open source tools. the cloud-native umbrella covers many tools and methods, including microservices, containers, CI/CD, multi-cloud The CNCF Cloud Native Interactive Landscape What is cloud-native DevOps? If we combine the two definitions, cloud-native DevOps turns into which result in improving the entire experience of the end-user. Cloud-native DevOps is all about You achieve this through technologies like Kubernetes and containers, which can automate processes and make applications more scalable. a set of practices that consists of constant development, automated actions, and cross-functional teams, making your company more flexible. All these practices apply to almost every aspect of the company. This means changing the entire culture which involves tools, people, and processes. When it comes to you need to . This will foster better alignment between people and tools. culture, adopt a DevOps-oriented approach You have to motivate your employees to collaborate more effectively, encouraging a faster feedback circle. Faster feedback will enable quick iterations and real-time improvements. You’ll also have to make some . This refers to adopting technologies that are in the center of cloud-native DevOps, such as containerization and microservices. technical changes The main technologies in Cloud-Native DevOps How to get started with cloud-native DevOps? These are the options to get started with a cloud-native strategy: Implementing a cloud-first strategy by choosing a cloud provider instead of an on-premises one Adopting a multi-cloud strategy if it meets business needs more closely Embracing DevOps, which is obviously what we’re going to focus on further in this post. When companies realize that agile development requires both automation and culture transformation to produce quality applications faster, DevOps becomes a priority. Handling multiple hybrid environments or streamlining infrastructure stuff can get very complicated. That’s why the popularity of tools that align cloud-native and DevOps processes is rising rapidly. To make the implementation of cloud-native DevOps easier, these are some things you should pay attention to: enables the creation of environment-agnostic software. It also eliminates implementation conflicts between developers and operation departments, enabling developers and testers to communicate easier. Containerization also makes the company more agile. This establishes a set of tools, terms, and processes for both development and operation teams. With microservices, you get sophisticated process automation and an easier transition towards agile product development, which is what continuous delivery requires. Going from monolith to microservices . The most popular is Kubernetes because it solves the challenges related to underlying computing, storage, and networking. Using a container orchestration platform in order to reduce complexity and the chances of errors. This automation can save a lot of headaches for developers. Moreover, it allows them to focus on the actual product instead of fixing issues. Writing code also makes things more transparent. Automating CI/CD pipelines according to the 12 factors Most common mistakes in adopting cloud-native DevOps Using too many tools Once you switch to cloud-native DevOps, you’ll try automating as many processes as possible. However, you don’t do this by just adding and adding tools. You have to pick the right tools and make the best combination that’s right for your application. Overusing tools will also cost you a lot of time and money. Another mistake here is that developers often get too reliable of a certain tool. The essence of DevOps, however, lies in the team spirit and the implementation of correct practices that contribute to increased productivity and enhanced processes. Avoiding continuous monitoring While testing inside the implementation pipelines makes the things more straightforward and flowing, it also makes them limited  and incomplete. Continuous monitoring, on the other hand, can optimize the entire process by highlighting each malfunction that happens even after the testing. Not paying enough attention to security Security checks can be quite time-consuming and costly. Teams often think of that security checking capabilities as something that is put to use within the CI/CD workflows. The implementation of a separate tool that takes care of security is a critical move for DevOps to turn and avoid any vulnerabilities in the process. Making the transition too fast Adopting cloud-native DevOps should be a slow process with a lot of learning on the way. Expecting a company that has been using on-premise applications to immediately align all their structures and platforms into a single cloud-native architecture is simply impossible. Maybe you can quickly make new cloud-native applications, but transitioning existing applications will take a while. The best solutions would be to go step-by-step. Start by migrating applications to the cloud and continue with transitioning from monolith to microservices. Then, you can set up containers and container orchestration platforms. However, none of this will be successful unless you undergo the culture shift we talk about so much. To wrap things up Effective implementation of cloud-native DevOps helps businesses to market fresh technologies more rapidly, speeding up their digital transformation. Through cutting enormous cloud costs and rising efficiency and performance, the cloud-native approach has become a blessing to companies with software products. Completely relying on the potential of cloud-native DevOps, unlocks on-demand, unlimited software development powers. Previously published at https://microtica.com/a-comprehensive-introduction-to-cloud-native-devops/

Cloud-Native DevOps: A Basic Primer

As a developer, when you think of security, what comes to mind? Is it clear what are different aspects that need to be handled to make a software secure? And why you need to do that? . Details of how to apply those controls are not covered as they would require separate articles - and lot of content is available on the web anyway. The objective of this article is to provide that view and articulate what controls should be in place and how What is Software Security? Security implementation of a software application can be classified in two parts: Pre-deployment - building a secure software Post-deployment - security of the environment where the software is running Software Security is pre-deployment. It is the process of identifying risks and building controls (or as it is called in security terminology) in the software itself while it is being built. Countermeasures Software Security is the focus of this post. We will see controls commonly used and what risks they mitigate. For more details, see What is Software Security by Gary McGraw Nature of security A software application would generally have two aspects: Services that provide some functionality a. Data generated and consumed by the services b. Security can be defined as . defending the services and the data from unauthorized and malicious usage at all times is the key word here. Defense literally means . That means attacks can happen anytime and any number of times, and we need to keep protecting the system from these attacks. Defending an act of resisting an attack That's what makes security of a software application very difficult because it is not easy to get it right all the time. Also, there is no room for error. One incident may be enough to destroy the reputation and business built over the years. How to approach security? The following is the minimum that should be done: Rather than inventing our own mechanisms, follow the best practices. These best practices have come out of years of learning and it is better to rely on that tried and tested knowledge. Follow industry best practices: : Make security a criteria in all aspects like Requirements, Architecture, DevOps, Engineering Practices like code reviews, unit testing, application testing, etc. This would ensure that everything is built secure by default - security is not thought of after developing the functionality. Secure by design What would make a software secure? CIA triad (or AIC triad) is commonly used for Information Security, but it also helps to better understand security in general. A software should have the following properties to be secure: : Authorized access to services and data e.g., to view your GMail mailbox and send emails, you must successfully login to GMail. Confidentiality : Integrity and authenticity of the data presented (security in transit) and stored (security at rest) e.g., the emails shown by GMail are exactly how they were sent by the sender and the content is same as that stored on the GMail servers. Integrity : Software application is always available to all authorized users for all legitimate use-cases e.g., GMail services are always available to its users. Availability Design Principles See by OWASP for details. Secure Design Principles is a very important principle to follow while designing the software. In simple terms, it means to have controls at different layers. It will ensure breach at one layer will not result in total failure. We'll see some examples later in the article. Defense in depth Controls The following are the controls that are commonly used. A combination of these controls can be implemented depending on the risk. NOTE: These are commonly used controls. This is not an exhaustive list e.g., though it is not included above, applying signatures on every request and response can be used if the risk is high and require that level of security. We will look at each of these. However, they build on the security building blocks like authentication, cryptography, certificates, etc. If you are not familiar with these concepts, it may be helpful to go through . Software Security Building Blocks Authentication Why? Authentication is required to perform authorization. Authentication will establish the identity of the entity trying to access the services, and based on that authorization can be applied. How? There can be two types of authentication: Authenticating an end user: and are widely used protocols for this. There are number of identity management services that support these protocols. OAuth2 OpenID Connect Using a standard protocol rather than custom authentication mechanism would make integrations with other systems easier. Authenticating a client: Two commonly used methods to authenticate a Client (a program calling an API): : This is an OAuth2 workflow that allows a client to authenticate itself using a and a and acquire a token which it can then use to call APIs. OAuth2 - Client Credentials Grant client-id client-secret : Both the Client and the Server trust each other's X.509 certificates used to establish TLS connection. Connection is established only if trusted parties try to connect. This is suitable in server-to-server communications e.g., when a backend service calls an external API. TLS Mutual Authentication For Web Applications that rely on cookie based session tracking, authenticating legitimate requests from client (Browser) is required to avoid . Watch this to see a demo of CSRF. Cross Site Request Forgery (CSRF) video Authorization - for Confidentiality Why? Authorization in this context is required for the following objective: Authenticated entity - a user or a client - can access only the data that they are supposed to access e.g., when you login to GMail, GMail does not show someone else's emails to you. How? All read operations on data associated with a user should have the following checks. The logged-in user is same as the user associated with data e.g., User Id on a payment transaction record must match the User Id of logged in user. The same check can be applied to clients if required. If #1 is not true, the logged-in user has enough permissions to access other user's data. Authorization - for Integrity Why? Authorization in this context is required for the following objective: Authenticated entity - a user or a client - can perform only those data manipulation actions that they are authorized to e.g., when you login to Outlook, you cannot manage someone else's calendar unless the other person has given you permissions to manage his or her calendar. How? All operations should have the following checks: Is the logged-in user authorized to perform the operation? If Role Based Access Control (RBAC) is used, is a role required to perform the operation assigned to the logged-in user? Spring Security and .Net Framework easily allow such access control using declarative methods - annotations \ attributes. The logged-in user is same as the user associated with data that is being modified e.g., the User Id on a payment transaction request must match the User Id of the logged in user. Client Authorization: In the case of APIs that can be consumed by multiple clients, authorizing clients is also important e.g., in a Microservice architecture, an API gateway can block requests if a client is not authorized to use an API. In the above design, an admin can disable a user so it is allowed from the Admin Portal client application only. If the Web Portal application tries to call the API, it is blocked. OAuth2 JWT tokens carry Client information and this can be done easily by inspecting tokens. NOTE: The same check should also be applied on the Microservice - remember Defense in depth? In case someone mistakenly removes the check on API Gateway, microservice will still block the client. Data Privacy Why? Personally Identifiable Information (PII) is any data that can identify an individual e.g., Mobile Number, Email address, etc. Unauthorized access to such information can be a breach of privacy. How? Identify PIIs captured by the software. Do not display PIIs without masking e.g., mobile number as 98XXX XXX76. All communications like email, SMS, etc. should mask PIIs. Do not write PIIs to the log files in plain text. Input Validation Why? Accepting invalid input can compromise the integrity of the data e.g., accepting a past date on a payment transaction request would result in incorrect state of the system. Lack of input validation can pose a vulnerability for Cross Site Scripting (XSS) attacks. See for a quick introduction to XSS. CROSS-SITE SCRIPTING (XSS) TUTORIAL How? Inputs should be validated at all layers: Defense in depth! Sanitize all text input to remove <script> tags. Otherwise the <script> tags would be stored with text and it might end-up in a malicious script being executed when the text is displayed in the browser later on - that's XSS in a nutshell. If a file is being uploaded as part of some functionality, it should also be treated as an input and be validated - scanned and its format validated if required. TLS \ HTTPS Why? For security of data when it is in transit. It protects against attack which can compromise both integrity and confidentiality of the system. Man-in-the-middle How? Always use TLS 1.2 (currently latest) and HTTPS for all internet traffic. In mobile applications, implement to make sure the mobile app always talks to "your" server only. See for details. SSL Pinning Android Security: SSL Pinning Encryption and Hashing Why? Security of data at rest (when stored) is as important as security in transit. Storing encrypted or hashed data will prevent unauthorized access to data at rest. How? Identify data that needs to be stored encrypted or as a hash value. Hash of a password is stored so it is irreversible even with the keys. Encryption key and the data should not be stored together e.g., in same database. Challenges: Encrypted data is as secure as the key it is encrypted with. Therefore, management of encryption key is very important when encrypting data at rest. Key Protection: Having to decrypt data on all read operations will impact the overall responsiveness of the system if frequently used data fields are stored encrypted. Performance trade-off: Once data is stored with an encryption key, it has to be decrypted with the same key. When data starts accumulating over the years, rotating the encryption key becomes a challenge because existing data needs to either (a) be re-encrypted with new key, which may be challenging in a live system or (b) handle multiple keys and key-data mapping in program itself, which can become very complicated. Key rotation: Cloud services like AWS Key Management Service (KMS) provide key protection and key rotation out of the box. Access to KMS keys and its encryption APIs can be controlled using IAM permissions and the client can  encrypt / decrypt  data without accessing the key itself. Auditing and Logging Why? Auditing helps to track the change in state and who caused that change e.g., if a user's name was updated, audit records will capture when it was updated and who updated it. This can be useful in event of an incident. Logging helps to trace errors and troubleshoot issues. How? Auditing is straightforward. Hibernate makes it very easy in Java. Logging too is straightforward, but in distributed architecture using or something similar is important to see all logs in one place. ELK Stack Secrets Management Why? Secrets are database password, API access credentials, etc.
 Most of these are configurations but if they are left in plain text, it poses a risk as they can be misused. Let's say an organization uses an external SMS service provider to send SMS. If the API access credentials for this external service are not protected, there is a risk of someone being able to send SMS with the identity of the organization. How? You can use AWS Secrets Manager, Vault, etc. to store such secrets. These services can also automatically rotate secrets. Alternative is to
 keep secrets encrypted in normal configuration - properties file or web.config, and the encryption key is stored in AWS Secrets Manager, Vault, etc. In Single Page Applications (SPA) and Mobile Applications, generally a JWT token would be used to call APIs: token is passed in Authorization header with every request. Since it is a bearer token, anyone with the token can use the APIs. So the token should not be stored on clients more than required
. NOTE: There is trade-off between security and usability. Most applications (web and mobile) keep users logged in forever, which means they have to keep a token with long validity on the client: in browser local storage or mobile. This is a risk. So a decision has to be made based on nature of the application. Sender Policy Framework Why? Let's say an application sends emails from address. If someone else is able to send emails using the same domain myapp.co, it can impact reputation and business of the organization. This is called Sender Policy Framework prevents such misuse of domain and assures authenticity of emails. connect@myapp.co Spoofing. How? SPF is configured with a DNS entry. See for details. What is SPF Resiliency Why? Resiliency here means ability of the application to be responsive (operational) in the event of a failure. See for more details. Reactive Manifesto In the context of security, we can loosely define resiliency to the following: In the event of a sudden increase in load (with malicious intent i.e. Denial of Service (DoS) attack), can the system remain operational?
 All cloud providers have services that provide protection against DoS \ DDoS attacks, but these are reactive in nature and may take a little time to react. The application should be able to withstand that otherwise the protection is of no use. How quickly the application can recover in the event of a downtime? How? There are different modern architecture patterns to achieve resiliency, but in any case - even for monolith architecture, the system can be resilient with following implementation: Elasticity means ability to scale up or down depending on the load.
 It is easy to implement with Load Balancing and Auto Scaling in any cloud environment. Elasticity: Automated builds and deployments is critical to quickly recover in the case of a failure.
 Unit testing and Test Automation would also help to do quick releases for urgent patches. Automation: Configurations keep getting added to a software as it evolves and often it become very difficult to correctly configure the software.
 This is important to have quick recovery in case of a failure. See for impact of configuration on operations and ways to mitigate the challenges. Configuration Simplicity: Application Configuration from Operations Viewpoint Conclusion Security is difficult if it is ignored, but Identifying risks and putting in place appropriate controls by following industry best practices and secure by design principles can make it a less daunting task. Also published at https://dev.to/pathiknd/software-security-overview-3ldi

Stories, News & Mentions about PRIMER

Primer’s Stories on HackerNoon