Kwilis in the Database Technology Industry

Kwil

Special thanks to Vishwanath Raman from Oasis Labs and David Whittington from AR.IO for feedback and insights. Can distributed ledger technology , helping companies save over per year? Or can blockchains help us build a , helping reduce the nearly ? Could we even use the same technology that powers Bitcoin and Ethereum to enable humans to ? decrease cloud outages $26.5 billion fully connected and autonomous driving network 1.9 million annual traffic deaths worldwide become an interplanetary species The unifying element in these use cases–cryptocurrency, cloud outages, autonomous vehicle networks, and even space travel–is overcoming byzantine failures. While blockchains can’t yet support the throughput or performance required for many of these applications, overcoming byzantine failures and allowing businesses and institutions to share open data in trustless environments has the potential to realize over . $3 trillion per year in economic value At Kwil, we often describe our database software as a “Byzantine Fault Tolerant Postgres.” This article will explain precisely and practically what “Byzantine Fault Tolerant Postgres” means and how it enables real-world, . trustless digital infrastructure “Byzantine Fault Tolerant Postgres” is the combination of two distinct technologies: byzantine fault tolerant (BFT) consensus algorithms and the PostgreSQL database engine. What is Byzantine Fault Tolerance? Byzantine fault tolerance arises from the “Byzantine Generals Problem,” a distributed systems problem by researchers in 1982. The Byzantine Generals Problem imagines a hypothetical scenario of Byzantine armies, each with its own general, camped outside an enemy city. Without a central authority to command and coordinate attacks, the generals must collectively vote on whether to attack or retreat via messengers. conceived The key challenge in the Byzantine Generals Problem is the generals have no way of knowing whether the other generals are loyal or traitors. For example, General 1 could tell Generals 2 and 3 to attack, and General 2 could tell General 3 that General 1 said to retreat (as shown below). General 3 cannot know whether General 1 or General 2 is the traitor, and thus, the three generals cannot progress safely. For a system of number of traitors to proceed safely, there must be 3 + 1 generals; therefore, >⅔ of the nodes must agree and act honestly. If ≥ ⅓ of the nodes in the system act deceptively, then the nodes will not be able to discern the safe path forward (i.e. the system loses liveness). I won’t go into the mathematical proof for the ⅔ threshold, but you can read a succinct and accessible explanation . n n here The key breakthrough with byzantine fault tolerance is that networks can be run and maintained by parties that do not trust each other. The most popular application of byzantine fault tolerance today is cryptocurrency networks; however, other real-world examples include enabling fault-tolerant safety systems on and protecting against nodes corrupted by radiation in SpaceX’s International Space Station . Boeing airplanes landing procedures In a Postgres database network, byzantine fault tolerance means that more than ⅔ of the nodes must agree on the safe path forward. Assuming a blockchain-based approach, this means that greater than ⅔ of the nodes must agree on each block, including all the transactions within the block, before progressing to the next block. Therefore, BFT Postgres networks can be run without trusting the other parties running nodes. What is Postgres? PostgreSQL, also known as Postgres, is an advanced, open-source, object-relational database engine. It is among the pieces of software worldwide, used in production at organizations ranging from the International Space Station to Reddit and Instagram. Relational databases of the total database market, with Postgres commanding 17.4% alone. In the last week, you have probably used or interacted with an application that relies on a Postgres database. most popular comprise 60% The key to Postgres (and other SQL databases) is that it uses the , storing data in structured tables. Tables are comprised of columns and rows, where each row represents an individual piece of data (also called a record), and each column represents an attribute. By storing data in a relational structure, data can be , enabling better maintainability as a dataset grows and the ability to combine data in ways that yield unique insights. For example, a university that needs to manage information about its academic staff and their professional activities could maintain a relational database with the following normalized structure: relational model normalized In the example above, storing data in a normalized structure offers several advantages. First, the normalized structure minimizes data redundancy: each piece of information is stored only once, reducing the likelihood of inconsistencies and errors. Second, it simplifies updates in the database: changes to a professor’s details, like a department change, only need to be made in one table, making the update process more efficient and less error-prone. Third, the relational structure makes it easier to combine relationships and queries across tables, such as the relationship between courses and publications that are relevant to the subject. Lastly, normalization makes database maintenance easier, especially as the volume of data grows over time. A further benefit of the relational model, especially in contrast to , is transactional integrity, guaranteeing that all or none of a database operation is executed and that concurrent transactions do not interfere (also known as ). This is critical in fields where precision, consistency, and reliability are key, such as banking and healthcare. NoSQL databases ACID transactions What is Byzantine Fault Tolerant Postgres? As inferred from the name, “Byzantine Fault Tolerant Postgres” combines the two technologies mentioned above: a byzantine fault tolerant consensus algorithm running over multiple Postgres databases. In a Kwil network, each node maintains a BFT blockchain and a Postgres database. When a state-changing transaction enters the system (e.g. database creation, database drop, insert/update/delete statement, or a funds transfer), the node first broadcasts the transaction to the other validator nodes. Each validator node then uses the transaction's digital signature to determine whether the transaction’s origin is legitimate. To guarantee and data storage and deletion, nodes must first achieve consensus on the valid transactions in a block. Once a block is confirmed, the nodes execute the transactions on their local Postgres database. To propose or validate the next block, a validator must prove that it correctly applied the changes from the previous block. This is proven by generating a proof containing the result of all the proposed changes from the previous block, applying a schema, and generating a resulting hash. The resulting hash is then included in the next block, and each node must have the correct hash to continue in the block mining process. This process ensures that each node stores the data on its Postgres database, guaranteeing that nodes properly insert, update, and delete data. strong consistency deterministic ordering How is BFT Postgres Useful? The idea of a byzantine fault tolerant relational database is not new; multiple have been published about how a generalizable BFT Database could help improve reliability and uptime while protecting against malicious actors in a distributed system. However, until recently, consensus algorithms were neither sophisticated enough nor could handle the data throughput required to be truly byzantine fault tolerant. Recent interest and funding for blockchain technology, largely due to public interest in cryptocurrencies, has spurred significant advances in BFT research and development. academic articles The main advantage of a byzantine fault tolerant database is that it allows for databases to be operated and shared among parties that may not “trust” each other. In a cryptocurrency network, parties may not “trust” each other because malicious actors seek to subvert the system for financial gain. In space landing systems, parties may not “trust” each other because of that corrupt a computer’s memory. Byzantine fault-tolerant databases enable trustless digital infrastructure. radiation-induced “bit flips” A key advantage of trustless digital infrastructure is that, for the first time, it establishes inviolable rules for how an application will behave. Because no centralized actor can subvert, change, or otherwise disrupt the infrastructure, users can confidently rely on that infrastructure without the de-platforming and high rent-seeking risks in centralized systems. For example, the uses Kwil to store user data across a multi-node, multi-party, BFT database network. The idOS’s top consumers—regulated DeFi protocols such as —require a stable and reliable identity , not an identity , to identify and KYC its users. Using a centralized identity would mean that Gnosis Pay could not make cryptographic guarantees about how its entire protocol would function; the identity company could suddenly disrupt the protocol. Using an identity means that Gnosis Pay has an inviolable guarantee for how its user onboarding and KYC will behave, and it can build its platform on the premise that those guarantees will not change. For the idOS, using a byzantine fault tolerant database allows it to make these inviolable guarantees needed to power regulated DeFi protocols. idOS Network Gnosis Pay protocol company company protocol Conclusion A “Byzantine Fault Tolerant Postgres” database is the unlock for trustless digital infrastructure, enabling a freer, fairer, and more resilient internet. As throughput, latency, and generalized computing capabilities continue to accelerate, trustless technologies like a BFT database will be able to reshape the ways we think about and require trust in technology. Does building with a BFT database sound interesting to you? Kwil’s binaries are available and free to use, and we are excited to go open source with our v0.7 release next month. Anyone can start a multi-node, BFT database network with . just four terminal commands Although the (v0.6) is based on the simpler, less feature-rich , the upcoming release (v0.7) uses the PostgreSQL engine. current Kwil release SQLite database engine Questions or comments? Disagree with this article? Join our , or reach out to luke@kwil.com. As always, we welcome community feedback and the opportunity to push decentralized technologies collaboratively into the future. Discord Footnotes [1] Image Source: https://www.cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdf [2] Image Source: https://www.cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdf

Fortifying Postgres With Byzantine Fault Tolerance

The world of web applications is rapidly undergoing a fundamental shift from web2 to web3 technologies, with decentralization and composability at the heart of this transformation. For dApp developers, this shift is not just a trend but a necessity. Decentralized databases, exemplified by permanent storage solutions like are becoming the backbone of secure, scalable, and user-centric applications. Arweave What Actually Is a Database? Technically, just an “organized set of data held in a computer.” In theory, this means that anything more structured than a pile of files on a disk somewhere can be called a database. In practice, it’s more specific than that. a database is Databases of course provide a place to put files, but most importantly, they provide . A schema plus a query layer. In this sense, Arweave itself is a database -- one big pool of data, searchable with GraphQL. tools to find the exact data an application needs The catch is that in permissionless systems, it’s a matter of bottom-up standards, not top-down schemas. The Atomic Asset (SWS-1) and Asset Discoverability (ANS-110) standards both specify ways to make Arweave data easy to find and use in any application, promoting interoperability. The Rise of Decentralized Databases Censorship, rent-seeking, data availability problems. Decentralized databases have emerged as the antidote to these pitfalls of centralization. They offer unparalleled security, data immutability, and user sovereignty. , for instance, epitomizes this with its "pay once, store forever" model, making it a cost-effective and pro-user solution for permanent dApp state storage. Arweave In the Arweave ecosystem alone, teams at , , , , and more are working to provide developers with a way to easily use and query Arweave with familiar syntax. Kwil WeaveDB HollowDB Glacier Empowering dApp Developers Security and Immutability The decentralized nature of these databases ensures that once data is stored, it becomes immutable. This is ideal for applications requiring secure, tamper-proof systems. Scalability and Flexibility Decentralized databases scale effortlessly with your application. The architecture of decentralized databases ensures data availability even if some nodes go offline, offering a reliable and scalable solution. User-Centric Design Decentralized databases give users unprecedented control over their data. It’s being stored with whatever database layer you choose, but it’s also there forever on Arweave, and no one can take that away. A compelling feature for dApps aiming to be more user-centric via data portability. Cost Efficiency The absence of central servers means no regular maintenance or need to provision infrastructure, reducing operational costs. The Evolution From Centralization In the Web2 era, centralized databases were the go-to solution for data storage and management. While they offered a controlled environment, their limitations were glaring: susceptibility to hacks, high maintenance costs, and being a single point of failure. However, certain centralized elements, such as APIs and caching layers, still hold value and have been integrated into this new generation of databases. The Hybrid Future: Best of Both Worlds While decentralization offers numerous advantages, a balanced approach that incorporates elements of centralization can be incredibly powerful. After all, the blockchain trilemma states that a system weighted towards decentralization and security will sacrifice scalability. Base layer Arweave is both secure and decentralized, but without centralized helpers (caches, gateway magic, APIs, etc.) databases that leverage it can’t provide the experience users or developers have grown to demand -- we’re accustomed to web2. What was once considered blazingly fast now ™, and anything less capable isn’t an acceptable choice for most developers. just works This hybrid model -- a decentralized base with centralized, hot-swappable helper layers -- acts as a bridge between what web2 expects and what web3 enables. Real 🌏 Impact Decentralized databases are making significant inroads in various sectors, from supply chain management and healthcare to financial services. Arweave's permanence is particularly useful in applications requiring long-term data availability. “WeaveDB can replace centralized databases in all cases,” says founder LensPlay Sahil Kakwani. “It's a standout choice for applications requiring decentralization, resilience, and high performance.” In another WeaveDB user interview, founder Harang Ju noted that decentralized solutions “will easily replace traditional centralized databases in places where transparency and openness are key.” Tepe While building the MEM explorer, the Decent Land Labs team implemented Kwil as an alternative to directly querying MEM interactions from the Arweave L1 chain. Compared to the GraphQL option, which takes several seconds, Kwil is able to return data almost instantly. While mainstream adoption may be a little while down the road, essential components are being built and battle-tested in preparation. Where We Stand… At Decent Land Labs, we believe in user sovereignty, data composability, and the . The future is not just decentralized, it’s also hybrid, leveraging the strengths of both centralization to create highly capable, scalable, and interoperable systems with a strong foundation. storage-based computing paradigm By embracing this shift and leveraging hybrid solutions along with the data permanence decentralization offers, developers are not just following a trend; they are preparing themselves for the future. Check out for more insight on how to integrate your dApps today! docs.mem.tech If you enjoyed our content, please and feel free to comment below! SUBSCRIBE Our new protocol release Molecular Execution Machine is now live! - Beta Testing Sign Up mem.tech

The Decentralized Database Revolution

La revolución de las bases de datos descentralizadas

La révolution des bases de données décentralisées

विकेंद्रीकृत डेटाबेस क्रांति

分散型データベース革命

A revolução do banco de dados descentralizado

Cuộc cách mạng cơ sở dữ liệu phi tập trung

去中心化数据库革命

While Web3 is screaming decentralization, the likes of Facebook, Amazon, and Google are hoping it's just a false alarm. As the internet moves closer to the era of decentralization with Web3, it's important to understand clearly how data control will be affected and the likely turn data engineering will take. The internet has welcomed many stages of evolution since its inception with the internet moving from being merely a news board to a place for interaction through the medium of social media. As human experiences are being gradually improved upon, there are concerns about how much data one can control in the bid to enjoy personalized services. What will be the fate of the numerous tech giants that depend on vast quantities of user data to offer personalized user experiences, which is undoubtedly a guaranteed strategy for success? This is the best time to quote Aaron Swartz briskly: Information is power, but like power, there are those who want to keep it to themselves. Web 3 appears to offer a safer online environment with improved data privacy and security. Although adoption is still in its early stages, it is crucial to prepare for what to anticipate in the wake of the next wave of the 21st-century internet. This is a tale of the data control dilemma with the encroachment of decentralization and loudly preached transparency. Let the tales begin! We will start with the basics. ? What Is Web3 Web3 is the next stage of the internet evolution. Powered by technologies like the blockchain, decentralized protocols, and cryptographic systems which store data using to ensure the data is secure, Web3 moves from the era of a centralized web as seen in Web2 to that of a decentralized web. cryptographic algorithms Still in doubt? Listen to Aya Miyaguchi, the Executive Director of Ethereum. Web 3 is about creating a more decentralized and open internet where and interactions, without relying on centralized intermediaries for access and validation. users have control over their data ? What Are Cryptographic Algorithms Imagine you have a secret message to send to your friend. You want to send it through the internet, but you're not sure of the safety of that message. You don't want anyone else to know what it entails, so you decide to use a special code that just you and your friend can comprehend to write the message. Based on the level of your friendship, you may decide to twist some words to mean something else that can only be decoded between you and your friend. That's where cryptography comes in. Adding an algorithm would be including a special set of rules to your codified message to enable your friend to understand it. Now, let's get a definition. Cryptographic algorithms are rules that turn secret messages into secret codes that only the parties involved in the interaction can understand. and cryptographic algorithms used to store data on the blockchain ensure the security and confidentiality of personal information. Privacy is a fundamental pillar of Web 3, By prioritizing privacy and security, Web 3 empowers individuals to take control of their data and protects them from the invasive practices of centralized entities. What Is a Decentralized Web? This term describes an internet environment that doesn't rely on a single entity like Google, Facebook, or Yahoo for control. Instead, it uses blockchain technology to enable direct interaction between users- all the computers and devices are equal and can communicate seamlessly. peer-to-peer In the Web 3 world, we don't need middlemen to trust each other. We can trust the system itself. - Joseph Lubin, Co-founder of Ethereum & CEO of ConsenSys Imagine walking into a room and interacting with everyone without hassle, that's what - feels like! peer to-peer Web3 Is Coming… Thanks to the early adopters of Web3, there are gradually more promising Web3 alternatives for day-to-day tasks that pledge to provide better experiences in terms of data security and speed such as: Brave for browsing Metamask to send money Opensea for buying and selling Mastodon to have more control over your data while still enjoying the benefits of social media. Filecoin to store and manage data rather than uploading to your Google Drive Data storage already has Web3 alternatives. Yeah! An anonymous person source says, " " Rightly so! Whoever controls the data, controls the truth. Web3 is giving more control to individuals so can have a stake in decision-making, and that's how life should be! Data is a valuable asset at times termed as especially in this present era where data is used to improve user experience and help businesses make decisions that lead to growth. you gold, The principles of centralized data management nullify the need for democracy and transparency when dealing with data in the Web2 space. Centralization is not bad We have all benefited from personalized services like having Amazon suggest what you might like based on the last thing you searched for or even bought. Web3 is a new layer of the internet that largely leans on the efforts of centralization to make data more accessible and secure. Decentralization doesn't always mean private Like many newbies in this ecosystem, it's easy to take on the thought that decentralization guarantees privacy. No, it doesn't. " ." Many decentralized applications (dApps) and smart contracts built on decentralized networks like Ethereum collect and store data in transparent, publicly accessible databases Too Many Broken Trusts Many times we have been forced to trust like Facebook with our data, and oftentimes they keep breaking it. tech lords Take the in March 2018 where millions of Facebook users had their data collected without their consent for political advertising reasons. Facebook Cambridge Analytica scandal It was quite easy for Cambridge Analytica, a British political consulting firm, to obtain tons of people’s data through a third-party app - . Your Digital Life The app helped the firm obtain personal details of individuals who signed up as well as details of their friends on Facebook all without their consent. After that incident, which had over 87 million users' data , it became more glaring to the public that Facebook owns and controls the data of everyone on its platform. Till now, users have little visibility on how accessible their personal data is. hijacked Web3 exists to prevent such scandals by giving you the power to control your data. It also promises you that your data is safe by eliminating third-party centralized networks for data storage which are vulnerable to attacks. Talking about centralized networks/servers, I think this is the best time to bring in the discussion on the synergy between big data, data engineering, and centralized servers. Data Engineering & How It Plays Out in Web2 Data engineering refers to the process involved in building an infrastructure with tools such as , , and to collect and transform big data for the benefit of business use which will result in growth. Apache Spark Hadoop Cassandra Big data, on the other hand, refers to large amounts of data- structured, unstructured, and in some cases, semi-structured, generated daily through the internet by individuals and businesses. The importance of data engineering cannot be overlooked- tons of data (big data) are constantly collected, harnessed, and processed through pipelines to provide valuable business insights and decisions. Now, a major limitation of this is that data engineering relies on centralized systems for data storage hence, is prone to a single point of failure - centralized networks have data processed and stored by a single node; in a case where the primary server crashes, there would be a general downtime which could lead to data loss. Also, individuals/businesses have little or no control over their data, and virtually every business now employs centralized servers. What do they say about times and seasons? Well… Here are some limitations of centralized cloud computing: Users' data can be stored and used without their consent. In the place of Google, personal data such as search history, personal preferences, and also location can be arbitrarily used. Data is prone to hacking by third parties just like the Facebook scandal. The operations will be slower leading to stifled scalability- following the dependency on primary servers(less division of labor). Servers are vulnerable to a single point of failure as explained above. Is that the path data management should continue in? I don't think so. There seems to be a way out -thankfully, with the "decentralized mandate" on Web3, multiple nodes can process and store large amounts of data. Now, let me help you understand this. Decentralized storage protocols can be likened to keeping your eggs safe in different baskets- all in the same room each stored in a well secure basket that can only be accessed with permission (on-chain). Centralized storage servers can be likened to keeping all your eggs in one basket relatively safe. The safety of the eggs can be easily tampered with when it's in one basket. That's how vulnerable centralized servers are. Notice that I highlighted that, in the first case scenario, these eggs are in the same room, so there's still a connection between them - that explains the interconnection between decentralized nodes. Just like the eggs in scenario 1, all placed in different baskets interconnected nodes or blocks are more secure and easy to detect when something goes wrong. But then, there's still a problem. Navigating the Early Stage of Innovation Illuminates a Myriad of Challenges The emergence of true data sovereignty in a decentralized environment has uncovered many holes that should be filled. At this current stage where Web2 is still fully in existence, many applications utilize centralized servers including Web3 dApps because there aren't yet fully equipped decentralized storage protocols to handle massive data . yet Web3 developers crave a more sophisticated decentralized database environment following the massive and complex nature of data generated in the ecosystem. Let's face it; with millions of transactions and blocks added daily, developers need to access and query data in real-time. Speed is of necessity for dApps to scale efficiently. Here are some of the problems faced by developers: Some existing decentralized storage protocols like and are limited to the storage of only static files. Arweave IPFS The abundance of Web3 applications means more complex data to deal with like NFT data, tokens, wallets, etc. Following the dynamic nature of Web3 data, these distributed protocols make it difficult for developers to easily query data, leaving most of the data stored in silos unused. Also, such a rigid storage environment slows down deployment. These vast amounts of data lying fallow in silos, which are not maximized are kept to waste because they can't be easily accessed when needed - in real-time. This affects the application lifecycle. : The time and expense required to build a robust database infrastructure to tally with the demands of Web3 complexities will give Web3 developers a second thought to continue with centralized databases which they are even more familiar with. It is a complex process to build from scratch, and there is scarce time to do that while trying to scale. Cost implications One of the pillars of a Web3 environment is . Developers need to trust that they can get reliable data from the emerging decentralized storage infrastructures available. trust Using blockchain specifically to store data takes a lot of time and isn't any different from using centralized servers; transactions are slower and take longer time to validate. In a bid to solve this, developers might resort to building a decentralized on-chain database. While this would be more effective, it is an extremely complex task to get started with. Who Owns the Data in Web3? Various Web3 projects may have distinct approaches to data ownership. However, ownership of data is distributed across nodes or computers to enable transparency, censorship resistance, and immutability. This distribution is carried out by decentralized systems including blockchain technology. Individuals have full control of their data and can decide which data they want to share. How Is Data Stored in Web3? Data storage in Web3 is usually carried out with decentralized technologies like the blockchain, distributed storage networks, and decentralized file systems like (InterPlanetary File System) Now we will take them in bits so you can understand: : A strong force to be reckoned with in the Web3 ecosystem that allows you to store data in blocks and link them securely through cryptography. (Remember the definition above) Blockchain It is usually used to store transactional data through an immutable chain, hence its name- blockchain. :  This decentralized storage type leverages the collective storage ownership of participants in the network and incentivizes them to share more of their unused storage space. The storage space is then broken into bits and distributed across multiple nodes. Distributed storage networks : A peer-to-peer distributed storage environment that leverages on content addressable model: a way to recognize files by their cryptographic hash. It helps in the storage and retrieval of files. Decentralized File Systems (IPFS) Which Web3 databases Are Available? As Web3 is buzzing, many developers have discovered a missing tech stack in the ecosystem, a database for Web3; below are a few emerging databases you can use: Polybase Glacier Ceramic WeaveDB IceFireDB Kwil Space and Time (Web3 data warehouse approach) Thus Far… Some Web3 databases have emerged to solve major pain points that Web3 developers face to scale their dApps efficiently. From the likes of a handful of examples listed above amongst others, their approach in ways is similar by providing: : A better interconnected organized space for Web3 data that allows for easy access and allows developers to build on the infrastructure seamlessly. Programmable composable data network :  With traditional databases rigid on structured data i.e., having rows, tables, and columns, it eliminates the chances to maximize Web3 data which comes in more complex forms. NoSQL databases will help handle Web3 data in any form with more flexibility and efficiency. Using the NoSQL database structure, different types of data can be stored without necessarily fitting into a predefined schema. NoSQL database : Just like building blocks, different parts of the environment can be easily put together or taken out to meet needs per time- it's customizable and dynamic in nature. Modularity : A robust tool that meets the heavy data demands of massive Web3 data without getting overwhelmed or reducing efficiency. Scalable infrastructure : This is one word you may struggle to pronounce but will regularly come across when talking about Web3 applications. I'll try to help you understand it in the best way possible 🙏 Interoperability Imagine that you have a flash drive that can easily connect to 3 computers. That drive is able to share information easily with each of the computers. That's how it works too in Web3- various dApps can share information with each other and work together even if the applications were built by different people. It's more like a meeting point of agreement despite the uniqueness of the applications. Emerging Web3 databases are receptive to allowing various dApps to interact and collaborate seamlessly on their platform. Business applications can interact without intermediaries. Glacier particularly, intends to with the ownership of Web3 data to bring to reality the monetization of Web3 data subsequently. Data monetization: combine NFT technology Exciting Times Ahead for Data Engineers As Web3 gradually encroaches into the way everything on the internet is done, data engineering will also experience interesting steps towards decentralization. Data engineers, the core architects of data pipelines, will have to learn new ways to design and maintain decentralized environments that meet the demands of data sovereignty which covers security, integrity, and privacy through immutability. Also, businesses will garner more trust seeing that their data is protected from unauthorized access, deletion, or manipulation. The smooth transition from Web2 to Web3 is also not leaving out Web2 applications, some of the databases listed above like IceFireDB, is providing a solution for Web2 and Web3 applications to access decentralized storage. Automation is a one-time button for repetitive tasks. Data engineers can employ smart contracts - programmable self-executing contracts to automate some data management practices. How Will Tech Giants React to the Full Unfolding of Web3? Like many things that trended within a particular time frame in the past, there came a time they also had to fade because they weren't able to successfully withstand change. That's the same way this will play out. Facebook, Amazon, and Google may not disappear entirely if they can adapt and implement the necessary changes to transition towards a more open and transparent internet. Facebook has gradually shown support for certain aspects of the Web3 sphere, such as the metaverse and NFTs, despite its to shut down its NFT platform for Facebook and its sister platform, Instagram. recent decision Everyone needs to learn constantly about the evolving direction of the internet because this next phase is really about and you do have a stake. you, Additionally, there would be a more transparent approach to handling consumer data as a business, ensuring direct consent through a decentralized web. It's truly an incredible journey to anticipate, albeit at a leisurely pace. Takeaway Just as you wouldn't be needing another person's phone to make calls, Web3 developers crave an environment for decentralized storage that fully exceeds expectations rather than blindfolding themselves to the many limitations of centralized facilities. Get excited about the new wave of the internet -Web3, where you are more involved and protected.

Who Will Eventually Control Big Data in Web3?

This all started with an email from a publicist offering to provide me with a sample of the in exchange for a review. The Kwilt Shoebox is a device designed to allow you to build a “personal cloud” — it connects to your smartphone and other devices via WiFi and allows you to offload files/photos to it in order to free up storage. Kwilt Shoebox Eventually I received the device. Upon opening the package, I was presented with some tasteful packaging and a series of adapters for the device. The Shoebox is designed to connect to USB storage mediums such as a flash drive or external hard drive. It only features a Micro-USB port for interfacing with peripherals, so a Micro-USB to USB-A adapter is included in the box. The Shoebox can also connect to a TV to display your photos; it comes equipped with a Mini-HDMI connector, so another adapter is included in the box. At this point something started to feel strange. For a device designed to connect directly to storage mediums and TVs, this choice of using adapters instead of just the correct connector was odd. The placement of the ports also felt a little weird — the power and USB port utilize the same connector, and are placed directly adjacent which could cause some confusion. The next thing that caught my eye was a large silver sticker not featured in promotional shots of the device. The sticker is plastered across the side of the Shoebox, not on the bottom like one might expect. Listed in very small print on the bottom of this sticker are the words “removing this label will void your warranty,” which is tech-speak for “something interesting is underneath here.” Naturally, I peeled it off. Seeing what was underneath the sticker made everything click into place. The green PCB, familiar layout of pins, and the word “Raspberry” peeking out from under the red case all immediately reveal that this device is in fact a Raspberry Pi Zero W. This is the first time I’ve ever seen a Raspberry Pi device make it into a consumer product without immediately revealing itself. I have definitely seen other RPi products before; for example, the 3D printer interface is based around the Zero’s bigger sibling. The difference here is that AstroPrint makes it very clear that the device is powered by a Raspberry Pi — it’s targeted towards makers and individuals that would likely want to modify the hardware. The Kwilt website, however, makes no mention of the Raspberry Pi anywhere. This makes sense considering the Shoebox’s targeted market of general consumers who wouldn’t have much of a desire to hack the device. AstroBox Using the Raspberry Pi as a platform is a smart decision on the part of Kwilt for numerous reasons. It has all the necessary capabilities to fulfill the goal of the company, and all of the Zero’s ports can be utilized for real purposes meaning there’s few wasted costs. Additionally, it’s likely that FCC and other licensing is much easier given that the Raspberry Pi Zero is already licensed. And, of course, there are no custom hardware fabrication and design costs. This allows Kwilt to focus their efforts on their well-done web app and mobile app that allows customers to interface with the device. Even is created by the Raspberry Pi foundation. I totally understand Kwilt’s approach to this project — they didn’t want to deal with creating a circuit, manufacturing that circuit, packaging the circuit, and more just to realize their initial concept. Kwilt can focus purely on creating a software interface for this project which is really heart of the whole system. No hardware experience necessary. the case that houses the device My next thought concerned the security of such a platform. I located the IP address of the shoebox and ran a scan via nmap to get networking information about the device. Sure enough, the Shoebox reveals to us that it’s running stock Raspian, and that port 22 is open for SSH like on a stock Raspberry Pi. Kwilt developers were smart enough to change the default SSH password away from “raspberry”, which would have granted easy access to the device. Next, I tried interrogating my poor shoebox for a few days using . Hydra is a password cracking tool that essentially tries to brute force its way into a device by trying different passwords over and over again. I made mine a little smarter by downloading a containing the most frequently used passwords. I got all the way from “123456” to “nachito” without finding a valid key, so the password securing SSH is at least relatively good. An analysis of all of the traffic coming into and out of the Shoebox reveals that all essential operations are performed via HTTPS, which is also good to see. Hydra 60Mb file All in all, the Shoebox appears to be a powerful device that well-utilizes the capabilities of the Raspberry Pi Zero W. The $60 price tag on the device is a significant markup over the $10 Zero W, but given that the company includes a bunch of adapters and a 32-Gb USB key and that they need to pay all their software engineers and marketers this makes sense. Obviously the Pi Zero platform is viable for consumer products, so I’m curious to see where other companies take this next.

The Rise of Raspberry Pi in Consumer Hardware

From Chat-Bots to Killer-Bots?

Yuan Yi Zhu, an Assistant Professor of International Relations and International Law at Leiden University, recently criticized an article published in The Times of London. The article was written by Matthew Parris, a former Member of the British Parliament. In the article, Parris argues in favor of legalizing euthanasia on the grounds that society simply cannot afford “desperate infirmity for as many such individuals as our society is producing.” Parris argues that it would allow the UK economy to better compete with China and people being coerced to being euthanised due to social pressure is “not a bad thing”. Parris says he does not apologise for how his article "treats human beings as units - in deficit or surplus to the collective". In February 2024, an article was published in the prestigious Scientific Reports journal by Nature, arguing that “AI holds the potential to carry out several major activities at much lower emission levels than can humans.” The piece argues that humans produce far greater CO2 emissions when doing writing or illustration work than AI Large Language Models. The article also notes that: “The freed human time may also incur new, unexpected environmental costs.” Advances in AI haven’t just led to humans being compared to machines but also being compared to animals. Late last year, according to The Guardian, BT’s Chief Innovation Officer, Harmeen Mehta ‘suggested workers whose jobs are threatened by AI accept their fate as “evolution,” comparing them to horses replaced by the car.’ Against this backdrop, one does wonder when we will inevitably see the rise of those who will argue for eugenics such that humans replace machines. This is nevertheless a stark reminder of how humans ultimately are behind killer computers. In a new book I authored called ‘ ,’ I have explored many catastrophic computer disasters. These disasters have included an aircraft entering ‘death dives’, fatal car crashes and lethal radiation doses. How to Protect Yourself from Killer Computers Recently, software reliability has been pushed to the forefront of the public conscience. March 2024 saw system failures, causing McDonald’s outlets in the UK, Australia, and Japan to go offline. Panama Bread in the US saw IT outages, as did the UK high-street giants Tesco, Sainsbury's, and Greggs. As several Boeing executives stood down from their jobs following the death of a whistleblower, it’s worth remembering how the first Boeing 737 MAX groundings began with a computer software problem. Previously unreported, it was also the case that March 2024 saw London police officers report chaos in jails and being unable to begin search warrant applications, as there was also an outage with the Metropolitan Police’s IT systems. The public inquiry into the UK Post Office’s Horizon IT Scandal is underway, a scandal where faulty accounting software resulted in miscarriages of justice, claiming around 700 victims and has been linked to four suicides. This included a pregnant mother being sent to jail. Engineers are now facing the consequences for being complicit in disasters; with FTX’s former Director of Engineering facing up to 75 years in prison, a VW engineer who was complicit in the emissions scandal having been sentenced to 40 months imprisonment despite co-operating with investigators and the architect of the Post Office Horizon IT system currently under investigation by UK police for perjury. In common across all these disasters, whilst the technical factors varied, I found a common theme of wrongdoing, retaliation, and cover-up. Issues were brushed under the carpet instead of addressed directly. For example, one engineering manager at Fujitsu, who built the software behind the Post Office scandal, told the public inquiry how he “was threatened with violence by an individual with a reputation for violence” during one of his early attempts to remedy the issues. This is an industry-wide problem. In November 2023, I led research that found that 53% of software engineers suspected wrongdoing at work, with 75% of software engineers in the UK having experienced retaliation the last time they reported it to their employers. This is a concern as computers do inevitably make mistakes as no software is perfect. Small things like a binary number being flipped (e.g., a “1” becoming a “0”) due to cosmic rays are enough to cause havoc in computer systems, particularly those that haven’t been designed with this risk in mind. There is, however, hope here. The research conducted in the new book finds that resilient systems can be built when humans are part of the system. In these , humans play a role in preventing disaster by raising the alarm and mitigating the issues. socio-technical systems For this to happen, humans must be free to raise the alarm to issues when they come across them. If the on-call engineers who keep our online services and critical national infrastructure running are not able to speak up and act to prevent disaster, these problems emerge. However, this applies to all software engineers and, indeed, everyone who is part of the software development process, including the customers. Killer computers are born in toxic cultures where engineers aren’t able to raise the alarm, and issues are brushed under the carpet. By making sure humans play a key role in the reliability of computer systems, we can help ensure technology serves society.

The Killer Humans Behind Killer Computers

Will a LinkedList be faster? Should I swap the `for each` with an `iterator`? Should this `ArrayList` be an `Array`? This article came to be in response to an optimization so malevolent it has permanently etched itself into my memory. Before going head-on into Java and the ways to tackle interference, either from the garbage collector or from context switching, let's first glance over the fundamentals of writing code for your future self. Premature optimization is the root of all evil. You've heard it before; premature optimization is the root of all evil. Well, sometimes. When writing software, I'm a firm believer in being: as ; you should try to narrate intentions as if you were writing a story. descriptive as possible as ; which means that you should know the fundamentals of the language and apply them accordingly. optimal as possible As Descriptive as Possible Your code should speak intention, and a lot of it pertains to the way you name methods and variables. int[10] array1;        // bad
int[10] numItems;      // better
int[10] backPackItems; // great Just by the variable name, you can already infer functionality. While is abstract, tells you a lot about expected behavior. numItems backPackItems Or say you have this method: List<Countries> visitedCountries() {
    if(noCountryVisitedYet)
        return new ArrayList<>(0);
    }
    // (...)
    return listOfVisitedCountries;
} As far as code goes, this looks more or less ok. Can we do better? We definitely can! List<Countries> visitedCountries() {
    if(noCountryVisitedYet)
        return Collections.emptyList();
    }
    // (...)
    return listOfVisitedCountries;
} Reading is much more descriptive than Collections.emptyList() new ArrayList<>(0); Imagine you're reading the above code for the first time and stumble on the that checks if the user has actually visited countries. Also, imagine this is buried in a lengthy class, reading is definitely more descriptive than , you're also making sure it's immutable making sure client code can't modify it. guard clause Collections.emptyList() new ArrayList<>(0) As Optimal as Possible Know your language, and use it accordingly. If you need a , there's no need to wrap it in a object. The same goes for using a if all you actually need is an . double Double List Array Know that you should concatenate Strings using or if you're sharing state between threads: StringBuilder StringBuffer // don't do this
String votesByCounty = "";
for (County county : counties) {
    votesByCounty += county.toString();
}

// do this instead
StringBuilder votesByCounty = new StringBuilder();
for (County county : counties) {
    votesByCounty.append(county.toString());
} Know how to index your database. Anticipate bottlenecks and cache accordingly. All the above are optimizations. They are the kind of optimizations that you should be aware of and implement as first citizens. How Do You Kill It First? I'll never forget about a hack I read a couple of years ago. Truth be said, the author backtracked quickly, but it goes to show how a lot of evil can spur from good intentions. // do not do this, ever!
int i = 0;
while (i<10000000) {
    // business logic
    
    if (i % 3000 == 0) { //prevent long gc
        try {
            Thread.sleep(0);
        } catch (Ignored e) { }
    }
} A garbage collector hack from hell! You can read more on why and how the above code works and, while the exploit is definitely interesting, this is one of those things you should ever do. in the original article never Works by side effects, has no purpose in this block Thread.sleep(0) Works by exploiting a deficiency of code downstream For anyone inheriting this code, it's obscure and magical Only start forging something a bit more involved if, after writing with all the , you've hit a bottleneck. But steer away from concoctions as the above. default optimizations the language provides How to Tackle Garbage Collector? That If after all's done, the Garbage Collector is still the piece that's offering resistance, these are some of the things you may try: If your service is so latency-sensitive that you can't allow for GC, run with . "Epsilon GC" and avoid GC altogether -XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC This will obviously grow your memory until you get an OOM exception, so either it's a short-lived scenario or your program is optimized not to create objects If your service is somewhat latency sensitive, but , run GC1 and feed it something like (default is 250ms) the allowed tolerance permits some leeway -XX:MaxGCPauseTimeMillis=100 , say one of them calls or which are stop-the-world garbage collectors, you can override offending behavior by running with If the issue spurs from external libraries System.gc() Runtime.getRuntime().gc() -XX:+DisableExplicitGC If you're running on a JVM above 11, do try the , performance improvements are monumental! . You may also want to check this . Z Garbage Collector (ZGC) -XX:+UnlockExperimentalVMOptions -XX:+UseZGC JDK 21 GC benchmark VERSION START VERSION END DEFAULT GC Java 1 Java 4 Serial Garbage Collector Java 5 Java 8 Parallel Garbage Collector Java 9 ongoing G1 Garbage Collector Note 1: since Java 15, is , but you still have to explicitly activate it with . ZGC production-ready -XX:+UseZGC Note 2: The VM considers machines as server-class if the VM detects more than two processors and a heap size larger or equal to 1792 MB. If not server-class, it . will default to the Serial GC In essence, opt for GC tuning when it's clear that the application's performance constraints are directly tied to garbage collection behavior and you have the necessary expertise to make informed adjustments. Otherwise, trust the JVM's default settings and focus on optimizing application-level code. - you'll want to read the u/shiphe full comment Other Relevant Libraries You May Want to Explore: Java Microbenchmark Harness (JMH) If you're out of feeling without any real benchmarking, you're doing yourself a disservice. JMH is the Java library to test your algorithms' performance. Use it. optimizing de facto Java-Thread-Affinity Pinning a process to a specific core may improve cache hits. It will depend on the underlying hardware and how your routine is dealing with data. Nonetheless, this library makes it so easy to implement that, if a CPU-intensive method is dragging you, you'll want to test it. LMAX Disruptor This is one of those libraries that, even if you don't need it, you'll want to study. The idea is to allow for ultra-low latency concurrency. But the way it's implemented, from to the brings a lot of new concepts. I still remember when I first discovered it, seven years ago, pulling an all-nighter to digest it. mechanical sympathy ring buffer, Netflix jvmquake The premise of is that when things go sideways with the JVM, you want it to die and not hang. A couple of years ago, I was running simulations on an HTCondor cluster that was on tight memory constraints, and sometimes, jobs would get stuck due to "out of memory" errors. jvmquake This library forces the JVM to die, allowing you to deal with the actual error. In this specific case, HTCondor would auto-re-schedule the job. Final Thoughts The code that made me write this post? I've written way worse. I still do. The best we can hope for is to continuously mess up less. I'm expecting to be disgruntled looking at my own code a few years down the road. And that's a good sign. You can find this and other articles at wasteofserver.com Edits & Thank You: to for catching the mutability error in and the detailed explanation. FlorianSchaetz visitedCountries() to and for explaining that , you get Serial GC u/brunocborges u/BikingSquirrel on lower-end machines to for taking the time to better explain when you should mess with the GC and u/shiphe when you shouldn't to for putting me on the right track regarding what practice u/tomwhoiscontrary should be considered standard to (again) for providing the link to the JDK 21 garbage collector benchmark u/BikingSquirrel Also published on wasteofserver.com

Software Thrives Unless You Kill it First: Premature Optimization and a Tale of Java GC

সফ্টওয়্যার উন্নতি লাভ করে যদি না আপনি এটিকে প্রথমে হত্যা করেন: অকাল অপ্টিমাইজেশান এবং জাভা জিসির একটি গল্প

Software gedeiht, wenn man sie nicht vorher tötet: Vorzeitige Optimierung und eine Geschichte von Java GC

El software prospera a menos que lo elimines primero: optimización prematura y una historia de Java GC

Le logiciel prospère à moins que vous ne le tuiez d’abord : optimisation prématurée et histoire de Java GC

सॉफ्टवेयर तब तक फलता-फूलता है जब तक आप उसे पहले खत्म नहीं कर देते: समयपूर्व अनुकूलन और जावा जीसी की कहानी

ソフトウェアは、最初に殺さない限り繁栄する: 時期尚早な最適化と Java GC の物語

소프트웨어는 먼저 죽이지 않으면 성공한다: 성급한 최적화와 Java GC 이야기

O software prospera a menos que você o elimine primeiro: otimização prematura e uma história de Java GC

Программное обеспечение процветает, если вы не убьете его первым: преждевременная оптимизация и рассказ о Java GC

Yazılım, Siz Önce Onu Öldürmediğiniz Sürece Büyür: Erken Optimizasyon ve Bir Java GC Hikayesi

Phần mềm phát triển mạnh trừ khi bạn tiêu diệt nó trước: Tối ưu hóa sớm và câu chuyện về Java GC

软件蓬勃发展，除非你先杀死它：过早优化和 Java GC 的故事

I’m trying to spend more time in Blazor, especially as I build out my next side project. Historically, I’ve spent a lot of time writing WinForms and WPF applications, but over the past few years, I’ve been mostly removed from any user interface development. With the rise of Blazor, I feel like I have another opportunity to jump back into things! As I dive a bit deeper, I’ve found MudBlazor has a great set of UI components — especially as I started building using the MudBlazor list items! In this article, I’ll share a quick overview of some of the ways that you can use MudBlazor list items, which will focus on MudList and MudListItem components. Check out the quick code tips and see if MudBlazor might be a fit for you! Overview of MudBlazor and List Components is a popular UI component library for Blazor development that provides a rich set of ready-to-use components. It simplifies the process of creating responsive and customizable user interfaces in Blazor applications. MudBlazor Coming from a WinForms and WPF background, I find I need to spend additional time getting familiar with the Blazor component libraries we have access to. Once you get a level of comfort, creating more powerful UIs feels a lot less restrictive! MudList is a key component in MudBlazor that allows us to create lists in our Blazor applications. MudList provides a flexible and configurable way to render list items. MudListItems are the individual items within a MudList. They allow for customization and configuration, enabling us to define the appearance and behavior of each list item according to our specific requirements. We’ll see some code examples coming up that allow us to customize the look and feel of MudListItems to get things the way we want! <MudList>
    <MudListItem>Item 1</MudListItem>
    <MudListItem>Item 2</MudListItem>
    <MudListItem>Item 3</MudListItem>
    <MudListItem>Item 4</MudListItem>
</MudList> Using MudBlazor’s MudList and MudListItems components, we can create dynamic and interactive list views in Blazor. MudBlazor takes care of the complexities of handling the rendering and responsiveness, allowing us to focus on the important stuff — building good Blazor apps. Configuring List Items in MudBlazor MudBlazor provides a powerful set of components for configuring list items, allowing you to customize their appearance and behavior to meet your specific needs. In this section, we’ll check these out by seeing how to add icons, text, and additional components to enhance your list view in Blazor. Adding Icons to MudBlazor List Items MudBlazor offers a wide range of icons that you can easily integrate into your list items. To add an icon, use the property of the component and provide the name of the desired icon. For example: Icon MudListItem <MudListItem Icon="Icons.Filled.CheckCircle" Text="Completed" /> I haven’t yet had a need for icons, but I know that when it comes to adding polish to a UI it can be really helpful. Especially when it comes to making functionality easier to identify and more accessible. Customizing Text and Content: You can customize the text displayed in a list item by setting the property. Additionally, MudBlazor allows you to include additional content within a list item using the property. For instance: Text ChildContent <MudListItem>
     <MudIcon Icon="Icons.Filled.Speaker" />
     <span>Notification</span>
     <MudBadge>5</MudBadge>
</MudListItem> In the above example, the list item contains an icon, a text span, and a badge component. This allows for a more visually appealing and informative list item. You can continue to nest all sorts of things in here! For one of my more recent UI screens, I’m working on a more complex item layout that requires a grid and multiple components within the MudItem. Responding to Events: MudBlazor provides event handlers that allow you to respond to user interactions with list items. For instance, you can handle the event to perform an action when a list item is clicked: OnClick <MudListItem OnClick="e => OnListItemClick(e)">
     <MudIcon Icon="Icons.Filled.Speaker" />
     <span>Notification</span>
     <MudBadge>5</MudBadge>
</MudListItem> In the above example, the method in your code-behind or component will be called when the list item is clicked. Here we can see the code for the handler: OnListItemClick @code {
    private async Task OnListItemClick(MouseEventArgs e)
    {
        // Handle the click event here
        Console.WriteLine("ListItem clicked!");

        // You can also implement other logic here, 
        // such as navigation or updating UI elements.

        // And... we can use async await!
        await Task.Yield();
    }
} Writing async event handlers this way is a nice touch compared to some of . And even though this is much nicer, I’ll have a follow-up article where I had problems getting this to fire because of some RenderMode issues conflicting with my paradigm! the nastiness we’re required to do traditionally with async event handlers dependency injection Wrapping Up MudBlazor List Items As I spend more time working in Blazor, I’m trying to build out more familiarity so I can be as proficient as I used to be with WPF and WinForms. That’s going to take some time. I think it’s important to get familiar with the awesome control libraries we have access to, and MudBlazor is a great fit so far. It’s extremely common for us as application developers to need to show data in lists, and we need to do so in a way that’s obvious to users. MudBlazor has the MudList with MudListItems, and this is a highly configurable way for us to do *just* that. If you found this useful and you’re looking for more learning opportunities, consider and check out my ! subscribing to my free weekly software engineering newsletter free videos on YouTube

KWIL

kwil.com

-- employees

Since n.d.

COMPANY RANKING

KWIL

EVERGREEN INDEX #6117

Representing this company? Click here to claim and customize this page!

Kwil WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

KWIL

kwil.com

-- employees

Since n.d.

COMPANY RANKING

KWIL

EVERGREEN INDEX #6117

RELATED COMPANIESRANK

STORIES ABOUT KWIL

Representing this company? Click here to claim and customize this page!

Kwil WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!