Federato's RiskOps underwriting platform for insurance carriers, MGAs and MGUs provides real-time risk selection and portfolio insights to empower underwriters to balance their book of business and make better underwriting decisions.

Federato Technologies

The Future of Social Networking: User Empowerment, Privacy, and Freedom From Censorship

This blog is the first in a series introducing Alluxio as the data platform to unify data silos across heterogeneous environments. The next blog will include insights from PrestoDB committer Beinan Wang to uncover the value for analytics use cases, specifically with PrestoDB as the compute engine. Alluxio started as a virtual distributed file system, a research project out of the AMPLab at U.C. Berkeley. Alluxio foresaw the need for agility when accessing large data stores separated from compute engines like Hadoop or Spark. Fast forward several years and over a thousand committers later, and Alluxio has blossomed into the industry’s leading platform for analytics and AI/ML. But as with any new type of technology, figuring out the best ways to use it depends on your data environment, computational workloads, issues, and goals. data orchestration So, after working with hundreds of organizations and fielding numerous inquiries, we thought it would be good to offer up a fresh take on the most popular use cases for Alluxio data orchestration. Data Orchestration: Bringing Data Closer to Compute The ability to quickly and easily access data and extract insights is increasingly important to any organization. With the explosion of data sources, the trends of cloud migration, and the fragmentation of technology stacks and vendors, there has been a huge demand for data infrastructure to achieve agility, cost-effectiveness, and desired performance. To meet the demand, provides a data abstraction layer that bridges computation frameworks and underlying storage systems. Alluxio is the only platform in the market today that brings together data locality with a distributed cache, API compatibility with popular data analytics and AI/ML frameworks, and a global namespace to federate data silos. Alluxio Alluxio is designed to help any framework access any data from any storage at high performance regardless of your environment, which enables your organization to remain agile and competitive in adopting and experimenting with new and existing technologies. The first use case is to run data-intensive workloads in a single cloud. Although more organizations are choosing cloud object storage for cost-effectiveness, scalability, and ease of use, they can hardly enjoy the same capabilities as a file system like HDFS. Thousands of companies leverage Alluxio at a large scale in production for modern data services. We have seen five common use cases to begin. With this overview, you will be able to tie this to your architecture and workloads and start using Alluxio. The first two use cases are single cloud or solely on-prem, while the following three use cases are for hybrid cloud environments. Cloud: achieve consistent SLAs and cost savings in a single cloud On-prem: speed-up analytics on on-prem object stores Hybrid cloud with compute in the cloud, data on-prem: burst compute to a public cloud and gradually migrate Hybrid cloud with data in the cloud and compute on-prem: gateway to utilize on-prem compute for data in the cloud Multi-datacenter: cross datacenter access without changing ingest pipeline across regions Use Case 1: Achieve Consistent SLAs and Cost Savings in a Single Cloud The first use case is to run data-intensive workloads in a single cloud. Although more organizations are choosing cloud object storage for cost-effectiveness, scalability, and ease of use, they can hardly enjoy the same capabilities as a file system like HDFS. For example, if you are running TensorFlow on top of AWS S3, one of the biggest challenges is the data access performance mismatch between the training frameworks on GPUs and cloud storage. Also, metadata operations are inefficient and expensive because of a pricing model based on the amount of access. Furthermore, when you are using ephemeral instances, the embedded caching solutions are not valid. As a data orchestration platform, Alluxio addresses these challenges by providing intelligent caching and metadata management. When deploying Alluxio with the compute node, you can see significant performance gains while accessing data ( , ) and cost savings ( ). 2-7x for analytics engines 1.7-2.3x for model training cut storage access cost up to 50% Alluxio not only handles the caching but also manages the corresponding metadata to avoid particular inefficient metadata operations and expensive metadata service costs. In addition, off-cluster caching for on-demand ephemeral workloads further makes consistent performance possible. Overall, with Alluxio as a data orchestration platform, you can achieve consistent SLAs and cost savings on cloud object stores. Alluxio can integrate with major cloud object stores, including AWS S3, Azure Blob Store, Google Cloud Storage. Electronic Arts, a leading company in the gaming industry, used Alluxio to achieve a high-performance platform on AWS to support real-time gaming services. Check out the case study . here Use Case 2: Speed-up Analytics on On-prem Object Stores The second use case is to run data-driven applications on top of an object store deployed on-premise. Similarly, it is hard to achieve the desired performance for analytics and AI, and metadata operations lack performance. In addition, on-prem object storage does not provide as many native APIs for popular frameworks as cloud object stores do. As a result, applications have to use less optimal connectors to access on-prem object stores. Alluxio, as a data orchestration platform, can improve the performance and handle the API translation, which will make onboarding to on-prem object storage seamless with a familiar security model. To access the underlying object storage, the applications only need to talk to Alluxio using conventional  APIs, and Alluxio handles the API translations behind the scenes. Thus, you can migrate workloads to the object store at your own pace without changing applications that affect the end-user experience. Furthermore, because Alluxio brings data closer to the compute frameworks, it will dramatically speed up Spark, Presto, Tensorflow, and other analytics frameworks on your object store with the flexibility of segregated storage. Alluxio supports any commercial object storage platform, including Ceph, MINIO, Cleversafe, Dell EMC ECS, and more. DBS, a top bank in Singapore, uses Alluxio to orchestrate data locally and makes numerous data applications accessible to the object storage. Check out the presentation . here If you are working in a large organization, you are likely to face more complex data infrastructure in a hybrid cloud environment with disaggregated compute and storage over a network connection. Use Case 3: Burst Compute to a Public Cloud and Gradually Migrate The third use case is to leverage the flexible compute resources on the public cloud so that you can migrate analytics or AI to the cloud and leave your data on-prem. This migration is not easy because of the limitation of network latency and bandwidth while running analytic workloads in the cloud with the data on-prem. The common solution is copying data into the cloud environment and maintaining the duplicate data on-prem (also called “lift and shift”). However, this solution is not only error-prone and also leads to issues in data consistency. Furthermore, some organizations may face compliance and data sovereignty requirements, preventing them from copying data into the cloud. The performance concern makes it difficult to progress with cloud migration. Alluxio tackles these challenges by providing the “Zero-Copy” burst feature – this is one of the most powerful features that organizations benefit from Alluxio on their journey to the cloud. Zero-copy burst enables compute engines in the cloud to access data on-prem without a persistent data copy or synchronization. You can scale analytics workloads with no change in the end-user experience or the security model. In terms of performance, by bringing the data close to data analytics and machine learning applications on-demand, the performance is much better than directly accessing the remote data ( ). Also, the on-prem data stores will have offloaded the computation and minimized the additional I/O overhead. an average of 3x improvement In summary, Alluxio, as a data orchestration platform in the middle, handles data movement for high efficiency so that workloads can be migrated to the public cloud on-demand without moving data between computing environments first. Alluxio takes charge of connecting to multiple stores so that you can gradually migrate the data using Alluxio to smooth your cloud migration journey. Walmart leverages Alluxio for this use case. They used Presto in the cloud to handle more capacity for data analytics and offload previously on-prem Hive workloads, which wouldn’t be as seamless, high-performing, and cost-effective without Alluxio’s zero-copy burst. Check out Walmart’s presentation . here Use Case 4: Hybrid Cloud Gateway to Utilize On-prem Compute for Data in the Cloud The fourth use case is to access cloud storage from a private data center, which is also a hybrid cloud architecture with decoupled object storage systems and compute nodes. Since the data is remote to the compute nodes, when running analytics directly on the object store, you will have to repeatedly pull the data from the storage nodes over the network, leading to inadequate performance and a high network egress cost. Alluxio remedies these performance and cost issues by acting as a hybrid cloud storage gateway that utilizes on-prem compute for data in the cloud. When deployed with the compute on-prem, Alluxio manages the compute cluster’s storage and provides data locality to the applications. Alluxio also helps by caching data, which avoids repeatedly fetching data directly from the cloud storage. Now you can keep the single source of truth in the cloud without sacrificing good performance or suffering from high egress costs by using Alluxio to fully leverage your existing compute-side resource deployments. reduce a large amount of network egress costs Comcast uses Alluxio to logically decouple compute and storage to enable rapid innovation in the storage and application layers. Check out the presentation . here Use Case 5: Cross Datacenter Access Without Changing Ingest Pipeline Across Regions The fifth use case is in multiple data centers where compute is in one data center, and the primary data lake is in another data center. Many organizations maintain satellite compute clusters that are independent of their main data cluster for the purposes of performance, security, or resource isolation. These satellite clusters need to access data remotely from the main cluster, which would become overloaded and dramatically impact the performance of existing workloads. Alluxio’s data orchestration platform makes the multi-datacenter architecture more manageable. When deployed on the compute nodes in the satellite cluster and configured to connect to the main data cluster, Alluxio serves as one logical copy of data that provides the unified namespace as the main data cluster. Alluxio can accelerate the remote data read from the main data cluster without adding extra ETL steps. Intelligent data caching eliminates the performance bottlenecks, decreasing the overall load of the main data cluster. Thus, data platforms can be self-service across business units without changes to the applications. At a leading SaaS company, the data application team used Alluxio to improve performance by ~3x and achieve self-service when adding satellite clusters to offload their central storage cluster. Summary With Alluxio, you can finally realize the technical and business benefits no matter if you are all on-prem, in the cloud, multi-cloud, or in a hybrid environment. Alluxio can help you achieve better agility, cost-effectiveness, and desired performance, which leads to faster time to insight to make better-informed decisions that provide a crucial competitive advantage. This is the reason why thousands of organizations choose Alluxio as the data orchestration platform to modernize their data infrastructure. If you are interested in learning more about how Alluxio can help solve your challenges, and . You can also join our community to engage with our users or developers. And, of course, you can download a copy of the free or a trial of the full to discover the benefits of data orchestration for your own use cases. contact us request a demo today slack channel Alluxio Community Edition Alluxio Enterprise Edition Also Published Here

How to Unify Silos With Data Orchestration

If 2020 has proven anything, it’s that predicting the future is a mug’s game. With that said, here are my predictions for where front-end development is heading over the next 12 months 🙄 React frameworks finally mature Frequently referred to as the ‘V’ in MVC, has both benefited and suffered from its lack of an opinionated framework. Years on and we’ve not seen the industry coalesce on a single stack, architecture or blessed toolchain. While this flexibility has undoubtedly aided its popularity, 2021 may be the year we finally see such frameworks reach maturity. React Over the last year has continued to dominate the React space, growing their framework, and their self-titled service offering — all while launching an , an , and hosting their largest (virtual) conference yet. While Next has been on the scene for a long time, it’s come a long way from its buggy beginnings. The batteries-included approach isn’t for everyone, but additions over the past year have culminated in a that’s both stable and surprisingly mature (sub-path routing at last 🙏). Vercel Next.js e-commerce starter kit analytics tool v10 release - Vercel was trying hard to channel Apple with their latest Keynote, complete with product ‘drops’ and all-black attire. Adding to the mix this year was the quiet ‘supporter preview’ launch of , a new closed-source React framework. Created by two of the biggest names in the community, Jackson and Florence of fame, Remix takes a different approach from Next — both in business model (charging a modest sum for an annual license fee) and philosophy (Remix is a product, not a platform). Instead of leaning into static page generation, Remix promises to leverage more traditional techniques, such as aggressive HTTP caching, and refocus on . Remix React Router web fundamentals It’s early days — Remix isn’t due to officially launch until next year — and we’re unlikely to ever approach the cohesive vision that an all-in-one tool such as enjoys. Nevertheless, it’s refreshing to see developments in what has long been a stagnant space. Ember We get a glimpse at container queries The desire for container queries has reached fever pitch. With Google finally announcing their , 2021 might be the year we get our first look at them. intent to prototype In the meantime, the increased adoption of combined with APIs such as CSS’s min/max/clamp and JavasScript’s ResiveObserver provide ever better methods of ‘faking it’. These kind of implementations have gotten increasingly (maybe even dangerously?) elaborate over the last few months, and it’s hard to imagine that this trend will stop next year. intrinsic web design I’m hopeful we’ll see more smart ways of approaching container-first design, such as Mathias’s and Heydon’s . Raven Watched Box WASM explodes The adoption of (WASM) has been slow, in no small part due to its positioning as a specialist tool (‘you’ll know if you need it’). However, in 2021 we’re likely to see that mindset shift. WebAssembly With the role of the front-end developer the domain of ‘full-stack’, we’re starting to see more engineers embrace lower-level systems languages such as , which is currently enjoying an unparalleled boom in , despite the initial uncertainty following the Mozilla layoffs. creeping towards Rust interest As a consequence, we’re seeing WASM-based solutions such as and enter the front-end framework wars. Benefiting from the performance and that Rust provides, it’s likely that 2021 will be the year we start to see a massive boom in production WASM. Yew Seed.rs safety We’re a long way off from any of these taking the spotlight away from the big players, but I’m confident adoption will spike in 2021. The monolith makes a come back The industry’s love of distributed services and carefully partitioned responsibilities continues, having evolved past the back-end ( ) and hit its unfortunate peak in 2019 with the introduction of (equally maligned and loved) micro-frontends. The UNIX-philosophy approach to architecture continues to rein — at least in the domain of tech reports and Medium articles. micro-services Over the last few years, the widespread use of micro-services has necessitated patterns such as (BFF), which introduces an intermediary between each client and the APIs which service it. Backend For Frontend - The BFF acts as the ‘glue’ that is the client’s single point of contact. This allows for it to be tightly coupled (and very well suited) to the client it services. This is particularly relevant in 2021 given the current interest in the , which, at its most complex, of its service . The answer to ‘how do I federate my APIs’ will be, amusingly: build one. JAMStack suffers under the weight dependencies We’re sure to see the adoption of the BFF technique increase in 2021, but — with a little luck, we might also see the gradual return to the monolith — or at least a slow retreat in the form of larger ‘ ‘ or compromises such as the . macro-services citadel No doubt Basecamp’s DHH, the most of the ‘majestic’ monolith (who recently launched yet another proudly traditional web-app, ) will be leading the charge. vocal proponent Hey In summary Interestingly, the biggest developments in the front-end are unlikely to be traditionally concerns. Back in our forecast, we noted that the role of the front-end developer was increasingly shifting towards ‘full-stack’, and this has borne out to be true. Even the evolution of our frameworks and tools suggests this, with an increased focus on data-fetching, concurrency, security and scalability. front-end 2019 There is a natural meeting point occurring, where engineers are faced with increasingly ‘back-end’ problems while tooling and services are becoming sophisticated enough to lower the barrier to entry. It’s a perfect storm, and I expect to see it continue throughout 2021. Previously published at https://www.browserlondon.com/blog/2021/01/04/front-end-predictions-2021/

Predicting The State of Front End Development: 2021 Edition

A React HOC for AWS Cognito to Simplify User Authentication

Aug 26, 2018 See update 2018-08-29 One of the key tradeoffs in blockchain design is whether to build more functionality into base-layer blockchains themselves (“layer 1”), or to build it into protocols that live on top of the blockchain, and can be created and modified without changing the blockchain itself (“layer 2”). The tradeoff has so far shown itself most in the scaling debates, with block size increases (and ) on one side and layer-2 solutions like Plasma and channels on the other, and to some extent blockchain governance, with loss and theft recovery being solvable by either or generalizations thereof such as , or by layer-2 solutions such as . So which approach is ultimately better? Those who know me well, or have seen me , know that I will inevitably say “some of both”. However, in the longer term, I do think that as blockchains become more and more mature, layer 1 will necessarily stabilize, and layer 2 will take on more and more of the burden of ongoing innovation and change. sharding the DAO fork EIP 867 Reversible Ether (RETH) out myself as a dirty centrist There are several reasons why. The first is that layer 1 solutions require ongoing protocol change to happen at the base protocol layer, base layer protocol change requires governance, and . it has still not been shown that, in the long term, highly “activist” blockchain governance can continue without causing ongoing political uncertainty or collapsing into centralization To take an example from another sphere, consider Moxie Marlinspike’s . A document by a company defending its right to maintain control over an ecosystem it depends on for its key business should of course be viewed with massive grains of salt, but one can still benefit from the arguments. Quoting: defense of Signal’s centralized and non-federated nature One of the controversial things we did with Signal early on was to build it as an unfederated service. Nothing about any of the protocols we’ve developed requires centralization; it’s entirely possible to build a federated Signal Protocol-based messenger, but I no longer believe that it is possible to build a competitive federated messenger at all. And: Their retort was “that’s dumb, how far would the internet have gotten without interoperable protocols defined by 3rd parties?” I thought about it. We got to the first production version of IP, and have been trying for the past 20 years to switch to a second production version of IP with limited success. We got to HTTP version 1.1 in 1997, and have been stuck there until now. Likewise, SMTP, IRC, DNS, XMPP, are all similarly frozen in time circa the late 1990s. To answer his question, that’s how far the internet got. It got to the late 90s. That has taken us pretty far, but it’s undeniable that once you federate your protocol, it becomes very difficult to make changes. And right now, at the application level, things that stand still don’t fare very well in a world where the ecosystem is moving … So long as federation means stasis while centralization means movement, federated protocols are going to have trouble existing in a software climate that demands movement as it does today. At this point in time, and in the medium term going forward, it seems clear that decentralized application platforms, cryptocurrency payments, identity systems, reputation systems, decentralized exchange mechanisms, auctions, privacy solutions, programming languages that support privacy solutions, and most other interesting things that can be done on blockchains are spheres where there will continue to be significant and ongoing innovation. Decentralized application platforms often need continued reductions in confirmation time, payments need fast confirmations, low transaction costs, privacy, and many other built-in features, exchanges are appearing in many shapes and sizes including , , and more. Hence, “building in” any of these into a base layer blockchain would be a bad idea, as it would create a high level of governance overhead as the platform would have to continually discuss, implement and coordinate newly discovered technical improvements. For the same reason federated messengers have a hard time getting off the ground without re-centralizing, blockchains would also need to choose between adopting activist governance, with the perils that entails, and falling behind newly appearing alternatives. on-chain automated market makers frequent batch auctions combinatorial auctions Even Ethereum’s limited level of application-specific functionality, precompiles, has seen some of this effect. Less than a year ago, Ethereum adopted the Byzantium hard fork, including operations to facilitate needed for ring signatures, ZK-SNARKs and other applications, using the curve. Now, Zcash and other blockchains are moving toward , and Ethereum would need to fork again to catch up. In part to avoid having similar problems in the future, the Ethereum community is looking to upgrade the EVM to , a virtual machine that is sufficiently more efficient that there is far less need to incorporate application-specific precompiles. elliptic curve operations alt-bn128 BLS-12-381 E-WASM But there is also a second argument in favor of layer 2 solutions, one that does not depend on speed of anticipated technical development: . This is less easily visible in Ethereum 1.0-style blockchains, where there are certain models that are reasonably universal (eg. Ethereum’s account-based model is one). In blockchains, however, one type of question that does exist in Ethereum today crops up: how to do cross-shard transactions? That is, suppose that the blockchain state has regions A and B, where few or no nodes are processing both A and B. How does the system handle transactions that affect both A and B? sometimes there are inevitable tradeoffs, with no single globally optimal solution sharded not The involves asynchronous cross-shard communication, which is sufficient for transferring assets and some other applications, but insufficient for many others. Synchronous operations (eg. to solve the ) can be bolted on top with , but this requires multiple rounds of cross-shard interaction, leading to significant delays. We can solve these problems with a , but this comes with several tradeoffs: current answer train and hotel problem cross-shard yanking synchronous execution scheme The system cannot process more than one transaction for the same account per blockTransactions must declare in advance what shards and addresses they affectThere is a high risk of any given transaction failing (and still being required to pay fees!) if the transaction is only accepted in some of the shards that it affects but not others It seems very likely that a better scheme can be developed, but it would be more complex, and may well have limitations that this scheme does not. There are known results preventing perfection; at the very least, puts a hard limit on the ability of some applications and some types of interaction to process more transactions per second through parallelization. Amdahl’s law So how do we create an environment where better schemes can be tested and deployed? The answer is an idea that can be credited to Justin Drake: layer 2 execution engines. Users would be able to send assets into a “bridge contract”, which would calculate (using some indirect technique such as or ) state roots using some alternative set of rules for processing the blockchain (think of this as equivalent to layer-two “meta-protocols” like and on top of Bitcoin, except because of the bridge contract these protocols would be able to handle assets whose “base ledger” is defined on the underlying protocol), and which would process withdrawals if and only if the alternative ruleset generates a withdrawal request. interactive verification ZK-SNARKs Mastercoin/OMNI Counterparty tooltip Note that anyone can create a layer 2 execution engine at any time, different users can use different execution engines, and one can switch from one execution engine to any other, or to the base protocol, fairly quickly. The base blockchain no longer has to worry about being an optimal smart contract processing engine; it need only be a data availability layer with execution rules that are quasi-Turing-complete so that any layer 2 bridge contract can be built on top, and that allow basic operations to carry state between shards (in fact, only ETH transfers being fungible across shards is sufficient, but it takes very little effort to also allow cross-shard calls, so we may as well support them), but does not require complexity beyond that. Note also that layer 2 execution engines can have different state management rules than layer 1, eg. not having storage rent; anything goes, as it’s the responsibility of the users of that specific execution engine to make sure that it is sustainable, and if they fail to do so the consequences are contained to within the users of that particular execution engine. In the long run, layer 1 would not be actively competing on all of these improvements; it would simply provide a stable platform for the layer 2 innovation to happen on top. Even if execution engines are something that gets partially or fully moved to layer 2, consensus on data ordering and availability is still a highly generalizable and necessary function; to see how difficult layer 2 execution engines are without layer 1 scalable data availability consensus, the in research, and its of naturally extending to fully general purpose blockchains, for an example. And if people want to throw a hundred megabytes per second of data into a system where they need consensus on availability, then we need a hundred megabytes per second of data availability consensus. Does this mean that, say, sharding is a bad idea, and we should keep the blockchain size and state small so that even 10 year old computers can process everyone’s transactions? Absolutely not. see difficulties Plasma difficulty Additionally, layer 1 can still improve on reducing latency; if layer 1 is slow, the only strategy for achieving very low latency is , which often have high capital requirements and can be difficult to generalize. State channels will always beat layer 1 blockchains in latency as state channels require only a single network message, but in those cases where state channels do not work well, layer 1 blockchains can still come closer than they do today. state channels Hence, the other extreme position, that blockchain base layers can be truly absolutely minimal, and not bother with either a quasi-Turing-complete execution engine or scalability to beyond the capacity of a single node, is also clearly false; there is a certain minimal level of complexity that is required for base layers to be powerful enough for applications to build on top of them, and we have not yet reached that level. Additional complexity is needed, though it should be chosen very carefully to make sure that it is maximally general purpose, and not targeted toward specific applications or technologies that will go out of fashion in two years due to loss of interest or better alternatives. And even in the future base layers will need to continue to make some upgrades, especially if new technologies (eg. STARKs reaching higher levels of maturity) allow them to achieve stronger properties than they could before, though developers today can take care to make base layer platforms maximally forward-compatible with such potential improvements. So it will continue to be true that a balance between layer 1 and layer 2 improvements is needed to continue improving scalability, privacy and versatility, though layer 2 will continue to take up a larger and larger share of the innovation over time. Justin Drake pointed out to me another good reason why some features may be best implemented on layer 1: those features are public goods, and so could not be efficiently or reliably funded with feature-specific use fees, and hence are best paid for by subsidies paid out of issuance or burned transaction fees. One possible example of this is secure random number generation, and another is generation of zero knowledge proofs for more efficient client validation of correctness of various claims about blockchain contents or state. Update 2018.08.29:

Layer 1 Should Be Innovative in the Short Term but Less in the Long Term

TL;DR we shard writes and avoid conflicts using an increment offset on PKs each data center contains local write servers we use multi-source replication on read servers so they contain all the data from all data centers, partitioned by month At the beginning, things were simple We were using : one master for writing, and a few slaves for reading. Besides (more on that later), it worked fine. And then you scale. And then it fails! The master is a SPOF — “ ingle oint f ailure”. MySQL replication lag S P O F Whether the process failed, the VM crashed, or we had to do maintenance on the server, having only one “master” to write to meant that the service was down for minutes. And this was not acceptable. The first thing we implemented was : there was the “stats” cluster (one master + N slaves), the “calls” cluster, the “customers” cluster, and so on. This allowed easier maintenance of the servers, as we could work on one cluster without impacting the others. sharding based on the “type” of data But there was still a SPOF in each cluster: the master. Options To get rid of the master-slave architecture where the master is a SPOF, we needed a solution where we could write on any node, failing over to another node if the previous one was not available. For reading, we needed a solution where we could run queries against “federated” data (reading from every node). This is not as simple as it sounds. When working on distributed systems, you have to work with: onsistency C vailability A artition tolerance P Yes, the . CAP theorem What should happen if you update a row on a node, and at the same time you update the same row on another node? Data reconciliation is hard (and sometimes impossible). What if you have a network failure, and the data is not consistent between the nodes? Which one do you trust? We looked at many technologies. There was the ‘master-master’ topology. MySQL Cluster was a good candidate. There was also Galera cluster, tungsten replicator, MySQL proxy… (At the time, MySQL group replication was not production ready). We also looked at the NoSQL world. Cassandra, MongoDB, HBase… but: we had a strong knowledge of MySQL in the team we were confident with it, it was stable, we knew how to run it we wanted to change our code as little as possible New technologies are sexy for sure. But stability and maintenance are key when you run a business. Note: there are many solutions now, especially if you are “in the cloud” and starting from scratch. The solution we expose here works pretty well if you have an existing MySQL infrastructure, an existing code base, people that have experience with MySQL, and the need to scale without changing the whole storage layer. Our solution There are two kinds of data we work with: data: carriers, phone numbers, routing profiles, dialcodes, etc. The data changes rarely, we must keep it “forever” (besides mandatory removal in respect of GDPR and other regulations), and it grows very slowly over time. “permanent” data: calls, SMS, API requests, events, webhooks, logs. . Written once. May be updated within hours. “usage” Grows very rapidly For the permanent data, we use a master-master topology. Nothing fancy here. We always write on the same master. If it fails, or if we need to do maintenance, we switch to another master. The story here is for the usage data. We call this the because we split reads/writes. “DBRW” architecture How it works Our implementation is pretty simple: when we need to insert data, (*). If it is not available, or if the error lets us failover, we try another write server. Boom! we choose a write server randomly The SPOF is gone! Write servers contain only the data that has been written on them. Their data is replicated asynchronously on the read servers. Read servers federate data from all the write servers, using multi-source replication. We apply the same rule for connection than with the write servers: we choose one server randomly (*), and fail over if it is not available. (*) Our system is a bit more clever: it avoid servers that are in maintenance or marked down by the supervision, the “random” is weighted, and we always try servers in the same datacenter first. We and on write servers. We on read servers. INSERT UPDATE SELECT Write servers Data purged daily, they need a limited storage space only We use very fast SSDs to get the best performance No indexes besides the primary key (so inserting is very fast) A server can be shut down for maintenance, it does not affect the system Read servers Same schema + indexes + partitions by month Indexes allow fast search queries (but take space) Partitions let us move data between disks, or discard old data in milliseconds Increment offsets explained Read servers receive and “merge” data from all write servers, by leveraging multi-source replication. To avoid conflicts on primary keys, we use an increment + on the write servers. For example: increment offset server DBW1 (increment 20, offset 1): 1, 21, 41, 61, 81 server DBW2 (increment 20, offset 2): 2, 22, 42, 62, 82 server DBW3 (increment 20, offset 3): 3, 23, 43, 63, 83 This way, INSERTs are not subject to conflict. When we need to UPDATE, . Write servers retain data for just a few days, older data is purged. Fortunately, once written, the , or it is within hours. we need to do it on the server that has the data usage data is rarely updated Obviously, this architecture does NOT work for data that needs to be updated continuously over time! To know which server holds the data with the primary key, all we need to do is apply a modulo of the increment, and it will give us the offset, thus the server. Example : (DBW1) or (DBW2). 41 mod 20 = 1 62 mod 20 = 2 Advantages of this architecture Any server can be turned off without impacting the service Write servers are light, and extremely fast (no indexes) Works well on a multi data center topology: ➡️ Write servers can be placed “locally” in each data center ➡️ Read servers can be “remote” as the MySQL replication works well on a WAN Read servers contain all the data, so search queries work as expected Maintenance on read servers is quite easy with the partitions Scaling writes is easy Cons Read servers require a lot of space (“vertical scaling”) but partitions makes it easy The primary key of write servers replicated and aggregated on read servers is not contiguous, and is not in order, because it is incremented independently on each write server We have been using this architecture for years, and it has been working great so far. Implementation Implementing the read/write split can be tricky. In our case, most of our projects had a “datastore” layer, in which we could easily implement the split, and the . sharding The most important part is not to forget to when doing updates, or if/when replication lag is an issue. read your own writes Recommended reading is a good book, unfortunately not up to date with the latest version of MySQL High performance MySQL , although not MySQL specific, is a great read Designing data-intensive applications is truely awesome The Percona blog Originally published at blog.callr.tech on August 29, 2018.

Sharding writes with MySQL and increment offsets

Blockchain technology is like one of those rapid fires in the Amazon that have all the eyes up in curiosity and now, since most of us have already tried to learn from one source or another what Blockchain means; it is as simply coined as the collection of data records in the form of blocks which are eventually combined to form a chain of database which henceforth cannot be subjected to any change. Some of the major user applications of blockchain platforms can be mentioned as their high utility in multiparty financial transactions and organized supply chains. They are also highly popular for their facilitation in the decentralization of business segments and processes. Blockchain technology is the contrivance of a group of people which goes by the pseudonym of Satoshi Nakamoto and was first made known in the year 2008 in a paper titled “Bitcoin: A Peer-to-Peer Electronic Cash System”. In the years that have followed, the undeniable growth of the crypto currencies is unmistakable. The Blockchain System, as high as they are evolving, provides not only dimensional scope for businesses to expand but also aid them to base on systems which are innovative and revolutionary. These blockchain systems or distribution ledgers can be placed in hierarchical orders in terms of their popularity and applications. In the last ten years, the Blockchain platforms are in a consistent state of a growth spurt and the ones that need you watching out are- #1. ETHEREUM Ethereum was founded in hopes to instrument a thoroughly trust less contract ledger generic in nature by Russian-Canadian VitalkButerin in 2014. Ethereum is a highly active and an open Blockchain platform that has been quite prominent owing to its potent functionality and the ability that it provides anyone to have an access to decentralized applications that function through the Blockchain technology. Ethereum USP’s ⦁ Vigorously ensued in Github ⦁ Open for public use ⦁ Multiple language applications such as Python, C++ ⦁ Proof-of-work based #2. HYPERLEDGER FABRIC Introduced first in 2016, Hyperledger Fabric is made known by the Linux Foundation. Its popularity is high due to the fact that it is one of the finest open-source synergetic intentions to boost the blockchain technologies of multiple industries. It backs the application of numerous networks with each admonishing various Assets and Agreements of different Member Nodes. Hyperledger USP’s ⦁ 185+ collaborating enterprises ⦁ Modular architecture ⦁ Read-only history inquiries ⦁ Production ready for enterprises #3. IBM BLOCKCHAIN The industry giant IBM opened spaces to the future by introducing its own Blockchain platform which garnered a lot of positive reviews for being the progressive and easy to use ledger for transactions. IBM boasts of a better consensus mechanism which has only helped it to gain much popularity. IBM USP’s ⦁ Permissioned network ⦁ Paid plans for upgraded application ⦁ Supported languages: Go, JAVA #4. MULTICHAIN Established upon the bitcoin’s blockchain, MultiChain is an open-source blockchain used to create permissioned blockchains within or amidst multiple enterprises. The huge popularity of this blockchain platform is derived by its motive to create much secure and private network in its enhanced versions. Multichain USP’s ⦁ Private permissioned network ⦁ Free and open source pricing ⦁ Actively updated in Github ⦁ Quick deployment ⦁ Supported Languages: Python, C#, C++, JavaScript #5. HYDRACHAIN This blockchain platform is a collaborative progressive initiative venture of Ethereum project and brainbot technologies. Being an extension of ethereum system, it is used to create private ledgers for organizational applications. The system is not much popular but is actively updated in Github. HydraChain USP’s ⦁ Open Source Pricing ⦁ Adherence to Ethereum Protocol ⦁ Permissioned network ⦁ Supported Language: Python #6. BIGCHAINDB Quite like the ones mentioned above, BigChainDB is also an open source blockchain technology platform that has each write documented in the database refraining the requirement of the sidechains. BigChainDB USP’s ⦁ Public and Private network ⦁ Custom Assets and transactions ⦁ No native currency #7. OPENCHAIN As the name is indicative enough, OpenChain is a popular open source blockchain platform that is widely used in organizations that aim to handle their digital assets. The platform was made known and progressed by Coinprism. The system has customized permissions on various levels which make it much favorable. Obenshain USP’s ⦁ Numerous degrees of authority ⦁ Private Network ⦁ Supported language: JavaScript #8. CHAIN CORE Chain Core is a rather new blockchain platform which is being highly brought into an application for its stupendous application in financial utilization. The authority and relocation of financial assets are dispersed among different organizations. Chain Core developer edition is public whereas the upgraded edition is utilized as the commercial product. ChainCore USP’s ⦁ Permissioned network ⦁ Native digital currencies ⦁ Multi-Signature accountability ⦁ Consensus mechanism is Federate #9. QUORUM Quorum is the brainchild of J.P. Morgan and is the customized Enterprise edition of Ethereum. Akin to many others in the domain Quorum is public, free to use and motivates innovation and experimentation. It is a smart contract system highly in use by coming of the age organizations. Quorum USP’s ⦁ Suitable for high-speed applications ⦁ Innovative graphics interface- CakeShop ⦁ Majority voting consensus mechanism #10. IOTA IOTA is the latest horse in the race and has proved its beneficiary in the initial year itself. It is an added application of open source blockchain technology, presently in beta. The blockchain platform commits to be a primary facilitator in the machine economy. IOTA USP’s ⦁ Both public and permissioned network ⦁ IOTA Token based pricing ⦁ Backs minuscule nano payments CHOOSING THE ONE: Obtaining knowledge regarding the best blockchain platforms comes rather handy in choosing the one most perfectly suitable for your organization. However, the parameters which aid in the selection of the one can be mentioned as- ⦁ Popularity ⦁ Kind of Network ⦁ Activity ⦁ Languages Used ⦁ Consensus Mechanism ⦁ Pricing James Tredwell is a Technical SEO and Content Developer with Hopinfirst.com, one of the top . James is passionate about SEO, Technology and management and blogs frequently on the topics. Author Bio. :- iOS app development company

Top 10 Blockchain Platforms to watch for in 2018

What will be the real theme to define the Third Generation of Blockchain Technology? There is a lot of new interesting projects and points of view in Blockchain Technology, everyone is trying to develop the next big thing and are trying to call themselves the Third Generation of Blockchain Technology, someone even calls theme-self like , a great project without a doubt but with a huge misunderstanding in their communication. PS: This is my response to it: “The Fourth Generation Of Blockchain Technology “ Multiversum https://medium.com/@alet89/mmm-eos-is-already-more-scalable-and-product-90-ready-the-fourth-gen-is-a-joke-are-you-serious-1ff986cad294 Defining a Generational Milestone To understand this theme as well we need to think about the evolution of the Blockchain Technology: First Gen: Trasfer Values with Decentralized Transactions and Ledger [Bitcoin, Dogecoin, Litecoin, BitcoinCash, Monero, Dash, Verge etc.]. Second Gen: Trasfer Values and Logic with Decentralized Transactions, Ledger, applications and Data. Third Gen: ???? If we define projects like EOS, NEO or even Multiversum the Third Generation of Blockchain is like to call BitcoinCash, LiteCoin or Monero, the Second Generation because they are Bitcoin’s improvements (good or not) or projects designed for a specific use case. Litecoin is designed for scalability, bitcoin Cash is designed to improve the block size and try to become a standard for payments, Monero for privacy-based transactions and so on… A generational milestone is about something that really change this technology with some new concept, like how Ethereum concept started the Second Generation, introducing Smart Contract and logic transaction, with a completely new way to intend Blockchain as an OS where people can decentralize and use Applications in a trustless way, without working on centralized costume layers but inside the network. The most talked trends in Blockchain technology today is scalability and decentralization, but as in the old Gen, these themes are improvements of the Second Generation Concept. The question is What’s next? The Second Generation of Blockchain limits to mass adoption In the Second Generation of Blockchain Technology the development and research is huge and there is a lot of projects that are trying to develop the best technology as possible to run Decentralized Applications in their network, everyone has different views and different killer features, EOS is trying to scale as much as they can with the DPOS consensus focussed on Dapps, Ethereum is trying to decentralize and connect the public Blockchain with the Privates with Plasma, Cardano is trying to develop a network focussed on security, NEO is trying to use the POS consensus, Nebulas are developing an incentive-based Blockchain for developers and so on. This scenario is amazing in terms of research, and for sure in 2/3 years we’ll have a lot of different Blockchain networks focussed on specific fields and to solve different problems. At the same time if you want to build a Decentralized Application today you have to choose a network like Ethereum or EOS etc and you can’t change that for wherever reasons it’s like a marriage and this is dangerous for any kinds of business. An example is what’s happened in the Ethereum network in 2017 the hype for Crypto Kitties had congested the entire network, unfortunately, this is not the dangerous risks that could happen, like network updates errors, forks, hack and more, si your business is completely in the hand of the network you chose. The Blockchain is different than the Cloud, building a network using a regular Cloud network like AWS or Google is more resilient because you can split your application into different networks or change network for any reason whenever you want. In the Blockchain field, things are more complicated, first of all, there’s not an exit strategy to change network and there is not any plan to solve this simple problem that could happen: You start a decentralized application in the Ethereum network for example, in a 100% Trustless-friendly way and someone will start a new blockchain network better fixed perfectly for your business or Ethereum makes some changes that you Don’t like or for some other reasons you want to change the network for your Dapp, you can’t. Yes you can make a fork, but forks are designed and it’s a useful method for blockchain networks especially in values transactions only, when you transact logic things are more complicated and you’ll have a lot of users and forks usually are somethings designed for tech guys, not for the masses, and the last but not the list, when you have users and you transact logic you have to deal with regulations, in this days this kind of stuff is not so simple. This is a huge problem for small and big business. To understand as well what could be the Third Gen Of Blockchain Technology we have to think about something new, a hard trend that really makes sense, I think this hard trend could be Interoperability. As Don Tapscott says: “ 10 years from now, the top cryptocurrency today, I’d be surprised if there were still three of them on the list. And there are all kinds of amazing new technologies that are linking various Blockchain together and creating all kinds of new capabilities. Companies like Cosmos, Polkadot, Aion or Icon. These are the next generation of companies and probably investments opportunity too” The Second Generation of Blockchain Network is a vertical network technology where you can build your own application for the first time in the history with a Decentralized Logic, the next step in this technology is to solve the interoperability and connect Blockchain 2.0 with a new technology that works like the internet for computers in the 90’s. This concept, in my opinion, is critical to developing a trustless based decentralized application and at the same to scale this technology to run interconnected and unstoppable applications free from risks of a vertical network, using more network based on specific needs. The concept of Interoperability Interoperability is the concept of interconnecting different Blockchains to build Decentralized Applications that interoperate with more Blockchains transferring Values and Logic in a decentralized and Trustless way.The concept of Interoperability in Blockchain Technology is about to build a network designed to translate a Dapp and run in into more Blockchain OS to transfer logic and values between different blockchains. This sounds like a new step in this technology and why I like to call it The Third Generation of Blockchain Technology. An example to understand how big this concept could be, is thinking about the evolution of the “Old Wide Web”, in this case, the Second Generation Blockchain Network is like an E-Commerce with his own catalog, the interoperability Blockchain network is like a platform when you can purchase items from more catalogs from different E-Commerce. The interoperability Ecosystem Current approaches in the Interoperability field are like The Intranet and The Internet of the 90s, they can be organized into two sections: A Blockchain network designed to build private/public Blockchain that interoperates and transact values and logic each other with the same OS (INTRANET). Single-OS Interoperability: A Blockchain Network designed to interoperate different Blockchain Technologies translating functions for different O.S. to transact Values and Logic between different technologies and standards (INTERNET). Multi-OS Interoperability: The Intranet vision is already on and the most used project is Cosmos, “The Internet of Blockchains”. This technology is interesting for some companies that need an intranet from different Blockchains. One of the most interesting projects between the Intranet vision and the Internet vision is Polkadot, developed from a co-founder of the Ethereum, Gavin Wood, that works for an interconnectivity consensus between different Blockchains: I’m focussed on the second vision the Multi-OS Interoperability concept where projects like AION, ICON, and WANCHAIN have developed the Interoperability Alliance to push standards in this field to help Blockchain networks and Dapp’s developers to build technology and simplify in the future the resilience in interoperation and transitions between this technologies. As wrote in this official post: The inability for isolated blockchains to communicate with one another is a recurring problem in the decentralized landscape. Isolated inter-blockchain communication can put a strain to blockchain’s scalability and mainstream adoption. The Alliance’s main priority is collaborate on research on interchain transactions and communication. The Alliance will focus on developing common industry standards, sharing researching, and protocol architecture. All three blockchain projects have the common goal of connecting blockchain protocols. ICON is an interconnecting blockchain network that connects independent blockchains without additional intermediaries. Aion aims to become the common protocol used for these blockchains, enabling more efficient and decentralized systems to be built. Wanchain is a distributed financial platform that enables private, cross-blockchain smart contract execution that aims to unite the world’s isolated digital assets. _ICON has joined the “Blockchain Interoperability Alliance”. The Alliance is backed by ICON, Aion, and Wanchain and has…_medium.com Blockchain Interoperability Alliance: ICON x Aion x Wanchain From the article “ ” Blockchain Interoperability Alliance (BIA) — Defining Blockchain 3.0 “ICON project was first started by the request of our enterprise blockchain clients to interconnect with other blockchain networks. The goal of this alliance is to create a globally accepted standard for connecting blockchains and to bring innovations together,” said JH Kim, Foundation Council, ICON “We believe that collaborating across projects will help set best practices for blockchain interoperability and encourage broader market adoption across enterprises and the public,” shared Matthew Spoke, Founder, Aion. “The Blockchain Interoperability Alliance will play a crucial role in increasing the connectivity of our digital assets and our lives,” said Wanchain President Dustin Byington. Looking AION in this field, for example, this kind of Blockchain Network have to: Transfer logic (data) and value (crypto) in a decentralized and fully trustless way between different blockchains technologies (Ethereum, Bitcoin and others). Federate: Provide fast transaction processing and increase data capacity to all Aion participating blockchains. Scale: Enable the creation of customized public or private blockchains that maintain interoperability with other blockchains, allowing publishers to choose governance, consensus mechanisms, issuance, and participation. Spoke: The vision of Aion Network is a multi-tier blockchain set of connections to solve the problem of Blockchains as ‘islands in isolation’, to scale this technology for the mass adoption. the vision M. Spoke the CEO of Aion, is interesting for the next decade of this technology, were in the future we’ll see a lot of new Blockchains focussed on solving specific problems, interconnected to scale. The BAAM (Blockchain as a Micro-service )Concept This new technology that is reinventing Blockchain ecosystem, will change how we build Dapps, for a new generation of interoperable decentralized Applications, that really can change every sector and that finally could build a competitive web 3.0. As I wrote in my article: Understanding The Gold Rush of Scalable and Validated Data powered by Blockchain and Decentralized AI The Interoperability infrastructure allows developers to create Decentralized Applications in a new smart way that I like to call “Blockchain as a Micro-service.” first of all it’s sounds very cool, and is similar to the way in which we create modern centralized applications with microservices-based on dedicated and replaceable APIs for each function. In Dapps we will use many dedicated and replaceable Blockchains focused on specific needs to perform each task. This type of infrastructure will create an exponential research on the evolution of this technology, especially in new Blockchains focused on specific use cases, without loosing the Trustless validation. The “BAAM” concept, The concept of BAAM, possible after this generational passage, is a huge milestone for Blockchain technology that finally could defeating every friction for enterprise or small businesses to adopt Blockchain technology in a trustless way.

Federato Technologies

federato.ai

11-50 employees

Since 2020

COMPANY RANKING

FEDERATO TECHNOLOGIES

EVERGREEN INDEX #4387

Federato Technologies WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

Federato Technologies

federato.ai

11-50 employees

Since 2020

COMPANY RANKING

FEDERATO TECHNOLOGIES

EVERGREEN INDEX #4387

RELATED COMPANIESRANK

STORIES ABOUT FEDERATO TECHNOLOGIES

Federato Technologies WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!