Flowco.io

flowcoio

Ory Hydra is a popular open-source OAuth2 and OpenID Connect that provides secure authentication and authorization for applications. One of the key challenges in building a scalable and performant OAuth2 server is managing the persistence layer, which involves storing and retrieving data from a database. server Motivation Ory was approached by a popular service provider to optimize performance in their auth system at high load. They often struggled to cope with the huge influx of authorization grants during peak times (over 600logins/sec). Looking for another solution, they started to evaluate Ory Hydra, investigating if it can handle this amount of grants. After reaching out to the team, we started to investigate ways to improve the overall performance to make Ory Hydra faster and more scalable than ever before. The key to success was to re-engineer parts of Hydra's persistence layer to reduce write traffic to the database, moving to a transient OAuth2 flow. Moving to a transient OAuth2 flow One of the core parts of this work was moving a large chunk of the transient OAuth2 flow state, which is exchanged between the three parties involved in an OAuth2 flow, from the server to the client. Instead of persisting transient state to the database, the state is now passed between the parties as either AEAD-encoded cookies or AEAD-encoded query parameters in redirect URLs. AEAD stands for authenticated encryption with associated data, which means that the data is confidential and also can't be tampered with without knowing a secret (symmetric) key. The flow is then only persisted in the database once when the final consent is given. This change has several benefits. First, it reduces the amount of data that needs to be stored in the database, which in turn reduces write traffic. Second, it eliminates the need for multiple indices on the flow table that were previously used during the exchange. The OAuth2 authorization code grant flow in detail The relevant part of the OAuth2 flow that we wanted to optimize is an exchange between the client (acting on behalf of a user), Hydra (Ory's OAuth2 Authorization Server), and the login and consent screens. When a client requests an authorization code through the , the user will be redirected first to the login UI to authenticate and then to the consent UI to grant access to the user's data (such as the email address or profile information). Authorization Code Grant Below is a sequence diagram of the exchange. Observe that each UI gets a as part of the URL parameters (steps 3 and 12) and then uses this as a parameter to retrieve more information (steps 4 and 13). Finally, both UIs either accept or reject the user request, usually based on user interaction with the UI (from steps 6 to 8 and 15 to 17). This API contract keeps Ory Hydra headless and decoupled from custom UIs. CHALLENGE CHALLENGE Optimization: Passing the AEAD-encoded flow in the URL parameters To reduce database access we now pass as the , , , and an AEAD-encoded flow. This way, we rely on the parties involved in the OAuth2 flow to pass the relevant state along. LOGIN_CHALLENGE LOGIN_VERIFIER CONSENT_CHALLENGE CONSENT_VERIFIER Before After The login and consent challenges and verifiers are random UUIDs stored in the database. The login and consent challenges and verifiers are the AEAD-encoded flow. Accepting or rejecting a request from the UI involves a database lookup for the specific challenge. Accepting or rejecting a request from the UI involves decrypting the flow in the challenge and generating an updated flow as part of the verifier. Implementation Since Ory Hydra is open source, you can review code changes in the Ory GitHub repositories. This is the relevant . commit Here is where we encode the flow in the specific challenges and verifiers: // ToLoginChallenge converts the flow into a login challenge.
func (f *Flow) ToLoginChallenge(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsLoginChallenge)
}

// ToLoginVerifier converts the flow into a login verifier.
func (f *Flow) ToLoginVerifier(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsLoginVerifier)
}

// ToConsentChallenge converts the flow into a consent challenge.
func (f *Flow) ToConsentChallenge(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsConsentChallenge)
}

// ToConsentVerifier converts the flow into a consent verifier.
func (f *Flow) ToConsentVerifier(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsConsentVerifier)
} In the persister (our database repository) we then decode the flow contained in the challenge. For example, here's the code for handling a consent challenge: func (p *Persister) GetFlowByConsentChallenge(ctx context.Context, challenge string) (*flow.Flow, error) {
    ctx, span := p.r.Tracer(ctx).Tracer().Start(ctx, "persistence.sql.GetFlowByConsentChallenge")
    defer span.End()

    // challenge contains the flow.
    f, err := flowctx.Decode[flow.Flow](ctx, p.r.FlowCipher(), challenge, flowctx.AsConsentChallenge)
    if err != nil {
        return nil, errorsx.WithStack(x.ErrNotFound)
    }
    if f.NID != p.NetworkID(ctx) {
        return nil, errorsx.WithStack(x.ErrNotFound)
    }
    if f.RequestedAt.Add(p.config.ConsentRequestMaxAge(ctx)).Before(time.Now()) {
        return nil, errorsx.WithStack(fosite.ErrRequestUnauthorized.WithHint("The consent request has expired, please try again."))
    }

    return f, nil
} Let's look at the impact of the changes when compared to the code without optimizations: The flows are now much faster and talk less to the database. Improved indices lead to further performance improvements By introducing a new index on the table, we were able to increase throughput and decrease CPU usage on PostgreSQL. The screenshot below shows the execution of the benchmarks without the improved indices where CPU usage spikes to 100%, and with improved indices, where CPU usage stays below 10%. hydra_oauth2_flow With the newly added indices, CPU usage (green bars) is removed, which reduces the likelihood of BufferLocks and related issues: Results The code and database changes reduced the total roundtrips to the database by 4-5x (depending on the amount of caching done) and reduced database writes by about 50%. Benchmarks Benchmarking the new implementation on Microsoft Azure with the following specifications: Services Configuration Total max SQL connections Notes Ory Hydra Consent App OAuth2 Client App rakyll/hey (http benchmark tool) 3x Standard_D32as_v4; South Central US 5x Standard_D8s_v3; South Central US 512 Every VM ran all the processes mentioned. PostgreSQL 14 in HA configuration Memory Optimized, E64ds_v4, 64 vCores, 432 GiB RAM, 32767 GiB storage; South Central US RAM beats CPU. Ory can perform up to at the peak and in the above configuration. This is possible by making the flow stateless and optimizing indices in frequently used queries. 1090 logins per second 800 logins / second consistently Conclusion The performance optimization work done by the Ory team has resulted in a significant improvement in Hydra's performance and scalability. By reducing write traffic to the database and improving the codebase and dependencies, Hydra is now faster and more responsive than ever before. By improving the indices, Hydra now scales much more efficiently with the number of instances. In the future, we will continue to optimize Ory's software to handle even more traffic. We believe that it's possible to get 5x more throughput on a single PostgreSQL node with data model optimizations. If you're building an OAuth2 server, we highly recommend giving Ory's fully certified OpenID Connect and OAuth2 implementations a try: Ory OAuth2 – our running on the global Ory Network, based on Ory Hydra – already utilizes the optimizations described in this article and setting it up only takes few minutes! fully managed service open source Also published . here

Scaling Ory Hydra to Two Billion Monthly OAuth2 Flows on a Single PostgreSQL DB

Escalamiento de Ory Hydra a dos mil millones de flujos mensuales de OAuth2 en una única base de datos de PostgreSQL

Mise à l'échelle d'Ory Hydra à deux milliards de flux OAuth2 mensuels sur une seule base de données PostgreSQL

सिंगल पोस्टग्रेएसक्यूएल डीबी पर ओरी हाइड्रा को दो बिलियन मासिक OAuth2 फ्लो तक स्केल करना

Ory Hydra を単一の PostgreSQL DB で月間 20 億の OAuth2 フローに拡張

Dimensionando o Ory Hydra para dois bilhões de fluxos OAuth2 mensais em um único banco de dados PostgreSQL

Mở rộng quy mô Ory Hydra lên hai tỷ luồng OAuth2 hàng tháng trên một cơ sở dữ liệu PostgreSQL

在单个 PostgreSQL 数据库上将 Ory Hydra 扩展到每月 20 亿个 OAuth2 流量

Web3 and crypto news can be overwhelming at times. GeN3 Magazine is launching its first issue that will share the positive story of web3 and the innovations in entertainment, lifestyle, culture, and finance emerging from blockchain technology. Print is dead. Or is it? Last year, No.3 Media decided it was time to bring print back to life, and they wanted to use blockchain to do it. Mark, the founder of , teamed up with from Agami Co., at TokenPr, and a host of other advisors to bring print to the era of Web3. This weekend web3 insiders got the first sneak peak of the premier issue at the CryptosRUs party in Las Vegas. Many attendees of the event were also featured in the magazine including Brian D. Evans, George Tung, Jett Tang, executive producer of , BeNFT, Flux, Rick Dugdale, and more. The event was a who's who in crypto and the first issue was received with rave reviews. No.3 Patrick Sarah The Next Crypto Gem While the focus always was and is, a medium to tell the story of Web3, the original concept has become a blockchain beast. The first issue will feature well-known brands like Animoca’s Mocaverse, Club3, Gala Games, and Vulcan Forged. With a focus on content with long term relevance to the crypto cultural movement, GeN3 Magazine aims to serve more or less as a time capsule of the technology as it evolves. Each issue will feature traditional and digital artists using emerging technologies to engage the community with their work, Lifestyle pieces on crypto clubs or destinations, industry changes that are taking place, and professional advice for those working or interested in the Web3 world.  New technologies that have long-lasting potential for positive change will be another crucial focus. For the first cover, we chose of BDE.ventures. His networking abilities have landed him in several unique positions in the Web3 world. Brian D. Evans not only serves as an advisor on multiple projects, but he also is a regular keynote speaker at conferences, a judge on series, and an active player in the NFT and crypto communities. Brian D. Evans The Next Crypto Gem TV Many have pointed out that print media cannot keep up with digital news with the fast pace that the market changes and technology advances. GeN3 Magazine is not building itself up to be another breaking news crypto site. While the brand will expand into GeN3 Media with more digital options available, the revolution that is taking place in finance, decentralization, and blockchain technology is world changing and deserves to be documented and encapsulated for all to witness. What better way to share the movement and help the rest of the world catch on, than to return to the familiar and trusted feel of print. But this won’t be the magazine of the 90’s we all remember. In order to tell the story of Web3, people need to experience the power of it. GeN3 magazine will showcase multiple interactive features that will be expanded upon with each issue. The first issue features secure QR code technology from Flowcode throughout the issue that allows users to move from physical print to digital content from their mobile phones. Looking ahead to Issue 2, the magazine will feature Agami Co’s NFC chip technology embedded in each copy.  Not only does the chip verify the authenticity of the magazine with a simple scan of a smartphone, it has the ability to unlock multiple features and benefits for magazine holders. As the magazine takes root, holders will eventually get exclusive access to partner brands and discounts, first launches, whitelists, and more. One of the highlights of issue 1 of GeN3 magazine is the 40 under 40 list highlighting Web3 leaders and innovators. This first list was carefully curated behind the scenes to highlight the talent in the industry, and show a wide selection of individuals working to move the space forward. Developers, artists, influencers, advisors, investors, executives, and more populate the list to provide the public with a snapshot of the breadth of Web3. A host of new advisors and investors have joined up since the project first started. Oliver Blakey from , from , and Fund Manager have all come on board to bring the story of Web3 to life. With a dedicated and well known team of advisors ready to see this project through, GeN3 Magazine is well situated to become Web3’s premier magazine. Ascensive Assets George Tung CryptosRus Sami Rusani Most of the population of the world does not understand what is taking place in the world of crypto. Mainstream media highlights scam and fraud as the main narrative of Web3 and ‘magic internet money’ is still the bulk of their understanding. This is a space many could get behind, if only they understood it. GeN3 is on a mission to change that narrative and share the intricate world of innovation and ownership. By offering a visual and easy to digest print Magazine, the goal is to not only help the world understand Web3, but get excited about it. The glossy, perfect bound issue will have a limited print run with an exclusive number of collectible versions of the magazine available to the public. Interested readers can put their name on the list at and will then be one of the first to be notified when the option to purchase an issue goes live. Visit their twitter to stay on top of all the latest updates. gen3mag.com @number3mag Disclaimer: TokenPR is a proud member of the GeN3 Magazine team.

Print Media Comes to Life With GeN3 Magazine Launch, Telling the Story of Web3 

Los medios impresos cobran vida con el lanzamiento de la revista GeN3, que cuenta la historia de Web3

La presse écrite prend vie avec le lancement du magazine GeN3, racontant l'histoire du Web3

GeN3 मैगज़ीन लॉन्च के साथ प्रिंट मीडिया जीवंत हो उठा है, जो Web3 की कहानी बताता है

Web3 のストーリーを伝える GeN3 マガジンの創刊で印刷メディアが活性化

A mídia impressa ganha vida com o lançamento da revista GeN3, contando a história da Web3

Phương tiện in trở nên sống động với việc ra mắt tạp chí GeN3, kể câu chuyện về Web3

随着 GeN3 杂志的推出，平面媒体焕发活力，讲述 Web3 的故事

Knowing how to find bottlenecks in your application is a valuable skill when trying to scale to thousands or even millions of users. Some useful techniques for achieving this goal are and profiling. load testing In this article, we will learn how to use JMeter to load test an e-commerce application and MongoDB Profiler to find slow queries. Additionally, we will apply the Extended Reference Pattern to optimize it. These techniques have been proven effective in one of the companies I worked for, reducing the latency of endpoints by 90% and decreasing MongoDB CPU usage from 20% to 3%. The results were instrumental in enabling an e-commerce platform to handle thousands of users during a Black Friday-like event. The e-commerce application MongoStore is a simple e-commerce platform that operates in two regions of the world. It currently offers approximately 100,000 products in each region and has plans to expand to other locations. The application is developed using , TypeScript, Serverless Framework, AWS, and . Node.js MongoDB Let’s examine its current implementation and architecture: You can view the complete code along with deployment instructions. here on GitHub The Product schema contains the , , and fields. sku name description const ProductSchema = new Schema<IProduct>(
  {
    sku: { type: String, required: true },
    name: { type: String, required: true },
    description: { type: String, required: true },
  },
  { timestamps: true },
); Each region can have multiple products, and the prices are specified per region. In this case, the schema references the and fields of the product. sku price const RegionSchema = new Schema<IRegion>(
  {
    name: { type: String, required: true, index: true, unique: true },
    products: [
      {
        _id: { id: false },
        sku: { type: String, required: true },
        price: { type: Number, required: true },
      },
    ],
  },
  { timestamps: true },
); For its architecture, the application uses a MongoDB Cluster hosted on 3 EC2 instances. The servers employ AWS Lambda and are behind an API Gateway. With this configuration, MongoStore can scale the backend servers using AWS Lambda. However, the database can still present scalability challenges since it does not scale automatically. Note: Auto-scaling has intentionally not been set up to identify database bottlenecks more effectively. Profiling with MongoDB Users have reported that the website is down, and CloudWatch data confirms that the database CPU usage has reached 100%. Consequently, it is likely that a database query is causing the issue. To investigate further, you can utilize the . The profiler collects data from MongoDB queries that exceed a specific threshold and stores it in the collection. MongoDB Database Profiler system.profile Connect to the primary MongoDB instance and use to execute the following command, which sets the data collection threshold to 100 ms: Mongo Shell db.setProfilingLevel(1, 100)
{
        "was" : 0,
        "slowms" : 100,
        "sampleRate" : 1,
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1683765280, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1683765280, 1)
} Now, retrieve the queries that have been executed and took longer than the threshold. db.system.profile.find().pretty()
{
        "op" : "command",
        "ns" : "database.regions",
        "command" : {
                "aggregate" : "regions",
                "pipeline" : [
                        {
                                "$match" : {
                                        "_id" : ObjectId("645c38d52f5c6ee098f9a392")
                                }
                        },
                        {
                                "$unwind" : "$products"
                        },
                        {
                                "$lookup" : {
                                        "from" : "products",
                                        "localField" : "products.sku",
                                        "foreignField" : "sku",
                                        "as" : "products.product"
                                }
                        },
                        {
                                "$unwind" : "$products.product"
                        },
                        {
                                "$limit" : 100
                        },
                        {
                                "$project" : {
                                        "_id" : "$products.product._id",
                                        "sku" : "$products.product.sku",
                                        "name" : "$products.product.name",
                                        "description" : "$products.product.description",
                                        "price" : "$products.price"
                                }
                        }
                ],
                "cursor" : {

                },
                "lsid" : {
                        "id" : UUID("d3e2a4d9-4970-4d80-917f-384be47acbd8")
                },
                "$clusterTime" : {
                        "clusterTime" : Timestamp(1684015798, 1),
                        "signature" : {
                                "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                                "keyId" : NumberLong(0)
                        }
                },
                "$db" : "database"
        },
        "keysExamined" : 1,
        "docsExamined" : 20000001,
        "cursorExhausted" : true,
        "numYield" : 20000,
        "nreturned" : 100,
        "queryHash" : "5B7E4F14",
        "planCacheKey" : "8928FB1D",
        "locks" : {
                "FeatureCompatibilityVersion" : {
                        "acquireCount" : {
                                "r" : NumberLong(20203)
                        }
                },
                "ReplicationStateTransition" : {
                        "acquireCount" : {
                                "w" : NumberLong(1)
                        }
                },
                "Global" : {
                        "acquireCount" : {
                                "r" : NumberLong(20203)
                        }
                },
                "Mutex" : {
                        "acquireCount" : {
                                "r" : NumberLong(202)
                        }
                }
        },
        "flowControl" : {

        },
        "readConcern" : {
                "level" : "local",
                "provenance" : "implicitDefault"
        },
        "writeConcern" : {
                "w" : "majority",
                "wtimeout" : 0,
                "provenance" : "implicitDefault"
        },
        "storage" : {

        },
        "responseLength" : 21893,
        "protocol" : "op_msg",
        "millis" : 12095,
        "planSummary" : "IDHACK"
} The whole operation took 12 seconds and by the variables and we can see that it only checked one index key and went through millions of documents. keysExamined docsExamined After some investigation, we noticed that we forgot to add the index to our collection. Let’s do it. sku products const ProductSchema = new Schema<IProduct>(
  {
    sku: { type: String, required: true, index: true, unique: true },
    name: { type: String, required: true },
    description: { type: String, required: true },
  },
  { timestamps: true },
); Now, let’s set the threshold to a lower time to get the results. But, first, clear the database by disabling the profiler and enabling again. db.system.profile db.setProfilingLevel(0) # disable profile
db.system.profile.drop() # drop profile database
db.setProfilingLevel(1, 10) # set threshold to 10ms db.system.profile.find().pretty()
{
        "op" : "command",
        "ns" : "database.regions",
        "command" : {
                "aggregate" : "regions",
                "pipeline" : [
                        {
                                "$match" : {
                                        "_id" : ObjectId("645c38d52f5c6ee098f9a392")
                                }
                        },
                        {
                                "$unwind" : "$products"
                        },
                        {
                                "$lookup" : {
                                        "from" : "products",
                                        "localField" : "products.sku",
                                        "foreignField" : "sku",
                                        "as" : "products.product"
                                }
                        },
                        {
                                "$unwind" : "$products.product"
                        },
                        {
                                "$limit" : 100
                        },
                        {
                                "$project" : {
                                        "_id" : "$products.product._id",
                                        "sku" : "$products.product.sku",
                                        "name" : "$products.product.name",
                                        "description" : "$products.product.description",
                                        "price" : "$products.price"
                                }
                        }
                ],
                "cursor" : {

                },
                "lsid" : {
                        "id" : UUID("cbbc0d5c-c35b-4428-8af6-d8b4a8ac0bd0")
                },
                "$clusterTime" : {
                        "clusterTime" : Timestamp(1684015558, 1),
                        "signature" : {
                                "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                                "keyId" : NumberLong(0)
                        }
                },
                "$db" : "database"
        },
        "keysExamined" : 101,
        "docsExamined" : 101,
        "cursorExhausted" : true,
        "numYield" : 0,
        "nreturned" : 100,
        "queryHash" : "5B7E4F14",
        "planCacheKey" : "2FBA11A3",
        "locks" : {
                "FeatureCompatibilityVersion" : {
                        "acquireCount" : {
                                "r" : NumberLong(203)
                        }
                },
                "ReplicationStateTransition" : {
                        "acquireCount" : {
                                "w" : NumberLong(1)
                        }
                },
                "Global" : {
                        "acquireCount" : {
                                "r" : NumberLong(203)
                        }
                },
                "Mutex" : {
                        "acquireCount" : {
                                "r" : NumberLong(202)
                        }
                }
        },
        "flowControl" : {

        },
        "readConcern" : {
                "level" : "local",
                "provenance" : "implicitDefault"
        },
        "writeConcern" : {
                "w" : "majority",
                "wtimeout" : 0,
                "provenance" : "implicitDefault"
        },
        "storage" : {

        },
        "responseLength" : 21893,
        "protocol" : "op_msg",
        "millis" : 29,
        "planSummary" : "IDHACK
} After adding the necessary index, the execution time was dramatically reduced to only 29 milliseconds. Additionally, the number of keys and documents examined also decreased significantly. That is a lot better! And finally, our users have stopped seeing crashes in the app. Load testing with jMeter Now that we fixed the bottleneck we had, we are confident that our system can handle thousands of users on a Black Friday, right? … WRONG! Improving just one endpoint and a database query does not guarantee we can handle the load we are expecting. One effective tool for load testing is , which can simulate multiple users accessing and making requests to the system simultaneously. Apache JMeter In this scenario, let’s assume we are expecting 10,000 users are expected, with each user making approximately 1 request every 10 seconds to the endpoint responsible for fetching products. This translates to a requirement of 1,000 requests per second. To validate the system’s ability to handle this expected throughput, follow these steps: Retrieve the ID of one of the regions using . cURL curl https://3mspdgu8cb.execute-api.us-east-1.amazonaws.com/staging/regions
{"regions":[{"_id":"645c38d52f5c6ee098f9a392","name":"Alaska"},{"_id":"645c39102f5c6ee098fcb0e9","name":"Rio"}]} Use the provided endpoint to fetch 100 products from the Alaska region. https://3mspdgu8cb.execute-api.us-east-1.amazonaws.com/staging/regions/645c38d52f5c6ee098f9a392/products\?limit\=100 Create a Test Plan in JMeter using the . Configure it to simulate 100 threads with a 30-second startup time, holding the load for 60 seconds. Ultimate Thread Group plugin Configure the HTTP Request sampler in JMeter to target the appropriate endpoint. Additionaly, add the View Results Tree and Aggregate Report. Run the load tests. Upon executing the load tests, it is observed that the system only reaches approximately 20 requests per second before encountering 502 (Bad Gateway) errors. What does this mean? If we SSH into the machine and run we can see that the service crashed, meaning it couldn’t handle the load. sudo service mongod status mongod [ec2-user@ip-10-0-0-100 ~]$ sudo service mongod status
Redirecting to /bin/systemctl status mongod.service
● mongod.service - MongoDB Database Server
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2023-05-16 23:28:20 UTC; 474ms ago The question is, but our query is already using the correct indexes, it is not doing a Collection Scan, and it is executing in 29ms. Can we do better than this? Yes! The Extended Reference Pattern When working with NoSQL databases, it is essential to understand that JOIN operations tend to be slow. In the context of this application, the step performed a join, negatively impacting the performance. $lookup To mitigate this, it is valuable to be aware of the . This pattern involves copying only the necessary data into the collection, eliminating the need for joins and allowing querying a single collection for the required information. Extended Reference Pattern It is also important to know that our first implementation had one drawback: having an array of products does not scale, because there is a limit of 16MB per document. Imagine if we had millions of products in each region. That array would grow quickly. To address these concerns, the Extended Reference Pattern can be applied by embedding the regions and prices directly into the product collection. This design change involves introducing a new array, , which contains the of the region and the corresponding product’s price in that region. regions _id const ProductSchema = new Schema<IProduct>(
  {
    sku: { type: String, required: true, index: true, unique: true },
    name: { type: String, required: true },
    description: { type: String, required: true },
    regions: [
      {
        _id: { type: Schema.Types.ObjectId, required: true, index: true },
        price: { type: Number, required: true },
      },
    ],
  },
  { timestamps: true },
); The application’s endpoint has been modified to utilize a new aggregate query that only queries the collection, eliminating the need for additional lookups. products export async function getProductsOptimized(
  regionId: string,
  limit: string | number,
): Promise<IProduct[]> {
  const mongoRegionId = new mongoose.Types.ObjectId(regionId);
  return Product.aggregate([
    { $match: { regions: { $elemMatch: { _id: mongoRegionId } } } },
    { $limit: Number(limit) },
    {
      $unwind: '$regions',
    },
    {
      $match: { 'regions._id': mongoRegionId },
    },
    {
      $project: {
        _id: 1,
        sku: 1,
        name: 1,
        description: 1,
        price: '$regions.price',
      },
    },
  ]);
} Upon retesting the endpoint with the optimized query using the query string parameter, the system demonstrates a significant improvement. The aggregate report shows that the throughput has increased to 80 requests per second, representing a 400% improvement. optimized=true While this improvement is substantial, it is still short of the required throughput of 1,000 requests per second. To handle such loads, additional strategies can be employed, such as vertically scaling the EC2 instances by adding more CPU and RAM, distributing the load across multiple replicas, or even sharding the database. Conclusion By utilizing MongoDB’s profiling capabilities for query optimization, load testing with tools like Apache JMeter, and applying effective design patterns like the Extended Reference Pattern, it is possible to significantly improve the performance and scalability of the application. Additionally, implementing observability practices, including monitoring, logging, and alerting, allows for proactive identification and resolution of performance bottlenecks. Also published here.

Finding Bottlenecks in MongoDB: JMeter Profiling and Load Testing Strategies

If you’re a developer working on a web3 app, you know that onboarding mainstream users into web3 is difficult. Even with the promise of truly owning your data, making near-free worldwide payments, and using a censorship-free system, the current process of creating and using a digital wallet is just too hard. Fortunately, there’s a better way. promises to streamline the process, making user onboarding seamless and simple. —a decentralized, public, layer-one blockchain designed for building web3 experiences for mainstream users—has used account abstraction to combine a model with to create an onboarding that is easy, simple, and as straightforward as traditional apps. Account abstraction Flow hybrid custody walletless onboarding Let's dive into the details to see how Hybrid Custody and Walletless Onboarding work together: the architecture, the code, and what we can expect moving forward. Why web3 onboarding is so difficult Digital wallets bring all the benefits of web3, including the assurance that no one can control your assets. There are no middlemen such as banks taking a percentage of your money, or third parties dictating what you can do. However, because most wallets are “self custody” (meaning only you have access), you—and only you—are in charge of your keys. There is no customer support if you make a mistake. This means you must store your keys safely. Don’t store them in clear text on your computer where they can be hacked. And don’t write them down where someone can read them. And don’t lose them, because there is no reset password in web3, so you’ll lose . You can see the issues. everything In addition to the challenges of self custody, you also need to understand tokens and payments. How do you pay for your transactions? What tokens do you need? How do you move from fiat to tokens? How do you transfer the tokens once you have them? Oh, you made a typo when you sent your tokens? Sorry, they’re lost forever. Again, you can see how difficult this all is. With all these obstacles, it’s easy to see why many people don’t make it past step one of signing up for your app. Custodial wallets: great UX, lacking ownership Developers tried to solve this issue with . These wallets manage the keys for you and have many benefits. custodial wallets Signing up is easy. You can use an email and password, as you would for a traditional service, and the wallet provider takes care of the rest. If you lose your login credentials, your keys are still accessible on the provider's servers – you can simply request a password reset. Since the wallet software runs on their servers, providers can even pay your fees. You don’t have to understand tokens, funding, and transfers. transaction But as the saying in web3 goes: “Not your keys, not your coins!” While custodial wallets solved many onboarding issues, the wallet providers, in the end, owned the keys. We’re right back to the problems we wanted to get away from: a centralized entity that can prevent you from using your tokens, can close your account at a whim, or can shut down completely, taking your funds with them. Wouldn’t it be nice if you could have the best of both worlds? Account abstraction to the rescue! solves one of the fundamental issues of blockchain networks: the distinction between externally owned accounts and contract accounts. Account abstraction An (EOA) is an account controlled by the user with a private key. These are the traditional web3 wallets that we talked about above. An EOA has a public address and initiates transactions (such as sending tokens) with the blockchain. externally owned account A (which is also just a smart contract) is a program deployed on-chain and executed by the network nodes. Contract accounts don’t initiate transactions. Instead they respond to a transaction with an action of their own. They have no private key—they are controlled by their code. This type of account could be said to have . contract account app-custody Account abstraction combines app-custody and self-custody into a new user experience (UX), giving users the benefits of both: control of their wallets, while abstracting away the wallet (and blockchain) altogether. On Flow, account abstraction is the basis for Walletless Onboarding using Hybrid Custody. Hybrid Custody gives you the best of both worlds allows developers to create an onboarding experience for users that is nearly seamless. Walletless Onboarding The walletless user journey (from flow.com) The hybrid custody model uses account abstraction to allow —where a child account delegates control to a parent account through a , essentially allowing one account to control another. account delegation Capability Now, the self-custody account can be a parent account, giving permissions to the app account, which is now a child account. With this setup, the accounts work together, and your app can work like this: Your app creates the wallets for the users. There is no barrier to entry here—no key management, no signing transactions, no understanding of blockchain. Users can use your app, and buy and sell assets, without knowing or caring about the technical details. Your app can even subsidize transaction fees or let them fund their account with fiat such as USD. And it all happens invisibly to the user. Once the user comes back with their own wallet-managed account, your app can delegate access to the user’s account. Once linked, the user has hybrid custody – their wallet manages their main (now “parent”) account which secondarily maintains access on the app-created (now “child”) account. Since their main account now has access on the app-created account, they can bring app assets anywhere their wallet-managed account goes. They can trade your app’s assets on marketplaces or share them with friends. Yet, at the same time, since the app also has access to the account, it can still facilitate seamless UX that acts on their behalf. With this hybrid custody model, users get to experience your app right away, with minimal barriers and no risk. And later on, if they choose, they can take control of the self-custody account themselves, enjoying all the real ownership and self-sovereignty it brings. Technical details of hybrid custody and account abstraction Account Creation How does all this work? On and many other blockchain networks, an EOA is created from a public key. The key is hashed, and the first 160 bits of this hash become the address of the account controlled by that key. This makes account creation independent from the network but irrevocably ties an account to that key. Ethereum Flow uses a particular on-chain function to create account addresses. You can then add any number of keys to a Flow account, adding and removing access based on the custodian of each key. This means a transaction is required to create a new address—so you can’t create accounts offline—but it also makes Flow account ownership much more flexible. You can see , but let’s check out the crucial parts here: a full account creation example on GitHub FromPrivateKey(userPrivateKey).
  SetHashAlgo(crypto.SHA3_256).
  SetWeight(flow.AccountKeyWeightThreshold) examples.ServiceAccount(flowClient) []*flow.AccountKey{userPublicKey}, 
  nil, 
  serviceAccountAddress
)


createAccountTx.SetProposalKey(
  serviceAccountAddress,
  serviceAccountKey.Index,
  serviceAccountKey.SequenceNumber
)


createAccountTx.SetReferenceBlockID(
  examples.GetReferenceBlockId(flowClient)
)


createAccountTx.SetPayer(serviceAccountAddress)

_ = createAccountTx.SignEnvelope(
  serviceAccountAddress,
  serviceAccountKey.Index,
  serviceSigner
)
_ = flowClient.SendTransaction(ctx, *createAccountTx) userPrivateKey := examples.RandomPrivateKey() userPublicKey := flow.NewAccountKey(). serviceAccountAddress, serviceAccountKey, serviceSigner := createAccountTx, _ := templates.CreateAccount( First, the user creates a key pair. Pretty basic stuff. Because on Flow, you create accounts by sending an account creation transaction, we need an existing account to send this transaction. This housekeeping is the job of the . ServiceAccount The receives the user’s public key to control the new account later, but the , which signs and pays for the transaction, handles the rest. In the case of your app, you would own this service account so you could sign account creation transactions on behalf of your user, also funding the account’s creation. createAccountTx ServiceAccount Hybrid Custody Let’s get back to the hybrid custody model. Your app can create app accounts for your users and link them to a public key that your app controls. The app can then handle your user's transactions and assets and even sponsor their transaction fees, just like any custodial account would do. Later, when a user decides to dive deeper into the web3 ecosystem and get their own wallet, you can create and save a link to the app-custodied account’s , and delegate control to the user’s wallet-managed account. AuthAccount Capability You can see an example of this in the where a multisig transaction is executed by the app to link a user's account to a walletless onboarding, dapp-custodied account to the signing user’s wallet-managed account. walletless arcade example Summary It’s worth driving the point home that delegating access on the app-created account is done via a linked Capability on the account which is then provided to the user. This mechanism for account delegation is completely new to blockchain accounts, and is only possible because of the account abstraction that’s been native to Flow for years now. This language API account linking feature is a huge advantage to accounts on Flow, in addition to many other features – m-of-n multisig, re-keying accounts, etc. – which are only now beginning to be realized on other chains with their nascent, non-native account abstraction implementations. Account abstraction is a powerful feature. On , it enables a hybrid custody model that allows users to enjoy the benefits of a custodial account when they sign up, while enabling them to take the account into self-custody at any time. It’s a new – and vastly improved – user experience that hopefully brings millions more users into web3. Flow Also published here .

Web3 Onboarding is Terrible — Make it Better With Account Abstraction and Flow

What is so unique about Flink, and what sets it apart from Storm and Spark? This article is part of Alibaba’s Flink series . Engineers at Alibaba’s food delivery app Ele.me(饿了吗) are finding themselves increasingly reliant on Apache Flink, an open source stream processing framework released in 2018. What is so unique about Flink, and what sets it apart from Storm and Spark? This article investigates how Ele.me’s big data platform operates in terms of real-time computing and assesses Flink’s various strengths and weaknesses. Ele.me’s Platform Today The following image illustrates the current architecture of the Ele.me platform. Current platform architecture On Ele.me’s current platform, data from multiple sources is first written into Apache Kafka. The main computation frameworks used for this are Storm, Spark, and Flink. The data results from these frameworks are then deposited onto various types of storage. There are currently over 100 tasks in Storm, roughly 50 in Spark, and a smaller number in Flink. The current cluster scale stands at 60TB of data a day, one billion computations, and 400 nodes. One thing to note here is that both Spark and Flink are run on YARN. While Flink on YARN is used mainly as the JobManager isolation between tasks, Storm on YARN is in standalone mode. This article will focus mainly on five areas — consistent semantics, Apache Storm, Apache Spark’s Spark Streaming and Structured Streaming, and Apache Flink. Consistent Semantics It is worth emphasizing here that maintaining the consistency of the following semantics is of paramount importance. · : Typically, when Java applications are being written, the simple “at-most-once” semantic is used, disregarding the source’s offset management and the downstream idempotence. When data comes in, there is no ACK mechanism for it — no matter the intermediate state and the writing data state. At-most-once (or fire-and-forget) · : A retransmission mechanism that retransmits data to ensure that each piece is processed at least once. At-least-once · : This is achieved by means of the coarse Checkpoint granularity control. Most exactly-once semantics refer to those in the computation framework, or in other words, whether the state inside the operator of each step can be replayed, and if the last job failed, whether it can recover from its previous state smoothly. This does not involve the idempotence that is output to sink. Exactly-once · : If it can be guaranteed that the downstream has idempotent operations such as “ON DUPLICATE KEY UPDATE” based on MySQL, or if you use ES/Cassandra etc., you can use the primary key to achieve the “upset” semantics. If you add idempotence into the mix when ensuring at-least-once, the result is exactly-once. Idempotence + At-least-once = Exactly-once Apache Storm While Spark Streaming and Structured Streaming were not fully put into practice until 2017, Ele.me has been using Apache Storm since before 2016. Storm boasts the following features. · Data is tuple-based · Latency is at the millisecond level · Java was supported to begin with, but now that Apache Beam is used, so too are Python and Go · The SQL function is incomplete. Typhon has been encapsulated internally, and users only need to extend some of our interfaces to use the numerous major functions. Flux is also a good tool for Storm. With it, to describe a Storm task, you only need to write a YAML file. This satisfies requirements to an extent, but users still essentially need to be writing Java at the engineer level, meaning that data analysts often cannot use it. Summary In summary, Storm has the following three main advantages. · : Because of the high threshold, its promotion is limited. Ease of use · : It requires more for external storage, including for key-value storage databases like Redis. State Backend · : Workers and slots are preset. Throughput is quite low, as only a small number of optimizations has been performed. Resource allocation Apache Spark: Spark Streaming At one point, the Ele.me team was asked whether it could write an SQL that could publish a real-time computing task in a matter of minutes. To do so, the team started working with Apache Spark Streaming, the main concepts of which are as follows. · Micro-batching: You have to preset a window and process data in that window. · Latency is at the second level, typically around 500ms. · Development languages include Java and Scala. · Streaming SQL: The Ele.me tech team hopes to provide a platform for streaming SQL in the near future. Key features · Spark Ecosystem and SparkSQL This represents a major advantage of Spark. The technology stack is unified, and SQL, graphical computation, and machine learning packages are all interoperable. Unlike Flink, with Spark, batch processing is done first, meaning that its real-time and offline APIs are unified. · Checkpoints on HDFS · Running on YARN Spark belongs to the Hadoop ecosystem and is highly integrated with YARN. · High throughput As it is a means of micro-batching, throughput is relatively high. Below is an image of the operations page on the Ele.me platform displaying the steps required when users publish a real-time task. On the page, there are a number of necessary parameters that the user has to select. First, the user has to select the Kafka cluster and the maximum consumer rate for each partition. Back pressure is enabled by default. For the shopping location, the user has to specify this each time. It is possible for a user, the next time they rewrite a real-time task, to select an offset shopping point according to their requirements. In the middle, the user can describe the pipeline. The SQL is multiple Kafka topics. When you select an output table, the SQL registers the Kafka DStream of the shopping as a table, and then writes a string of pipelines. Finally, some external sinks are encapsulated for the user. All storage types mentioned are supported. Storage is supported if it can implement upsert semantics. Multi-Stream joining Some users may still be wondering how to perform multi-stream joining. For Spark 1.5, refer to Spark Streaming SQL. In this open-source project, DStream is registered as a table, and the join operation is done on the table. But this operation is only supported in versions prior to 1.5. After the introduction of Structured Streaming in Spark 2.0, the project was discarded. The method is tricky: Unique method While allowing Spark Streaming to shop multiple topics, each batch of RDDs in the shopping DStream is converted into DataFrame so that DataFrame can be registered as a table. Then the table is divided in two so that a join can be made. This join is completely dependent on the data of this shopping, and the conditions of the join are uncontrollable. That is why this method is tricky. Take the following case as an example. Two topics are shopped. The table is split in two under the filer condition. Then a join can be made on these two tables. However, it is essentially a stream. Exactly-once semantics For exactly-once, it is important to note that data must be required to sink to external storage before offset commits. Whether the data sinks to ZK or MySQL, it is important to ensure that it is in a transaction. It is also necessary to output it to external storage before the source driver generates the Kafka RDD and the executor consumes the data. If these conditions are met, an end-to-end exactly-once semantic can be achieved. This is a major premise. Summary · : If you want to implement cross-batch stateful computation with the 1.X version, do it through these two interfaces. You still need to save this state to HDFS or external storage. Therefore, implementation is a bit complicated. Stateful processing SQL (<2.x mapWithState, updateStateByKey) · : The true semantics of multi-stream joins are unachievable. Real multi-stream joining · : These are difficult to implement. You have to sink them to external storage, and then manually submit the offset in the transaction. End-to-end exactly-once semantics Apache Spark: Structured Streaming In versions later than Spark 2.X, incremental stateful computations are used. The following picture is from the official website. All stream computations here refer to Google’s data flow, which has one important concept — a gap between the time an event occurs and the time data is processed. There is a watermark in the field of StreamCompute, which can specify the range of time delay. Data outside the delay window can be discarded. The following graph illustrates the architecture of Structured Streaming. In the graph, steps 1, 2, and 3 of exactly-once semantics are implemented. In essence, the implementation uses the batch method. With this method, the offset maintains itself and HDFS is used for state storage. The external sink does not run a similar idempotent operation, nor does it commit offset after writing. The implementation achieves the exactly-once only of the internal engine, all the while guaranteeing fault tolerance. Key features · : This can satisfy the stateful FlowCompute. Stateful processing SQL and DSL · : Multi-stream joins can be implemented through Spark 2.3. The method of implementation is similar to that for Flink. You have to define the conditions of two streams (this mainly involves defining time as a condition). For example, two topics stream in and you limit the data that needs a buffer through a field in a specific schema (usually the event time). By doing so, you join the two streams. Real multi-stream joining · . To support idempotent operations, you only need to extend the sink interface, which results in exactly-once semantics. It is easier to implement end-to-end exactly-once semantics Structured Streaming differs from native streaming APIs most notably in that, when Structured Streaming creates a table’s DataFrame, the schema of the table must be specified. This means that you have to specify the schema in advance. In addition, its watermark does not support SQL. At Ele.me, the tech team adds an extension to achieve the full SQL write and the conversion from left to right (as shown in the following graph). This is hopefully applicable not only to programmers, but also to data analysts who do not know how to program. Summary · : Before version 2.3, Structured Streaming was mainly based on processing time. With this, the completion of each batch of data processing would trigger the computation of the next batch. Version 2.3 introduced the trigger function for continuous processing, record by record. Trigger · : Currently, only map-like operations are supported, and support for SQL is also limited. Continuous processing · : The guarantee of end-to-end exactly-once semantics requires additional extensions. The Ele.me tech team found that Kafka version 0.11 provides transactional functions, based on which we can consider achieving end-to-end exactly-once semantics from the source to the engine and the sink. Low end-to-end latency with exactly-once guarantees · : For users who require functions such as CEP to handle complex events, the Drools rule engine can be run on each executor. CEP (Drools) With Apache Spark’s aforementioned shortcomings in Structured Streaming in mind, the engineers at Ele.me decided to start using Flink. Apache Flink Flink, an open source stream processing framework, is a leader in the streaming field. It boasts excellent graph computing and machine learning functions and its underlay supports YARN, Tez, among others. Flink’s framework The JobManager in Flink is similar to the driver in Spark, the TaskManager is similar to the executor, and tasks in both are the same. However, the RPC used by Flink is Akka, and the Flink core customizes the memory’s serialization framework. Moreover, tasks in Flink do not need to wait for each other, as tasks at each stage of Spark do, instead sending data to the downstream after processing. Flink’s binary data processing operator For serialization in Spark, users typically use the default Kryo or Java serialization. Project Tungsten has also optimized a JVM layer and code generation for Spark programs. Flink, meanwhile, implements a memory-based serialization framework that maintains the concepts of key and pointer. Its key is continuous storage, and is optimized at the CPU level. The probability of cache miss is extremely low. When comparing and sorting data, you do not compare the real data — you first compare the key. Only when the result is equal will the data be deserialized from the memory. At this point, you can compare the specific data. This represents a reliable performance optimization. Flink’s task chain With OperatorChain, if the upstream and downstream data distribution does not need to be shuffled again, the map followed is just a simple data filter. If we put it in a thread, the cost of thread context switching can be reduced. Concept of parallelism To illustrate this concept, imagine that five tasks need a number of concurrent threads to run for them. If the tasks are chained, they can be run in one thread, which improves the data transmission performance. With Spark, operators cannot set their degree of concurrency, whereas with Flink they can, making Flink more flexible, and resource utilization more efficient. Spark generally adjusts the degree of parallelism through Spark.default.parallelism. During a shuffle operation, parallelism is generally adjusted by the Spark.sql.shuffle.partitions parameter. In real-time computation, this parameter should be set to a smaller value. For example, in production, Kafka and partition parameters are set at almost the same, while the batch is adjusted to a slightly larger value. In the left graph, the Ele.me engineers have set the concurrency to 2 (up to 10). They then ran two concurrent batches, and made a group (up to 10) according to the key. This way, data can be dispersed as much as possible. States & checkpoints Flink processes data piece by piece, and after each piece has been processed, it is immediately sent to the downstream. In contrast, data in Spark has to wait until all the tasks in the stage that the operator is located in are completed. Flink has a coarse-grained checkpoint mechanism, which gives each element a snapshot at a low cost. Only when all the data belonging to this snapshot comes in will the computation be triggered. After the computation, the buffer data is sent to the downstream. Currently, Flink SQL does not provide an interface to control the buffer timeout — in other words, how long it takes for the data to be buffered. When you construct a Flink context, you can specify the buffer timeout as 0, so that data will be sent immediately after it is processed. You do not need to wait until a certain threshold is reached before sending it down. By default, the backend is maintained in the JobManager memory. Each operator’s state is written to RocksDB, and the asynchronous cycle is synchronized to external storage incrementally. Error tolerance The red node on the left side of the graph has a failover. If it is at-least-once, then retransmitting the data at the furthest upstream can correct an error. However, if it is exact-once, each compute node has to be played back from the time of the last error. Exactly-once two-phase commit After Flink 1.4, a two-phase commit protocol was introduced to support exactly-once. After data is consumed from the upstream Kafka, a vote will be initiated at each step to record the state, and the mark is handled through the checkpoint barrier. The state is only written to Kafka at the end (this applies only to Kafka 0.11 or later). Only after final completion will the state of each step be notified to the coordinator in the JobManager and solidified, thereby achieving exactly-once. Savepoints Another advantage of Flink is that the savepoint function can be achieved using Flink’s checkpoint. A savepoint is not just a means of data recovery, but also of recovering the computing state. Final analysis: Flink benefits vs. drawbacks Flink’s benefits can be summed up as the following key features: · : In contrast to Spark, Flink supports richer streaming semantics, including processing time, event time, and ingestion time. Trigger · : Flink supports continuous processing, and performs better with windows than Spark does. Continuous processing & windows · : With the two-phase commit protocol, users can choose to sacrifice throughput to adjust and ensure the end-to-end exactly-once semantics, according to their business needs. Low end-to-end latency with exactly-once guarantees · CEP · : Users can perform version control according to their business needs. Savepoints However, there are also some shortcomings in the following areas: 1. : The SQL feature is incomplete. Most users are migrating from Hive, where Spark has a coverage of over 99%. Currently, SQL functions are not supported, and neither is the setting of parallelism for a single operator. SQL 2. : Flink’s performance here is weaker than Spark’s, but the community is working to improve in these areas. Machine learning, graph computing This article is part of Alibaba’s Flink series . Alibaba Tech First hand and in-depth information about Alibaba’s latest technology → Facebook: . Twitter: . “Alibaba Tech” “AlibabaTech”

Flink or Flunk? Why Ele.me Is Developing a Taste for Apache Flink

Photo by on Andras Vas Unsplash As a dev, React has been an integral part of my learning experience. I started learning and writing Javascript in October of 2017, primarily in the context of Rails. Needless to say, there was a lot of involved and the experience was not particularly pleasant. My work with React actually started in React Native, but this article will strictly be for React web. Most of the concepts are applicable, but the presentation semantics will change in Native (the HTML elements must be switched for React Native <View>s <Text>s, <FlatList>, etc.) Javascript jQuery What are we building Today we will be building a Todo List. This article is meant to illustrate the setup of a React app with Babel/Webpack and Flow for type checking. The finished repo for part one is here: React Todo In this article we will primarily focus on using Flow types for Components. Some knowledge of React is assumed. Disclaimer: I will not be going in-depth into configuring webpack. I mostly use a boilerplate configuration from , which is a fantastic starting point for production-ready React apps, but not intended for beginners. If you are interested in learning more about Webpack configuration, here is a great resource: React-Redux Boilerplate How to Develop React.js Apps Fast Using Webpack 4 Why am I doing this This article will serve as an introduction to Flow types and static type-checking more generally. It is assumed you have some familiarity with React, but a great deal is not necessary. Getting started To start coding right away, run this command in your terminal: Some explanation behind the base configuration: For this tutorial I am going to take some steps so we can use webpack hot reloading in development. First I copied over my usual webpack and directories, then I copy over the folder from a application and add a small . (All of these are found in the repo) config/ server/ src/ create-react-app index.html I keep a private repo with my preferred base configuration (I didn’t use it here because it includes extras like Redux Saga and reselect that we will be adding in Part 2 of this series) that I clone to bootstrap a new app with these defaults. Next, I set up my as follows: package.json I also copy over my personal .eslintrc and the following .babelrc: Flow Types To begin development, create a .flowconfig and then start the flow server Put the following into your .flowconfig to stop flow from type-checking node_modules: Now start the server using: $ yarn start and go to in your browser. http://localhost:3000 Next, let’s build our first Flow typed component. Building a list So our todo list requires some essential functionality: Displays a list of items with completed status (true/false) and some text description Should be able to mark items completed or not Should be able to add new items Should be able to remove items So where should we begin? Since we are not using redux (yet), we need one of our components to have and keep track of state. I think in this case we are best served tracking state in the root App. So lets change App.js to this: For now, we’ve just declared our root component: React Apps always load using a “Root” component, generally named . A Flow-typed component carries the following class signature: Our root component is not going to receive any Props, and the State type declaration looks like this: App class ComponentName extends Component<Props, State> type AppState = {| todos: Array<any> |} For now, we are using because we haven't really determined what a Todo will be yet. We're using the pipes to tell Flow this is the exact shape of AppState - it should not contain any other keys or be missing the key. any {| |} todos Also notice the initialization of default state as a property rather than in the constructor. Let’s go ahead and define the behaviors for updating our App state here as well: const flipCompleted: (item: any) => * = item => ({...item,completed: !item.completed,}) class App extends PureComponent<any, AppState> {state: AppState = {todos: [],} addTodo = (todo: any): void => this.setState(state => ({todos: state.todos.concat(todo),})) removeTodo = (todoID: number): void => this.setState(state => ({todos: state.todos.filter((todo: any): boolean => todo.id !== todoID)})) completeTodo = (todo: any): void => {const updateTodos = existing => (existing.id === todo.id? flipCompleted(todo): existing)const newTodosState: (state: AppState) => AppState = state => ({todos: state.todos.map(updateTodos)}) this.setState(newTodosState) } render() {const { todos } = this.state return (  
  <div className="App">  
    <span className="status-text">  
      Total: {todos.length}  
    </span>  
  </div>  
) }} These are a little weird looking, so I’ll explain each one: : addTodo Our addTodo function just accepts a todo item and updates state to include it. the type signature lets us know we shouldn't expect a return value when we call . (todo: any): void this.addTodo(todo) : removeTodo Remove todo is going to accept a todo ID and iterate over the array of todos to remove the matching ID. Not the most performant solution, but we aren’t building for speed yet. : completeTodo Complete todo is going to accept an entire todo item and create a new todo item with its property flipped from true => false and vice versa (completed: !todo.completed) and then replace the item in the list of todos. completed A note on the type signature: type UpdateFunction = (item: any) => * A is what’s known in Flow as the . This is a placeholder type. It is similar to , but allows you to maintain a certain amount of type-safety, by telling the type checker to infer the returned type, as opposed to which Flow will ignore altogether. A related concept is types, which we will go over later. * Existential Type any any Generic Our removeTodo and completeTodo functions have some issues. Both and have dependencies on the Todo item having an property and in the case of a property. Instead of defining the TodoType here, let's go ahead and start building our component. removeTodo completeTodo id completeTodo completed TodoList To build a List Let’s make a directory in : src/components/ Let’s start with a single Todo and then work our way up to the logic we need for the list. TodoList/TodoItem.js In here we’ve finally defined our : TodoType Notice we’re again using pipes to tell Flow this is the exact shape of a TodoType. Here we also define our presentation logic for Todos, like connecting the ability to remove one to a button, or associating a checkbox with the property and having it handle : completed this.onCompleteTodo As far as , we're going to call the function passed in by props with the todo itself as the argument. To make sure the function passed into props has the right signature, we defined our 's prop types: onCompleteTodo TodoItem Here we declare that we want our component to receive 3 props, and none of them can be : TodoItem undefined todo: an object of shape TodoType onCompleted: a function that accepted a object as an argument, with a void return type TodoType onRemove: a function with no required argument (more on this later) with a void return type To clarify the function we'll look at the implementation of our : onRemove TodoList TodoList/TodoList.js Ok, let’s go through the props our List wants to receive first: Looks familiar, right? the props here correspond directly with the state/behavior we defined in our Root component. The component that maintains state is generally referred to as a component. Since we only have one in this app i haven't bothered, but generally good practice is to separate stateful and presentational into different directories. In any case, this component is going to receive the list of todos and the behaviors associated with manipulating it as props. container containers components We our (using so the dependency is stripped by babel, in cases where the only dependency is on type definitions) import type { TodoType } from './TodoItem' import TodoType import type This function probably makes more sense in the form component itself, but it could be argued that the logic of collecting the data that corresponds to an item shouldn’t be coupled to the shape of the item or the way you persist it. In any case, this accepts some text as a todo item, assigns it an ID (the current date as milliseconds since epoch) and gives it the default state of . This function gets called here: completed: false A few notes: : a function that accepts any number of (unary) functions, and returns a function that is the result of composing the function returns from right to left. so in this case, compose It may look a little strange, but composition is a very powerful concept. the shorthand works because in javascript functions are . Read more about functional programming in javascript in this excellent series: first class Composing Software: An Introduction also has some functional flavor, courtesy of currying: onRemoveTodo Remember that prop we referenced in the ? Well it doesn't need any arguments because the event handler we passed in was a curried function that we defined here. when we render a todo item we pass this function as and the return value (that the component receives as a prop) is a function with this signature . Because the function retains lexical scope, the we pass in the definition here will be the value receives. onRemove TodoItem this.onRemoveTodo(todo.id) () => void todoID this.props.onRemoveTodo(todoID) This means our renderList function looks like this: We take each todo in the , and create a with the required props. Array<TodoType> Todo the reason I used a curried function for onRemove is so the onClick handler in checkbox could be defined point-free ( ). This same strategy can be used to refactor our TodoItem into a stateless functional component, by currying onCompleted: onClick={this.props.onRemove} Finally, our render function simply invokes the renderTodos function as well as placing our above the list for easy access. NewTodoForm TodoList/TodoForm.js Component props/state: This is what’s known as a input, meaning the state of the input field is controlled by our application, rather than accepting the user’s input and directly assigning it to the attribute of the input element in the DOM. This allows you to intercept the value, reformat it, validate, or do whatever you want. In our case we aren't doing anything but saving it in state. This allows us to easily clear the input when the user submits by setting the state field to an empty string: controlled value Because we de-coupled our todo item logic from our form, our form isn’t responsible for building the todo. it simply calls the onSubmit function with the text from the form. If you wanted to add some validation for the form, you would have the option of doing so in the passed in . It isn't perfect, but it gets the job done here (if you wanted to validate this form, it'd probably be best to use a function as a prop that the form could call before deciding to call and clear its state). onSubmit validate onSubmit TodoList/index.js Here we are just exposing our TodoList as the default export and defining a type export for TodoType (so our App component can use it). Now our TodoList is assembled, it’s time to use it in our component. We can also update our type signatures: App On our way out I decided to spice things up with a custom count function. It simply accepts an array of generic objects and a predicate function, and returns the number of elements for which the function returns true. This lets us display the count of the completed items as well as the total. Some notes on the Flow types used here: type Predicate<T> = (item: T) => boolean In this type declaration, T is a . When we define our count function, we use another generic type this way: generic type const count = <K>(list: Array<K>, predicate: Predicate<K>): number => list.reduce((acc, item: K): number => (predicate(item) ? acc + 1 : acc),0,) So this is saying: A Predicate<T> is a function that accepts an (item: T) and returns a booleancount<K> is a function that accepts (list: Array<K>, predicate: Predicate<K>), and returns a number Generic types are a way to describe function behavior in static type checking systems. It uses what’s called a to indicate the actual type will be provided on invocation. Flow can use this definition and understand that if count is called with an Array of K, the Predicate function needs to accept an item of type K as an argument, and return a boolean. type argument This helps Flow to understand what you want your code to do, so later on if I try this: // in some other filetype Stuff = { item: string }const arrayOfStuff: Array<Stuff> = [{ item: 'hey' }, { item: 'hi' }]export default arrayOfStuff // in yet another fileimport array from './someOtherFile'import count from './App.js' count(array, => !!thing.notItem)// cannot get thing.notItem because property notItem is missing in 'Stuff' thing Flow will let me know this code I’m trying to type makes no sense. Otherwise here we are simply adding the call to our new TodoList component and passing in the props it expects. Finished, for now I hope I have illustrated some of the benefits of a static type-checking system for development. While this is hardly a comprehensive guide, it should serve to introduce some of the basic and important concepts of static types, and (hopefully) provide reason to use a type-checker for your JS projects.

Building a Modern React.js app with Flow

How to Build an AngularJS Events App

This short guide will provide you a consistent and reusable development workflow for all / projects. The more effort you put into writing quality code, the less time you spend on debugging. You can increase your code quality and reduce the time spent on debugging with a consistent development workflow. In this guide I will show you how to configure your editor to handle your code formatting, linting, and type checking. React React Native Test Driven Development and a preconfigured build configuration are recommended. I won’t go into much detail on either of these. I recommend for the web and using the for mobile development. Both require zero build configuration. is fantastic for testing . create-react-app React Native CLI Jest React Installation To get started download and install . You can launch from the terminal, . VS Code VS Code here’s how I use VS Code for my editor of choice. If you use Sublime Text or Atom this article still applies but you’ll need to use the relevant text editor extensions for your text editor. Let’s generate our app without any configuration! For this article we’ll use the , but you can follow along with as well. React Native CLI create-react-app $ react-native init SweetApp && cd SweetApp Open the SweetApp project with . VS Code $ code . Once you have open, click the Extensions button in the Activity Bar. Install the following extensions: VS Code ESLint Prettier-ESLint Flow Language Support Babel ES6/ES7 ESLint Setup We will install using Airbnb’s linter rules. Refer to for the installation instructions. ESLint Airbnb’s JavaScript Github Repo Then install $ npm install babel-eslint --save-dev babel-eslint You should now see an .eslintrc.json file in your project. Open the eslintrc file and configure it like so: Prettier Setup Since we are using there is very little setup required to get working. We’ve installed the extension, now we need to tell to format our code using a code formatter ( ) after saving a file. Prettier with ESLint Prettier Prettier-ESLint VS Code Prettier Press CMD + , if you’re on a Mac to open your and enable format on save. VS Code User Settings // Format a file on save. A formatter must be available, the file must not be auto-saved, and editor must not be shutting down.

"editor.formatOnSave": true Flow Setup is a static type checker for JavaScript and is included in your project if you use the or build configurations. Flow React Native CLI create-react-app From your terminal install using . Flow Homebrew $ brew install flow “Flow works best when installed per-project with explicit versioning rather than globally.” https://flow.org/en/docs/install/ To run per project we will install from . Later we will configure to run from node_modules in lieu of the global installation. Flow flow-bin npm VS Code Flow Flow Before installing we need to figure out which version to install. Open the .flowconfig file and scroll to the very bottom where you’ll find the version number. This is the version of we want to install from NPM. As of the day this article was published, the version to use is 0.42.0. flow-bin flow-bin Flow We also need to install the . babel-preset for Flow What are babel-presets? https://babeljs.io/docs/plugins/#presets $ npm install flow-bin@0.42.0 babel-preset-flow --save-dev Open the .babelrc file and add then add the config [ ](http://Retain line numbers. This will lead to wacky code but is handy for scenarios where you can’t use source maps. (NOTE: This will not retain the columns)) Flow as a preset “retainLines”: true Your .babelrc file should look like this: {

  "presets": [

    "react-native",

    "flow"

  ],

  "retainLines": true

} Now we want to tell our editor to use from the projects node_modules directory. We also want to disable JavaScript validation for this project only to . Flow fix a known issue Open the settings.json file and add this: Workspace // Support using flow through your node_modules folder, WARNING: Checking this box is a security risk. When you open a project we will immediately run code contained within it.
"flow.useNPMPackagedFlow": true,

// Enable/disable JavaScript validation. (For Flow)
"javascript.validate.enable": false

// Enable/disable default JavaScript formatter (For Prettier)
"javascript.format.enable": false Disabling the default JavaScript formatter allows to handle our code formatting. Prettier Let’s create some for . npm scripts Flow Open the package.json file and add the following scripts. // start the flow server
"flow start": "flow start",

// stop the flow server
"flow stop": "flow stop",

// check the flow status
"flow status": "flow status",

// check the flow coverage percentage
"flow coverage": "flow coverage" Your scripts in package.json should now look like this: From the same directory as your package.json file run the following terminal command. $ npm run flow start The is running and will perform type checking on any file with a // @flow annotation at the top of the file. Flow server Testing your new development workflow Open the index.ios.js file in the SweetApp project. Don’t use the index.android.js file as the .flowconfig file ignores it. to use Flow with .android files. See this post on Stack Overflow ESLint Linter You should notice red squiggly lines in your code. You can hover your mouse cursor over the red squiggly line and will tell you the linter rule causing the warning. ESLint Delete the semicolon at the end of the return statement inside the class render method. Hover your mouse cursor over the red squiggly line and you’ll see warns you about the missing semicolon! ESLint If you are not receiving any linter warnings please review the ESLint Setup portion of this guide. Prettier Code Formatter will auto-format your code based on it’s rules whenever you save a file. Prettier Add an array of numbers and save the array to a variable called test. Format the array of numbers like so: Now save the file and watch the magic of take effect! Prettier The test array is now formatted correctly! Flow Type Checking To ensure is working run the flow start npm script to start the that runs in the background. Flow Flow server $ npm run flow start Create an add function that returns the sum of two numbers. Now let’s add the WRONG types (string) to the add function parameters and return type. Notice after saving the file red squiggly lines appear under the number parameters of the add function invocation. Hover your mouse cursor over the red squiggly lines and informs you: Flow [flow] this type is incompatible with the expected parameter type of string Let’s fix the add function to use the correct number types. The type errors are gone! Flow A cool feature of the is the always up-to-date coverage percentage in the Status Bar. Flow VS Code extension Flow 🎉 You’re all set**!** 🎉 Hopefully this guide saves you the headache I experienced and makes for a great reference guide now and in the future. Happy coding! 😀

Configure ESLint, Prettier, and Flow in VS Code for React Development

Just flow with the flow. Designed by the Facebook team(open source) and for JavaScript developers, Flow is a static type checker that catches common errors in your application before they run. What is meant by a Static type checker? This generally means that the type of a variable is known at compile time. For some languages this means that you as the programmer must specify what type each variable is (e.g.: Java, C, C++); other languages offer some form of type inference, the capability of the type system to deduce the type of a variable (e.g.: OCaml, Haskell, Scala) — stackoverflow . Flow integrates with your code editor by checking changes and analyzing correctness of your code while you write code. Flow not only points out the error in your code, it also provides context as to what is wrong with the code. In this article we’d see dive into the smoothness of Flow and see how it could be beneficial for us. Make sure you have installed. NodeJS Pull out your code editor, I would be using as that is my code editor. There’s support for other editors such as , and . Sublime-text vim emacs nuclide Using Sublime-text, you would need to install a sublimeLinter package to have Flow working correctly which is . SublimeLinter-Flow Getting started is easy: Create a new project folder or open your already existing project. file in your project root. Create a _.flowconfig_ _npm install --save-dev flow-bin_ Create a new file or open any of your JS files. Just right at the top add this tiny JavaScript comment and that’s all. index.js // @flow In the file add this snippet for a start: index.js var str: number = 'hello world!';console.log(str); In your code editor, if you’re using Sublime text, you should see something like this below. Error messages displayed. What we did here was add a trivial type error, we expect to be a number but instead we assigned it a string. str To show that Flow constantly guards your code, take out the comment and see what happens. The error highlights are gone. //@flow Removing the flow comments triggers either to watch for errors or not. You can also check for errors in your terminal by running the command , you would get a log showing where the error occurred. flow Error log typing flow in the command line. Now if you had an existing project or you created a new project. If you ran your project or in your terminal, you would get a different error from what shows you. node index.js flow var str: number = 'hello world!';^SyntaxError: Unexpected token : This is because we need to add and a to help us transpile our code. Babel Babel plugin Install if you don’t already have it babel-cli npm install -g babel-cli npm install —-save-dev babel-plugin-transform-flow-strip-types Create a file in your project root and add .babelrc {"plugins: ["transform-flow-strip-types"]} In your terminal, run babel-node index.js If you will notice, even with the error Flow complains about, it was ignored. Meaning Flow does not stop you from going further with other things. But it is advisable to fix your errors before pushing to production :). Now to correct our error and let Flow stop barking at us for sending the wrong type. All we have to do is change to number string var str: string = 'hello world!';console.log(str); Flow supports a bunch of other types such as: boolean number string null void And , be sure to check them out. more One last example: Flow also supports type checking in function parameters. function doggy(x: number) {return x + 1;} What this means is that, we declare a function and we expect a number as the parameter. if we called the function: doggy(3) // flow displays no errorsdoggy("hola!") // we seem to enjoy flows error colors here. Just one last example. Just One Last Example: Having worked with Flow for a while now, i haven’t covered all the goodness of what Flow offers but here’s my favourite so far, feel free to share what you like about Flow in the comments to help others learn more. Flow allows us to define a function along with the return value we expect. What ever the function type may be, Async, Generators, Flow is compatible. function doggy(x: number): number {return "c";// throws error because a string is returned.} function catty(x: number): number {return 3; //throws no error because a number is returned} Finally! Check out t as well. Flow for Reac With this Type Checking, there are some advantages such as better productivity, more readable code and autocompletion by flow. Cheers!

Stories, News & Mentions about FLOWCOIO

Flowcoio’s Stories on HackerNoon