OpenRent is the way people rent property online

OpenRent

openrent

Lua-Land CPU Flame Graphs in OpenResty XRay

Selenium has been the de facto automation standard/tool for browser/UI automation for quite sometime. Creators of selenium also created selenium grid for scaling the UI testing. The grid consists of a selenium hub and several selenium nodes which connects/hosts the browsers and thus providing a way to run several UI tests simultaneously. Selenium also provides some easy way to customize/extend existing capabilities. For most of the use cases the selenium hub /node based architecture scales pretty well and doesn’t need much customization. Need for higher availability and scale The architecture, although supports scaling the selenium nodes, it doesn’t make it easy to scale the hub itself. On a typical deployment there would be a single hub and several nodes attached to it. The mapping of sessions/browser instances is maintained in memory inside the hub. So, if the hub fails, the whole grid will become unusable. A simple way to enable some level of HA is to run at least 2 hubs. Challenge with multiple hubs Requirement of sticky session Since selenium hubs can’t talk to each other, the distribution of load to different hubs need some kind of stickiness. Stickiness here needs to make sure that once the session is created on one hub, all subsequent request for that specific session comes to the same hub. Session stickiness is not a new thing. A typical stickiness for browser based clients is done via cookies. The LB looks at the cookie information being passed to decide which upstream server the request would be forwarded. Cookie based stickiness ruled out Passing this extra bit of information using cookie is not possible in selenium automation scenarios, so this was not an option for our use case. Client ip based stickiness One option that we have been using so far is to create stickiness based on client ip. So all calls originating from one client ip will always land to a given hub. While this works for most of the scenarios, the load on hubs are not distributed evenly. This particular technique solves the HA problem as such, but this may not be a great solution for scaling if the load is not even. Uneven load and uneven test cases in test suites A large test suite with huge number of UI tests can completely chock the one off hub where all tests are landing. For making our test infrastructure more reliable and robust, we may have to explore some different avenue. Openresty to the rescue Openresty is a software bundle of nginx with some high quality Lua libraries included. It allows you to customize load balancing behavior and add custom behaviors. Design We can add openresty as a proxy layer before the hub. We can deploy couple of instances of openresty in front of hub and the LB can forwards the request to one of these openresty instances in a round robin fashion ( earlier the LB was forwarding the request to hub directly using the client ip hashing). Openresty instances works as a reverse proxy and all instances of hub are upstream server for each openresty instances. The Magic The way the request forwarding works is: Each new session request will be forwarded to one of the hub in a round robin fashion. Once the session is created at hub and response comes back to the openresty proxy instance, it inspects the response body, looks for session id and appends the upstream host ip(host ip in number format) to the existing session id and responds back to the client. The client receives a new session id which has the upstream hub node embedded in it. Any subsequent operation on that session from client uses this session id in the url path, which now has the upstream hub’s ip embedded in the session id. The proxy inspects the request url and extracts the session id from the request. It rewrites the url path by removing the embedded host ip part. It also sets the upstream host to the specific host found in the ip section of the session id. Then it forwards the request to the desired hub. The hub sees the session id in the exact same format as it had generated and continues to process as usual. Sample nginx conf with embedded lua code: _Contribute to Mitendra/code_samples development by creating an account on GitHub._github.com Mitendra/code_samples Taking a step further While solving the scale and availability with openresty proxy, we can use this proxy for some more features built on top of Selenium hub. Rate limiting: Even though this would help in scaling the hub and nodes, there are scenarios where the Spike in number of new sessions may exceed the number of available resources. Currently all requests are put in the queue and processed when the resources are available which results into delayed session creation. Different test cases/frameworks have different timeout settings and while some of them proceed, many of them would get stuck. Many times, while the sessions get times out at client/test case layer, the hub still will go on and create a session. This unused sessions later gets cleaned up but it makes resource crunch even worse. To solve this we can add a rate limiting feature, where based on the capacity and traffic pattern only certain number of sessions in the queue would be allowed and beyond that we can return a http 429 status with some details like current active sessions and an estimated time after which it can be retried. Monitoring data: We can add a lot of monitoring data about sessions bring requested, time to create a new session, active session time etc from proxy layer, which can give further insight on how to optimize the infrastructure usage. References Image source: https://pixabay.com/en/skyscraper-highrise-building-tall-1209407/ Selenium: https://www.seleniumhq.org/ Selenium grid: https://www.seleniumhq.org/docs/07_selenium_grid.jsp Openresty: https://openresty.org/en/ _Embed the Power of Lua into NGINX HTTP servers. Contribute to openresty/lua-nginx-module development by creating an…_github.com openresty/lua-nginx-module _Embed the Power of Lua into NGINX HTTP servers. Contribute to openresty/lua-nginx-module development by creating an…_github.com openresty/lua-nginx-module

Scaling selenium with Openresty

In my talk, “Evolving your APIs,” I mention that an API gateway is a Reverse Proxy "on steroids.” One key difference between the former and the latter is that the API Gateway is not unfriendly to business logic. The poster child is rate-limiting. Rate-limiting is an age-old Reverse Proxy feature focused on protecting against Distributed Denial of Service (DDoS) attacks. It treats all clients the same and is purely technical. In this day and age, most API providers offer different subscription tiers; the higher the tier, the higher the rate limit, and the more you pay incidentally. It's not technical anymore and requires to differentiate between clients. In this post, I want to detail how to do it with Apache APISIX. Note I take most of the material from the workshop. Rate-limiting for the masses Apache APISIX offers no less than three plugins to rate limit requests: limit conn: limits the number of concurrent requests
limit req: limits the number of requests based on the Leaky Bucket algorithm
limit count: limits the number of requests based on a fixed time window The limit-count plugin is a good candidate for this post. Let's configure the plugin for a route: routes:
  - uri: /get
    upstream:
      nodes:
        "http://httpbin.org:80": 1
    plugins:
      limit-count:                     #1
        count: 1                       #2
        time_window: 60                #2
        rejected_code: 429             #3
#END Set the limit-count plugin
Limit requests to one every 60 seconds
Override the default HTTP response code, i.e., 503 At this point, we configured regular rate limiting. curl -v http://localhost:9080/get
curl -v http://localhost:9080/get If we execute the second request before a minute has passed, the result is the following: HTTP/1.1 429 Too Many Requests
Date: Tue, 09 Jul 2024 06:55:07 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 241
Connection: keep-alive
X-RateLimit-Limit: 1                   #1
X-RateLimit-Remaining: 0               #2
X-RateLimit-Reset: 59                  #3
Server: APISIX/3.9.1

<html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>openresty</center>
<p><em>Powered by <a href="https://apisix.apache.org/">APISIX</a>.</em></p></body>
</html> Configured limit
Remaining quota
Waiting time in seconds before quota replenishment Per-consumer rate limiting To configure per-consumer rate limiting, we first need to implement request authentication. APISIX offers many authentication plugins; we shall use the simplest one, key-auth. key-auth checks a specific HTTP request header - apikey by default. Here's how we configure consumers: consumers:
  - username: johndoe                  #1
    plugins:
      key-auth:
        key: john                      #2
  - username: janedoe                  #1
    plugins:
      key-auth:
        key: jane                      #2 Users
HTTP header request value curl -H 'apikey: john' localhost:9080/get #1
curl -H 'apikey: jane' localhost:9080/get #2 Authenticate as johndoe
Authenticate as janedoe In general, you attach plugins to APISIX routes but can also attach them to consumers. We can now move the limit-count plugin. routes:
  - uri: /get
    upstream:
      nodes:
        "httpbin:80": 1
    plugins:
      key-auth: ~                      #1

consumers:
  - username: johndoe
    plugins:
      key-auth:
        key: john
      limit-count:
        count: 1                       #2
        time_window: 60
        rejected_code: 429
  - username: janedoe
    plugins:
      key-auth:
        key: jane
      limit-count:
        count: 5                       #2
        time_window: 60
        rejected_code: 429
#END The route is only accessible to requests authenticating with key-auth
johndoe has a lower limit count than janedoe. Did he forget to pay his subscription fees? curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get The second request gets rate-limited. Per-group rate limiting We never attach permissions directly to identities in Identity Management systems. It's considered bad practice because when a person moves around the organization, we need to add and remove permissions one by one. The good practice is to attach permissions to groups and set the person in that group. When the person moves, we change their group; the person loses permissions from the old group and gets permissions from the new group. People get their permissions transitively via their groups. Apache APISIX offers an abstraction called a Consumer Group for this. Let's create two consumer groups with different rate limit values: consumer_groups:
  - id: 1
    plugins:
      limit-count:
        count: 1
        time_window: 60
        rejected_code: 429
  - id: 2
    plugins:
      limit-count:
        count: 5
        time_window: 60
        rejected_code: 429 The next step is to attach consumers to these groups: consumers:
  - username: johndoe
    group_id: 1
    plugins:
      key-auth:
        key: john
  - username: janedoe
    group_id: 2
    plugins:
      key-auth:
        key: jane curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get The second request gets rate-limited. We have the same results as before with two benefits. The first one is as I wrote above: when consumers move in and out, they change their permissions accordingly. The second benefit is that the limit count is shared among all consumers of a group.  Indeed, when you set a limit, you don't want each consumer to be rate limited at X requests per Y second; you want the group as a whole to share the limit. In this way, if a single consumer is very active, they will naturally cap the rate of other consumers who share the same group. Of course, you can set a limit on both a consumer and the group it belongs to. In this case, the lowest limit will apply first. consumers:
  - username: johndoe
    group_id: 2                        #1  
    plugins:
      key-auth:
        key: john
      limit-count:
        count: 1                       #2
        time_window: 60
        rejected_code: 429
  - username: janedoe
    group_id: 2
    plugins:
      key-auth:
        key: jane Move johndoe to group 2
Limit him individually curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: john' localhost:9080/get #1 johndoe hits the limit here, but janedoe now only has four requests left from this minute, as the former used one request Conclusion In this post, we implement rate limiting with Apache APISIX. We set the rate limit on a route and moved it to individual consumers. Then we moved it to consumer groups, so all consumers in a group share the same "pool". The complete source code for this post can be found on GitHub. To go further: Consumer
Consumer Group
Apache APISIX Hands-on Lab In my talk, “Evolving your APIs,” I mention that an API gateway is a Reverse Proxy "on steroids.” One key difference between the former and the latter is that the API Gateway is not unfriendly to business logic. The poster child is rate-limiting. Rate-limiting is an age-old Reverse Proxy feature focused on protecting against Distributed Denial of Service (DDoS) attacks. It treats all clients the same and is purely technical. In this day and age, most API providers offer different subscription tiers; the higher the tier, the higher the rate limit, and the more you pay incidentally. It's not technical anymore and requires to differentiate between clients. In this post, I want to detail how to do it with Apache APISIX. Note I take most of the material from the workshop . workshop Rate-limiting for the masses Apache APISIX offers no less than three plugins to rate limit requests: limit conn: limits the number of concurrent requests limit req: limits the number of requests based on the Leaky Bucket algorithm limit count: limits the number of requests based on a fixed time window limit conn : limits the number of concurrent requests limit conn limit req : limits the number of requests based on the Leaky Bucket algorithm limit req Leaky Bucket limit count : limits the number of requests based on a fixed time window limit count The limit-count plugin is a good candidate for this post. limit-count Let's configure the plugin for a route: routes:
  - uri: /get
    upstream:
      nodes:
        "http://httpbin.org:80": 1
    plugins:
      limit-count:                     #1
        count: 1                       #2
        time_window: 60                #2
        rejected_code: 429             #3
#END routes:
  - uri: /get
    upstream:
      nodes:
        "http://httpbin.org:80": 1
    plugins:
      limit-count:                     #1
        count: 1                       #2
        time_window: 60                #2
        rejected_code: 429             #3
#END Set the limit-count plugin Limit requests to one every 60 seconds Override the default HTTP response code, i.e., 503 Set the limit-count plugin limit-count Limit requests to one every 60 seconds Override the default HTTP response code, i.e. , 503 i.e. 503 At this point, we configured regular rate limiting. curl -v http://localhost:9080/get
curl -v http://localhost:9080/get curl -v http://localhost:9080/get
curl -v http://localhost:9080/get If we execute the second request before a minute has passed, the result is the following: HTTP/1.1 429 Too Many Requests
Date: Tue, 09 Jul 2024 06:55:07 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 241
Connection: keep-alive
X-RateLimit-Limit: 1                   #1
X-RateLimit-Remaining: 0               #2
X-RateLimit-Reset: 59                  #3
Server: APISIX/3.9.1

<html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>openresty</center>
<p><em>Powered by <a href="https://apisix.apache.org/">APISIX</a>.</em></p></body>
</html> HTTP/1.1 429 Too Many Requests
Date: Tue, 09 Jul 2024 06:55:07 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 241
Connection: keep-alive
X-RateLimit-Limit: 1                   #1
X-RateLimit-Remaining: 0               #2
X-RateLimit-Reset: 59                  #3
Server: APISIX/3.9.1

<html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>openresty</center>
<p><em>Powered by <a href="https://apisix.apache.org/">APISIX</a>.</em></p></body>
</html> Configured limit Remaining quota Waiting time in seconds before quota replenishment Configured limit Remaining quota Waiting time in seconds before quota replenishment Per-consumer rate limiting To configure per-consumer rate limiting, we first need to implement request authentication. APISIX offers many authentication plugins; we shall use the simplest one, key-auth . key-auth checks a specific HTTP request header - apikey by default. key-auth key-auth apikey Here's how we configure consumers: consumers:
  - username: johndoe                  #1
    plugins:
      key-auth:
        key: john                      #2
  - username: janedoe                  #1
    plugins:
      key-auth:
        key: jane                      #2 consumers:
  - username: johndoe                  #1
    plugins:
      key-auth:
        key: john                      #2
  - username: janedoe                  #1
    plugins:
      key-auth:
        key: jane                      #2 Users HTTP header request value Users HTTP header request value curl -H 'apikey: john' localhost:9080/get #1
curl -H 'apikey: jane' localhost:9080/get #2 curl -H 'apikey: john' localhost:9080/get #1
curl -H 'apikey: jane' localhost:9080/get #2 Authenticate as johndoe Authenticate as janedoe Authenticate as johndoe johndoe Authenticate as janedoe janedoe In general, you attach plugins to APISIX routes but can also attach them to consumers. We can now move the limit-count plugin. limit-count routes:
  - uri: /get
    upstream:
      nodes:
        "httpbin:80": 1
    plugins:
      key-auth: ~                      #1

consumers:
  - username: johndoe
    plugins:
      key-auth:
        key: john
      limit-count:
        count: 1                       #2
        time_window: 60
        rejected_code: 429
  - username: janedoe
    plugins:
      key-auth:
        key: jane
      limit-count:
        count: 5                       #2
        time_window: 60
        rejected_code: 429
#END routes:
  - uri: /get
    upstream:
      nodes:
        "httpbin:80": 1
    plugins:
      key-auth: ~                      #1

consumers:
  - username: johndoe
    plugins:
      key-auth:
        key: john
      limit-count:
        count: 1                       #2
        time_window: 60
        rejected_code: 429
  - username: janedoe
    plugins:
      key-auth:
        key: jane
      limit-count:
        count: 5                       #2
        time_window: 60
        rejected_code: 429
#END The route is only accessible to requests authenticating with key-auth johndoe has a lower limit count than janedoe. Did he forget to pay his subscription fees? The route is only accessible to requests authenticating with key-auth key-auth johndoe has a lower limit count than janedoe . Did he forget to pay his subscription fees? johndoe janedoe curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get The second request gets rate-limited. Per-group rate limiting We never attach permissions directly to identities in Identity Management systems. It's considered bad practice because when a person moves around the organization, we need to add and remove permissions one by one. The good practice is to attach permissions to groups and set the person in that group. When the person moves, we change their group; the person loses permissions from the old group and gets permissions from the new group. People get their permissions transitively via their groups. transitively Apache APISIX offers an abstraction called a Consumer Group for this. Consumer Group Let's create two consumer groups with different rate limit values: consumer_groups:
  - id: 1
    plugins:
      limit-count:
        count: 1
        time_window: 60
        rejected_code: 429
  - id: 2
    plugins:
      limit-count:
        count: 5
        time_window: 60
        rejected_code: 429 consumer_groups:
  - id: 1
    plugins:
      limit-count:
        count: 1
        time_window: 60
        rejected_code: 429
  - id: 2
    plugins:
      limit-count:
        count: 5
        time_window: 60
        rejected_code: 429 The next step is to attach consumers to these groups: consumers:
  - username: johndoe
    group_id: 1
    plugins:
      key-auth:
        key: john
  - username: janedoe
    group_id: 2
    plugins:
      key-auth:
        key: jane consumers:
  - username: johndoe
    group_id: 1
    plugins:
      key-auth:
        key: john
  - username: janedoe
    group_id: 2
    plugins:
      key-auth:
        key: jane curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get
curl -H 'apikey: jane' localhost:9080/get The second request gets rate-limited. We have the same results as before with two benefits. The first one is as I wrote above: when consumers move in and out, they change their permissions accordingly. The second benefit is that the limit count is shared among all consumers of a group.  Indeed, when you set a limit, you don't want each consumer to be rate limited at X requests per Y second; you want the group as a whole to share the limit. In this way, if a single consumer is very active, they will naturally cap the rate of other consumers who share the same group. Of course, you can set a limit on both a consumer and the group it belongs to. In this case, the lowest limit will apply first. consumers:
  - username: johndoe
    group_id: 2                        #1  
    plugins:
      key-auth:
        key: john
      limit-count:
        count: 1                       #2
        time_window: 60
        rejected_code: 429
  - username: janedoe
    group_id: 2
    plugins:
      key-auth:
        key: jane consumers:
  - username: johndoe
    group_id: 2                        #1  
    plugins:
      key-auth:
        key: john
      limit-count:
        count: 1                       #2
        time_window: 60
        rejected_code: 429
  - username: janedoe
    group_id: 2
    plugins:
      key-auth:
        key: jane Move johndoe to group 2 Limit him individually Move johndoe to group 2 johndoe Limit him individually curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: john' localhost:9080/get #1 curl -H 'apikey: john' localhost:9080/get
curl -H 'apikey: john' localhost:9080/get #1 johndoe hits the limit here, but janedoe now only has four requests left from this minute, as the former used one request johndoe hits the limit here, but janedoe now only has four requests left from this minute, as the former used one request johndoe janedoe Conclusion In this post, we implement rate limiting with Apache APISIX. We set the rate limit on a route and moved it to individual consumers. Then we moved it to consumer groups, so all consumers in a group share the same "pool". The complete source code for this post can be found on GitHub . GitHub To go further: To go further: Consumer Consumer Group Apache APISIX Hands-on Lab Consumer Consumer Consumer Group Consumer Group Apache APISIX Hands-on Lab Apache APISIX Hands-on Lab

Advanced Rate Limiting with Apache APISIX: Optimizing API Gateway Performance

How to Implement the Idempotency-Key Specification on Apache APISIX

Apache APISIX-এ আইডেমপোটেন্সি-কী স্পেসিফিকেশন কীভাবে প্রয়োগ করবেন

So implementieren Sie die Idempotency-Key-Spezifikation auf Apache APISIX

Cómo implementar la especificación de clave de idempotencia en Apache APISIX

Comment implémenter la spécification Idempotency-Key sur Apache APISIX

Apache APISIX पर Idempotency-Key विनिर्देश को कैसे लागू करें

Apache APISIX で Idempotency-Key 仕様を実装する方法

Apache APISIX에서 멱등성 키 사양을 구현하는 방법

Como implementar a especificação de chave de idempotência no Apache APISIX

Как реализовать спецификацию идемпотентного ключа в Apache APISIX

Idempotency-Key Belirtimi Apache APISIX'te Nasıl Uygulanır?

Cách triển khai Đặc tả khóa Idempotency trên Apache APISIX

如何在 Apache APISIX 上实现幂等密钥规范

In early 2021, I started to work on the project. I have to admit that I had never heard about it before. In this post, I’d like to introduce some Apache projects that are less well-known than HTTPD or Kafka. Apache APISIX Apache APISIX is an . It builds upon , a Lua layer built on top of the famous reverse-proxy. APISIX adds abstractions to the mix, , , , , and offers a plugin-based architecture. APISIX API Gateway OpenResty nginx e.g. Route Service Upstream Lots of plugins are provided out of the box: Transformation: , , gRPC, , etc. response-rewrite proxy-rewrite body-transformer Authentication: JWT, OPA, Keycloak, OpenID Connect, etc. Observability: metrics, logging, and traces Traffic: rate limiting, request validation, canary release, etc. Serverless: Azure functions, AWS Lambdas, OpenWhisk, etc. Messaging: Kafka, Dubbo, and MQTT Pre- and post-processing If no plugin fits your requirements, writing your own is possible. You can leverage APISIX on Kubernetes as an Ingress Controller. APISIX provides a Helm Chart for this. Apache ShardingSphere claims to offer an ecosystem able to transform any database into a distributed database system. It acts as a proxy between your code and your database(s). It comes in two flavors: ShardingSphere ShardingSphere-JDBC: a JDBC driver that acts as a proxy to your database(s). It’s only available for JVM-based applications. ShardingSphere-Proxy: a technology-independent deployable component. ShardingSphere offers several core features: Data Sharding is the core feature, as the project’s name implies. Most use cases focus on scaling purposes, but there are others, , requirements. e.g. data residency XA transactions for distributed transactions Read/write splitting Data encryption etc. Apache SeaTunnel is a data integration platform that offers the three pillars of data pipelines: sources, transforms, and sinks. It offers an abstract API over three possible engines: the Zeta engine from SeaTunnel or a wrapper around or . Be careful, as each engine comes with its own set of features. Apache SeaTunnel Apache Spark Apache Flink The power of SeaTunnel comes from its rich connector ecosystem. It does provide traditional SQL connectors, , Oracle, PostgreSQL, and MySQL, and NoSQL ones, , MongoDB, Cassandra, and Elasticsearch. However, it also comes bundled with some original ones, including Jira, Google Sheets, and Notion. I have a particular fondness for the CDC connector sources over MongoDB, MySQL, and Microsoft SQL Server. e.g. e.g. SeaTunnel comes with a web UI, which provides visual management of jobs, scheduling, running, and monitoring capabilities. Apache SkyWalking is an APM tool, focusing on microservices, Cloud Native apps, and Kuernetes architectures. It builds its architecture on four kinds of components: Apache SkyWalking collect telemetry data (metrics, logs, traces, and events). They support multiple output formats, including OpenTelemetry. Probes The aggregates and processes data platform The offers an interface over a supported backend. Supported backends include ElasticSearch, H2, MySQL, TiDB, and BanyanDB, a custom storage engine developed for SkyWalking storage Finally, a web allows visualizing SkyWalking’s data UI Skywalking supports a couple of formats, including OpenTelemetry. Given the industry’s current focus on OpenTelemetry, I recommend seriously considering this option. Apache Doris is a real-time data warehouse. Apache Doris Doris promotes four primary scenarios: Reporting analysis Ad-Hoc query Unified data warehouse construction Data lake query Doris is mostly MySQL compliant so that you can use a regular MySQL client. Discussion The Apache Foundation hosts the projects above, but they have another thing in common: they were all incepted in China. Have a look at the . You’ll probably be amazed at the sheer number; it’s close to 300! Apache project list In recent years, the number of projects incepted at the Apache Foundation has increased drastically. Look again at the list; I’m sure you only know a few of them - lots come from China. The trend is only growing; it’s a great move to integrate China with the OpenSource world! Just as I finish this post, my friend Stefano Fago has posted on another relevant project, , a streaming data lake platform. Apache Paimon To go further: Apache APISIX Apache ShardingSphere Apache SeaTunnel Apache SkyWalking Apache Doris Originally at A Java Geek on December 17th, 2023 published

Five Apache Projects You Don't (But Should) Know About

is an API Gateway, which builds upon the reverse-proxy to offer a plugin-based architecture. The main benefit of such an architecture is that it brings structure to the configuration of routes. It’s a help at scale, when managing hundreds or thousands of routes. Apache APISIX OpenResty In this post, I’d like to describe how plugins, priority, and phases play together and what pitfalls you must be aware of. APISIX plugin’s priority When you configure a route with multiple plugins, Apache APISIX needs to execute them in a order so that the results are the same over time. For this reason, every APISIX plugin has a . You can check a plugin priority directly in the code. For example, here’s the relevant code fragment for the plugin: consistent hardcoded priority basic-auth local _M = {
    version = 0.1,
    priority = 2520,                                 #1
    type = 'auth',
    name = plugin_name,
    schema = schema,
    consumer_schema = consumer_schema
} Priority is 2520 For documentation purposes, the file lists the priority of all out-of-the-box plugins. Here’s an excerpt: default-config.yaml plugins:
  - ldap-auth                      # priority: 2540
  - hmac-auth                      # priority: 2530
  - basic-auth                     # priority: 2520
  - jwt-auth                       # priority: 2510
  - key-auth                       # priority: 2500
  - consumer-restriction           # priority: 2400
  - forward-auth                   # priority: 2002
  - opa                            # priority: 2001
  - authz-keycloak                 # priority: 2000
  #- error-log-logger              # priority: 1091
  - proxy-cache                    # priority: 1085
  - body-transformer               # priority: 1080
  - proxy-mirror                   # priority: 1010
  - proxy-rewrite                  # priority: 1008
  - workflow                        # priority: 1006
  - api-breaker                    # priority: 1005
  - limit-conn                     # priority: 1003
  - limit-count                    # priority: 1002
  - limit-req                      # priority: 1001 Imagine a route configured with and . Because of their respective priority, would run before . proxy-mirror proxy-rewrite proxy-mirror proxy-rewrite upstream:
  - id: 1
    nodes:
      "oldapi:8081": 1

routes:
  - id: 1
    uri: "/v1/hello*"
    upstream_id: 1
    plugins:
      proxy-rewrite:                                 #1-3
        regex_uri: ["/v1(.*)", "$1"]
      proxy-mirror:                                  #2
        host: http://new.api:8082 The plugin ordering in this file is not relevant has priority 1010 and will execute first proxy-mirror has priority 1008 and will run second proxy-rewrite The above setup has an issue. For example, if we call , APISIX will first mirror the request, and then remove the prefix. Hence, the new API receives instead of . It’s possible to override the default priority to fix it: localhost:9080/v1/hello /v1 /v1/hello /hello routes:
  - id: 1
    uri: "/v1/hello*"
    upstream_id: 1
    plugins:
      proxy-rewrite:
        _meta:
          priority: 1020                             #1
        regex_uri: ["/v1(.*)", "$1"]
      proxy-mirror:
        host: http://new.api:8082 Override the default priority Now, has higher priority than : the former runs before the latter. proxy-rewrite proxy-mirror In this case, it works flawlessly; in others, it might not. Let’s dive further. APISIX phases APISIX runs plugins not only according to their priorities but also through dedicated phases. Because APISIX builds upon OpenResty, which builds upon ngxinx, the phases are very similar to the phases of these two components. nginx defines several an HTTP request goes through: phases NGX_HTTP_POST_READ_PHASE NGX_HTTP_SERVER_REWRITE_PHASE NGX_HTTP_FIND_CONFIG_PHASE NGX_HTTP_REWRITE_PHASE NGX_HTTP_POST_REWRITE_PHASE NGX_HTTP_PREACCESS_PHASE NGX_HTTP_ACCESS_PHASE NGX_HTTP_POST_ACCESS_PHASE NGX_HTTP_PRECONTENT_PHASE NGX_HTTP_CONTENT_PHASE NGX_HTTP_LOG_PHASE Each phase focuses on a task, , verifies that the client is authorized to make the request. i.e. NGX_HTTP_ACCESS_PHASE In turn, OpenResty offers similarly named phases. From nginx documentation: Finally, here are the phases of Apache APISIX: rewrite access before_proxy header_filter body_filter log We have seen two ways to order plugins: and Now comes the most important rule: ! by priority by phase. order by priority only works inside a phase For example, the plugin has a priority of 900 and runs in phase ; the plugin has a priority of 995 and runs in phase . Regardless of their respective priorities, will happen before , because happens before . Likewise, changing a priority won’t "move" a plugin out of its phase. redirect rewrite gzip body_filter redirect gzip rewrite body_filter The example above with and worked because both run in the phase. proxy-mirror proxy-rewrite rewrite The main issue is that priority is documented in the file, while the phase is buried in the code. Worse, some plugins run across different phases. For example, let’s check the proxy plugin and, more precisely, the functions defined there: config-default.yaml proxy-rewrite local function is_new_headers_conf(headers) local function check_set_headers(headers) function _M.check_schema(conf) function _M.rewrite(conf, ctx) The name of the function is suspiciously similar to one of the phases above. Looking at other plugins, we see the same pattern repeat. Apache APISIX runs plugin functions with the same name as the phase. rewrite() I took the liberty of summarizing all plugins and their respective phases in a table. +----------------------+---------+--------+-------------+--------------+------------+-----+
| Plugin               | Phase                                                            |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| General              | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| redirect             | X       |        |             |              |            |     |
| echo                 |         |        |             | X            | X          |     |
| gzip                 |         |        |             | X            |            |     |
| real-ip              | X       |        |             |              |            |     |
| ext-plugin-pre-req   | X       |        |             |              |            |     |
| ext-plugin-post-req  |         | X      |             |              |            |     |
| ext-plugin-post-resp |         |        | X           |              |            |     |
| workflow             |         | X      |             |              |            |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Transformation       | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| response-rewrite     |         |        |             | X            | X          |     |
| proxy-rewrite        | X       |        |             |              |            |     |
| grpc-transcode       |         | X      |             | X            | X          |     |
| grpc-web             |         | X      |             | X            | X          |     |
| fault-injection      | X       |        |             |              |            |     |
| mocking              |         | X      |             |              |            |     |
| degraphql            |         | X      |             |              |            |     |
| body-transformer     | X       |        |             | X            | X          |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Authentication       | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| key-auth             | X       |        |             |              |            |     |
| jwt-auth             | X       |        |             |              |            |     |
| basic-auth           | X       |        |             |              |            |     |
| authz-keycloak       |         | X      |             |              |            |     |
| authz-casdoor        |         | X      |             |              |            |     |
| wolf-rbac            | X       |        |             |              |            |     |
| openid-connect       | X       |        |             |              |            |     |
| cas-auth             |         | X      |             |              |            |     |
| hmac-auth            | X       |        |             |              |            |     |
| authz-casbin         | X       |        |             |              |            |     |
| ldap-auth            | X       |        |             |              |            |     |
| opa                  |         | X      |             |              |            |     |
| forward-auth         |         | X      |             |              |            |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Security             | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| cors                 | X       |        |             | X            |            |     |
| uri-blocker          | X       |        |             |              |            |     |
| ua-restriction       |         | X      |             |              |            |     |
| referer-restriction  |         | X      |             |              |            |     |
| consumer-restriction |         | X      |             |              |            |     |
| csrf                 |         | X      |             | X            |            |     |
| public-api           |         | X      |             |              |            |     |
| chaitin-waf          |         | X      |             |              |            |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Traffic              | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| limit-req            |         | X      |             |              |            |     |
| limit-conn           |         | X      |             |              |            | X   |
| limit-count          |         | X      |             |              |            |     |
| proxy-cache (init)   |         | X      |             | X            | X          |     |
| proxy-cache (disk)   |         | X      |             | X            |            |     |
| proxy-cache (memory) |         | X      |             | X            | X          |     |
| request-validation   | X       |        |             |              |            |     |
| proxy-mirror         | X       |        |             |              |            |     |
| api-breaker          |         | X      |             |              |            | X   |
| traffic-split        |         | X      |             |              |            |     |
| request-id           | X       |        |             | X            |            |     |
| proxy-control        | X       |        |             |              |            |     |
| client-control       | X       |        |             |              |            |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Observability        | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| zipkin               | X       | X      |             | X            |            | X   |
| skywalking           | X       |        |             |              |            | X   |
| opentelemetry        | X       |        |             |              |            | X   |
| prometheus           |         |        |             |              |            |     |
| node-status          |         |        |             |              |            |     |
| datadog              |         |        |             |              |            | X   |
| http-logger          |         |        |             |              | X          | X   |
| skywalking-logger    |         |        |             |              |            | X   |
| tcp-logger           |         |        |             |              |            | X   |
| kafka-logger         |         |        |             |              | X          | X   |
| rocketmq-logger      |         |        |             |              | X          | X   |
| udp-logger           |         |        |             |              |            | X   |
| clickhouse-logger    |         |        |             |              | X          | X   |
| syslog               |         |        |             |              |            | X   |
| log-rotate           |         |        |             |              |            |     |
| error-log-logger     |         |        |             |              |            |     |
| sls-logger           |         |        |             |              |            | X   |
| google-cloud-logging |         |        |             |              |            | X   |
| splunk-hec-logging   |         |        |             |              |            | X   |
| file-logger          |         |        |             |              | X          | X   |
| loggly               |         |        |             |              | X          | X   |
| elasticsearch-logger |         |        |             |              |            | X   |
| tencent-cloud-cls    |         | X      |             |              | X          | X   |
| loki-logger          |         |        |             |              | X          | X   |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Others               | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| serverless           |         | X      |             |              |            |     |
| openwhisk            |         | X      |             |              |            |     |
| dubbo-proxy          |         | X      |             |              |            |     |
| kafka-proxy          |         | X      |             |              |            |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+ Conclusion I’ve detailed Apache APISIX plugin phases and priorities in this post. I’ve explained their relationship with one another. Icing on the cake, I documented each out-of-the-box plugin’s phase(s). I hope it will prove helpful. To go further: Default plugin priorities Apache APISIX plugin execution lifecycle Plugin execution code Originally published at , on December 10th, 2023 A Java Geek

Navigating Apache APISIX: A Strategic Approach to Plugin Priorities and Execution Phases

Let's learn about via these 74 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Observability /Learn Repo 1. So, How are Observability and Monitoring Different, Actually? The similarities and differences between monitoring and observability and how to pair the two strategies 2. Observability Tips and Tricks For Using Grafana and Prometheus Rapid prototyping tips when using Grafana and Prometheus observability and monitoring stack. 3. Observability and Monitoring in a nutshell A guide to understanding the concepts and differences between monitoring and observability. 4. Debugging AWS Lambda Runtime Errors Debugging AWS Lambda can get tricky in Amazon CloudWatch. KloudMate lets you get to Lambda errors quickly and effectively using Lambda error logs. 5. An Intro to Dynamic CDK Dashboards How to create a construct based on other constructs in the same stack without forcing the developer to call a method for each? That's exactly what aspects do! 6. N/A please delete this 7. Developer Trends from Thoughtworks’ Technology Radar 27 Thoughtworks’ Technology Radar is a regular time to take a look at what trends are changing our industry. 8. How Observability and Monitoring Produces Better Software An article focused on deep diving into observability and its significance in software. Its history, goals, the importance of observability, and the issues that 9. How to Build Your Own Observability Platform on Kubernettes End-to-end full observability using Odigos and other back-end observability tools. 10. Maintaining Quality When Transitioning from Monolith to Microservices Piece by piece, legacy monolith applications are being broken down and replaced by microservices. 11. How to Monitor Apache Flink with OpenTelemetry Apache Flink monitoring support is now available in the open source OpenTelemetry collector. 12. Observability vs Monitoring: What's the Difference? Monitoring has been a basic system to track the health of servers for years. Now it is not enough. 13. Observability and Monitoring Have a Symbiotic Relationship, but They Are Different Observability vs. monitoring, what is the difference? Monitoring is the what to observability’s why. Here we dig into the differences. 14. Microservice.add(observability) != Microservice.add(monitoring) You are reading this content, which means that you are not novice to the microservices field. So let me just  scratch the surface of it before moving to Observable Microservices.  Once upon a time Monolith Application was now transformed into Microservices based application. 15. How Do You Make Cloud Migration Easier? Engage in intelligent change. 16. Application Monitoring: Closing Observability Gaps with Custom Metrics Which application metrics should you collect for your microservices? 17. The Challenges of Modern Observability It’s tough, impossible, to get a holistic view of your entire application when it’s running on 200 pods deployed to 12 nodes distributed across the world. 18. Why OpenTelemetry Should Matter to Network and Systems Admins OpenTelemetry is young, even by internet standards. Born out of the merger of OpenTracing and OpenCensus projects at the Cloud Native Computing Foundation CNCF 19. What is the ideal memory size to lower costs of running a task on Lambda? Should you increase Lambda memory? It might sound crazy, but increasing your AWS Lambda memory could actually lower your bills. Find out how. 20. What "Shifting Left" in Software Really Means for Blameless DevOps A truly "blameless culture" in software must to evolve from incident reporting, to telemetry aimed at Proactive Observability in DevOps 21. CI/CD/CF — The "Missing-Link" in the DevOps Toolchain? Everyone is familiar with CI and CD processes, but whatever happened to Continuous Feedback? Leveraging observability in dev creates a new type of dev process 22. Why Your Monitoring Dashboard May Be Feeding You Phantom Metrics We trust our metrics to show us the status of our system and where it misbehaves. But do our metrics show us what really happened? 23. Cloud Native Applications: 4 Building Blocks You Need in Your Platform To create amazing cloud native applications your platform needs these components: Kubernetes, Continuous Delivery Pipelines, Edge Stack, and Observability. This article will explain the importance of each component. 24. Comparing Different Serverless Monitoring Platforms Technology touches almost every corner of the world economy. Even when it’s an indirect relation, in many cases tech is an essential, vital part of our societies. It just can’t fail without causing too much distress and losses. Not only financially, but especially to the human aspect. 25. Get Started With Sidekick Open Source Live Debugger in 5 Mins Sidekick is a live application debugger that lets you troubleshoot your applications while they keep on running. Here is how you can start using it in 5 minutes 26. Logging in Observability - Part 1 rsyslog vs journald vs filebeat 27. Managing Collectors on K8 – Why We Chose the OpenTelemetry Collector for Odigos Without proper implementation and continuous configuration of your collectors, observability tools will be limited at best at best and many times ineffectual. 28. On Dynamic Observability and Team Culture with Liran Haimovitch, Rookout CTO Rookout Co-Founder and CTO, Liran Haimovitch, shares the origin story of their debugging tool, what excites him about the startup life, PLG, and more. 29. 9 Data Trends You’ll See in 2023 2022 saw the data space grow by leaps and bounds. Here are the top 9 things our team of data experts expects to see in 2023. 30. Observability and AI Partner Well to Tackle All of That Data Gone are the days when operations and development were a back-office affair. 31. Logging in Observability - Part 2 In this article, I want to describe how to work with logs, analyze them in the command line, and we will consider new modern tools to visualize logs. 32. Tracing: An Essential Component of Observability In this article, I am going to shed light on one important aspect of observability - tracing. 33. Microservice Observability Patterns [Part 1] Logging is one of the most important parts of software systems. Whether you have just started working on a new piece of software, or your system is running in a large scale production environment, you’ll always find yourself seeking help from log files. Logs are the first thing people look for when something goes wrong, or something doesn’t work as expected. 34. What is Observability? Is it Cultural? In order to leverage observability we need a significant shift in our corporate culture that encapsulates the entire company & goes beyond the tools. 35. Murre - the lightweight K8s metrics monitoring tool 36. Introductory Guide to Debugging Lambda Configuration Errors The “Lambda configuration error” is as generic as it gets but at the end of the day, it's a pathing issue. Let's go over how to resolve this. 37. Contextual Intelligence and Observability: Without the Former, You Really Don’t Have the Latter Observability is a hot term in the industry, but don’t let it fool you: having visibility into your organization's apps and services only gives you a piece. 38. The Everything Guide to Data Collection in DevSecOps Collecting wide swaths of observability and security data is key to a high-quality digital experience. Find out what you need to know to get started. 39. Better Performance and Security by Monitoring Logs, Metrics, and More Monitoring is a crucial part of observability. Learn how monitoring can specifically improve security, performance, and reliability. 40. Goodbye, Cost-depth Trade Off: Cloud-Native Observability, Redefined It turns out that, by rethinking the fundamentals of your approach to monitoring, you can have it all 41. Prometheus — Easy In, Difficult Out Setting up a Prometheus server can be easy, but getting actionable information out is a very long and never-ending journey. 42. Choosing a Computing Method: a Serverless SWOT Analysis If you ever find yourself deciding for or against serverless the following tries to make the decision easier for you. 43. Setting Up AWS CloudWatch Alerts (vs Dashbird Alerts) To Monitor Your Applications Learn about the best practices for AWS Cloudwatch Alerts and Dashbird Alarms, to not miss out on critical info about your serverless app. 44. Exploring Differences Between Monitoring And Observability Monitoring vs Observability: in this article, we're explaining what is observability exactly and how does it differ from monitoring. 45. Kuma and Prometheus for Observability in Kubernetes Microservices Clusters A year ago, Harry Bagdi wrote an amazingly helpful blog post (link at bottom of article) on observability for microservices. And by comparing titles, it becomes obvious that my blog post draws inspiration from his work. 46. To Log Or Not To Log, Is The (Eternal) Question An alternative logging strategy to make loggers your friends, not enemies 47. 2023 Observability Predictions: Open Source is the Key to Explosive Growth The observability industry is projected to grow about 8% in 2023, a continuation of a strong growth trend since the 90’s. 48. How Goji Investments Enhances Developer Experience via Observability This post was written by Dean Record, Engineer at Goji Investments. 49. What is API Observability? APIs are the blood vessels of digital business. 50. A Guide to Deploying Jaeger on Kubernetes in Production Logs, metrics, and traces are the three pillars of the Observability world. The distributed tracing world, in particular, has seen a lot of innovation in recent months, with OpenTelemetry standardization and with Jaeger open source project graduating from the CNCF incubation. 51. How To Use a FastAPI Abstraction To Query AWS Data Lake Our guide in creating FastAPI abstraction that allows us to query the AWS Glue metadata catalog - a Data Discovery API for AWS Data Lake. 52. Sizing Your Cloud Resources: 10 Common Mistakes to Avoid Learn the common pitfalls of moving to the cloud and how you can avoid them to truly benefit from the cloud’s elasticity. 53. CNCF Tools Overview: Are You Cloud-Native? Cloud computing is becoming more and more of a household name, with even the most conservative fields of business figuring out how to make the best use of it. Cloud computing usually starts with running a private cloud solution on premises before venturing onto the public cloud. Of course, the cloud is not a single uniform being. It may come from different providers, Amazon Web Services, Google Cloud Platform, and Microsoft Azure being the biggest players here. Or it may come with different visibility and hosting, that is, public (resides with the provider), private (self-hosted), or hybrid (which uses a bit of both). And the cloud can use different tools and APIs for management as well. 54. Microservice Observability Patterns [Part 2] In my previous article, I talked about the importance of logs and the differences between structured and unstructured logging. Logs are easy to integrate into your application and provide the ability to represent any type of data in the form of strings. 55. Improving Observability Through Service Level Metrics 56. Observability on Heroku: How to Monitor Apps on a Managed Infrastructure Cloud platforms like Heroku make it easier than ever to host applications: just upload your code, and they’ll deploy it for you. But a common misconception is that, because you don’t own the infrastructure, you can’t really monitor your applications or see under the hood. 57. Setting Up Notification Systems to Observe and Analyze Your Application By building observability and analytics into your notification system, you can identify and quickly resolve issues by monitoring how your product is performing. 58. Better Failure Detection in Serverless Apps Having reliable failure detection in place is a must for any application in production. Here's how to best achieve it for AWS Lambda. 59. 4 Key Observability Metrics for Distributed Applications A common architectural design pattern these days is to break up an application monolith into smaller microservices. Each microservice is then responsible for a specific aspect or feature of your app. For example, one microservice might be responsible for serving external API requests, while another might handle data fetching for your frontend. 60. Lua-Land CPU Flame Graphs in OpenResty XRay This post will introduce the idea of Lua-land CPU flame graphs and use OpenResty XRay to produce real flame graphs for several small and standalone Lua examples 61. 5 Key Takeaways from New Relic’s 2022 State of Logs Report The 2022 State of Logs Report provides a detailed view of how this practice is shaping engineering and the technologies of the future. 62. Introduction to Observability in ITOM and AIOps Observability is a best practice implemented by AIOps, enabling automation and expanding visibility into the entire organizational ecosystem. 63. Internal AWS Monitoring is Hard to Grasp Learm how a serverless monitoring solution can catch problems for you without the painful learning curve connected to serverless failure detection. 64. Jaeger Persistent Storage with Elasticsearch, Cassandra and Kafka Running systems in production involve requirements for high availability, resilience and recovery from failure. When running cloud-native applications this becomes even more critical, as the base assumption in such environments is that compute nodes will suffer outages, Kubernetes nodes will go down and microservices instances are likely to fail, yet the service is expected to remain up and running. 65. How to Monitor Serverless Applications With AWS CloudWatch Alarms Running any application in production assumes reliable monitoring to be in place and serverless applications are no exception. 66. How to Optimize Large Scale Serverless Applications for Operational Excellence Managing applications at scale often comes up as one of the biggest concerns for businesses; How can it work smoothly? How do we monitor so many resources? How do we maintain best practices with constantly evolving infrastructure? In this article, we run through the best approach for operational excellence looking at serverless monitoring strategy, serverless alerting strategy, and security and compliance best practices. 67. Fixing The ClickHouse Node Failure On Distributed Systems - A How-To Guide Part One: ClickHouse Failures, by Marcel Birkner 68. Introduction To Distributed Tracing Pattern A distributed architecture brings in several challenges when it comes to operability and monitoring. Here, one may be dealing with tens if not hundreds of microservices, each of which may or may not have been built by the same team. 69. 10 Threats to an Open API Ecosystem Despite tight economic situations worldwide, the API economy continues to grow. 70. Data Access for Microservices If you want to access data in a distributed environment such as in a microservice architecture, then data services are the way to go. The idea is to create a data abstraction layer (DAL) that the rest of the system’s applications and services can share. Thus, a data service gives you a generalized interface to the data you’re exposing and provides access to it in a standard manner. This would be in a well-understood protocol and a known data format. For example, a popular approach is to use JSON via HTTP/S. 71. Rethinking Programming: Automated Observability Introduction 72. Watching the Requests Go By: Reconstructing an API Spec with APIClarity In this post, we’re going to look at what API reconstruction is and how APIClarity solves the API observability problem. 73. Navigate Your Way to Production Bliss with Caretta The easiest way to map our cluster: Caretta - a standalone OSS tool, leveraging eBPF technology to be lightweight and frictionless 74. The Importance of Monitoring Big Data Analytics Pipelines In this article, we first explain the requirements for monitoring your big data analytics pipeline and then we go into the key aspects that you need to consider to build a system that provides holistic observability. Thank you for checking out the 74 most read stories about Observability on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

74 Stories To Learn About Observability

171 Stories To Learn About Monitoring

2022 is over, and not a moment too soon. I'll never forget it: some of my friends had to flee their own country; others are fighting for their freedom as I write this post. I hope they will be safe and that their wishes will come true in 2023. On the personal and technical side, here's a summary of the past year from my perspective. Job Change First and foremost, I changed jobs. I worked for for 3½ years. However, I started to become dissatisfied with the company's direction and my position within it. Hazelcast I looked for a couple of months for my next gig. In February, I started to work on : Apache APISIX Full Lifecycle API Management API Gateway, Ingress Controller, etc. Apache APISIX provides rich traffic management features like Load Balancing, Dynamic Upstream, Canary Release, Circuit Breaking, Authentication, Observability, etc. When I joined Hazelcast, I had to distributed systems and concepts related to them: clocks, order, split brain, leader, etc. In comparison, an API Gateway such as APISIX is pretty simple. Hence, my focus is more on underlying technologies: understand learning NGINX OpenResty Lua Kubernetes Ingress etc. It means I had to shift the subjects I write about: you won't find any Java and hardly any Kotlin. My Blog In 2022, I published 50 blog posts on : one each week on Sunday, but on Christmas and the New Year - for obvious reasons. Here are the top 5 most viewed pages: https://blog.frankel.ch/ Rank Post Views % Avg. time on page 1 Hard things in Computer Science 26,123 5.29 00:05:35 2 A poor man's API 21,297 4.32 00:05:04 3 Discussing Backend For Front-end 18,626 3.78 00:04:57 4 Homepage 18,062 3.66 00:00:59 5 A list of cache providers 14,646 2.97 00:03:32 Two things are of interest: The 4th most viewed page is the homepage. The 5th, about cache providers, was published in October 2021. It's a long tail, indeed! Views are broken down as the following during the year: You can notice a couple of huge spikes. They are front page occurrences. I advertise on HN when I believe the post has potential; sometimes, others do. Most viewed posts correlate somewhat with referrals from Hacker News. Hacker News Rank Post Views % 1 A poor man's API 10,924 38.57 2 Discussing Backend For Front-end 8,128 28.69 3 Hard things in Computer Science 4,072 14.38 4 What I miss in Java, the perspective of a Kotlin developer 2,180 7.70 5 Toying with Kotlin's context receivers 1,023 3.61 Cross-Posting 2022 also saw consistent cross-posting on third-party content aggregator sites, which allow linking to the original post. As a reminder, Google (and other search sites) flag similar content as duplicate. If the aggregator site has more "page rank" than your site, then the content on the latter will be marked as such. The attribute is the way to tell Google: here's the original content, don't flag either as duplicate. I only post on sites with such a feature. rel canonical That being said, here are my followers depending on the site: : 564 Medium : 1,838 Dev.to : 80 Hashnode : no follow feature DZone : no follow feature Hacker Noon : no follow feature either foojay.io It's interesting to see that though I thought focused on web and front-end, I got many followers though I rarely write on such subjects. Conversely, I still need to understand why my follower base on Hashnode doesn't take off. dev.to Public Speaking Indeed, 2022 was not my best year in terms of conference attendance. Here's a comparison of the previous years: Raw numbers are a bit misleading: because of Covid, I spoke at many events online in 2020 and 2021. It's easier to do so than traveling to another country (or continent!); on the other hand, engagement is much lower, not to mention the lack of social interactions. much Anyway, 2022 numbers are below 2019's, the last "regular" year. Despite this, I'm ok with it: 2019 was hard on me, and conference attendance is more of a marathon than a sprint. I think I've found the sweet spot between public speaking and other activities. Regarding content, I mainly did two talks: Evolving your APIs Chopping the monolith It's below the number of talks I usually try to achieve, but they were pretty popular, especially the first one. I'm working on more diversity for 2023. GitHub Contributions As a Developer Advocate, my are different from the ones of a regular software developer, especially one working on Open Source. GitHub contributions Most of my contributions are commits because I'm the only one working on my demos. Furthermore, I regularly update them with the latest version of whatever dependency I use. I'm trying to open issues on Apache APISIX since I'm a huge user. Eating one's dog food is a great way to uncover either bugs or usability improvements. Finally, pull requests are for blog posts and reviews for my colleagues' blog posts. Conclusion Years when one changes jobs are always challenging but very interesting; 2022 was no different. My resolution for 2023 are: To deepen my understanding of the Apache APISIX ecosystem. Write as many blog posts as in 2022 - it's hard to do better, anyway. Design at least three new talks What are yours? Originally published at on January 8th, 2023 A Java Geek Photo by on Michael Skok Unsplash

Looking Back on My 2022: A Year in Review

The subject of Web resource caching is as old as the World Wide Web itself. However, I'd like to offer an as-exhaustive-as-possible catalog of how one can improve performance by caching. Web resource caching can happen in two different places: client-side - on the browser and server side. In my , I explained the former; this post focuses on the latter. previous post While client-side caching works well, it has one central issue: to serve the resource locally, it must first have it in the cache. Thus, each client needs its cached resource. If the requested resource is intensive to compute, it doesn't scale. The idea behind server-side caching is to compute the resource once and serve it from the cache to all clients. A couple of dedicated server-side resource caching solutions have emerged over the years: , , , etc. Other solutions are less focused on web resource caching and more generic, , or . Memcached Varnish Squid e.g. Redis Hazelcast If you want to dive deeper into generic caching solutions, please check these on the subject. two posts To continue with the sample from last week, I'll use Apache APISIX to demo server-side caching. APISIX relies on the plugin for caching. Unfortunately, at the moment, APISIX doesn't integrate with any third-party caching solution. It offers two options: memory-based and disk-based. proxy-cache In general, the former is faster, but memory is expensive, while the latter is slower, but disk storage is cheap. Within OpenResty, however, the disk option may be faster because of how LuaJIT handles memory. You should probably start with the disk, and if it's not fast enough, mount . /dev/shm Here are my new routes: routes:
  - uri: /cache
    upstream_id: 1
    plugins:
      proxy-rewrite:
        regex_uri: ["/cache(.*)", "/$1"]
      proxy-cache: ~ Note that the default cache key is the host and the request URI, which includes query parameters. The default configuration uses the default disk-based : proxy-cache configuration proxy_cache:                      # Proxy Caching configuration
    cache_ttl: 10s                  # The default caching time in disk if the upstream does not specify the cache time
    zones:                          # The parameters of a cache
      - name: disk_cache_one        # The name of the cache, administrator can specify
                                    # which cache to use by name in the admin api (disk|memory)
        memory_size: 50m            # The size of shared memory, it's used to store the cache index for
                                    # disk strategy, store cache content for memory strategy (disk|memory)
        disk_size: 1G               # The size of disk, it's used to store the cache data (disk)
        disk_path: /tmp/disk_cache_one  # The path to store the cache data (disk)
        cache_levels: 1:2           # The hierarchy levels of a cache (disk)
      - name: memory_cache
        memory_size: 50m We can test the setup with : curl curl -v localhost:9080/cache The response is interesting: < HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 147
< Connection: keep-alive
< Date: Tue, 29 Nov 2022 13:17:00 GMT
< Last-Modified: Wed, 23 Nov 2022 13:58:55 GMT
< ETag: "637e271f-93"
< Server: APISIX/3.0.0
< Apisix-Cache-Status: MISS                      #1
< Accept-Ranges: bytes Because the cache is empty, APISIX has a cache miss. Hence, the response is from the upstream If we again before the default cache expiration period (300 seconds), the response is from the cache: curl < HTTP/1.1 200 OK
...
< Apisix-Cache-Status: HIT After the expiration period, the response is from the upstream, but the header is explicit: < HTTP/1.1 200 OK
...
< Apisix-Cache-Status: EXPIRED Note that we can explicitly purge the entire cache by using the custom HTTP method: PURGE curl localhost:9080/cache -X PURGE After purging the cache, the above cycle starts anew. Note that it's also possible to bypass the cache, , for testing purposes. We can configure the plugin accordingly: e.g. routes:
  - uri: /cache*
    upstream_id: 1
      proxy-cache:
        cache_bypass: ["$arg_bypass"]       #1 Bypass the cache if you send a query parameter with a non- value bypass 0 curl -v localhost:9080/cache?bypass=please It serves the resource from the upstream regardless of the cache status: < HTTP/1.1 200 OK
...
< Apisix-Cache-Status: BYPASS For more details on all available configuration parameters, check the plugin. proxy-cache Conclusion This post was relatively straightforward. The most challenging issue with server-side caching is the configuration: what to cache, for how long, etc. Unfortunately, it depends significantly on your context, problems, and available resources. You probably need to apply Plan-Do-Check-Act: guesstimate a relevant configuration, apply it, measure the performance, and rinse and repeat until you find your sweet spot. I hope that with an understanding of both client-side and server-side caching, you'll be able to improve the performance of your applications. The source code is available on . GitHub To go further: Cache API responses proxy-cache plugin Originally published at on December 4<sup>th</sup>, 2022 A Java Geek

Understanding Web Resource Caching: Server-side

Creating a full-fledged API requires resources, both time and money. You need to think about the model, the design, the REST principles, etc., without writing a single line of code. Most of the time, you don't know whether it's worth it: you'd like to offer a Minimum Viable Product and iterate from there. I want to show how you can achieve it without writing a single line of code. The solution The main requirement of the solution is to use the . It's a well-established Open Source SQL database. PostgreSQL database Instead of writing our REST API, we use the PostgREST component: PostgREST is a standalone web server that turns your PostgreSQL database directly into a RESTful API. The structural constraints and permissions in the database determine the API endpoints and operations. -- PostgREST Let's apply it to a simple use case. Here's a table that I want to expose via a CRUD API: product Note that you can find the whole source code on to follow along. GitHub PostgREST's is pretty complete and works out of the box. Yet, I didn't find any ready-made Docker image, so I created my own: Getting Started guide FROM debian:bookworm-slim                                                   #1

ARG POSTGREST_VERSION=v10.1.1                                               #2
ARG POSTGREST_FILE=postgrest-$POSTGREST_VERSION-linux-static-x64.tar.xz     #2

RUN mkdir postgrest

WORKDIR postgrest

ADD https://github.com/PostgREST/postgrest/releases/download/$POSTGREST_VERSION/$POSTGREST_FILE \
    .                                                                       #3

RUN apt-get update && \
    apt-get install -y libpq-dev xz-utils && \
    tar xvf $POSTGREST_FILE && \
    rm $POSTGREST_FILE                                                      #4 Start from the latest Debian Parameterize the build Get the archive Install dependencies and unarchive The Docker image contains a executable in the folder. We can "deploy" the architecture via Docker Compose: postgrest /postgrest version: "3"
services:
  postgrest:
    build: ./postgrest                                   #1
    volumes:
      - ./postgrest/product.conf:/etc/product.conf:ro    #2
    ports:
      - "3000:3000"
    entrypoint: ["/postgrest/postgrest"]                 #3
    command: ["/etc/product.conf"]                       #4
    depends_on:
      - postgres
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD: "root"
    volumes:
      - ./postgres:/docker-entrypoint-initdb.d:ro       #5 Build the above Dockerfile Share the configuration file Run the executable postgrest With the configuration file Initialize the schema, the permissions, and the data At this point, we can query the table: product curl localhost:3000/product We immediately get the results: [{"id":1,"name":"Stickers pack","description":"A pack of rad stickers to display on your laptop or wherever you feel like. Show your love for Apache APISIX","price":0.49,"hero":false}, 
 {"id":2,"name":"Lapel pin","description":"With this \"Powered by Apache APISIX\" lapel pin, support your favorite API Gateway and let everybody know about it.","price":1.49,"hero":false}, 
 {"id":3,"name":"Tee-Shirt","description":"The classic geek product! At a conference, at home, at work, this tee-shirt will be your best friend.","price":9.99,"hero":true}] That was a quick win! Improving the solution Though the solution works, it has a lot of room for improvement. For example, the database user cannot change the data, but everybody can actually access it. It might not be a big issue for product-related data, but what about medical data? The PostgREST documentation is aware of it and explicitly advises using a reverse proxy: PostgREST is a fast way to construct a RESTful API. Its default behavior is great for scaffolding in development. When it’s time to go to production it works great too, as long as you take precautions. PostgREST is a small sharp tool that focuses on performing the API-to-database mapping. We rely on a reverse proxy like Nginx for additional safeguards. -- Hardening PostgREST Instead of nginx, we would benefit from a full-fledged API Gateway: enters . We shall add it to our Docker Compose: Apache APISIX version: "3"
services:
  apisix:
    image: apache/apisix:2.15.0-alpine                              #1
    volumes:
      - ./apisix/config.yml:/usr/local/apisix/conf/config.yaml:ro
    ports:
      - "9080:9080"
    restart: always
    depends_on:
      - etcd
      - postgrest
  etcd:
    image: bitnami/etcd:3.5.2                                       #2
    environment:
      ETCD_ENABLE_V2: "true"
      ALLOW_NONE_AUTHENTICATION: "yes"
      ETCD_ADVERTISE_CLIENT_URLS: "http://0.0.0.0:2397"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2397" Use Apache APISIX APISIX stores its configuration in etcd We shall first configure Apache APISIX to proxy calls to : postgrest curl http://apisix:9080/apisix/admin/upstreams/1 -H 'X-API-KEY: 123xyz' -X PUT -d ' #1-2
{
  "type": "roundrobin",
  "nodes": {
    "postgrest:3000": 1                                                             #1-3
  }
}'

curl http://apisix:9080/apisix/admin/routes/1 -H 'X-API-KEY: 123xyz' -X PUT -d '    #4
{
  "uri": "/*",
  "upstream_id": 1
}' Should be run in one of the Docker nodes, so use the Docker image's name. Alternatively, use but be sure to expose the ports localhost Create a reusable upstream Point to the PostgREST node Create a to the created route upstream We can now query the endpoint via APISIX: curl localhost:9080/product It returns the same result as above. DDoS protection We haven't added anything, but we're ready to start the work. Let's first protect our API from Distributed Denial of Service attacks. Apache APISIX is designed around a plugin architecture. To protect from DDoS, we shall use a plugin. We can set plugins on a specific when it's created or on every ; in the latter case, it's a . We want to protect every route by default, so we shall use one. route route global rule curl http://apisix:9080/apisix/admin/global_rules/1 -H 'X-API-KEY: 123xyz' -X PUT -d '
{
  "plugins": {
    "limit-count": {                 #1
      "count": 1,                    #2
      "time_window": 5,              #2
      "rejected_code": 429           #3
    }
  }
}' limits the number of calls in a time window limit-count Limit to 1 call per 5 seconds; it's for demo purposes Return ; the default is 429 Too Many Requests 503 Now, if we execute too many requests, Apache APISIX protects the upstream: curl localhost:9080/product <html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>openresty</center>
</body>
</html> Per-route authorization PostgREST also offers an Open API endpoint at the root. We thus have two routes: for the Open API spec and for the products. Suppose we want to disallow unauthorized people to access our data: Regular users can access products, while admin users can access both the Open API spec products. / /product and APISIX offers several . We will use the simplest one possible, . It relies on the abstraction. requires a specific header: the plugin does a reverse lookup on the value and finds the consumer whose key corresponds. authentication methods key-auth Consumer key-auth Here's how to create a consumer: curl http://apisix:9080/apisix/admin/consumers -H 'X-API-KEY: 123xyz' -X PUT -d '    #1
{
  "username": "admin",                                                               #2
  "plugins": {
    "key-auth": {
      "key": "admin"                                                                 #3
    }
  }
}' Create a new consumer Consumer's name Consumer's key value We do the same with consumer and key . Now, we can create a dedicated route and configure it so that only requests from pass through: user user admin curl http://apisix:9080/apisix/admin/routes -H 'X-API-KEY: 123xyz' -X POST -d ' #1
{
  "uri": "/",
  "upstream_id": 1,
  "plugins": {
    "key-auth": {},                                                             #2
    "consumer-restriction": {                                                   #2
      "whitelist": [ "admin" ]                                                  #3
    }
  }
}' Create a new route Use the and plugins key-auth consumer-restriction Only -authenticated requests can call the route admin Let's try the following: curl localhost:9080 It doesn't work as we are not authenticated via an API key header. {"message":"Missing API key found in request"} curl -H "apikey: user" localhost:9080 It doesn't work as we are authenticated as , but the route is not authorized for but for . user user admin {"message":"The consumer_name is forbidden."} curl -H "apikey: admin" localhost:9080 This time, it returns the Open API spec as expected. Monitoring A much-undervalued feature of any software system is monitoring. As soon as you deploy any component in production, you must monitor its health. Nowadays, many services are available to monitor. We will use as it's Open Source, battle-proven, and widespread. To display the data, we will rely on for the same reasons. Let's add the components to the Docker Compose file: Prometheus Grafana version: "3"
services:
  prometheus:
    image: prom/prometheus:v2.40.1                                    #1
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml    #2
    depends_on:
      - apisix
  grafana:
    image: grafana/grafana:8.5.15                                     #3
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning              #4
      - ./grafana/dashboards:/var/lib/grafana/dashboards              #4
      - ./grafana/config/grafana.ini:/etc/grafana/grafana.ini         #4-5
    ports:
      - "3001:3001"
    depends_on:
      - prometheus Prometheus image Prometheus configuration to scrape Apache APISIX. See the full file here Grafana image Grafana configuration. Most of it comes from the configuration by APISIX. provided Change the default port from to to avoid conflict with the PostgREST service 3000 3001 Once the monitoring infrastructure is in place, we only need to instruct APISIX to provide the data in a format that Prometheus expects. We can achieve it through configuration and a new global rule: plugin_attr:
  prometheus:
    export_addr:
      ip: "0.0.0.0"             #1
      port: 9091                #2 Bind to any address Bind to port . Prometheus metrics are available on on the Docker network 9091 http://apisix:9091/apisix/prometheus/metrics We can create the global rule: curl http://apisix:9080/apisix/admin/global_rules/2 -H 'X-API-KEY: 123xyz' -X PUT -d '
{
  "plugins": {
    "prometheus": {}
  }
}' Send a couple of queries and open the Grafana dashboard. It should look similar to this: Conclusion Creating a full-fledged REST(ful) API is a huge investment. One can quickly test a simple API by exposing one's database in a CRUD API via PostgREST. However, such an architecture is not fit for production usage. To fix it, you need to set a façade in front of PostgREST, a reverse proxy, or even better, an API Gateway. Apache APISIX offers a wide range of features, from authorization to monitoring. With it, you can quickly validate your API requirements at a low cost. The icing on the cake: when you've validated the requirements, you can keep the existing façade and replace PostgREST with your custom-developed API. The source code is available on . GitHub To go further: PostgREST Getting started with Apache APISIX Apache APISIX plugins Originally published at on November 20th, 2022 A Java Geek

Creating a Poor Man's API Using PostgREST 

Microservice architecture is nowadays almost a standard for backend development. An API gateway is an excellent way to connect a group of microservices to a single API accessible to users. API gateways are available from cloud providers such as AWS/Azure/Google Cloud Platform and Cloudflare. Kong is a scalable API gateway built on open source and can be an excellent alternative if you don't want to have your system locked into a particular vendor. This tutorial shows an example using Kong API gateway, , and . The illustration below shows you the final architecture we are going to build in this guide Ory Kratos Ory Oathkeeper The full source code for this tutorial is available on github What we will use gateway can be an excellent solution for an ingress load balancer and API gateway if you do not want vendor lock-in of any cloud API Gateways in your application. Kong uses and Lua. OpenResty extends Nginx with Lua scripting to use Nginx's event model for non-blocking I/O with HTTP clients and remote backends like PostgreSQL, Memcached, and Redis. OpenResty is not an Nginx fork, and Kong is not an Openresty fork. Kong uses OpenResty to enable . Kong OpenResty API gateway features acts like an identity and access proxy for our microservices. It allows us to proxy only authenticated requests to our microservices, so we don't need to implement middleware to check authentication. It can also transform requests, for example, convert session auth into JWT for a back-end service. Oathkeeper is the authentication provider; it handles all first-party authentication flows: username/password, forgot password, MFA/2FA, . It also provides OIDC/social login capabilities for example, "Login with GitHub". Kratos and more Building simple microservices Let's say we have two microservices: and . They are pretty simple and serve only to test our API gateway, but you can switch them out for more complex components. hello world The "World" microservice exposes a API endpoint and returns a simple JSON message: /world package main

import (
	"encoding/json"
	"log"
	"net/http"
)

type Response struct {
	Message string `json:"message"`
}

func helloJSON(w http.ResponseWriter, r *http.Request) {
	response := Response{Message: "World microservice"}
	w.Header().Set("Content-type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(response)
}
func main() {
	http.HandleFunc("/world", helloJSON)
	log.Fatal(http.ListenAndServe(":8090", nil))

} The "Hello" microservice exposes a API endpoint and returns a simple JSON message: /hello package main

import (
	"encoding/json"
	"log"
	"net/http"
)

type Response struct {
	Message string `json:"message"`
}

func helloJSON(w http.ResponseWriter, r *http.Request) {
	response := Response{Message: "Hello microservice"}
	w.Header().Set("Content-type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(response)
}
func main() {
	http.HandleFunc("/hello", helloJSON)
	log.Fatal(http.ListenAndServe(":8090", nil))

} We now want to secure access to these microservices and let only authenticated users access these endpoints. Okay. Let's start hacking, shall we? Ory Kratos setup Follow the guide to set up Ory Kratos. In this tutorial, you only need a docker-compose file with the following configuration: Quickstart postgres-kratos:
    image: postgres:9.6
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=kratos
      - POSTGRES_PASSWORD=secret
      - POSTGRES_DB=kratos
    networks:
      - intranet

  kratos-migrate:
    image: oryd/kratos:v0.8.0-alpha.3
    links:
      - postgres-kratos:postgres-kratos
    environment:
      - DSN=postgres://kratos:secret@postgres-kratos:5432/kratos?sslmode=disable&max_conns=20&max_idle_conns=4
    networks:
      - intranet
    volumes:
      - type: bind
        source: ./kratos
        target: /etc/config/kratos
    command: -c /etc/config/kratos/kratos.yml migrate sql -e --yes

  kratos:
    image: oryd/kratos:v0.8.0-alpha.3
    links:
      - postgres-kratos:postgres-kratos
    environment:
      - DSN=postgres://kratos:secret@postgres-kratos:5432/kratos?sslmode=disable&max_conns=20&max_idle_conns=4
    ports:
      - '4433:4433'
      - '4434:4434'
    volumes:
      - type: bind
        source: ./kratos
        target: /etc/config/kratos
    networks:
      - intranet
    command: serve -c /etc/config/kratos/kratos.yml --dev --watch-courier

  kratos-selfservice-ui-node:
    image: oryd/kratos-selfservice-ui-node:v0.8.0-alpha.3
    environment:
      - KRATOS_PUBLIC_URL=http://kratos:4433/
      - KRATOS_BROWSER_URL=http://127.0.0.1:4433/
    networks:
      - intranet
    ports:
      - "4455:3000"
    restart: on-failure


  mailslurper:
    image: oryd/mailslurper:latest-smtps
    ports:
      - '4436:4436'
      - '4437:4437'
    networks:
      - intranet Some notes on the network architecture: HTTP and are the public and admin API's of Ory Kratos. :4433 :4434 HTTP for Mailslurper - a mock Email server. You can get an activation link by accessing . :4436 http://127.0.0.1:4436 HTTP for the UI interface that allows one to start sign-up/login/recovery flows. :4455 After running you can open to test your configuration. docker-compose up http://127.0.0.1:4455/welcome Configuring Ory Oathkeeper Now we can start configuring our gateways for this example. Kong is the entry point for the network traffic. Ory Oathkeeper would be accessible from the internal network only in this case. Let's review our architecture diagram from before: Oathkeeper checks sessions and proxies traffic to our microservice while Kong provides ingress load balancing. We can even set up to have a more robust configuration for our service. Here is how we configure the access rules for Ory Oathkeeper: Round-Robin DNS -
  id: "api:hello-protected"
  upstream:
    preserve_host: true
    url: "http://hello:8090"
  match:
    url: "http://oathkeeper:4455/hello"
    methods:
      - GET
  authenticators:
    -
      handler: cookie_session
  mutators:
    - handler: noop
  authorizer:
    handler: allow
  errors:
    - handler: redirect
      config:
        to: http://127.0.0.1:4455/login
-
  id: "api:world-protected"
  upstream:
    preserve_host: true
    url: "http://world:8090"
  match:
    url: "http://oathkeeper:4455/world"
    methods:
      - GET
  authenticators:
    -
      handler: cookie_session
  mutators:
    - handler: noop
  authorizer:
    handler: allow
  errors:
    - handler: redirect
      config:
        to: http://127.0.0.1:4455/login The Ory Oathkeeper configuration: log:
  level: debug
  format: json

serve:
  proxy:
    cors:
      enabled: true
      allowed_origins:
        - "*"
      allowed_methods:
        - POST
        - GET
        - PUT
        - PATCH
        - DELETE
      allowed_headers:
        - Authorization
        - Content-Type
      exposed_headers:
        - Content-Type
      allow_credentials: true
      debug: true

errors:
  fallback:
    - json

  handlers:
    redirect:
      enabled: true
      config:
        to: http://127.0.0.1:4455/login
        when:
          -
            error:
              - unauthorized
              - forbidden
            request:
              header:
                accept:
                  - text/html
    json:
      enabled: true
      config:
        verbose: true

access_rules:
  matching_strategy: glob
  repositories:
    - file:///etc/config/oathkeeper/access-rules.yml

authenticators:
  anonymous:
    enabled: true
    config:
      subject: guest

  cookie_session:
    enabled: true
    config:
      check_session_url: http://kratos:4433/sessions/whoami
      preserve_path: true
      extra_from: "@this"
      subject_from: "identity.id"
      only:
        - ory_kratos_session

  noop:
    enabled: true

authorizers:
  allow:
    enabled: true

mutators:
  noop:
    enabled: true Ory Oathkeeper now looks up a valid session in the request cookies and proxies only authenticated requests. It redirects to login UI if there's no cookie available. ory_kratos_session Adding Kong Now all that is needed is to configure Kong: services:
  kong-migrations:
    image: "kong:latest"
    command: kong migrations bootstrap
    depends_on:
      - db
    environment:
      <<: *kong-env
    networks:
      - intranet
    restart: on-failure

  kong:
    platform: linux/arm64
    image: "kong:latest"
    environment:
      <<: *kong-env
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_PROXY_LISTEN: "${KONG_PROXY_LISTEN:-0.0.0.0:8000}"
      KONG_ADMIN_LISTEN: "${KONG_ADMIN_LISTEN:-0.0.0.0:8001}"
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_PREFIX: ${KONG_PREFIX:-/var/run/kong}
      KONG_DECLARATIVE_CONFIG: "/opt/kong/kong.yaml"
    networks:
      - intranet
    ports:
      # The following two environment variables default to an insecure value (0.0.0.0)
      # according to the CIS Security test.
      - "${KONG_INBOUND_PROXY_LISTEN:-0.0.0.0}:8000:8000/tcp"
      - "${KONG_INBOUND_SSL_PROXY_LISTEN:-0.0.0.0}:8443:8443/tcp"
      - "127.0.0.1:8001:8001/tcp"
      - "127.0.0.1:8444:8444/tcp"
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 10s
      timeout: 10s
      retries: 10
    restart: on-failure:5
    read_only: true
    volumes:
      - kong_prefix_vol:${KONG_PREFIX:-/var/run/kong}
      - kong_tmp_vol:/tmp
      - ./config:/opt/kong
    security_opt:
      - no-new-privileges

  db:
    image: postgres:9.6
    environment:
      POSTGRES_DB: kong
      POSTGRES_USER: kong
      POSTGRES_PASSWORD: kong
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "kong"]
      interval: 30s
      timeout: 30s
      retries: 3
    restart: on-failure
    networks:
      - intranet

  hello: The docker-compose creates three containers db container with PostgreSQL database to store the configuration of services/routes for our API gateway. kong-migrate to run migrations against the database. kong container that exposes port for proxying traffic and port with admin API. 8000 8001 As the last step, we need to create a service for Kong and configure routes. #!/bin/bash

# Creates an secure-api service
# and proxies network traffic to oathkeeper
curl -i -X POST \
  --url http://localhost:8001/services/ \
  --data 'name=secure-api' \
  --data 'url=http://oathkeeper:4455'

# Creates routes for secure-api service
curl -i -X POST \
  --url http://localhost:8001/services/secure-api/routes \
  --data 'paths[]=/'\ Testing You can open or in your browser and there are two possible scenarios: http://127.0.0.1:8000/hello http://127.0.0.1:8000/world You receive (or ). {"message": "Hello microservice"} "World microservice" The browser redirects you to . http://127.0.0.1:4455/login Further steps mutator to have the identity accessible as JWT token for your microservices. Configure id_token the password policy to better suit your use case. Configure . Add two-factor authentication Consider using instead of having an additional reverse proxy inside your network. authentication based on subrequest result can be an excellent plugin to use oathkeeper as a decision API for Kong. Kong auth request

Secure Microservices with Kong and Ory

, take a back seat, there's a new player in town! NGINX For many developers out there, NGINX would be the primary choice for reverse proxy. However at , our API platform for banking integrations, we moved on to something more exciting! Decentro API development has moved from a single stack monolithic architecture to the majestic microservices architecture. This requires a gateway server that should manage redirection, proxying, and security over multiple requests. This means our old faithful NGINX has become a little outdated. (Let’s not even talk about Apache HTTP Server here!) Even if you run a monolithic application, The API Gateway is still a good pick. Apart from proxying, there are many other benefits. The API Gateways should give us the developer the ability to: Scale: As you grow, so should your application Provide security: Applying SSL is just scratching the surface Have a GUI: This would make life 10 times easier Be Intuitive: Should be self-explanatory “ ” But what does it do? You may ask. Good question. Suppose we have an API stack with multiple APIs spanning over a bunch of microservices. Let’s look at the diagram: Here you can see the API Gateway is the lord and the savior for the application instances running behind it. It proxies the requests to the requisite application instance (as a reverse proxy). Now your endpoint can be independent of the microservice it caters to. So if you want to have multiple endpoints (e.g., /docs, /documents, /documentation) connected to the same application, you can. Just reverse proxy it, baby! Farewell, NGINX! Here at Decentro, we used to work with NGINX as our reverse proxy server. But we grew out of it. We had to log in to our instance to reload NGINX via CLI and had to learn to manage configurations when using NGINX. It didn’t have consumer-level handling for API management. We needed something extra. After logging our heads together for quick brainstorming and crossing the Is’ & Ts’ on the Pros/Cons list, we found the one! Dhumm… Dhumm… Dhumm… Did you feel it? Here comes ! KONG Source: Github As is evident from the image above, Kong provides a lot of features out of the box. No recompilation. No command-line configuration. No extra cost. Kong comes with its own RESTful Admin APIs. It is built on top of OpenResty (an extension of NGINX) and LuaJIT. These provide access to all kinds of configurations that can be made available. We have done another addition on top of it by adding the Konga dashboard to manage Kong via a GUI. Kong API Gateway Structure The API structure in Kong is pretty simple. Services – The logical representation of your microservice or monolithic application Routes – The logical representation of endpoints that are accessible within the service. Consumers – The clients that are going to be using the services.Plugins – This is where the magic happens. Plugins can be applied to either of the above entities mentioned. So let’s talk about an example. We have a microservice that takes care of sending emails.  We'll declare a Service with the name “emails” (Pretty innovative.. yep!). This will contain the details of the application server, which it will reverse proxy for. Now let’s make Routes for this service. So we have two kinds of emails to be sent. Transactional – Sent by the API Marketing – Sent by the marketing department to let the clients know about the new features of our awesome product. Now, we will put an API Key authentication plugin on top of the “emails” service. This will ensure that no bad actors try to get access to our APIs. Rate-limit Plugins We will also add a rate-limit plugin (to limit the number of API hits) on the different routes. Transactional (600 per minute) – Since this API is triggered by other flows, we need a high rate limit. Marketing (1 per minute) – Since this is manually triggered, we don’t need to go beyond 1 per minute In order to further enhance our security, we will allow only certain IP addresses to be able to access our transactional email route as it will be triggered only by our internal systems, and we know the IP address range for our internal systems. We will set it open to the world (Potentially insecure) for the marketing emails as the marketing peeps should be able to access it from any cafe that they wish to work out of. Now we want only authorized consumers to access the APIs, so we will set up a consumer and add the API authentication credentials for them to use. Now our email service is ready to serve consumers. We can integrate it into our code and give it to the marketing peeps for them to use it manually. Wrapping Things Up As you can see, using Kong as an API Gateway is simple and takes the hassle of managing APIs. Kong provides many other features like upstreams, certificate management, request and response transformation, serverless code execution, logging, and CORS, which are very helpful when you wish to develop APIs quickly. NGINX and other reverse proxy servers are simply blown out of the water by the ease of configurability and sheer dexterity that KONG provides out of the box. We rest our case, thus!

Why We Switched from NGINX Is to KONG API Gateway

Por qué cambiamos de NGINX a KONG API Gateway

is a remote procedure call (RPC) framework initially developed by Google circa 2015 that has seen growing adoption in recent years. Based on HTTP/2 for transport and Protobuf as Interface Definition Language (IDL), gRPC has several capabilities that traditional REST APIs struggle with, such as bi-directional streaming and efficient binary encoding. gRPC Native gRPC Support What follows is a step-by-step tutorial on how to set up Kong to proxy gRPC services, demonstrating two possible scenarios. In the first scenario, a single Route entry in Kong matches all gRPC methods from a service. In the second one, I have per-method Routes, which allows me to apply different plugins to specific gRPC methods. Before starting, install Kong Gateway , if you haven’t already. As gRPC uses HTTP/2 for transport, it is necessary to enable HTTP/2 proxy listeners in Kong. To do so, add the following property in your Kong configuration: proxy_listen = : http2, : http2 ssl 0.0 .0 .0 9080 0.0 .0 .0 9081 Alternatively, you can also configure the proxy listener with environment variables: KONG_PROXY_LISTEN= bin/kong restart "0.0.0.0:9080 http2, 0.0.0.0:9081 http2 ssl" In this guide, I'll assume Kong is listening for HTTP/2 proxy requests on port 9080 and secure HTTP/2 on port 9081. I'll use the command-line client and the collection of mock gRPC services. gRPCurl grpcbin Case 1: Single Service and Route I'll begin with a simple setup with a single gRPC Service and Route; all gRPC requests sent to Kong’s proxy port will match the same route. Issue the following request to create a gRPC Service (assuming your gRPC server is listening in localhost, port 15002): $ curl -XPOST localhost: /services \
  --data name=grpc \
  --data protocol=grpc \
  --data host=localhost \
  --data port= 8001 15002 Issue the following request to create a gRPC Route: $ curl -XPOST localhost: /services/grpc/routes \
  --data protocols=grpc \
  --data name= -all \
  --data paths= 8001 catch / Using gRPCurl, issue the following gRPC request: $ grpcurl -v -d -plaintext localhost: hello.HelloService.SayHello '{"greeting": "Kong 1.3!"}' 9080 The response should resemble the following: Resolved method descriptor:
rpc SayHello ( .hello.HelloRequest ) returns ( .hello.HelloResponse );
Request metadata to send:
(empty)
Response headers received:
content-type: application/grpc
date: Tue, Jul : : GMT
server: openresty/ via: kong/ x-kong-proxy-latency: x-kong-upstream-latency: Response contents:
{ : }
Response trailers received:
(empty)
Sent request and received response 16 2019 21 37 36 1.15 .8 .1 1.2 .1 0 0 "reply" "hello Kong 1.3!" 1 1 Notice that Kong response headers, such as and , were inserted in the response. via x-kong-proxy-latency Case 2: Single Service, Multiple Routes Now I'll move on to a more complex use-case, where requests to separate gRPC methods map to different Routes in Kong, allowing for more flexible use of Kong plugins. Building on top of the previous example, let’s create a few more routes for individual gRPC methods. The gRPC “HelloService” service being used in this example exposes a few different methods, as you can see in its Protobuf definition (obtained from the gRPCbin ): repository syntax = ; hello; { ; ; ; ;
} { greeting = ;
} { reply = ;
} "proto2" package service HelloService SayHello(HelloRequest) (HelloResponse) rpc returns LotsOfReplies(HelloRequest) (stream HelloResponse) rpc returns LotsOfGreetings(stream HelloRequest) (HelloResponse) rpc returns BidiHello(stream HelloRequest) (stream HelloResponse) rpc returns message HelloRequest optional string 1 message HelloResponse required string 1 I'll create individual routes for its “SayHello” and “LotsOfReplies” methods. Create a Route for “SayHello”: $ curl -XPOST localhost: /services/grpc/routes \
  --data protocols=grpc \
  --data paths= SayHello \
  --data name=say-hello 8001 /hello.HelloService/ Create a Route for “LotsOfReplies”: $ curl -XPOST localhost: /services/grpc/routes \
  --data protocols=grpc \
  --data paths= LotsOfReplies \
  --data name=lots- -replies 8001 /hello.HelloService/ of With this setup, gRPC requests to the “SayHello” method will match the first Route, while requests to “LotsOfReplies” will be routed to the latter. Issue a gRPC request to the “SayHello” method: $ grpcurl -v -d \
  -H -plaintext \
  localhost: hello.HelloService.SayHello '{"greeting": "Kong 1.3!"}' 'kong-debug: 1' 9080 (Notice I'm are sending a header , which causes Kong to insert debugging information as response headers.) kong-debug The response should look like this: Resolved method descriptor:
rpc SayHello ( .hello.HelloRequest ) returns ( .hello.HelloResponse );
Request metadata to send:
kong-debug: Response headers received:
content-type: application/grpc
date: Tue, Jul : : GMT
kong-route-id: ef3d1-d092 ca b4e42453d97
kong-service-id: d82736b7-a4fd -b575-c68d94c3493a
kong-service-name: s1
server: openresty/ via: kong/ x-kong-proxy-latency: x-kong-upstream-latency: Response contents:
{ : }
Response trailers received:
(empty)
Sent request and received response 1 16 2019 21 57 00 390 -4401 -99 -0 -4530 1.15 .8 .1 1.2 .1 0 0 "reply" "hello Kong 1.3!" 1 1 Notice the Route ID refers to the first route I created. Similarly, let’s issue a request to the “LotsOfReplies” gRPC method: $ grpcurl -v -d \
  -H -plaintext \
  localhost: hello.HelloService.LotsOfReplies '{"greeting": "Kong 1.3!"}' 'kong-debug: 1' 9080 The response should look like the following: Resolved method descriptor:
rpc LotsOfReplies ( .hello.HelloRequest ) returns ( stream .hello.HelloResponse );
Request metadata to send:
kong-debug: Response headers received:
content-type: application/grpc
date: Tue, Jul : : GMT
kong-route-id: bb ac5-b177-bc04b3974c87
kong-service-id: a87674-f984 f75 abc da478e204f
kong-service-name: grpc
server: openresty/ via: kong/ x-kong-proxy-latency: x-kong-upstream-latency: Response contents:
{ : }
Response contents:
{ : }
Response contents:
{ : }
Response contents:
{ : }
Response contents:
{ : }
Response contents:
{ : }
Response contents:
{ : }
Response contents:
{ : }
Response contents:
{ : }
Response contents:
{ : }
Response trailers received:
(empty)
Sent request and received responses 1 30 2019 22 21 40 133659 -7e88 -4 31 -4 -8 -85 1.15 .8 .1 1.2 .1 14 0 "reply" "hello Kong 1.3!" "reply" "hello Kong 1.3!" "reply" "hello Kong 1.3!" "reply" "hello Kong 1.3!" "reply" "hello Kong 1.3!" "reply" "hello Kong 1.3!" "reply" "hello Kong 1.3!" "reply" "hello Kong 1.3!" "reply" "hello Kong 1.3!" "reply" "hello Kong 1.3!" 1 10 Notice that the response header now carries a different value and refers to the second Route created on this page. kong-route-id Note: gRPC will still be routed to the first route I created (the “catch-all” route) since the request matches neither nor routes. reflection requests SayHello LotsOfReplies Logging and Observability Plugins As I mentioned earlier, Kong 1.3 gRPC support is compatible with logging and observability plugins. For example, let’s try out the and plugins with gRPC. File Log Zipkin File Log Issue the following request to enable File Log on the “SayHello” route: $ curl -X POST localhost: /routes/say-hello/plugins \
  --data name=file-log \
  --data config.path=grpc-say-hello.log 8001 Follow the output of the log as gRPC requests are made to “SayHello”: $ tail -f grpc-say-hello.log
{ :{ : , : , : }, :{ : , : , : , : , : , : , : , : , : , : , : }, :{ :{}, : , : , : , :{ : , : , : , : , : }, : }, : , :[{ : , : , : , : }], :{ :{ : , : , : , : , : , : , : , : , : }, : , : }, :{ : , : , :[ ], : , :{ : }, : , : , : , : , :[ ], : }, : }
{ :{ : , : , : }, :{ : , : , : , : , : , : , : , : , : , : , : }, :{ :{}, : , : , : , :{ : , : , : , : , : }, : }, : , :[{ : , : , : , : }], :{ :{ : , : , : , : , : , : , : , : , : }, : , : }, :{ : , : , :[ ], : , :{ : }, : , : , : , : , :[ ], : }, : } "latencies" "request" 8 "kong" 5 "proxy" 3 "service" "host" "localhost" "created_at" 1564527408 "connect_timeout" 60000 "id" "74a95d95-fbe4-4ddb-a448-b8faf07ece4c" "protocol" "grpc" "name" "grpc" "read_timeout" 60000 "port" 15002 "updated_at" 1564527408 "write_timeout" 60000 "retries" 5 "request" "querystring" "size" "46" "uri" "/hello.HelloService/SayHello" "url" "http://localhost:9080/hello.HelloService/SayHello" "headers" "host" "localhost:9080" "content-type" "application/grpc" "kong-debug" "1" "user-agent" "grpc-go/1.20.0-dev" "te" "trailers" "method" "POST" "client_ip" "127.0.0.1" "tries" "balancer_latency" 0 "port" 15002 "balancer_start" 1564527732522 "ip" "127.0.0.1" "response" "headers" "kong-route-id" "e49f2df9-3e8e-4bdb-8ce6-2c505eac4ab6" "content-type" "application/grpc" "connection" "close" "kong-service-name" "grpc" "kong-service-id" "74a95d95-fbe4-4ddb-a448-b8faf07ece4c" "kong-route-name" "say-hello" "via" "kong/1.2.1" "x-kong-proxy-latency" "5" "x-kong-upstream-latency" "3" "status" 200 "size" "298" "route" "id" "e49f2df9-3e8e-4bdb-8ce6-2c505eac4ab6" "updated_at" 1564527431 "protocols" "grpc" "created_at" 1564527431 "service" "id" "74a95d95-fbe4-4ddb-a448-b8faf07ece4c" "name" "say-hello" "preserve_host" false "regex_priority" 0 "strip_path" false "paths" "/hello.HelloService/SayHello" "https_redirect_status_code" 426 "started_at" 1564527732516 "latencies" "request" 3 "kong" 1 "proxy" 1 "service" "host" "localhost" "created_at" 1564527408 "connect_timeout" 60000 "id" "74a95d95-fbe4-4ddb-a448-b8faf07ece4c" "protocol" "grpc" "name" "grpc" "read_timeout" 60000 "port" 15002 "updated_at" 1564527408 "write_timeout" 60000 "retries" 5 "request" "querystring" "size" "46" "uri" "/hello.HelloService/SayHello" "url" "http://localhost:9080/hello.HelloService/SayHello" "headers" "host" "localhost:9080" "content-type" "application/grpc" "kong-debug" "1" "user-agent" "grpc-go/1.20.0-dev" "te" "trailers" "method" "POST" "client_ip" "127.0.0.1" "tries" "balancer_latency" 0 "port" 15002 "balancer_start" 1564527733555 "ip" "127.0.0.1" "response" "headers" "kong-route-id" "e49f2df9-3e8e-4bdb-8ce6-2c505eac4ab6" "content-type" "application/grpc" "connection" "close" "kong-service-name" "grpc" "kong-service-id" "74a95d95-fbe4-4ddb-a448-b8faf07ece4c" "kong-route-name" "say-hello" "via" "kong/1.2.1" "x-kong-proxy-latency" "1" "x-kong-upstream-latency" "1" "status" 200 "size" "298" "route" "id" "e49f2df9-3e8e-4bdb-8ce6-2c505eac4ab6" "updated_at" 1564527431 "protocols" "grpc" "created_at" 1564527431 "service" "id" "74a95d95-fbe4-4ddb-a448-b8faf07ece4c" "name" "say-hello" "preserve_host" false "regex_priority" 0 "strip_path" false "paths" "/hello.HelloService/SayHello" "https_redirect_status_code" 426 "started_at" 1564527733554 Notice the gRPC requests were logged, with info such as the URI, HTTP verb, and latencies. Zipkin Start a Zipkin server: $ docker run -d --name zipkin -p 9411:9411 openzipkin/zipkin Enable the Zipkin plugin on the grpc Service: curl -X POST localhost:8001/services/grpc/plugins \
    --data name=zipkin \
    --data config.http_endpoint=http://127.0.0.1:9411/api/v2/spans \
    --data config.sample_ratio=1 As requests are proxied, new spans will be sent to the Zipkin server and can be visualized through the Zipkin Index page, which is, by default, http://localhost:9411/zipkin: To display Traces, click “Find Traces”, as shown above. The following screen will list all traces matching the search criteria: A trace can be expanded by clicking on it: Spans can also be extended, as displayed below: Notice that, in this case, it’s a span for a gRPC reflection request. Previously published at https://konghq.com/blog/manage-grpc-services-kong/

Essential Tips To Manage Your gRPC Services With Kong Like A Pro

by Robert L. Read and Nariman Poushin Learnings In just the last four weeks, thousands of engineers have self-organized around the problem of designing, testing, manufacturing and deploying ventilators to address a present and possibly worsening shortfall of mechanical ventilators required by the unusually virulent and transmissive COVID-19 pandemic. An even greater effort has worked to provide personal protective equipment, as is fitting; we each have our own way of helping. As is also fitting, large firms have begun manufacturing ventilators, and existing manufacturers have increased production, either by themselves or in partnerships. This increase in production may meet the need, at least in the wealthier nations. Personally, I believe that even if this is probable, it is far from certain. Therefore as of April 7th, there remains a pressing need for open source pandemic ventilators. In less than a month, there has been a lot of learning. No less than 70 projects world-wide were begun and made some progress, which we have attempted to in various forms. One of these, the AmboVent from Israel, is close to being both clinically acceptable and easily mass-produced. Its plans are freely open-sourced so that any nation or firm may build them. analyze and track I have personally learned a lot about pulmonology. I daresay many other engineers like me began working on the problem before fully understanding the subtlety of attempting to build a life-critical ventilator to treat a desperately sick person with the worst COVID-19 symptoms, the ARDS-like phase. You can’t solve this problem with an oscillating leaf-blower. This is natural, because when they began, the communities supporting and connecting the engineers were less strong. People felt like they could be “the“ solution instead of being “part of” the solution. A Huge Community But since then many organizations have been established or refocused: (at least 16,000 people) and (4,400 people), , , , , , , , , , and the , are the ones we know of. (We are probably unintentionally more aware of English-language projects.) These organization are world-wide and often international in their focus. Please report additional ones to us at our of ventilators projects and resources. HelpfulEngineering EndCoronaVirus.org DIY Ventilators Pak Innovation Club Ventilator Crowd #industryvirus 1 million Ventilators Project 1M Ventilators Global Ventilator & PPE DIY Collaboration #EngineersAssemble The Ventilator Project Ventilators Collaboration Network open source list When you can count twenty thousand people in a few slack channels, it is fair to say you are a community. Would a community of 20,000 engineers attack the problem of building open source ventilators by creating as many projects as possible? Of course not. Modularization They would do what engineers do: break the problem into smaller pieces. They would in fact, modularize the problem, and let different teams work on different pieces in parallel to shorten the time to first life saved. Nine women cannot make a baby in one month; but 20,000 engineers may be able to save lives quickly, if they do compete or pull in separate directions. Making disparate pieces work harmoniously together is not simple, but engineers have gotten good at it. Openness is now seen as our greatest tool. It allows massive reuse and massive crowd-sourced brainpower. Other tools are standards, protocols, testing regimes, and different methodologies — all forms of communication, however specialized. In hindsight, we should have divided up into teams, or even whole communities staffed by thousands, working on separate pieces. But a month ago the pieces were not clear. I believe now the pieces are becoming clearer, and I would like to try to articulate them, in hopes that the overall community will self-organize more efficiently. A Model of All Ventilators Ventilators must deliver air in a precisely controlled manner. However, they are fundamentally simple. For example, they are far simpler than an internal combustion engine. In fact, we have started an open-source project called COROVAMP (COROnaVirus Abstract Physical Model). It is at the moment just a diagram, shown at the top of this article. Believe it or not, every mechanical ventilator can be mapped onto this diagram; some even have fewer conceptual parts. Thinking about the physical machine, the airway, the patient, the control, and, importantly, the exhausted clinicians wearing PPE who may have to use a pandemic ventilator, allows us to identify several independent modules. Testing, Monitoring and Alarming The key to deploying pandemic ventilators is testing. But the act of testing is closely related to the monitoring which we have all seen in movies: the machine gives feedback to the clinicians. In this case, that feedback is about pressure, volume, respiration rate, and a few other things. The machine has to sound an alarm when something goes wrong, either with the patient, or the machine. One thing that can go wrong is that is too high or too low. pressure or the volume Because the act of testing is so closely related to monitoring and monitoring is the basis of alarming, these functions are closely related. Testing is done on a bench with an artificial lung; monitoring is done in an ICU or critical care ward. So you need monitoring for every ventilator connected to a patient; but with one tester you can test many ventilators; so of course, testing precedes monitoring. In a community of 20,000, there should probably be at least 4,000 working on testing, monitoring, and alarming. Three of us at Public Invention have started one such project ( ), but there is room for far more projects. VentMon VentMon will plug into the airway of ANY ventilator to provide (possibly redundant) monitoring and alarming. Control Panel Most ventilators have a physical control panel and status display. The has a simple one that should be easy for any experienced clinician to use: AmboVent The AmboVent has a straight-foward control panel I doubt the AmboVent control panel is perfect; but the beauty of an open source design is that we can improve on it. Indeed, we can go a step further: we can modularize it. Some games companies, such as Nintendo and Atari, allowed thousands of complex games to be played with a simple controller. There is no reason we could not build a “ ” that could be reused by other teams, so they don’t have to invent their own. In fact Sunday morning the Public Invention VentMon team met a team led by Prof. John Bennett of UC Boulder who had this idea, and we are attempting to cooperate, in part by defining a standard. Universal Pandemic Ventilator Controller A Data Standard The internet runs on standards, although users don’t have to know that. In order for a to be a module that works with different ventilators, there must be a standard for how it communicates with other components. Some of what must be standardized is the respiratory data itself: the pressure, flow, temperature and O2 measurements taken by the machine that are clinically important. In an attempt to cooperate with Prof. Bennett’s team, we began such a standard Sunday night: . It is of course, not very good yet, but it is open-source: you can take it and improve it, and hopefully, give your improvements back to us. Universal Pandemic Ventilator Controller Public Invention Respiratory Data Standard (PIRDS) (v0.1) The Drive Mechanism Every ventilator has a source of air. Many squeeze an AmbuBag. This has the advantage that the volume is easy to control. It may or may not have the drawback that the bag doesn’t last 14 days through the course of the illness; we will see. Perhaps a fan is better; perhaps a pump. The Brazilian uses a bellows, which can be manufactured anywhere. OpenVentilator But the truth is: the patient doesn’t care. Even the doctors don’t care. What matters is the delivery of an air/oxygen mixture at pressures, tidal volumes, and rates that a doctors thinks best for the patient. There is no reason that the drive mechanism could not be interchangeable, or to get a little extreme: why not hot-swappable? If you’re compressor goes bad and your supply chain has been disrupted, why not swap in an AmuBag drive? The answer is, of course, it probably wouldn’t work without extensive testing, physical, electrical, and computer standards. In other words, without engineers doing a lot of what they love to do to make it possible. Others Possibly a humidifier, an oxygen mixer, and an exhalation sanitizer should be modules. The modularization itself should be an open source project that 20,000 engineers can review and possibly extend and improve. Nariman Poushin has suggested that there should be a small subset of the Robotic Operating System (ROS) that is specific to ventilators: a VOS. An operating system for ventilators would use software to tie together many different components elegantly. Supply Chain Resilience If we are to save lives on a large scale, we must plan for manufacturing at a large scale. The meaning of “large scale” may vary depending on your nation and region. But every effort, including those undertaken by Dyson, Tesla, Ford and GM must be concerned with a breakdown in the supply chain of parts and components. This may be caused by the pandemic itself or the economic or political disruption it causes. It seems to me the only way to have supply chain resilience is to have a modular design where different parts can be used. This must not be “dual-sourced” but “poly-sourced”. I suspect large ventilator manufacturers have an internal modular design. However, until now they would have had a disincentive to make that open. The pandemic has changed that. In the presence of extreme supply chain disruption, economic catastrophe, and international strife, the only way to have modularity is to be open about every aspect of the design. Transparency and Openness There are projects, possibly very good projects, that are not open right now. They are in no way benefiting the community, and they are in no way leveraging the community. I believe policy makers should be skeptical and mistrustful of every pandemic ventilator project, but they should be doubly skeptical of any project that is not a transparent open source project. A closed system forces them to rely on the authority and reputation of its makers rather than the evaluation of a community. We have proposed a distributed process by which confidence can be built by having different teams build and test open designs: I recommend every project be fully open from day one with a license and a document repository. Use an . License your documentation with CC0 or CC-BY or CC-BY-SA or make a . Use the or some other license for code. open source hardware license different choice Gnu GPL Don’t imagine that you are stronger by yourself. Publish your work. You don’t have to respond to anybody’s comment on it. If someone says something useful, use it; if not, keep your head down. I can accept no argument that says more lives will be saved by a project keeping their work secret until some milestone is reached or some test is passed or some committee approves. The pandemic demands transparency and openness. ( is a US 501(c)3 public charity.) Public Invention

Lets Modularize the Open Source Ventilator Problem

COVID19 Technology Trends Affecting the Spread of the Pandemic 

Building Webiny — A Serverless CMS Challenges and Lessons Learned Over the Last 12 Months First, we had metal, then it all went virtual and, finally, we moved to containers. Now, a new era of “servers” is emerging (if we can even call them that); one that is born in the cloud and designed for the cloud. Evolution of serverless — Jason McGee Werner Vogels, CTO of Amazon, said: “Serverless is the future of development” — and I couldn’t agree more. However, not everything is peachy. The serverless environment is young(ish), and not all systems we use today are optimized for it, especially the content management systems. Let me explain. Revolution over evolution We are still chasing faster horses rather than thinking about how to teleport. If the previous sentence confused you, let me give you an analogy. Upgrading your Wordpress version 4 to version 5 is evolution; you have just been given a faster horse. This horse can jump higher, go for longer, and can even walk backward — but it’s still a horse. It’s still the same stack that was invented 15 years ago (initial release: May 27, 2003). In the meantime, bright new minds have invented the serverless technology, new API standards (like ), cool and new ways to build and construct the UI with libraries (like ). Today, we even have a . The non-existence of all these technologies was how the previous CMSs were designed, and those limitations are still baked into them. GraphQL React serverless database The path of revolution requires us to build new things with technologies (and mindsets) that are more powerful, faster, safer, and much more scalable. This is a story about one of those trips. Why build a serverless CMS? The answer is simple — I had an itch :) Joking aside, if you are looking for a CMS built with technologies like React, Node, and GraphQL, and is also designed for the serverless environment — well, you’re in luck! As of now, there is only one option that’s available today — . Webiny <a href="https://medium.com/media/931d737448d15cf9a035b99b7008508d/href">https://medium.com/media/931d737448d15cf9a035b99b7008508d/href</a> Webiny is a CMS for the serverless era. It’s open-source, licensed under MIT, and it also features a hosted version, where you can get your very own serverless environment in just a few mouse clicks. Webiny has numerous cool features and is aimed primarily at developers. You can learn more about that on the aforementioned website. Now, let’s get back on track with the original title of this post. Challenges along the way The challenges we faced can be categorized into three different groups: CMS related Architecture related Business/finance related We will discuss each category in more detail and then share some useful tips — especially about the business/financial side. Now, let’s dive deeper. CMS challenges A CMS is a rather complex set of features that need to work together. Here are some examples. Page builder You can’t just create a form with an input field for the title and a for the page content and call it CMS. That’s not nearly enough. You need a full-featured page builder — and we’ve built one! CKEditor But boy, the effort required to create one that actually works and is optimized is sure hard to describe. Webiny Page Builder The first main challenge was managing the data inside the page builder. The interface is complex — each page item has over 20 different properties that you can adjust. In a single page you can easily have over 100 elements, bringing the total amount of props that you need to manage, store, and retrieve up to thousands. We used to control all that, and it’s not just Redux as a data-layer that saves you loads of trouble, but the debugger that comes with it as well. This wouldn’t have been possible without it. You can move up or down the stack of your events, see how the whole data structure looks in real time, and many other things without which this would’ve been way harder. Redux Redux flow Rendering performance Our CMS has over 100 different moving parts — some hold state, some do major state updates every time you make a change. Propagating that through the UI can sometimes cause the interface to lag and just not feel “fast” enough. In the beginning, everything was nice and smooth. You develop one element, you test it, you move on to the next. Everything seems to be working until you finally begin building real pages with tons of elements. Slowly but surely, everything becomes slower and slower. An obvious first thought is the It has the “Highlight Updates” option, which will help you identify the re-rendered elements. React Developer Tools. React Developer Tools Unfortunately, this will only help you with a very simple UI hierarchy. In our case, this barely scratched the surface. So, we had to turn to the Chrome Dev Tools, the in particular. This tool is priceless when debugging rendering performance in your React app. The recorded snapshots let you see exactly how much time was spent to render each particular element in your app hierarchy (don’t forget to open the section). You can, then, go directly to the specific component’s code and start optimizing things: Performance Snapshots Timings Chrome Dev Tools — Performance Snapshot This is still a very manual process since you have to think about the output and why certain components get re-rendered, but you can quickly find different problems in your app and in some cases solving one problem can fix rendering performance in many places ( and Redux’s will be your best friends to get the job done). shouldComponentUpdate connect Very daunting at the beginning, this process will make you happy in the end. CSS libraries Here is a helpful tip which is not very obvious straight away: If you are using a library for CSS, like , make sure you do not update props too often on components that control the CSS, as that will create and insert a new style DOM element on each update. Emotion A typical example is a tool to resize elements: the size of the element must under no circumstances be an Emotion prop — it has to be a simple React element style value. Otherwise, you will bombard your DOM with new style elements while resizing is taking place (that’s just how Emotion works). User interface Building a UI is a story of challenges enough to fill a whole book, if not several ones. Luckily, when we started, Google had just launched , and there were a few react libraries popping up that had those components ready to be used. The one we used is called , and, together with the on the website that teaches you how to use their components in the right way, we managed to create a pretty decent UI for our CMS. Material Design RMWC Design section material.io Homepage Themes and plugins What is a CMS if you can’t build your own theme, or customize it with your own plugins and add-ons? We wanted our theme system to be super simple. Since the page builder is actually the place where you build the content and generate the HTML, we managed to create a theme library where all you have is a small JSON config and everything else is done through (S)CSS. You can learn more about our theme setup on our . documentation website {
    layouts: [
        // defines a list of layouts
    ],
    fonts: {
        // defines font faces
    },
    colors: {
        // defines a list of default colors
    },
    elements: {
        // defines element settings
    },
    typography: {
        // defines typography styles
    }
} As for the plugin system, it’s hard to find any good advice or best practices on how to make a plugin system. So, what we actually did was we made the whole CMS as a set of plugins. Every button, menu item, and form element is actually a plugin. This made us feel comfortable in the scalability and the possibilities of what you can do with plugins. On the other hand, this made our code very modular and decoupled. Finally, this has a great effect on developer experience (DX) since there is no difference in how the code is written for the “core” of the system versus how the code will be written by other developers when . building their own plugins Architectural challenges Architecture for a serverless application is rather simple, right? You have an API Gateway, some Lambdas, a database on the other end, an S3 bucket with static files. But, here’s a question for you — what about a multi-tenant serverless application? Basically where each user can have its own domain name, its own SSL certificate, static file hosting, and a database. The hosted version of Webiny is exactly that. And, I can tell you that, today, there isn’t a cloud provider that will allow you to have 10.000 different SSL certificates on an API gateway or serve static assets from the same bucket using multiple domain names. CloudFront and similar CDNs have hard limits on the number of SSL certificates and domain names that they can serve. We talked to several cloud providers and they just don’t support that. So, although serverless is great — it’s got its limitations. Or does it?! ;) We took this problem as a challenge and, to get around it, we used an OpenResty Nginx fork and wrote our own API gateway for the AWS Lambda, designed our own SSL certificate management system in the front, created a proxy to the S3 buckets, and also added a bunch of usage and performance monitoring agents, — we can now support as many serverless tenants as we want. et voila OpenResty is great; it just takes a bit of practice with coding in Lua. https://github.com/openresty/lua-nginx-module You might argue that this is not a true Serverless CMS since it uses a “server” in the form of a proxy to make it work. Technically, the proxy is not required; it’s only needed for the hosted environment with multiple tenants. But, also, for the sake of argument, if you use a CDN in front of your serverless page, like , those are technically OpenResty reverse proxies. Cloudflare Don’t get me wrong, this definitely is an area where big cloud providers like AWS, Google, and Azure need to improve since those platforms, today, don’t support a multi-tenant serverless architecture. Business challenges By this point, you’ve hopefully gotten the picture of the effort required to build a serverless CMS, but the story doesn’t really end there — I’ve got one more thing to share. Webiny is , but it’s taken a lot of time, dedication, and money to create Webiny. When we started the project, we knew doing this on the side was never an option; we would most likely fail in our attempt. So, the only way to build something as large and complex as this required financial support. But, no investor will give you money if you go and approach him with an elevator pitch to build a CMS, which has 0 users and no revenue — it would just never be convincing enough to raise investment. open source Also, creating a CMS 10 years ago was way easier because the bar to enter that market was much lower, but that’s no longer the case. Several people told us — “Hey, just build an MVP.” Well, you go and try to build an MVP for, let’s say, an electric car, and let me know which things you will leave out that your competition already has today. Let me help you with answering that: you won’t leave out a single thing! Your car will have everything the competition has and then some — and that’s when you’ll launch! Otherwise, nobody will buy it. No matter what you do in life, if you want to be great at something, you need to reach the level of the “current greatness” and then be slightly better than that. So, how did we solve our financial problem? Well, the team originally started as a small web agency. We earned some money by doing standard web-agency stuff and then, at one point, we decided to stop accepting new work, all the money left on the bank account was used as an investment to ensure salaries for the team so that we can be fully committed to this project. Had it been any other way, Webiny would never have seen the light of day. One small tip in this area. AWS has the Activate program, you can apply it and get $1000 in AWS credits, which is great for covering the cost of the infrastructure while developing. Another thing to explore is things like ProductHunt Ship. If you subscribe for the annual membership ($600 — $1500), you can get between $5000 and $7500, also in AWS credits, alongside . Note that if you apply for the AWS activate, the initial credit amount might be deducted from the credit amount you get from ProductHunt Ship. other benefits AWS Activate What’s next If you got this far … I admire your focus. I blame the length of this post on my long plane flight. Anyhow, hope you’ve enjoyed the read. I would really appreciate it if you’d give a spin, and let me know what you think of it. You can reach out to me via twitter . Webiny @SvenAlHamad https://www.webiny.com

Stories, News & Mentions about OPENRENT

Openrent’s Stories on HackerNoon