Get the most recent info and news about LOKI on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

From only one laptop to global branch offices, automate and secure your network in a so simple way that anyone can do it.

LOKI

loki

Marvel Rivals&#x27; portrayal of <strong>Loki</strong> has already been shown to take a more than a little bit of inspiration from his MCU counterpart.

Comparing Marvel Rivals' Loki to the MCU Version

One of my current talks focuses on Observability in general and Distributed Tracing in particular, with an implementation. In the demo, I show how you can see the traces of a simple distributed system consisting of: the API Gateway, a Kotlin app with Spring Boot, a Python app with Flask, and a Rust app with Axum. OpenTelemetry Apache APISIX Earlier this year, I spoke and attended the Observability room at FOSDEM. One of the talks demoed the Grafana stack: Mimir for metrics, Tempo for traces, and Loki for logs. I was pleasantly surprised how one could move from one to the other. Thus, I wanted to achieve the same in my demo but via OpenTelemetry to avoid coupling to the Grafana stack. In this blog post, I want to focus on logs and Loki. Loki basics and our first program At its core, Loki is a log storage engine: Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. Loki Loki provides a to store and read logs. Let's push a log from a Java app. Loki expects the following payload structure: RESTful API I'll use Java, but you can achieve the same result with a different stack. The most straightforward code is the following: public static void main(String[] args) throws URISyntaxException, IOException, InterruptedException {
    var template = "'{' \"streams\": ['{' \"stream\": '{' \"app\": \"{0}\" '}', \"values\": [[ \"{1}\", \"{2}\" ]]'}']'}'"; //1
    var now = LocalDateTime.now().atZone(ZoneId.systemDefault()).toInstant();
    var nowInEpochNanos = NANOSECONDS.convert(now.getEpochSecond(), SECONDS) + now.getNano();
    var payload = MessageFormat.format(template, "demo", String.valueOf(nowInEpochNanos), "Hello from Java App");           //1
    var request = HttpRequest.newBuilder()                                                                                  //2
            .uri(new URI("http://localhost:3100/loki/api/v1/push"))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(payload))
            .build();
    HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());                                         //3
} This is how we did String interpolation in the old days Create the request Send it The prototype works, as seen in Grafana: However, the code has many limitations: The label is hard-coded. You can and must send a single label Everything is hard-coded; nothing is configurable, e.g., the URL The code sends one request for every log; it's hugely inefficient as there's no buffering HTTP client is synchronous, thus blocking the thread while waiting for Loki No error handling whatsoever Loki offers both gzip compression and Protobuf; none are supported with my code Finally, it's completely unrelated to how we use logs, : e.g. var logger = // Obtain logger
logger.info("My message with parameters {}, {}", foo, bar); Regular logging on steroids To use the above statement, we need to choose a logging implementation. Because I'm more familiar with it, I'll use SLF4J and Logback. Don't worry; the same approach works for Log4J2. We need to add relevant dependencies: <dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>             
    <version>2.0.7</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>       
    <version>1.4.8</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>com.github.loki4j</groupId>
    <artifactId>loki-logback-appender</artifactId> 
    <version>1.4.0</version>
    <scope>runtime</scope>
</dependency> SLF4J is the interface Logback is the implementation Logback appender dedicated to SLF4J Now, we add a specific Loki appender: <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">                   
    <http>
        <url>http://localhost:3100/loki/api/v1/push</url>                                 
    </http>
    <format>
        <label>
            <pattern>app=demo,host=${HOSTNAME},level=%level</pattern>                     
        </label>
        <message>
            <pattern>l=%level h=${HOSTNAME} c=%logger{20} t=%thread | %msg %ex</pattern>  
        </message>
        <sortByTime>true</sortByTime>
    </format>
</appender>
<root level="DEBUG">
    <appender-ref ref="STDOUT" />
</root> The loki appender Loki URL As many labels as wanted Regular Logback pattern Our program has become much more straightforward: var who = //...
var logger = LoggerFactory.getLogger(Main.class.toString());
logger.info("Hello from {}!", who); Grafana displays the following: Docker logging I'm running most of my demos on Docker Compose, so I'll mention the Docker logging trick. When a container writes on the standard out, Docker saves it to a local file. The command can access the file content. docker logs <CONTAINER> However, other options than saving to a local file are available, , , Google Cloud, Splunk, etc. To choose a different option, one sets a logging driver. One can configure the driver at the overall Docker level or per container. e.g. syslog Loki offers its own . To install it: plugin docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions At this point, we can use it on our container app: services:
  app:
    build: .
    logging:
      driver: loki                                                    #1
      options:
        loki-url: http://localhost:3100/loki/api/v1/push              #2
        loki-external-labels: container_name={{.Name}},app=demo       #3 Loki logging driver URL to push to Additional labels The result is the following. Note the default labels. Conclusion From a bird's eye view, Loki is nothing extraordinary: it's a plain storage engine with a RESTful API on top. Several approaches are available to use the API. Beyond the naive one, we have seen a Java logging framework appender and Docker. Other approaches include scraping the log files, , Promtail, via a Kubernetes sidecar. You could also add an Opentelemetry Collector between your app and Loki to perform transformations. e.g. Options are virtually unlimited. Be careful to choose the one that fits your context the best. To go further: Push log entries to Loki via API Loki Clients Originally published at on August 27th, 2023 A Java Geek

Why You Should Send Loki All Your Logs

Grafana Loki is a log aggregation system that stores and queries logs from applications and infrastructure. Although commonplace, logs hold critical information about system operations and are a valuable source of debugging and troubleshooting information. Logs are frequently used to identify and track malicious activity, or simply to track user activity to provide business intelligence. In an earlier blog post, , we provided an overview of Loki’s components and their overall architecture. Think of Loki for logs as analogous to Prometheus for metrics. Loki is lightweight and cost-effective because it only indexes and queries metadata. Promtail agents collect, label and transform logs before sending them to Loki. Then Loki indexes metadata and groups entries into streams that are indexed with labels. Grafana is then used to visualize and query log information from Loki. As of version 2.0, Loki stores data in a single object storage backend, such as MinIO. Whereas the previous blog post explained the advantages of saving Loki data to MinIO, this blog post is a tutorial that teaches you how. Logging with Grafana Loki and MinIO First, we’ll teach you how to deploy Loki and MinIO using Docker containers, followed by instructions on how to install from source. Learn Loki and Master MinIO Using Docker Containers We built a demo using Docker containers, some scripting and a Git repository. We started with the Grafana Loki repository, and added configuration yaml and containers for MinIO and a script to create and expose a MinIO bucket for Loki data. The following steps will result in these five images running locally in Docker containers: Loki Promtail Grafana MinIO Create buckets The first three are obviously needed for Loki, Promtail and Grafana. They were already available from Grafana under the (there is a short ). MinIO is object storage for Loki, and the final container will run a script that creates buckets as Loki targets. Loki repository video explainer Loki has been configured to save log data to MinIO using . Of particular importance is the section: loki.yaml storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    resync_interval: 5s
    shared_store: s3
  aws:
    s3: http://minioadmin:minioadmin@minio.:9000/loki
    s3forcepathstyle: true Note the dot in the S3 address for MinIO. This is used because there is no need to specify AWS Region. We create a Docker environment using the file. We will configure containers, volumes, ports, networks and provide startup commands. docker-compose.yaml In order for the image to run as a container with access to Loki configuration via volumes: loki services:
  loki:
    image: grafana/loki:latest
    volumes:
      - <your-local-path>/loki/production:/home/loki/production
    ports:
      - "3100:3100"
    command: -config.file=/home/loki/production/loki.yaml
    networks:
      - loki The first configuration you’ll need to make is to make sure that your local folder of is being shared with the container. Please edit the section of your to refer to the local folder on the host system where you have downloaded the folder. We are mapping the host path to the container path in order to read . production volumes docker-compose.yaml production loki.yaml In , we also define a network and expose ports of the container running MinIO server. In this case, port 9000 and 9001 will be exposed to your local machine in order for you to access MinIO server with a browser. docker-compose.yaml Console UI:
  minio:
    image: minio/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      - loki
    command: server ~ --address ':9000' --console-address ':9001' We configure the network with all containers sharing a network. Each container can ping the others and use their APIs. Don’t use fixed IP addresses for your containers. To make it easier, we’ve configured and in our environment to resolve to the appropriate container. For example: loki loki minio http://minio:9000    for MinIO
http://loki:3100     for Loki
http://loki:3000     for Grafana The next action taken by is to run MinIO Client (mc) in a container to configure MinIO Server, create the destination bucket for Loki data and set the access policy to as required. docker-compose.yaml public createbuckets:
    image: minio/mc
    networks:
      - loki
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc config host add myminio http://minio:9000 minioadmin minioadmin;
      /usr/bin/mc rm -r --force myminio/loki;
      /usr/bin/mc mb myminio/loki;
      /usr/bin/mc policy set public myminio/loki;
      exit 0;
      " Promtail, a small Go program, is used to tail or collect the distributed log files and deliver them to Loki.  For this demo Promtail is deployed in its container with a configuration file that scrapes the /var/log directory.  No changes are required in order to run this demo and have it process the log files at that location - /var/log. At the end of this tutorial we will cover how to have Promtail scrape other locations on the system for log files. Steps Now that you understand how to configure Loki, Promtail and your Docker environment, please  follow these steps to configure and run the demo environment. The following steps use the , and the default Promtail configuration to demonstrate how Loki works with MinIO. After downloading our files, edit them for your environment. docker-compose.yaml loki.yaml Clone this repository : https://github.com/cniackz/loki git clone https://github.com/cniackz/loki.git minio-loki-tutorial Change directory to your local folder: production https://github.com/cniackz/loki/tree/main/production cd <your-local-path>/minio-loki-tutorial/production
Edit docker-compose.yaml to reference your local home directory
If necessary, edit promtail.yaml Build your local Docker environment: docker-compose build --no-cache Launch local Docker containers: docker-compose up -d To confirm that Loki data is being saved to MinIO, log into MinIO Console at or . This tutorial uses the following credentials: http://localhost:9001 http://minio:9001 user: minioadmin
password: minioadmin After logging in, click Buckets. You should see that the Loki bucket has been created. On the right side, click Browse to see the contents of the newly created bucket. You should see the directory under . This is where Loki will save data. fake loki Click its name to open the directory. Loki holds logs in memory for a configured interval and then writes them to object storage. The default is an interval of 5 minutes, so please wait 5 minutes for data to appear in your bucket. fake At this point, Promtail is sending logs to Loki and Loki is saving data to MinIO. Now, let’s set Grafana up to view Loki logs. Grafana 6.0 and higher include built-in support for Loki. Grafana 6.3 and higher includes support for functionality. LogQL Log into Grafana at (default credentials are admin:admin). http://loki:3000 In Grafana, click the cog icon on the left sidebar to go to Configuration and then Data Sources. Click the big Add Data Source button and then select Loki. Edit the http URL field to be for our Loki server running locally using Docker port mapping: . http://loki:3100 Then click Save & Test. To view logs immediately, click explore. You can also view logs by clicking Explore in the left sidebar. However you explore, then select the Loki datasource in the top-left dropdown, and then choose a log stream using the Log Labels button. You can type a LogQL query and click on Run Query. However, if you don’t yet know LogQL, you can use the GUI to select a log and query parameters. To quickly see the logs that are in Loki, click Log browser, then under and choose , then under click the name of your job (from ). In this case I’ve clicked on . Then click the Show Logs button. 1.  Select labels to search in job 2. Find values for the selected labels promtail-local-config.yaml varlogs You can select the time range and set the refresh interval of the query on the top right of the browser window. To see more details of the logs, scroll down and click on one of the log entries, it will provide additional information related to the log entry. A really powerful feature is the ability to filter or see statistics about and directly from log details by clicking on the icons. This simplifies troubleshooting as it makes it easier to look for recurring errors and pivot between search terms. labels fields Extending the Demo Once the demo is running, the next step is to have Promtail scrape a different set of files that might be more interesting for your use case. In order to do this we need Promtail to run using a configuration file that we can edit. One way to achieve this is using volumes in Docker. Edit the file to create 2 new volumes accessible from the Promtail container. The first provides access to the directory on the host system where the new config file will be created. The second provides access to a directory that will contain the log files of interest.  Additionally, Promtail will be launched referencing the new config file, which we have called : docker-compose.yaml promtail-local-config.yaml version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:latest
    volumes:
     - <your-local-path>/loki/minio-loki-tutorial/production:/home/loki/production
    ports:
      - "3100:3100"
    command: -config.file=/home/loki/production/loki.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.4.2
    volumes:
      - /var/log:/var/log
      -- <your-local-path>/loki/minio-loki-tutorial/production:/home/loki/production
      -- <your-local-path>/access_logs:/home/loki/access_logs
    command: -config.file=/home/loki/production/promtail-local-config.yaml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

  minio:
    image: minio/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      - loki
    command: server ~ --address ':9000' --console-address ':9001'

createbuckets:
    image: minio/mc
    networks:
      - loki
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc config host add myminio http://minio:9000 minioadmin minioadmin;
      /usr/bin/mc rm -r --force myminio/loki;
      /usr/bin/mc mb myminio/loki;
      /usr/bin/mc policy set public myminio/loki;
      exit 0;
      " There’s a lot more that you can do with log data in Grafana. For starters, you can install promtail in more places to tail more logs to Loki. Copy the executable and to other machines/instances/containers, edit the configuration as described in and run it. promtail-local-config.yaml Promtail Configuration We now need to create the Promtail config file, to send local system logs to Loki. Download and edit a from Grafana. The section to focus on is because this is where promtail is told which logs to pull, how to format them and where to send them. Please see for more information about configuring promtail. promtail-local-config.yaml sample config file scrape_configs Get logs into Loki The section includes the following: scrape_configs job_name - This differentiates the logs collected from other log groups. targets - Optional for . However, is often defined because in older versions of Promtail it was not optional. This was an artifact from directly using the Prometheus service discovery code, which required this entry. static_configs labels - Static label to apply to every log line scraped by this definition. Good examples include the environment name, job name, or app name. path - The path to where the logs that Loki is to consume are stored. server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: apache
  static_configs:
  - targets:
      - localhost
    labels:
      job: access_logs
      __path__: /home/loki/access_logs/*log It’s important to understand different configuration options for scraping in Promtail, and Grafana provides . plenty of details It may be helpful to consult the full to see the application’s full capabilities. promtail configuration reference Finally, place some Apache web server access logs in the directory specified in the file on the host system.  These are the logs that Promtail will ingest and send to Loki for processing: docker-compose.yaml -- <your-local-path>/access_logs:/home/loki/access_logs Once the edits have been made and the sample Apache access logs put in place, bring the containers down and back up with docker compose: docker-compose down
docker-compose up -d Promtail will subsequently load the new Apache access log files and make them available to Loki. The environment that we’ve created in this tutorial is helpful for getting started with Loki, but it is not production-ready. Next steps would be to leave Docker for Kubernetes and use distributed MinIO instead of a single instance. Eventually, Loki data gets big enough that it benefits from an external database for quick index searches. Please see for more information. Scalability - Scaling with Grafana Loki Conclusion The observability stack of Grafana, Prometheus and AlertManager gained a powerful addition with Loki (BTW, we also have a tutorial for ). Distributed systems, especially when containerized and orchestrated by Kubernetes, have many logs for their applications and microservices. Loki combined with MinIO is a cost-effective way to collect, store and query logs. Grafana, Prometheus and AlertManager on MinIO If you have any questions, please send us an email at , or join the and ask away. hello@min.io MinIO slack channel Also published . here

How To Deploy Grafana Loki and Save Data to MinIO Using Docker Containers or Direct from Source

Cómo implementar Grafana Loki y guardar datos en MinIO usando contenedores Docker o directamente desde la fuente

Comment déployer Grafana Loki et enregistrer des données sur MinIO à l'aide de conteneurs Docker ou directement depuis la source

ग्राफाना लोकी को कैसे तैनात करें और डॉकर कंटेनरों का उपयोग करके या स्रोत से सीधे डेटा को मिनियो में कैसे सहेजें

Grafana Loki をデプロイし、Docker コンテナーまたはソースから直接使用してデータを MinIO に保存する方法

Como implantar o Grafana Loki e salvar dados no MinIO usando contêineres do Docker ou direto da fonte

Cách triển khai Grafana Loki và lưu dữ liệu vào MinIO bằng Docker Container hoặc trực tiếp từ nguồn

如何使用 Docker 容器或直接从源代码部署 Grafana Loki 并将数据保存到 MinIO

Grafana Loki: Architecture Summary and Running in Kubernetes

It has been God backing the global financial system, if US Dollars can be trusted. The medieval sentiment ‘ ’ snuck into the original 1776 design of the Great Seal of the United States and was raised to adorn the currency during the height of the United States economic power in 1956. In God We Trust Although the most famous pronouncement of Gods death was in 1882 in Friedrich Nietzsche’ ‘The Gay Science’, the death of God had been discussed earlier in the 19th century, notably by the lesser-known Max Stirner whose philosophies were a similar type of existentialist psychology. Regardless of potentially spooky similarities between the two, that God may no longer be with us was something they both agreed upon in the 19th century. So it was a strange addition to the currency of the global superpower, strange enough to feel as if it may potentially have been in bad faith. Potentially that the money was the God and there was little behind that but puppets and shadows. So is it coincidence that the sentiment matches the neo-liberalism of the United States in the second half of the 20th century, which placed the power of money at the centre of social, cultural and political objectives. The pursuit of money was an end unto itself, a self-realising end, that if pursued, would make-right for its imperfections accrued along the way – after the fact. With money achieving a transcendence above scrutiny from its physical realities (that the pure pursuit of money was not self-realising, that there could be negative consequences of material greed) before the fact in the late 1940’s, it was money we trusted in 1956, having achieved a supernatural status. What had been implicit then went on to formally enter culture in the 1980’s, when Gordon Gekko announced that ‘Greed was good’. Money, US Dollars to be precise, was God, and the blind pursuit of Gods will – neo-liberalism – was morality. Having earned an absolute moral right in neo-liberalism over the course of 50 years, money was almost immediately degraded in a series of arrogant responses to events that set the global financial system on a course to total failure. After a large and compelling flag was raised in the Global Financial Crises, Bitcoin emerged shortly afterwards as an anarchistic type of money, uncontrollable and de-centralised and seemingly to make fun of the entire enterprise of finance. Finance in kind, has rarely taken Bitcoin, Cryptocurrencies or Digital assets seriously, seeing them as a kind of bad joke. Myth Mythological stories are supposed to reach across time, through culture and to express universal human experiences in nuanced ways that continue to hold value after the zeitgeist has changed. Mythology is supposed as unavoidable, the tragic or comedic boundaries of the human experience through which we bounce off like a pinball careening from Icarus to Theseus, King Arthur to Scheherazade. Mythology had embedded itself into the human experience with its 1000 illustrious fables until Christianity began to dominate myth with its own powerful take that included a commandment that instructed ‘thou shalt have no other Gods before me’. Soon after the death of God, first Nietzsche and then Freud brought the non-Christian abstract reality back to life with the radical subjectivity of Nietzsche and the ‘unconscious mind’ of Freud. But it was in Carl Jung, one of Freud's young contemporaries, an intelligence he thought would be an heir to his ‘new science’ of psychoanalysis, that resurrected Mythology, returning it to its previous glory in his work. After a fierce intellectual union the two encountered irreconcilable differences. One of these differences was Freuds inability to escape his conditioning, promoting a judeo-christian, monothestic view of reality that saw humans as base animals with vaguely suppressed violent tendencies and sexual obsessions, where Jung took a less dichotomous approach to psychology and developed foundational ideas of the human experience that led to interpretations of mythology – to understand myth with the broad and universal rules of his ‘collective unconscious’. The first thing that Jung had done was to take Freud's unconscious mind and view it outside of the dichotomy of good and evil – a post-religion take.  Within the individual unconscious mind, Jung proposed a ‘collective unconscious’ in which a shared human experience played itself out at the level of archetypal metaphysical models, of the mythological boundaries re-emerging to guide and fortell a recurring human experience. Jung’s archetypes were recurring cultural manifestations, from events and characters to motifs. For example, a masculine force was a single thing with its own essence, its own identifiable thread of consistency embedded in masculine things called the anima with a similar force for femininity he called the animus. His archetypal characters included the great mother, father, child, devil, god, wise old man, wise old woman, the hero and The Trickster. Modern story-telling, especially todays movies have Jungian archetypes (as well as Joseph Cambell’s heroes journey story-telling structure) consciously embedded into them - as recipe now for hollywood - but they also turn up as Delueze’ Rhizomes, a recurring, broad, stateless phenomenon that takes form in the nature of reality, within subjectivity rather than as a result of subjectivity. The Trickster According to Jung, the trickster is ‘ ’ and for Byrd Gibbens – Professor at the University of Arkansas at Little Rock ‘ .’ an archetypal psychic structure of extreme antiquity Many native traditions held clowns and tricksters as essential to any contact with the sacred. People could not pray until they had laughed, because laughter opens and frees from rigid preconception. Humans had to have tricksters within the most sacred ceremonies lest they forget the sacred comes through upset, reversal surprise The Jesters role is to create the comedic moment, where comedy is the moment of relief from a set of social and cultural realities that, taken to their natural conclusion, would lead to inevitable tragedy.  The ability to perform this role is no small task and involves firstly a deep understanding of what those realities are, and secondly to be capable of disregarding all of its dire consequences to find the logic that can be reversed, the paradox to explore, the stupid detail to magnify or the unstoppable force to discount with a goal to manifest absurdity, comedy and existential reflection. The ability to take tragedy and turn it into something comedic is a type of shape-shifting and results in a fundamental and important change in state. Alchemists, who took elemental changes from chemical reaction as types of ritualistic changes in the nature of reality had imbued the chemical mercury with the qualities of the Roman trickster God – Mercurius. Mercury is highly transformative – interchangeable between seemingly radically different forms as if to undergo a ‘tortuous ordeal of purification and renewal’. Jung gives a concrete example of the trickster with alchemical Mercurius with ‘his fondness for sly jokes, malicious pranks, his powers as a shape-shifter, his dual nature, half animal, half divine’. But the trickster is also embodied in other objects and symbols as well. In Killorglin, Irelands oldest annual event is the 3-day ‘puck fair’, where pubs stay open longer than usual and in the town centre a goat is raised on top of a scaffold. That goat is ‘King Puck’ for the duration of the fair (a goat is known as a puck). The entire festival was borne of the bloodthirsty Oliver Cromwell’s conquest of Ireland during a period that Ireland did not have her own king. In raising a wild goat as their king, the inhabitants of Killorglin mounted a criticism of Cromwell as well as English rule, simultaneously making the idea of a King ridiculous. It is this abstract mechanism that is embodied in the trickster archetype. And to underwrite Jung while reiterating him, Jesters were consistent features of empires, from Ancient Egypt, the Roman Empire, Aztecs, Medieval Europe and so on, often with a special political role.  The archetypal tricksters way of viewing reality would be contextualised for each individual cultural reality but perform very similar tasks, as a universal human phenomenon. The dichotomous, paradoxical mind of the Jester could also be used to deliver bad news to a king. When the French fleet was destroyed by the English, Philllipe VI’s jester concluded that English Sailors ‘don’t have the guts to jump into the water like our brave French’. This type of role had been eventually taken over by satirists in plays and even later as newspaper cartoonists with their brand of political satire. The archetype was not lost on Dostoevsky either, who wrote a book titled ‘The Idiot’ with a goal to contrast the paradoxes of Christianity with Russian socio-cultural norms.  Dostoevsky’s idiot held his own deeply felt values and explored his own experience of living upto those values in contrast with his surroundings. It is instructive in how the literary giant saw himself – as an idiot, a fool, as the trickster archetype who would emerge as one of the most important writers in history, influencing Nietzsche as well as Jean-Paul Sartre. One of the most famous and popular today is the Norse God Loki, of which recent movies have constructed his character in full light of these well-developed modern conceptions. Loki is taken as a trickster-God, as the archetypal trickster, pointing out the paradoxes of reality, of challenging the Gods, of breaking the rules and taking things to places that nobody thought possible Most tellingly, in Richard Wagners masterpiece ‘Ring of the Nibelung’, Loki sets Valhalla alight, destroying the Gods in the event of Ragnarok –  the cataclysmic destruction of the cosmos and everything in it. Not including the Gods, but especially the Gods. And perhaps this – the ability to destroy the Gods - is the most important quality of The Trickster. ‘In God We Trust’ So when the excesses of the US Dollar global financial system began to emerge in enormous moral hazard, creating existential systematic risk during the Global Financial Crises (and again after Covid-19) –we conclude that trust in the US Dollar God has failed because the US Dollar system has failed. After a period of corporate excess in the late 90’s and early 00’s, the Global Financial Crises was the final flag to be raised to realise that this Emperor was no longer wearing clothes – but it would take a Jester to point this out. And so out of the very serious, philosophical and absolute critique of the US Dollar global financial system that is Bitcoin, the Jester archetype fully embodied itself in Bitcoin, cryptocurrencies and digital assets. Bitcoin was a new form of money – intelligently designed! Ridiculous!  Secretly traded over the internet to purchase illegal goods! It boggled the mind! An industry of criminals and outsiders; how could it be money? It was for fools, ‘probably rat poison squared’ according to worlds greatest investor Warren Buffet who didn’t get it – neither did the eminent Jamie Dimon. But an enormous, global community did, who viewed the moral hazard of the US Federal Reserve banks control over the monetary supply as an existential threat.  They had an instinct that bailing out the politically connected Goldman Sachs and JP Morgan with billions of dollars while allowing Bear Sterns to fail was a corruption through which they could no longer trust in God. Although there were efforts to ‘occupy wall street’ and political intentions to slap banks on the wrist, some idiots, jesters and fools decided to do something crazy, to do a wild thing and start to build a new financial system.  This undertaking would undermine the ideological raison d’etre of the United States and its dominance of the global financial system with its paradoxes revealed far and wide, its excesses parodied, its nakedness laughed at. It was an announcement to burn down Valhalla, to invite Ragnarok. So if there is any doubt about the virility of the Jester archetype, take a look at crypto culture, especially its meme culture. It is in the duality of holding paradoxical, dichotomous views simultaneously. To take nothing about the very serious nature of finance seriously, to lampoon the US Dollar and offer a challenge to it called Doge. To joke, to ridicule and to offhandedly discount the omnipotence of the US Dollar, against most experts and especially against the eminent figures in global finance.  To completely disrespect and disregard it, and build a new system around it as if the global financial system could be so easily improved by outsiders. Ragnarok : Covid-19 ‘ ’. In Norse mythology, Ragnarok is a series of events, including a great battle, foretold to lead to the death of a number of great figures (including the gods), natural disaster and the submersion of the world in water. After these events, the world will resurface anew and fertile, the surviving and returning gods will meet and the world will be repopulated When the US Federal Reserve Bank announced that the Repo markets required $500 billion of liquidity late December 2019, this commentator took that to mean that the Quantitative Easing of the Global Financial Crises had not just failed, but had magnified the problem it had hoped to solve. It was simply a matter of time after that liquidity entered the system before more would be required. Extraneous liquidity had become an exponential requirement of insatiable global financial markets. God here is the absolute trust in the US Dollar, in its role in the Global Financial System, in its infallibility and the infallibility of its priests – Alan Greenspan, Ben Bernanke, Milton Friedman et al Their actions in the global financial system began to look more like the corruption of the system rather than an efficient and properly functioning system, creating points of interest for Loki, the trickster archetype. The lure was the infallible power of the system, of the emperor-making Federal Reserve Bank Governors, of Milton Friedmans monetarist economics and his and others neo-liberalism that was supposed to self-correct – that was suggested as inherently morally self-correcting! The trickster archetype gains momentum as the logic of the system is used against it, its deficiencies highlighted and its absolute pre-eminence challenged. As this rationale emerges, and especially when it is reinforced in practice, the archetype gains relevance and power. This has been highlighted in cryptocurrency recently where the mood is always at-arms-length to what is important.  The crypto community on twitter is indulgent to the point of hubris in playing the role of trickster. But there is more to consider on the subject. The tricksters role is to provide comedy – not in the modern sense of the word but in the true sense of what comedy is - of unlocking the rigid state of reality that leads to negative and terminal outcomes to create space for new horizons. The role is to glean comedy from tragedy, to facilitate a looping progression that underwrites the human experience – from tragedy to comedy and comedy to tragedy – a Sisyphean process endured for eternity. But from within the view of the trickster, this utility is neither sought nor gained.  After performing their task and achieving their end, tricksters are no longer required – their job is simply to view potential tragedy, illuminate the inconsistencies, laugh at the paradoxes, and make merry while the sun shines. After Ragnarok With some irony (and perhaps without any at all) the Norse’ Ragnarok, which translates to ‘Twilight of the Gods’, is a book title of Nietzsche, who was inspired by Wagner (before falling out).  For Nietzsche, after the death of God there would be a ‘revaluation of all values’ in which new ideals would be discovered, new value systems would be built. Similarly to the Norse Ragnarok, potentially new Gods to discover? For all of the historical sentiment of the crypto community and its value in the context of the Jester archetype, it is this new step that is most important now – to discover and build new value systems. For this, there are instructive elements already existing in crypto. De-centralisation Vs centralisation includes a socio-cultural theory of power. Bitcoin includes a philosophy of monetary value – to answer the question of what money is for the first time in history. The tokenomics incentive structures and systematic game theory are some of the most sophisticated systems of their kind that we have ever seen There are likely others as well as other ways to view things. So perhaps embedded within the trickster archetype and its ability to be a God-killer are some of the values we should consider in a revaluation of values – in the world after Ragnarok – after we realise in the coming period that the USD has failed as a global reserve currency. That we cannot trust in God. After all, it is an historical problem of authoritative hierarchies accruing power and consolidating it until power becomes a self-realising end-goal. Tragedy inevitably emerges from a reality unable to contradict itself, with too much power to be contradicted by others. This is how neo-liberalism, the belief in competitive free-markets as an inherently moral end-goal in itself requires trillions of extraneous liquidity to support the ‘free’ market. The moral hazard is in an action that utterly annihilates the foundation of the reasoning behind that action. A circular logic, a feedback loop within a system - an ideological inbreeding. Symbolised by the pyramid also represented on the US Dollar, authoritative hierarchy is inevitably degenerative, inevitably increasingly centralised. Through this process it always and must become ridiculous, highlighted by whomever is willing to ridicule – unless it employs some trickster/jester mechanism. So can maintaining a trickster archetype inoculate us against this ancient and recurring problem - that make realising the trickster necessary?

Cryptocurrencies and Loki's Part in the Current Global Financial System

As many of you may have been tracking, the Australian Parliament has been deliberating on a piece of legislation for the last couple of months called the . This bill seeks to give Australian law enforcement and intelligence agencies powers to force tech companies and telcos operating in Australia to do any number of things. Here’s a partial list: Telecommunications and Other Legislation Amendment (Assistance and Access) Bill 2018 (a) removing one or more forms of electronic protection that are or were applied by, or on behalf of, the provider; or (b) providing technical information; or © installing, maintaining, testing or using software or equipment; or (d) ensuring that information obtained in connection with the execution of a warrant or authorisation is given in a particular format; or (f) assisting with the testing, modification, development or maintenance of a technology or capability; or (h) modifying, or facilitating the modification of, any of the characteristics of a service provided by the designated communications provider; or (j) an act or thing done to conceal the fact that any thing has been done covertly in the performance of a function, or the exercise of a power, conferred by a law of the Commonwealth, a State or a Territory, so far as the function or power relates to: (i) enforcing the criminal law and laws imposing pecuniary penalties; or (ii) assisting the enforcement of the criminal laws in force in a foreign country; or (iii) the interests of Australia’s national security, the interests of Australia’s foreign relations or the interests of Australia’s national economic well-being. Pretty scary stuff, huh? We thought so too. We’ve been tracking the progress of the bill and have been trying to analyse it to work out what the implications might be for software companies both inside and outside of Australia, and especially for Loki. The Assistance and Access Bill 2018 (Shortened to AAA18 for the rest of the article) gives Australian agencies the ability to issue 3 types of notices to ‘communications service providers.’ The definition of ‘provider’ in the legislation is very broad. Pretty much anyone that provides any service or product that involves the internet could fall under its scope, and the notices that can be issued increase in scope and obligation, and are called Technical Assistance Requests (TARs), Technical Assistance Notices (TANs), and Technical Capability Notices (TCNs). The latter is a legally enforced instruction to create or modify features to give an agency a new technical capability. Although this notice must come from the Attorney General of Australia, the scope for new espionage tools to be created by this notice is extremely broad. The scariest thing about this bill is the penalties given to providers who leak information about the investigation or notice, or refuse to comply with the notice. Jail sentences as long as 10 years could apply to a whistleblower, and given that these notices could be issued to companies and individuals that provide services to Australians, these notices could be issued to almost anyone around the world. With strong extradition treaties, this legislation could reach people across the nations of the Five Eyes Alliance (UK, US, AU, NZ, CA) and beyond. Some, myself included, strongly suspect that this is a coordinated effort by the Five Eyes alliance to gain access to the world’s most popular applications, as the UK recently to the already controversial Investigatory Powers Act of 2016, so that it is now quite closely aligned with this new Australian legislation. They even use similar terms, with the main similarity being the Technical Capability Notice. pushed through an amendment Thankfully, AAA18 does explicitly state that these notices cannot be used to force a company to break its own encryption, introduce security flaws, or deliberately ignore existing flaws. It also explicitly says that these notices cannot be used to introduce a ‘systemic weakness’ into the product or service. There were a great many concerns about the definition of ‘systemic weakness’ being too vague in the legislation. The final amendment to the bill gave us the following definitions: systemic vulnerability means a vulnerability that affects a whole class of technology, but does not include a vulnerability that is selectively introduced to one or more target technologies that are connected with a particular person. For this purpose, it is immaterial whether the person can be identified. systemic weakness means a weakness that affects a whole class of technology, but does not include a weakness that is selectively introduced to one or more target technologies that are connected with a particular person. For this purpose, it is immaterial whether the person can be identified. target technology : (a) for the purposes of this Part, a particular carriage service, so far as the service is used, or is likely to be used, (whether directly or indirectly) by a particular person, is a target technology that is connected with that person; and (b) for the purposes of this Part, a particular electronic service, so far as the service is used, or is likely to be used, (whether directly or indirectly) by a particular person, is a target technology that is connected with that person; and © for the purposes of this Part, particular software installed, or to be installed, on: (i) a particular computer; or (ii) a particular item of equipment; used, or likely to be used, (whether directly or indirectly) by a particular person is a target technology that is connected with that person; and (d) for the purposes of this Part, a particular update of software that has been installed on: (i) a particular computer; or (ii) a particular item of equipment; used, or likely to be used, (whether directly or indirectly) by a particular person is a target technology that is connected with that person; and (e) for the purposes of this Part, a particular item of customer equipment used, or likely to be used, (whether directly or indirectly) by a particular person is a target technology that is connected with that person; and (f) for the purposes of this Part, a particular data processing device used, or likely to be used, (whether directly or indirectly) by a particular person is a target technology that is connected with that person. For the purposes of paragraphs (a), (b), ©, (d), (e) and (f), it is immaterial whether the person can be identified. What this effectively means is that providers can not be forced to make changes to their products that negatively affect every user of that product. Instead, they can only be ordered to create the means by which they could selectively inject weaknesses or vulnerabilities into a specific product in use by a specific targeted person. However, AAA18 gives the legislative authority for agencies to create and install monitoring tools and other intrusive mechanisms into all kinds of software and hardware. For this iteration of the bill, these tools can only be switched on and used to target crimes with minimum sentences of 3 years or more. However, there is nothing to say that won’t change and very little oversight is required by this bill. It is also extremely problematic that these tools will even exist in the first place. If they fall into the wrong hands, the effects will be devastating. The NSA in the US developed a range of surveillance techniques that were eventually leaked and used against American interests by criminals and foreign governments, and the same will likely happen here. The introduction of AAA18 and its UK equivalent means that applications such as WhatsApp, Signal, Facebook Messenger, Gmail, and any other popular communications medium can silently turn into a monitoring device for ASIO, ASDS, AFP, GCHQ, MI5, and so on. The companies behind these products can’t utter a peep about these notices or their repercussions, or warn their users. They are even given protections to indemnify themselves against any civil cases caused by any later discovery of eroded privacy. This bill does not require these companies to break the encryption of their systems, but there are plenty of other things these agencies could force companies to create and install, such as tools that would allow them to remotely pull information from a specific user’s device post-encryption, or gain access to these services’ servers where treasure troves of metadata could be harvested. Every ‘private’ messaging application out there could now be forced to send data back to intelligence agencies about who is communicating to who and when the communications are taking place. , and companies across the world can be compelled to follow them. This isn’t hypothetical anymore — these powers are now law We will not know how widely these powers will be deployed until the first annual report is released. All it takes is one TCN for each of Facebook, Google, and WhatsApp, and the vast majority of private communications will be corruptible at the whim of the UK/Australian Intelligence and Law Enforcement community. The information that they can gather from this can easily be shared across the Five Eyes Alliance, effectively giving these tools to the governments of the whole English speaking world. For advocates of digital privacy such as myself, this is a very concerning development. If anything, it only strengthens the need to shift the paradigm in communication tools so that they are decentralised, open source and private by default. If we succeed, laws such as these can’t dissolve our ability to collectively access private spaces online. I have said this before, but access to these online spaces is critical to a healthy 21st century democracy, and our governments are too quick to dismiss the need for widespread digital security. How it will affect Loki Obviously, we were terrified when we first saw this bill. The potential for the project to be entirely undermined by this legislation did not go unnoticed. We had begun to consider how we might set up failsafes to allow people to catch bad code being injected into our codebase, or to pay someone external to Loki to do regular inspections of our binaries that we release and ensure they are not leaking extra information or mismatching the codebase in some way. If we were to be issued a TCN, we would not be able to tell anyone about it. If we set up some sort of canary system, we could be imprisoned. So whatever failsafe we did set up would have to be external to Loki, and would have to be regularly auditing us to make sure we haven’t been compromised before a TCN was issued. We had also considered that we may have to leave Australia altogether. However, given the legislation allows them to target any company whether they are in Australia or not, and given most of the Loki team currently lives in Australia and has family here, this would be a very extreme and difficult move to make. None of us would ever be able to return home if we were issued a notice and refused to comply. We may have simply been extradited out of wherever we moved to anyway, so this option seemed extreme and ineffective. New amendments to the bill make this move unnecessary. Our analysis of the bill and its proposed amendments lead us to the following conclusions: With the addition of the amendment defining ‘systemic weakness,’ any modification to the Loki source code that gives authorities new capabilities would almost certainly be classed as a systemic weakness. Thus, we see it is a nearly impossible that we would be forced to implement any privacy degrading code into the software that we release to the public. It is feasible that we may be forced to develop an alternative client software for authorities to use with additional data collecting features or something of the sort. Thankfully, given the network is regulated by Service Nodes, the authorities would still need to own 40%+ of the entire Service Node network to make this effective for widespread surveillance. Given our economic assumptions as outlined in our , this is likely to be prohibitively expensive for the government, particularly if the usage of Loki is so high that it is worth the government issuing a TCN in the first place. Cryptoeconomics proposal As long as we are able to keep our code open source, we can guarantee that our code will always be auditable. As such, it may make sense to go over our existing licenses and convert them to GPLv3 to prevent closed source software derived from our implementation from appearing. We will require written permission from the copyright holders of the projects we have forked in order to achieve this. As Loki is a decentralised system with extensive privacy protections, there is very little we as the Loki team can do to de-anonymise our users even if we wanted to. The extent of the information we might be able to provide to authorities are all known attacks such as DPI and traffic shape correlation, however we don’t have any additional information at hand than any other node operator on the network. This means that if we are issued a notice, we won’t be much use to these agencies anyway. The chances that Loki will eventually be issued a notice are fairly high, however such a notice would not result in compromising the privacy that our system provides. The same can not be said of other communication platforms, where the service provider has central control over the data being transported. Your phone number is attached to your WhatsApp and Signal accounts, and all of the metadata that you create is now up for grabs. Only by decentralising the routing and storage of communications can it become truly private under this new legislation. The case for a system like Loki has never been stronger. What you can do to stay private on the internet With this new legislation, it is important to understand and be vigilant about our privacy on the internet. Wherever practical, using open source software is a good start to protecting one’s privacy. Open source software is much more trustworthy than closed source applications as the code is generally reviewed by people form all over the world. If any backdoors or questionable features are added to this software, alarm bells should be ringing shortly afterwards. Whenever possible, you should also always build the open source software you are using from source. This will remove another layer of trust, as you’ll know the application you build is running exactly what is in the codebase you want to use. Nothing extra, nothing less. It is entirely feasible that modified versions of applications can put in place instead of the real ones if a notice compels a developer to do so. Reproducible builds help to mitigate against this, and this is something we are now striving towards, but it is your responsibility to check you didn’t get served a fake version of the application you want to use. VPNs are a decent first step in protecting network privacy, but this legislation could severely undermine their utility. If the VPN you use is compelled to install monitoring systems in their service, you might just be handing your browsing and connection information straight over to the authorities and be none the wiser. A better approach is to use some sort of mixnet. Tor is the obvious choice for the time being, but when Lokinet is launched, you should use it, as you’ll will be able to use essentially any program you like straight out of the box (it’s both TCP and UDP friendly). You’ll enjoy low latency, and a dedicated network of incentivised nodes that you can reasonably trust to not be largely comprised of nodes run by surveillance agencies. I hope this article has clarified the situation for you, and given you some insight to why decentralised privacy is so important. If you’d like to follow what we’re doing at Loki, head to https://loki.network Originally published at loki.network .

Loki’s Response to the Assistance and Access Bill 2018

Net neutrality via blockchain: analyzing Loki Network’s onion routing mixnet

Five years ago, I co-founded a company — — with some friends. We raised some money and built a game — an iPhone-based Pokemon-esque creature-collecting game set in a world that dynamically evolved to match your real-world surroundings. (Yes, I pitched that >1000 times). We made a bit of money. We rented out a house in Silicon Valley (with a pool). And we were acquired by Yahoo in 2013. Loki Studios I’ve spent a lot of time processing the many, many things that happened. I finally sat down and wrote it all out, for my own records. What went right, wrong, what I’d do differently in the future, whatever. A lot of what’s written below may be well-known knowledge — clichés, even. But it’s a collection of my own learnings and growth, so I wrote it anyway. It’s written conversationally, as advice for future me. After spewing it all out, I thought I might as well share. On co-founders We started as four. We lost two. One to misalignment of goals, another to misalignment of lifestyle expectations. A year in, we added one. You don’t need to be best friends with your co-founders. But you do require mutual respect, and a lot of trust. Try finding a point on which you disagree — ethics, politics, religion, whatever. do you disagree? Would you be comfortable with that at 50x intensity, when everything around you is going down in flames? How You don’t need to co-found with clones of yourself, aligned on everything. Solo founding is difficult because has weaknesses. So when founding with someone else, do you balance one another’s strengths and weaknesses? Enough to get to the milestones you want to achieve? You may not need a financial controller now, but maybe one of you should know how to use Photoshop. And one of you should be decent at external communications. everyone And while skill sets are important, look beyond the obvious. Are you cautious and strategic? Can your co-founder be the voice of recklessness, pushing you forward? Are you loud and emotional? Is someone on your team thoughtful and quiet? But see above note on disagreeing with mutual respect. On ownership Don’t split ownership equally. Yes, everyone told us this. We did it anyway. We were 4 Stanford CS grads and nothing differentiated us in terms of experience. How could it have gone any other way?? Doesn’t matter, don’t do it. Work together a couple months, if you have to. See what you’re each ready to contribute to the company. On listening to users Your most vocal customers represent 1% of your company, within an order of magnitude in either direction. Either way, it’s a small minority. — they represent the early adopters who will spread the word; trumpet your triumphs; disavow and curse you on your mistakes. Listen to them — they represent the early adopters who represent extremes of your user base. We expected Geomon to be playable over the course of 2–3 months. Some users hit max level in the course of a weekend and thirst for more. . Don’t listen to them The end-game features for your fanatics will not be the ones to help your game grow (Fun fact: we took engaging our users seriously and replied to each user as an in-game character — the equivalent of Professor Oak. Each email started with: “Hey Field Agent! Thanks for reaching out to us…”) On users hating change Everyone knows this too. But I’ll re-iterate it here. Whether you’re adding a requested new feature or improving your graphics, some very vocal users will always eviscerate you. Very publicly. Take the feedback under consideration. Do what you think is best. Move on. On building games (mostly for you lifelong gamers, like me) It’s been a lifelong dream of mine to build games. Every game we played, my brother and I would ruminate on how to improve the game, or what kind of characters would make the game awesome. This is indicative of an awesome game foundation. When you have built a game inherently logical and fun, anyone — every 8 year old kids — can come up with great ideas on how to improve. We learned the hard way that building a fundamentally sound game is incredibly difficult. How many elements should exist? 12 sounds like a nice, round number. Nope! Way too complex, nobody remembers what counters what. How much experience should gathered level up, allowing for a nice feeling of progression while also not getting people to the end game too quickly? How much should the world be dynamic and responsive to your actions? What percent of the game should be “on rails” and scripted to ensure a good first experience? How do you ensure the single-player mode is fun for those who prefer PvE, while the multi-player mode is enjoyable in the long term? We spent hundreds of hours interacting with top players in the game who understand newly evolving strategies and tactics than we did ourselves. Gamers are clever and bright and, in the age of the internet and forums and guides, new edge case tactics spread like wildfire and can rapidly destroy the precarious balance of your game. There are a hundred aspects that require a dedicated game architect. We jumped into it headfirst slapping together a bunch of mechanics that sounded , and had to backtrack later to ensure a cohesive game. awesome On revenue I “grew up” in an era where it seemed sites and apps could seemingly ignore revenue. As long as you grew your user base like wildfire, there would be financial support from eager investors to get you to the next growth stage. ! We spent 18 months without putting much effort into revenue. We were growing comfortably. We were profitable on weekends and holidays, when people decided to spend. But this wasn’t the trajectory I promised my investors, my employees, or myself. This wasn’t the promised land of hockey sticks! You could IPO without ever having been profitable So we spent 6 months focusing on revenue. And through a lot of studying, experimentation and hard work we achieved ramen profitability (made enough money to feed ourselves pizza and instant ramen). Had we started this a year earlier, who knows what would have happened? Monetization may never have worked without the first 18 months we put into building the game. Then again, maybe we could have entered a healthy, profitable growth loop allowing us to create the series of games we had originally envisioned. Who knows?? On design So… design matters. A lot. We had a kickass idea and every gamer we pitched was sold in 3 words — “pokemon for iOS”. But honestly, when they saw our game, I distinctly remember the slightly scrunched faces concealing a thinly veiled disappointment. Our game was fun! But we turned away a lot of mainstream gamers when they saw the geeky graphics we had come up with. By year two, I understood it was very important to re-skin our game. But it was hard. There were dozens of pages to redo and a hundred thousand users we might lose (see “On users hating change”). Too little, too late, and no one was experienced with visual design. So… next time :) The concept was awesome (in my humble opinion). The design did not appeal to the mainstream. On startups We started this company because we were so excited about the concept of the game. We wanted it to exist. We wanted to play it. We spent a summer working on it, foregoing a summer internship and salary. It was so much fun we dropped out of school to keep working on it. We survived on prior internship savings. When one co-founder ran out, the rest of us chipped in. When that ran out, I raised some money. When we could no longer keep up with the work, we hired more people. That’s all to say… we didn’t do it all to . We did it because we wanted to create something we were really stoked about. And without that, I don’t think we’d have made it through the roadblocks, disappointments, and emergencies. I lived a decade in 2.5 years. I grew a lot of white hair (true story). If (when) I do another project in the future, it’ll be when I find another passion like this. do a start-up On teams It’s all true what they say. When you’re going through the toughest of times your team’s all you’ve got. A great hire is worth 20x a decent hire. Don’t settle for less. And to steal a quote from someone wise, when you’re interviewing someone “if there’s any doubt, there’s no doubt. Don’t hire.” On getting help I got help from so many people so many times. On every subject imaginable. And I don’t think I appropriately took steps to thank many of the people involved. Something I hope to take steps to remedy. Thank you Ravi for teaching me how to fundraise. Thank you Shawn for lessons in game design. Thank you James for teaching me a little about art — sorry we didn’t get to work together more. Thank you Katy, John, Kyle, Tim, Amy, Annlyn, Richard and Jeremy for your help in designing our creatures and world. Thank you Dan, Jessica and Jason for your help in testing the earliest versions of the game. Thank you Yu-kai on helping build our competitive scene. Thank you Damon and Alex for making the introductions that changed the course of our careers. Thank you Osuke, Arthur, Josh, Matt, Osman, David and Miguel for believing in a handful of nobodies. Thank you Pa, Ma, Chris, and all my relatives and friends I leaned on while going through this. Thank you to our users for supporting our dreams. And thank you to the team — Chris, Brian, Sam, Gene, Bart, Jess, Russ, Marco, Juan and Rico for a whirlwind of an unforgettable couple of years.

10 lessons from Loki Studios

True Anonymity, Privacy, and SNApps With Simon Harman of Loki

This paper is available on under CC 4.0 license. Arxiv Authors: (1) Md Masud Rana, Department of Mathematics, University of Kentucky; (2) Duc Duy Nguyen, Department of Mathematics, University of Kentucky & ducnguyen@uky.edu. Table of Links Abstract & Introduction Datasets and Results Methods Conclusion, Data and Software Availability, Competing interests, Acknowledgments & References 4 Conclusion The study of protein-protein interactions (PPIs) and the prediction of mutation-induced binding free energy changes are of great importance in understanding the molecular basis of biological processes. The application of geometric graph theory and atom-level graph coloring techniques provides a powerful framework for analyzing biomolecules and capturing their intricate relationships. By utilizing the concept of geometric subgraphs and constructing multi-scale weighted colored geometric subgraphs (MWCGS), we can effectively represent the structural and functional properties of PPIs. The site-specific MWCGS features allow us to extract meaningful patterns and characteristics, shedding light on the effects of mutations and the underlying molecular interactions. In this work, we developed a mutation-induced binding free energy change predictor, called GGL-PPI, by incorporating site-specific MWCGS features for PPIs and gradient-boosting trees. Our method demonstrates superior performance compared to existing methods. The model was validated on three datasets: AB-Bind S645, SKEMPI 1.0 S1131, and SKEMPI 2.0 S4169 and S8338, showcasing its robustness and effectiveness. Furthermore, GGL-PPI was evaluated on a blind test set, the Ssym dataset. To prevent data leakage between the test and training sets, the model was trained on a homology-reduced balanced training set Q3488. This approach ensures the reliability and fairness of the evaluation process. GGL-PPI exhibits the most unbiased and superior performance in predicting binding free energy changes for both direct and reverse mutations, outperforming other existing methods, particularly for reverse mutations. Overall, the results highlight the potential of the GGL-PPI approach in accurately predicting mutation-induced binding free energy changes in protein-protein interactions, providing valuable insights into the molecular mechanisms underlying protein-protein interactions and facilitating drug design and discovery efforts. 5 Data and Software Availability The source code is available at Github: https://github.com/NguyenLabUKY/GGL-Mutation. 6 Competing interests No competing interest is declared. 7 Acknowledgments This work is supported in part by funds from the National Science Foundation (NSF: # 2053284, # 2151802, and # 2245903), and the University of Kentucky Startup Fund. References [1] Dana Chuderland and Rony Seger. Protein-protein interactions in the regulation of the extracellular signal-regulated kinase. Mol. Biotechnol., 29:57–74, 2005. [2] Harry Jubb, Tom L Blundell, and David B Ascher. Flexibility and small pockets at protein–protein interfaces: new insights into druggability. Prog. Biophys. Mol., 119(1):2–9, 2015. [3] Mileidy W Gonzalez and Maricel G Kann. Chapter 4: Protein interactions and disease. PLoS Comput. Biol., 8(12):e1002819, 2012. [4] Cunliang Geng, Li C Xue, Jorge Roel-Touris, and Alexandre MJJ Bonvin. Finding the δδg spot: are predictors of binding affinity changes upon mutations in protein–protein interactions ready for it? Wiley Interdiscip. Rev. Comput. Mol. Sci., 9(5):e1410, 2019. [5] Rocco Moretti, Sarel J Fleishman, Rudi Agius, Mieczyslaw Torchala, Paul A Bates, Panagiotis L Kastritis, Joao PGLM Rodrigues, Mika¨el Trellet, Alexandre MJJ Bonvin, Meng Cui, et al. Community-wide evaluation of methods for predicting the effect of mutations on protein–protein interactions. Proteins: Struct. Funct. Genet., 81(11):1980–1987, 2013. [6] Jeffrey R Brender and Yang Zhang. Predicting the effect of mutations on protein-protein binding interactions through structure-based interface profiles. PLoS Comput. Biol., 11(10):e1004494, 2015. [7] Ning Zhang, Yuting Chen, Haoyu Lu, Feiyang Zhao, Roberto Vera Alvarez, Alexander Goncearenco, Anna R Panchenko, and Minghui Li. Mutabind2: predicting the impacts of single and multiple mutations on protein-protein interactions. Iscience, 23(3):100939, 2020. [8] Carlos HM Rodrigues, Yoochan Myung, Douglas EV Pires, and David B Ascher. mcsm-ppi2: predicting the effects of mutations on protein–protein interactions. Nucleic Acids Res., 47(W1):W338–W344, 2019. [9] Minghui Li, Marharyta Petukh, Emil Alexov, and Anna R Panchenko. Predicting the impact of missense mutations on protein–protein binding affinity. J. Chem. Theory Comput., 10(4):1770–1780, 2014. [10] Hafumi Nishi, Manoj Tyagi, Shaolei Teng, Benjamin A Shoemaker, Kosuke Hashimoto, Emil Alexov, Stefan Wuchty, and Anna R Panchenko. Cancer missense mutations alter binding properties of proteins and their interaction networks. PloS one, 8(6):e66273, 2013. [11] Mu Gao, Hongyi Zhou, and Jeffrey Skolnick. Insights into disease-associated mutations in the human proteome through protein structural analysis. Structure, 23(7):1362–1369, 2015. [12] Alessia David, Rozami Razali, Mark N Wass, and Michael JE Sternberg. Protein–protein interaction sites are hot spots for disease-associated nonsynonymous snps. Hum. Mutat., 33(2):359–363, 2012. [13] H Billur Engin, Jason F Kreisberg, and Hannah Carter. Structure-based analysis reveals cancer missense mutations target protein interaction interfaces. PLoS One, 11(4):e0152929, 2016. [14] Amelie Stein, Douglas M Fowler, Rasmus Hartmann-Petersen, and Kresten Lindorff-Larsen. Biophysical and mechanistic models for disease-causing protein variants. Trends Biochem. Sci., 44(7):575–588, 2019. [15] Stephanie Portelli, Jody E Phelan, David B Ascher, Taane G Clark, and Nicholas Furnham. Understanding molecular consequences of putative drug resistant mutations in mycobacterium tuberculosis. Sci. Rep., 8(1):1–12, 2018. [16] Sundeep Chaitanya Vedithi, Sony Malhotra, Madhusmita Das, Sheela Daniel, Nanda Kishore, Anuja George, Shantha Arumugam, Lakshmi Rajan, Mannam Ebenezer, David B Ascher, et al. Structural implications of mutations conferring rifampin resistance in mycobacterium leprae. Sci. Rep., 8(1):5016, 2018. [17] Vytautas Gapsys, Servaas Michielssens, Daniel Seeliger, and Bert L de Groot. Accurate and rigorous prediction of the changes in protein free energies in a large-scale mutation scan. Angew. Chem., Int. Ed., 55(26):7364–7368, 2016. [18] Elizabeth H Kellogg, Andrew Leaver-Fay, and David Baker. Role of conformational sampling in computing mutationinduced changes in protein structure and stability. Proteins: Structure, Function, and Bioinformatics, 79(3):830–838, 2011. [19] Brian J Bender, Alberto Cisneros III, Amanda M Duran, Jessica A Finn, Darwin Fu, Alyssa D Lokits, Benjamin K Mueller, Amandeep K Sangha, Marion F Sauer, Alexander M Sevy, et al. Protocols for molecular modeling with rosetta3 and rosettascripts. Biochemistry, 55(34):4748–4763, 2016. [20] Kurt S Thorn and Andrew A Bogan. Asedb: a database of alanine mutations and their effects on the free energy of binding in protein interactions. Bioinformatics, 17(3):284–285, 2001. [21] MD Shaji Kumar and M Michael Gromiha. Pint: protein–protein interactions thermodynamic database. Nucleic Acids Res., 34(suppl 1):D195–D198, 2006. [22] MD Shaji Kumar, K Abdulla Bava, M Michael Gromiha, Ponraj Prabakaran, Koji Kitajima, Hatsuho Uedaira, and Akinori Sarai. Protherm and pronit: thermodynamic databases for proteins and protein–nucleic acid interactions. Nucleic Acids Res., 34(suppl 1):D204–D206, 2006. [23] Iain H Moal and Juan Fern´andez-Recio. Skempi: a structural kinetic and energetic database of mutant protein interactions and its use in empirical models. Bioinformatics, 28(20):2600–2607, 2012. [24] Justina Jankauskait˙e, Brian Jim´enez-Garc´ıa, Justas Dapk¯unas, Juan Fern´andez-Recio, and Iain H Moal. Skempi 2.0: an updated benchmark of changes in protein–protein binding energy, kinetics and thermodynamics upon mutation. Bioinformatics, 35(3):462–469, 2019. [25] Sarah Sirin, James R Apgar, Eric M Bennett, and Amy E Keating. Ab-bind: antibody binding mutational database for computational affinity predictions. Protein Sci., 25(2):393–409, 2016. [26] Cunliang Geng, Anna Vangone, and Alexandre MJJ Bonvin. Exploring the interplay between experimental methods and the performance of predictors of binding affinity change upon mutations in protein complexes. Protein Eng. Des. Sel., 29(8):291–299, 2016. [27] Sherlyn Jemimah, K Yugandhar, and M Michael Gromiha. Proximate: a database of mutant protein–protein complex thermodynamics and kinetics. Bioinformatics, 33(17):2787–2788, 2017. [28] Douglas EV Pires, David B Ascher, and Tom L Blundell. mcsm: predicting the effects of mutations in proteins using graph-based signatures. Bioinformatics, 30(3):335–342, 2014. [29] Cunliang Geng, Anna Vangone, Gert E Folkers, Li C Xue, and Alexandre MJJ Bonvin. isee: Interface structure, evolution, and energy-based machine learning predictor of binding affinity changes upon mutations. Proteins: Structure, Function, and Bioinformatics, 87(2):110–119, 2019. [30] Sherlyn Jemimah, Masakazu Sekijima, and M Michael Gromiha. Proaffimuseq: sequence-based method to predict the binding free energy change of protein–protein complexes upon mutation using functional classification. Bioinformatics, 36(6):1725–1730, 2020. [31] Guangyu Zhou, Muhao Chen, Chelsea JT Ju, Zheng Wang, Jyun-Yu Jiang, and Wei Wang. Mutation effect estimation on protein–protein interactions using deep contextualized representation learning. NAR genom. bioinform., 2(2):lqaa015, 2020. [32] Alexey Strokach, Tian Yu Lu, and Philip M Kim. Elaspic2 (el2): combining contextualized language models and graph neural networks to predict effects of mutations. J. Mol. Biol., 433(11):166810, 2021. [33] Xianggen Liu, Yunan Luo, Pengyong Li, Sen Song, and Jian Peng. Deep geometric representations for modeling effects of mutations on protein-protein binding affinity. PLoS Comput. Biol., 17(8):e1009284, 2021. [34] Piero Fariselli, Pier Luigi Martelli, Castrense Savojardo, and Rita Casadio. Inps: predicting the impact of nonsynonymous variations on protein stability from sequence. Bioinformatics, 31(17):2816–2821, 2015. [35] Grant Thiltgen and Richard A Goldstein. Assessing predictors of changes in protein stability upon mutation using self-consistency. PLoS One, 7(10):e46084, 2012. [36] Fabrizio Pucci, Katrien V Bernaerts, Jean Marc Kwasigroch, and Marianne Rooman. Quantification of biases in predictions of protein stability changes upon mutations. Bioinformatics, 34(21):3659–3665, 2018. [37] Dinara R Usmanova, Natalya S Bogatyreva, Joan Ari˜no Bernad, Aleksandra A Eremina, Anastasiya A Gorshkova, German M Kanevskiy, Lyubov R Lonishin, Alexander V Meister, Alisa G Yakupova, Fyodor A Kondrashov, et al. Self-consistency test reveals systematic bias in programs for prediction change of stability upon mutation. Bioinformatics, 34(21):3653–3658, 2018. [38] Duc D Nguyen, Tian Xiao, Menglun Wang, and Guo-Wei Wei. Rigidity strengthening: A mechanism for protein– ligand binding. J. Chem. Inf. Model., 57(7):1715–1721, 2017. [39] Md Masud Rana and Duc Duy Nguyen. Geometric graph learning with extended atom-types features for proteinligand binding affinity prediction. Comput. Biol. Med., 164:107250, 2023. [40] Jian Jiang, Rui Wang, and Guo-Wei Wei. Ggl-tox: geometric graph learning for toxicity prediction. J. Chem. Inf. Model., 61(4):1691–1700, 2021. [41] Menglun Wang, Zixuan Cang, and Guo-Wei Wei. A topology-based network tree for the prediction of protein–protein binding affinity changes following mutation. Nat. Mach. Intell., 2(2):116–123, 2020. [42] Duan Chen, Zhan Chen, Changjun Chen, Weihua Geng, and Guo-Wei Wei. Mibpb: a software package for electrostatic analysis. J. Comput. Chem., 32(4):756–770, 2011. [43] Xiang Liu, Huitao Feng, Jie Wu, and Kelin Xia. Hom-complex-based machine learning (hcml) for the prediction of protein–protein binding affinity changes upon mutation. J. Chem. Inf. Model., 62(17):3961–3969, 2022. [44] Douglas EV Pires and David B Ascher. mcsm-ab: a web server for predicting antibody–antigen affinity changes upon mutation with graph-based signatures. Nucleic Acids Res., 44(W1):W469–W473, 2016. [45] Peng Xiong, Chengxin Zhang, Wei Zheng, and Yang Zhang. Bindprofx: assessing mutation-induced binding affinity change by protein interface profiles with pseudo-counts. J. Mol. Biol., 429(3):426–434, 2017. [46] Quanya Liu, Peng Chen, Bing Wang, Jun Zhang, and Jinyan Li. dbmpikt: a database of kinetic and thermodynamic mutant protein interactions. BMC bioinformatics, 19(1):1–7, 2018. [47] Bian Li, Yucheng T Yang, John A Capra, and Mark B Gerstein. Predicting changes in protein thermodynamic stability upon point mutation with deep 3d convolutional neural networks. PLoS Comp. Bio., 16(11):e1008291, 2020. [48] Lijun Quan, Qiang Lv, and Yang Zhang. Strum: structure-based prediction of protein stability changes upon single-point mutation. Bioinformatics, 32(19):2936–2946, 2016. [49] Leo Breiman. Random forests. Machine learning, 45:5–32, 2001. [50] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273–297, 1995. [51] Duc Duy Nguyen, Kaifu Gao, Menglun Wang, and Guo-Wei Wei. Mathdl: mathematical deep learning for d3r grand challenge 4. J. Comput. Aided. Mol. Des., 34(2):131–147, 2020. [52] Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189–1232, 2001. This paper is available on under CC 4.0 license. Arxiv

PPIs and the Prediction of Mutation-Induced Binding Free Energy Changes: What It All Means

To unlock the main character’s theurgy moves in P3Reload, you have to have certain personas unlocked. So, it’s important to keep expanding the compendium. After explaining what they are and how to use them, the game gives you 1 to start with, but 6 more can be unlocked. Here’s how: Jack Brothers Personas you need: Jack o’Lanter, Jack Frost Its ability: “Deals medium Almighty damage to all foes. High chance of inflicting Down.” King and I Personas you need: Black Frost, King Frost Its ability: “Heavy Ice damage to all foes, ignoring resistances. Medium chance of Freeze.” Best Friends Personas you need: Decarabia, Forneus Its ability: “Next physical/magic attack will deal more than double damage for 1 ally.” Scarlet Havoc Personas you need: Siegfried, Mithras Its ability: “Deals severe Slash damage to all foes, ignoring resistances.” Trickster Personas you need: Susano-o, Loki Its ability: “Massive Almighty damage to all foes. High chance of inflicting an ailment.” Armageddon Personas you need: Helel, Satan Its ability: “Ultimate Almighty damage to all foes, but reduce’s user’s HP and SP to 1.” Are the Theurgy Special Moves Necessary? Necessary? No, you can get by without unlocking them or even using them. But if you’re struggling to get through Tartarus, especially the Gatekeepers, the Theurgy moves really come in handy. Regardless of the boss, its health, or how high a level it is, the Theurgy can put a sizable dent into their life bar. On top of the Gatekeepers, they work really well on the Monad bosses. Those bosses aren’t particularly hard but when you have to fight 3 back to back, having some extra firepower doesn’t hurt. And if you’re worried about these Theurgy moves making the experience too easy for you, you can just not use them. That’s the great thing about them. Which Theurgy Special Move Is the Best? The one that I pulled out the most was Scarlet Havoc. It really came in clutch when I wanted to steamroll Gatekeeper after Gatekeeper, and it’s legitimately one of the best if not the best in the game. However, the most powerful by far is Armageddon. It does 9,999 damage to all enemies. Which sounds great, until you realize it essentially wipes out your character’s HP and SP; it completely reduces both to 1. This isn’t really that big of a deal if you have a healer on your team, and you can regain SP if you have an invigorate ability, but it can be risky if you’re careless. So, while the other special moves aren’t bad, for my buck, Scarlet Havoc and Armageddon are the two best Theurgy Special Moves in Persona 3 Reload. Read More: The MCU Phase 4 Movies in Order The MCU Phase 3 Movies in Order The Apollo Justice Trilogy: The 5 Best Cases Feature image source

How to Unlock All the MC's Theurgy Super Moves in Persona 3 Reload

পারসোনা 3 রিলোড-এ সমস্ত MC এর Theurgy সুপার মুভগুলি কীভাবে আনলক করবেন

So schalten Sie alle Theurgy-Supermoves des MC in Persona 3 Reload frei

Cómo desbloquear todos los súper movimientos de teurgia del MC en Persona 3 Reload

Comment débloquer tous les super mouvements de théurgie du MC dans Persona 3 Reload

पर्सोना 3 रीलोड में एमसी के सभी थर्गी सुपर मूव्स को कैसे अनलॉक करें

ペルソナ 3 リロードですべての MC のテウルギー スーパー ムーブのロックを解除する方法

페르소나 3 리로드에서 모든 MC의 Theurgy 슈퍼 무브를 잠금 해제하는 방법

Como desbloquear todos os super movimentos de teurgia do MC no Persona 3 Reload

Как разблокировать все суперприемы теургии MC в Persona 3 Reload

Persona 3 Reload'da MC'nin Tüm Theurgy Süper Hareketlerinin Kilidini Açma

Cách mở khóa tất cả các chiêu thức siêu phàm của MC trong Persona 3 Tải lại

如何解锁《女神异闻录 3：重装上阵》中 MC 的所有神术超级动作

A couple of months ago, I stumbled upon this list of  Secure your API with these : 16 practices to secure your API Authentication 🕵️️ - Verifies the identity of users accessing APIs. Authorization 🚦 - Determines permissions of authenticated users. Data Redaction 🖍️ - Obscures sensitive data for protection. Encryption 🔒 - Encodes data so only authorized parties can decode it. Error Handling ❌ - Manages responses when things go wrong, avoiding revealing sensitive info. Input Validation & Data Sanitization 🧹 - Checks input data and removes harmful parts. Intrusion Detection Systems 👀 - Monitor networks for suspicious activities. IP Whitelisting 📝 - Permits API access only from trusted IP addresses. Logging and Monitoring 🖥️ - Keeps detailed logs and regularly monitors APIs. Rate Limiting ⏱️ - Limits user requests to prevent overload. Secure Dependencies 📦 - Ensures third-party code is free from vulnerabilities. Security Headers 📋 - Enhances site security against types of attacks like XSS. Token Expiry ⏳ - Regularly expiring and renewing tokens prevents unauthorized access. Use of Security Standards and Frameworks 📘 - Guides your API security strategy. Web Application Firewall 🔥 - Protects your site from HTTP-specific attacks. API Versioning 🔄 - Maintains different versions of your API for seamless updates. While it's debatable whether some points relate to security, versioning, the list is a good starting point anyway. In this two-post series, I'd like to describe how we can implement each point with Apache APISXI (or not). e.g., Authentication Authentication is about identifying yourself with a system. It requires a proof. provides two kinds of authentications: internal, with APISIX checking credentials, and external, when delegated to a third party. All authentication mechanisms work via plugins. Here's the current list of available authentication plugins. Apache APISIX Type Name Description Internal key-auth Authenticate via an HTTP Header Internal basic-auth Relies on a browser callback Internal jwt-auth Uses a JWT token to authenticate External authz-keycloak Delegates to Keycloak External authz-casdoor Delegates to Casdoor External wolf-rbac Delegates to wolf External openid-connect Delegates to an -compliant third-party OpenID Connect External cas-auth Delegates to a -compliant third-party CAS External hmac-auth Delegates to an -compliant third-party HMAC External authz-casbin APISIX assigns authenticated calls to a . For example, we can create a consumer authenticated with the plugin: consumer key-auth consumers:
  - username: john
    plugins:
      key-auth:
        key: mykey Every request containing the header with the key will be assigned to the consumer . apikey mykey john Authorization Authentication alone isn't enough. Once a request to a URL has been authenticated, we need to decide whether it's allowed to proceed further. That's the role of authorization. Authorization [...] is the function of specifying access rights/privileges to resources, which is related to general information security and computer security, and to access control in particular. More formally, "to authorize" is to define an access policy. -- Authorization on Wikipedia Apache APISIX implements authorization mainly via the plugin. Here's the most straightforward usage of the plugin: consumer-restriction consumer-restriction consumers:
  - username: johndoe                     #1
    plugins:
      keyauth:
        key: mykey

routes:
  - upstream_id: 1                        #2
    plugins:
      keyauth: ~
      consumer-restriction:
        whitelist:                        #3
          - johndoe Define a consumer Reference an already existing upstream Only allows defined consumers to access the route Most real-world authorization models avoid binding an identity directly to a permission. They generally bind a group (and even a role) so that it becomes easier to manage many identities. Apache APISIX provides the abstraction for this. consumer group consumer_groups:
  - id: accountants                      #1

consumers:
  - username: johndoe
    group_id: accountants                #2
    plugins:
      keyauth:
        key: mykey

routes:
  - upstream_id: 1
    plugins:
      keyauth: ~
      consumer-restriction:
        type: consumer_group_id          #3
        whitelist:
          - accountants Define a consumer group Assign the consumer to the previously defined consumer group Restrict the access to members of the defined consumer group, , i.e. accountants Input validation With Apache APISIX, you can define a set of JSON schemas and validate a request against any of them. My colleague Navendu has written an exhaustive blog post on the subject: . Your API Requests Should Be Validated I think it's not the API Gateway's responsibility to handle request validation. Each upstream has specific logic, and moving the validation responsibility from the upstream to the Gateway ties the latter to the logic for no actual benefit. In any case, the checkbox is ticked. IP Whitelisting Apache APISIX implements IP Whitelisting via the plugin. You can define either regular IPs or CIDR blocks. ip-restriction routes:
  - upstream_id: 1
    plugins:
      ip-restriction:
        whitelist:
          - 127.0.0.1
          - 13.74.26.106/24 Logging and Monitoring Logging and Monitoring fall into the broader category, also encompassing . Apache APISIX offers a broad range of Observability plugins in each category. Observability Tracing Type Name Description Tracing zipkin Collect and send traces according to the Zipkin specification Tracing skywalking Integrate with the project Apache SkyWalking Tracing opentelemetry Report data according to the OpenTelemetry specification Metrics prometheus Expose metrics in the Prometheus format Metrics node-status Expose metrics in JSON format Metrics datadog Integrate with Datadog Logging file-logger Push log streams to a local file Logging syslog Push logs to a Syslog server Logging http-logger Push JSON-encoded logs to an HTTP server Logging tcp-logger Push JSON-encoded logs to a TCP server Logging udp-logger Push JSON-encoded logs to a UDP server Logging kafka-logger Push JSON-encoded logs to a Kafka cluster Logging rocketmq-logger Push JSON-encoded logs to a RocketMQ cluster Logging loki-logger Push JSON-encoded logs to a Loki instance Logging splunk-hec-logging Push logs to a Splunk instance Logging loggly Push logs to a Loggly instance Logging elasticsearch-logger Push logs to an Elasticsearch instance Logging sls-logger Push logs to Alibaba Cloud Log Service Logging google-cloud-logging Push access logs to Google Cloud Logging Service Logging tencent-cloud-cls Push access logs to Tencent Cloud CLS Rate Limiting Rate Limiting protects upstreams from Distributed Denial of Services attacks, DDoS. It's one of the main features of reverse proxies and API Gateways. APISIX implements rate limiting through three different plugins: a.k.a The Plugin limits the number of concurrent requests to your services limit-conn The Plugin limits the number of requests to your service using the limit-req leaky bucket algorithm The Plugin limits the number of requests to your service by a given count per time. The plugin uses the Fixed Window algorithm. limit-count Let's use for the sake of example: limit-count routes:
  - upstream_id: 1
    plugins:
      limit-count:
        count: 10
        time_window: 1
        rejected_code: 429 The above configuration snippet protects the upstream from being hit by more than ten requests per second. It applies to every IP address because of the default configuration. The complete snippet would look like the following: routes:
  - upstream_id: 1
    plugins:
      limit-count:
        count: 10
        time_window: 1
        rejected_code: 429
        key_type: var
        key: remote_addr When dealing with APIs, there's a considerable chance you want to differentiate between your clients. Some might get a better rate for different reasons: they paid a premium offer, they are considered strategic, they are internal clients, etc. The same consumer could also use different IP addresses because they run on various machines with other APIs. Allowing the same consumer more calls because they execute their requests on a distributed infrastructure would be unfair. As it stands, the IP is not a great way to assign the limit; we prefer to use a named consumer or, even better, a consumer group. It's perfectly possible with APISIX: consumer_groups:
  - id: basic
    plugins:
      limit-count:
        count: 1
        time_window: 1
        rejected_code: 429
  - id: premium
    plugins:
      limit-count:
        count: 10
        time_window: 1
        rejected_code: 429

consumers:
  - username: johndoe
    group_id: basic
    plugins:
      keyauth:
        key: mykey1
  - username: janedoe
    group_id: premium
    plugins:
      keyauth:
        key: mykey2

routes:
  - upstream_id: 1
    plugins:
      key-auth: ~ Now, can only send a request every second, as he's part of the plan, while can request ten times as much as part of the premium plan. johndoe basic janedoe Security Headers The OWASP lists plenty of you can set to improve the security of your web apps and APIs. Apache APISIX provides two dedicated plugins for specific security risks: HTTP Headers CORS CSRF For any other header, you can use the more generic plugin to add them. Finally, we can also remove default HTTP response headers, such as , to make targeted attacks less likely. response-rewrite Server global_rules:                               #1
  - id: 1
    plugins:
      response-rewrite:
        headers:
          set:
            X-Content-Type-Options: nosniff #2
          remove:
            - Server                        #3 Do it on every route - security by default! It still can be overridden on a per-route basis in case of need. Tell the browser not to infer the content type if it's not explicitly set Don't advertise the server Conclusion We've seen how to configure Apache APISIX to secure your APIs against 7 of the 16 rules in the original list. The rules left could be less straightforward to implement; we will cover them in the second installment. Originally published at on February 20th, 2024 A Java Geek

16 Best Practices For Securing Your APIs with Apache APISIX - Part 1

After Avengers: Endgame, it seemed that a lot of the storylines in the Marvel Cinematic Universe (MCU) were coming to a close. However, not all of them. And on top of that, new ones were already on the horizon. These storylines would come to fruition in the next phase of the MCU: phase 4. This phase saw the continuation of Black Widow’s, Black Panther’s, Doctor Strange’s, and Spider-Man’s stories, and it also saw the introduction of Shang-Chi and the Eternals. If this sounds like a lot, don’t worry. Because here are the MCU Phase 4 movies in order. Phase 4 Movies in Order Black Widow Shang-Chi and the Legend of the Ten Rings Eternals Spider-Man: No Way Home Doctor Strange in the Multiverse of Madness Thor: Love and Thunder Black Panther: Wakanda Forever 1. Black Widow (2021) Although Black Widow was introduced early in the MCU, it would take over 10 years for her to get her own stand-alone movie. The movie would give fans more of an insight into her character, her past, and would cover time periods unexplored by other movies. It would also introduce her family: Yelena Belova, the Red Guardian, and Melina Vostokoff as they reunited to take down the leader of the Red Room. But they’ll have to take care of his number one assassin first: Taskmaster. 2. Shang-Chi and the Legend of the Ten Rings (2021) A son going against his father; A tale as old as time. But the reason this type of story has remained prominent is because it’s a good one, and Shang-Chi managed to not only reach its expectations but exceed them. Shang-Chi was living in San Francisco, wanting to put his past behind him. But the past reaches out to him and forces him to not only confront it but also to confront his father: Xu Wenwu. However, this isn’t your typical father; he has immense power thanks to the ten rings. Shang-Chi will have to put all his heart and soul to overcome his dad. 3. Eternals (2021) Just like Shang-Chi’s dad has lived for over a 1,000 years, the Eternals have lived for a very long time. Except they lived for even longer, hiding in the shadows of history. What’s their purpose? To protect humans from the creatures known as Deviants. But when the seemingly extinct Deviants return from slumber, the Eternals will once again have to unite to take them down. 4. Spider-Man: No Way Home (2021) At the end of the last Spider-Man movie, the whole world finds out that Peter Parker is Spider-Man. And, of course, his life changes immediately. He’s being interrogated by government officials, he’s being treated differently in school, and he finally has to come clean to his Aunt May about his secret double life. However, Peter can live with all of that. What he can’t live with, however, is the lives of his friends being impacted by his decision to be a superhero. To fix this, he turns to one Doctor Strange for help. The good doctor proposes a spell that will make the whole world forget that Peter Parker is Spider-Man, but things don’t go as planned. What happens next is something that fans have been wanting to see for many years. 5. Doctor Strange in the Multiverse of Madness (2022) Coming right off the heels of No Way Home is Doctor Strange in the Multiverse of Madness. By this point, the multiverse concept was already something that the MCU was expanding on with shows like Loki and What If? But now, it was Doctor Strange’s turn to jump head-first into the multiverse. After he’s blindsided by a new threat, Doctor Strange and his new ally, America Chavez, accidentally stumble into another universe. They’ll have to fight their way back home to help out Wong and the rest of the sorcerers. 6. Thor: Love and Thunder (2022) After the events of Avengers: Infinity War and Endgame, Thor is going through some things. He lost his realm, his friend Heimdall, his brother Loki, and he feels responsible for the Blip. But in Love and Thunder, he’s finding his groove back. Although the realm of Asgard is gone, the people remain, and they need help. Asgardian children are kidnapped, and Thor and Thor have to save them. No, that’s not a typo. There are two Thors: Thor Odinson and the Might Thor who is revealed to be the returning Jane Foster. And good thing that there’s two of them because the villain who kidnapped the kids is Gorr the God Butcher 7. Black Panther: Wakanda Forever (2022) After the unfortunate passing of actor Chadwick Boseman, who played Black Panther/ T’Challa, plans had to pivot. In Wakanda Forever, T’Challa also passes away, which leaves Wakanda without a Black Panther. Although reluctant at first, T’Challa’s younger sister Shuri, is forced to take the mantle after Wakanda is threatened by Namor and the Atlanteans. What ensues is a battle between two powerful nations, both fighting to ensure the future of their people. Conclusion Marvel’s Phase 4 may have had its ups and downs, but overall, it was another fun section in the MCU’s history. Read More: The MCU Phase 3 Movies in Order The Apollo Justice Trilogy: The 5 Best Cases The New Tomb Raider Games in Order

The MCU Phase 4 Movies in Order

Let's learn about via these 67 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Media /Learn Repo 1. The Last of Us HBO Series; The Nth of Many As Naughty Dog's Award-winning game title, The Last of Us, makes its way to TV screens. I take a look at the relationships between gaming and TV. 2. Corona Porn: The Arousal of a Pandemic The media calamity of “The Corona” is disturbing and dismal, but even still, many are buzzed. It’s so titillating that we just can’t help but… participate ourselves. 3. 10 Advantages of Using a Crypto Advertising Network Starting from June I’m in charge of marketing at Squilla Capital, and we have two exciting releases hitting the market in September this year. Hence, I had to compile a marketing strategy. 4. "The Simpsons" May Indicate Mainstream Bitcoin Adoption by 2036 One of my colleagues recently shared an anecdote with me about ‘The Simpsons.’ During a chat about politics, her son told her, “Well, it was on ‘The Simpsons’ and then it came true.” And then he asked her, “How do they do that?” 5. How to Change Location on Hulu Are you looking for a way to change location on Hulu? Then, you have come to the right place because this content will tell you the easiest way to change Hulu 6. The Rise of Refurbished Phones — Except in the U.S. Refurbished phones are becoming more and more popular, but the U.S. is lagging behind in sales. Here's a look at why. 7. The 59 Questions From the Will Ferrell Joaquin Phoenix Interview On March 28 2018, Interview Magazine published Joaquin Phoenix Interview by Will Ferrell, and it became known as the greatest interview ever published. 8. Responsive Web Design: Understand And Apply It Once And For All As a Microverse student for one month, I realized that my first clone webpages didn’t have a responsive layout. I was using pixels in the Navbars, percentages in one section and rem in another, I didn’t have a rule or a standard procedure. The goal was to make it look like the original webpage IN MY SCREEN. As I advanced through the course, I learned that these units made all the
difference to the webpage’s response in different screen sizes. My code
reviewers would tell me to adjust my webpage because it looked different when they were viewing it. When I started having these issues, I made a lot of research on the subject,
but I was still a little confused about it. This article’s goal is to create a
simple and direct rule to achieve the required responsiveness any developer needs in their code. 9. How We Revamped an Online Newspaper with Ruby on Rails, React, and NodeJS The Village is an online city newspaper owned by the Redefine brand. It reports on important events of the day, culture, and entertainment in Moscow and St. Petersburg. Under the franchise, editorial offices also operate in other Russian cities, as well as in Belarus, Kazakhstan, and Ukraine. 10. Three Reasons Behind Ludicrous Spending in Streaming Media Netflix spent $16 billion in 2019 on producing and acquiring new content. Amazon pursued its originals journey with $6 billion. Quibi, a yet to launch streaming service raised a billion dollars to create a catalog for millennials. 11. Here's Why Journalists Won't Write About Your Startup The media is a key point of contact for the marketing team of any startup. As a tech journalist and freelance writer, I talk to hundreds of SMEs, startups, fortune 500 companies, investors and industry experts every year. I receive thousands of press releases and invitations to events and conferences. Many businesses do themselves and their product a real disservice meaning we can't or won't write about them. Let's walk through some of the most common mistakes so you can avoid them: 12. Video Transcoding And Optimization For Web With ffmpeg Made Easy In this short article, I follow the main steps of video transcoding and optimization for the web. I share basic tips and examples to consistently automate these processes inhouse using ffmpeg, a powerful and open source package. 13. The CEO of my company, Genius, must be Black When Tom and I started Genius in 2009, we wanted to build something that would grow bigger than rap lyrics. The first 4 songs I annotated on RapExegesis, as it was originally called, were rap songs. However, the 5th one was Elton John's "Rocket Man", the 6th was The Eagles' "Hotel California", and the 7th was a poem, the first non-lyrics post on the site - Emily Dickinson's "I'm Nobody!  Who Are You?" 14. Gitcoin CLR Matching Review Media Grants Gitcoin Media and Tech grants both need to be separated on the basis of matching algorithms. Media mostly requires fixed funding except when its expanding or its combination of media+tech project, whereas tech projects require more and more funding as they grow rapidly or succeeding. 15. The Top 8 Social Media Data Charts and Live Charts You Need to Know Much like that of many other industries, the landscape of marketing has changed radically in the past two decades 16. Top 22 NFT Podcasts For NFT Novices & Masterminds This Nifty 2022 We know you’re curious to learn more about NFTs. 2021 was a HUGE BOOM of nothing but ‘NFT this' and NFT that'- so which 22 podcasts should you listen to!? 17. Modern PR: Results Driven Brand Awareness & Demand Generation Modern PR: The Next Generation Of Brand Awareness & Demand Generation 18. How Has Traditional Media and Social Media Helped To Boost The Popularity Of Cryptocurrency In today’s digital world, both standard and social media channels are renowned for having a huge effect on our daily lives. From the ways we communicate and share content, to the ways we keep up to date with the world around us, social media has, and continues to have a disruptive effect on how we view the world. Cryptocurrencies have been no exception. 19. All of the Marvel Phase 5 TV Shows Announced Marvel's Phase 5 is packed to the brim with exciting projects. Everything from Secret Invasion to Loki season 2. 20. 6 Dark Web Myths Perpetuated by the Mainstream Media How do you know the dark web? Many people know this term from movies or news reports. 21. Responding to Online War Propaganda on LinkedIn: Who Do the Rules Protect? I am from Ukraine, and I’m trying to use my LinkedIn in order to spread the truth about the war that is going on right now. 22. Why The Queen’s Gambit Was the #1 Show in 12 Countries The Chess Show on Netflix | The Queen's Gambit 23. I am a Big Believer in Deep Work - Interview With Omri Hurwitz, Tech Marketer & Media Strategist An Interview with Omri Hurwitz, Tech Marketer & Media Strategist 24. Guide on How To Eliminate COVID-19 News According to the American press institute, the purpose of news and journalism is to provide citizens with the information they need to make the best possible decisions about their lives, their communities, their societies, and their governments. 25. How the Noonies is Like the Dundies [or at least I told my team that] Managing people, meh. Building teams, now that's fun. It is really interesting to think about how individuals can make others better. 26. The First Week of YourStack 2,281 days ago Product Hunt launched as a tiny newsletter. The early days were especially chaotic. I didn’t get much sleep, skipped the gym, and buried my head into work. While unsustainable (I'm not advocating overworking yourself!), it was sooo fun and energizing to build something brand new. It was also scary. New products – especially those driven by community – are incredibly fragile. 7 days ago we launched a beta of YourStack (with a grammatically incorrect, non-editable tweet – lol). Instantly those same nostalgic feelings from the early days of Product Hunt re-emerged. The response has been very encouraging. Alex was kind enough to write about it on BuzzFeed. Lucas followed with a story on TechCrunch. And a bunch of folks shared support on Twitter and Product Hunt. But the product is very early and our roadmap is deep. So far we've onboarded only ~4% of those that have signed up as we build tooling to scale (thanks for all your patience!). I wake up each morning nervous but excited to read your feedback and wild ideas as the team and I continue building. If you get a chance, create your account and start stacking. We're whitelisting new friendly folks each day. Feedback? I'd love to hear it at ryan@yourstack.com. Cheers,
Ryan Hoover (@rrhoover) [27. The Internet of Money: Lubricant for the Internet of Things](https://hackernoon.com/the-internet-of-money-lubricant-for-the-internet-of-things) The history, condition, and direction of communication, media, civilization, and the whole of humankind is not complete without discussing the essence and history of money. The history of human civilization can be explained by the constant struggle between two tendencies--centralization vs. decentralization—exemplified by what has unfolded in the development and evolution of communication media technologies. The same is true for money; or what we should more accurately refer to as currency. You might ask, what does money have to do with communication, marketing, and media? Everything. Money is what makes the world go ‘round.  Contrary to the popular song, money CAN buy you love and unfortunately, as we will soon see, money hasn’t exactly made the world go ‘round lately. 28. What the West Can Learn From Vietnam’s Response to Covid-19 Thank you so much for the outpouring support for my story. It's been viewed ~50k times and shared by hundreds. You can comment below with Hacker Noon's brand new commenting system, or join the conversation in various places around the internet: 29. The Two Most Important Things To Remember When Doing Modern PR 30. Why Home Media Servers Are Worth Your Time Files are getting larger and space for your favorite content can be at a premium. Getting your own server can make storing data so much easier. 31. Future of News: Top Journalism Trends and Insights for 2023 Did you know? Two out of three journalists say their work has been impacted by economic uncertainty. 32. Do Not Buy a Computer from Walmart, and Related Advice How do you feel about your ability to give advice? Depends on the topic. Do you have advice for me? Do not buy a computer from Walmart. 33. The News DAO? Form a News DAO against the usual news agencies where we can find out the truth about things, without censorship from the 'owners'. 34. Debunking the Top 5 PR Myths Getting press coverage for your startup can be tricky.  Avoid these 5 PR misconceptions. 35. Every DC Movie That Got Canceled in 2022 Here are all of the DC films that were canceled in 2022: Batgirl, Man of Steel 2, Wonder Woman 3, and a Black Adam sequel. 36. 5 Foolproof Ways to Communicate PR Metrics to Any Stakeholders A look at how PR and media coverage reporting can and must evolve. 37. 3 Daily Newsletters Worth Subscribing 📰 On any given day, you'll find tons of stories that are seemingly important, some moving, others hilarious/weird. 38. What's the Future of Media? Substack, Decentralized Media, and MultiTouchpoint Marketing On today’s episode of Startups On Demand, I am joined by Yoel Israel, CEO of WadiDigital, a results-driven and customer service-centered B2B digital marketing a 39. The CEO of My Company, Genius, Must Be Black When Tom and I started Genius in 2009, we wanted to build something that would grow bigger than rap lyrics. The first 4 songs I annotated on RapExegesis, as it was originally called, were rap songs. However, the 5th one was Elton John's "Rocket Man", the 6th was The Eagles' "Hotel California", and the 7th was a poem, the first non-lyrics post on the site - Emily Dickinson's "I'm Nobody!  Who Are You?" 40. Could COVID-19 Help Bring About The Media Revolution We Need? On friday 20th of March, I and many thousands of UK business owners, self-employed workers and unemployed workers eagerly tuned into the 5pm news, hoping to catch a vital update on whether the government would be offering us the essential aid that we need to survive the coming 3 months. What followed was an ‘unprecedented’ broadcast (I’m going to take a shot every time Rishi Sunak says this word from now on) promising support. 41. Stop Using Fake Quotations You Found on the Internet The Internet is full of people seeking and sharing useful information. Yet, news stories are subject to constant skepticism — the political views and motivations of legitimate news outlets are under constant scrutiny. 42. "They're Noncreative People Who Are Controlling Creative Things" Joe Rogan on Suits [2007 Interview] Joe Rogan and Tom Green joke about how big the internet could become over skype interviews with fans in 2007. 43. The Metaverse Insider Partners with MILC Platform to Revolutionise the Global Media Industry The Metaverse Insider is proud to announce a partnership with MILC as part of its evolving Metaverse ecosystem. 44. Learnings From Managing a News Publishing Platform There are many options when it comes to publishing a news product. Some publishers choose to collaborate with other publishers to develop their own platform, 45. Why Media and Entertainment Can’t Do without Mobile Apps There are over 2 million apps available for download both for Android and iOS platforms, and the average smartphone owner uses at least 30 of them each month. The most popular app categories are social media, messengers, games, business, education, and lifestyle. 46. How the Washington Post Uses TikTok, and What That Means for GenZ News Consumption GenZ-ers seem to like political and international news so journalists are now moving to TikTok. 47. Hallyu for Idiots: Understand the Korean Wave via Parasite and K-dramas Feature Picture: Parasite - Best movie of the decade, in my opinion. As well as 2019 Best Picture according to the Academy Awards. 48. Podcasts have Enabled Newer Insights into Sex Positivity Last year was a gamechanger for the entertainment industry. Why so? Because it was the year that podcasts hit their stride, with more than half of Americans having listened to a podcast and an estimated 32% listening monthly as of March of last year - the largest annual increase since podcast-related data started being tracked in 2008. 49. AI in Media: Can a Silicon-Brained Journo Beat a Flesh ’n Blood One? This year brought us two Nobel prize nominees in literature. Feels like there’s a “buy one get two” promotion going on, and that’s good! More awarded authors - better stories. 50. The Doe Discuss How Tech and Social Media Has Changed Journalism Social media over the last five years has become a growing and dominant source of information and news for hundreds of millions of individuals all over the world. Social media is always in your pocket on your smartphone. 51. Media Products vs The Rest: The High Leverage Phenomenon In Media Products Hulu purchased the rights of Seinfeld in 2015 for $150 Million. For the same deal after 4 years Netflix wrote a check of $400 Million. During these 4 years, neither Larry David nor Jerry Seinfeld (creators of Seinfeld) has done any new work for the show. 52. Why a Career in Motion Graphics Is a Good Option for Tech Savvy Students in 2023 Today, there’s no shortage of roles in the tech space for young aspiring professionals to pursue. From becoming a coder with a lucrative salary to an app developer with room to grow at a company, there are seemingly endless career options for students interested in technological processes. 53. Avatars Will Emerge as Storytellers Humans rule the world. 54. What's Next for Media - A Conversation With Sean McDonnell 55. Top Articles This Week In Tech: WN 14 We hope you did have an awesome week! Here is your dose of the best
articles that came up over the last week. Have fun reading them! 56. #FreeYakutia: Ukraine Plans to Help Yakutia to Become an Independent Country in 2022 This Slogging thread by Arthur Tkachenko occurred in slogging's official #random channel, and has been edited for readability. 57. Here We Go Again: 21 Saucy Predictions for 2021 21 Predictions for 2021 58. 20 Data Science Podcasts You Don’t Want to Miss Podcasts have unequivocally become one of the most dominant forms of media consumption in recent years. 59. Facebook: Brazen Censorship or a Tactical Solution to a Sophisticated Social Ask? We are entering a completely new supra-state level of regulation. This function that has suddenly gone to Facebook as a state has all the chances to contribute 60. The 59 Best Technology Journalism Stories Published in 2020 In a year dominated by pandemic, there was also a lot of notable technology news and analysis. 61. The Disintermediation of the Media: How Bad Journalism Came To Be And Why It Persists Multiple times a day, the President of the United States types a thought into his phone, my pocket buzzes, and I read his message. I don’t watch the news anymore. 62. My Experience With a Global News Organization and a Trillion-Dollar Corporation Julian Gough recounts their experience dealing with big organizations. 63. Stop Compromising Your UX for Display Ad Revenue: It's Not Worth It Banner ads can make website visitors uncomfortable, but they are a big source of revenue. This article covers how to balance your user's experience & your ads. 64. "Why Don't I Have a Computer?" David Letterman Asked Bill Gates in 1995 Bill Gates explains the origins of Microsoft and tries to convince Dave to get a computer. (Air Date 11/27/1995) 65. How AI and Machine Learning Are Infiltrating the Media Industry For the past few years now, the increasing digitalization of customer journeys and the exponential improvement of cloud technologies and computing capacities have invited media groups to rethink the way they do business. If it sounds like I’m using long words to say “digital disruption”, trust your instinct. Many of these disruptions have been centered around the mountains of data media groups have access to, and what Artificial Intelligence (AI) (and machine learning more specifically) could do with it. Indeed, while artificial intelligence has been fully embraced by a plethora of pure players (Spotify, Netflix, Buzzfeed, Disney…) traditional actors are still lagging, and now see the technology as a shortcut to a much-needed renewed growth. 66. From Wojcicki to Mohan: A Look at the Future of YouTube Under the New CEO In light of Susan Wojcicki's departure as the CEO of Youtube, what is the future of the company like? 67. Media Conglomerates Are Taking Over The World That's right, media conglomerates are taking over the world... or at least are trying to. This Week on Planet Internet, David Smooke, Amy Tom, and Ustav Jaiswal dive into the world of big media - think Facebook, Disney, Comcast, ATT - about the policies, who owns them, and more. Thank you for checking out the 67 most read stories about Media on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

67 Stories To Learn About Media

Since 2008, the Marvel Cinematic Universe (MCU) has been a Hollywood juggernaut. And even 15 years later, it still has millions of fans around the world. But many would agree that the peak of the MCU’s popularity is already in the past, particularly, it seemed to peak in the third phase. The MCU’s phase 3 was a time of extreme hype. There really was nothing quite like it, and it’s something that could most likely never be successfully replicated again. If you missed out on it, don’t worry. Here are all the MCU Phase 3 movies in chronological order. The MCU Phase 3 Movies in Order 1. Captain America: Civil War 2. Doctor Strange 3. Guardians of the Galaxy Vol. 2 4. Spider-Man: Homecoming 5. Thor: Ragnarok 6. Avengers: Infinity War 7. Ant-Man and the Wasp 8. Captain Marvel 9. Avengers: Endgame 10. Spider-Man: Far From Home 1. Captain America: Civil War (2016) The beginning of the phase started off with a bang because in Captain America: Civil War, trouble was brewing, and not just because of the supervillains. There was trouble brewing within the Avengers. The government and the public were fed up with the mistakes of the Avengers, and they wanted to hold them accountable by forcing them to work for the United Nations. Captain America and a few other heroes were extremely against this while Iron Man and his side were in favor. The two sides can’t see eye-to-eye and clash in epic fashion. 2. Doctor Strange (2016) After an accident leaves him with shaky hands, Doctor Stephen Strange’s days as a surgeon are over forever. However, what happens next no one could expect as Strange then begins his journey to be a sorcerer. First, he only does so for selfish reasons. But the more time he spends training, the more he realizes that he wasn’t meant to be a rich and selfish surgeon; he was meant to be the Sorcerer Supreme. In simpler terms, he was destined to be a superhero. 3. Guardians of the Galaxy Vol. 2 (2017) Jumping from the mystical side of the MCU to the cosmic one, everyone’s favorite rag-tag team of mischiefs returns for a second adventure. In the first movie, it’s heavily implied that Peter’s father is an important figure and somebody we’ll one day get to see. Well, in Vol. 2, we finally meet the mysterious man. And nothing is ever the same for the Guardians of the Galaxy. But beyond that, we get to meet the interesting Mantis, flesh out more of Gamora and Nebula’s relationship, and get to know another side of Rocket. 4. Spider-Man: Homecoming (2017) It was a long time coming, but it finally happened. One of the most popular superhero characters of all time finally got his own movie in the MCU. After making his debut in Captain America: Civil War, Spider-Man goes back to living a “regular” life as a teenager in Brooklyn. As regular as can be when you’re a superhero that is. But he quickly realizes that he won’t just be nabbing purse snatchers and petty thieves. He’ll have to deal with his own supervillain in the form of the Vulture. Even though he’s already gone toe-to-toe with some of the Avengers, this may be his biggest challenge yet. 5. Thor: Ragnarok (2017) Although this was the third movie for Thor, it felt like a fresh beginning. This version of Thor was more silly and comedic, and although future installments played too much into this, he was perfect in Thor: Ragnarok. In fact, everything about this movie is amazing. After their sister returns, Thor and his brother Loki find themselves in a strange new world that they can’t seem to escape of. That doesn’t matter though because they need to figure out a plan quickly before their sister destroys all of Asgard. Luck is on their side, however, as two allies are also stranded on the strange planet. Together, the 4 of them do everything they can to stop Hela. 6. Avengers: Infinity War (2018) Since his small teaser at the end of the first Avengers movie, Thanos had been built up as the final boss of the Avengers. A bigger threat than they could ever dream of. And in Infinity War, we find out if they’re up to the task. Still reeling from the prior events of Civil War, the Avengers (alongside the Guardians of the Galaxy) fight the Mad Titan before he can reclaim all of the Infinity Stones. Because it’s just not Earth that is being threatened but the entire universe. 7. Ant-Man and the Wasp (2018) While the Avengers are fighting for their lives, Ant-Man and the Wasp have their own mini (ha) adventure in San Francisco. The Wasp’s mom, Janet van Dyne, is stuck in the Quantum Realm after she shrunk too small. Decades later, Ant-Man and the Wasp believe they have the power and know-how to bring her back. Unfortunately, it won’t be an easy journey. Along the way, they’ll have to deal with both Sonny Burch and Ghost, 2 opponents who have their own agendas. 8. Captain Marvel (2019) Captain Marvel was released toward the tail-end of Phase 3, but it actually takes place before most of the other movies in the MCU. It follows a mysterious Kree soldier who suffers from memory loss. As more of her memories get uncovered, she has to ask herself, “Who can she trust?”  This question itself opens up doors and reveals secrets she never thought possible and allows her to become Captain Marvel. 9. Avengers: Endgame (2019) This was it. The culmination of the 21 movies that came before it. The Avengers once again have to come together and join forces to fight Thanos for a second and hopefully final time. But a lot has changed since they last encountered him, and many of the Avengers have changed (some for better and some for worse). None of that matters, however, because the stakes are higher than ever, and only the Avengers can make everything right. Although there was one more movie in Phase 3, Avengers: Endgame seemed to be a fitting conclusion to the saga. 10. Spider-Man: Far From Home (2019) Even though Peter Parker isn’t finished with high school, he’s had enough adventure and excitement for a lifetime. But the hits keep on coming for Spidey. On a school trip to Europe, he’ll come across other superpowered beings such as the Elementals and Mysterio who will turn his life upside down. And if that wasn’t enough, he still has to deal with being a teenager: getting bullied, having girl problems, and having a love-sick best friend. Conclusion Phase 3 of the MCU was one of its best, and this article should help you get on track to relive the journey or start it for the first time. Feature image source Also published here Read More: The Apollo Justice Trilogy: The 5 Best Cases The New Tomb Raider Games in Order Persona 3 Remake Officially Announced: Everything We Know So Far

The MCU Phase 3 Movies in Order

138 Stories To Learn About Startups Top Story

is an API Gateway, which builds upon the reverse-proxy to offer a plugin-based architecture. The main benefit of such an architecture is that it brings structure to the configuration of routes. It’s a help at scale, when managing hundreds or thousands of routes. Apache APISIX OpenResty In this post, I’d like to describe how plugins, priority, and phases play together and what pitfalls you must be aware of. APISIX plugin’s priority When you configure a route with multiple plugins, Apache APISIX needs to execute them in a order so that the results are the same over time. For this reason, every APISIX plugin has a . You can check a plugin priority directly in the code. For example, here’s the relevant code fragment for the plugin: consistent hardcoded priority basic-auth local _M = {
    version = 0.1,
    priority = 2520,                                 #1
    type = 'auth',
    name = plugin_name,
    schema = schema,
    consumer_schema = consumer_schema
} Priority is 2520 For documentation purposes, the file lists the priority of all out-of-the-box plugins. Here’s an excerpt: default-config.yaml plugins:
  - ldap-auth                      # priority: 2540
  - hmac-auth                      # priority: 2530
  - basic-auth                     # priority: 2520
  - jwt-auth                       # priority: 2510
  - key-auth                       # priority: 2500
  - consumer-restriction           # priority: 2400
  - forward-auth                   # priority: 2002
  - opa                            # priority: 2001
  - authz-keycloak                 # priority: 2000
  #- error-log-logger              # priority: 1091
  - proxy-cache                    # priority: 1085
  - body-transformer               # priority: 1080
  - proxy-mirror                   # priority: 1010
  - proxy-rewrite                  # priority: 1008
  - workflow                        # priority: 1006
  - api-breaker                    # priority: 1005
  - limit-conn                     # priority: 1003
  - limit-count                    # priority: 1002
  - limit-req                      # priority: 1001 Imagine a route configured with and . Because of their respective priority, would run before . proxy-mirror proxy-rewrite proxy-mirror proxy-rewrite upstream:
  - id: 1
    nodes:
      "oldapi:8081": 1

routes:
  - id: 1
    uri: "/v1/hello*"
    upstream_id: 1
    plugins:
      proxy-rewrite:                                 #1-3
        regex_uri: ["/v1(.*)", "$1"]
      proxy-mirror:                                  #2
        host: http://new.api:8082 The plugin ordering in this file is not relevant has priority 1010 and will execute first proxy-mirror has priority 1008 and will run second proxy-rewrite The above setup has an issue. For example, if we call , APISIX will first mirror the request, and then remove the prefix. Hence, the new API receives instead of . It’s possible to override the default priority to fix it: localhost:9080/v1/hello /v1 /v1/hello /hello routes:
  - id: 1
    uri: "/v1/hello*"
    upstream_id: 1
    plugins:
      proxy-rewrite:
        _meta:
          priority: 1020                             #1
        regex_uri: ["/v1(.*)", "$1"]
      proxy-mirror:
        host: http://new.api:8082 Override the default priority Now, has higher priority than : the former runs before the latter. proxy-rewrite proxy-mirror In this case, it works flawlessly; in others, it might not. Let’s dive further. APISIX phases APISIX runs plugins not only according to their priorities but also through dedicated phases. Because APISIX builds upon OpenResty, which builds upon ngxinx, the phases are very similar to the phases of these two components. nginx defines several an HTTP request goes through: phases NGX_HTTP_POST_READ_PHASE NGX_HTTP_SERVER_REWRITE_PHASE NGX_HTTP_FIND_CONFIG_PHASE NGX_HTTP_REWRITE_PHASE NGX_HTTP_POST_REWRITE_PHASE NGX_HTTP_PREACCESS_PHASE NGX_HTTP_ACCESS_PHASE NGX_HTTP_POST_ACCESS_PHASE NGX_HTTP_PRECONTENT_PHASE NGX_HTTP_CONTENT_PHASE NGX_HTTP_LOG_PHASE Each phase focuses on a task, , verifies that the client is authorized to make the request. i.e. NGX_HTTP_ACCESS_PHASE In turn, OpenResty offers similarly named phases. From nginx documentation: Finally, here are the phases of Apache APISIX: rewrite access before_proxy header_filter body_filter log We have seen two ways to order plugins: and Now comes the most important rule: ! by priority by phase. order by priority only works inside a phase For example, the plugin has a priority of 900 and runs in phase ; the plugin has a priority of 995 and runs in phase . Regardless of their respective priorities, will happen before , because happens before . Likewise, changing a priority won’t "move" a plugin out of its phase. redirect rewrite gzip body_filter redirect gzip rewrite body_filter The example above with and worked because both run in the phase. proxy-mirror proxy-rewrite rewrite The main issue is that priority is documented in the file, while the phase is buried in the code. Worse, some plugins run across different phases. For example, let’s check the proxy plugin and, more precisely, the functions defined there: config-default.yaml proxy-rewrite local function is_new_headers_conf(headers) local function check_set_headers(headers) function _M.check_schema(conf) function _M.rewrite(conf, ctx) The name of the function is suspiciously similar to one of the phases above. Looking at other plugins, we see the same pattern repeat. Apache APISIX runs plugin functions with the same name as the phase. rewrite() I took the liberty of summarizing all plugins and their respective phases in a table. +----------------------+---------+--------+-------------+--------------+------------+-----+
| Plugin               | Phase                                                            |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| General              | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| redirect             | X       |        |             |              |            |     |
| echo                 |         |        |             | X            | X          |     |
| gzip                 |         |        |             | X            |            |     |
| real-ip              | X       |        |             |              |            |     |
| ext-plugin-pre-req   | X       |        |             |              |            |     |
| ext-plugin-post-req  |         | X      |             |              |            |     |
| ext-plugin-post-resp |         |        | X           |              |            |     |
| workflow             |         | X      |             |              |            |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Transformation       | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| response-rewrite     |         |        |             | X            | X          |     |
| proxy-rewrite        | X       |        |             |              |            |     |
| grpc-transcode       |         | X      |             | X            | X          |     |
| grpc-web             |         | X      |             | X            | X          |     |
| fault-injection      | X       |        |             |              |            |     |
| mocking              |         | X      |             |              |            |     |
| degraphql            |         | X      |             |              |            |     |
| body-transformer     | X       |        |             | X            | X          |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Authentication       | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| key-auth             | X       |        |             |              |            |     |
| jwt-auth             | X       |        |             |              |            |     |
| basic-auth           | X       |        |             |              |            |     |
| authz-keycloak       |         | X      |             |              |            |     |
| authz-casdoor        |         | X      |             |              |            |     |
| wolf-rbac            | X       |        |             |              |            |     |
| openid-connect       | X       |        |             |              |            |     |
| cas-auth             |         | X      |             |              |            |     |
| hmac-auth            | X       |        |             |              |            |     |
| authz-casbin         | X       |        |             |              |            |     |
| ldap-auth            | X       |        |             |              |            |     |
| opa                  |         | X      |             |              |            |     |
| forward-auth         |         | X      |             |              |            |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Security             | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| cors                 | X       |        |             | X            |            |     |
| uri-blocker          | X       |        |             |              |            |     |
| ua-restriction       |         | X      |             |              |            |     |
| referer-restriction  |         | X      |             |              |            |     |
| consumer-restriction |         | X      |             |              |            |     |
| csrf                 |         | X      |             | X            |            |     |
| public-api           |         | X      |             |              |            |     |
| chaitin-waf          |         | X      |             |              |            |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Traffic              | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| limit-req            |         | X      |             |              |            |     |
| limit-conn           |         | X      |             |              |            | X   |
| limit-count          |         | X      |             |              |            |     |
| proxy-cache (init)   |         | X      |             | X            | X          |     |
| proxy-cache (disk)   |         | X      |             | X            |            |     |
| proxy-cache (memory) |         | X      |             | X            | X          |     |
| request-validation   | X       |        |             |              |            |     |
| proxy-mirror         | X       |        |             |              |            |     |
| api-breaker          |         | X      |             |              |            | X   |
| traffic-split        |         | X      |             |              |            |     |
| request-id           | X       |        |             | X            |            |     |
| proxy-control        | X       |        |             |              |            |     |
| client-control       | X       |        |             |              |            |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Observability        | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| zipkin               | X       | X      |             | X            |            | X   |
| skywalking           | X       |        |             |              |            | X   |
| opentelemetry        | X       |        |             |              |            | X   |
| prometheus           |         |        |             |              |            |     |
| node-status          |         |        |             |              |            |     |
| datadog              |         |        |             |              |            | X   |
| http-logger          |         |        |             |              | X          | X   |
| skywalking-logger    |         |        |             |              |            | X   |
| tcp-logger           |         |        |             |              |            | X   |
| kafka-logger         |         |        |             |              | X          | X   |
| rocketmq-logger      |         |        |             |              | X          | X   |
| udp-logger           |         |        |             |              |            | X   |
| clickhouse-logger    |         |        |             |              | X          | X   |
| syslog               |         |        |             |              |            | X   |
| log-rotate           |         |        |             |              |            |     |
| error-log-logger     |         |        |             |              |            |     |
| sls-logger           |         |        |             |              |            | X   |
| google-cloud-logging |         |        |             |              |            | X   |
| splunk-hec-logging   |         |        |             |              |            | X   |
| file-logger          |         |        |             |              | X          | X   |
| loggly               |         |        |             |              | X          | X   |
| elasticsearch-logger |         |        |             |              |            | X   |
| tencent-cloud-cls    |         | X      |             |              | X          | X   |
| loki-logger          |         |        |             |              | X          | X   |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| Others               | rewrite | access | beforeproxy | headerfilter | bodyfilter | log |
+----------------------+---------+--------+-------------+--------------+------------+-----+
| serverless           |         | X      |             |              |            |     |
| openwhisk            |         | X      |             |              |            |     |
| dubbo-proxy          |         | X      |             |              |            |     |
| kafka-proxy          |         | X      |             |              |            |     |
+----------------------+---------+--------+-------------+--------------+------------+-----+ Conclusion I’ve detailed Apache APISIX plugin phases and priorities in this post. I’ve explained their relationship with one another. Icing on the cake, I documented each out-of-the-box plugin’s phase(s). I hope it will prove helpful. To go further: Default plugin priorities Apache APISIX plugin execution lifecycle Plugin execution code Originally published at , on December 10th, 2023 A Java Geek

Navigating Apache APISIX: A Strategic Approach to Plugin Priorities and Execution Phases

Prometheus and Grafana are two big names in the open-source world of observability. Both are widely liked and used, with vibrant, opinionated communities, and they routinely build on top of each other. So, how do Prometheus and Grafan stack up against each other? In this blog, we'll compare them and examine - How their offerings overlap and differ How they perform against each other on a variety of criteria How they’re commonly used - together and separately, and why Introduction to Prometheus and Grafana Prometheus is a monitoring solution. An open-source project, it was started by SoundCloud in 2012 and has since gained immense popularity and traction. One reason for its widespread adoption is its seamless integration with Kubernetes. Prometheus is the de facto monitoring standard for a Kubernetes environment. Prometheus Prometheus offering At its core, Prometheus is a time-series DB that uses a pull mode to fetch metrics from instrumented jobs. With its multidimensional data model and flexible query language, Prometheus allows devs to easily get, store, and work with metrics data. : Prometheus discovers and scrapes metrics from predefined targets, typically service endpoints or infra components. Data Collection : Prometheus has a time-series DB that allows for highly efficient storage and querying of metrics data. Data Storage : PromQL (Prometheus Query Language) is used to retrieve and analyze metrics. It's a flexible query language allowing for precise slicing, dicing, and aggregation of data, ideal for deep performance analysis. Querying with PromQL : Prometheus comes with a built-in visualization interface, but it is basic and primarily intended for ad-hoc querying. For a richer, more robust visualization experience, Prometheus recommends using Grafana. Visualization Prometheus Expression browser: Image Source In contrast, the Grafana visualization of Prometheus data is much richer Image Source Grafana Grafana started as a visualization tool. However, over the years, Grafana has evolved into a full-stack observability platform. It not only helps users visualize data but also assists in collecting and aggregating it. Grafana can be used not just for metrics but also for other observability data (logs and traces). See the image below for the difference between Prometheus and Grafana offerings. In summary, the primary difference is that Prometheus is primarily a monitoring solution, while Grafana is a more comprehensive, full-stack solution that can be used across metrics, traces, and logs. Prometheus vs. Grafana: Detailed assessment Now that we understand what each of Prometheus and Grafana offers, let us compare them across the following criteria Core observability functions (Data collection, processing & storage) Scalability Querying Alerting Visualization (Visualization, UI/ UX, collaboration) Others (Documentation, ease of deployment, integrations, and pricing) Summary assessment Features Prometheus Grafana Breadth of solution ✓(Only metrics) ✓✓ ( across metrics, logs, traces) Data collection/ instrumentation ✓✓ (metrics) ✓✓ (also has logs/ traces; metrics agent similar to Prometheus) Data Storage ✓ (purpose-built for metrics;) ✓✓ (across metrics, logs, traces; metrics DB built on top of Prometheus) Scalability ✓ ✓✓(Mimir more scalable) Alerting ✓✓ (built-in AlertManager) ✓ (slightly less performant) Querying ✓✓ (PromQL) ✓✓ (Built on PromQL) Visualization & User Flows Visualization ✓ ✓✓ UI & UX ✓ ✓✓ Collaboration ✗ ✓✓ Other Documentation ✓✓ ✓✓ Easy Deployment ✓✓ ✓ Integration with other tools ✓✓ ✓✓ Free Plan ✓✓ (open-source) ✓✓ (open-source, plus paid cloud version) - Best-in-class ✓✓ - Good enough ✓ - Poor ✗ Detailed assessment Data collection/ instrumentation The main difference today is that Prometheus supports data collection for just metrics, while Grafana agent can be used for collection & forwarding of traces and logs as well. Note that for metrics data collection, Prometheus (Prom agent) in 2021, to make the solution more scalable. The Prom agent was inspired by the Grafana agent and mainly from it. introduced an agent mode takes the code related to metrics functionality In summary, the Grafana agent trumps for a few reasons - Allows you to collect & forward as well traces and logs You can send data to OTel systems as well (not just Prometheus-based ones) Allows more control over the agent’s components with Grafana’s rich UI debugging capabilities Prometheus agent is preferred in situations where teams are only focused on metrics data or are in the process of switching between standard Prometheus to prom agent. Data Storage Prometheus shines within metrics data storage with its efficient time-series database, optimized for the retention and querying of time-stamped metrics. Its unique storage model ensures that older data is compacted and can be efficiently queried over long periods. Grafana now has data storage back-ends across metrics, traces, and logs.  Loki for log aggregation and storage, Tempo for distributed traces, and  Mimir for metrics. For metrics itself, should you use Grafana Mimir or Prometheus? Note that Grafana Mimir builds  on Prometheus and many pieces of it have Prometheus code so there is some overlap :) In general, Prometheus is more widely used/ popular. That said, is a more modern metrics solution that addresses many of the challenges with Prometheus (like multi-tenancy, longer retention, and faster queries). See here for a Mimir more detailed comparison They’re also compatible with each other, so if you have a Prometheus agent, you could just set it to send data to a Mimir cluster so they’re . compatible with each other Scalability When it comes to scalability, Prometheus adopts a pull-based, single-tenant model which, while straightforward, poses challenges as systems grow. To handle vast amounts of data, Prometheus typically requires sharding and federation, adding some complexity. Grafana Mimir, on the other hand, is built for scalability and high performance. It has a distributed multi-tenant model that allows you to scale horizontally seamlessly, and a dedicated long-term storage solution, to store and process vast amounts of data. Grafana wins on scalability here. Querying Functional query language, PromQL, is both robust and expressive, allowing users to extract intricate details from their metrics. Alerts in Prometheus are defined using the same query language, ensuring precision. Grafana can leverage PromQL as well. In keeping with the theme of both companies building on top of each other, Grafana has also built its own . Prometheus query builder, which improves on PromQL Alerting Prometheus has a separate component called the Prometheus Alert Manager, that allows you to create and manage any alerts based on Prometheus data. It’s widely used, proven and well-liked. Historically, Grafana alerting was limited to data on the dashboards. However, with Grafana’s evolution into full-stack, Grafana alerting has become more comprehensive. Grafana Alerting now allows you to define alerts based on any Grafana data (Loki logs, Mimir, Tempo traces). The engine allows you to define alert criteria, evaluation frequency, time duration for evaluation, and composite criteria and also set notification policies like where and to whom the alerts are routed to. You could mute alerts for a while, or stop receiving notifications for a specific alert altogether. That said, Prometheus AlertManager still has an edge within metrics as it allows for more complex alerts with complex queries and calculations, with better performance. Grafana Alerting uses a SQL database so performance may not be great. Visualization For , Grafana is the star. Its dashboards are customizable, intuitive, and designed for a great user experience. Prometheus, on the other hand, has a basic visualization interface. It's functional but lacks the polish and flexibility Grafana offers. \ data visualization If rich visuals and dashboards are your focus, Grafana is the clear choice. Prometheus provides the data; Grafana makes it look good. UI & UX Diving into UI and UX, Grafana offers a sleek, user-friendly interface, making dashboard creation and navigation a breeze. In contrast, Prometheus focuses more on its core functionalities, with a UI that's straightforward but not as refined. For those prioritizing a smooth user experience and intuitive layout, Grafana has the edge. However, if you're looking purely for functionality and don't mind a steeper learning curve, Prometheus gets the job done. Collaboration and team management With built-in features like user roles, permissions, and team-centric dashboards, Grafana enables easy collaboration. Prometheus, on the other hand, leans heavily on its robust metrics collection, lacking advanced team features. If seamless team coordination is your goal, Grafana takes the cake. Documentation Both provide thorough resources. Prometheus distinctly carves a niche with detailed help on the metric collection, including best practices and common pitfalls. Grafana, on the other hand, hosts an extensive library of resources, spanning tutorials on dashboards, panels, and its expanding list of plugins. While Prometheus's documentation reads like a deep, technical manual, Grafana offers a blend of user guides, tutorials, and community-contributed content. Both projects are very well-documented and have vibrant communities. Deployment Prometheus is straightforward to deploy banking on its standalone nature with configurations primarily via YAML files. This minimalism makes its initial setup somewhat swift. Grafana, conversely, offers a lot of integrations, making it versatile but forcing a steeper initial learning curve. Though Prometheus speaks the language of simplicity, Grafana whispers promises of adaptability. As for teams preferring a plug-and-play approach, Grafana might demand a bit more patience, but its flexibility is worth the elbow grease. Integrations Prometheus, with its dedicated exporters, zeroes in on extracting metrics from various services, ensuring a tailored fit. It excels within metrics. Grafana, however, plays a broader game. Its vast array of plugins supports numerous data sources, helping in seamless integration. This is just a function of whether you’re looking for metrics alone, or also for other observability. Pricing Both projects are 100% open-source. Prometheus has an Apache v2.0 license, while Grafana has an . AGPL license Prometheus does not have a cloud version. However, several other players offer hosted Prometheus- e.g., Amazon-managed service for Prometheus, Google Cloud-managed service for Prometheus, and many other independent players. Grafana on the other hand offers its which is paid for. It’s a robust, tightly integrated offering that brings the best of the proven Grafana stack and makes it available as a hosted solution, own cloud version Better Together? As we saw above, Grafana and Prometheus build on each other a lot and are happy partners in the open-source observability ecosystem. The decision is often not really Prometheus vs. Grafana, but how to use Prometheus and Grafana together in the best way possible. Grafana and Prometheus in Practice: Typical Combinations & Configurations In real-world observability scenarios, the flexibility of Prometheus and Grafana allows for a range of configurations, each tailored to suit different requirements. Here's a quick dive into how these tools are commonly set up together for metrics: Grafana-Prometheus configurations in Monitoring Within monitoring, companies do Grafana-only, Prometheus-only, or. combination of the two (see image below). : This setup isn't just popular—it's a powerhouse. Prometheus, with its focused metric scraping, provides raw, granular data. Grafana takes this data and transforms it into actionable insights via its advanced visualization. It's not just about collecting metrics but understanding them, and this combination excels here. Prometheus metrics server + Grafana visualization : Increasingly popular. Teams adopting this are looking for cohesion—Grafana not just as a dashboard tool, but as an all-encompassing observability platform. Mimir + Grafana visualization : This combo is less common. It's typically adopted by teams with specific needs or those that are in the nascent stages of their observability journey. However, as organizations scale and demand more intricate visualizations, they often switch to Grafana for a broader palette of visualization tools. Prometheus server + Prometheus visualization Grafana-Prometheus configurations in overall Observability stack Prometheus for metrics alone, Grafana for the rest This is where teams use Prometheus for just metrics back-end and Grafana for traces, and logs, with an integrated visualization layer. This allows for a single-pane-of-glass experience, where the developer sees all observability data on the same dashboard. It's also one of the most commonly preferred configurations. Most teams already have Prometheus setup as their monitoring tool and are used to it, so tend to prefer this model. The native compatibility between Prometheus and Grafana visualization makes this a popular choice. Grafana stack for everything This is the full Grafana observability option, widely known as the LGTM stack (Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics). This is being adopted by much more modern teams who’re either setting up their observability anew, or refreshing their stack, and are looking for less expensive options vs. the commercial players. This offers a tightly integrated experience much like a Datadog or NewRelic, while having the advantages of being open-source and flexible. What’s Next? AI Once you have your basic observability set up, what next? Recent developments in AI are set to dramatically change how we implement observability. Even with a strong observability stack, developers still need to navigate large volumes of data to zero in on incident-specific data that they’re looking for. When a production incident occurs, these AI observability workspaces pull from across Prometheus, Grafana, and the rest of your observability stack, and generate AI inferences on the most probable root causes. This helps drastically reduce MTTR and also offers a unified incident-specific dashboard for troubleshooting. You can sign up for early access . incident-specific data here Summary We looked at a comprehensive assessment of Prometheus vs. Grafana — their offerings, where they overlap and how they differ, how they perform across different dimensions, and how they’re often used together. They’re both robust offerings within their own categories and liberally borrow from each other. Both have contributed significantly to advancing the open-source observability ecosystem. Also published here.

A Comprehensive Comparison of Prometheus and Grafana in 2023

The OpenTelemetry Collector sits at the center of the OpenTelemetry architecture but is unrelated to the W3C Trace Context. In my , I use Jaeger instead of the Collector. Yet, it's ubiquitous, as in every OpenTelemetry-related post. I wanted to explore it further. tracing demo In this post, I explore the different aspects of the Collector: The data kind: logs, metrics, and traces Push and pull models Operations: reads, transformations, and writes First steps A long time ago, as we know it didn't exist; what we had instead was . Back then, monitoring was a bunch of people looking at screens displaying dashboards. Dashboards themselves consisted of metrics and only system metrics: mainly CPU, memory, and disk usage. For this reason, we will start with metrics. observability monitoring is one of the primary monitoring solutions. It works on a pull-based model: Prometheus scrapes compatible endpoints of your application(s) and stores them internally. Prometheus We will use the OTEL Collector to scrape a Prometheus-compatible endpoint and print out the result in the console. Grafana Labs offers a that generates random metrics to play with. For simplicity's sake, I'll use Docker Compose; the setup looks like the following: project version: "3"

services:
  fake-metrics:
    build: ./fake-metrics-generator                                         #1
  collector:
    image: otel/opentelemetry-collector:0.87.0                              #2
    environment:                                                            #3
      - METRICS_HOST=fake-metrics
      - METRICS_PORT=5000
    volumes:
      - ./config/collector/config.yml:/etc/otelcol/config.yaml:ro           #4 No Docker image is available for the fake metrics project; hence, we need to build it Latest version of the OTEL Collector at the time of this writing Parameterize the following configuration file Everything happens here As I mentioned above, the OTEL Collector can do a lot. Hence, configuration is everything. receivers:                                                                  #1
  prometheus:                                                               #2
    config:
      scrape_configs:                                                       #3
        - job_name: fake-metrics                                            #4
          scrape_interval: 3s
          static_configs:
            - targets: [ "${env:METRICS_HOST}:${env:METRICS_PORT}" ]

exporters:                                                                  #5
  logging:                                                                  #6
    loglevel: debug

service:
  pipelines:                                                                #7
    metrics:                                                                #8
      receivers: [ "prometheus" ]                                           #9
      exporters: [ "logging" ]                                              #9 List of receivers. A receiver reads data; it can be either push-based or pull-based. We use the pre-defined receiver prometheus Define pull jobs Job's configuration List of exporters. In contrast to receivers, an exporter writes data. The simplest exporter is to write data on the standard out Pipelines assemble receivers and exporters Define a metric-related pipeline The pipeline gets data from the previously-defined receiver and sends it to the exporter, , prints them prometheus logging i.e. Here's a sample of the result: 2023-11-11 08:28:54 otel-collector-collector-1     | StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Timestamp: 2023-11-11 07:28:54.14 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Value: 83.090000
2023-11-11 08:28:54 otel-collector-collector-1     | NumberDataPoints #1
2023-11-11 08:28:54 otel-collector-collector-1     | Data point attributes:
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__embrace_world_class_systems: Str(concept)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__exploit_magnetic_applications: Str(concept)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__facilitate_wireless_architectures: Str(extranet)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__grow_magnetic_communities: Str(challenge)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__reinvent_revolutionary_applications: Str(support)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__strategize_strategic_initiatives: Str(internet_solution)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__target_customized_eyeballs: Str(concept)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__transform_turn_key_technologies: Str(framework)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__whiteboard_innovative_partnerships: Str(matrices)
2023-11-11 08:28:54 otel-collector-collector-1     | StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Timestamp: 2023-11-11 07:28:54.14 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Value: 53.090000
2023-11-11 08:28:54 otel-collector-collector-1     | NumberDataPoints #2
2023-11-11 08:28:54 otel-collector-collector-1     | Data point attributes:
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__expedite_distributed_partnerships: Str(approach)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__facilitate_wireless_architectures: Str(graphical_user_interface)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__grow_magnetic_communities: Str(policy)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__reinvent_revolutionary_applications: Str(algorithm)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__transform_turn_key_technologies: Str(framework)
2023-11-11 08:28:54 otel-collector-collector-1     | StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Timestamp: 2023-11-11 07:28:54.14 +0000 UTC
2023-11-11 08:28:54 otel-collector-collector-1     | Value: 16.440000
2023-11-11 08:28:54 otel-collector-collector-1     | NumberDataPoints #3
2023-11-11 08:28:54 otel-collector-collector-1     | Data point attributes:
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__exploit_magnetic_applications: Str(concept)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__grow_magnetic_communities: Str(graphical_user_interface)
2023-11-11 08:28:54 otel-collector-collector-1     |      -> fake__target_customized_eyeballs: Str(extranet) Beyond printing The above is an excellent first step, but there's more than printing to the console. We will expose the metrics to be scraped by a regular Prometheus instance; we can add a to visualize them. While it may seem pointless, bear with it, as it's only a stepstone. Grafana dashboard To achieve the above, we only change the OTEL Collector configuration: exporters:
  prometheus:                                                               #1
    endpoint: ":${env:PROMETHEUS_PORT}"                                     #2

service:
  pipelines:
    metrics:
      receivers: [ "prometheus" ]
      exporters: [ "prometheus" ]                                           #3 Add a exporter prometheus Expose a Prometheus-compliant endpoint Replace printing with exposing That's it. The OTEL Collector is very flexible. Note that the Collector is multi-input, multi-output. To both print data and expose them via the endpoint, we add them to the pipeline: exporters:
  prometheus:                                                               #1
    endpoint: ":${env:PROMETHEUS_PORT}"
  logging:                                                                  #2
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [ "prometheus" ]
      exporters: [ "prometheus", "logging" ]                                #3 Expose data Print data The pipeline will both print data and expose them With the Prometheus exporter configured, we can visualize metrics in Grafana. Note that receivers and exporters specify their type every one of them must be unique. To comply with the last requirement, we can append a qualifier to distinguish between them, , and and i.e. prometheus/foo prometheus/bar. Intermediary data processing A valid question would be why the OTEL Collector is set between the source and Prometheus, as it makes the overall design more fragile. At this stage, we can leverage the true power of the OTEL Collector: data processing. So far, we have ingested raw metrics, but the source format may not be adapted to how we want to visualize data. For example, in our setup, metrics come from our fake generator, "business", and the underlying NodeJS platform, "technical." It is reflected in the metrics' name. We could add a dedicated source label and remove the unnecessary prefix to filter more efficiently. You declare data processors in the section of the configuration file. The collector executes them in the order they are declared. Let's implement the above transformation. processors The first step toward our goal is to understand that the collector has two flavours: a "bare" one and a contrib one that builds upon it. Processors included in the former are limited, both in number and in capabilities; hence, we need to switch the contrib version. collector:
  image: otel/opentelemetry-collector-contrib:0.87.0                        #1
  environment:
    - METRICS_HOST=fake-metrics
    - METRICS_PORT=5000
    - PROMETHEUS_PORT=8889
  volumes:
    - ./config/collector/config.yml:/etc/otelcol-contrib/config.yaml:ro     #2 Use the flavour contrib For added fun, the configuration file is on another path At this point, we can add the processor itself: processors:
  metricstransform:                                                         #1
    transforms:                                                             #2
      - include: ^fake_(.*)$                                                #3
        match_type: regexp                                                  #3
        action: update
        operations:                                                         #4
          - action: add_label                                               #5
            new_label: origin
            new_value: fake
      - include: ^fake_(.*)$
        match_type: regexp
        action: update                                                      #6
        new_name: $${1}                                                     #6-7
# Do the same with metrics generated by NodeJS Invoke the metrics transform processor List of transforms applied in order Matches all metrics with the defined regexp List of operations applied in order Add the label Rename the metric by removing the regexp group prefix Fun stuff: syntax is $${x} Finally, we add the defined processor to the pipeline: service:
  pipelines:
    metrics:
      receivers: [ "prometheus" ]
      processors: [ "metricstransform" ]
      exporters: [ "prometheus" ] Here are the results: Connecting receivers and exporters A connector is both a receiver an exporter and connects two pipelines. The example from the documentation receives the number of spans (tracing) and exports the count, which has a metric. I tried to achieve the same with 500 errors - spoiler: it doesn't work as intended. and Let's first add a log receiver: receivers:
  filelog:
    include: [ "/var/logs/generated.log" ] Then, we add a connector: connectors:
  count:
    requests.errors:
      description: Number of 500 errors
      condition: [ "status == 500 " ] Lastly, we connect the log receiver and the metrics exporter: service:
   pipelines:
     logs:
       receivers: [ "filelog" ]
       exporters: [ "count" ]
     metrics:
       receivers: [ "prometheus", "count" ] The metric is named , but its value stays at 1. log_record_count_total Logs manipulation Processors allow data manipulation; operators are specialized processors that work on logs. If you're familiar with the Elasticsearch Logstash Kibana stack, they are the equivalent of Logstash. As of now, the log timestamp is the ingestion timestamp. We shall change it to the timestamp of its creation. receivers:
  filelog:
    include: [ "/var/logs/generated.log" ]
    operators:
      - type: json_parser                                                   #1
        timestamp:                                                          #2
          parse_from: attributes.datetime                                   #3
          layout: "%d/%b/%Y:%H:%M:%S %z"                                    #4
        severity:                                                           #2
          parse_from: attributes.status                                     #3
          mapping:                                                          #5
            error: 5xx                                                      #6
            warn: 4xx
            info: 3xx
            debug: 2xx
      - id: remove_body                                                     #7
        type: remove
        field: body
      - id: remove_datetime                                                 #7
        type: remove
        field: attributes.datetime
      - id: remove_status                                                   #7
        type: remove
        field: attributes.status The log is in JSON format; we can use the provided JSON parser Metadata attributes to set Fields to read from Parsing pattern Mapping table Accept a range, , . The operator has a special interpreted value (and similar) for HTTP statuses. e.g. 501-599 5xx Remove duplicated data Logs At this point, we can send the logs to any log aggregation component. We shall stay in the Grafana Labs sphere and use Loki. exporters:
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push" We can also use logs from the collector itself: service:
  telemetry:
    logs: Finally, let's add another pipeline: service:
  pipelines:
    logs:
      receivers: [ "filelog" ]
      exporters: [ "loki" ] Grafana can also visualize the logs. Choose Loki as a datasource. Conclusion In this post, we delved into the OpenTelemetry collector. While it's not a mandatory part of the OTEL architecture, it's a useful Swiss knife for all your data processing needs. In case you're not stuck to a specific stack or don't want to, it's a tremendous help. The complete source code for this post can be found on . GitHub To go further: OpenTelemetry Collector OpenTelemetry Operators Originally at on November 12th, 2023 published A Java Geek

Stories, News & Mentions about LOKI

Loki’s Stories on HackerNoon

Loki's News & Mentions