Resync

resync

Table of Links Abstract and I. Introduction II. Expectations of Blockchain Accessibility III. Approach I: Maintain a Ledger Locally - Run a Full Node IV. Approach II: Query a Third-Party Ledger-Node-As-A-Service (NAAS) V. Approach III: Light Node - External Query & Local Verification VI. Concluding Remark and References V. APPROACH III: LIGHT NODE - EXTERNAL QUERY & LOCAL VERIFICATION With the cost of running a full node continuously rising, the Ethereum community is seeking to develop a solution that can ensure data integrity and reduce system requirements. The result is the light node [23]: A light node is a server that only synchronizes limited information (such as block headers) with the blockchain network instead of block contents. When a light node encounters an RPC query, it pulls related contents from the blockchain network, verifies the contents with the synchronized information, and sends the result back to the client (see Fig. 4). Traditionally, a light node synchronizes all block headers with the network and pulls related blocks to answer the query when it receives one. However, the recent innovation of ultralight nodes, such as Flyclient [10], allows a light node to synchronize less data with the blockchain but still be able to verify data related to the query, provided that some node in the blockchain network is willing to compute a piece of proof for the correctness of the data. When the required data on the blockchain is very specific and limited, like when the users only care about one address that they own, specific light nodes, in the form of wallet software, are also available to the users. For zero-knowledgebased blockchains (such as Zcash [21] and Monero [14]), the wallet software can synchronize with the blockchains by downloading and analyzing blocks one by one to extract all transactions related to the address, without actually storing the blocks. These specific light nodes can be much more efficient for new blockchain users because they only care about blocks that come after the creation of the address, so they do not have to analyze any data before the creation time. There are also many studies on universal light nodes in the research area, but few have been deployed to real-world blockchains [12]. Some notable examples in active development include Flyclient [10] and IBC [30]. A light node is expected to ensure data integrity because it can check the block contents with the node’s synchronized headers. However, a recent study shows that the security guarantee is weakened when the adversary controls the network [44]. A. The Case of Ethereum We notice that support for light nodes is minimal as of 2022 for Ethereum, one of the most popular blockchains [24]: there is only one execution client and no production-ready light clients on the consensus layer. The proper functioning of light nodes depends on their ability to pull blocks from full nodes in the network, but there is no incentive for full nodes to provide such service, and as a result, light nodes often fail to find peers. Future development on light nodes focuses on allowing light nodes to relay information among themselves [22]: it is planned not to provide financial incentives for nodes, and the network is designed to work like BitTorrent and IPFS. B. Open Questions While significant engineering effort is still necessary before light clients become universal, there is still room for innovations to improve the client’s performance. A recent SoK paper has suggested privacy-preserving, efficient resynchronization after offline, and cross-chain interoperability as potential future research directions [12]. Based on the premise that proving some statement to be true is more difficult than proving it to be false, some recent study also focuses on fraud proof [1]. The idea is that the light node does not need to verify the information it receives to be true, but can simply ask a few full nodes to make sure that it is not false. We also notice that no matter the design of the light client, if it needs a full node to serve as a proofer for additional efforts beyond simply broadcasting a block, then incentives might be necessary to ensure that the full node renders the service. Toward that end, we think that the light node mechanism may merge with the node-as-a-service providers, with the providers serving the light nodes in a way that ensures integrity. In this sense, light nodes may evolve as a way to a verifiable RPC protocol. Authors:
(1) Zhongtang Luo, Purdue University (luo401@purdue.edu);
(2) Rohan Murukutla, Supra (r.murukutla@supraoracles.com);
(3) Aniket Kate, Purdue University / Supra (aniket@purdue.edu). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Table of Links Abstract and I. Introduction Abstract and I. Introduction II. Expectations of Blockchain Accessibility II. Expectations of Blockchain Accessibility III. Approach I: Maintain a Ledger Locally - Run a Full Node III. Approach I: Maintain a Ledger Locally - Run a Full Node IV. Approach II: Query a Third-Party Ledger-Node-As-A-Service (NAAS) IV. Approach II: Query a Third-Party Ledger-Node-As-A-Service (NAAS) V. Approach III: Light Node - External Query & Local Verification V. Approach III: Light Node - External Query & Local Verification VI. Concluding Remark and References VI. Concluding Remark and References V. APPROACH III: LIGHT NODE - EXTERNAL QUERY & LOCAL VERIFICATION With the cost of running a full node continuously rising, the Ethereum community is seeking to develop a solution that can ensure data integrity and reduce system requirements. The result is the light node [23]: A light node is a server that only synchronizes limited information (such as block headers) with the blockchain network instead of block contents. When a light node encounters an RPC query, it pulls related contents from the blockchain network, verifies the contents with the synchronized information, and sends the result back to the client (see Fig. 4). Traditionally, a light node synchronizes all block headers with the network and pulls related blocks to answer the query when it receives one. However, the recent innovation of ultralight nodes, such as Flyclient [10], allows a light node to synchronize less data with the blockchain but still be able to verify data related to the query, provided that some node in the blockchain network is willing to compute a piece of proof for the correctness of the data. When the required data on the blockchain is very specific and limited, like when the users only care about one address that they own, specific light nodes, in the form of wallet software, are also available to the users. For zero-knowledgebased blockchains (such as Zcash [21] and Monero [14]), the wallet software can synchronize with the blockchains by downloading and analyzing blocks one by one to extract all transactions related to the address, without actually storing the blocks. These specific light nodes can be much more efficient for new blockchain users because they only care about blocks that come after the creation of the address, so they do not have to analyze any data before the creation time. There are also many studies on universal light nodes in the research area, but few have been deployed to real-world blockchains [12]. Some notable examples in active development include Flyclient [10] and IBC [30]. A light node is expected to ensure data integrity because it can check the block contents with the node’s synchronized headers. However, a recent study shows that the security guarantee is weakened when the adversary controls the network [44]. A. The Case of Ethereum A. The Case of Ethereum We notice that support for light nodes is minimal as of 2022 for Ethereum, one of the most popular blockchains [24]: there is only one execution client and no production-ready light clients on the consensus layer. The proper functioning of light nodes depends on their ability to pull blocks from full nodes in the network, but there is no incentive for full nodes to provide such service, and as a result, light nodes often fail to find peers. Future development on light nodes focuses on allowing light nodes to relay information among themselves [22]: it is planned not to provide financial incentives for nodes, and the network is designed to work like BitTorrent and IPFS. B. Open Questions B. Open Questions While significant engineering effort is still necessary before light clients become universal, there is still room for innovations to improve the client’s performance. A recent SoK paper has suggested privacy-preserving, efficient resynchronization after offline, and cross-chain interoperability as potential future research directions [12]. Based on the premise that proving some statement to be true is more difficult than proving it to be false, some recent study also focuses on fraud proof [1]. The idea is that the light node does not need to verify the information it receives to be true, but can simply ask a few full nodes to make sure that it is not false. We also notice that no matter the design of the light client, if it needs a full node to serve as a proofer for additional efforts beyond simply broadcasting a block, then incentives might be necessary to ensure that the full node renders the service. Toward that end, we think that the light node mechanism may merge with the node-as-a-service providers, with the providers serving the light nodes in a way that ensures integrity. In this sense, light nodes may evolve as a way to a verifiable RPC protocol. Authors: (1) Zhongtang Luo, Purdue University (luo401@purdue.edu); (2) Rohan Murukutla, Supra (r.murukutla@supraoracles.com); (3) Aniket Kate, Purdue University / Supra (aniket@purdue.edu). Authors: Authors: (1) Zhongtang Luo, Purdue University (luo401@purdue.edu); (2) Rohan Murukutla, Supra (r.murukutla@supraoracles.com); (3) Aniket Kate, Purdue University / Supra (aniket@purdue.edu). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Light Nodes: Balancing Cost and Data Integrity in Blockchain Accessibility 

deployments come in all shapes and sizes. We support bare metal installs on any version of , containerized installs on any version of (including Red Hat OpenShift), and installs just about anywhere you can deploy a small lightweight single binary. But with flexibility comes the inevitability that edge case issues will require debugging. MinIO Linux Kubernetes In this blog post, we’ll show you how to debug a MinIO install running in Kubernetes and also some of the common issues you might encounter when doing bare metal installation and how to rectify them. Kubernetes Debugger Pod There are a few ways to access the MinIO API running inside a Kubernetes cluster. We can use or set up a listening on to be able to access the API. Both of these methods offer a way to access the service from outside the network, but they do come with one major downside: You can only access the Service that the NodePort or Port Forwarding references on an available port (not the usual configuration for the application). For example, you have to access the MinIO API, usually found on port , via a randomly assigned port. kubectl port-forwarding Service NodePort 9000 3xxxx What if I told you there was a better way – and it's not novel? When debugging applications you want to have full access to the native run-time environment so you can use various tools to troubleshoot and debug the cluster. One way to do that is launching a “busybox” style pod and installing all the required tools needed to debug the application. First launch a into the same namespace as your MinIO install. In order to do this create a yaml file called with the following yaml. Pod debugger-pod.yaml apiVersion: v1

kind: Pod

metadata:

  name: mc

  labels:

    app: mc

spec:

  containers:

  - image: minio/mc:latest

    command:

      - "sleep"

      - "604800"

    imagePullPolicy: IfNotPresent

    name: mc

  restartPolicy: Always The above Pod configuration is pulling the image for MinIO utility. In order to ensure the pod doesn’t just launch and then exit, we’ve added a command. mc sleep Once the yaml is saved apply the configuration to the Kubernetes namespace where the MinIO cluster is running kubectl apply -f debugger-pod.yaml Once the pod is up and running access it via shell $ kubectl exec -i -t -n default mc -c mc -- sh -c "(bash || ash || sh)"

[root@mc /]# Then with you can access the MinIO cluster mc [root@mc /]# mc alias set myminio    --insecure

Added `myminio` successfully. Now that we have a debugger pod up and running, you can perform action on the cluster directly within the same network. For example, if the replication was broken due to an site going offline or a hardware failure, you can resync any pending objects to be replicated using the following command [root@mc /]# mc admin replicate resync start minio1 minio2


[root@mc /]# mc admin replicate resync status minio1 minio2


✔ ✔ ✔

ResyncID: 		 2248d1d1-633f-4d61-b938-d8ea0b9b2d31

Status:   		 Completed

Objects:  		 2225

Versions: 		 2225

FailedObjects:     0

Throughput:   	 5.3 MiB/s

IOPs:     		 124.23 objs/s

Transferred:  	 94 MiB

Elapsed:  		 17.909833202s

CurrObjName:  	 testbucket/web-app/tsconfig.json Another reason you would run the debugger pod is if there are some file system permissions or invalid groups configurations in your pod, you can update them using the debugger pod [root@mc /]# chgrp -R 1000780050 .minio.sys/ The above debugging method can also be used in bare metal environments. For instance you can launch a busybox or bastion node with installed and follow the same instructions as above. mc Debugging Bare Metal Bare metal Linux installs are the most straightforward. In fact it just takes a few commands to get MinIO installed and running with SystemD. For details, please see . Configuring MinIO with SystemD Once in a great while, bare metal installs go awry. Here are some of the (not-so-common) pitfalls that we are asked about  in or . These pitfalls are not hardware or operating specific but can be useful to know in any kind of environment. SUBNET Slack File Permission One of the most common pitfalls is the file permissions of the MinIO binary and the configuration file. If this occurs, when you start MinIO using SystemD you will see and here is the full stack trace Assertion failed for MinIO. # systemctl status minio.service

● minio.service - MinIO

     Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: enabled)

     Active: inactive (dead)

     Assert: start assertion failed at Tue 2023-12-26 18:21:38 PST; 8s ago

             AssertFileIsExecutable=/usr/local/bin/minio was not met

       Docs: https://docs.min.io


Dec 26 18:13:37 minio1 systemd[1]: minio.service: Starting requested but asserts failed.

Dec 26 18:17:50 minio1 systemd[1]: Assertion failed for MinIO.

Dec 26 18:21:38 minio1 systemd[1]: minio.service: Starting requested but asserts failed.

Dec 26 18:21:38 minio1 systemd[1]: Assertion failed for MinIO. This could be caused by a number of reasons, let's go down the list and check for each of them. : The binary, in this example located at needs to have permission for user and group, respectively. MinIO Binary /usr/local/bin/minio root:root # ll /usr/local/bin/minio

total 93804

-rwxr-xr-x 1 root    root    96018432 Nov 15 16:35  minio* : The MinIO service needs to run under a unique Linux user and group for security purposes, never run as a user. By default we use for the user and group names. In the SystemD service config file you should see something like this MinIO Service User and Group root minio-user User=minio-user

Group=minio-user : The directory where MinIO data will be stored needs to be owned by or whichever user you decide to run the MinIO service as above. MinIO Data Dir minio-user:minio-user # ls -l /mnt

total 4

drwxrwxr-x 2 minio-user minio-user 4096 Dec 27 09:58 data : Both the config files should have permissions for user and group like so SystemD and MinIO config root:root # ls -l /etc/default/minio

-rw-r--r-- 1 root root 1330 Dec 27 09:52 /etc/default/minio


# ls -l /etc/systemd/system/minio.service

-rw-r--r-- 1 root root 941 Dec 26 17:13 /etc/systemd/system/minio.service : The entire install process should be run as . You can also try if your user has permissions but the recommendation is to run as root as the install needs to place files in a bunch of places that only the user can access. Your bash prompt should have a and not a like so Run as Root root sudo root # $ vs # $ If none of the above work, the best approach is to remove the app, directories and configs and start a fresh install as a root user. Port conflict Another common issue related to deleted files which still hold on to the process, which causes port conflicts. Even when a service is not running, you may be unable to start a new service on the existing port or the service that is running will misbehave (such as not allowing you to login). # lsof -n | grep (deleted)

COMMAND PID USER FD TYPE DEVICE   SIZE 	NODE NAME

nginx   	13423 root 5u  	REG 253,3   42949672960 	17 (deleted)

minio 	13423 minio 6u    REG 253,3 	0         	      18 (deleted)

minio       13423 minio 7u    REG 253,3 	0         	      19 (deleted) You might see errors such as those below on a MinIO install Login Failed net::ERR_FAILED 500 Internal Server Error 401 Unauthorized The screenshot above shows an internal server error and an unauthorized error. While looking at the surface it looks unclear what has caused this error, we can debug with a little linux knowledge what to look for that could cause this, let's take a gander. There are several ways to debug this issue, first lets check to see if multiple MinIO processes are running on the same node # ps aux | grep -i minio

minio-u+    5048  0.3  1.7 1594008 144384 ?      Ssl  11:03   0:01 /usr/local/bin/minio server --console-address :9001 /mnt/data/disk1/minio


minio-u+    9276  0.3  1.7 1594208 144301 ?      Ssl  11:25   0:01 /usr/local/bin/minio server --console-address :9001 /mnt/data/disk1/minio As we can see above there are 2 MinIO processes running. Start by killing the process that is older or has been running the longest, in this case it seems to be the process ID . 5048 kill -9 5048 Sometimes even after killing the process the service might still not start or might still get hung up because it has reserved a process number but not let it go. This can be caused by files that have been deleted but are still being tracked by the operating system. You can find the deleted files via LSOF lsof -n | grep '(deleted)' Last but not least, if there are no deleted files left over or hung processes and if everything looks absolutely clean, the last resort is to quickly restart the node. This is a no-nonsense method that  shuts down and clears any pending files and processes so you start a fresh install. SUBNET to the rescue Although rare, installation edge cases will always exist. MinIO customers know that they have nothing to worry about because they can quickly message our engineers – who have written the code – via the portal. We've seen almost everything under the sun, so while the issue might look cryptic or mind boggling at first glance, we'll put our years of expertise debugging installations in many varied environments to work and help you in a jiffy. SUBNET If you have any questions on troubleshooting and debugging installs be sure to reach out to us on ! MinIO Slack

Pro Tips for Debugging MinIO: Overcoming Common Issues in Kubernetes and Bare Metal Environments

MinIO ডিবাগ করার জন্য প্রো টিপস: কুবারনেটস এবং বেয়ার মেটাল পরিবেশে সাধারণ সমস্যাগুলি অতিক্রম করা

Profi-Tipps zum Debuggen von MinIO: Überwindung häufiger Probleme in Kubernetes- und Bare-Metal-Umgebungen

Consejos profesionales para depurar MinIO: superar problemas comunes en Kubernetes y entornos bare metal

Conseils de pro pour déboguer MinIO : surmonter les problèmes courants dans les environnements Kubernetes et Bare Metal

मिनियो डिबगिंग के लिए प्रो टिप्स: कुबेरनेट्स और बेयर मेटल वातावरण में आम समस्याओं पर काबू पाना

MinIO デバッグのプロのヒント: Kubernetes とベアメタル環境での一般的な問題の克服

MinIO 디버깅을 위한 전문가 팁: Kubernetes 및 베어메탈 환경의 일반적인 문제 극복

Dicas profissionais para depuração do MinIO: superando problemas comuns em ambientes Kubernetes e Bare Metal

Советы для профессионалов по отладке MinIO: решение распространенных проблем в Kubernetes и средах с голым железом

MinIO'da Hata Ayıklamaya Yönelik Profesyonel İpuçları: Kubernetes ve Bare Metal Ortamlarında Yaygın Sorunların Aşılması

Mẹo chuyên nghiệp để gỡ lỗi MinIO: Khắc phục các sự cố thường gặp trong môi trường Kubernetes và Bare Metal

调试 MinIO 的专业技巧：解决 Kubernetes 和裸机环境中的常见问题

Delivering Cloud Infrastructure the GitOps Way with Crossplane and ArgoCD

Grafana Loki is a log aggregation system that stores and queries logs from applications and infrastructure. Although commonplace, logs hold critical information about system operations and are a valuable source of debugging and troubleshooting information. Logs are frequently used to identify and track malicious activity, or simply to track user activity to provide business intelligence. In an earlier blog post, , we provided an overview of Loki’s components and their overall architecture. Think of Loki for logs as analogous to Prometheus for metrics. Loki is lightweight and cost-effective because it only indexes and queries metadata. Promtail agents collect, label and transform logs before sending them to Loki. Then Loki indexes metadata and groups entries into streams that are indexed with labels. Grafana is then used to visualize and query log information from Loki. As of version 2.0, Loki stores data in a single object storage backend, such as MinIO. Whereas the previous blog post explained the advantages of saving Loki data to MinIO, this blog post is a tutorial that teaches you how. Logging with Grafana Loki and MinIO First, we’ll teach you how to deploy Loki and MinIO using Docker containers, followed by instructions on how to install from source. Learn Loki and Master MinIO Using Docker Containers We built a demo using Docker containers, some scripting and a Git repository. We started with the Grafana Loki repository, and added configuration yaml and containers for MinIO and a script to create and expose a MinIO bucket for Loki data. The following steps will result in these five images running locally in Docker containers: Loki Promtail Grafana MinIO Create buckets The first three are obviously needed for Loki, Promtail and Grafana. They were already available from Grafana under the (there is a short ). MinIO is object storage for Loki, and the final container will run a script that creates buckets as Loki targets. Loki repository video explainer Loki has been configured to save log data to MinIO using . Of particular importance is the section: loki.yaml storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    resync_interval: 5s
    shared_store: s3
  aws:
    s3: http://minioadmin:minioadmin@minio.:9000/loki
    s3forcepathstyle: true Note the dot in the S3 address for MinIO. This is used because there is no need to specify AWS Region. We create a Docker environment using the file. We will configure containers, volumes, ports, networks and provide startup commands. docker-compose.yaml In order for the image to run as a container with access to Loki configuration via volumes: loki services:
  loki:
    image: grafana/loki:latest
    volumes:
      - <your-local-path>/loki/production:/home/loki/production
    ports:
      - "3100:3100"
    command: -config.file=/home/loki/production/loki.yaml
    networks:
      - loki The first configuration you’ll need to make is to make sure that your local folder of is being shared with the container. Please edit the section of your to refer to the local folder on the host system where you have downloaded the folder. We are mapping the host path to the container path in order to read . production volumes docker-compose.yaml production loki.yaml In , we also define a network and expose ports of the container running MinIO server. In this case, port 9000 and 9001 will be exposed to your local machine in order for you to access MinIO server with a browser. docker-compose.yaml Console UI:
  minio:
    image: minio/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      - loki
    command: server ~ --address ':9000' --console-address ':9001' We configure the network with all containers sharing a network. Each container can ping the others and use their APIs. Don’t use fixed IP addresses for your containers. To make it easier, we’ve configured and in our environment to resolve to the appropriate container. For example: loki loki minio http://minio:9000    for MinIO
http://loki:3100     for Loki
http://loki:3000     for Grafana The next action taken by is to run MinIO Client (mc) in a container to configure MinIO Server, create the destination bucket for Loki data and set the access policy to as required. docker-compose.yaml public createbuckets:
    image: minio/mc
    networks:
      - loki
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc config host add myminio http://minio:9000 minioadmin minioadmin;
      /usr/bin/mc rm -r --force myminio/loki;
      /usr/bin/mc mb myminio/loki;
      /usr/bin/mc policy set public myminio/loki;
      exit 0;
      " Promtail, a small Go program, is used to tail or collect the distributed log files and deliver them to Loki.  For this demo Promtail is deployed in its container with a configuration file that scrapes the /var/log directory.  No changes are required in order to run this demo and have it process the log files at that location - /var/log. At the end of this tutorial we will cover how to have Promtail scrape other locations on the system for log files. Steps Now that you understand how to configure Loki, Promtail and your Docker environment, please  follow these steps to configure and run the demo environment. The following steps use the , and the default Promtail configuration to demonstrate how Loki works with MinIO. After downloading our files, edit them for your environment. docker-compose.yaml loki.yaml Clone this repository : https://github.com/cniackz/loki git clone https://github.com/cniackz/loki.git minio-loki-tutorial Change directory to your local folder: production https://github.com/cniackz/loki/tree/main/production cd <your-local-path>/minio-loki-tutorial/production
Edit docker-compose.yaml to reference your local home directory
If necessary, edit promtail.yaml Build your local Docker environment: docker-compose build --no-cache Launch local Docker containers: docker-compose up -d To confirm that Loki data is being saved to MinIO, log into MinIO Console at or . This tutorial uses the following credentials: http://localhost:9001 http://minio:9001 user: minioadmin
password: minioadmin After logging in, click Buckets. You should see that the Loki bucket has been created. On the right side, click Browse to see the contents of the newly created bucket. You should see the directory under . This is where Loki will save data. fake loki Click its name to open the directory. Loki holds logs in memory for a configured interval and then writes them to object storage. The default is an interval of 5 minutes, so please wait 5 minutes for data to appear in your bucket. fake At this point, Promtail is sending logs to Loki and Loki is saving data to MinIO. Now, let’s set Grafana up to view Loki logs. Grafana 6.0 and higher include built-in support for Loki. Grafana 6.3 and higher includes support for functionality. LogQL Log into Grafana at (default credentials are admin:admin). http://loki:3000 In Grafana, click the cog icon on the left sidebar to go to Configuration and then Data Sources. Click the big Add Data Source button and then select Loki. Edit the http URL field to be for our Loki server running locally using Docker port mapping: . http://loki:3100 Then click Save & Test. To view logs immediately, click explore. You can also view logs by clicking Explore in the left sidebar. However you explore, then select the Loki datasource in the top-left dropdown, and then choose a log stream using the Log Labels button. You can type a LogQL query and click on Run Query. However, if you don’t yet know LogQL, you can use the GUI to select a log and query parameters. To quickly see the logs that are in Loki, click Log browser, then under and choose , then under click the name of your job (from ). In this case I’ve clicked on . Then click the Show Logs button. 1.  Select labels to search in job 2. Find values for the selected labels promtail-local-config.yaml varlogs You can select the time range and set the refresh interval of the query on the top right of the browser window. To see more details of the logs, scroll down and click on one of the log entries, it will provide additional information related to the log entry. A really powerful feature is the ability to filter or see statistics about and directly from log details by clicking on the icons. This simplifies troubleshooting as it makes it easier to look for recurring errors and pivot between search terms. labels fields Extending the Demo Once the demo is running, the next step is to have Promtail scrape a different set of files that might be more interesting for your use case. In order to do this we need Promtail to run using a configuration file that we can edit. One way to achieve this is using volumes in Docker. Edit the file to create 2 new volumes accessible from the Promtail container. The first provides access to the directory on the host system where the new config file will be created. The second provides access to a directory that will contain the log files of interest.  Additionally, Promtail will be launched referencing the new config file, which we have called : docker-compose.yaml promtail-local-config.yaml version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:latest
    volumes:
     - <your-local-path>/loki/minio-loki-tutorial/production:/home/loki/production
    ports:
      - "3100:3100"
    command: -config.file=/home/loki/production/loki.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.4.2
    volumes:
      - /var/log:/var/log
      -- <your-local-path>/loki/minio-loki-tutorial/production:/home/loki/production
      -- <your-local-path>/access_logs:/home/loki/access_logs
    command: -config.file=/home/loki/production/promtail-local-config.yaml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

  minio:
    image: minio/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      - loki
    command: server ~ --address ':9000' --console-address ':9001'

createbuckets:
    image: minio/mc
    networks:
      - loki
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc config host add myminio http://minio:9000 minioadmin minioadmin;
      /usr/bin/mc rm -r --force myminio/loki;
      /usr/bin/mc mb myminio/loki;
      /usr/bin/mc policy set public myminio/loki;
      exit 0;
      " There’s a lot more that you can do with log data in Grafana. For starters, you can install promtail in more places to tail more logs to Loki. Copy the executable and to other machines/instances/containers, edit the configuration as described in and run it. promtail-local-config.yaml Promtail Configuration We now need to create the Promtail config file, to send local system logs to Loki. Download and edit a from Grafana. The section to focus on is because this is where promtail is told which logs to pull, how to format them and where to send them. Please see for more information about configuring promtail. promtail-local-config.yaml sample config file scrape_configs Get logs into Loki The section includes the following: scrape_configs job_name - This differentiates the logs collected from other log groups. targets - Optional for . However, is often defined because in older versions of Promtail it was not optional. This was an artifact from directly using the Prometheus service discovery code, which required this entry. static_configs labels - Static label to apply to every log line scraped by this definition. Good examples include the environment name, job name, or app name. path - The path to where the logs that Loki is to consume are stored. server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: apache
  static_configs:
  - targets:
      - localhost
    labels:
      job: access_logs
      __path__: /home/loki/access_logs/*log It’s important to understand different configuration options for scraping in Promtail, and Grafana provides . plenty of details It may be helpful to consult the full to see the application’s full capabilities. promtail configuration reference Finally, place some Apache web server access logs in the directory specified in the file on the host system.  These are the logs that Promtail will ingest and send to Loki for processing: docker-compose.yaml -- <your-local-path>/access_logs:/home/loki/access_logs Once the edits have been made and the sample Apache access logs put in place, bring the containers down and back up with docker compose: docker-compose down
docker-compose up -d Promtail will subsequently load the new Apache access log files and make them available to Loki. The environment that we’ve created in this tutorial is helpful for getting started with Loki, but it is not production-ready. Next steps would be to leave Docker for Kubernetes and use distributed MinIO instead of a single instance. Eventually, Loki data gets big enough that it benefits from an external database for quick index searches. Please see for more information. Scalability - Scaling with Grafana Loki Conclusion The observability stack of Grafana, Prometheus and AlertManager gained a powerful addition with Loki (BTW, we also have a tutorial for ). Distributed systems, especially when containerized and orchestrated by Kubernetes, have many logs for their applications and microservices. Loki combined with MinIO is a cost-effective way to collect, store and query logs. Grafana, Prometheus and AlertManager on MinIO If you have any questions, please send us an email at , or join the and ask away. hello@min.io MinIO slack channel Also published . here

How To Deploy Grafana Loki and Save Data to MinIO Using Docker Containers or Direct from Source

Cómo implementar Grafana Loki y guardar datos en MinIO usando contenedores Docker o directamente desde la fuente

Comment déployer Grafana Loki et enregistrer des données sur MinIO à l'aide de conteneurs Docker ou directement depuis la source

ग्राफाना लोकी को कैसे तैनात करें और डॉकर कंटेनरों का उपयोग करके या स्रोत से सीधे डेटा को मिनियो में कैसे सहेजें

Grafana Loki をデプロイし、Docker コンテナーまたはソースから直接使用してデータを MinIO に保存する方法

Como implantar o Grafana Loki e salvar dados no MinIO usando contêineres do Docker ou direto da fonte

Cách triển khai Grafana Loki và lưu dữ liệu vào MinIO bằng Docker Container hoặc trực tiếp từ nguồn

如何使用 Docker 容器或直接从源代码部署 Grafana Loki 并将数据保存到 MinIO

Technical scalability is one of the main drivers of a data platform, as mentioned in the . But what are the options? In this blog we’ll evaluate the direction of scalability and the tradeoffs you might encounter in the decision process. SOFT methodology The main directions of technical scalability Vertical scalability Vertical scalability generally means “buy a bigger box”: what can’t be achieved with a server sized X, could be achieved with a server sized 2X. There’s plenty of data platforms that allow this type of scaling and, with all the options available now in the cloud market, the vertical scalability can take you loooooooong way. Pro: No changes in the architecture Easy estimation of new performance Cons: Resource constraints on a single machine The cost of a single big machine is greater than the cost of more smaller machines The load is parallelized only within the machine Horizontal scalability Horizontal scalability goes on the opposite direction, instead of having a bigger machine, we spread the workload across several smaller machines. It’s usually available in distributed systems where the workload is shared across nodes. Apache Kafka is a perfect example: its distributed nature allows you to plug in and out nodes from the cluster and have an automatic rebalancing of the data and responsibilities across nodes. Pro: Easy/Fast plug in of new resources Cost of more smaller machines is usually less of the cost of a single big machine The load is parallelized across the cluster Cons: Only available in distributed systems Change in the cluster architecture for every new node added Data is replicated Might lead to eventual consistency Diagonal scalability A mix of the two, is of course possible, in situations where we could both add nodes to a cluster, and at the same time, replace the existing with new nodes of enhanced capacity. Typically the scenario is similar to the Horizontal scalability, the only change is that the new nodes have different (usually increased) specs compared to the previous ones. Another way of scaling “diagonally” is to create different nodes for different purposes. Taking PostgreSQL as example, one might choose to scale vertically on the primary, but also scale horizontally to create one or more “read only replicas” that could fulfill all the read-only traffic, relieving the primary from part of the workload. Pro: Freedom of scaling in both directions Cons: Not always available For some of the technologies, the workload (e.g. write operations) can’t really be shared across all nodes Cross tech scalability Not all the workloads need to be performed in one single data technology: As the previous example with PostgreSQL read-only replicas, different nodes of a cluster can fulfil different workloads. Taking the example further, one could scale by storing the transactional data in PostgreSQL but then integrating ClickHouse for analytical workloads. This cross tech scalability allows to optimize each component for a single usage and scale independently. Pro: Workload-optimised data platforms Independent, workload-related, scaling Cons: Data synch between technologies Challenges of data platform scaling If data platform scaling was an easy task, we wouldn’t talk a lot about it. There are some challenges that the scaling operation brings with it. Where does the data reside? To achieve optimal performance and minimize any network latency, data platforms historically stored co-located data storage and compute. In recent times, a new trend of detaching compute from storage (see Snowflake, neon) allowed further scalability options to increase the disk and the CPU independently at the cost of (a small) additional latency. The tradeoff is between performance (immediate local access), workload isolation/independent scaling. Furthermore, if the data is co-located with the compute, it needs to be moved when performing, for example, a vertical scaling. Check this youtube video from Gwen Shapira to understand more Time to scale Depending on the architecture, the time to scale can vary between fractions of a second (in case only a new compute instance needs to be spun up) to hours (in case huge backups need to be restored). Evaluating the time to scale is necessary to understand the scaling thresholds: e.g. if one can scale in 10 seconds, the scaling option could be triggered at 95% capacity, if the scaling takes a few hours, the threshold needs to be lowered. Downtime Nobody wants downtime, specifically when needing to scale up to follow a rise in demand. However one must understand and evaluate: how much work can be done without changes in the original system how long is the switchover downtime (if any) how and how fast clients can re-sync connection parameters Answering the above queries will help understand how to minimize the unavailability of the system. Scaling-induced additional load The scaling operation might induce additional load on the system, e.g. when streaming changes from one node to the other. Evaluating how the additional load could impact system performances needs to be done upfront to ensure to provide a sufficient service during the scaling procedure. Not all workloads are equal As mentioned before, the type of scaling depends on the architectures and workloads: adding a read-only replica in a specific cloud region is a fairly low cost solution to alleviate some load from a primary PostgreSQL database. That said, the solution doesn’t work when a lot of write workloads are in the game. Backups/Restore What does it have to do “backup/restore” with scaling? Well, some of the technologies (hi PostgreSQL!) will initially restore a database copy from a backup and then start streaming the delta from the primary. Therefore, in such cases, having a ready to go, and up to date backup allows you to cut the time to scale. More data jumps more risks In some of the scenarios presented above, scaling is performed by adding new nodes to a cluster, or adding new technologies in the mix. Even if these options provide great benefit in the parallelism and the separation of workloads they are adding additional steps in the data journey. Every time there’s more than one server storing the data, there’s always the chance of not-in-sync replicas. Every time you add a new data hop, there are always additional complexities with network, integration, etc. Conclusion This article aims at covering a wide spectrum of scaling possibilities and related issues. When designing a data platform, the scalability problem (not only technical, check out the to know more) should be an important decision factor to evaluate a solution based on the underline technologies, the expected workloads and the business constraints. SOFT methodology Also published . here

Understanding the Directions of Technical Data Platforms Scalability

This tutorial demonstrates how to use C/C++ code in your KMM shared module. You'll learn how to embed a simple SHA-256 implementation on C-language in the KMM library and use it for iOS and Android target platforms. Kotlin Multiplatform Mobile (KMM) is an SDK designed to simplify the development of cross-platform mobile applications. You can share common code between iOS and Android apps and write platform-specific code only where it's necessary You can use C and C++ code both with Android and iOS . KMM utilizes this functionality and allows you to use C and C++ code in your shared module. (Xcode supports C, C++, and out-of-the-box, and Android has ) Android NDK It can be useful in app development; there are a lot of libraries that have already been written in C/C++. A great example of this technique is the which embeds the engine in the KMM code. Zipline project QuickJs In this article, we won’t talk about how it works under the hood and will instead concentrate on writing the code and setting up the Gradle build. The code for this article could be found in the repository Start Building a Library for iOS and Android We won’t talk about how to create a KMM library as it is very well explained in the official Kotlin tutorial. Let’s take the SHA-256 code written on C as an example, and copy it into our folder. The sample code for the SHA-256 implementation was taken from this . native/sha256 repository Because Android and iOS applications support C-code differently, we need to use the . expected/actual approach First of all, let’s set up our common part: /**
 * Encoder interface which can encode byte arrays to Sha256 format.
 */
interface Sha256Encoder {
    fun encode(src: ByteArray): ByteArray

    fun encodeToString(src: ByteArray): String {
        val encoded = encode(src)
        return buildString(encoded.size) {
            encoded.forEach { append(it.toUByte().toString(16).padStart(2, '0')) }
        }
    }
}

expect object Sha256Factory {
    /**
     * Creates a new instance of [Sha256Encoder]
     */
    fun createEncoder(): Sha256Encoder
} Then, continue to platform-specific implementations. Embed C-code for iOS Target Platforms For iOS target platforms, we will set up the Gradle plugin, which will build our C-code as LLVM bitcode. C Klib C Klib is an experimental library with limited support from the authors. All we need to do is provide a path to the source folder with C-code; in our example, it will be : native/sha256 cklib {
    config.kotlinVersion = libs.versions.kotlin.get()
    create("sha256") {
        language = C
        srcDirs = project.files(file("native/sha256"))
    }
} Then we need to create a file for the KMM tool which generates Kotlin bindings from a C-header file. .def cinterop To learn more about , you could read .def the official cinterop tutorial In our case, we just need to create our in the folder (default searching place for tool) and name package where Kotlin bindings will be: sha256.def nativeInterop/cinterop cinterop package = com.ttypic.clibs.sha256 Finally, we need to add the cinterop step for our target platforms and specify the header file: kotlin {
    // ...
    listOf(
        iosX64(),
        iosArm64(),
        iosSimulatorArm64()
    ).forEach {
        it.compilations {
            val main by getting {
                cinterops {
                    create("sha256") {
                        header(file("native/sha256/sha256.h"))
                    }
                }
            }
        }
    }
    // ...
} After resyncing our Gradle build, we will see Kotlin bindings in the package, and we are ready to invoke native code. Our implementation will look like this: com.ttypic.clibs.sha256 actual actual object Sha256Factory {
    actual fun createEncoder(): Sha256Encoder = NativeSha256Encoder
}

object NativeSha256Encoder : Sha256Encoder {
    @OptIn(ExperimentalUnsignedTypes::class)
    override fun encode(src: ByteArray): ByteArray =
        memScoped {
            val ctx = alloc<SHA256_CTX>()
            sha256_init(ctx.ptr)
            val srcPointer = src.toUByteArray().toCValues().ptr
            sha256_update(ctx.ptr, srcPointer, src.size.toULong())
            UByteArray(32).apply {
                usePinned {
                    sha256_final(ctx.ptr, it.addressOf(0))
                }
            }.toByteArray()
        }
} That’s it; now we can run tests to check if everything is OK: class NativeSha256Test {

    @Test
    fun `should pass library sha256 checks`() {
        checkEncodeToString("Kotlin", "c78f6c97923e81a2f04f09c5e87b69e085c1e47066a1136b5f590bfde696e2eb")
        checkEncodeToString("abc", "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad")
        checkEncodeToString("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq", "248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1")
        checkEncodeToString("aaaaaaaaaa", "bf2cb58a68f684d95a3b78ef8f661c9a4e5b09e82cc8f9cc88cce90528caeb27")
    }

    private fun checkEncodeToString(input: String, expectedOutput: String) {
        assertEquals(expectedOutput, Sha256Factory.createEncoder().encodeToString(input.asciiToByteArray()))
    }

}

private fun String.asciiToByteArray() = ByteArray(length) {
    get(it).code.toByte()
} Embed C-code for Android Target Platforms For Android, we will be using . Android NDK uses under the hood; that’s why we need to provide the build file: Android NDK CMake CMakeList.txt cmake_minimum_required(VERSION 3.4.1)
# setup C standard we used
set(CMAKE_C_STANDARD 99)
# source code file mask
file(GLOB_RECURSE sources "../../native/*.c")
# build sources as dynamic library 
add_library(sha256 SHARED ${sources})
# link library
target_link_libraries(sha256) Then we have to specify this file in the Android plugin and setup the target platforms: android {
    // ...
    defaultConfig {
        // ...
        ndk {
            // target platforms
            abiFilters += listOf("x86", "x86_64", "armeabi-v7a", "arm64-v8a")
        }
    }
    // ...
    externalNativeBuild {
        cmake {
            path = file("src/androidMain/CMakeLists.txt")
        }
    }
} To invoke our native code, we need to use to map native code to JVM. Android Studio has great JNI support. Let’s create file, then include JNI and our library header files: JNI sha256-jni.c #include "jni.h"
#include "sha256/sha256.h" Now, we are ready to write our code: actual actual object Sha256Factory {
    actual fun createEncoder(): Sha256Encoder = AndroidSha256Encoder
}

object AndroidSha256Encoder : Sha256Encoder {
    init {
        System.loadLibrary("sha256")
    }
    external override fun encode(src: ByteArray): ByteArray
} It is very straightforward. First, we load our dynamic library, and next, we use to invoke native code. Now, all we need is to write JNI-bindings. You can do it yourself using the fully qualified class name or invoke the Android Studio context menu and choose JNI create JNI funtion JNI function in sha256-jni.c JNIEXPORT jbyteArray JNICALL Java_com_ttypic_clibs_AndroidSha256Encoder_encode(JNIEnv *env, jclass _, jbyteArray src) {
    BYTE hash[SHA256_BLOCK_SIZE];
    SHA256_CTX ctx;

    size_t len = (size_t) ((*env)->GetArrayLength(env, src));
    jboolean copied;
    jbyte* bytes = (*env)->GetByteArrayElements(env, src, &copied);

    sha256_init(&ctx);
    sha256_update(&ctx, (const BYTE *) bytes, len);
    sha256_final(&ctx, hash);

    (*env)->ReleaseByteArrayElements(env, src, bytes, JNI_ABORT);

    jbyteArray result = (*env)->NewByteArray(env, SHA256_BLOCK_SIZE);
    (*env)->SetByteArrayRegion(env, result, 0, SHA256_BLOCK_SIZE, (const jbyte *) hash);
    return result;
} And finally, we are ready to write tests: We need to write the instrumental test on the simulator; regular Unit-test won’t load our native library. class AndroidSha256Test {

    @Test
    fun should_pass_library_sha256_checks() {
        checkEncodeToString("Kotlin", "c78f6c97923e81a2f04f09c5e87b69e085c1e47066a1136b5f590bfde696e2eb")
        checkEncodeToString("abc", "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad")
        checkEncodeToString("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq", "248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1")
        checkEncodeToString("aaaaaaaaaa", "bf2cb58a68f684d95a3b78ef8f661c9a4e5b09e82cc8f9cc88cce90528caeb27")
    }

    private fun checkEncodeToString(input: String, expectedOutput: String) {
        assertEquals(
            expectedOutput,
            Sha256Factory.createEncoder().encodeToString(input.asciiToByteArray())
        )
    }
}

private fun String.asciiToByteArray() = ByteArray(length) {
    get(it).code.toByte()
} We successfully embedded C-code in our Kotlin Multiplatform Library, which can be used in iOS and Android target platforms. Useful Links Github repository with final code Get started KMM Official KMM library tutorial Zipline project Native dependencies in KMM

How to Extend a KMM Shared Module With C/C++ Code

This is our final post in our three-part series demonstrating how to use the Salesforce Mobile SDK to build an Android app that works with the Salesforce platform. , we showed you how to connect to your org. showed you how to edit and add data to your org from your app. This post will show you how to synchronize data from your Salesforce org to your mobile device and handle scenarios such as network loss. Let’s get right into it! In our first post Our second post Working with Mobile Sync One of the hardest aspects of mobile development is dealing with data synchronization. How do you handle the situation when you need to add a new broker, but you’re offline? Or what if two agents are updating the same broker—how can you handle the merging of those two changes? With the Salesforce Mobile SDK, these real-world issues are handled for you by a system called . Mobile Sync has you map your phone’s local data to the data in Salesforce; it also requires you to define operations for fetching and pushing data—what it calls and . Mobile Sync syncDown syncUp Define the shape of data to be synced To get started with Mobile Sync, create a file in called : res/raw brokerstore.json {
 "soups": [
   {
     "soupName": "brokers",
     "indexes": [
       { "path": "Id", "type": "string"},
       { "path": "Name", "type": "string"},
       { "path": "Title__c", "type": "string"},
       { "path": "Phone__c", "type": "string"},
       { "path": "Mobile_Phone__c", "type": "string"},
       { "path": "Email__c", "type": "string"},
       { "path": "Picture__c", "type": "string"},
       { "path": "__local__", "type": "string"},
       { "path": "__locally_created__", "type": "string"},
       { "path": "__locally_updated__", "type": "string"},
       { "path": "__locally_deleted__", "type": "string"},
       { "path": "__sync_id__", "type": "integer"}
     ]
   }
 ]
} This file defines the shape of the data on your phone, as well as some additional metadata that’s necessary for the sync. Next, create a file called : brokersync.json {
 "syncs": [
   {
     "syncName": "syncDownBrokers",
     "syncType": "syncDown",
     "soupName": "brokers",
     "target": {"type":"soql", "query":"SELECT Name, Title__c, Phone__c, Mobile_Phone__c, Email__c, Picture__c FROM Broker__c LIMIT 10000"},
     "options": {"mergeMode":"OVERWRITE"}
   },
   {
     "syncName": "syncUpBrokers",
     "syncType": "syncUp",
     "soupName": "brokers",
     "target": {"createFieldlist":["Name", "Title__c", "Phone__c", "Mobile_Phone__c", "Email__c", "Picture__c"]},
     "options": {"fieldlist":["Id", "Name", "Title__c", "Phone__c", "Mobile_Phone__c", "Email__c", "Picture__c"], "mergeMode":"LEAVE_IF_CHANGED"}
   }
 ]
} These are the operations that Mobile Sync will use when syncing data down and up. The code to complete the Mobile Sync process depends on several factors, such as when you want to perform a synchronization, as well as hooking into the Android event cycle when a device loses (and regains) connectivity. The following code samples will show you a complete example of what needs to happen to get synchronization working, but they should be considered high-level concepts, and not necessarily production-ready enterprise-grade code. Set up periodic sync With that said, let’s take a look at how to implement synchronization. First, add this line to the end of our method in : onResume(client: RestClient) MainActivity.kt setupPeriodicSync(); Next, we’ll add a new variable and a new function to the class: MainActivity private val SYNC_CONTENT_AUTHORITY =
   "com.salesforce.samples.mobilesyncexplorer.sync.brokersyncadapter"

private fun setupPeriodicSync() {
   val account = MobileSyncSDKManager.getInstance().userAccountManager.currentAccount

   ContentResolver.setSyncAutomatically(account, SYNC_CONTENT_AUTHORITY, true)
   ContentResolver.addPeriodicSync(
       account, SYNC_CONTENT_AUTHORITY,
       Bundle.EMPTY, 10
   )
} Since we’re using in our function, let’s make sure to import it by adding this line alongside the other import statements near the top of : ContentResolver MainActivity.kt import.android.content.ContentResolver We have defined two methods that trigger synchronization. will run a sync every seconds. This is much too frequent for a production environment, but we’ll set it this way for demonstration purposes. setupPeriodicSync 10 Mapping sync operations to our data and UI We will show the next few code samples all at once, and discuss what they’re doing afterward. In , create a new file called and paste these lines into it: app/java/com.example.sfdc BrokerSyncAdapter.kt package com.example.sfdc

import android.accounts.Account
import android.content.AbstractThreadedSyncAdapter
import android.content.ContentProviderClient
import android.content.Context
import android.content.SyncResult
import android.os.Bundle
import com.salesforce.androidsdk.accounts.UserAccount
import com.salesforce.androidsdk.accounts.UserAccountManager
import com.salesforce.androidsdk.app.SalesforceSDKManager
import com.example.sfdc.BrokerListLoader

class BrokerSyncAdapter
   (
   context: Context?, autoInitialize: Boolean,
   allowParallelSyncs: Boolean
) :
   AbstractThreadedSyncAdapter(context, autoInitialize, allowParallelSyncs) {
   override fun onPerformSync(
       account: Account, extras: Bundle, authority: String,
       provider: ContentProviderClient, syncResult: SyncResult
   ) {
       val syncDownOnly = extras.getBoolean(SYNC_DOWN_ONLY, false)
       val sdkManager = SalesforceSDKManager.getInstance()
       val accManager = sdkManager.userAccountManager
       if (sdkManager.isLoggingOut || accManager.authenticatedUsers == null) {
           return
       }
       if (account != null) {
           val user = sdkManager.userAccountManager.buildUserAccount(account)
           val contactLoader = BrokerListLoader(context, user)
           if (syncDownOnly) {
               contactLoader.syncDown()
           } else {
               contactLoader.syncUp() // does a sync up followed by a sync down
           }
       }
   }

   companion object {
       // Key for extras bundle
       const val SYNC_DOWN_ONLY = "syncDownOnly"
   }
} Now, in that same folder, create with these lines: BrokerListLoader.kt package com.example.sfdc

import android.content.AsyncTaskLoader
import android.content.Context
import android.content.Intent
import android.util.Log
import com.salesforce.androidsdk.accounts.UserAccount
import com.salesforce.androidsdk.app.SalesforceSDKManager
import com.salesforce.androidsdk.mobilesync.app.MobileSyncSDKManager
import com.salesforce.androidsdk.mobilesync.manager.SyncManager
import com.salesforce.androidsdk.mobilesync.manager.SyncManager.MobileSyncException
import com.salesforce.androidsdk.mobilesync.manager.SyncManager.SyncUpdateCallback
import com.salesforce.androidsdk.mobilesync.util.SyncState
import com.salesforce.androidsdk.smartstore.store.QuerySpec
import com.salesforce.androidsdk.smartstore.store.SmartSqlHelper.SmartSqlException
import com.salesforce.androidsdk.smartstore.store.SmartStore
import org.json.JSONArray
import org.json.JSONException
import java.util.ArrayList

class BrokerListLoader(context: Context?, account: UserAccount?) :
   AsyncTaskLoader<List<String>?>(context) {
   private val smartStore: SmartStore
   private val syncMgr: SyncManager
   override fun loadInBackground(): List<String>? {
       if (!smartStore.hasSoup(BROKER_SOUP)) {
           return null
       }
       val querySpec = QuerySpec.buildAllQuerySpec(
           BROKER_SOUP,
           "Name", QuerySpec.Order.ascending, LIMIT
       )
       val results: JSONArray
       val brokers: MutableList<String> = ArrayList<String>()
       try {
           results = smartStore.query(querySpec, 0)
           for (i in 0 until results.length()) {
               brokers.add(results.getJSONObject(i).getString("Name"))
           }
       } catch (e: JSONException) {
           Log.e(TAG, "JSONException occurred while parsing", e)
       } catch (e: SmartSqlException) {
           Log.e(TAG, "SmartSqlException occurred while fetching data", e)
       }
       return brokers
   }

   @Synchronized
   fun syncUp() {
       try {
           syncMgr.reSync(
               SYNC_UP_NAME
           ) { sync ->
               if (SyncState.Status.DONE == sync.status) {
                   syncDown()
               }
           }
       } catch (e: JSONException) {
           Log.e(TAG, "JSONException occurred while parsing", e)
       } catch (e: MobileSyncException) {
           Log.e(TAG, "MobileSyncException occurred while attempting to sync up", e)
       }
   }

   /**
    * Pulls the latest records from the server.
    */
   @Synchronized
   fun syncDown() {
       try {
           syncMgr.reSync(
               SYNC_DOWN_NAME
           ) { sync ->
               if (SyncState.Status.DONE == sync.status) {
                   fireLoadCompleteIntent()
               }
           }
       } catch (e: JSONException) {
           Log.e(TAG, "JSONException occurred while parsing", e)
       } catch (e: MobileSyncException) {
           Log.e(TAG, "MobileSyncException occurred while attempting to sync down", e)
       }
   }

   private fun fireLoadCompleteIntent() {
       val intent = Intent(LOAD_COMPLETE_INTENT_ACTION)
       SalesforceSDKManager.getInstance().appContext.sendBroadcast(intent)
   }

   companion object {
       const val BROKER_SOUP = "brokers"
       const val LOAD_COMPLETE_INTENT_ACTION =
           "com.salesforce.samples.mobilesyncexplorer.loaders.LIST_LOAD_COMPLETE"
       private const val TAG = "BrokerListLoader"
       private const val SYNC_DOWN_NAME = "syncDownBrokers"
       private const val SYNC_UP_NAME = "syncUpBrokers"
       private const val LIMIT = 10000
   }

   init {
       val sdkManager = MobileSyncSDKManager.getInstance()
       smartStore = sdkManager.getSmartStore(account)
       syncMgr = SyncManager.getInstance(account)
       // Setup schema if needed
       sdkManager.setupUserStoreFromDefaultConfig()
       // Setup syncs if needed
       sdkManager.setupUserSyncsFromDefaultConfig()
   }
} What did we just do? Each file has a specific role, and while the objects and fields will certainly be different for your app, the functions and philosophies of these classes are the same: is responsible for mapping the sync operations you defined in with the Kotlin code that will actually perform the work. Notice that there are and methods that use the Mobile SDK’s to load the JSON files and communicate back and forth with Salesforce. BrokerListLoader brokersync.json syncUp syncDown SyncManager can perhaps best be thought of as the code that’s responsible for scheduling the synchronization. That is, it’s the entry point for , and can be called by UI elements (such as during a refresh button click) or Android system events (such as connectivity loss). BrokerSyncAdapter BrokerListLoader Lastly, we need to add one line to our file (in ). Our new sync functionality will need special Android permissions, which we ask the user to allow upon app installation at runtime. Add the following line with to the end of your manifest: AndroidManifest.xml app/manifests WRITE_SYNC_SETTINGS <uses-permission android:name="com.example.sfdc.C2D_MESSAGE" />
  <uses-permission android:name="android.permission.WRITE_SYNC_SETTINGS" />
</manifest> Testing sync Our last step is to test that mobile sync works. With the emulator running, you should be logged in and see a list of brokers. This list should mirror the broker list that you see in your web browser on your local machine. In your web browser, edit one of the broker names, and save that change. Then, in your emulator, you can power off the phone or switch to another application and then switch back to your . You should see your list of broker names updated with the change you made in your browser: sfdc-mobile-app And that’s all! Again, this is just a foundational building block for you to learn from. In just about 150 lines of code, we’ve devised a solution for a rather complicated series of moving parts: Mapping Salesforce custom objects to JSON Setting up a timer to periodically sync data back and forth between a Salesforce org and a mobile device Handling errors (and recovering from issues) around network connectivity Conclusion The Salesforce Mobile SDK makes it tremendously easy to connect mobile devices with Salesforce data. In this post, you learned how to query and manipulate data from your phone, and saw the results reflected instantaneously on Salesforce. You also learned about Mobile Sync and its role in anticipating issues with connectivity. And yet, with all of this information, there’s still that the Salesforce Mobile SDK can do. Take a look at the on working with the SDK. Or, if you prefer to do more coding, there are to keep you busy. We can’t wait to see what you build! so much more complete documentation plenty of Trailhead tutorials

Building a Kotlin Mobile App with the Salesforce SDK: Synchronizing Data

By: David Zhu Metadata synchronization (sync) is a core feature in Alluxio that keeps files and directories consistent with their source of truth in under storage systems, thus making it simple for users to retrieve the latest version of data from Alluxio. Meanwhile, understanding the internal process is important in order to tune the performance. This article describes the design and the implementation in Alluxio to keep metadata synchronized. Why is Metadata Sync Critical in Alluxio In Alluxio, metadata refers to the information of files and directories in the Alluxio file system, including information of their owners, groups, permission, creation and modification time, etc. Metadata is independent of their content — even if a file or directory is empty, it still has associated metadata. Alluxio maintains a copy of the file system or the object store namespace of the under storage system. Metadata consistency is important in Alluxio, especially when different clusters write/read data in the data pipeline and make changes outside of Alluxio. A typical scenario in the above diagram is a data pipeline with the combination of Spark ETL and Presto SQL. The ETL cluster (without Alluxio) writes data, followed by the analytics cluster with Alluxio reading the transformed data. Since Alluxio maintains a copy of the metadata from under storage and manages the metadata, when data changes in under storage with the ETL step, the instance of Alluxio on the analytics cluster must be made aware and kept consistent with the metadata in the underlying storage system to proceed correctly. How Does Metadata Sync Work in Alluxio Alluxio provides a file system abstraction in a unified namespace on one or more under storage systems. Accessing files or directories through Alluxio is expected to get the same result as directly accessing the under storage. For example, if the under storage mounted to Alluxio root is , listing the directory in Alluxio yields the same result as listing objects in and printing in Alluxio should return the same result as . s3://bucket/data "/" s3://bucket/data "/file" s3://bucket/data/file In Alluxio, metadata is only stored and served from Alluxio masters, whereas the content of individual files is served from Alluxio workers. By default, Alluxio loads the metadata from the under storage on demand. In the above example, an Alluxio master starting from empty does not have any information about after startup. This file is only recognized when some user lists directory in Alluxio or tries to access . This “lazy” behavior prevents unnecessary effort and significantly improves performance because metadata operations in under storage can be slow. s3://bucket/data/file "/" "/file" Note that updating metadata can be bidirectional. If all modifications to the file system occur through Alluxio, Alluxio only needs to scan under storage once to retrieve the initial state, then apply changes in both Alluxio and under storage synchronously as a part of the file system RPC calls. This will provide users with a consistent view of under storage. In reality, however, changes are often made to the under storage outside of Alluxio. As a result, Alluxio master must discover the addition, removal, and updates to files and directions in under storage, and apply the changes in the Alluxio file system. This process to synchronize the two namespaces is called metadata sync. How to Trigger Metadata Sync When applications make changes to the metadata of an Alluxio file, and this file is persisted, the change will always be propagated to under storage synchronously, and there is no need to trigger metadata sync. When applications update under storage files without letting Alluxio know, there are two approaches to govern the timing of metadata sync. 1. Rely on a time-based automatic sync One can set a sync Interval as an Alluxio property key “ ”. alluxio.user.file.metadata.sync.interval When the value is -1 (current default value), Alluxio will never resync with under storage after initial load. When its value is set to 0, Alluxio will always resync with under storage whenever metadata is accessed. When the value is positive (default unit is ms), Alluxio will (best effort) not re-sync a path within that time interval. Note that using this approach, if a path in Alluxio has never been accessed, it will never trigger a sync. Once the path is accessed after the interval of sync expires, Alluxio will sync with the under storage again. For example, in a Presto job, the query planning stage lists all the files necessary for the job, which will trigger a sync if those paths have not been accessed recently. However, subsequent stages of the job will not sync unless the job duration exceeds the sync interval. So we can technically resync more frequently than the sync interval in this case. This property key can be customized with a new global default value (when set in ) or on a directory basis recursively to apply all its children. alluxio-site.properties 2. Sync manually using LoadMetadata flags If sync metadata does not happen because of the sync interval, most Alluxio operations will continue to execute with the current metadata in the Alluxio file system with some exceptions: For most of the users, Alluxio CLI “ ” is the easiest way to trigger sync manually. For example, one can run “ ” to force update the metadata of Alluxio path “ “. loadMetadata bin/alluxio fs loadMetadata /path/to/sync /path/to/sync For applications built on Alluxio file system (Java), there are two API methods and which retrieve the metadata for a path or a directory. When calling these methods, there is an additional field of in the option for each call, which may trigger a “loadMetadata” process by the master on the Alluxio path being inquired. This process can be considered as a simplified version of the sync by only loading file metadata from the under storage but not modifying a file’s metadata if it is already in the Alluxio. if is set to , then nothing will be loaded, and if the file does not already exist, a FileNotFound exception will be thrown. When is , we will only load the metadata once for each directory. This affects these two file system calls only, and this option is only considered when sync does not happen. SDK getStatus listStatus LoadMetadataPType LoadMetadataPType NEVER LoadMetadataPType ONCE How is Metadata Sync Implemented Alluxio master potentially triggers metadata sync on an Alluxio path when the Alluxio master receives an RPC request to retrieve the metadata of this path. Rather than having a dedicated service to traverse the entire file system inode tree and keep it in sync, this effort is amortized by each individual Alluxio file system operation on master. The high-level process for the syncing inside the RPC request is: Given the Alluxio path, determine if it is not consistent with the corresponding under storage path.  This means the under storage path does not exist, or has metadata that differs from Alluxio. This part is done using the RPC thread. The steps from 1 should populate the sync queue, and we iterate through the sync queue and process each path in a worker thread from a separate thread pool. The order of traversal is a BFS order, because additional paths are added at the end of the queue. Concurrency and executors will be discussed in the parallelism section in more detail. This part is carried out by the sync threads, and under storage, information is read using the under storage prefetch thread. The reason for this is to overlap communication with computation. Sync threads need to manipulate inode tree, and under storage, prefetch can start as soon as we determine that we need the information at some point in the future. Prefetching threads load the under storage status information into the under storage status cache, which is discussed in the cache section. Note that the metadata sync process can be relatively expensive and block other operations if they touch the same part of the inode tree. This is because the sync process may write-lock the part of file system metadata it is updating. Particularly, when syncing a particular path in a tree, the RPC handling thread will first acquire the read lock on the entire path to the file. Because the syncing thread needs the ability to create paths as well, it needs a write lock for the sync root path. As the syncing thread processes each path under the root path, additional locks are acquired. A write lock to the file path is acquired by the syncing thread and released as soon as the path is processed. Performance Optimization Tuning parallelism One can tune the level of parallelism to sync metadata by controlling three configuration parameters: This indicates the number of individual files to sync concurrently within a single metadata sync request (e.g., on a directory). alluxio.master.metadata.sync.concurrency.level This indicates the number of concurrent threads for all sync operations alluxio.master.metadata.sync.executor.pool.size This indicates the number of concurrent threads that can perform under storage prefetch operations for all sync operations alluxio.master.metadata.sync.ufs.prefetch.pool.size Caching result There are three types of different caches, with different goals and uses in the metadata syncing process. Here is a quick summary of all of them. is a negative cache, used to avoid checking the under storage for those paths that have been known to not exist. It uses prefix matching to determine if a path is in the under storage or not. For example, if the path is in the absent cache, we know must not exist on the under storage either. In addition, Absent Cache entries have a timestamp attached. So that we know the time it was last checked in the under storage. This is useful when the sync interval is some time period, we use the timestamp to determine whether we need to recheck the under storage for the existence of the file or directory. AbsentCache /a/b /a/b/c is a cache used to prefetch under storage statuses during the sync process. Instead of fetching the path information when needed, we can often prefetch some file statuses while we process the current directory. UfsStatusCache is a positive cache and contains paths that have been synced with the under storage recently. When we receive a metadata operation, we will check with this cache to determine whether we need to sync a particular path. UfsSyncPathCache Summary Metadata synchronization is one of the most important features in Alluxio. There are multiple different ways to trigger synchronization with different performance tradeoffs. Inside Alluxio master, there is a list of optimizations applied to speed up the synchronization. More Info Documentation of Alluxio Unified Namespace Two Ways to Keep Files in Sync Between Alluxio and HDFS Also Published Here

Metadata Synchronization: Design, Implementation and Optimization

The blog provides information on the build & deploy tools that are centered around microservices application delivery and are being popularly used in the industry today. Continuous Integration/Continuous Deployment, or CI/CD, has become a key cornerstone for building, testing, and deploying applications across environments these days. Here, we talk about the popular build & deploy tools that are centered around microservices application delivery and are being popularly used in the industry today. Take a look! While CI helps to mitigate the risks and enables a quick and seamless code release by automating multiple code changes from varied developers of the project. On the other hand, CD enables DevOps teams to deliver the integrated code to production seamlessly, thus providing a quick and effective automated process to release new features and updates to customers without much hassles. The amazing benefits of Microservices include improved agility, enhanced productivity, faster development time, and much more. No doubt, this is the reason why Microservices architecture is the next wave of today’s dynamic market space. To leverage the most out of the CI/CD pipelines and to overcome complex Microservices challenges, it’s important for enterprises to adopt the best practices and tools. Here are 5 popular build & deploy tools for seamless and secure delivery of CI/CD pipelines. 1. GitLab Known as a “complete DevOps platform,” is an open-source web-based service that helps the development, security, and Ops teams to collaborate in a better way. GitLab improves the pace of the development cycle, reduces development costs, renders faster time to market, and increases developer productivity. Source: GitLab GitLab Just like GitHub, one of the main features of GitLab is the ability to act as a source code repository. However, the approach behind GitLab goes beyond just being a code repository and spans various DevOps CI/CD features to help software teams in automating delivery pipelines. GitLab was brought into being with the basic idea of bringing all the features of source code management and Microservices CI/CD tools that build, test, and deploy code under one roof. Key Features: - Well-orchestrated Source-code management. - Quick and hassle-free build/deploy/test functionality. - Cloud-based SaaS (software as a service) and on-premises services are offered. - Time-tracking ability with granular details. - Automatically administered Analytics report. - Complete security testing of static applications. 2. CircleCI Source: CircleCI is one of the most popular cloud-based CI/CD tools used by enterprises as an alternative to Jenkins, an open-source CI/CD and deployment automation DevOps tool. CircleCI helps in the automated and hassle-free setup of CI/CD pipelines. CircleCI It is a well-reputed tool known for its incredible features, including speed, performance, and scalability. Leading companies such as Facebook and Spotify use CircleCI for implementing their CI/CD workflows. Leveraging integration with BitBucket and GitHub, CircleCI helps DevOps teams in automating the build & deploy process of an app. CircleCI enables the development teams to test their code after every commit and create workflows for large-scale CI/CD pipelines. The tool enables hassle-free automation of the Microservices CI/CD pipelines at a great pace. Key Features: - Simple and basic initial configuration. - Default Integration with GitHub and BitBucket. - Scalable cloud platform. - Can seamlessly deploy to a wide variety of environments, including AWS, NPM. - Ability to select “Build Environment.” 3. BuildPiper Source: BuildPiper As Microservices CI/CD workflows scale, managing them becomes much more complex. is a next-gen CI/CD pipeline automation and workflow orchestration tool. The main idea behind the development of BuildPiper is to enable a highly intuitive and easy setup of feature-rich delivery pipelines for a seamless and secure product release. As the pipeline events are logged at every step, BuildPiper renders comprehensive visibility allowing DevOps teams to analyze pipeline efficiency and troubleshoot effectively. BuildPiper One thing that makes BuildPiper the most coveted platform for enabling enterprise-grade Microservices is its ability to build & deploy multiple microservices in a single pipeline through its robust and secured Macro pipelines. Leveraging the out-of-the-box capabilities of Macro pipelines, BuildPiper helps in setting up Microservices CI/CD pipelines with much reduced time and effort. Automated CI checks and comprehensive CI analysis offered by ‘BuildPiper’ is what sets it apart from the league and helps the software teams to identify defects and vulnerabilities in the initial stages of the pipeline itself, ensuring a quick, compliant, and hassle-free code release. Source: BuildPiper BuildPiper helps you in a simple pipeline construct of stages and jobs. The four types of stages it offers include Edit Stage, Add Job, Delete Stage, and Delete all jobs. Jobs types that users can perform with BuildPiper include, Build, Deploy, Promote, and API Call. Automated CI Checks - Comprehensive CI Analysis - Multi-Language CI Support - Intuitive and hassle-free Setup of Feature-rich Delivery Pipelines - Multiple Options to Trigger the Pipeline that includes Manual, Schedule, and SCM Poll. - Simple and Uncomplicated Pipeline Construct of Stages &  Jobs - Real-Time Metrics via an Interactive and User-friendly UI - Easy to understand and see UI and YAML view - A Kubernetes dashboard for a 360-degree cluster view. Source: BuildPiper Key Features - 4. GitHub Actions Source: Github is an event-driven automation tool that enables teams to seamlessly implement CI/CD workflows natively on the GitHub platform. GitHub Actions Using the tool, developers can automatically trigger workflows for any GitHub event — such as a commit, issue creation, or pull request. Say, for instance, a pull request can be performed to automatically run and execute a suite of tests. GitHub Actions makes it easy to automate software workflows with the help of best-in-class CI/CD practices. Development teams have complete flexibility in doing code reviews, handling branch management, and implying issue triaging in a way they want to. A major advantage of using GitHub Actions is CI/CD processes that earlier required multiple integrations and services can run directly in the GitHub repository now. This means less complexity and less friction in the Microservices CI/CD pipelines. Key Features: - Ability to build, test, and deploy directly from Github. - Automatic workflows can be initiated by any GitHub event such as push, pull, issue creation, etc. - Real-time logging of events. - Built-in secret stores to manage database credentials and keys. - A large library for creating custom actions. 5. Argo CD Source: CNCF/Argo Logo is an open-source tool for Kubernetes to execute workflows, manage Kubernetes clusters, and implement GitOps in the right manner. It has become one of the most popular “GitOps” tools for Kubernetes (K8s) in the past few years. One of the reasons for its burgeoning popularity is that it’s a cloud-native and open-source tool. Argo uses git-based repositories (like GitHub) as a source of truth that provides a clear picture of the desired state of Kubernetes-based applications. It acts as a Kubernetes controller that monitors the app and can identify and resync Kubernetes clusters that are not in their desired states. Argo CD Key Features: - Automate K8s deployment. - Supports a variety of configuration management templates such as YAML, Helm, etc. - Rollback to any desired configuration in a supported repository. - Simplifies complex and advanced Kubernetes deployment strategies such as Canary and Blue-Green made easy. - Continuous reporting of the health status. CI/CD has now become an integral part of modern software development methods and practices. A secure and well-rounded CI/CD pipeline is critical to keeping an organization at par with other software companies in this competitive market space. This is why tech leaders and entrepreneurs across the globe are looking for the right tools for their businesses. So, if you’re planning to make your next move for CI/CD implementation, this blog will definitely help you out in identifying the different build & deploy tools available for implementing CI/CD. Wrapping it All Up Also Published On: https://blog.opstree.com/2021/07/27/top-5-microservices-ci-cd-tools-to-look-out-for/

5 Best Microservices CI/CD Tools You Need to Check Out

In today’s research article, we’ll focus on dissecting Bitcoin’s Four Year Cycle in an effort to better understand Bitcoin’s current price predicament and gain insight into some of the more important technical steps price needs to achieve to ensure future exponential growth. First, we’ll focus on breaking down the Bitcoin Four Year Cycle into distinct phases in the market cycle. We’ll emphasise the psychology that underpins each of these phases. Secondly, we’ll discuss how the pandemic-induced crash in March 2020 was in fact a theoretically sound correction through the lens of Bitcoin’s historically recurring pre-Halving price tendencies. Then, we’ll emphasise the importance of the Bitcoin Halving in shaping Bitcoin’s Four Year Cycles. Later, we’ll talk about what Bitcoin needs to technically achieve to open itself up to significant upside in 2021 so as to reach a new All Time High. And finally, we’ll investigate how the Four Year Cycle brings to light some crucial observations about Bitcoin’s price behaviour and discuss how these observations could inform us as to its potential price growth in the future This content was originally featured in the Rekt Capital newsletter Bitcoin - The Four Year Cycle The twelve month candlesticks on Bitcoin’s price chart showcase a recurring theme to form Four Year Cycles. Each twelve month candle denotes a specific phase in the Four Year Cycle. Here is a breakdown of each of these phases: Phase 1 — Exponential Highs This candle highlights the exponential nature of the uptrend that leads Bitcoin to new All Time Highs. It is at this stage in the market where investors are at their most euphoric and experience the most extreme form of “the fear of missing out.” It is these extreme emotions that compel investors to buy at higher and higher prices. This prolongs the bull trend and presses price to new exponential highs. This phase is also the “topping out” phase where Bitcoin reaches its new All Time High and price peaks. Phase 2 — Correction Following the euphoria of the previous twelve month candle, this phase corrects for the over-exuberance and overoptimism in the market. It is during this time that Bitcoin sheds considerable valuation as a result of cascading sell-side pressure from profit-taking investors. Bitcoin experiences a healthy retracement after an exponential growth cycle. Phase 3 — Accumulation After an extended correction in price, Bitcoin experiences waning sell-side momentum and begins to bottom out. This is where bargain buyers tend to step in and accumulate Bitcoin at a deeply discounted prices. It is these types of buyers that form the very beginnings of new demand for Bitcoin. Phase 3 happens to be the point of maximum financial opportunity for Bitcoin investors as the prospect of reward greatly exceeds the risk of downside. Phase 4 — Recovery and Continuation The bottoming out period is over and a new uptrend has been confirmed as price experiences trend continuation towards the upside. In the context of Bitcoin’s twelve month candlesticks, historically this phase has been all about eclipsing a previous area of sellside pressure (i.e. “resistance” whereby sellers resist price from continuing to the upside; this level is denoted by a black horizontal line). It is during this period where a twelve month candle close above that level of resistance is an important technical step in confirming a shift in market psychology at said level. This level is a key psychological inflexion point that tends to spur strong emotions in the market and resulting buy-side momentum. Upon a successful twelve month candle close, this is when market psychology at this key switches from selling to buying. Did The March 2020 Crash Distort The Four Year Cycle? Pre-Halving Retrace Complete ✅ It’s true that the pandemic-induced sell-off caused Bitcoin to deviate from its historically recurring tendencies. In the grander scheme of Bitcoin’s overall price history however, this deviation wasn’t as substantial as most might think. In fact, this retrace was historically sound; after all, pre-Halving retraces occur a few weeks prior to the Halving. This was a pre-Halving retrace. Let’s briefly recount Bitcoin’s pre-Halving retraces across time: Bitcoin experienced a -50% retrace approximately 100 days prior to Halving #1 in 2012 and lasted 2 days. Bitcoin experienced a -38% retrace approximately 24 days prior to Halving #2 in 2016 and lasted 44 days. Bitcoin experienced a -63% retrace approximately 88 days prior to Halving #3 in 2020 and lasted 29 days. So while Bitcoin’s crash in March was anomalous in terms of how deep it was, . the fact that a significant correction occurred prior to the Halving was theoretically sound and in line with Bitcoin’s historically recurring pre-Halving price tendencies I first emphasised this “pre-Halving retrace” price tendency in my September 6th, 2019 article called “ Bitcoin Halving - Everything You Need To Know ”. Here’s the relevant excerpt from that article So while the March 2020 crash was anomalous to some degree (i.e. retrace depth), it was generally in line with historically recurring price tendencies Bitcoin tends to showcase prior to its Halvings. And as I mentioned in my recent research article (which is an up-to-date analysis of all Bitcoin Halvings), Bitcoin has in fact reclaimed its key higher-timeframe market structure since the March 2020 crash and will now be challenging key resistances to actually resynchronise with its previous market cycles. Bitcoin Halving: A Historical Analysis Bitcoin Halving and The Four Year Cycle The Bitcoin Halving is an event whereby the amount of new Bitcoin that gets created every 10 minutes gets cut in half (denoted in blue on the price chart). In May of this year, Bitcoin’s third Halving took place, reducing the block reward of 12.5 Bitcoin every 10 minutes to 6.25 Bitcoin. Not only does the Bitcoin Halving mark a fundamental change in the Bitcoin protocol, but it also as well. has tremendous ramifications for Bitcoin’s price Historically, Bitcoin has rallied exponentially after each of its Halvings. Bitcoin rallied +3,400% after Halving 1 and 4,080% after Halving 2. It is a fundamental catalyst that has played a significant role in propelling Bitcoin’s price to exponential highs. And the fact that I’ve discussed the pre-Halving retrace in the context of Bitcoin’s Four Year Cycle is no coincidence. Because the Bitcoin Halving events occupy a curious price positioning if we analyse them through the lens of the Four Year Cycle. In the context of the Four Year Cycle, the Bitcoin Halving occurs in the “Recovery and Continuation” candlestick (i.e. Candle 4) and precedes the new exponential Candle 1 in the new Bitcoin Four Year Cycle. In other words - the Bitcoin Halvings tend to occur a year prior to Bitcoin’s exponential rallies to new All-Time Highs. Bitcoin Set For New All Time High In 202 According to the 4 Year Cycle, Bitcoin is currently in Phase 4. That means Bitcoin is in the final phase of its current Four Year Cycle. And the primary aim of Phase 4 is to eclipse the previous resistance that effectively prompted a bear market for Bitcoin’s price a few years earlier (i.e. “Candle 2”). That resistance is ~$13,900. So for the 4 Year Cycle to continue to play out, Bitcoin needs to manage a twelve month candle close above the ~$13,900 price level by the end of 2020 so as to precede an exponential Candle 1. At the time of this writing, Bitcoin is really not that far away from eclipsing ~$13,900. Bitcoin’s current twelve-month candle still looks healthy, with a great deal of bullish momentum behind it. It has taken the shape of a bullish hammer candlestick, resembling the bottoming out candle from the previous Four Year Cycle which preceded immense upside in of itself. And hammer-shaped candlesticks tend to precede bullish trend reversals towards the upside. As per the historical tendencies of the 4 Year Cycle, a twelve month candle close above ~$13,900 is the technical confirmation that is necessary to open Bitcoin up to a challenge and subsequent eclipse of $20,000 to reach a . new All-Time High in 2021 How Much Could Bitcoin Rally In 2021 To address the question of where Bitcoin could rally to in 2021, we have to take into account a few noteworthy technical tendencies that can be observed by dissecting Bitcoin’s Four Year Cycle: If BTC Breaks $13900 in 2020, BTC Will Not Eclipse $20000 in 2020 If Bitcoin manages to perform a twelve-month candle close above $13,900 that would mean Bitcoin would eclipse $13,900 this year to make a new cycle high, but eclipse the old All Time High of $20,000. not Nonetheless, a twelve-month candle close above $13,900 that would open Bitcoin up to enter a new exponential Phase 1 in a new Four Year Cycle. That is, a new All Time High would await in 2021. Bitcoin To Turn $13,900 Into Support in Early 2021 Upon successful twelve-month candle close above $13900 in 2020, Bitcoin would attempt a retest of $13900 as support very early on in 2021, before embarking on a new exponential rally to establish a new All Time High. This is a necessary technical prerequisite that Bitcoin has performed throughout history. This is when key Four Year Cycle resistances (i.e. previously black horizontal levels) change from being a source of sell-side momentum (i.e. red horizontal level) to a source of buy-side pressure (i.e. green horizontal level) Diminishing Rate of Return on Bitcoin Investments It is Phase 1 candles that springboard Bitcoin to new All Time Highs. At a glance at the price chart above however, the exponential Phase 1 candles appear to produce less exponential rallies over time (i.e. black arrows). That means Bitcoin’s next exponential Phase 1 in the new Four Year Cycle will likely be less exponential than previous Phase 1’s. It is likely there are diminishing returns after each Bitcoin Halving. Much like the actual Phase 1 twelve-month candlesticks, the rate of overextension in the form of upside wicks past these crucial Four Year Cycle resistances (i.e. black horizontal levels in the chart below) are also getting shorter and less volatile over time. Where Could Bitcoin Rally To In 2021? If we assume the rate of diminishing Post-Halving returns to remain constant across all exponential “Candle 1’s”, Bitcoin could rally exponentially to a relatively conservative new All Time High of ~$90,000. That said, this conservative extrapolation of current Post-Halving bull trends by standards of Bitcoin’s price history doesn’t account for upside wicks past key Four Year Cycle resistances (i.e. black horizontals). This means Bitcoin could very well overextend past $90,000 and even beyond the psychological level of $100,000 before finally rejecting. P.S. Thank you for reading and feel free to sign up to my newsletter for more crypto insights. Also published at https://rektcapital.substack.com/p/fouryearcycle

Stories, News & Mentions about RESYNC

Resync’s Stories on HackerNoon