Our mission is to enable companies to turn their data into an asset

Meru Data

merudata

It is difficult for the metadata in dNFT(dynamic NFT) to achieve “decentralization” and “dynamic change” at the same time. Decentralized metadata storage About (most NFTs are static) metadata is decentralized and stored on IPFS. The format suffix is *,* where refers to the NFT collection, where is a number, such as “26”. An example of such a project is , the metadata of which is stored in IPFS, although the BaseURI used for IPFS can be changed by the contract owner. 48% of NFT ipfs/your_collection_file_hash/tokenid your_collection_file tokenid Bored Ape Yacht Club Suppose the platform wants to provide NFT customization capabilities and put metadata in decentralized storage. In that case, developers need to exhaust all possible combinations of attributes, put all the varieties of metadata on IPFS, and then index according to the combination of attributes to find the in IPFS to get the metadata. tokenURI The disadvantages of this approach are: First, many metadata combinations lead to much storage cost in advance. You have to store them all in advance; Second, if the platform wants to add a Smartglasses option to the feature of Glasses, or when it provides the ability to change dynamically, it can change the metadata information in the NFT about the game level and other metadata information in the NFT according to the goals of the World Cup during a match. At this point, the timeliness requirements for metadata changes are higher. For example, this dynamic change almost equals replacing a new NFT, and the cost of changing NFT metadata will rise sharply. Dynamically update first The second method is to abandon the decentralized metadata storage and only put the ownership on the Chain to ensure timely and rapid dynamic changes. Some projects generate metadata through a centralized server, and the metadata information of the customized NFT will be combined and generated in a centralized manner. To be able to display different houses in 2022 and 2027, NFT minter should first put other roofs on the Chain, store metadata off the Chain, and the URL of a centralized service to do the correlation, pointing to one of them in 2022, and replacing it in 2037 to another one. However, they store metadata on their central storage, and these servers can fail in the blink of an eye, costing NFT owners millions worldwide. More importantly, it isn’t easy to guarantee that such storage will not be tampered with. Especially for some game NFTs, their value does not lie in the displayed pictures but in the information that is crucial in the game, such as force values, levels, etc. This information is stored in metadata. It is impossible to guarantee that only the owner of the NFT owner who can change the metadata based on centralized storage. And when the metadata storage is separated from data on the Chain, the dynamic metadata history does not leave traces on the Chain, and the two are unrelated, so it isn’t easy to carry out practical analysis and insight. Year Transaction price Metadata history 2021 $40,000 old roof; old door 2025 $45,000 old roof; old door 2027 $48,000 old roof—>new roof; old door 2029 $70,000 2031 old door—>new door; new roof 2037 ? The history of the house Currently, only the first two columns of data are kept on-chain, which you can see directly from platforms like OpenSea. But you only know that the price is increasing over time, but you need to understand why. The last data column is in the platform’s centralized server, which is unreliable. For example, a sharp price increase in 2029 is due to the repair of a new roof — the information about this is in the third column of metadata. Because the metadata is changing and is not stored on the Chain, it may be tampered with, so you cannot know, nor is it challenging to analyze and understand what a more reasonable price in 2037 is. These problems may lead to a poor display experience of dNFT in the NFT trading market, making it difficult for dNFT to circulate and prosper in the secondary market. For example, in OpenSea, a dNFT that changes with the daily sunshine conditions rotates the animation every 8 hours, the change history of metadata is not presented on the website, and the dynamic feature is completely invisible when you first enter it. You have no way of knowing this is dynamic. Put all the things on the Chain. Therefore, it is best to put all metadata, images, transaction information, etc., on the Chain and can add the metadata and change it conveniently and dynamically. Your signature should perform the changes, and the NFT platform as an intermediary minimizes the direct touch of data (reading and writing, caching, etc.). DB3 network Here, let me introduce the , a decentralized storage solution that adopts the key-value storage paradigm and has advanced database properties, such as mutation and programmability. DB3 network Change metadata quickly based on decentralized storage. DB3 can support convenient CRUD-like operations, insert, update, read and delete stored data. This is easy to achieve for centralized storage, but it is challenging for decentralized storage. The reason is that the update operation conflicts with the immutable characteristics of the blockchain. But in use cases such as the dynamic NFT mentioned above, immutable storage is important from a security point of view. Still, it limits the number of possible applications (such as the evolutionary ability of NFT). A mutation is defined in the DB3 network as our core concept. All operations on data modification are defined as mutation. You can use mutation to insert, update and delete data. For example, you can easily complete the data renewal when you want to modify roof color metadata. >put house roofcolor offwhite
submit mutation to mempool done!
>get house roofcolor
roof -> offwhite At the same time, mutation can perform data update operations, and the other three: insert/update/delete and the hash returned by these operations will be submitted to the block. the operation is recorded into blocks. Analyze metadata history onchain Every metadata change of dNFT will be recorded on the chain like a snapshot. At the same time, because we have optimized the efficiency of our storage engine, we can achieve high-performance queries, not inferior to the web2 database. Therefore, in the future, analysis of dNFT will be very easy to read and display a more credible metadata history. Claim metadata ownership Secondly, the description information of dNFT will be stored in a concept similar to the “table” in web2. The operation and modification of this table need to obtain the user’s permission, which the user completely controls, ensuring the ownership of dNFT. The centralized service should barely touch metadata. Since the metadata defines the asset, securing metadata is as important as securing ownership. Our ： Github Browse some design docs; Start coding from ; goodfirstissue Discuss with us according the . WIPissue Also published . here

Storing Dynamic NFT Metadata on the DB3 Network

Dear Bankless Nation, Some popular NFT platforms from the 2017-2018 era, like Ascribe, Digital Objects, and Editional, eventually went under. These projects stored their metadata on their own servers, so when they went bust, their tokens’ visuals were lost to time. Lost NFTs are nothing new, then. But these early losses later spurred lots of advances around decentralized storage networks as well as totally on-chain NFTs, whose metadata are forever retrievable from the blockchain. Not everyone pays attention to the lessons of crypto history, though. For example, consider Sam Bankman-Fried of FTX notoriety. This is what his first NFT minted on FTX looked like circa September 2021: No Picasso, right? Certainly not, but at least it had visuals and metadata then. Fast forward to today and in the wake of FTX’s collapse that same NFT, plus all the other NFTs minted at FTX, now have broken metadata: Why? Because SBF and FTX relied on a centralized server system and it’s not running those servers anymore. This system was “good enough” before, but it’s nonexistent now! As such, this episode is the latest big reminder that metadata — whether that be through nodes on decentralized storage networks like Arweave and IPFS or on-chain through Ethereum nodes — is part of the magic of NFTs. decentralizing All that said, . In the meantime let’s catch you up on all the latest headlines in the space, have a great weekend everyone ✌️ backup what NFTs you can when you can -WMP 📣 General NFT News held its third drop ft. Li Jin: Bankless Bankless Collectibles commenced claims for its . Blur second airdrop introduced the : CoinLedger NFT Tax Loss Harvestooor , a new general NFT marketplace, . GigaMart launched introduced the Ledger Stax, a new crypto hardware wallet that’s capable of displaying NFTs: Ledger introduced . Mirror improved NFT embeds added to its community marketplace generator. Rarible Polygon support , a membership rewards program in collaboration with Polygon, . Starbucks Odyssey kicked off , an early-stage web3 fund, . Variant acquired multiple Nouns 🏆 Digital Collectibles launched oracle connectivity, meaning finiliars now respond with emotional animations to their native cryptos’ prices: Finiliar reached a 4Ξ floor for the . Pudgy Penguins first time this year mints and holders stats . Reddit Avatar continue to boom 🎨 Cryptoart selections began, a project he collaborated on with the estate of artist László Moholy-Nagy: Dmitri Cherniak’s Light Years added a , for tracking recent payments and royalties, to Manifold Studio. Manifold Token Activity dashboard , a new marketplace for generative art, has arrived: sansa 🎮 Decentralized Gaming introduced , a new series of expanded gameplay mechanics. Axie Infinity Axie Core , a decentralized esports team, entered into a . BLKHVND partnership with ParagonsDAO introduced code that allows projects to receive royalties from . Pirate Nation both Blur and OpenSea 💽 Music NFTs celebrated its first birthday: Sound 🌐 Virtual Worlds , the parent entity of Aave, acquired mobile with plans to integrate web3 social infra Lens Protocol into the platform. Aave Companies mobiel metaverse Sonar added support for to its marketplace. Decentraland LAND rentals 💎 DeFi x NFTs borrowers can now mint, transfer, or sell their . NFTfi loan obligations , a protocol for collective ownership of NFTs, . Tessera launched Action steps 🆕 Read and in (newsletter) The Status of Web3 Social The Perks of Decentralized Perps Bankless 🗣️ Watch on (YouTube) The Bull Case for Modular Blockchains Bankless 🌱 Watch on Circles UBI with Martin Koppelmann Green Pill 🙇 Catch up on my other write-ups from this week! ✧ NFT TCRs ✧ Collective NFT experiences with Tessera ✧ The Bankless Holiday Crypto Gift Guide Author Bio William M. Peaster is a professional writer and creator of Metaversal —a Bankless newsletter focused on the emergence of NFTs in the crypto economy. He’s also recently been contributing content to Bankless, JPG, and beyond! First Published Here .

Where Has Your NFT Metadata Gone?

The first step you’ll take to building an NFT minting Dapp is to prepare the artwork. Without the artwork, you can’t have your NFT project actualized. Another reason for preprocessing your artworks is for generating metadata for each of the images. Without this metadata, your NFT cannot be sold in any of the big secondary markets, such as Opensea. For example, the image below is an NFT in the Opensea marketplace bearing some metadata which can also be seen below. The above information including the artwork, its details, and traits can be seen on the IPFS image below. Setting up Project Dependencies Open your terminal and navigate to your project directory or create a project folder at a specific location. For example, running on the terminal will create a folder called “ on your specified location. **mkdir preprocessor** preprocessor” Next, run the following commands on the terminal. cd preprocessor
npm init -y
npm install sharp @faker.js/faker --save-dev The above commands will install both and libraries on your project. The faker library will help us generate random information. Whereas, the sharp library will help us process the image to a certain dimension, quality, and format. sharp faker Next, create a folder in your project called and another called . In the “arts” folder, put all the images for processing inside it. arts outputs With the above steps accomplished, open the project on “VS code”. The project structure should look like the one below. - preprocessor/
  - arts/
  - node_modules/
  - outputs/
  - package.json
  - package-lock.json Great, let’s move on to coding the engine responsible for processing the images. Prerequisites You’ll need the following installed on your computer to complete this tutorial. NodeJs IPFS Desktop app VS Code Coding the Processing Engine Create a file at the root of your project called and paste the following code **imagify.js** const fs = require('fs')
const path = require('path')
const sharp = require('sharp')
const { faker } = require('@faker-js/faker')
const input = './arts'
const output = './outputs'

let img_counter = 1
const imgSize = { width: 500, height: 500 }
const desired_ext = '.webp'
const base_url = 'https://ipfs.io/ipfs/REPLACE_WITH_IPFS_CID/'
const attributes = {
  weapon: [
    'Stick',
    'Knife',
    'Blade',
    'Clube',
    'Ax',
    'Sword',
    'Spear',
    'Gun',
    'Craft',
  ],
  environment: [
    'Space',
    'Sky',
    'Desert',
    'Forest',
    'Grassland',
    'Moiuntains',
    'Oceans',
    'Rainforest',
  ],
  rarity: Array.from(Array(10).keys()),
}

fs.readdirSync(input).forEach((file) => {
  const orginal_ext = path.extname(file)
  const orginal_file_name = path.basename(file).split('.')[0]

  if (['.jpg', '.jpeg', '.png', '.gif', '.webp'].includes(orginal_ext)) {
    const id = img_counter

    const metadata = {
      id,
      name: `Adulam NFT #${id}`,
      description:
        'A.I Arts NFTs Collection, Mint and collect the hottest NFTs around.',
      price: 1,
      image: base_url + id + desired_ext,
      demand: faker.random.numeric({ min: 10, max: 100 }),
      attributes: [
        {
          trait_type: 'Environment',
          value: attributes.environment.sort(() => 0.5 - Math.random())[0],
        },
        {
          trait_type: 'Weapon',
          value: attributes.weapon.sort(() => 0.5 - Math.random())[0],
        },
        {
          trait_type: 'Rarity',
          value: attributes.rarity.sort(() => 0.5 - Math.random())[0],
          max_value: 10,
        },
        {
          display_type: 'date',
          trait_type: 'Created',
          value: Date.now(),
        },
        {
          display_type: 'number',
          trait_type: 'generation',
          value: 1,
        },
      ],
    }

    if (fs.existsSync(`${input}/${orginal_file_name + orginal_ext}`)) {
      sharp(`${input}/${orginal_file_name + orginal_ext}`)
        .resize(imgSize.height, imgSize.width)
        .toFile(`${output}/images/${id + desired_ext}`, (err, info) =>
          console.log(err),
        )

      fs.writeFileSync(`${output}/metadata/${id}.json`, JSON.stringify(metadata), {
        encoding: 'utf-8',
        flag: 'w',
      })
    }
    console.log(metadata)
    img_counter++
  }
}) The following steps will help you understand how this metadata processing engine works. Importing Essential Libraries const fs = require('fs')
const path = require('path')
const sharp = require('sharp')
const { faker } = require('@faker-js/faker')
const input = './arts'
const output = './outputs' The represents the file system, it is an inbuilt module that came with NodeJs. It has the responsibility of managing file reading and writing activities on your machine. fs The is another node module that helps you navigate through the directory structures on your machine. This will be instrumental in locating where our images are kept. path Sharp is the module we use for processing the image such as resizing and transforming to a different file type. We’ll use the faker to generate a random number. Lastly, the variable contains the location where the images to be processed are located and the points to the location where the processed images will be saved. input output Defining Essential Variables let img_counter = 1
const imgSize = { width: 500, height: 500 }
const desired_ext = '.webp'
const base_url = 'https://ipfs.io/ipfs/REPLACE_WITH_IPFS_CID/'
const attributes = {
  weapon: [
    'Stick',
    'Knife',
    'Blade',
    'Clube',
    'Ax',
    'Sword',
    'Spear',
    'Gun',
    'Craft',
  ],
  environment: [
    'Space',
    'Sky',
    'Desert',
    'Forest',
    'Grassland',
    'Moiuntains',
    'Oceans',
    'Rainforest',
  ],
  rarity: Array.from(Array(10).keys()),
} The above codes contain important variables to be used in the course of generating our metadata. helps us to number the images consistently with the current iteration. **Image_counter** defines the dimension of the width and height of each image to be processed. **ImgSize** speaks of the file format you want your processed images to bear. **Desired_ext** specifies the location where the images are to be stored on the IPFS. **Base_url** holds further information about each image’s metadata. **Attributes** Executing Recursive Task fs.readdirSync(input).forEach((file) => {
  if(['.jpg', '.jpeg', '.png', '.gif', '.webp'].includes(orginal_ext)) {
    // Images and metadata tasks are recursively performed here...
  }
}) In the above block of code, we used the file system library (fs) to loop through all the images in the location (arts). And for each of the images, we made sure our engine is only selecting images from an approved list of extensions. **input** Performing The Metadata Task const id = img_counter
const metadata = {
  id,
  name: `Adulam NFT #${id}`,
  description:
    'A.I Arts NFTs Collection, Mint and collect the hottest NFTs around.',
  price: 1,
  image: base_url + id + desired_ext,
  demand: faker.random.numeric({ min: 10, max: 100 }),
  attributes: [
    {
      trait_type: 'Environment',
      value: attributes.environment.sort(() => 0.5 - Math.random())[0],
    },
    {
      trait_type: 'Weapon',
      value: attributes.weapon.sort(() => 0.5 - Math.random())[0],
    },
    {
      trait_type: 'Rarity',
      value: attributes.rarity.sort(() => 0.5 - Math.random())[0],
      max_value: 10,
    },
    {
      display_type: 'date',
      trait_type: 'Created',
      value: Date.now(),
    },
    {
      display_type: 'number',
      trait_type: 'generation',
      value: 1,
    },
  ],
} In the code block above, we supplied values for each metadata space. For example, environments, weapons, and all the trait values are randomly and dynamically supplied. Performing The Image Transformation Task if (fs.existsSync(`${input}/${orginal_file_name + orginal_ext}`)) {
  sharp(`${input}/${orginal_file_name + orginal_ext}`)
    .resize(imgSize.height, imgSize.width)
    .toFile(`${output}/images/${id + desired_ext}`, (err, info) =>
      console.log(err),
    )

  fs.writeFileSync(`${output}/metadata/${id}.json`, JSON.stringify(metadata), {
    encoding: 'utf-8',
    flag: 'w',
  })
}

console.log(metadata)
img_counter++ In the snippet above, we used the file system module again to first locate each one of our artwork and resized it to our specified dimension (500 x 500). Also, we supplied a new name in line with the current iteration and gave it our desired extension (webp). Resizing and transforming the images into greatly optimized our artworks to an astonishing height. webp For example, where I subjected this image preprocessing engine to images totaling a size of . The size went down to for the extension and an astonishing for the extension. That huge size reduction accounts for a big leap in the load time of a Minting website built with my images. in this video 99 111MB 62MB .png 4.5MB .webp Lastly from the block of code above, we created metadata for each image processed, bearing both an identical name and a URL pointing to the image’s location. This metadata is what we’ll deploy to IPFS after processing the images. JSON Now, run the command below to have your image transformed. Be sure you are in your project folder. node imagify.js At this point, we are done with our image engine, our output folder should have the following file structure as the result. - output/
  - images
    - 1.webp
    - 2.webp
    - ......
  - metadata
    - 1.json
    - 2.json
    - ...... Uploading Images and Metadata to IPFS IPFS stands for the interplanetary file system, it is peer-to-peer and decentralized. There isn’t an easy way to pull out data stored on the IPFS and as such, it is a near-perfect peer to use along with your blockchain applications for storing media contents. To use the easy and less confusing way, head to the IPFS Desktop app installation page and follow the instructions specified there. After the installation is successful, open up the IPFS app and upload FIRST, and I repeat, FIRST upload the images folder. A unique CID (content Identification) string will be generated as part of the folder name, see the image below. Now, copy the images folder CID as can be seen from the image above, and replace it in your code. See the code below. **imagify.js** const base_url = "https://ipfs.io/ipfs/REPLACE_WITH_IPFS_CID/" //old string
const base_url = "https://ipfs.io/ipfs/QmY1zrFibpdHQ7qcqZqq7THsqTremZYepXNWR5Au3MF1ST/" //new string Now, run the again to include the accurate location of each image to your metadata. See an example of the generated JSON metadata before and after the replacement of the CID. **node imagify.js** JSON You can watch this video to understand how I used these Images and metadata on a full NFT minting Project. https://www.youtube.com/watch?v=QN3wb_mXBjY?embedable=true Before CID Replacement {
  id: 97,
  name: 'Adulam NFT #97',
  description: 'A.I Arts NFTs Collection, Mint and collect the hottest NFTs around.',
  price: 1,
  image: 'https://ipfs.io/ipfs/REPLACE_WITH_IPFS_CID/97.webp',
  demand: '4',
  attributes: [
    { trait_type: 'Environment', value: 'Forest' },
    { trait_type: 'Weapon', value: 'Craft' },
    { trait_type: 'Rarity', value: 4, max_value: 10 },
    {
      display_type: 'date',
      trait_type: 'Created',
      value: 1664478034024
    },
    { display_type: 'number', trait_type: 'generation', value: 1 }
  ]
} After CID Replacement {
  id: 97,
  name: 'Adulam NFT #97',
  description: 'A.I Arts NFTs Collection, Mint and collect the hottest NFTs around.',
  price: 1,
  image: 'https://ipfs.io/ipfs/QmY1zrFibpdHQ7qcqZqq7THsqTremZYepXNWR5Au3MF1ST/97.webp',
  demand: '7',
  attributes: [
    { trait_type: 'Environment', value: 'Moiuntains' },
    { trait_type: 'Weapon', value: 'Clube' },
    { trait_type: 'Rarity', value: 2, max_value: 10 },
    {
      display_type: 'date',
      trait_type: 'Created',
      value: 1664478110287
    },
    { display_type: 'number', trait_type: 'generation', value: 1 }
  ]
} Finally, as shown in the image below, upload the metadata folder to IPFS alongside the images folder. Fantastic, now let’s pin it on the web for the world to see. Currently, both folders are sitting on your local IPFS node (Your Computer), for it to be accessible worldwide, you need to use a Pinning service such as . Pinata Pinning your folders to IPFS First, head to and sign up if you haven’t done that before. Then click on the account icon and select API Keys. See the image below. Pinata pin manager On the keys creation page, click on create a new key and enter the name for the key. Observe the image below. Now copy the JWT key on your clipboard. This is what we’ll use to link our IPFS Desktop with our Pinata account. See the image below. Next, open up your IPFS desktop application, head to the settings tab and add a new service, select Pinata and paste your JWT token to the space provided. Refer to the image below. The last thing to do is to actually pin your folders to Pinata using the instruction below. Head to the files tab, click on the triple dotted line, and select set pinning. This will pop up in the image below. Select Pinata and apply, by so doing, your images folder will be accessible globally. Confirming Global Image Accessibility Head to this website, copy and paste your CID on the IPFS input field and click on the cache button. This scans the entire set of publicly available IPFS gateways in search of your images. See the image below. The results from the above image show that many IPFS nodes now have copies of the images folder available and accessible globally even if you deleted the original copy on your local node. With the steps clearly outlined for you, also Pin the metadata folder to make them publicly available online. Now you can use any of the links in the image above as your base URL for tokens. See the image below. ERC721 And there you have it, that is how to prepare and upload your artworks on the IPFS. Conclusion You will always encounter the need to understand how to preprocess and upload artworks on the IPFS at a batch scale. Once you understand how to work and process images to the IPFS you can start using it on your special web3 projects. Till next time, keep crushing it! See live demo usage on my NFT project on the Goerli testnet . See Opensea’s location . You can watch the video version on my YouTube Channel. here here https://www.youtube.com/watch?v=YzurgMLwFVQ?embedable=true About the Author I am a full-stack blockchain developer with years of experience in the software development industry. 6+ By combining Software Development, writing, and teaching, I demonstrate how to build decentralized applications on EVM-compatible blockchain networks. My stacks include , , , , , , , , and more. JavaScript React Vue Angular Node React Native NextJs Solidity to learn how to build a Web3 app from scratch. I also offer private and specialized classes for serious folks who want to learn one-on-one from a mentor. Subscribe to my YouTube channel . Book your Web3 classes here Now, let me show you how to represent the above image and metadata in your code.

How to Richly Preprocess your IPFS NFT Images and Metadata

Cómo preprocesar enriquecidamente sus imágenes y metadatos IPFS NFT

Comment prétraiter de manière riche vos images et métadonnées IPFS NFT

अपने आईपीएफएस एनएफटी छवियों और मेटाडेटा को बड़े पैमाने पर प्रीप्रोसेस कैसे करें

IPFS NFT 画像とメタデータを豊富に前処理する方法

Como pré-processar ricamente suas imagens e metadados IPFS NFT

Cách tiền xử lý phong phú Hình ảnh và Siêu dữ liệu IPFS NFT của bạn

如何丰富地预处理您的 IPFS NFT 图像和元数据

So far in this blog series, we've created our first live streaming channel, learned how to create a playback experience in the browser, and enhanced that experience with the Amazon Interactive Video Service (Amazon IVS) player SDK. In the next few posts, we're going to look at adding some interactivity to our playback experience. Live streaming is clearly popular, but it's more than just the video that appeals to viewers. Interacting with the broadcaster and other viewers is a major reason so many people keep coming back to certain streams and streamers. Amazon IVS gives us a few pretty amazing features that enable interactivity, and one of the cooler elements that we can use to enhance our live stream is (obligatory " "). timed metadata official doc link Timed Metadata? Timed metadata is a way to embed information in your live stream at a specific point in time. The metadata that you publish becomes a part of the video stream itself since it uses ID3 tags embedded directly into the video segments. That means they are available in later on if we record our live streams. ID3 tags? Ever wonder how your music player knows the artist, song title, album and more about the song it’s playing? Audio files contain a small chunk of data that contains this information. Audio players read this data and display it! What Is It Good For? All kinds of stuff! Using timed metadata, we can create an interactive experience for our viewers outside of the video player. Imagine a game show live stream that can render questions in the browser when the host asks them. Think about a live shopping experience that updates the page with product images, reviews, pricing information, or discount codes as the presenter is demoing the product. What about updating leaderboards, box scores, detailed stats during a live streamed sporting (or e-sports) event? Picture a live streamed bike race that broadcasts the rider’s GPS location with the stream and display it in real time on a map. There are so many use cases, and I’m sure you’ve already thought of a few for your own applications. Enough talk, let’s build something! Listening For Timed Metadata Let's head back once again to the Amazon IVS page. Notice that there is a test stream listed here that contains timed metadata. Nice! We'll plug this into our Amazon IVS player example that we've been working with and build upon it. Here's how that looks: test streams <video id="video-player" controls autoplay playsinline></video> const videoEl = document.getElementById('video-player');
const streamUrl = 'https://fcc3ddae59ed.us-west-2.playback.live-video.net/api/video/v1/us-west-2.893648527354.channel.xhP3ExfcX8ON.m3u8';
const ivsPlayer = IVSPlayer.create();
ivsPlayer.attachHTMLVideoElement(videoEl);
ivsPlayer.load(streamUrl);
ivsPlayer.play(); This stream is an example of a “live trivia” stream where the host reads a few trivia questions. A timed metadata event is embedded at the exact point in time that the host reads a trivia question. We can listen for, and respond to this event. If you remember from our last post, we can use on the Amazon IVS player to listen for various events, and ( ) is one of those events. addEventListener() TEXT_METADATA_CUE docs ivsPlayer.addEventListener(IVSPlayer.PlayerEventType.TEXT_METADATA_CUE, (e) => {
    console.log(e);
}); If we play our stream, we can check the console after the host reads a question and observe the logged event. It should look similar to this: {
    "startTime": 61.04233333333333,
    "endTime": 61.04233333333333,
    "type": "TextMetadataCue",
    "description": "",
    "text": "{\"question\": \"From what language does the term R.S.V.P. originate from?\",\"answers\": [ \"Russian\", \"Italian\", \"French\", \"Portuguese\" ],\"correctIndex\": 2}",
    "owner": "metadata.live-video.net"
} The key represents the time since the stream started the event took place (61 seconds in the example above). The interesting bit we're after here is the key, which contains the metadata. Here, it's a JSON string, but it can be whatever you need it to be. startTime text The only limitation on timed metadata is that the payload must be less than 1kb. That might sound small, but metadata isn't intended to deliver large payloads. We can perform additional steps on the client side if necessary. For example, we could pass a question ID in the example above instead of passing the question, answers, and correct answer and fetch the rest of the information from client memory or a remote API call. But since we've got the entire trivia payload, let's parse the metadata. const metadata = JSON.parse(e.text); The parsed metadata structure uses the following structure: {
    "question": "From what language does the term R.S.V.P. originate from?",
    "answers": [
        "Russian",
        "Italian",
        "French",
        "Portuguese"
    ],
    "correctIndex": 2
} Rendering the Question and Answers So we've got the questions, answers, and the index of the correct answer. Certainly our viewers would be really honest and never poke around in the browser's developer tools to view the correct answer, so we'll proceed with our trivia app on the honor system. Let's render the question and answers. First, we'll add some HTML markup (styled with Bootstrap) below the video player. <div class="card mt-3">

    <div class="card-header bg-dark text-white">
        <h4>Question</h4>
    </div>

    <div class="card-body">

        <div class="card-title">
            <h4 id="question"></h4>
        </div>

        <div id="answerContainer">
            <ul id="answers" class="list-group"></ul>
        </div>

    </div>

    <div class="card-footer text-center">
        <button class="btn btn-dark" id="check-answer-btn" disabled>Submit</button>
    </div>

</div> So we've got a container for our question, and the answers. We've also added a button to check the answer that is initially . Let's add some code inside of our event handler to store the correct answer in the global scope so that we can check the user's answer later on. disabled window.correctIndex = metadata.correctIndex; Now we can enable the submit button, display the question, and clear out the answer container. // enable the submit button
document.getElementById('check-answer-btn').removeAttribute('disabled');

// display the question
document.getElementById('question').innerHTML = metadata.question;

// clear previous answers
const answersEl = document.getElementById('answers');
answersEl.replaceChildren(); Now we can loop over all the answers in the metadata object and render them to the page. // render the answers for this question
metadata.answers.forEach((a, i) => {
    // create an answer container
    const answerEl = document.createElement('li');
    answerEl.classList.add('list-group-item');

    // radio button to select the answer
    const answerRadio = document.createElement('input');
    answerRadio.setAttribute('name', 'answer-input');
    answerRadio.type = 'radio';
    answerRadio.id = `answer-input-${i}`;
    answerRadio.classList.add('me-2', 'form-check-input');
    answerRadio.dataset.index = i;
    answerEl.appendChild(answerRadio);

    // label to display the answer text
    const answerLbl = document.createElement('label');
    answerLbl.setAttribute('for', `answer-input-${i}`);
    answerLbl.innerHTML = a;
    answerEl.appendChild(answerLbl);

    answersEl.appendChild(answerEl);
}); Checking for the Correct Answer Now our application will display the question and answers on the page. We'll need a function that can check whether the user's answer is correct that will provide user feedback, so let's create that now. const checkAnswer = () => {
    // disable the submit btn
    document.getElementById('check-answer-btn').setAttribute('disabled', 'disabled');

    // check the current answer
    const selectedAnswer = document.querySelector('input[name="answer-input"]:checked');
    const selectedIdx = parseInt(selectedAnswer.dataset.index);

    // highlight the correct answer
    document.querySelector(`input[data-index="${window.correctIndex}"]`).nextSibling.classList.add('text-success');

    // if they're wrong, highlight the incorrect answer
    if (selectedIdx !== window.correctIndex) {
        selectedAnswer.nextSibling.classList.add('text-danger');
    }
} Don't forget to attach an event listener for the event on our button. We'll add that in our function. click Submit init() document.getElementById('check-answer-btn').addEventListener('click', checkAnswer); And we're ready to see it in action! If you're playing along at home, check your code against the code in the demo below. If not, check it out anyway to see the final solution. Give it a shot and see how many questions you can get right! Note, you may have to scroll down in the demo to see the question. Producing Timed Metadata We've looked at how to consume metadata, but we've not yet seen how to produce it. No worries - it's not that hard to do! There are several ways to produce timed metadata, and all of them require the . You can get this value from the Amazon IVS Management Console in the channel details, or via the AWS CLI. Channel ARN $ aws \
    ivs \
    list-channels \
    --filter-by-name [YOUR CHANNEL NAME] \
    --query "channels[0].arn" \
    --output text Check out the . Need to Install the CLI? install guide With the AWS CLI One way to produce timed metadata is to use the AWS CLI. I don't know if this is a method that you'd use very often, but for testing and debugging purposes, it's good to know how to do it. $ aws \
    ivs \
    put-metadata \
    --channel-arn [YOUR CHANNEL ARN] \
    --metadata '{"test": true}' . AWS CLI Docs With the Amazon IVS SDK Node SDK import AWS from 'aws-sdk';
const Ivs = new AWS.IVS({ region: 'us-east-1', credentials });

const putMetadata = async (metadata) => {
    const input = {
        channelArn: '[YOUR CHANNEL ARN]',
        metadata: JSON.stringify(metadata)
    };
    let output;
    try {
        output = await Ivs.putMetadata(input).promise();
    }
    catch (e) {
        console.error(e);
        if (e.name === 'ChannelNotBroadcasting') {
            output = { offline: true };
        }
        else {
            throw new Error(e);
        }
    }
    return output;
} . Node SDK docs Python SDK import boto3
client = boto3.client('ivs')

response = client.put_metadata(
    channelArn='[YOUR CHANNEL ARN]',
    metadata='{"python": true}'
)
print(response) . Python SDK Docs Other Languages Check the for the syntax to publish metadata events in your favorite language. SDK specific documentation Summary Timed metadata is infinitely cool and has many applications to make our live stream engaging and interactive. In our next post, we’re going to look at another way we can add interactivity to our live streams - Amazon IVS Chat! Also Published Here

Using Amazon IVS to Create Interactive Live Streaming Experiences with Timed Metadata

Uso de Amazon IVS para crear experiencias interactivas de transmisión en vivo con metadatos cronometrados

With the use of blockchain explorer or by retrieving it from the smart contracts that manage them, you may locate the metadata for your NFTs. is data that provides information about other data; in the case of an NFT, it describes that NFT's essential properties, including its name, description, and anything else its creator feels is important. In many cases, an NFT's metadata also contains links to the images and other "primary" digital assets that give an NFT its value. Metadata (opens new window) Individually identifiable digital representations of blockchain-based tangible or intangible objects called are not typically divided into smaller parts. Instead, structured metadata related to actual or virtual things is represented by NFTs. Often, the tokens are not connected to the objects and serve as distinct identifiers, advancing the interoperable commercialization of the physical or digital items, according to their proponents. nonfungible tokens (NFTs) Hashes of files or other data were added to in 2012 or 2013 to demonstrate their existence or legitimacy at a particular moment. This development served as the foundation for creating "Colored Coins," uniquely identified tokens, which are recognized by adding metadata to Bitcoin transactions. the Bitcoin blockchain What is Metadata? So, what is this metadata, and how is it generated? In this article, we will explore these questions and also deep dive into how you can retrieve metadata of the NFT from the contract; how you read NFT files; and how you can view the metadata of any NFT. What exactly is the location of NFTs that are stored off-chain? Is it anything like Amazon Web Services or Google Drive? This is a crucial point that causes confusion surrounding NFT information. Who else is in charge of the NFT metadata storage online? Since NFT marketplaces use metadata to show nonfungible tokens to buyers and sellers, metadata must be in a format that marketplaces can understand. In other words, you're writing computer code—a smart contract—that instructs the blockchain to initialise a table on your behalf. Your NFTs' ownership and metadata are kept in this table. More particular, each table row has the following details: The Token Identifier (or ID) The Owner of the Token The Metadata associated with the token How can you view the metadata of any NFT? When an NFT project is made available for the minting and you buy it, a random number generator will give your NFT a set of characteristics. The metadata contains this information. The blockchain contains an immutable record of the token's info. This record contains information about the token's purpose, present owner, and previous transactional activity. Consequently, how can you create NFT metadata? Viewing an NFT's metadata, confirming its ownership and tracing its transaction history are all possible using an NFT tracking and verification service through which you can verify that the token ID and contract address are connected to the legitimate owner of the content using a marketplace database. If you select a service that complies with NFT's coding standard, one of the many NFT verification tools available can search for you. An Ethereum blockchain explorer called is a well-known tool for locating and confirming NFTs. The platform allows users to verify transaction histories, wallet addresses, metadata, smart contracts and other on-chain data. Similarly, you can access all the NFT metadata you require on the BNB Chain network by using a blockchain explorer like BscScan. Etherscan However, to retrieve the metadata of a nonfungible token from the smart contracts that control it, you'll need to access the NFT's smart contract. The metadata should be accessible for inspection and verification under the contract's "Details" section. If the NFT complies with the current nonfungible token standard, the following details ought to be shown: Here is a JSON metadata file for a sample NFT. ash-nft.json{ "description": "Friendly OpenSea Creature", "image": "https://opensea-prod.appspot.com/puffs/3.png", "name": "Dave Starbelly", "attributes": [ { "trait_type": "Base", "value": "Starfish" }, { "trait_type": "Eyes", "value": "Big" }, { "trait_type": "Mouth","value": "Surprised" }, ] } The second essential component of NFTs that lets them work is NFT metadata. NFTs have the ability to link to data outside of their smart contract, effectively enabling the network to refer to off-chain data. As a result, executing NFTs on a network like expenses than it otherwise would. Ethereum has lower computing The asset itself often resides off-chain, whereas the Non-Fungible Token that defines the provenance of an asset resides on the blockchain. There are a few exceptions; one is the collection OnChain Monkeys, which was totally built on chain in a single transaction. Since the complete collection is hosted on-chain, no file storage solution is required.

Easy Way to Find Your NFT’s Metadata

This article introduces the design and implementation of metadata storage in Alluxio Master, either on heap and off heap (based on RocksDB). Background Alluxio is the world’s first open source data orchestration platform for analytics and AI for the cloud. Alluxio bridges the gap between data driven applications and storage systems. It brings data from the storage tier closer to the data driven applications and makes it easily accessible enabling applications to connect to numerous storage systems through a common interface. Alluxio’s architecture enables data access at speeds orders of magnitude faster than existing solutions. As a distributed system, Alluxio employs a Master-Worker architecture. An Alluxio cluster consists of one or more Master nodes and several Worker nodes. The Alluxio Master serves all user requests and journals file system metadata changes. Alluxio workers are responsible for managing user-configurable local resources allocated to Alluxio (e.g. memory, SSDs, HDDs). Alluxio workers store data as blocks and serve client requests that read or write data by reading or creating new blocks within their local resources. This article introduces how Alluxio implements metadata storage in Alluxio Master. Two Ways of Storing Metadata in Alluxio Inside Alluxio master, each file or a directory is represented by a data structure called inode which contains attributes like file permissions, and other metadata like block locations. Alluxio master stores the metadata of the entire file system as an inode tree, similar to HDFS and other UNIX-based file systems. Alluxio provides two ways to store the metadata: ROCKS: an on-disk, RocksDB-based metastore HEAP: an on-heap metastore The default is the ROCKS. The Alluxio source code provides two interfaces, InodeStore and BlockStore, in the /core/server/master/src/main/java/alluxio/master/metastore directory. The metadata managed by InodeStore includes individual inodes and the parent-child relationship between different inodes. BlockStore is responsible for managing the block size and block location of file data blocks. The cachingInodeStore implements an on java heap configurable cache in front of RocksDB to improve performance. In HEAP, these interfaces are implemented through HeapInodeStore and HeapBlockStore. The overall dependency relationships are as follows. During the AlluxioMasterProcess, two Factories are generated to corresponding stores, and different InodeStore and BlockStore are generated in the Factories based on the configurations. ROCKS Metastore RocksDB is an embedded Key-Value database. Users can achieve efficient KV data storage and access by calling the API interface. The BlockStore interface is implemented by RocksBlockStore, with the process as follows. Factory is added to the mContext in AlluxioMasterProcess. MasterUtils.createMasters() will create all the Master threads in order. When creating DefaultBlockMaster, it will call BlockStore.Factory to create a RocksBlockStore instance. RocksDB.loadLibrary() will be called when initiating RocksBlockStore to load the dependent libraries and then create an instance of RocksStore class according to the configuration file. RocksStore is used to operate the RocksDB database, including initializing the database, backing up and restoring the database, etc. When creating RocksStore, an instance of RocksDB class is created by calling RocksDB.open() method in the create() method. DefaultBlockMaster reads and writes to the RocksDB database by using this RocksDB class. RocksBlockStore uses the following main methods: getBlock(long id) putBlock(long id, BlockMeta meta) removeBlock(long id) getLocations(long id) addLocation(long id, BlockLocation location) removeLocation(long blockId, long workerId) Let’s take getBlock() as an example. getBlock() is used to get the metadata of the corresponding block by blockId as follows. @Override
  public Optional<BlockMeta> getBlock(long id) {
    byte[] meta;
    try {
  meta = db().get(mBlockMetaColumn.get(), Longs.toByteArray(id));
    } catch (RocksDBException e) {
      throw new RuntimeException(e);
    }
    if (meta == null) {
      return Optional.empty();
 }
    try {
      return Optional.of(BlockMeta.parseFrom(meta));
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  } Below is to interact with RocksDB: meta = db().get(mBlockMetaColumn.get(), Longs.toByteArray(id)); The db() method in the above code will return the previously created RocksDB class and then call the RocksDB.get() method. get() method needs two parameters, the first one is the ColumnFamilyHandle and the second is blockId. Each K-V pair in RocksDB corresponds to a ColumnFamily, and ColumnFamily is equivalent to the logical partition in RocksDB. When we need to query the data in a ColumnFamily, we need to operate the underlying database through ColumnFamilyHandle, which is created while creating RocksDB instance. The K-V data stored in RocksDB are stored as byte strings, so we need to convert the blockId to byte[], and convert the value returned from RocksDB back into a BlockMeta Java object using Google Protocol buffers. The InodeStore interface in ROCKS method is implemented by CachingInodeStore and RocksInodeStore. The CachingInodeStore uses memory to store the metadata cache, while the RocksInodeStore is a metastore implemented by RocksDB as the backing store of CachingInodeStore. When the metadata of the Alluxio cluster can be completely stored in the CachingInodeStore, Alluxio does not interact with the RocksInodeStore, but uses CachingInodeStore to get better performance instead. When the storage capacity of the CachingInodeStore reaches a threshold, Alluxio automatically migrates the metadata from the CachingInodeStore to the RocksInodeStore. At this point, the performance of metadata access depends on the cache hit rate of CachingInodeStore and the performance of RocksDB. The process of creating and using RocksInodeStore is similar to RocksBlockStore. If MASTER_METASTORE_INODE_CACHE_MAX_SIZE is set to 0, then it uses RocksInodeStore. if it is not 0, then it requires creating both CachingInodeStore and RocksInodeStore. case ROCKS:
InstancedConfiguration conf = ServerConfiguration.global();
  if (conf.getInt(PropertyKey.MASTER_METASTORE_INODE_CACHE_MAX_SIZE) == 0) {
    return lockManager -> new RocksInodeStore(baseDir);
  } else {
    return lockManager -> new CachingInodeStore(new RocksInodeStore(baseDir), lockManager);
  } Heap Metastore Alluxio uses heap memory as storage in the Heap method. When creating the AlluxioMasterProcess, the HeapInodeStore and HeapBlockStore are built to implement the InodeStore and BlockStore interfaces. When creating the HeapInodeStore, a ConcurrentHashMap called mInodes is created to store the Inode information of files and folders. At the same time, a TwoKeyConcurrentMap called mEdges will be created to store the parent-child relationship of different nodes. private final Map<Long, MutableInode<?>> mInodes = new ConcurrentHashMap<>();
  // Map from inode id to ids of children of that inode. The inner maps are ordered by child name.
  private final TwoKeyConcurrentMap<Long, String, Long, Map<String, Long>> mEdges =
      new TwoKeyConcurrentMap<>(() -> new ConcurrentHashMap<>(4)); TwoKeyConcurrentMap is a class defined in Alluxio, which implements a ConcurrentMap supporting two keys with the following logical structure: <k1, <k2, value>>. Each key in the mEdges data structure maps an a Inode ID to a map of its children, which then maps each child’s name to its Inode ID. In HeapBlockStore, a ConcurrentHashMap called mBlocks is created to store the metadata of each block, and a TwoKeyConcurrentMap called mBlockLocations is created to store the location of the block in the worker. // Map from block id to block metadata.
  public final Map<Long, BlockMeta> mBlocks = new ConcurrentHashMap<>();
  // Map from block id to block locations.
  public final TwoKeyConcurrentMap<Long, Long, BlockLocation, Map<Long, BlockLocation>>
      mBlockLocations = new TwoKeyConcurrentMap<>(() -> new HashMap<>(4)); The two keys in mBlockLocations are blockId and workerId, and the value is the specific locations on the worker where the block is stored. The blockId can be used to get the location of the block stored in the worker. Summary Alluxio provides two ways to store metadata, ROCKS and HEAP, and the default storage is the ROCKS. For ROCKS, in addition to using RocksDB, Alluxio provides a memory-based cache to improve metadata read and write performance. As a result, we can get high performance when the size of the metastore is limited. With RocksDB, we can store metadata to the hard disk to get more storage space. In general, we should use the RocksDB to store metadata in Alluxio Master. If you only need to store a small amount of metadata which requires very high metadata read and write performance, you can also consider using the HEAP method. About the Author Changsheng Gu, Big Data Engineer at China Mobile. Gu works in China Mobile Cloud Center, developing cloud-native data lake with a focus on HDFS and Alluxio.

Deep Dive into the Implementation of Metadata Storage

We cannot expect doctors to remember every detail about each patient throughout their careers and diagnose illnesses instantly as it is not conceivable. However, if the doctor has instant access to all of the data, he will be able to provide personalized care for all of his patients and eliminate the need for new tests. Healthcare data must be appropriately classified and safeguarded to enable easy data access. A solid Metadata management solution is required to achieve these goals. Healthcare metadata management ensures that the data is easily integrated, shared, linked, analyzed, and maintained to foster more innovation in the healthcare industry. Whether you are a doctor, nurse, or CEO, metadata management solutions can significantly impact patients' lives. How can metadata management make a difference in the healthcare industry? Whenever someone utters the word data, our mind usually jumps to numbers that can improve marketing strategies, attract customers, and boost revenue. However, have you ever thought of data as a lifesaver? According to , 3,03,000 women die during labor and birth each year, 2.7 million babies die within the first one month, and 2.6 million babies are stillborn. If we can pinpoint where, when, and to whom these incidents occur, it will be much easier to conduct research and improve maternal health. WHO Healthcare metadata management will do the trick! It helps you search and find relevant information across unstructured data like X-rays, tissue samples, and other images. Metadata management solution gives a clear understanding of every data in the hospital industry. Healthcare sectors are trying to bring in more innovation and improve the success rate of clinical trials. With a robust metadata management solution, doctors can quickly identify the factors contributing to a trial's success or failure by tracking down the data to its root. It enables you to understand where and when various factors such as medications, recommended therapy, or food habits contributed to the trial's outcome, reducing the likelihood of the same error occurring twice. Manually determining the reason will take a significant amount of resources and time from your business intelligence team. An AI-powered metadata management solution can help you with quick and accurate conclusions while enabling transparency between teams. Let us take a look at the healthcare metadata management solution use cases. Understand your data's lineage Everyone's primary concern in the aftermath of the pandemic is healthcare. Many patients are being admitted and discharged from the hospital, resulting in a massive data volume. A robust metadata management solution will assist the business intelligence team in identifying the data source, detecting errors in the report, and correcting them at an earlier stage, much to the delight of the healthcare industry. For example, suppose anybody accuses your hospital of any mishaps. In that case, you can quickly identify the source of the problem and who is responsible for it without panicking and losing millions of dollars as a penalty with a healthcare metadata management solution. It records who last accessed the data, when, where, and who changed it. It enables you to track the data as it evolves over time and quickly identifies any problems in the hospital. With the right solution, you can get accurate answers and deal with day-to-day challenges in minutes. Find your data instantly Everyone wants to access relevant data with the swipe of a finger. However, it's like trying to find a needle in a haystack without its metadata. A sound healthcare metadata management system will assist you in making the most of your data collection. With automated data discovery, you can find your data in seconds. Quick data discovery allows doctors to prescribe more effective treatments and make clinical decisions that are more accurate and timely. It is a mandate in the healthcare industry to comply with regulatory requirements. To achieve compliance, the data team will need to put in a lot of effort during auditing to determine the data source. With a metadata management solution in place, you can quickly create a visual representation of data that will assist you in identifying each data set and its source. You can save your data team a lot of time searching for numbers, allowing them to focus more on analysis. Enable collaboration across your organization Data flow from various sources, including pharmacies, patients, wearables, billing and finance, and human resources, are labeled differently. The issue with this is that it results in data redundancy, which wastes resources in managing and analyzing the data. The metadata management solution creates a clearly defined glossary term for each information set in the organization, allowing for data collaboration across departments to address this issue. If business terminologies are not correctly named and defined, it will lead to confusion and mistakes. A hospital business glossary provides a common understanding of every term in the hospital industry, improving the quality of care and patient safety. Patients in the hospital sector are routinely referred to specialists for accurate diagnosis. The only issue is that vital information gets lost in the process. With a defined business glossary, metadata in the healthcare industry aims to solve this issue and enable effective collaboration of patient data across various healthcare organizations to use the patient history in emergencies, thereby avoiding catastrophes. Drive more significant insights and transform the healthcare industry with a metadata management solution The healthcare industries need to make rapid and intelligent decisions based on precise patient data. Between 20% and 25% of US healthcare spending is squandered. That's about a trillion dollars, and according to study, better data utilization could eliminate 50 to 75% of that waste. Automated metadata management is your go-to solution to achieve this instantly. McKinsey's At Xtract.io, we help the healthcare industry save money, prevent corruption, innovate, enable fast diagnosis and provide exemplary patient care with our robust . metadata management solution

Three ways Metadata Management is Transforming the Healthcare Industry

By: David Zhu Metadata synchronization (sync) is a core feature in Alluxio that keeps files and directories consistent with their source of truth in under storage systems, thus making it simple for users to retrieve the latest version of data from Alluxio. Meanwhile, understanding the internal process is important in order to tune the performance. This article describes the design and the implementation in Alluxio to keep metadata synchronized. Why is Metadata Sync Critical in Alluxio In Alluxio, metadata refers to the information of files and directories in the Alluxio file system, including information of their owners, groups, permission, creation and modification time, etc. Metadata is independent of their content — even if a file or directory is empty, it still has associated metadata. Alluxio maintains a copy of the file system or the object store namespace of the under storage system. Metadata consistency is important in Alluxio, especially when different clusters write/read data in the data pipeline and make changes outside of Alluxio. A typical scenario in the above diagram is a data pipeline with the combination of Spark ETL and Presto SQL. The ETL cluster (without Alluxio) writes data, followed by the analytics cluster with Alluxio reading the transformed data. Since Alluxio maintains a copy of the metadata from under storage and manages the metadata, when data changes in under storage with the ETL step, the instance of Alluxio on the analytics cluster must be made aware and kept consistent with the metadata in the underlying storage system to proceed correctly. How Does Metadata Sync Work in Alluxio Alluxio provides a file system abstraction in a unified namespace on one or more under storage systems. Accessing files or directories through Alluxio is expected to get the same result as directly accessing the under storage. For example, if the under storage mounted to Alluxio root is , listing the directory in Alluxio yields the same result as listing objects in and printing in Alluxio should return the same result as . s3://bucket/data "/" s3://bucket/data "/file" s3://bucket/data/file In Alluxio, metadata is only stored and served from Alluxio masters, whereas the content of individual files is served from Alluxio workers. By default, Alluxio loads the metadata from the under storage on demand. In the above example, an Alluxio master starting from empty does not have any information about after startup. This file is only recognized when some user lists directory in Alluxio or tries to access . This “lazy” behavior prevents unnecessary effort and significantly improves performance because metadata operations in under storage can be slow. s3://bucket/data/file "/" "/file" Note that updating metadata can be bidirectional. If all modifications to the file system occur through Alluxio, Alluxio only needs to scan under storage once to retrieve the initial state, then apply changes in both Alluxio and under storage synchronously as a part of the file system RPC calls. This will provide users with a consistent view of under storage. In reality, however, changes are often made to the under storage outside of Alluxio. As a result, Alluxio master must discover the addition, removal, and updates to files and directions in under storage, and apply the changes in the Alluxio file system. This process to synchronize the two namespaces is called metadata sync. How to Trigger Metadata Sync When applications make changes to the metadata of an Alluxio file, and this file is persisted, the change will always be propagated to under storage synchronously, and there is no need to trigger metadata sync. When applications update under storage files without letting Alluxio know, there are two approaches to govern the timing of metadata sync. 1. Rely on a time-based automatic sync One can set a sync Interval as an Alluxio property key “ ”. alluxio.user.file.metadata.sync.interval When the value is -1 (current default value), Alluxio will never resync with under storage after initial load. When its value is set to 0, Alluxio will always resync with under storage whenever metadata is accessed. When the value is positive (default unit is ms), Alluxio will (best effort) not re-sync a path within that time interval. Note that using this approach, if a path in Alluxio has never been accessed, it will never trigger a sync. Once the path is accessed after the interval of sync expires, Alluxio will sync with the under storage again. For example, in a Presto job, the query planning stage lists all the files necessary for the job, which will trigger a sync if those paths have not been accessed recently. However, subsequent stages of the job will not sync unless the job duration exceeds the sync interval. So we can technically resync more frequently than the sync interval in this case. This property key can be customized with a new global default value (when set in ) or on a directory basis recursively to apply all its children. alluxio-site.properties 2. Sync manually using LoadMetadata flags If sync metadata does not happen because of the sync interval, most Alluxio operations will continue to execute with the current metadata in the Alluxio file system with some exceptions: For most of the users, Alluxio CLI “ ” is the easiest way to trigger sync manually. For example, one can run “ ” to force update the metadata of Alluxio path “ “. loadMetadata bin/alluxio fs loadMetadata /path/to/sync /path/to/sync For applications built on Alluxio file system (Java), there are two API methods and which retrieve the metadata for a path or a directory. When calling these methods, there is an additional field of in the option for each call, which may trigger a “loadMetadata” process by the master on the Alluxio path being inquired. This process can be considered as a simplified version of the sync by only loading file metadata from the under storage but not modifying a file’s metadata if it is already in the Alluxio. if is set to , then nothing will be loaded, and if the file does not already exist, a FileNotFound exception will be thrown. When is , we will only load the metadata once for each directory. This affects these two file system calls only, and this option is only considered when sync does not happen. SDK getStatus listStatus LoadMetadataPType LoadMetadataPType NEVER LoadMetadataPType ONCE How is Metadata Sync Implemented Alluxio master potentially triggers metadata sync on an Alluxio path when the Alluxio master receives an RPC request to retrieve the metadata of this path. Rather than having a dedicated service to traverse the entire file system inode tree and keep it in sync, this effort is amortized by each individual Alluxio file system operation on master. The high-level process for the syncing inside the RPC request is: Given the Alluxio path, determine if it is not consistent with the corresponding under storage path.  This means the under storage path does not exist, or has metadata that differs from Alluxio. This part is done using the RPC thread. The steps from 1 should populate the sync queue, and we iterate through the sync queue and process each path in a worker thread from a separate thread pool. The order of traversal is a BFS order, because additional paths are added at the end of the queue. Concurrency and executors will be discussed in the parallelism section in more detail. This part is carried out by the sync threads, and under storage, information is read using the under storage prefetch thread. The reason for this is to overlap communication with computation. Sync threads need to manipulate inode tree, and under storage, prefetch can start as soon as we determine that we need the information at some point in the future. Prefetching threads load the under storage status information into the under storage status cache, which is discussed in the cache section. Note that the metadata sync process can be relatively expensive and block other operations if they touch the same part of the inode tree. This is because the sync process may write-lock the part of file system metadata it is updating. Particularly, when syncing a particular path in a tree, the RPC handling thread will first acquire the read lock on the entire path to the file. Because the syncing thread needs the ability to create paths as well, it needs a write lock for the sync root path. As the syncing thread processes each path under the root path, additional locks are acquired. A write lock to the file path is acquired by the syncing thread and released as soon as the path is processed. Performance Optimization Tuning parallelism One can tune the level of parallelism to sync metadata by controlling three configuration parameters: This indicates the number of individual files to sync concurrently within a single metadata sync request (e.g., on a directory). alluxio.master.metadata.sync.concurrency.level This indicates the number of concurrent threads for all sync operations alluxio.master.metadata.sync.executor.pool.size This indicates the number of concurrent threads that can perform under storage prefetch operations for all sync operations alluxio.master.metadata.sync.ufs.prefetch.pool.size Caching result There are three types of different caches, with different goals and uses in the metadata syncing process. Here is a quick summary of all of them. is a negative cache, used to avoid checking the under storage for those paths that have been known to not exist. It uses prefix matching to determine if a path is in the under storage or not. For example, if the path is in the absent cache, we know must not exist on the under storage either. In addition, Absent Cache entries have a timestamp attached. So that we know the time it was last checked in the under storage. This is useful when the sync interval is some time period, we use the timestamp to determine whether we need to recheck the under storage for the existence of the file or directory. AbsentCache /a/b /a/b/c is a cache used to prefetch under storage statuses during the sync process. Instead of fetching the path information when needed, we can often prefetch some file statuses while we process the current directory. UfsStatusCache is a positive cache and contains paths that have been synced with the under storage recently. When we receive a metadata operation, we will check with this cache to determine whether we need to sync a particular path. UfsSyncPathCache Summary Metadata synchronization is one of the most important features in Alluxio. There are multiple different ways to trigger synchronization with different performance tradeoffs. Inside Alluxio master, there is a list of optimizations applied to speed up the synchronization. More Info Documentation of Alluxio Unified Namespace Two Ways to Keep Files in Sync Between Alluxio and HDFS Also Published Here

Metadata Synchronization: Design, Implementation and Optimization

Integration with blockchain still remains a mystery for many engineers, even for those who might be familiar with it as end users. While smart contract development often attracts most of the attention and coverage in technical write-ups, developers are often left to their own devices when it comes to figuring out how to integrate blockchain technology with the rest of their application. The Casper Ecosystem thankfully offers solutions that eliminate  problems around getting started and deploying live dApps on Casper in one complementary product suite, making it one of the most developer-friendly blockchain ecosystems. Developed by MAKE and granted by Casper Association, CSPR.click, CSPR.cloud and CSPR.build address user interactions with the blockchain and provide tools to read and observe the network activity in real time. ** ** CSPR.click elegantly solves the end user authentication problem by welcoming users with a familiar-looking Single-Sign-On experience where they can connect to any Casper-enabled application using their preferred wallet. Users who are entirely new to the blockchain can enjoy a smooth onboarding experience by using self-custodial wallets linked to social logins, their email address or cell phone number. When it comes to being able to pay for contract calls, integration with multiple on-ramps guarantees that all users will be able to obtain tokens quickly and easily, regardless of their geographical region. Developers can quick-start their CSPR.click integration, and complete customization is also possible using the lower-level SDK. Mobile Casper Wallet is automatically supported, making your Casper-enabled application truly cross-platform out of the gate. Most developers, regardless of their prior experience with blockchain integrations, can bootstrap a functional front-end application with CSPR.click in a matter of minutes. REST APIs, WebSockets and Node RPC with CSPR.cloud Applications, including those integrated using CSPR.click, can rely on CSPR.cloud to observe on-chain activity on the Casper Network. CSPR.cloud is an enterprise-grade middleware platform that can be used separately from CSPR.click, and equips developers with highly available and scalable REST APIs that provide low-latency access to normalized and indexed blockchain data, fungible and non-fungible token activity, historical pricing data, and related off-chain metadata. Real-time WebSocket streams make it possible to granularly subscribe to the data you are interested in, allowing your application to react immediately whenever your smart contracts emit an event or when an on-chain entity you observe changes its state. Both the REST API and the Real-time WebSocket streams rely entirely on widely used internet standards, making it exceedingly easy to integrate with existing and new applications without the need to know or learn any new technologies. This unlocks Casper blockchain development to the tens of millions of software engineers at businesses and enterprises who’ve never  interacted with a blockchain before.  And for those applications that need direct access to the native Casper Node RPC API, CSPR.cloud provides a cache-enabled, high-performance and highly-available managed node infrastructure for maximally performant access without the associated overhead. Both CSPR.click and CSPR.cloud come with comprehensive documentation and examples. Developers can start building on the Casper network with CSPR.click and CSPR.cloud by registering on CSPR.build and generating their access keys for instant access. Casper is Solving Real World Problems For Builders In most environments, to be able to interact with smart contracts, end users must possess blockchain accounts with sufficient token balances to cover smart contract execution costs. Getting users through the process of creating and managing private and public keys, keeping their recovery phrases safe, as well as funding their accounts with tokens, already represents a serious UX challenge. And this corresponds to a significant bounce rate of new users, and lower retention of existing ones. For developers, the underlying technical integration with multiple crypto wallets and fiat on-ramps adds another layer of complexity, demanding considerable initial engineering effort and time, as well as ongoing maintenance and up-keep with ever- and rapidly evolving standards, regulatory constraints, APIs and SDKs of third party wallet and on-ramp providers. CSPR.click solves all this for application developers. Casper also helps developers address that smart contracts are executed on-chain asynchronously, i.e. the end user submits a transaction to the network, the transaction gets processed when a new block is created, and that results in a change of state. Applications must constantly monitor network activity to properly respond to the contract logic and any state changes it may produce that are relevant to their overall application. This requires not only an in-depth understanding of how the blockchain works but also a reliable and scalable infrastructure. While theoretically, the application can connect to any public node, in practice, the reliability of these nodes remains uncertain. A serious application or business will certainly not want to rely on just any public node, without assured redundancy, maintenance, uptime and disaster recovery processes in place, and without dedicated domain experts to ensure their application always runs smoothly. The suite of building tools in CSPR.build, CSPR.click and CSPR.cloud ensures developers do not have to worry about the integration process, and can exclusively focus on their creativity and core business. Build with Casper by registering on CSPR.build and generate your access keys for instant access. This story was distributed as a release by Btcwire under HackerNoon’s Business Blogging Program. Learn more about the program here. Integration with blockchain still remains a mystery for many engineers, even for those who might be familiar with it as end users. While smart contract development often attracts most of the attention and coverage in technical write-ups, developers are often left to their own devices when it comes to figuring out how to integrate blockchain technology with the rest of their application. The Casper Ecosystem thankfully offers solutions that eliminate  problems around getting started and deploying live dApps on Casper in one complementary product suite, making it one of the most developer-friendly blockchain ecosystems. Developed by MAKE and granted by Casper Association, CSPR.click, CSPR.cloud and CSPR.build address user interactions with the blockchain and provide tools to read and observe the network activity in real time. MAKE MAKE ** ** CSPR.click elegantly solves the end user authentication problem by welcoming users with a familiar-looking Single-Sign-On experience where they can connect to any Casper-enabled application using their preferred wallet. CSPR.click CSPR.click Users who are entirely new to the blockchain can enjoy a smooth onboarding experience by using self-custodial wallets linked to social logins, their email address or cell phone number. When it comes to being able to pay for contract calls, integration with multiple on-ramps guarantees that all users will be able to obtain tokens quickly and easily, regardless of their geographical region. Developers can quick-start their CSPR.click integration, and complete customization is also possible using the lower-level SDK . Mobile Casper Wallet is automatically supported, making your Casper-enabled application truly cross-platform out of the gate. Most developers, regardless of their prior experience with blockchain integrations, can bootstrap a functional front-end application with CSPR.click in a matter of minutes. quick-start quick-start lower-level SDK lower-level SDK REST APIs, WebSockets and Node RPC with CSPR.cloud Applications, including those integrated using CSPR.click, can rely on CSPR.cloud to observe on-chain activity on the Casper Network. CSPR.cloud is an enterprise-grade middleware platform that can be used separately from CSPR.click, and equips developers with highly available and scalable REST APIs that provide low-latency access to normalized and indexed blockchain data, fungible and non-fungible token activity, historical pricing data, and related off-chain metadata. CSPR.cloud CSPR.cloud REST APIs REST APIs Real-time WebSocket streams make it possible to granularly subscribe to the data you are interested in, allowing your application to react immediately whenever your smart contracts emit an event or when an on-chain entity you observe changes its state. Both the REST API and the Real-time WebSocket streams rely entirely on widely used internet standards, making it exceedingly easy to integrate with existing and new applications without the need to know or learn any new technologies. Real-time WebSocket streams Real-time WebSocket streams This unlocks Casper blockchain development to the tens of millions of software engineers at businesses and enterprises who’ve never  interacted with a blockchain before.  And for those applications that need direct access to the native Casper Node RPC API, CSPR.cloud provides a cache-enabled, high-performance and highly-available managed node infrastructure for maximally performant access without the associated overhead. node infrastructure node infrastructure Both CSPR.click and CSPR.cloud come with comprehensive documentation and examples. Developers can start building on the Casper network with CSPR.click and CSPR.cloud by registering on CSPR.build and generating their access keys for instant access. registering on CSPR.build registering on CSPR.build Casper is Solving Real World Problems For Builders In most environments, to be able to interact with smart contracts, end users must possess blockchain accounts with sufficient token balances to cover smart contract execution costs. Getting users through the process of creating and managing private and public keys, keeping their recovery phrases safe, as well as funding their accounts with tokens, already represents a serious UX challenge. And this corresponds to a significant bounce rate of new users, and lower retention of existing ones. For developers, the underlying technical integration with multiple crypto wallets and fiat on-ramps adds another layer of complexity, demanding considerable initial engineering effort and time, as well as ongoing maintenance and up-keep with ever- and rapidly evolving standards, regulatory constraints, APIs and SDKs of third party wallet and on-ramp providers. CSPR.click solves all this for application developers. Casper also helps developers address that smart contracts are executed on-chain asynchronously, i.e. the end user submits a transaction to the network, the transaction gets processed when a new block is created, and that results in a change of state. Applications must constantly monitor network activity to properly respond to the contract logic and any state changes it may produce that are relevant to their overall application. This requires not only an in-depth understanding of how the blockchain works but also a reliable and scalable infrastructure. While theoretically, the application can connect to any public node, in practice, the reliability of these nodes remains uncertain. A serious application or business will certainly not want to rely on just any public node, without assured redundancy, maintenance, uptime and disaster recovery processes in place, and without dedicated domain experts to ensure their application always runs smoothly. The suite of building tools in CSPR.build, CSPR.click and CSPR.cloud ensures developers do not have to worry about the integration process, and can exclusively focus on their creativity and core business. Build with Casper by registering on CSPR.build and generate your access keys for instant access. registering on CSPR.build registering on CSPR.build This story was distributed as a release by Btcwire under HackerNoon’s Business Blogging Program. Learn more about the program here. This story was distributed as a release by Btcwire under HackerNoon’s Business Blogging Program. Learn more about the program here . here here

Casper Blockchain Lightning Fast dApp Deployment With The CSPR.build Suite

A Comprehensive Guide to Building DolphinScheduler 3.2.0 Production-Grade Cluster Deployment

I've always liked GUI, both desktop-based and browser-based, before you needed five years of training on the latter. That's the reason I loved and still love Vaadin: you can develop web UIs without writing a single line of HTML, JavaScript, and CSS. I'm still interested in the subject; a couple of years ago, I analyzed the state of JVM desktop frameworks. I also like the Rust programming language a lot. Tauri is a Rust-based framework for building desktop applications. Here's my view. Overview Build an optimized, secure, and frontend-independent application for multi-platform deployment.
-- Tauri website A Tauri app is composed of two modules: the client-side module in standard Web technologies (HTML, Javascript, and CSS) and the backend module in Rust. Tauri runs the UI in a dedicated Chrome browser instance. Users interact with the UI as usual. Tauri offers a binding between the client-side JavaScript and the backend Rust via a specific JS module, i.e, window.__TAURI__.tauri. It also offers other modules for interacting with the local system, such as the filesystem, OS, clipboard, window management, etc. Binding is based on strings. Here's the client-side code: const { invoke } = window.__TAURI__.tauri;

let greetInputEl;
let greetMsgEl;

greetMsgEl.textContent = await invoke("greet", { name: greetInputEl.value });  //1 Invoke the Tauri command named greet Here's the corresponding Rust code: #[tauri::command]                                                              //1
fn greet(name: &str) -> String {                                               //1
    format!("Hello, {}! You've been greeted from Rust!", name)
} Define a Tauri command named greet In the following sections, I'll list Tauri's good, meh, and bad points. Remember that it's my subjective opinion based on my previous experiences. The good Getting started:
Fortunately, it is becoming increasingly rare, but some technologies need to remember that before you're an expert, you're a newbie. The first section of any site should be a quick explanation of the technology, and the second a getting started. Tauri succeeds in this; I got my first Tauri app in a matter of minutes by following the Quick start guide.


Documentation:
Tauri's documentation is comprehensive, extensive (as far as my musings browsed them), and well-structured.


Great feedback loop:
I've experienced exciting technologies where the feedback loop, the time it takes to see the results of a change, makes the technology unusable. GWT, I'm looking at you. Short feedback loops contribute to a great developer experience.
In this regard, Tauri scores points. One can launch the app with a simple cargo tauri dev command. If the front end changes, Tauri reloads it. If any metadata changes, e.g., anything stored in tauri.conf.json, Tauri restarts the app. The only downside is that both behaviors lose the UI state.


Complete lifecycle management:
Tauri not only helps you develop your app but also provides the tools to debug, test, build, and distribute it. The meh At first, I wanted to create my usual showcase for desktop applications, a file renamer app. However, I soon hit an issue when I wanted to select a directory using the file browser button. First, Tauri doesn't allow the use of the regular JavaScript file-related API; Instead, it provides a more limited API. Worse, you need to explicitly configure which file system paths are available at build time, and they are part of an enumeration. I understand that security is a growing concern in modern software. Yet, I fail to understand this limitation on a desktop app, where every other app can access any directory. The bad However, Tauri's biggest problem is its design, more precisely, its separation between the front and the back end. What I love in Vaadin is its management of all things frontend, leaving you to learn the framework only. It allows your backend developers to build web apps without dealing with HTML, CSS, and JavaScript. Tauri, though a desktop framework, made precisely the opposite choice. Your developers will need to know frontend technologies. Worse, the separation reproduces the request-response model created using browser technologies to create UIs. Reminder: early desktop apps use the Observer model, which better fits user interactions. We designed apps around the request-response model only after we ported them on the web. Using this model in a desktop app is a regression, in my opinion. Conclusion Tauri has many things to like, mainly everything that revolves around the developer experience. If you or your organization uses and likes web technologies, try Tauri. However, it's a no-go for me: to create a simple desktop app, I don't want to learn how to center a div or about the flexbox layout, etc. To go further: Tauri I've always liked GUI, both desktop-based and browser-based, before you needed five years of training on the latter. That's the reason I loved and still love Vaadin : you can develop web UIs without writing a single line of HTML, JavaScript, and CSS. I'm still interested in the subject; a couple of years ago, I analyzed the state of JVM desktop frameworks . Vaadin state of JVM desktop frameworks I also like the Rust programming language a lot. Tauri is a Rust-based framework for building desktop applications. Here's my view. Overview Build an optimized, secure, and frontend-independent application for multi-platform deployment. -- Tauri website Build an optimized, secure, and frontend-independent application for multi-platform deployment. -- Tauri website Tauri website A Tauri app is composed of two modules: the client-side module in standard Web technologies (HTML, Javascript, and CSS) and the backend module in Rust. Tauri runs the UI in a dedicated Chrome browser instance. Users interact with the UI as usual. Tauri offers a binding between the client-side JavaScript and the backend Rust via a specific JS module, i.e , window.__TAURI__.tauri . It also offers other modules for interacting with the local system, such as the filesystem, OS, clipboard, window management, etc. i.e window.__TAURI__.tauri Binding is based on strings. Here's the client-side code: const { invoke } = window.__TAURI__.tauri;

let greetInputEl;
let greetMsgEl;

greetMsgEl.textContent = await invoke("greet", { name: greetInputEl.value });  //1 const { invoke } = window.__TAURI__.tauri;

let greetInputEl;
let greetMsgEl;

greetMsgEl.textContent = await invoke("greet", { name: greetInputEl.value });  //1 Invoke the Tauri command named greet Invoke the Tauri command named greet greet Here's the corresponding Rust code: #[tauri::command]                                                              //1
fn greet(name: &str) -> String {                                               //1
    format!("Hello, {}! You've been greeted from Rust!", name)
} #[tauri::command]                                                              //1
fn greet(name: &str) -> String {                                               //1
    format!("Hello, {}! You've been greeted from Rust!", name)
} Define a Tauri command named greet Define a Tauri command named greet greet In the following sections, I'll list Tauri's good, meh, and bad points. Remember that it's my subjective opinion based on my previous experiences. The good Getting started:
Fortunately, it is becoming increasingly rare, but some technologies need to remember that before you're an expert, you're a newbie. The first section of any site should be a quick explanation of the technology, and the second a getting started. Tauri succeeds in this; I got my first Tauri app in a matter of minutes by following the Quick start guide. Documentation:
Tauri's documentation is comprehensive, extensive (as far as my musings browsed them), and well-structured. Great feedback loop:
I've experienced exciting technologies where the feedback loop, the time it takes to see the results of a change, makes the technology unusable. GWT, I'm looking at you. Short feedback loops contribute to a great developer experience.
In this regard, Tauri scores points. One can launch the app with a simple cargo tauri dev command. If the front end changes, Tauri reloads it. If any metadata changes, e.g., anything stored in tauri.conf.json, Tauri restarts the app. The only downside is that both behaviors lose the UI state. Complete lifecycle management:
Tauri not only helps you develop your app but also provides the tools to debug, test, build, and distribute it. Getting started: Fortunately, it is becoming increasingly rare, but some technologies need to remember that before you're an expert, you're a newbie. The first section of any site should be a quick explanation of the technology, and the second a getting started. Tauri succeeds in this; I got my first Tauri app in a matter of minutes by following the Quick start guide. Getting started: Fortunately, it is becoming increasingly rare, but some technologies need to remember that before you're an expert, you're a newbie. The first section of any site should be a quick explanation of the technology, and the second a getting started. Tauri succeeds in this; I got my first Tauri app in a matter of minutes by following the Quick start guide. Quick start Documentation: Tauri's documentation is comprehensive, extensive (as far as my musings browsed them), and well-structured. Documentation: Tauri's documentation is comprehensive, extensive (as far as my musings browsed them), and well-structured. Great feedback loop: I've experienced exciting technologies where the feedback loop, the time it takes to see the results of a change, makes the technology unusable. GWT, I'm looking at you. Short feedback loops contribute to a great developer experience. In this regard, Tauri scores points. One can launch the app with a simple cargo tauri dev command. If the front end changes, Tauri reloads it. If any metadata changes, e.g., anything stored in tauri.conf.json, Tauri restarts the app. The only downside is that both behaviors lose the UI state. Great feedback loop: I've experienced exciting technologies where the feedback loop, the time it takes to see the results of a change, makes the technology unusable. GWT , I'm looking at you. Short feedback loops contribute to a great developer experience. GWT In this regard, Tauri scores points. One can launch the app with a simple cargo tauri dev command. If the front end changes, Tauri reloads it. If any metadata changes, e.g. , anything stored in tauri.conf.json , Tauri restarts the app. The only downside is that both behaviors lose the UI state. cargo tauri dev e.g. tauri.conf.json Complete lifecycle management: Tauri not only helps you develop your app but also provides the tools to debug, test, build, and distribute it. Complete lifecycle management: Tauri not only helps you develop your app but also provides the tools to debug , test , build , and distribute it. debug test build distribute The meh At first, I wanted to create my usual showcase for desktop applications, a file renamer app . However, I soon hit an issue when I wanted to select a directory using the file browser button. First, Tauri doesn't allow the use of the regular JavaScript file-related API; Instead, it provides a more limited API. Worse, you need to explicitly configure which file system paths are available at build time, and they are part of an enumeration. file renamer app I understand that security is a growing concern in modern software. Yet, I fail to understand this limitation on a desktop app, where every other app can access any directory. The bad However, Tauri's biggest problem is its design, more precisely, its separation between the front and the back end. What I love in Vaadin is its management of all things frontend, leaving you to learn the framework only. It allows your backend developers to build web apps without dealing with HTML, CSS, and JavaScript. Tauri, though a desktop framework, made precisely the opposite choice. Your developers will need to know frontend technologies. Worse, the separation reproduces the request-response model created using browser technologies to create UIs. Reminder: early desktop apps use the Observer model , which better fits user interactions. We designed apps around the request-response model only after we ported them on the web. Using this model in a desktop app is a regression, in my opinion. Observer model Conclusion Tauri has many things to like, mainly everything that revolves around the developer experience. If you or your organization uses and likes web technologies, try Tauri. However, it's a no-go for me: to create a simple desktop app, I don't want to learn how to center a div or about the flexbox layout, etc. div To go further: To go further: Tauri Tauri Tauri

Why I Love Vaadin but Tauri Isn't for Me

Load Balancing Strategies for Applications: From Infrastructure to Code

The Data Revolution: AI Takes the Wheel

Ordinals. The Origin Ordinals became a significant game changer in the NFT segment in 2023. Since their introduction to the Bitcoin blockchain, the daily inscriptions have shown a consistent increase. As of now, there have been a total of 66,348,042 inscriptions. Source: Dune Ordinals introduced an innovative approach to creating NFTs, meme tokens, and other digital assets on the Bitcoin blockchain. However, it caused lots of discussions within the community. Ordinals utilize inscriptions to store content directly on the blockchain, leading to skepticism among many users about its overall benefit to the ecosystem. Concerns were raised that the block space occupied by these NFTs could potentially increase fees and reduce transaction speeds. Over time, it became clear that Ordinals have become a crucial component of the Bitcoin blockchain, laying the foundation for their integration into decentralized exchanges, lending protocols, and yield farming platforms. Michael Saylor, the founder of MicroStrategy, stated, "Bitcoin-NFT technology has the potential to foster innovation in the digital asset market." He also mentioned that the creation of applications and further development of the blockchain would accelerate its mass adoption. As observed, the number of application inscriptions grew significantly in 2024. Source: Dune Taking that into account, it was only a matter of time before Ordinals and immersive augmented reality (AR) merged. The Jade98 platform introduces the first-ever AR/VR inscriptions (Jade ARdinals), set to redefine the standards of digital collectibles. It was created to enhance efficiency, transparency, and unique user engagement in the DeFi segment. With the Jade98 platform, you can not only upgrade your existing BRC collections but also craft ARdinal collections right from scratch. Let's explore the groundbreaking Jade98 platform, which allows its users to create Jade ARdinals and earn rewards for their activity. About JD98 token To understand the whole Jade98 Platform ecosystem, it’s important to learn the main functionality of its basic element. The JD98 token is the cornerstone of the entire ecosystem. It facilitates payments for core functionalities, such as using AR technologies, on the platform. Creators also receive staking rewards in $JD98 tokens. Its primary utility is to support commissions and transactions throughout the platform, serving as its vital lifeblood. Epoch-Based Calculations System: Enhancing Control and Simplicity At the heart of the Jade98 platform lies the concept of epochs — fixed intervals of time 1999998 sec. At the end of each epoch, significant events occur, such as stakers receiving their JD98 rewards or the burning of the revenue treasury. Check out the detailed scheme of the platform below: These are the main tokenomics processes that maintain the functionality and financial payback of the system. Unlike the tokenomics of many other DeFi platforms, all processes on the Jade98 Platform are fully transparent, with no hidden pitfalls. Now, let's review each of these processes individually. Platform Service Revenue: Monetizing AR Services Central to Jade's platform revenue model is the provision of augmented reality (AR) services. You have the freedom to craft an exciting ARdinal collection from scratch, complete with all metadata. Overall, you get the full package: create your AR, mint it on the blockchain, and showcase your unique digital masterpiece. For these innovative opportunities, users pay a service fee in BTC upon initiating the mint transactions (similar to Luminex/Unisat inscription service). This revenue is collected in the platform’s treasury for subsequent distribution through various mechanisms: rewards pool for stakers, Buyback and Burning mechanism, and other processes that ensure sustained growth and profitability of the Jade98 platform. Buyback and Burning Mechanism Another important feature of the Jade98 platform is the Buyback and Burning mechanism, wherein 20% of incoming revenue is allocated towards purchasing $JD98 tokens from the market. The process is depicted in detail in the upper part of the whole Jade98 platform tokenomics scheme. As you can see, the Service Revenue users pay for ARdinals creation, which is then converted to JD98 for further burning and then accumulated in the Burning Treasury to be systematically burned at the end of each epoch. The system sends them a Bitcoin Eater address, thereby enhancing the token value. Staking: Empowering Token Holders Stakers on the platform can enjoy multiple avenues for staking, enabling them to extract tokens from circulation and lock them within the platform in exchange for a share of profits generated during the epoch. As you can see from another part of the tokenomics scheme, there are several ways of staking available. Single-asset Staking (JD98 token) In this case, you'll receive your rewards at the end of the epoch. Multi-asset Staking (JD98 + ARdinals) For the ​​Jade Pair Staking, you’ll need to get an ARdinal from Jade ARdinals Collection. This will increase your rewards by staking a special Jade ARdinal with JD98$. The payment amounts and, accordingly, the APY indicator are affected by: Number of stacked $JD98
Type of staking you chose
Rarity of staked ARdinals
Number of days in the epoch your assets were staked
Number of epochs before this position was already staked
Number of referrals a user has and how much they have staked Initially, Simple Staking will be developed and launched using a simplified formula. Over time, it will be enhanced by refining the formula and adding additional assets for boosting. Staking Rewards Distribution: Ensuring Fairness and Efficiency At the end of each epoch, rewards from the previous epoch are calculated and disbursed from the Reward Pool in $JD98 tokens. Thus, new rewards are sent to users every three to five weeks. Regular revenues enable you to redistribute your income correctly and reinvest a portion for additional earnings. Users have the option to receive payouts directly to their addresses or initiate transactions to claim their rewards, with the latter option to minimize transaction fees. Additionally, accrued rewards incur compound interest if left unclaimed. Conclusion At Jade98 Platform, we strive to develop an effective and transparent tokenomics system that will not only enhance AR/VR functionality but also contribute to our users' success and increase their income. The groundbreaking integration of AR services with Ordinals will boost the incorporation of the Bitcoin blockchain into the DeFi sector and its mass adoption worldwide. This represents a perfect new direction for immersive, real-world applications to develop and a powerful means to foster a sustainable digital economy. Follow our X(Twitter) to get more information about upcoming events. Join Telegram chat for discussions. Ordinals. The Origin Ordinals became a significant game changer in the NFT segment in 2023. Since their introduction to the Bitcoin blockchain, the daily inscriptions have shown a consistent increase. As of now, there have been a total of 66,348,042 inscriptions. Source: Dune Source: Dune Dune Dune Ordinals introduced an innovative approach to creating NFTs, meme tokens, and other digital assets on the Bitcoin blockchain. However, it caused lots of discussions within the community. Ordinals utilize inscriptions to store content directly on the blockchain, leading to skepticism among many users about its overall benefit to the ecosystem. Concerns were raised that the block space occupied by these NFTs could potentially increase fees and reduce transaction speeds. Over time, it became clear that Ordinals have become a crucial component of the Bitcoin blockchain, laying the foundation for their integration into decentralized exchanges, lending protocols, and yield farming platforms. Michael Saylor, the founder of MicroStrategy, stated, " Bitcoin-NFT technology has the potential to foster innovation in the digital asset market. " He also mentioned that the creation of applications and further development of the blockchain would accelerate its mass adoption. As observed, the number of application inscriptions grew significantly in 2024. Bitcoin-NFT technology has the potential to foster innovation in the digital asset market. Source: Dune Source: Dune Dune Dune Taking that into account, it was only a matter of time before Ordinals and immersive augmented reality (AR) merged. The Jade98 platform introduces the first-ever AR/VR inscriptions (Jade ARdinals), set to redefine the standards of digital collectibles. It was created to enhance efficiency, transparency, and unique user engagement in the DeFi segment. With the Jade98 platform, you can not only upgrade your existing BRC collections but also craft ARdinal collections right from scratch. Let's explore the groundbreaking Jade98 platform, which allows its users to create Jade ARdinals and earn rewards for their activity. About JD98 token To understand the whole Jade98 Platform ecosystem, it’s important to learn the main functionality of its basic element. The JD98 token is the cornerstone of the entire ecosystem. It facilitates payments for core functionalities, such as using AR technologies, on the platform. Creators also receive staking rewards in $JD98 tokens. Its primary utility is to support commissions and transactions throughout the platform, serving as its vital lifeblood. Epoch-Based Calculations System: Enhancing Control and Simplicity At the heart of the Jade98 platform lies the concept of epochs — fixed intervals of time 1999998 sec. At the end of each epoch, significant events occur, such as stakers receiving their JD98 rewards or the burning of the revenue treasury. Check out the detailed scheme of the platform below: These are the main tokenomics processes that maintain the functionality and financial payback of the system. Unlike the tokenomics of many other DeFi platforms, all processes on the Jade98 Platform are fully transparent, with no hidden pitfalls. Now, let's review each of these processes individually. Platform Service Revenue: Monetizing AR Services Central to Jade's platform revenue model is the provision of augmented reality (AR) services. You have the freedom to craft an exciting ARdinal collection from scratch, complete with all metadata. Overall, you get the full package: create your AR, mint it on the blockchain, and showcase your unique digital masterpiece. For these innovative opportunities, users pay a service fee in BTC upon initiating the mint transactions (similar to Luminex/Unisat inscription service). This revenue is collected in the platform’s treasury for subsequent distribution through various mechanisms: rewards pool for stakers, Buyback and Burning mechanism, and other processes that ensure sustained growth and profitability of the Jade98 platform. Buyback and Burning Mechanism Another important feature of the Jade98 platform is the Buyback and Burning mechanism, wherein 20% of incoming revenue is allocated towards purchasing $JD98 tokens from the market. The process is depicted in detail in the upper part of the whole Jade98 platform tokenomics scheme. As you can see, the Service Revenue users pay for ARdinals creation, which is then converted to JD98 for further burning and then accumulated in the Burning Treasury to be systematically burned at the end of each epoch. The system sends them a Bitcoin Eater address, thereby enhancing the token value. Staking: Empowering Token Holders Stakers on the platform can enjoy multiple avenues for staking, enabling them to extract tokens from circulation and lock them within the platform in exchange for a share of profits generated during the epoch. As you can see from another part of the tokenomics scheme, there are several ways of staking available. Single-asset Staking (JD98 token) In this case, you'll receive your rewards at the end of the epoch. Multi-asset Staking (JD98 + ARdinals) For the ​​Jade Pair Staking, you’ll need to get an ARdinal from Jade ARdinals Collection. This will increase your rewards by staking a special Jade ARdinal with JD98$. The payment amounts and, accordingly, the APY indicator are affected by: Number of stacked $JD98 Type of staking you chose Rarity of staked ARdinals Number of days in the epoch your assets were staked Number of epochs before this position was already staked Number of referrals a user has and how much they have staked Number of stacked $JD98 Type of staking you chose Rarity of staked ARdinals Number of days in the epoch your assets were staked Number of epochs before this position was already staked Number of referrals a user has and how much they have staked Initially, Simple Staking will be developed and launched using a simplified formula. Over time, it will be enhanced by refining the formula and adding additional assets for boosting. Staking Rewards Distribution: Ensuring Fairness and Efficiency At the end of each epoch, rewards from the previous epoch are calculated and disbursed from the Reward Pool in $JD98 tokens. Thus, new rewards are sent to users every three to five weeks. Regular revenues enable you to redistribute your income correctly and reinvest a portion for additional earnings. Users have the option to receive payouts directly to their addresses or initiate transactions to claim their rewards, with the latter option to minimize transaction fees. Additionally, accrued rewards incur compound interest if left unclaimed. Conclusion At Jade98 Platform, we strive to develop an effective and transparent tokenomics system that will not only enhance AR/VR functionality but also contribute to our users' success and increase their income. The groundbreaking integration of AR services with Ordinals will boost the incorporation of the Bitcoin blockchain into the DeFi sector and its mass adoption worldwide. This represents a perfect new direction for immersive, real-world applications to develop and a powerful means to foster a sustainable digital economy. Follow our X(Twitter) to get more information about upcoming events. X(Twitter) X(Twitter) Join Telegram chat for discussions. Telegram chat Telegram chat

Defining the Bitcoin DeFi Future: Jade98 Platform's Transparent Tokenomics

বিটকয়েন ডিফাই ভবিষ্যত সংজ্ঞায়িত করা: Jade98 প্ল্যাটফর্মের স্বচ্ছ টোকেনমিক্স

Die Bitcoin-DeFi-Zukunft definieren: Transparente Tokenomics der Jade98-Plattform

Definiendo el futuro de Bitcoin DeFi: la tokenómica transparente de la plataforma Jade98

Définir l'avenir du Bitcoin DeFi : les tokenomiques transparentes de la plateforme Jade98

बिटकॉइन डीफाई भविष्य को परिभाषित करना: जेड98 प्लेटफॉर्म का पारदर्शी टोकनॉमिक्स

ビットコイン DeFi の未来を定義する: Jade98 プラットフォームの透明なトークノミクス

비트코인 DeFi 미래 정의: Jade98 플랫폼의 투명한 토큰경제학

Definindo o futuro do Bitcoin DeFi: Tokenomics transparente da plataforma Jade98

Определение будущего Bitcoin DeFi: прозрачная токеномика платформы Jade98

Bitcoin DeFi Geleceğini Tanımlamak: Jade98 Platformunun Şeffaf Tokenomikleri

Xác định tương lai Bitcoin DeFi: Tokenomics minh bạch của nền tảng Jade98

定义比特币 DeFi 的未来：Jade98 平台的透明代币经济学

Hacking Backward and Forward-Compatible Application Releases

How to Streamline CI/CD With Octopus and TeamCity

অক্টোপাস এবং টিমসিটির সাথে কীভাবে সিআই/সিডি স্ট্রিমলাইন করবেন

So optimieren Sie CI/CD mit Octopus und TeamCity

Cómo optimizar CI/CD con Octopus y TeamCity

Comment rationaliser le CI/CD avec Octopus et TeamCity

ऑक्टोपस और टीमसिटी के साथ CI/CD को कैसे सुव्यवस्थित करें

Octopus と TeamCity で CI/CD を効率化する方法

Octopus와 TeamCity를 사용하여 CI/CD를 간소화하는 방법

Como simplificar CI/CD com Octopus e TeamCity

Как оптимизировать CI/CD с помощью Octopus и TeamCity

Octopus ve TeamCity ile CI/CD Nasıl Kolaylaştırılır

Cách hợp lý hóa CI/CD với Octopus và TeamCity

如何使用 Octopus 和 TeamCity 简化 CI/CD

How Cryptocurrency APIs Drive Growth in the Blockchain Industry

The rationale behind this project stems from the various stability issues encountered with Harbor, other registries, prompting a transition away from it. Project Plan Set up ECR infrastructure using Terraform, This includes private repos creation. provider "aws" 
{region = "us-east-1" # Change to your AWS region# 
Optionally, specify a profile name if you're using AWS named profiles
# profile = "your-profile-name"}resource "aws_ecr_repository" "my_ecr_repo" {name = "my-ecr-repo" # Change this to your desired repo nameimage_tag_mutability = "MUTABLE"image_scanning_configuration {scan_on_push = true}
# If you need to create more repositories, you can either duplicate this block# and change the name, or use a more dynamic approach with `count` or `for_each`.# Optional: Enable lifecycle policy to manage image count or untagged imageslifecycle_policy {policy = jsonencode({rules = [{rulePriority = 1description = "Expire untagged images"selection = {==tagStatus = "untagged"countType = "imageCountMoreThan"countNumber = 100}action = {type = "expire"} }, ] })}} 2. Implement replication rules for cross-region and cross-account functionality. mutable-repos = ["apps/signatures",]
ecr_replication_destinations_rules = 
{lab1 = {region = "us-west-2"id = "<>"}
lab2 = {region = "us-east-1"id = "<>"}
stage1 = {region = "ap-southeast-1"id = "<>"}
stage2 = {region = "us-east-1"id = "<>"}
prod = {region = "us-west-2"id = "<>"}} 3. Once the Infrastructure is available, adapt application pipelines to push images and Helm charts to ECR along with Harbor. Activities To be Performed Prepare User “ecr_role” with necessary below permissions, facilitating automated image/chart pushes to ECR in the CI/CD section. Established ECR private repositories for all applications, replicating the setup across remaining environments to manage replication costs.
Updated and executed pipelines for all application components to ensure visibility of images in both locations (Harbor and ECR).
Provided implementation guidelines to the Application team, detailing necessary changes and their locations.
Transitioned to ECR from Harbor across all environments (Lab, Stage, Production). Moving Forward with Helm Our idea is to maintain the whole CD through ArgoCD. So for that, we need to add this as a repository in Argo. A new Challenge emerges with OCI calls in Helm implementation as it will not use HTTPs calls. Resolution Install External secrets in the cluster where we have Argo helm repo add external-secrets https://charts.external-secrets.iohelm install external-secrets \external-secrets/external-secrets \-n external-secrets \--create-namespace \--set installCRDs=true 2. Create AWS credentials in K8 secrets, or if you are using gitlab for pulling the data to argo, then you can configure the same under export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLEexport AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYecho -n AWS_SECRET_ACCESS_KEY > ./secret-access-keyecho -n AWS_ACCESS_KEY_ID > ./access-key kubectl create secret generic awssm-secret --from-file=./access-key --from-file=./secret-access-key 3. The secret created in previous step, swasm-secret used as below apiVersion: generators.external-secrets.io/v1alpha1kind: ECRAuthorizationTokenmetadata: name: ecr-genspec: # specify aws region (mandatory) region: us-west-1 auth:   # 1: static credentials   # point to a secret that contains static credentials   # like AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY   secretRef:     accessKeyIDSecretRef:        name: "awssm-secret"    key: "access-key" secretAccessKeySecretRef: name: "awssm-secret" key: "secret-access-key" Challenges The predominant challenge involved handling Helm charts from ArgoCD when storing an OCI Helm repo in ECR, necessitating token rotation using “External Secrets Operator”. Achievements: Achieved zero downtime across all environments despite numerous redeployments and adherence to application team timelines. Progress made on the migration from Harbor to ECR, with no disruptions in operations. If this post was helpful, please do follow and click the clap 👏 button below to show your support. Follow me on Linked-in for further updates- https://www.linkedin.com/in/chaitanya-kanth-tummalachervu-296956121/ The rationale behind this project stems from the various stability issues encountered with Harbor, other registries, prompting a transition away from it. Project Plan Project Plan Set up ECR infrastructure using Terraform, This includes private repos creation. Set up ECR infrastructure using Terraform, This includes private repos creation. Set up ECR infrastructure using Terraform, This includes private repos creation. provider "aws" 
{region = "us-east-1" # Change to your AWS region# 
Optionally, specify a profile name if you're using AWS named profiles
# profile = "your-profile-name"}resource "aws_ecr_repository" "my_ecr_repo" {name = "my-ecr-repo" # Change this to your desired repo nameimage_tag_mutability = "MUTABLE"image_scanning_configuration {scan_on_push = true}
# If you need to create more repositories, you can either duplicate this block# and change the name, or use a more dynamic approach with `count` or `for_each`.# Optional: Enable lifecycle policy to manage image count or untagged imageslifecycle_policy {policy = jsonencode({rules = [{rulePriority = 1description = "Expire untagged images"selection = {==tagStatus = "untagged"countType = "imageCountMoreThan"countNumber = 100}action = {type = "expire"} }, ] })}} provider "aws" 
{region = "us-east-1" # Change to your AWS region# 
Optionally, specify a profile name if you're using AWS named profiles
# profile = "your-profile-name"}resource "aws_ecr_repository" "my_ecr_repo" {name = "my-ecr-repo" # Change this to your desired repo nameimage_tag_mutability = "MUTABLE"image_scanning_configuration {scan_on_push = true}
# If you need to create more repositories, you can either duplicate this block# and change the name, or use a more dynamic approach with `count` or `for_each`.# Optional: Enable lifecycle policy to manage image count or untagged imageslifecycle_policy {policy = jsonencode({rules = [{rulePriority = 1description = "Expire untagged images"selection = {==tagStatus = "untagged"countType = "imageCountMoreThan"countNumber = 100}action = {type = "expire"} }, ] })}} 2. Implement replication rules for cross-region and cross-account functionality. mutable-repos = ["apps/signatures",]
ecr_replication_destinations_rules = 
{lab1 = {region = "us-west-2"id = "<>"}
lab2 = {region = "us-east-1"id = "<>"}
stage1 = {region = "ap-southeast-1"id = "<>"}
stage2 = {region = "us-east-1"id = "<>"}
prod = {region = "us-west-2"id = "<>"}} mutable-repos = ["apps/signatures",]
ecr_replication_destinations_rules = 
{lab1 = {region = "us-west-2"id = "<>"}
lab2 = {region = "us-east-1"id = "<>"}
stage1 = {region = "ap-southeast-1"id = "<>"}
stage2 = {region = "us-east-1"id = "<>"}
prod = {region = "us-west-2"id = "<>"}} 3. Once the Infrastructure is available, adapt application pipelines to push images and Helm charts to ECR along with Harbor. Activities To be Performed Activities To be Performed Prepare User “ecr_role” with necessary below permissions, facilitating automated image/chart pushes to ECR in the CI/CD section. Prepare User “ecr_role” with necessary below permissions, facilitating automated image/chart pushes to ECR in the CI/CD section. Established ECR private repositories for all applications, replicating the setup across remaining environments to manage replication costs. Updated and executed pipelines for all application components to ensure visibility of images in both locations (Harbor and ECR). Provided implementation guidelines to the Application team, detailing necessary changes and their locations. Transitioned to ECR from Harbor across all environments (Lab, Stage, Production). Established ECR private repositories for all applications, replicating the setup across remaining environments to manage replication costs. Updated and executed pipelines for all application components to ensure visibility of images in both locations (Harbor and ECR). Provided implementation guidelines to the Application team, detailing necessary changes and their locations. Transitioned to ECR from Harbor across all environments (Lab, Stage, Production). Moving Forward with Helm Our idea is to maintain the whole CD through ArgoCD. So for that, we need to add this as a repository in Argo. A new Challenge emerges with OCI calls in Helm implementation as it will not use HTTPs calls. Resolution Install External secrets in the cluster where we have Argo helm repo add external-secrets https://charts.external-secrets.iohelm install external-secrets \external-secrets/external-secrets \-n external-secrets \--create-namespace \--set installCRDs=true Install External secrets in the cluster where we have Argo helm repo add external-secrets https://charts.external-secrets.iohelm install external-secrets \external-secrets/external-secrets \-n external-secrets \--create-namespace \--set installCRDs=true Install External secrets in the cluster where we have Argo helm repo add external-secrets https://charts.external-secrets.io helm install external-secrets \ external-secrets/external-secrets \ -n external-secrets \ --create-namespace \ --set installCRDs=true helm repo add external-secrets https://charts.external-secrets.io helm install external-secrets \ external-secrets/external-secrets \ -n external-secrets \ --create-namespace \ --set installCRDs=true 2. Create AWS credentials in K8 secrets, or if you are using gitlab for pulling the data to argo, then you can configure the same under export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY echo -n AWS_SECRET_ACCESS_KEY > ./secret-access-key echo -n AWS_ACCESS_KEY_ID > ./access-key kubectl create secret generic awssm-secret --from-file=./access-key --from-file=./secret-access-key export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY echo -n AWS_SECRET_ACCESS_KEY > ./secret-access-key echo -n AWS_ACCESS_KEY_ID > ./access-key kubectl create secret generic awssm-secret --from-file=./access-key --from-file=./secret-access-key 3. The secret created in previous step, swasm-secret used as below apiVersion: generators.external-secrets.io/v1alpha1 kind: ECRAuthorizationToken metadata: name: ecr-gen spec: # specify aws region (mandatory) region: us-west-1 auth: # 1: static credentials # point to a secret that contains static credentials # like AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY secretRef: accessKeyIDSecretRef: name: "awssm-secret" key: "access-key" secretAccessKeySecretRef: name: "awssm-secret" key: "secret-access-key" kind: ECRAuthorizationToken metadata: name: ecr-gen spec: # specify aws region (mandatory) region: us-west-1 auth: # 1: static credentials # point to a secret that contains static credentials # like AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY secretRef: accessKeyIDSecretRef: name: "awssm-secret" key: "access-key" secretAccessKeySecretRef: name: "awssm-secret" key: "secret-access-key" Challenges The predominant challenge involved handling Helm charts from ArgoCD when storing an OCI Helm repo in ECR, necessitating token rotation using “External Secrets Operator”. Achievements: Achieved zero downtime across all environments despite numerous redeployments and adherence to application team timelines. Progress made on the migration from Harbor to ECR, with no disruptions in operations. If this post was helpful, please do follow and click the clap 👏 button below to show your support. If this post was helpful, please do follow and click the clap 👏 button below to show your support. Follow me on Linked-in for further updates- https://www.linkedin.com/in/chaitanya-kanth-tummalachervu-296956121/ https://www.linkedin.com/in/chaitanya-kanth-tummalachervu-296956121/

Migrating Container Images and Helm Charts from Harbor to ECR

Stories, News & Mentions about MERUDATA

Merudata’s Stories on HackerNoon