pieFi

piefi

When working with text in JavaScript, you may come across situations where you need to add a prefix or suffix to each new line in a given text. In this tutorial, we'll walk you through a simple and effective way to achieve this using HTML, CSS, and JavaScript. We'll create an easy-to-use online tool that adds a specified prefix and suffix to each new line of text entered by the user. Step 1: Create the HTML Structure First, let's create the basic for our tool. This includes a text area for inputting the text, two input fields for the prefix and suffix, a button to trigger the , and an output text area to display the processed text. HTML structure JavaScript function <!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Add Prefix and Suffix</title>
</head>
<body>
    <h1>Add Prefix and Suffix to Each New Line</h1>
    <textarea id="inputText" rows="10" cols="50" placeholder="Enter your text here..."></textarea>
    <br>
    <label for="prefix">Prefix:</label>
    <input type="text" id="prefix" />
    <br>
    <label for="suffix">Suffix:</label>
    <input type="text" id="suffix" />
    <br>
    <button onclick="addPrefixAndSuffix()">Add Prefix and Suffix</button>
    <br>
    <h2>Output:</h2>
    <textarea id="outputText" rows="10" cols="50" readonly></textarea>
</body>
</html> Step 2: Add CSS for Styling Although not required, you may want to add some basic the tool. Here's a simple example: CSS to style <style>
    body {
        font-family: Arial, sans-serif;
        max-width: 800px;
        margin: 0 auto;
        padding: 20px;
    }

    textarea, input {
        display: block;
        width: 100%;
        margin-bottom: 10px;
    }

    button {
        background-color: #4CAF50;
        color: white;
        padding: 8px 16px;
        border: none;
        cursor: pointer;
        text-decoration: none;
    }

    button:hover {
        background-color: #45a049;
    }
</style> Add the above CSS code inside the tag of the HTML code to apply the styles. <head> Step 3: Write the JavaScript Function Now, let's write the JavaScript function that will add the specified prefix and suffix to each new line of the input text. Be sure to include this code within tags at the end of your HTML file, just before the closing tag. <script> </body> function addPrefixAndSuffix() {
    const inputText = document.getElementById('inputText').value;
    const prefix = document.getElementById('prefix').value;
    const suffix = document.getElementById('suffix').value;
    const lines = inputText.split('\n');

    const outputLines = lines.map(line => {
        if (line.trim() === '') {
            return line;
        }
        return prefix + line + suffix;
    });

    const outputText = outputLines.join('\n');
    document.getElementById('outputText').value = outputText;
} This function retrieves the input text, prefix, and suffix from the HTML elements. It then splits the input of lines, processes each line by adding the prefix and suffix (skipping empty or whitespace-only lines), and finally combines the processed lines back into a single string. text into an array Step 4: Test the Tool To test the , save the HTML, CSS, and JavaScript code in a new file and open it in your web browser. You should see a simple interface with input fields for the text, prefix, and suffix, as well as a button to trigger the function. tool .html Enter your text in the text area, input your desired prefix and suffix, and click the "Add Prefix and Suffix" button. The processed text with the added prefix and suffix should appear in the output text area. Conclusion In this tutorial, we have demonstrated how to create a user-friendly online tool that adds a prefix and suffix to each new line of text using HTML, CSS, and JavaScript. This tool can be helpful for tasks such as formatting lists, adding , or enhancing the appearance of your text. code snippets With this simple yet effective solution, you can save time and effort when working with text in JavaScript.

How to Add Prefix or Suffix to Each New Line in JavaScript

Task description: A (pronounced as “try”) or is a tree data structure used to efficiently store and retrieve keys in a dataset of strings. There are various applications of this data structure, such as autocomplete and spellchecker. trie prefix tree Implement the Trie class: Initializes the trie object. Trie() Inserts the string into the trie. void insert(String word) word Returns if the string is in the trie (i.e., was inserted before), and otherwise. boolean search(String word) true word false Returns if there is a previously inserted string that has the prefix , and otherwise. boolean startsWith(String prefix) true word prefix false Example 1: Input
["Trie", "insert", "search", "search", "startsWith", "insert", "search"]
[[], ["apple"], ["apple"], ["app"], ["app"], ["app"], ["app"]]
Output
[null, null, true, false, true, null, true] Explanation
Trie trie = new Trie();
trie.insert("apple");
trie.search("apple");   // return True
trie.search("app");     // return False
trie.startsWith("app"); // return True
trie.insert("app");
trie.search("app");     // return True Constraints: 1 <= word.length, prefix.length <= 2000 and consist only of lowercase English letters. word prefix At most calls will be made to , , and . 3 * 10^4 in total insert search startsWith Reasoning: is the classic data structure that is widely used. One of the most famous uses is text searching. Let’s take Google.com as an example. Trie Google’s example Have you ever noticed whenever you start typing a search phrase to a search bar you almost immediately get some suggestion from the searching system? I bet the answer is “Yes”. The Trie data structure makes it possible. Let’s add some words to Trie and see how it looks like this: Example of Trie with some words The Trie above contains words: Algorithm, Animation, Ant, Apache, Append, Apple, Application, Apricot. As you may notice we treat every word as a sequence of characters. Starting from the first character of each word we attach it to the root of the and move down till we reach the last character of each word. Trie Once we reached the end of a word we mark that last as the end of the word. I hope now it’s clear to you how it works in general. If you still struggle with it, I recommend you to read . Node Wikipedia’s article Solution: We are to implement Trie with the following method: https://gist.github.com/RakhmedovRS/a53ea510ad97088fdfd4829209d405b2 As we saw in the example of real we had to notice that as with any other data structure it consists of smaller pieces which are usually called . Let’s introduce this small part: Trie Tree Node https://gist.github.com/RakhmedovRS/bb2a188cd156565b692af1f5262d96a3 A few things we need to have a closer look at here: We have the boolean array which will tell us if we have particular characters in a particular node. boolean variable which answers just one question, whether or not the current node is the end of a word in our Trie. Last but not least TrieNode array stores child TrieNodes of the current TrieNode. chars isEnd children Let’s look at other methods, one by one: Method returns nothing and accepts word to insert into the Trie: void insert(String word) https://gist.github.com/RakhmedovRS/b460db4fd040a47ca33e8afc401af7c0 Starting from the root of the Trie we iterate through every char in provided word and create a new TrieNode every time if we don’t have one. At the last character in word, we mark TrieNode as isEnd. The next method in our list is which searches for a particular word in the Trie and returns a boolean value depending on whether or not the Trie contains it. As input, it takes a word. boolean search(String word) https://gist.github.com/RakhmedovRS/18bb68c4e83f3c602afa4a338e96fbc2 Here we do the same as in the previous method except for a few things. Iterate through every character in provided word and if we don’t have a child with the next character we return false. At the end of a word, we return the current node value of . isEnd Lastly method takes prefix and returns true if we have a prefix in our Trie if not we return false. boolean startsWith(String prefix) https://gist.github.com/RakhmedovRS/d25842ef37c339e6260ef50097713e84 This method does completely the same as the previous one except for only one thing. If we managed to iterate through the whole prefix we return true, because we’re looking for the prefix and not an entire word, otherwise we return false. As always at the end of the article, I prove the full source code of the problem https://gist.github.com/RakhmedovRS/cff7b0ea83d0a598c86c7b9aac834425 It has linear time and space complexity and has the following position among other people's solutions. Also published . here

How to Implement Trie (Prefix Tree) - Blind 75 LeetCode Questions

Cómo implementar Trie (Árbol de prefijos) - Preguntas ciegas de 75 LeetCode

In this post, we will look at prefix sums and how they can be used to solve a common coding problem, that is, calculating the sum of an array (segment). This article will use Java for the code samples but the concept should apply to most programming languages. Before looking into prefix sums, let’s look at the function below that calculates the sum of an array using a standard for loop. The function iterates the array and adds each value to a variable to get the sum: Array Segments Now for array segments, array segments can also be referred to as a of an array. Specifically a of an array. With some modifications, the previous function can be used to also calculate the array segment sum by adding starting and ending array indexes: slice continuous slice Let’s say we have an array with values and we want to get the sum of a segment starting from values . The sum should be . [ 10, 7, 22, 2, 13, 15, 6 ] 7 to 13 44 (7 + 22 + 2 + 13) Now, let’s use our modified function to calculate the sum of array segment providing the starting index for 7(index=1) and ending index for 13(index=4): Prefix sums Now for prefix sums, we can use prefix sums as an alternative approach to the same problem. Prefix sums is a simple yet powerful technique that we can use to easily calculate the sum of a segment or an array. It allows us to lookup the sum of an array segment or for the whole array in constant time, by introducing a reusable lookup array. Given the same array from above, its prefix sum should be like: Now that we know what prefix sums look like, let’s use it to calculate the sum of an array segment of the original array. Using the same example array segment as earlier, the segment that starts from index 1 and ends in index 4 that have the sum of 44. When using prefix sums, iterating each array value won’t be needed anymore, we only need two elements for the calculation and we can get those in constant time. We can get the two values using elements from prefix sums array in certain indexes. The first element is at the index of starting index of the array segment, in this case, it’s , which has the value of . The second element is at the index of the ending index of the array segment, in this case, it’s , so the final index is , which has the value of . Now that we have our two elements, we can subtract the first element to the second element to get the array segment sum which is . index 1 10 +1 index 4 + 1 5 54 44 (54 - 10) That is prefix sums, in theory, now let’s write some Java code to create a prefix sum of an array. The input would be the original array and the output would be an array containing values similar to the example above. There are slight variations on how one should build a prefix sum. In our example, we’re going with initializing it with an array size based from the original array size (n+1). Then each prefix sum value of will have a value of + in order for us to have a running sum for the original array: +1 prefixSum[i] prefixSum[i - 1] originalNums[i - 1] After creating our prefix sum, we can use it to look up the sum of an array segment(or whole) in constant O(1) time. Let’s build a helper function for that lookup to make our lives easier: Putting it all together, let’s use prefix sums to calculate the array segment sum for the same sample array above: In instances where different array segment sums are needed for the same array, prefix sums are most useful. The first approach would have been , where m is how many times we need to recalculate different array segments. With prefix sums, our time complexity is reduced to . O(n * m) O(n + m) End In this post, we learned what are prefix sums, how to implement them, and how they can be applied and used to calculate the sum of an array segment. Also published at: https://micogongob.com/array-segment-sum-using-prefix-sums

Prefix Sums and How They Can be Used to Solve Coding Problems

GraphQL’s concept spans from data store to client. What happens along the way? Tooling in the world has come a long way and spinning up an application is easier than ever. By stepping through each component, you’ll have a better understanding of GraphQL as a whole. GraphQL To follow along with this tutorial, clone this . It’s a bare-bones app using sessions to authenticate users, a battle tested and secure option for persisting user auth. fullstack example Our Stack Image from https://www.howtographql.com Looking at the image above from left to right, the first represents our client application built on (our GraphQL data cache) and . I’m using to build the client bundle. Apollo React Parcel Next, the “GraphQL Server” will be , a server based on which interfaces with Apollo and provides us with a lot of sensible defaults. graphql-yoga Express will stand between the server and the database as our data layer, similar to an ORM for writing and accessing data to and from our database. Prisma is open source and supports all of the major (the stack of pancake on the right of the image). Prisma databases Setup Start by installing some global packages. yarn global add prisma graphql-cli nodemon Create a file in our project folder, we’ll fill in the DB_URL variable once we deploy Prisma: .env APP_SECRET=mysecret123DB_URL= Next, create a then log into the Prisma CLI with so you can spin up free dev servers for your projects. Don’t worry, Prisma can be moved to the hardware of your choice at any time and won’t lock you in. Prisma account prisma login We’re going to generate our user models and deploy them to our test database. Inside of you’ll see we have one model, . When we deploy, Prisma will create the bindings based on this schema. Our in the root of our project will explain to Prisma how we want our setup deployed. Now, run: /database User .graphqlconfig.yml prisma deploy database Select “ , choose whichever region makes sense, enter a name like “ and set the stage as “ . Prisma will generate an endpoint like , drop it into your . Demo server” fullstack” dev” [https://us1.prisma.sh/_myusername-12345_/fullstack/dev](https://us1.prisma.sh/myusername-2acca9a/my-project/dev.) .env One last thing, Prisma will comment out your to: database/prisma.yml endpoint: #endpoint: ${env:DB_URL} [https://us1.prisma.sh/myusername-12345/fullstack/dev](https://us1.prisma.sh/myusername-2acca9a/my-project/dev.) ... So switch it back to: endpoint: ${env:DB_URL} ... How it’ll Work After installing dependencies with or fire it up your local server with and navigate to . The command will simultaneously run our server and frontend build. yarn install npm install npm run dev localhost:1234 dev Try Out Sign up and Log in! After a user is successfully created, it will kick you to the dashboard. Once you’re logged in, the protected routes won’t allow authenticated users to the or routes. Once logged out, you can’t go to . /signup /login /dashboard Try refreshing while logged in. On page load the app will make a request to check user’s authentication state. If authenticated, the page will then fetch the user’s data. The Backend Open you’ll see the server instantiated with . It takes a number of arguments: /src/index.js graphql-yoga GraphQLServer : This is our GraphQL schema. Inside the file you’ll see we’ve listed the queries and mutations that will be handled by our resolvers. Typedefs : These are the methods that do the work of logging in, fetching our user, etc. Their inputs and return values must match up with what’s defined in our typeDef’s . Resolvers schema.graphql this takes a function that we’ll use to set the database to point to our Prisma set up. Pass in the generated typeDefs from the as well as the endpoint and secret from our . Context: prisma deploy .env As mentioned above, is based on Express so we can use common packages like to handle our sessions. graphql-yoga express-session After we define our sessions implementation, we start the server with CORS settings that allow our frontend on (via parcel) to get our session cookies. localhost:1234 The Client Side Our front end is a barebones React application bundled up with Parcel. Starting in we set up the to connect to our server running on port and assign the client to our . client/index.jsx ApolloClient 4000 ApolloProvider In the component you’ll see we’re calling the query which checks if there’s a session on the backend. It’ll determine whether to show the links or . Header GET_AUTH_STATUS login / signup logout Further down in you’ll see protected routes for our and auth routes. I created a special route component, , which calls to decide whether to display the corresponding component. Even though we call our auth status query in a few different places, when you open “ ” in developer tools you’ll see that it is only requested once. This is because Apollo Cache handles the data between different components . index.jsx Dashboard ProtectedRoute GET_AUTH_STATUS Network automatically The component handles both our signup and login mutations. On successful login or signup, we need to update our because and mutations only return the User object. Instead of making another request for auth status, we’ll just update the status manually: Auth GET_AUTH_STATUS LOGIN SIGNUP <Mutationmutation={isSignup ? SIGNUP : LOGIN}variables={isSignup ? { email, name, password } : { email, password }} > update={(cache) => {cache.writeQuery({query: GET_AUTH_STATUS,data: {isLoggedIn: {status: true,__typename: 'AuthStatus',},},});}} Wrap Up Now’s a good time to dive in to GraphQL if you’ve been watching from the sidelines. The GraphQL eco-system has come a long way in the few years it’s been around. Each of the layers in the stack has matured enough that it’s relatively straightforward to build an app from top to bottom. Also, interest is on the rise. The most recent “ ” reflects this sentiment and we’ll only see more in the future. State of Javascript Well, can explain better than me, but suffice to say using server side sessions to save sensitive data is safer (and easy). Why not store JWTs in local storage or cookies? these articles Again, here’s a link to the example . repository

Piecing Together GraphQL

Wish me luck, I’m diving into JavaScript! As much as I want to start using right away, and create applications, I know that I won’t be able to fully grasp the language unless I understand the fundamentals. Consequently, I’ve been following the chapters from which has been a great source so far. JavaScript https://javascript.info For my own reference, I thought I would write about interesting tidbits I’m , or topics I’m struggling with along the way. I hope this will also be of some use to others who are leaning JavaScript as well. learning Increment/Decrement This numerical operation increases or decreases a variable by 1. It’s important to remember that this can only be applied to variables, and applying this operation to numerical values will return an error. Increment ++: Increases variable by 1 Decrement — — : Decreases variable by 1 The ++ or — — can be applied both before and after the variable. This is where it gets a bit tricky. Postfix Form: counter++ Syntax Prefix Form: ++counter Although both forms increase the variable by 1, there is a difference. The Postfix Form returns the original value of the variable, before the increment/decrement The Prefix Form returns the value after the increment/decrement. This difference can be seen if we are using the returned value of the increment/decrement. Prefix Example let counter = 2;alert(++counter); // 3 incremented value has been returned Postfix let counter = 2;alert(counter++); // 2 Returns the original value prior to the increment If we are using the value of the increment/decrement at a later point in time however, there is no difference between the forms. Prefix Example let counter = 2;++counter; // _3 The incremented value_alert(counter); // 3 Incremented value has been returned Postfix let counter = 2;counter++; // _2 The original value_alert(counter); // 3 Value has now been incremented and returns the new value It took me a bit of time to wrap my head around this so I hope this was a clear enough explanation. If you liked this article, give it a few claps. I would greatly appreciate it!

JavaScript — Back to Basics: Prefix vs. Postfix

Recommendations by Concise User Profiles from Review Text: Experimental Results

Sentiment Analysis through LLM Negotiations: Conclusion and References

AI-Powered Localization Using an Open-source Project as a Case Study

Hateful Meme Detection: Leveraging PVLMs for Zero-shot VQA Probing

Using MinIO and Python to Create Scalable and Efficient Event-driven Applications

স্কেলযোগ্য এবং দক্ষ ইভেন্ট-চালিত অ্যাপ্লিকেশন তৈরি করতে MinIO এবং Python ব্যবহার করে

Verwenden von MinIO und Python zum Erstellen skalierbarer und effizienter ereignisgesteuerter Anwendungen

Uso de MinIO y Python para crear aplicaciones basadas en eventos escalables y eficientes

Utiliser MinIO et Python pour créer des applications événementielles évolutives et efficaces

स्केलेबल और कुशल इवेंट-संचालित एप्लिकेशन बनाने के लिए मिनियो और पायथन का उपयोग करना

MinIO と Python を使用してスケーラブルで効率的なイベント駆動型アプリケーションを作成する

MinIO 및 Python을 사용하여 확장 가능하고 효율적인 이벤트 기반 애플리케이션 만들기

Usando MinIO e Python para criar aplicativos orientados a eventos escalonáveis e eficientes

Использование MinIO и Python для создания масштабируемых и эффективных приложений, управляемых событиями

Ölçeklenebilir ve Verimli Olay Odaklı Uygulamalar Oluşturmak için MinIO ve Python'u Kullanmak

Sử dụng MinIO và Python để tạo các ứng dụng hướng sự kiện có khả năng mở rộng và hiệu quả

使用 MinIO 和 Python 创建可扩展且高效的事件驱动应用程序

Managing Large Data Volumes With MinIO, Langchain and OpenAI

MinIO, Langchain এবং OpenAI এর মাধ্যমে বড় ডেটা ভলিউম পরিচালনা করা

Große Datenmengen verwalten mit MinIO, Langchain und OpenAI

Gestión de grandes volúmenes de datos con MinIO, Langchain y OpenAI

Gestion de gros volumes de données avec MinIO, Langchain et OpenAI

मिनियो, लैंगचेन और ओपनएआई के साथ बड़े डेटा वॉल्यूम का प्रबंधन

MinIO、Langchain、OpenAI による大容量データの管理

MinIO, Langchain 및 OpenAI를 사용하여 대용량 데이터 관리

Gerenciando grandes volumes de dados com MinIO, Langchain e OpenAI

Управление большими объемами данных с помощью MinIO, Langchain и OpenAI

MinIO, Langchain ve OpenAI ile Büyük Veri Hacimlerini Yönetmek

Quản lý khối lượng dữ liệu lớn với MiniIO, Langchain và OpenAI

使用 MinIO、Langchain 和 OpenAI 管理大量数据

Make a Code Review Great Again: Patterns of Quick and Effective Code Quality Control

The modern data landscape demands a new breed of infrastructure – one that seamlessly integrates structured and unstructured data, scales effortlessly, and empowers efficient AI/ML workloads. This is where come in, providing a central hub for all your data needs. However, building and managing an effective data lake can be complex. modern datalakes This blog post dives deep into three powerful tools that can optimize your current approach: , and MinIO. The steps below will walk you through how these services seamlessly combine to create a robust, cloud-native data lake architecture specifically optimized for AI/ML workloads. Apache Iceberg Tabular, What is Tabular? Tabular is a data platform created by the original creators of Apache Iceberg. It is designed to provide an independent, universal storage platform that connects to any compute layer, eliminating data vendor lock-in. This feature is critical to the modern data stack, it allows users to pick best-in-class compute and storage tools without being forced into a particular vendor's aged and or mismatched tool set. In an of MinIO and Iceberg and can be enhanced by Tabular. Tabular can be used to manage and query Iceberg data stored in MinIO, allowing for the storage and management of structured data in a scalable, high-performance, and cloud-native manner. These Kubernetes native, components work together smoothly with very little friction and build on each other’s capabilities to perform at scale. architecture Why S3FileIO instead of Hadoop's file-io? This implementation leverages Iceberg's S3FileIO. S3FileIO is considered better than Hadoop's file-io for several reasons. Some of which we’ve already discussed : elsewhere : Iceberg's S3FileIO is designed to work with cloud-native storage. Optimized for Cloud Storage Iceberg uses an to distribute files across multiple prefixes in a MinIO bucket, which helps to minimize throttling and maximize throughput for S3-related IO operations. Improved Throughput and Minimized Throttling: ObjectStoreLocationProvider Iceberg has been to fully leverage strict consistency by eliminating redundant consistency checks that could impact performance. Strict Consistency: updated Iceberg's S3FileIO implements a algorithm, which uploads data files parts in parallel as soon as each part is ready, reducing local disk usage and increasing upload speed. Progressive Multipart Upload: progressive multipart upload Iceberg allows for for S3 API writes to ensure the integrity of uploaded objects, which can be enabled by setting the appropriate catalog property. Checksum Verification: checksum validations Iceberg supports adding to objects during write and delete operations with the S3 API, which can be useful for cost tracking and management. Custom Tags: custom tags The FileIO interface in Iceberg does not require as strict guarantees as a Hadoop-compatible FileSystem, which allows it to that could otherwise degrade performance. Avoidance of Negative Caching: avoid negative caching In contrast, Hadoop's S3A FileSystem, which was used before S3FileIO, does not offer the same level of optimization for cloud storage. All this to say: Don’t hobble your future-facing data lake infrastructure with the trappings of the past. Prerequisites Before you begin, make sure your system meets the following requirements: Docker Docker Compose If you're starting from scratch, you can install both using the installer for your specific platform. It is often easier than downloading Docker and Docker Compose separately. Verify if Docker is installed by running the following command: Docker Desktop docker-compose --version Getting started To get started, clone or copy the YAML file in Tabular’s . You just need the YAML for this tutorial. Feel free to explore the rest of the repository at a later time. git repository Breaking it Down The provided YAML file is a Docker Compose configuration file. It defines a set of services and their configurations for a multi-container Docker application. In this case, there are two services: Spark-Iceberg and MinIO. Let's break down each section: 1. Spark-Iceberg Service: spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    build: spark/
    networks:
      iceberg_net:
    depends_on:
      - rest
      - minio
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    ports:
      - 8888:8888
      - 8080:8080
      - 10000:10000
      - 10001:10001
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      iceberg_net:
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000 Specifies the Docker image to use for the spark-iceberg service. In this case, it uses the tabulario/spark-iceberg:latest image. image: Specifies that the spark-iceberg service depends on the rest and minio services. depends_on: Assigns a specific name (spark-iceberg) to the container. container_name: Sets environment variables for the container, including Spark and AWS credentials. environment: Mounts local directories (./warehouse and ./notebooks) as volumes inside the container. volumes: Maps container ports to host ports for accessing Spark UI and other services. ports: 2. Minio Service: minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=minio
    networks:
      iceberg_net:
        aliases:
          - warehouse.minio
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"] Specifies the Docker image for the MinIO service. image: Assigns a specific name (MinIO) to the container. container_name: Sets environment variables for configuring MinIO, including root user credentials. environment: Maps container ports to host ports for accessing MinIO UI. ports: Specifies the command to start the MinIO server with specific parameters. command: Another aspect of the MinIO service is , MinIO’s command line tool. mc mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      " Specifies that the mc service depends on the MinIO service. depends_on: Specifies the Docker image for the mc service. image: Assigns a specific name (mc) to the container. container_name: Sets environment variables for configuring the MinIO client. environment: Defines the entry point command for the container, including setup steps for the MinIO client. entrypoint: /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      " This sequence of commands essentially performs the following tasks: Removes the existing warehouse directory and its contents from the MinIO server. Creates a new bucket named warehouse. Sets the access policy of the warehouse bucket to public. This Docker Compose file orchestrates a multi-container environment with services for Spark, PostgreSQL, MinIO. It sets up dependencies, environment variables, and commands necessary for running the services together. The services work in tandem to create a development environment for data processing using Spark and Iceberg with MinIO as the object storage backend. Starting Up In a terminal window, cd into the tabular-spark-setup directory in the repository and run the following command: docker-compose up Login into MinIO at with the credentials to see that the warehouse bucket has been created. http://127.0.0.1:9001 admin:password Once all the containers are up and running, you can access your Jupyter Notebook server by navigating to http://localhost:8888 Run one of the sample notebooks and return to MinIO at to see your warehouse populated with data. http://127.0.0.1:9001 Building Your Modern Datalake This tutorial on building a modern datalake with Iceberg, Tabular, and MinIO is just the beginning. This powerful trio opens doors to a world of possibilities. With these tools, you can seamlessly integrate and analyze all your data, structured and unstructured, to uncover hidden patterns and drive data-driven decisions that fuel innovation. Leverage the efficiency and flexibility of this architecture in production to expedite your AI/ML initiatives and unlock the true potential of your machine learning models, accelerating your path to groundbreaking discoveries. Reach out to us at or on our channel if you have any questions as you build. hello@min.io Slack

Building Effective Modern Data Architectures with Iceberg, Tabular and MinIO

আইসবার্গ, ট্যাবুলার এবং মিনিও সহ কার্যকরী আধুনিক ডেটা আর্কিটেকচার তৈরি করা

Erstellen effektiver moderner Datenarchitekturen mit Iceberg, Tabular und MinIO

Creación de arquitecturas de datos modernas y eficaces con Iceberg, Tabular y MinIO

Construire des architectures de données modernes et efficaces avec Iceberg, Tabular et MinIO

आइसबर्ग, टेबुलर और मिनियो के साथ प्रभावी आधुनिक डेटा आर्किटेक्चर का निर्माण

Iceberg、Tabular、MinIO を使用した効果的な最新データ アーキテクチャの構築

Iceberg, Tabular 및 MinIO를 사용하여 효과적인 최신 데이터 아키텍처 구축

Construindo arquiteturas de dados modernas e eficazes com Iceberg, Tabular e MinIO

Создание эффективных современных архитектур данных с помощью Iceberg, Tabular и MinIO

Iceberg, Tabular ve MinIO ile Etkili Modern Veri Mimarileri Oluşturma

Xây dựng kiến trúc dữ liệu hiện đại hiệu quả với Iceberg, Tabular và MinIO

使用 Iceberg、Tabular 和 MinIO 构建有效的现代数据架构

How to Use BigQuery Efficiently: Your Guide to Cost Optimization and Efficient Data Management

Apache Cassandra—the immensely popular open-source, distributed NoSQL database management system—is approaching the general availability of its 5.0 version (the 5.0 beta is already available). Whether you’re an existing Cassandra user or considering a migration to the open-source database with the release of this new version, there’s a lot to be excited about with 5.0. I spoke with Mo Ansari, a Product Manager at Instaclustr by NetApp, about what developers, engineers, DBAs, and other users should know about Cassandra 5.0—from database upgrades to new use cases to open-source community support. Mo knows Cassandra particularly well (Instaclustr has long had a managed platform around the fully open-source version of the database, which Mo works on). Here’s what Mo had to say about Cassandra 5.0: Cassandra 5.0 promises improved database performance and scalability—what are some specific optimizations or architectural changes that have been made to achieve this? Cassandra 5.0 adds in the advantages of trie (prefix tree)-based Memtables and SSTables—meaning the new version of the open source NoSQL database offers significant potential for improved database performance and memory optimization. Cassandra’s performance for reads and modification operations (and its ability to correctly size structures to data) benefit from these storage formats, which make use of tries and byte-comparable representations of database keys. Memory management overhead and garbage collection also face fewer burdens with trie Memtables and trie-indexed SSTables, delivering even more benefits for Cassandra users (especially those using the database at scale). Aside from performance, are there other changes with Cassandra 5.0 that developers should be excited about? Developers will want to get their hands on Cassandra 5.0’s vector support, including Vector Search for locating content in large datasets. They can also utilize new CQL vector functions and a new vector data type designed for saving and retrieving embeddings vectors. Because of these improvements, Cassandra 5.0 is well-positioned as a data layer technology for supporting AI/ML application development. ML models function by comparing similarities among data and putting data connections in context. Embeddings vectors power similarity comparisons by offering arrays of floating-point numbers representing how similar particular objects are to each other. With Cassandra 5.0, developers now have a powerful (and open source) database with that specific functionality that’s so important to AI/ML applications. Developers should also be hyped about storage-attached indexing, which makes secondary indexes far more usable and efficient. On a Cassandra database table, developers can now easily create one or more secondary indexes, each based on a single column they choose. The result is massively scalable and globally distributed indexing with search throughput that can’t be beat (useful with Vector Search), modular extensibility (also demonstrated by Vector Search), and stunning indexing functionality that captures semantics via queries and content (including large documents and images). Last but not least, developers using Cassandra 5.0 will like the ability to build their own user-defined functions, and to utilize the range of useful new native CQL aggregation and math functions available with the new version. How has the developer experience changed—if it has—with Cassandra 5.0? Cassandra is bringing significant enhancements to the developer experience. The new release features open the door for a vast number of use cases and balance the developer experience. The new version focuses on ease of use, performance, and security and includes features such as storage-attached indexes that make queries on non-primary key columns more efficient. This reduces the complexity and overhead associated with secondary indexes. Additionally, the new version is anticipated to support ACID transactions with version 5.1, which will bring SQL-like functionality to Cassandra—making it more approachable for developers familiar with relational databases. Furthermore, the new version has more guardrails and enhanced tooling, including a new virtual table to view system lows, which will aid in the development process. Overall, Cassandra 5.0 is packed with features that will empower developers to work faster, create more efficient queries, and manage data with increased security and control! Do you anticipate the use cases for Cassandra to change or evolve based on what’s included in 5.0? Yes, with a new vector datatype and storage-attached indexes improving the performance of the usage of vectors, Apache Cassandra is positioning itself as a contender in the AI market. Over the years, Cassandra’s exceptional write throughput and ability to handle large volumes of data made it ideal for several use cases and applications, such as scalable web applications, messaging systems, and event logging and monitoring systems. However, Cassandra 5.0 is going to significantly alter the landscape. Apart from all the use cases that Cassandra supported and suited in the past, version 5.0 will extend its applicability to more complex, transactional, and analytical applications, bridging the gap between NoSQL flexibility and the rigorous demands of modern data-intensive applications. I see several new use cases, such as analytics and machine learning, niche financial services requiring ACID transactions, applications requiring complex querying capabilities, etc. As the Apache Cassandra project calls it, “moving towards an AI-driven future.” With increasing adoption of cloud and containerized environments, how has Cassandra 5.0 been designed to support these deployment models? Apache Cassandra has always been designed with distributed and fault-tolerant computing and scaling in mind. It has always been a good choice for cloud-based workloads. While not introducing specific features under the “cloud-native” label, Cassandra 5.0 continues to support and enhance deployments in cloud and containerized environments through its scalable, resilient, and community-driven projects. Features included in the 4.1 version release have already paved the way for a more cloud-native future for Cassandra. The K8ssandra project is another great example of a commitment to a cloud-native future. With any new release, there’s always a learning curve. How do you see the adoption process going for Cassandra 5.0, and what advice would you give to developers who are looking to migrate? The transition to Cassandra 5.0 does bring a learning curve, but it’s designed to be as smooth as possible. I anticipate gradual adoption as developers familiarize themselves with new features and improvements. I expect the adoption will be pretty quick in dev and testing environments; however, it will be gradual in production environments. Given the focus on cost reduction and AI readiness, there are a lot more reasons, use cases, motivations, and other drivers for adoption. Based on its several utilities, I'm guessing that Apache Cassandra will become the preferred choice for AI workloads. For those looking to migrate, my advice is to leverage the extensive resources available on the Apache website, including its documentation and community forums. Planet Cassandra is a fantastic place to look at the use cases. I would also advise the developers to engage in the community for shared insights and best practices through mailing lists, project town halls, and other events such as contributor meetings. Cassandra has a large and active open-source community. How has the community been involved in the development of 5.0? Apache Cassandra has a large and vibrant open-source community, playing a crucial role in the development of 5.0. Members have been actively involved in proposing new features and improving existing ones based on real-world use cases and future needs. Feedback was collected through various channels, such as mailing lists, Jira tickets, and Cassandra Enhancement Proposals (CEPs), and it has been instrumental in shaping the release to meet the evolving needs of users. The community has been actively contributing to many more features that are currently in the draft stage and will bring many more benefits to Cassandra in the future. Apache Cassandra—the immensely popular open-source, distributed NoSQL database management system—is approaching the general availability of its 5.0 version (the 5.0 beta is already available). 5.0 beta Whether you’re an existing Cassandra user or considering a migration to the open-source database with the release of this new version, there’s a lot to be excited about with 5.0. I spoke with Mo Ansari, a Product Manager at Instaclustr by NetApp , about what developers, engineers, DBAs, and other users should know about Cassandra 5.0—from database upgrades to new use cases to open-source community support. Instaclustr by NetApp Mo knows Cassandra particularly well (Instaclustr has long had a managed platform around the fully open-source version of the database, which Mo works on). Here’s what Mo had to say about Cassandra 5.0: Cassandra 5.0 promises improved database performance and scalability—what are some specific optimizations or architectural changes that have been made to achieve this? Cassandra 5.0 adds in the advantages of trie (prefix tree)-based Memtables and SSTables—meaning the new version of the open source NoSQL database offers significant potential for improved database performance and memory optimization. Cassandra’s performance for reads and modification operations (and its ability to correctly size structures to data) benefit from these storage formats, which make use of tries and byte-comparable representations of database keys. Memory management overhead and garbage collection also face fewer burdens with trie Memtables and trie-indexed SSTables, delivering even more benefits for Cassandra users (especially those using the database at scale). Aside from performance, are there other changes with Cassandra 5.0 that developers should be excited about? Developers will want to get their hands on Cassandra 5.0’s vector support, including Vector Search for locating content in large datasets. They can also utilize new CQL vector functions and a new vector data type designed for saving and retrieving embeddings vectors. Because of these improvements, Cassandra 5.0 is well-positioned as a data layer technology for supporting AI/ML application development. ML models function by comparing similarities among data and putting data connections in context. Embeddings vectors power similarity comparisons by offering arrays of floating-point numbers representing how similar particular objects are to each other. With Cassandra 5.0, developers now have a powerful (and open source) database with that specific functionality that’s so important to AI/ML applications. Developers should also be hyped about storage-attached indexing, which makes secondary indexes far more usable and efficient. On a Cassandra database table, developers can now easily create one or more secondary indexes, each based on a single column they choose. The result is massively scalable and globally distributed indexing with search throughput that can’t be beat (useful with Vector Search), modular extensibility (also demonstrated by Vector Search), and stunning indexing functionality that captures semantics via queries and content (including large documents and images). Last but not least, developers using Cassandra 5.0 will like the ability to build their own user-defined functions, and to utilize the range of useful new native CQL aggregation and math functions available with the new version. How has the developer experience changed—if it has—with Cassandra 5.0? Cassandra is bringing significant enhancements to the developer experience. The new release features open the door for a vast number of use cases and balance the developer experience. The new version focuses on ease of use, performance, and security and includes features such as storage-attached indexes that make queries on non-primary key columns more efficient. This reduces the complexity and overhead associated with secondary indexes. Additionally, the new version is anticipated to support ACID transactions with version 5.1, which will bring SQL-like functionality to Cassandra—making it more approachable for developers familiar with relational databases. Furthermore, the new version has more guardrails and enhanced tooling, including a new virtual table to view system lows, which will aid in the development process. Overall, Cassandra 5.0 is packed with features that will empower developers to work faster, create more efficient queries, and manage data with increased security and control! Do you anticipate the use cases for Cassandra to change or evolve based on what’s included in 5.0? Yes, with a new vector datatype and storage-attached indexes improving the performance of the usage of vectors, Apache Cassandra is positioning itself as a contender in the AI market. Over the years, Cassandra’s exceptional write throughput and ability to handle large volumes of data made it ideal for several use cases and applications, such as scalable web applications, messaging systems, and event logging and monitoring systems. However, Cassandra 5.0 is going to significantly alter the landscape. Apart from all the use cases that Cassandra supported and suited in the past, version 5.0 will extend its applicability to more complex, transactional, and analytical applications, bridging the gap between NoSQL flexibility and the rigorous demands of modern data-intensive applications. I see several new use cases, such as analytics and machine learning, niche financial services requiring ACID transactions, applications requiring complex querying capabilities, etc. As the Apache Cassandra project calls it, “ moving towards an AI-driven future .” moving towards an AI-driven future With increasing adoption of cloud and containerized environments, how has Cassandra 5.0 been designed to support these deployment models? Apache Cassandra has always been designed with distributed and fault-tolerant computing and scaling in mind. It has always been a good choice for cloud-based workloads. While not introducing specific features under the “cloud-native” label, Cassandra 5.0 continues to support and enhance deployments in cloud and containerized environments through its scalable, resilient, and community-driven projects. Features included in the 4.1 version release have already paved the way for a more cloud-native future for Cassandra. The K8ssandra project is another great example of a commitment to a cloud-native future. 4.1 version release K8ssandra project With any new release, there’s always a learning curve. How do you see the adoption process going for Cassandra 5.0, and what advice would you give to developers who are looking to migrate? The transition to Cassandra 5.0 does bring a learning curve, but it’s designed to be as smooth as possible. I anticipate gradual adoption as developers familiarize themselves with new features and improvements. I expect the adoption will be pretty quick in dev and testing environments; however, it will be gradual in production environments. Given the focus on cost reduction and AI readiness, there are a lot more reasons, use cases, motivations, and other drivers for adoption. Based on its several utilities, I'm guessing that Apache Cassandra will become the preferred choice for AI workloads. For those looking to migrate, my advice is to leverage the extensive resources available on the Apache website , including its documentation and community forums. Planet Cassandra is a fantastic place to look at the use cases. I would also advise the developers to engage in the community for shared insights and best practices through mailing lists, project town halls, and other events such as contributor meetings. Apache website Planet Cassandra Cassandra has a large and active open-source community. How has the community been involved in the development of 5.0? Apache Cassandra has a large and vibrant open-source community , playing a crucial role in the development of 5.0. Members have been actively involved in proposing new features and improving existing ones based on real-world use cases and future needs. Feedback was collected through various channels, such as mailing lists, Jira tickets, and Cassandra Enhancement Proposals (CEPs), and it has been instrumental in shaping the release to meet the evolving needs of users. open-source community The community has been actively contributing to m any more features that are currently in the draft stage and will bring many more benefits to Cassandra in the future. any more features

Here's What to Know About Apache Cassandra 5.0

How to Implement the Idempotency-Key Specification on Apache APISIX

Apache APISIX-এ আইডেমপোটেন্সি-কী স্পেসিফিকেশন কীভাবে প্রয়োগ করবেন

So implementieren Sie die Idempotency-Key-Spezifikation auf Apache APISIX

Cómo implementar la especificación de clave de idempotencia en Apache APISIX

Comment implémenter la spécification Idempotency-Key sur Apache APISIX

Apache APISIX पर Idempotency-Key विनिर्देश को कैसे लागू करें

Apache APISIX で Idempotency-Key 仕様を実装する方法

Apache APISIX에서 멱등성 키 사양을 구현하는 방법

Como implementar a especificação de chave de idempotência no Apache APISIX

Как реализовать спецификацию идемпотентного ключа в Apache APISIX

Idempotency-Key Belirtimi Apache APISIX'te Nasıl Uygulanır?

Cách triển khai Đặc tả khóa Idempotency trên Apache APISIX

如何在 Apache APISIX 上实现幂等密钥规范

Migration Background Some users originally used Airflow as their scheduling system. However, Airflow only allows workflow definition through code and lacks resource and project granularity which fails to meet customer needs in scenarios requiring strong permission control. Consequently, some users need to migrate their scheduling system from Airflow to Apache Dolphinscheduler. With a commitment to addressing user needs, Whaleops developed the Air2phin migration tool to facilitate a smoother transition to DolphinScheduler. Since Airflow workflows are defined using Python code, and some metadata only exists in the Python code without being persisted to the database, it’s necessary to parse Python code for analysis and migration. Why Migrate to DolphinScheduler Both Airflow and DolphinScheduler are task scheduling systems addressing workflow orchestration, each with its strengths. This section will focus on DolphinScheduler’s advantages over Airflow. A detailed comparison will be covered in future articles: Both are mature open-source workflow scheduling systems with robust communities. The subtle differences include: Apache DolphinScheduler: Emphasizes visualization with supplementary API, offers granular permission management, more workflow levels, lower usage costs, and democratizes data. Airflow: Defines workflows through code, catering to advanced developers with high flexibility but higher usage costs, primarily targeting developers. 2. Since DolphinScheduler stores workflow definitions, task definitions, and task relationships in the database, it has several advantages: No extra steps are needed when adding master or worker nodes; in contrast, Airflow requires copying DAG files to new nodes. Eliminates latency issues inherent in file parsing, unlike Airflow’s loop-based scheduling which consumes substantial CPU resources. Retains complete historical data on workflow and task statuses. In Airflow, deleted tasks in the latest definitions cannot be traced back. Natively supports versioning information, which requires Git logs and reverting in Airflow. 3. DolphinScheduler features a resource center, making it easier for users to manage local and remote resources. In Airflow, external resources typically need version control alongside Git. 4. Besides offline scheduling tasks, DolphinScheduler supports real-time tasks, data materials, and physical resource monitoring, focusing on practical scheduling functionalities. Airflow currently concentrates more on offline workflow scheduling. 5. DolphinScheduler is a distributed, centerless system with higher master server resource utilization. Airflow’s scheduler consumes more CPU resources in scanning and scheduling tasks. Demands and Challenges Demands: The core demand for a migration tool is to transform Airflow DAGs into DolphinScheduler workflows with minimal human intervention. However, a balance is needed to avoid overly complex, less maintainable, and less generalizable programs, especially when accommodating different Airflow versions. Challenges Airflow and DolphinScheduler’s Python SDKs share basic Python syntax (for, if, else) but differ in task names and parameters. For example, Airflow’s bash operator corresponds to Shell in DolphinScheduler’s SDK, with different parameters. Syntax Differences: Both platforms might allow some degree of customization, like custom plugins. However, they differ in the number and abstraction of task types, with some unique to each platform. Task Type Variance: : Airflow uses Cron expressions (e.g., “5 4 * * *”) or Python’s datetime.timedelta, while DolphinScheduler employs more detailed Cron expressions, like “0 5 4 * * ? *”. Scheduling Variance : Airflow handles time parameters using macros and provides a Jinja2 template for calculations (e.g., ds_add(‘2015–01–06’, -5)). DolphinScheduler has its own definitions and rules, like using “yyyy-MM-dd” for run times and “yyyy-MM-dd+1” for time adjustments. Built-in Time Parameter Differences : Both Airflow and DolphinScheduler’s Python SDK may update their APIs over time. Migration tool rules need to be simple to modify and add, minimizing maintenance costs. Migration Rule Extension Variations between Airflow versions might impact syntax, e.g., airflow.operators.bash_operator in versions before 2.0.0 and airflow.operators.bash in 2.0.0 and later. Different Airflow Versions: Introduction to the Migration Tool What Is Air2phin? Air2phin is a rule-based AST (Abstract Syntax Tree) transformer that converts Airflow DAG files into pydolphinscheduler definition files. It uses LibCST for parsing and transforming Python code and YAML files to define transformation rules. It’s a tool to assist users in conversion, not an automatic one. Air2phin Data Flow! Retrieves original Airflow DAGs definitions from standard input or files. Loads conversion rules from a YAML file into air2phin. Parses Airflow DAGs content into a CST tree. Modifies the CST tree based on transformation rules. Outputs the transformed result to standard output or files. How to Use Air2phin？ Air2phin, being a Python package, requires installation via pip. Once installed, it can be used to convert all the DAGs in Airflow into definitions for the DolphinScheduler Python SDK. After this transformation, executing these SDK scripts with Python will submit the converted workflows to DolphinScheduler. # Install package python -m pip install --upgrade air2phin # Migrate Airflow's DAGs air2phin migrate -i ~/airflow/dags In a production environment, the directory may contain numerous DAGs. Air2phin, by default, processes these DAGs sequentially. For more efficient processing, you can use the option to enable multi-process execution. ~/airflow/dags --multiprocess # Use multiprocess to convert the Airflow DAGs files air2phin migrate -i --multiprocess 12 ~/airflow/dags After completing the above transformation, the migration from Airflow DAG files to DolphinScheduler Python SDK scripts is accomplished. These scripts can then be submitted to DolphinScheduler for execution. # Install apache-dolphinscheduler according to the Apache DolphinScheduler server you use, ref: https://dolphinscheduler.apache.org/ python/main/#versionpython -m pip install apache-dolphinscheduler # Submit your DolphinScheduler Python SDK definition python ~/airflow/dags/tutorial.py Defining Custom Conversion Rules in Air2phin？ Most Airflow users customize some operators. To convert these operators, users need to define rules. Fortunately, Air2phin’s rules are based on YAML files, which means that users can easily add new rules. The following is a rule conversion YAML file that converts a user-defined Redshift operator into a DolphinScheduler SQL task type. It is assumed here that the user has customized a redshift operator based on airflow.providers.postgres.operators.postgres. The operator code is as follows: from airflow.providers.postgres.operators.postgres import PostgresOperator class RedshiftOperator(PostgresOperator): def __init__( self, *, sql: str | Iterable[str], my_custom_conn_id: str = 'postgres_default', autocommit: bool = False, parameters: Iterable | Mapping | None = None, database: str | None = None, runtime_parameters: Mapping | None = None, **kwargs, ) -> None: super().__init__( sql=sql, postgres_conn_id=my_custom_conn_id, autocommit=autocommit, parameters=parameters, database=database, runtime_parameters=runtime_parameters, **kwargs, ) Since this is a user-defined operator, it is definitely not in the built-in conversion rules of air2phin, so we need to customize a YAML file of conversion rules. name: RedshiftOperator migration: module: - action: replace src: utils.operator.RedshiftOperator.RedshiftOperator dest: pydolphinscheduler.tasks.sql.Sql - action: add module: pydolphinscheduler.resources_plugin.Local parameter: - action: replace src: task_id dest: name - action: add arg: datasource_name default: type: str value: "redshift_read_conn" - action: add arg: resource_plugin default: type: code value: Local(prefix="/path/to/dir/") Customers only need to add this file to the rule path of air2phin to realize the conversion action of the custom operator. air2phin migrate --custom-rules /path/to/RedshiftOperator.yaml ~/airflow/dags How Does Air2phin Address Migration Challenges？ Air2phin addresses the various challenges posed by the migration process: Syntax Differences Since both Airflow and DolphinScheduler Python SDK are written in Python, the basic syntax related to Python is similar. However, since Airflow and DolphinScheduler Python SDK are two unrelated sets of APIs, there are some differences between the two in terms of specific parameters, classes, and functions. air2phin is used to solve this problem. It resolves the differences and implements the migration process from one platform to another by defining appropriate YAML conversion rules for this part of the difference. YAML File Conversion Rules • For different naming or structures of parameters, mapping rules can be defined in the YAML file to map the parameter names of Airflow to the corresponding parameters of the DolphinScheduler Python SDK. Parameter mapping: • : If Airflow and DolphinScheduler Python SDK use different class names or functions, you can define class and function conversion rules in a YAML file to map Airflow’s class names and functions to their DolphinScheduler Python SDK equivalents. Class and function conversion • Given that the two platforms may have different error handling and alerting mechanisms, how to map Airflow’s error handling to the equivalent mechanism of DolphinScheduler can be defined in the YAML file. Error handling and alerting: By formulating these conversion rules, you can ensure that during the migration process, Airflow’s task code is converted into the code required by the DolphinScheduler Python SDK platform according to the definition of the YAML file to adapt to the differences between platforms and ensure that new and modified tasks flexibility. Timing Scheduling Differences There are also some differences between Airflow and DolphinScheduler Python SDK in terms of scheduled scheduling configuration. Airflow uses standard Cron expressions to define task scheduling, while DolphinScheduler Python SDK adopts a more precise Cron scheduling strategy. This difference may affect the precise scheduling and execution frequency of tasks. : Airflow uses general Cron expressions to define the scheduling frequency of tasks. A Cron expression consists of five or six fields, representing minutes, hours, dates, months, and days of the week. It allows users to define relatively loose scheduling rules, such as once an hour, once a day, etc. • Airflow’s Cron expressions : DolphinScheduler introduces a more precise Cron scheduling strategy. It splits Cron expressions into two parts: Basic Cron and Advanced Cron. Basic Cron is used to define rough scheduling rules for tasks, such as minutes, hours, dates, etc. Advanced Cron is used to define more precise scheduling rules, including second-level accuracy. This allows DolphinScheduler to implement more fine-grained task scheduling, which is suitable for scenarios with higher execution time requirements, such as the financial field. • Precise Cron scheduling of DolphinScheduler Python SDK Since the accuracy of DolphinScheduler Python SDK is higher than that of Airflow, there will be no problem of accuracy loss during conversion, and this problem will be easily solved. Built-in Time Parameter Differences Built-in time parameter differences refer to the different ways in which Airflow and DolphinScheduler Python SDK use built-in time parameters in task scheduling. Airflow uses Jinja2’s macro function to implement built-in time parameters, while DolphinScheduler’s Python SDK uses a custom way to implement these parameters. These two implementation methods may lead to differences in usage and understanding. • Airflow’s Jinja2 macro: Airflow’s built-in time parameters are implemented through Jinja2 macros. Jinja2 macros allow the use of special placeholders and functions in DAGs files for dynamically generating scheduling times. For example, you can use {{ macros.ds_add(ds, 1) }} to add one day to the scheduling time. • Custom implementation of DolphinScheduler Python SDK: When implementing built-in time parameters, DolphinScheduler’s Python SDK may use some custom functions or classes instead of directly using Jinja2 macros. These custom implementation methods may require specific configuration and processing on the DolphinScheduler platform. Therefore, you need to pay attention to the following when migrating: 1. Different syntax and methods: The syntax and usage of Airflow’s Jinja2 macro are very different from the custom implementation of the DolphinScheduler Python SDK, which may cause parameters not to be migrated correctly part of the time. Air2phin will retain its original values for some parameters that cannot be automatically migrated. 2. Functional similarity: Although implemented differently, both aim to provide built-in time parameters for task scheduling. Ensure that migrated tasks correctly use the new platform’s built-in timing parameters. Extension of Migration Rules Airflow allows users to define and use custom Operators, Hooks, Sensors, etc. as needed to meet specific task requirements. These custom components may be used in DAGs, and the way they are implemented and called may require special handling during the migration process. The simplest way to deal with it is to use the method mentioned in the above question “How Air2phin defines its own conversion rules.” As long as the custom task can be defined in DolphinScheduler, the task can be migrated from Airflow to DolphinScheduler. Different Versions of Airflow Migration Different versions of Airflow have different operator syntax. In versions before 2.0.0, Airflow only supports bash with the class airflow.operators.bash_operator.BashOperator. However, in versions 2.0.0 and later, Airflow More recommended for bash is the airflow.operators.bash.BashOperator class which is also compatible with Airflow.operators.bash_operator.BashOperator. There are many similar situations, so Air2phin needs to be compatible with the shell task type converted from the above two types into DolphinScheduler. We implement the transformation of multiple classes by supporting lists in YAML. For details, see the migration.module.* nodes below. migration: module: - action: replace src: - airflow.operators.bash.BashOperator - airflow.operators.bash_operator.BashOperator dest: pydolphinscheduler.tasks.shell.Shell parameter: - action: replace src: task_id dest: name - action: replace src: bash_command dest: command User Benefits The Air2phin migration tool allows users to convert Airflow’s DAGs code to DolphinScheduler Python SDK through simple configuration, bringing many benefits to users: • Simplify the migration process: The migration tool can automatically handle code conversion, avoiding the complex process of manual line-by-line migration, greatly reducing the workload of developers. • Save time and costs: Manually migrating code requires a significant investment of time and human resources. Use migration tools to complete your migration quickly and efficiently, saving time and money. • Reduce errors: Manual migration easily introduces errors, but migration tools can automatically perform conversions based on predefined rules, reducing potential human errors. • Standardize code style: The migration tool can generate code based on predefined rules and templates to ensure consistent code style and reduce maintenance costs. • Lower the technical threshold: The migration tool can hide the underlying technical details, allowing developers who are not familiar with DolphinScheduler to easily migrate tasks. • Flexibility and customizability: Good migration tools usually offer some customizable options to meet the needs of different projects while maintaining flexibility. Overall, using Air2phin can significantly improve the efficiency and quality of the migration process, reduce risks, and at the same time, reduce the workload of developers, bringing time and resource savings to the team, as well as a better development experience. Problems that Air2phin Cannot Solve Yet Air2phin is a tool that helps users migrate from Airflow to Apache DolphinScheduler more easily. The keyword for this is “assistance”, which means that it can reduce the user’s migration cost, but it cannot be fully automated. The currently known unsolvable problems are as follows: • Task types that do not exist in DolphinScheduler cannot be migrated: Some task types only exist in Airflow, but do not exist in DolphinScheduler. These tasks cannot be automatically migrated and need to be processed manually. For example, the Discord operator does not exist in DolphinScheduler, so the original Discord operator definition will be retained and needs to be processed manually by the user. • Some task attributes cannot be migrated to DolphinScheduler: Some task attributes in Airflow do not exist in DolphinScheduler, such as successc_callback and retry_callback. These attributes will be directly abandoned during the migration process. Reference Air2phin Documentation Air2phin on PyPI Air2phin GitHub Repository Cited Source [1] AWS Performance Evaluation: Link

How to Seamlessly Transfer From Airflow to Apache DolphinScheduler With Air2phin

This New Data Type Is 8 Times Faster Than JSON: Improve Your Semi-Structured Data Analysis

Stories, News & Mentions about PIEFI

Piefi’s Stories on HackerNoon