MinIO is a high-performance, S3 compatible object store. It is built for large scale AI/ML, data lake and database workloads. It runs on-prem and on any cloud (public or private) and from the data center to the edge. MinIO is software-defined and open source under GNU AGPL v3.

Min.io Github

MinIO Reviews and Ratings

Introduction to MinIO

MinIO Wikipedia

What is MinIO?

MinIO

We are back with another article for the HackerNoon Company of the Week series! Every week we feature an awesome tech brand from our tech company database, making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike.This week, we present our business blogging partnerMinIO - a high-performance, S3 compatible object store, built for large scale AI/ML, data lake and database workloads. It runs on-prem and on any cloud (public or private) and from the data center to the edge. MinIO also offers features like data replication, lifecycle management and high availability, making it a top choice for modern cloud-native applications. Wanna be part of HackerNoon’s evergreen tech company database? Request Your Tech Company Page on HackerNoon! Meet MinIO: #FunFact Do you know that MinIO is open source under GNU AGPL v3? Open source is the key driver of technological innovation and here's how the company shares its support and belief in the role of open source in AI revolution:"Embracing a philosophy of extreme open innovation, companies that truly participate in open-source software development challenge traditional notions of competitive advantage by acknowledging that not all good code or great ideas reside within their organization. This shift supports the argument that shared innovations within the open-source ecosystem lead to faster market growth, providing even smaller software firms with more limited R&D funds the opportunity to benefit from R&D spillovers present in open-source software. This is because, in contrast to traditional outsourcing, open innovation enhances internal resources by leveraging the collective intelligence of the community, without diminishing internal R&D efforts. Meaning that open-source software companies don’t have to sacrifice their budgets to pursue thought leadership and code outside their organization." MinIO’s Top Stories on HackerNoon MinIO has published 43 stories via HackerNoon Business Blogging Program, and has generated 2 years 23 days and 29 minutes of reading time! Here are our top 3 favorite stories by MinIO: Open-Source: The Next Step in AI Revolution
Quantum State: How Two Things Can Be True at the Same Time
Analyzing the Pros, Cons, and Risks of LLMs We are back with another article for the HackerNoon Company of the Week series ! Every week we feature an awesome tech brand from our tech company database , making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. This week, we present our business blogging partner MinIO - a high-performance, S3 compatible object store, built for large scale AI/ML, data lake and database workloads. It runs on-prem and on any cloud (public or private) and from the data center to the edge. MinIO also offers features like data replication, lifecycle management and high availability, making it a top choice for modern cloud-native applications. Company of the Week series Company of the Week tech company database tech company database MinIO MinIO Wanna be part of HackerNoon’s evergreen tech company database? Request Your Tech Company Page on HackerNoon! Wanna be part of HackerNoon’s evergreen tech company database? Request Your Tech Company Page on HackerNoon ! Request Your Tech Company Page on HackerNoon Request Your Tech Company Page on HackerNoon Meet MinIO: #FunFact Do you know that MinIO is open source under GNU AGPL v3? Open source is the key driver of technological innovation and here's how the company shares its support and belief in the role of open source in AI revolution : "Embracing a philosophy of extreme open innovation, companies that truly participate in open-source software development challenge traditional notions of competitive advantage by acknowledging that not all good code or great ideas reside within their organization. This shift supports the argument that shared innovations within the open-source ecosystem lead to faster market growth, providing even smaller software firms with more limited R&D funds the opportunity to benefit from R&D spillovers present in open-source software. This is because, in contrast to traditional outsourcing, open innovation enhances internal resources by leveraging the collective intelligence of the community, without diminishing internal R&D efforts. Meaning that open-source software companies don’t have to sacrifice their budgets to pursue thought leadership and code outside their organization." role of open source in AI revolution role of open source in AI revolution "Embracing a philosophy of extreme open innovation, companies that truly participate in open-source software development challenge traditional notions of competitive advantage by acknowledging that not all good code or great ideas reside within their organization. This shift supports the argument that shared innovations within the open-source ecosystem lead to faster market growth, providing even smaller software firms with more limited R&D funds the opportunity to benefit from R&D spillovers present in open-source software. This is because, in contrast to traditional outsourcing, open innovation enhances internal resources by leveraging the collective intelligence of the community, without diminishing internal R&D efforts. Meaning that open-source software companies don’t have to sacrifice their budgets to pursue thought leadership and code outside their organization." MinIO’s Top Stories on HackerNoon MinIO has published 43 stories via HackerNoon Business Blogging Program , and has generated 2 years 23 days and 29 minutes of reading time! Here are our top 3 favorite stories by MinIO: MinIO MinIO HackerNoon Business Blogging Program HackerNoon Business Blogging Program Open-Source: The Next Step in AI Revolution Quantum State: How Two Things Can Be True at the Same Time Analyzing the Pros, Cons, and Risks of LLMs Open-Source: The Next Step in AI Revolution Open-Source: The Next Step in AI Revolution Quantum State: How Two Things Can Be True at the Same Time Quantum State: How Two Things Can Be True at the Same Time Analyzing the Pros, Cons, and Risks of LLMs Analyzing the Pros, Cons, and Risks of LLMs

Meet MinIO: The HackerNoon Company of the Week

মিট মিনিও: সপ্তাহের হ্যাকারনুন কোম্পানি

Lernen Sie MinIO kennen: Das HackerNoon-Unternehmen der Woche

Conozca MinIO: la empresa de la semana de HackerNoon

Rencontrez MinIO : l'entreprise HackerNoon de la semaine

मिलिए मिनियो से: इस सप्ताह की हैकरनून कंपनी

MinIO のご紹介: 今週の HackerNoon 企業

MinIO를 만나보세요: 금주의 HackerNoon 회사

Conheça MinIO: a empresa HackerNoon da semana

Встречайте MinIO: компания недели HackerNoon

MinIO ile Tanışın: Haftanın HackerNoon Şirketi

Gặp gỡ MinIO: Công ty HackerNoon của tuần

认识 MinIO：本周 HackerNoon 公司

Event notifications in MinIO may not seem thrilling at first, but once you harness their power, they illuminate the dynamics within your storage buckets. Event notifications are critical components of a full-fledged, efficient object storage system. Webhooks are my personal . They are like a Swiss Army knife in the world of events, offering a universal solution to various challenges. favorite tool for integrating with MinIO MinIO's user-friendly UI provides seamless service , but we're diving deeper in this guide. We're building services from scratch in Python, using their client credentials, to walk you through the nuts and bolts of MinIO integration. integrations In our exploration, we'll focus on deploying using docker-compose, a method that offers streamlined and efficient orchestration. This approach will involve setting up a cohesive environment for MinIO and Flask, enabling them to interact seamlessly. By integrating the MinIO service with its appropriate credentials and configurations, we aim to create a systematic workflow that effectively demonstrates the practical application of managing and responding to MinIO bucket events. Once we set up client configurations and define the structure for response data, as always, the real fun begins. This demonstration will highlight several key aspects of how you can connect your MinIO client to a Flask application where the event notification data can further be processed. We want you to feel comfortable in developing your own event-driven systems with MinIO, so utilize the services we've provided in the resource hosted on . blog-assets/flask-webhook-event-notifications MinIO’s GitHub Page Prepare to dive into a world where data handling is both an art and a science, made simpler with MinIO. It's an opportunity to innovate, create and revolutionize the way your apps work with data. MinIO and Integrated Services MinIO's integration into the exemplifies its adaptability across various cloud technologies. Webhooks are pivotal, offering developers the flexibility to craft custom integrations, whether for managing data across diverse cloud platforms or for local home lab setups. Kubernetes ecosystem This guide goes beyond theoretical concepts, providing you with practical, executable code snippets to build your integrations. It's an invitation to explore the boundless potential of your creativity in leveraging MinIO event notifications. Laying the Groundwork for a Python Application with Docker The initial phase of our journey is dedicated to harnessing the power of Docker's containerization to craft a robust Python application environment. Our approach centers around deploying with Docker-compose, a method chosen for its simplicity and effectiveness. This choice is designed to cater to a wide range of developers, prioritizing ease of use and rapid deployment while ensuring a high level of functionality. Utilizing Docker-compose, we create a user-friendly, configuration-driven setup. This environment is perfect for those seeking quick deployment without sacrificing the depth of their project's capabilities. It provides a straightforward pathway to integrate advanced webhook features and fine-tune MinIO settings, catering to the specific requirements of your project. Each step we take in setting up this environment is crucial. It's not just about getting the services up and running; it's about understanding and leveraging components to create a comprehensive system. Developing your own systems can be the spark that ignites your innovation, enhancing your overall data management strategy, and eventually turns your raw data into actionable, insightful information. Deploying MinIO and Integrated Services Deploy with Docker-Compose: Flask App and MinIO We will begin by setting up a Python application and its environment. This involves deploying MinIO with docker compose and the services to be integrated. To set up MinIO with a Flask application, we will be using the command to clone the repository to your local environment: git minio/blog-assets git clone https://github.com/minio/blog-assets.git

cd flask-webhook-event-notifications

docker-compose up This will clone the repository from GitHub, navigate to the directory containing the file, and start the MinIO and Flask services. minio/blog-assets /flask-webhook-event-notifications/ docker-compose.yaml Directory Structure This docker-compose structure outlines two services and their respective configuration variables. For visualization purposes, I have provided a tree view of the desired directory structure here: /flask-webhook-event-notifications
├── Dockerfile
├── app
│       └── main.py
└── docker-compose.yaml Setting up Webhooks in MinIO Configuring a webhook in MinIO can be accomplished through various methods ranging from using the user-interface, by using (the MinIO client utility), or by way of scripting with various programming languages. mc MinIO supports a , including: , , , , , , , , , and . variety of external services for event notifications AMQP (RabbitMQ) MQTT NATS NSQ Elasticsearch Kafka MySQL PostgreSQL Redis webhook services Setting up MinIO to utilize these involves a series of well-defined steps, ensuring that your MinIO instance not only captures but also effectively communicates important event data as  an interactive, responsive part of your application ecosystem. event notifications Understanding the Data Structure of MinIO Event Notifications The S3 event notifications from MinIO include a , essential for a comprehensive understanding and effective management of events. Below I have listed some of the values found from within the event data: detailed JSON data structure : The unique identifier of the object in the bucket. Key : The object’s version identifier for integrity and version control. eTag : The size of the object in bytes, indicating its scale. Size : Ensures the events are processed in the exact sequence they occurred. Sequencer : The media type of the object, specifying how to handle or display the file. ContentType : User-defined metadata attached to the object, providing additional context. UserMetadata : Bucket Details : The unique identifier for the bucket in AWS. ARN (Amazon Resource Name) : The name of the bucket where the object is stored. Name : Information about the owner of the bucket. OwnerIdentity : Indicates the version of the S3 event notification schema used. s3SchemaVersion : Identifier for the specific notification configuration that triggered this event. ConfigurationId This structure is particularly effective for , enabling systematic logging, parsing and analysis of interactions with the MinIO bucket. Flask applications Setting Up MinIO for Webhooks and Event-Driven Operations After deploying the outlined above, continue by using the MinIO Client, , command-line utility. This setup involves creating an alias in MinIO, configuring the endpoint and setting bucket notifications. docker-compose.yaml mc We will be working inside of an interactive terminal for the container, which we can spawn by running a single command: “minio” docker exec -it minio /bin/sh The reason for running our mc commands from within this shell is because the Docker image already has installed and ready to go. minio/minio mc Once inside the container’s interactive terminal, the process of configuring MinIO for event notifications using the MinIO Client (mc) involves the following key steps: : The first step involves creating an alias for your MinIO server using the MinIO Client (mc). This alias is a shortcut to your MinIO server, allowing you to easily execute further mc commands without repeatedly specifying the server’s address and access credentials. This step  simplifies the management of your MinIO server through the client. Setting Up MinIO Alias mc alias set myminio http://localhost:9000 minio minio123 : Configure a new webhook service endpoint in MinIO. This setup is done using either environment variables or runtime configuration settings, where you define important parameters such as the endpoint URL, an optional authentication token for security, and client certificates for secure connections. Adding the Webhook Endpoint to MinIO mc admin config set myminio notify_webhook:1 endpoint="http://flaskapp:5000/minio-event" queue_limit="10" : Once you have configured the settings, restart your MinIO deployment to ensure the changes take effect. Restarting the MinIO Deployment mc admin service restart myminio Expect: Restart command successfully sent to myminio. Type Ctrl-C to quit or wait to follow the status of the restart process....Restarted myminio successfully in 1 seconds : The next step involves using the mc event add command. This command is used to add new bucket notification events, setting the newly configured Webhook service as the target for these notifications. Configuring Bucket Notifications mc event add myminio/mybucket arn:minio:sqs::1:webhook --event put,get,delete Expect: Successfully added arn:minio:sqs::1:webhook : Run this command to list the event assigned to myminio/mybucket: List Bucket Subscribed Events minio mc event list myminio/mybucket Expect: arn:minio:sqs::1:webhook s3:ObjectCreated:*,s3:ObjectAccessed:*,s3:ObjectRemoved:*  Filter: : Run this command to list the event assigned to myminio/mybucket in JSON format: List Bucket Assigned Events (in JSON) minio mc event list myminio/mybucket arn:minio:sqs::1:webhook --json Expect: { "status": "success", "id": "", "event": ["s3:ObjectCreated:","s3:ObjectAccessed:", "s3:ObjectRemoved:*"], "prefix": "", "suffix": "", "arn": "arn:minio:sqs::1:webhook"} The Structure of the Event Notification Data Received by Flask Depending on the services or integration you are building, you may need to identify the event_data from your Flask app, and this requires a good understanding of the data your event provides. {
  "s3": {
    "bucket": {
      "arn": "arn:aws:s3:::mybucket",
      "name": "mybucket",
      "ownerIdentity": {
        "principalId": "minio"
      }
    },
    "object": {
      "key": "cmd.md",
      "eTag": "d8e8fca2dc0f896fd7cb4cb0031ba249",
      "size": 5,
      "sequencer": "17A9AB4FA93B35D8",
      "contentType": "text/markdown",
      "userMetadata": {
        "content-type": "text/markdown"
      }
    },
    "configurationId": "Config",
    "s3SchemaVersion": "1.0"
  },
  "source": {
    "host": "127.0.0.1",
    "port": "",
    "userAgent": "MinIO (linux; arm64) minio-go/v7.0.66 mc/RELEASE.2024-01-11T05-49-32Z"
  },
  "awsRegion": "",
  "eventName": "s3:ObjectCreated:Put",
  "eventTime": "2024-01-12T17:58:12.569Z",
  "eventSource": "minio:s3",
  "eventVersion": "2.0",
  "userIdentity": {
    "principalId": "minio"
  },
  "responseElements": {
    "x-amz-id-2": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8",
    "x-amz-request-id": "17A9AB4FA9328C8F",
    "x-minio-deployment-id": "c3642fb7-ab2a-44a0-96cb-246bf4d18e84",
    "x-minio-origin-endpoint": "http://172.18.0.3:9000"
  },
  "requestParameters": {
    "region": "",
    "principalId": "minio",
    "sourceIPAddress": "127.0.0.1"
  }
} By following these steps, you can effectively utilize MinIO event notifications, significantly automating data workflow processes. For more detailed guidance and information, please refer to the MinIO documentation on and . Bucket Notifications Monitoring Bucket and Object Events command-line tool for a more detailed, scriptable setup. The blog post further rounds out your understanding of the topic by emphasizing the importance of properly configuring PostgreSQL in the MinIO UI and the significance of restarting the MinIO server for changes to take effect. If you're interested in setting up , take a look at , where I’ve covered MinIO's extensive configuration and management of data events. These configurations range from using the for a user-friendly graphical interface to the MinIO with PostgreSQL Streamlining Data Events with MinIO and PostgreSQL MinIO Console mc Developing a Webhook with Flask to Receive Event Notifications Following the deployment of our environment, we now shift our focus to the integration of MinIO with Python, a key aspect of our data handling and processing system. This integration is pivotal in creating a cohesive ecosystem, where MinIO can seamlessly collaborate with Flask. Importing the Necessary Packages In our demonstration code, we carefully select Python imports to ensure the application's functionality aligns with its intended purpose. The package creates the web server infrastructure, defining endpoints to handle incoming HTTP requests. Then the application can be coded to handle the MinIO event notifications in any desired manner. flask from flask import  Flask, jsonify, request These imports collectively form the foundation of the application, enabling it to receive and process MinIO event notifications. Flask Application and Event Handling Endpoint in Python A Flask application is instantiated, and an endpoint is set up to handle POST requests at the route . Flask is a micro web framework in Python, ideal for setting up web applications and API endpoints. /minio-event app = Flask(__name__)

@app.route('/minio-event', methods=['POST'])
def handle_minio_event():
    event_data = request.json
    app.logger.info(f"Received MinIO event: {event_data}")
    return jsonify(event_data), 200 The function in the Flask app processes POST requests containing MinIO event data, and returns the received from the MinIO event notification. handle_minio_event event_data This approach facilitates real-time processing and response to storage events, enabling dynamic interaction between the MinIO storage system and the Flask application. Integrating Services with MinIO Bucket Events via Python Scripting This blog post used MinIO and Python in a Docker environment to demonstrate the power and flexibility of MinIO bucket event notifications, and demonstrates a strategic approach for creating scalable, efficient event-driven applications. The use of Docker, with its containerization technology, stands out for enabling components like MinIO and Flask to work independently yet cohesively. Of course, this containerized cloud-native setup minimizes conflicts and dependencies, highlighting Docker and Docker containers significance in modern software architecture. In conclusion of our exploration of MinIO webhook event notifications, I am convinced that the synergy of a dynamic programming language and MinIO’s formidable strength presents an unparalleled toolkit. This pairing paves the way for boundless opportunities in application development. It empowers us to not only innovate and streamline but also to expand our capabilities with remarkable efficiency and adaptability. This guide has demonstrated the simplicity and efficacy of API development using Python, laying a solid foundation for ongoing innovation and meeting ever-changing demands. It underscores the adaptability that is essential for the continued advancement in both data management and the evolution of application development. This approach is not just a methodology; it’s a pathway to future-proofing our technological endeavors.

Using MinIO and Python to Create Scalable and Efficient Event-driven Applications

In the rapidly evolving world of data storage and processing, combining efficient cloud storage solutions with advanced AI capabilities presents a transformative approach to handling vast volumes of data. This article demonstrates a practical implementation using MinIO, Langchain and OpenAI’s GPT-3.5 model, focusing on summarizing documents stored in MinIO buckets. The Power of MinIO MinIO is open-source, high-performance object storage that is fully compatible with the Amazon S3 API. Known for its scalability, MinIO is ideal for storing unstructured data such as photos, videos, log files, backups and container images. It’s not just about storage; MinIO also offers features like data replication, lifecycle management and high availability, making it a top choice for modern cloud-native applications. Integrating Langchain and OpenAI , a Python-based tool, facilitates the interaction between document loaders and AI models. In our use case, we combine Langchain with OpenAI’s model to summarize documents from MinIO buckets. This setup exemplifies how AI can extract essential information from extensive data, simplifying data analysis and interpretation. For additional information and supporting materials related to this article such as notebooks and loaded documents, please visit the in the directory. Langchain gpt-3.5-turbo-1106 MinIO Github repository langchain-s3-minio Installing Langchain Before diving into the implementation, ensure you have Langchain installed. Install it via pip: pip install --upgrade langchain This will encapsulate all the required libraries we will be using for our S3 loaders and OpenAI model. Step 1: Langchain S3 Directory and File Loaders Initially, we focus on loading documents using Langchain's and . These loaders are responsible for fetching multiple and single documents from specified directories and files in MinIO buckets. S3DirectoryLoader S3FileLoader MinIO Configurations and Langchain S3 File Loader from langchain_community.document_loaders.s3_file import S3FileLoader

# MinIO Configuration for the public testing server
endpoint = 'play.min.io:9000'
access_key = 'minioadmin'
secret_key = 'minioadmin'
use_ssl = True

# Initialize and load a single document
file_loader = S3FileLoader(
    bucket='web-documentation',
    key='MinIO_Quickstart.md',
    endpoint_url=f'http{"s" if use_ssl else ""}://{endpoint}',
    aws_access_key_id=access_key,
    aws_secret_access_key=secret_key,
    use_ssl=use_ssl
)

document = file_loader.load() Python Langchain Example - S3 File Loader Langchain S3 Directory Loader: from langchain_community.document_loaders.s3_directory import S3DirectoryLoader

# Initialize and load documents
directory_loader = S3DirectoryLoader(
    bucket='web-documentation',
    prefix='',
    endpoint_url=f'http{"s" if use_ssl else ""}://{endpoint}',
    aws_access_key_id=access_key, 
    aws_secret_access_key=secret_key, 
    use_ssl=use_ssl
)

documents = directory_loader.load() Python Langchain Example - S3 Directory Loader Summarizing with OpenAI Step 2: After loading the documents, we use OpenAI's GPT-3.5 model (which are included in the Langchain library via ) to generate summaries. This step illustrates the model's capability to understand and condense the content, providing quick insights from large documents. ChatOpenAI To access the OpenAI API, you can acquire an API key by visiting the . Once you have the key, integrate it into the code below, to harness the power of GPT-3.5 for document summarization. OpenAI platform Code Example for Document Summarization: from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnableLambda
import os

# Set your OpenAI API key
os.environ['OPENAI_API_KEY'] = 'your-openai-api-key'
model = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-1106")

prompt = ChatPromptTemplate.from_template(
    "Summarize the following document '{document_name}':{context}Please provide the summary and key points."
)

loaded_documents = [documents, document]  # From S3 Loaders
flattened_documents = [doc for sublist in loaded_documents for doc in sublist] 

for loaded_document in flattened_documents:
    document_text = loaded_document.page_content
    document_name = getattr(loaded_document, 'name', 'Unknown Document')  # Assuming each document has a 'name' attribute
    chain = (
        RunnableLambda(lambda x: {"context": document_text, "document_name": document_name})
        | prompt
        | model
        | StrOutputParser()
    )
    summary = chain.invoke(None)
    print("Summary:", summary) Python Langchain Example - Summarizing Documents with OpenAI API Below is the output from running this demo, and is a result of integrating LangChain with OpenAI’s GPT-3.5 and MinIO S3 storage; the output has been shortened for demonstrative purposes: Summary: The document is a quickstart guide for MinIO, a high-performance object storage system that is compatible with Amazon S3. It explains how to run MinIO on bare metal hardware or in containers. For Kubernetes environments, it recommends using the MinIO Kubernetes Operator. The key points are:

- MinIO is a high-performance object storage system.
- It is released under the GNU Affero General Public License v3.0.
- MinIO is API compatible with Amazon S3.
- It can be used to build high-performance infrastructure for machine learning, analytics, and application data workloads.
- The guide provides quickstart instructions for running MinIO on bare metal hardware or in containers.
- For Kubernetes environments, the MinIO Kubernetes Operator is recommended. Response from OpenAI API This method highlights an interesting way to load documents from S3 storage into an LLM using the Langchain framework to process them, while OpenAI’s GPT-3.5 model generates a concise summary and key points of the which is fetched from the server. The use of AI to analyze and condense extensive documentation, provides users with a quick and thorough understanding of essential aspects like installation, server configuration, SDKs and other MinIO features. It showcases the capability of AI in extracting and presenting critical information from comprehensive data sources. MinIO_Quickstart.md play.min.io Loading Documents from MinIO Buckets with Langchain The integration of MinIO, Langchain and OpenAI offers a compelling toolset for managing large data volumes. While Langchain's S3 loaders, S3DirectoryLoader and S3FileLoader, play an important role in retrieving documents from MinIO buckets, they are solely for loading data into Langchain. These loaders do not perform actions related to uploading data into buckets. For tasks like uploading, modifying or managing bucket policies, the is the appropriate tool. This SDK provides a comprehensive set of functionalities for interacting with MinIO storage, including file uploads, bucket management and more. For additional information, please see , . MinIO Python SDK Quickstart Guide — MinIO Object Storage for Linux Python Client API Reference — MinIO Object Storage for Linux While Langchain streamlines the process of fetching and processing data using AI models, the heavy lifting of data management within the MinIO buckets is dependent on the MinIO Python SDK. This is an important distinction that must be understood by developers and data engineers building efficient, AI-integrated storage solutions. For a thorough understanding of MinIO's capabilities and how to utilize its Python SDK for various storage operations, refer to MinIO's official . documentation By using MinIO object storage as the primary data repository for AI and ML processes, you can simplify your data management pipeline. MinIO excels as a one-stop solution for , which is crucial for effective AI and ML operations. This streamlined approach reduces complexity and overhead, potentially accelerating insights by ensuring swift access to data. storing, managing, and retrieving large datasets For those interested in delving deeper into the integration of MinIO with Langchain to enhance LLM tool-use, the article “ ” offers a comprehensive exploration of the subject. Developing Langchain Agents with MinIO SDK for LLM Tool-Use Good luck in your development endeavors! We hope continues to play a key role in your AI/ML journey. Reach out to us on and share your insights and discoveries! MinIO Slack

Managing Large Data Volumes With MinIO, Langchain and OpenAI

The modern data landscape demands a new breed of infrastructure – one that seamlessly integrates structured and unstructured data, scales effortlessly, and empowers efficient AI/ML workloads. This is where come in, providing a central hub for all your data needs. However, building and managing an effective data lake can be complex. modern datalakes This blog post dives deep into three powerful tools that can optimize your current approach: , and MinIO. The steps below will walk you through how these services seamlessly combine to create a robust, cloud-native data lake architecture specifically optimized for AI/ML workloads. Apache Iceberg Tabular, What is Tabular? Tabular is a data platform created by the original creators of Apache Iceberg. It is designed to provide an independent, universal storage platform that connects to any compute layer, eliminating data vendor lock-in. This feature is critical to the modern data stack, it allows users to pick best-in-class compute and storage tools without being forced into a particular vendor's aged and or mismatched tool set. In an of MinIO and Iceberg and can be enhanced by Tabular. Tabular can be used to manage and query Iceberg data stored in MinIO, allowing for the storage and management of structured data in a scalable, high-performance, and cloud-native manner. These Kubernetes native, components work together smoothly with very little friction and build on each other’s capabilities to perform at scale. architecture Why S3FileIO instead of Hadoop's file-io? This implementation leverages Iceberg's S3FileIO. S3FileIO is considered better than Hadoop's file-io for several reasons. Some of which we’ve already discussed : elsewhere : Iceberg's S3FileIO is designed to work with cloud-native storage. Optimized for Cloud Storage Iceberg uses an to distribute files across multiple prefixes in a MinIO bucket, which helps to minimize throttling and maximize throughput for S3-related IO operations. Improved Throughput and Minimized Throttling: ObjectStoreLocationProvider Iceberg has been to fully leverage strict consistency by eliminating redundant consistency checks that could impact performance. Strict Consistency: updated Iceberg's S3FileIO implements a algorithm, which uploads data files parts in parallel as soon as each part is ready, reducing local disk usage and increasing upload speed. Progressive Multipart Upload: progressive multipart upload Iceberg allows for for S3 API writes to ensure the integrity of uploaded objects, which can be enabled by setting the appropriate catalog property. Checksum Verification: checksum validations Iceberg supports adding to objects during write and delete operations with the S3 API, which can be useful for cost tracking and management. Custom Tags: custom tags The FileIO interface in Iceberg does not require as strict guarantees as a Hadoop-compatible FileSystem, which allows it to that could otherwise degrade performance. Avoidance of Negative Caching: avoid negative caching In contrast, Hadoop's S3A FileSystem, which was used before S3FileIO, does not offer the same level of optimization for cloud storage. All this to say: Don’t hobble your future-facing data lake infrastructure with the trappings of the past. Prerequisites Before you begin, make sure your system meets the following requirements: Docker Docker Compose If you're starting from scratch, you can install both using the installer for your specific platform. It is often easier than downloading Docker and Docker Compose separately. Verify if Docker is installed by running the following command: Docker Desktop docker-compose --version Getting started To get started, clone or copy the YAML file in Tabular’s . You just need the YAML for this tutorial. Feel free to explore the rest of the repository at a later time. git repository Breaking it Down The provided YAML file is a Docker Compose configuration file. It defines a set of services and their configurations for a multi-container Docker application. In this case, there are two services: Spark-Iceberg and MinIO. Let's break down each section: 1. Spark-Iceberg Service: spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    build: spark/
    networks:
      iceberg_net:
    depends_on:
      - rest
      - minio
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    ports:
      - 8888:8888
      - 8080:8080
      - 10000:10000
      - 10001:10001
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      iceberg_net:
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000 Specifies the Docker image to use for the spark-iceberg service. In this case, it uses the tabulario/spark-iceberg:latest image. image: Specifies that the spark-iceberg service depends on the rest and minio services. depends_on: Assigns a specific name (spark-iceberg) to the container. container_name: Sets environment variables for the container, including Spark and AWS credentials. environment: Mounts local directories (./warehouse and ./notebooks) as volumes inside the container. volumes: Maps container ports to host ports for accessing Spark UI and other services. ports: 2. Minio Service: minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=minio
    networks:
      iceberg_net:
        aliases:
          - warehouse.minio
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"] Specifies the Docker image for the MinIO service. image: Assigns a specific name (MinIO) to the container. container_name: Sets environment variables for configuring MinIO, including root user credentials. environment: Maps container ports to host ports for accessing MinIO UI. ports: Specifies the command to start the MinIO server with specific parameters. command: Another aspect of the MinIO service is , MinIO’s command line tool. mc mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      " Specifies that the mc service depends on the MinIO service. depends_on: Specifies the Docker image for the mc service. image: Assigns a specific name (mc) to the container. container_name: Sets environment variables for configuring the MinIO client. environment: Defines the entry point command for the container, including setup steps for the MinIO client. entrypoint: /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      " This sequence of commands essentially performs the following tasks: Removes the existing warehouse directory and its contents from the MinIO server. Creates a new bucket named warehouse. Sets the access policy of the warehouse bucket to public. This Docker Compose file orchestrates a multi-container environment with services for Spark, PostgreSQL, MinIO. It sets up dependencies, environment variables, and commands necessary for running the services together. The services work in tandem to create a development environment for data processing using Spark and Iceberg with MinIO as the object storage backend. Starting Up In a terminal window, cd into the tabular-spark-setup directory in the repository and run the following command: docker-compose up Login into MinIO at with the credentials to see that the warehouse bucket has been created. http://127.0.0.1:9001 admin:password Once all the containers are up and running, you can access your Jupyter Notebook server by navigating to http://localhost:8888 Run one of the sample notebooks and return to MinIO at to see your warehouse populated with data. http://127.0.0.1:9001 Building Your Modern Datalake This tutorial on building a modern datalake with Iceberg, Tabular, and MinIO is just the beginning. This powerful trio opens doors to a world of possibilities. With these tools, you can seamlessly integrate and analyze all your data, structured and unstructured, to uncover hidden patterns and drive data-driven decisions that fuel innovation. Leverage the efficiency and flexibility of this architecture in production to expedite your AI/ML initiatives and unlock the true potential of your machine learning models, accelerating your path to groundbreaking discoveries. Reach out to us at or on our channel if you have any questions as you build. hello@min.io Slack

Building Effective Modern Data Architectures with Iceberg, Tabular and MinIO

আইসবার্গ, ট্যাবুলার এবং মিনিও সহ কার্যকরী আধুনিক ডেটা আর্কিটেকচার তৈরি করা

Erstellen effektiver moderner Datenarchitekturen mit Iceberg, Tabular und MinIO

Creación de arquitecturas de datos modernas y eficaces con Iceberg, Tabular y MinIO

Construire des architectures de données modernes et efficaces avec Iceberg, Tabular et MinIO

आइसबर्ग, टेबुलर और मिनियो के साथ प्रभावी आधुनिक डेटा आर्किटेक्चर का निर्माण

Iceberg、Tabular、MinIO を使用した効果的な最新データ アーキテクチャの構築

Iceberg, Tabular 및 MinIO를 사용하여 효과적인 최신 데이터 아키텍처 구축

Construindo arquiteturas de dados modernas e eficazes com Iceberg, Tabular e MinIO

Создание эффективных современных архитектур данных с помощью Iceberg, Tabular и MinIO

Iceberg, Tabular ve MinIO ile Etkili Modern Veri Mimarileri Oluşturma

Xây dựng kiến trúc dữ liệu hiện đại hiệu quả với Iceberg, Tabular và MinIO

使用 Iceberg、Tabular 和 MinIO 构建有效的现代数据架构

deployments come in all shapes and sizes. We support bare metal installs on any version of , containerized installs on any version of (including Red Hat OpenShift), and installs just about anywhere you can deploy a small lightweight single binary. But with flexibility comes the inevitability that edge case issues will require debugging. MinIO Linux Kubernetes In this blog post, we’ll show you how to debug a MinIO install running in Kubernetes and also some of the common issues you might encounter when doing bare metal installation and how to rectify them. Kubernetes Debugger Pod There are a few ways to access the MinIO API running inside a Kubernetes cluster. We can use or set up a listening on to be able to access the API. Both of these methods offer a way to access the service from outside the network, but they do come with one major downside: You can only access the Service that the NodePort or Port Forwarding references on an available port (not the usual configuration for the application). For example, you have to access the MinIO API, usually found on port , via a randomly assigned port. kubectl port-forwarding Service NodePort 9000 3xxxx What if I told you there was a better way – and it's not novel? When debugging applications you want to have full access to the native run-time environment so you can use various tools to troubleshoot and debug the cluster. One way to do that is launching a “busybox” style pod and installing all the required tools needed to debug the application. First launch a into the same namespace as your MinIO install. In order to do this create a yaml file called with the following yaml. Pod debugger-pod.yaml apiVersion: v1

kind: Pod

metadata:

  name: mc

  labels:

    app: mc

spec:

  containers:

  - image: minio/mc:latest

    command:

      - "sleep"

      - "604800"

    imagePullPolicy: IfNotPresent

    name: mc

  restartPolicy: Always The above Pod configuration is pulling the image for MinIO utility. In order to ensure the pod doesn’t just launch and then exit, we’ve added a command. mc sleep Once the yaml is saved apply the configuration to the Kubernetes namespace where the MinIO cluster is running kubectl apply -f debugger-pod.yaml Once the pod is up and running access it via shell $ kubectl exec -i -t -n default mc -c mc -- sh -c "(bash || ash || sh)"

[root@mc /]# Then with you can access the MinIO cluster mc [root@mc /]# mc alias set myminio    --insecure

Added `myminio` successfully. Now that we have a debugger pod up and running, you can perform action on the cluster directly within the same network. For example, if the replication was broken due to an site going offline or a hardware failure, you can resync any pending objects to be replicated using the following command [root@mc /]# mc admin replicate resync start minio1 minio2


[root@mc /]# mc admin replicate resync status minio1 minio2


✔ ✔ ✔

ResyncID: 		 2248d1d1-633f-4d61-b938-d8ea0b9b2d31

Status:   		 Completed

Objects:  		 2225

Versions: 		 2225

FailedObjects:     0

Throughput:   	 5.3 MiB/s

IOPs:     		 124.23 objs/s

Transferred:  	 94 MiB

Elapsed:  		 17.909833202s

CurrObjName:  	 testbucket/web-app/tsconfig.json Another reason you would run the debugger pod is if there are some file system permissions or invalid groups configurations in your pod, you can update them using the debugger pod [root@mc /]# chgrp -R 1000780050 .minio.sys/ The above debugging method can also be used in bare metal environments. For instance you can launch a busybox or bastion node with installed and follow the same instructions as above. mc Debugging Bare Metal Bare metal Linux installs are the most straightforward. In fact it just takes a few commands to get MinIO installed and running with SystemD. For details, please see . Configuring MinIO with SystemD Once in a great while, bare metal installs go awry. Here are some of the (not-so-common) pitfalls that we are asked about  in or . These pitfalls are not hardware or operating specific but can be useful to know in any kind of environment. SUBNET Slack File Permission One of the most common pitfalls is the file permissions of the MinIO binary and the configuration file. If this occurs, when you start MinIO using SystemD you will see and here is the full stack trace Assertion failed for MinIO. # systemctl status minio.service

● minio.service - MinIO

     Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: enabled)

     Active: inactive (dead)

     Assert: start assertion failed at Tue 2023-12-26 18:21:38 PST; 8s ago

             AssertFileIsExecutable=/usr/local/bin/minio was not met

       Docs: https://docs.min.io


Dec 26 18:13:37 minio1 systemd[1]: minio.service: Starting requested but asserts failed.

Dec 26 18:17:50 minio1 systemd[1]: Assertion failed for MinIO.

Dec 26 18:21:38 minio1 systemd[1]: minio.service: Starting requested but asserts failed.

Dec 26 18:21:38 minio1 systemd[1]: Assertion failed for MinIO. This could be caused by a number of reasons, let's go down the list and check for each of them. : The binary, in this example located at needs to have permission for user and group, respectively. MinIO Binary /usr/local/bin/minio root:root # ll /usr/local/bin/minio

total 93804

-rwxr-xr-x 1 root    root    96018432 Nov 15 16:35  minio* : The MinIO service needs to run under a unique Linux user and group for security purposes, never run as a user. By default we use for the user and group names. In the SystemD service config file you should see something like this MinIO Service User and Group root minio-user User=minio-user

Group=minio-user : The directory where MinIO data will be stored needs to be owned by or whichever user you decide to run the MinIO service as above. MinIO Data Dir minio-user:minio-user # ls -l /mnt

total 4

drwxrwxr-x 2 minio-user minio-user 4096 Dec 27 09:58 data : Both the config files should have permissions for user and group like so SystemD and MinIO config root:root # ls -l /etc/default/minio

-rw-r--r-- 1 root root 1330 Dec 27 09:52 /etc/default/minio


# ls -l /etc/systemd/system/minio.service

-rw-r--r-- 1 root root 941 Dec 26 17:13 /etc/systemd/system/minio.service : The entire install process should be run as . You can also try if your user has permissions but the recommendation is to run as root as the install needs to place files in a bunch of places that only the user can access. Your bash prompt should have a and not a like so Run as Root root sudo root # $ vs # $ If none of the above work, the best approach is to remove the app, directories and configs and start a fresh install as a root user. Port conflict Another common issue related to deleted files which still hold on to the process, which causes port conflicts. Even when a service is not running, you may be unable to start a new service on the existing port or the service that is running will misbehave (such as not allowing you to login). # lsof -n | grep (deleted)

COMMAND PID USER FD TYPE DEVICE   SIZE 	NODE NAME

nginx   	13423 root 5u  	REG 253,3   42949672960 	17 (deleted)

minio 	13423 minio 6u    REG 253,3 	0         	      18 (deleted)

minio       13423 minio 7u    REG 253,3 	0         	      19 (deleted) You might see errors such as those below on a MinIO install Login Failed net::ERR_FAILED 500 Internal Server Error 401 Unauthorized The screenshot above shows an internal server error and an unauthorized error. While looking at the surface it looks unclear what has caused this error, we can debug with a little linux knowledge what to look for that could cause this, let's take a gander. There are several ways to debug this issue, first lets check to see if multiple MinIO processes are running on the same node # ps aux | grep -i minio

minio-u+    5048  0.3  1.7 1594008 144384 ?      Ssl  11:03   0:01 /usr/local/bin/minio server --console-address :9001 /mnt/data/disk1/minio


minio-u+    9276  0.3  1.7 1594208 144301 ?      Ssl  11:25   0:01 /usr/local/bin/minio server --console-address :9001 /mnt/data/disk1/minio As we can see above there are 2 MinIO processes running. Start by killing the process that is older or has been running the longest, in this case it seems to be the process ID . 5048 kill -9 5048 Sometimes even after killing the process the service might still not start or might still get hung up because it has reserved a process number but not let it go. This can be caused by files that have been deleted but are still being tracked by the operating system. You can find the deleted files via LSOF lsof -n | grep '(deleted)' Last but not least, if there are no deleted files left over or hung processes and if everything looks absolutely clean, the last resort is to quickly restart the node. This is a no-nonsense method that  shuts down and clears any pending files and processes so you start a fresh install. SUBNET to the rescue Although rare, installation edge cases will always exist. MinIO customers know that they have nothing to worry about because they can quickly message our engineers – who have written the code – via the portal. We've seen almost everything under the sun, so while the issue might look cryptic or mind boggling at first glance, we'll put our years of expertise debugging installations in many varied environments to work and help you in a jiffy. SUBNET If you have any questions on troubleshooting and debugging installs be sure to reach out to us on ! MinIO Slack

Pro Tips for Debugging MinIO: Overcoming Common Issues in Kubernetes and Bare Metal Environments

MinIO ডিবাগ করার জন্য প্রো টিপস: কুবারনেটস এবং বেয়ার মেটাল পরিবেশে সাধারণ সমস্যাগুলি অতিক্রম করা

Profi-Tipps zum Debuggen von MinIO: Überwindung häufiger Probleme in Kubernetes- und Bare-Metal-Umgebungen

Consejos profesionales para depurar MinIO: superar problemas comunes en Kubernetes y entornos bare metal

Conseils de pro pour déboguer MinIO : surmonter les problèmes courants dans les environnements Kubernetes et Bare Metal

मिनियो डिबगिंग के लिए प्रो टिप्स: कुबेरनेट्स और बेयर मेटल वातावरण में आम समस्याओं पर काबू पाना

MinIO デバッグのプロのヒント: Kubernetes とベアメタル環境での一般的な問題の克服

MinIO 디버깅을 위한 전문가 팁: Kubernetes 및 베어메탈 환경의 일반적인 문제 극복

Dicas profissionais para depuração do MinIO: superando problemas comuns em ambientes Kubernetes e Bare Metal

Советы для профессионалов по отладке MinIO: решение распространенных проблем в Kubernetes и средах с голым железом

MinIO'da Hata Ayıklamaya Yönelik Profesyonel İpuçları: Kubernetes ve Bare Metal Ortamlarında Yaygın Sorunların Aşılması

Mẹo chuyên nghiệp để gỡ lỗi MinIO: Khắc phục các sự cố thường gặp trong môi trường Kubernetes và Bare Metal

调试 MinIO 的专业技巧：解决 Kubernetes 和裸机环境中的常见问题

Leveraging MinIO and Apache Tika for Automated Text Extraction and Analysis

স্বয়ংক্রিয় পাঠ্য নিষ্কাশন এবং বিশ্লেষণের জন্য MinIO এবং Apache Tika ব্যবহার করা

Nutzung von MinIO und Apache Tika zur automatischen Textextraktion und -analyse

Aprovechando MinIO y Apache Tika para la extracción y el análisis de texto automatizados

Tirer parti de MinIO et Apache Tika pour l’extraction et l’analyse automatisées de texte

स्वचालित पाठ निष्कर्षण और विश्लेषण के लिए मिनियो और अपाचे टिका का लाभ उठाना

MinIO と Apache Tika を活用した自動テキスト抽出と分析

자동화된 텍스트 추출 및 분석을 위해 MinIO 및 Apache Tika 활용

Aproveitando MinIO e Apache Tika para extração e análise automatizada de texto

Использование MinIO и Apache Tika для автоматического извлечения и анализа текста

Otomatik Metin Çıkarma ve Analizi için MinIO ve Apache Tika'dan Yararlanma

Tận dụng MinIO và Apache Tika để trích xuất và phân tích văn bản tự động

利用 MinIO 和 Apache Tika 进行自动文本提取和分析

Backing Up Weaviate with MinIO S3 Buckets to Achieve Strategic Enhancement to Data Management

The response to our previous post, , was extraordinary - we’ve fielded dozens of calls from enterprises asking us for repatriation advice. We have aggregated those responses into this new post, where we dig a little deeper into the costs and savings associated with repatriation to make it easier for you to put together your own analysis. Migration of data is a daunting task for many. In practice, they target new data to come to MinIO and take their sweet time to migrate old data from the cloud or leave it in place and not grow. How to Repatriate From AWS S3 to MinIO Repatriation Overview To repatriate data from AWS S3, you will follow these general guidelines: Determine the specific buckets and objects that need to be repatriated from AWS S3. Make sure you understand business needs and compliance requirements on a bucket-by-bucket basis. Review Data Requirements: You’ve already decided to repatriate to MinIO, now you can choose to run MinIO in an on-premises data center or at another cloud provider or colocation facility. Using the requirements from #1, you will select hardware or instances for forecasted storage, transfer and availability needs. Identify Repatriation Destination: Plan and execute the transfer of data from AWS S3 to MinIO. Simply use MinIO's built-in Batch Replication or mirror using the MinIO Client (see for details). There are several additional methods you can use for data transfer, such as using AWS DataSync, AWS Snowball or , or directly using AWS APIs. Data Transfer: How to Repatriate From AWS S3 to MinIO TD SYNNEX data migration Ensure that appropriate access controls and permissions are set up for the repatriated data on a per-bucket basis. This includes IAM and bucket policies for managing user access, authentication, and authorization to ensure the security of the data. Data Access and Permissions: It is critical to preserve the object lock retention and legal hold policies after the migration. The target object store has to interpret the rules in the same way as Amazon S3. If you are unsure, ask for the on the target object store implementation. Object Locks: Cohasset Associates Compliance Assessment Define and implement a data lifecycle management strategy for the repatriated data. This includes defining retention policies, backup and recovery procedures, and data archiving practices on a per-bucket basis. Data Lifecycle Management: Validate the transferred data to ensure its integrity and completeness. Perform necessary checks and tests to ensure that the data has been successfully transferred without any corruption or loss. After the transfer, the object name, ETag and metadata, checksum and the number of objects all match between the source and destination. Data Validation: The good news is that if you follow cloud-native principles to build your applications, then all you will have to do is reconfigure them for the new MinIO endpoint. However, if your applications and workflows were designed to work with the AWS ecosystem, make the necessary updates to accommodate the repatriated data. This may involve updating configurations, reconfiguring integrations or in some cases modifying code. Update Applications and Workflows: Continuously monitor and optimize the repatriated data environment to ensure optimal performance, cost-efficiency, and adherence to data management best practices. Monitor and Optimize: Repatriation Steps There are many factors to consider when budgeting and planning for cloud repatriation. Fortunately, our engineers have done this with many customers and we’ve developed a detailed plan for you. We have customers that have repatriated everything from a handful of workloads to hundreds of petabytes. The biggest planning task is to think through choices around networking, leased bandwidth, server hardware, archiving costs for the data not selected to be repatriated, and the human cost of managing and maintaining your own cloud infrastructure. Estimate these costs and plan for them. Cloud repatriation costs will include data egress fees for moving the data from the cloud back to the data center. These fees are intentionally high enough to compel cloud lock-in. Take note of these high egress fees - they substantiate the economic argument to leave the public cloud because, as the amount of data you manage grows, the egress fees increase. Therefore, if you’re going to repatriate, it pays to take action sooner rather than later. We’re going to focus on data and metadata that must be moved – this is eighty percent of the work required to repatriate. Metadata includes bucket properties and policies (access management based on access/secret key, lifecycle management, encryption, anonymous public access, object locking and versioning). Let’s focus on data (objects) for now. For each namespace you want to migrate, take inventory of the buckets and objects you want to move. It is likely that your DevOps team already knows which buckets hold important current data. You can also use . At a high level, this will look something like: Amazon S3 Inventory Namespace Total Buckets Total Object Count Total Object Size (GB) Daily Total Upload (TB) Daily Total Download (TB) ns-001 166 47,751,258 980,014.48 50.04 14.80 ns-002 44 24,320,810 615,033.35 23.84 675.81 ns-002 648 88,207,041 601,298.91 328.25 620.93 ns-001 240 68,394,231 128,042.16 62.48 12.45 The next step is to list, by namespace, each bucket and its properties for every bucket you’re going to migrate. Note the application(s) that store and read data in that bucket. Based on usage, classify each bucket as hot, warm or cold tier data. In an abridged version, this will look something like Bucket Name Properties App(s) Hot/Warm/Cold Tier A Copy and paste JSON here Spark, Iceberg, Dremio Hot B Copy and paste JSON here Elastic Warm C Copy and paste JSON here Elastic (snapshots) Cold You have some decisions to make about data lifecycle management at this point and pay close attention because here’s a great way to save money on AWS fees. Categorize objects in each bucket as hot, warm or cold based on how frequently they are accessed. A great place to save money is to migrate cold tier buckets directly to S3 Glacier – there’s no reason to incur egress fees to download just to upload again. Depending on the amount of data you’re repatriating, you have a few options to choose how to migrate. We recommend that you load and work with new data on the new MinIO cluster while copying hot and warm data to the new cluster over time. The amount of time and bandwidth needed to copy objects will, of course, depend on the number and size of the objects you’re copying. Here’s where it will be very helpful to calculate the total data that you’re going to repatriate from AWS S3. Look at your inventory and total the size of all the buckets that are classified as hot and warm. Total Hot and Warm Tier Data = 1,534,096.7 GB Available bandwidth = 10 Gbps Minimum Transfer Time required (total object size / available bandwidth) = 14.2 days Calculate data egress fees based on the above total. I’m using , but your organization may qualify for a discount from AWS. I’m also using 10 Gbps as the connection bandwidth, but you may have more or less at your disposal. Finally, I’m working from the assumption that one-third of S3 data will merely be shifted to S3 Glacier Deep Archive. list price Total Data Tiered to S3 Glacier = 767,048.337 GB S3 to S3 Glacier transfer fees ($0.05/1000 objects) = $3,773.11 S3 Glacier Deep Archive monthly storage fee = $760 Don’t forget to budget for S3 Glacier Deep Archive usage moving forward. Total Data to be Transferred = 1,534,096.7 GB First 10 TB at $0.09/GB = $900 Next 40 TB at $0.085/GB = $3,400 Next 100 TB at $0.07/GB = $70,000 Additional over 150 TB at $0.05/GB = $69,205 Total Egress Fees = $143,504 For the sake of simplicity, the above calculation includes neither the fee for per object operations ($0.40/1m) nor the cost of LISTing ($5/1m). For very large repatriation projects, we can also compress objects before sending them across the network, saving you some of the cost of egress fees. Another option is to use AWS Snowball to transfer objects. Snowball devices are each 80TB, so we know up front that we need 20 of them for our repatriation effort. The per-device fee includes 10 days of use, plus 2 days for shipping. Additional days are available for $30/device. 20 Snowball Devices Service Fee ($300 ea) = $6,000 R/T shipping (3-5 days at $400/device) = $8,000 S3 data out ($0.02/GB) = $30,682 Total Snowball Fees = $38,981.93 AWS will charge you standard request, storage, and data transfer rates to read from and write to AWS services including and . There are further considerations when working with . For S3 export jobs, data transferred to your Snow Family device from S3 are billed at standard S3 charges for operations such as LIST, GET, and others. You are also charged standard rates for Amazon CloudWatch Logs, Amazon CloudWatch Metrics, and Amazon CloudWatch Events. Amazon S3 AWS Key Management Service (KMS) Amazon S3 storage classes Now we know how long it will take to migrate this massive amount of data and the cost. Make a business decision as to which method meets your needs based on the combination of timing and fees. At this point, we also know the requirements for the hardware needed to run MinIO on-prem or at a colocation facility. Take the requirement above for 1.5PB of storage, estimate data growth, and consult our page and . Recommended Hardware & Configuration Selecting the Best Hardware for Your MinIO Deployment The first step is to recreate your S3 buckets in MinIO. You’re going to have to do this regardless of how you choose to migrate objects. While both S3 and MinIO store objects using server-side encryption, you don’t have to worry about migrating encryption keys. You can connect to your KMS of choice using . This way, new keys will be automatically generated for you as encrypted tenants and buckets are created in MinIO. MinIO KES to manage encryption keys You have multiple options to copy objects: Batch Replication and . My previous blog post, included detailed instructions for both methods. You can copy objects directly from S3 to on-prem MinIO, or use a temporary MinIO cluster running on EC2 to query S3 and then mirror to on-prem MinIO. mc mirror How to Repatriate From AWS S3 to MinIO Typically, customers use tools we wrote combined with AWS Snowball or TD SYNNEX’s data migration hardware and services to move larger amounts of data (over 1 PB). MinIO recently partnered with Western Digital and TD SYNNEX to field a Snowball alternative. Customers can schedule windows to take delivery of the Western Digital hardware and pay for what they need during the rental period. More importantly, the service is not tied to a specific cloud - meaning the business can use the service to move data into, out of, and across clouds - all using the ubiquitous S3 protocol. Additional details on the service can be found on the page on the TD SYNNEX site. Data Migration Service Bucket metadata, including policies and bucket properties, can be read using and then set up in MinIO. When you sign up for MinIO SUBNET, our engineers will work with you to migrate these settings from AWS S3: access management based on access key/secret key, lifecycle management policies, encryption, anonymous public access, immutability and versioning. One note about versioning, AWS version ID isn’t usually preserved when data is migrated because each version ID is an internal UUID. This is largely not a problem for customers because objects are typically called by name. However, if AWS version ID is required, then we have an extension that will preserve it in MinIO and we’ll help you enable it. get-bucket S3 API calls Pay particular attention to . S3 isn’t going to be the only part of AWS’s infrastructure that you leave behind. You will have a lot of service accounts for applications to use when accessing S3 buckets. This would be a good time to list and audit all of your service accounts. Then you can decide whether or not to recreate them in your identity provider. If you choose to automate, then use Amazon Cognito to share IAM information with external OpenID Connect IDPs and AD/LDAP. IAM and bucket policies Pay particular attention to Data Lifecycle Management, such as object retention, object locking and archive/tiering. Run a on each bucket to obtain a human-readable JSON list of lifecycle rules. You can easily recreate AWS S3 settings using MinIO Console or MinIO Client (mc). Use commands such as and to pinpoint objects that require special security and governance treatment. get-bucket-lifecycle-configuration get-object-legal-hold get-object-lock-configuration While we’re on the subject of lifecycle, let’s talk about backup and disaster recovery for a moment. Do you want an additional MinIO cluster to replicate to, for backup and disaster recovery? After objects are copied from AWS S3 to MinIO, it’s important to validate data integrity. The easiest way to do this is to use the MinIO Client to run against old buckets in S3 and new buckets on MinIO. This will compute the difference between the buckets and return a list of only those objects that are missing or different. This command takes the arguments of the source and target buckets. For your convenience, you may want to create for S3 and MinIO so you don’t have to keep typing out full addresses and credentials. For example: mc diff aliases mc diff s3/bucket1 minio/bucket1 The great news is that all you have to do is point existing apps at the new MinIO endpoint. Configurations can be rewritten app by app over a period of time. Migrating data in object storage is less disruptive than a filesystem, just change the URL to read/write from a new cluster. Note that if you previously relied on AWS services to support your applications, those won’t be present in your data center, so you’ll have to replace them with their open-source equivalent and rewrite some code. For example, Athena can be replaced with Spark SQL, Apache Hive and Presto, Kinesis with Apache Kafka, and AWS Glue with Apache Airflow. If your S3 migration is part of a larger effort to move an entire application on-prem, then chances are you used to call downstream services when new data arrived. If this is the case, then do not fear - MinIO supports as well. The most straightforward migration here would be to implement a custom webhook to receive the notification. However, if you need a destination that is more durable and resilient, then use messaging services such as Kafka or RabbitMQ. We also support sending events to databases such as PostgreSQL and MySQL. S3 event notifications event notification Now that you’ve completed repatriating, it’s time to turn your attention to storage operation, monitoring and optimization. The good news is that no optimization is needed for MinIO – we’ve built optimization right into the software so you know you’re getting the best performance for your hardware. You’ll want to start monitoring your new MinIO cluster to assess resource utilization and performance on an ongoing basis. MinIO exposes via a Prometheus endpoint that you can consume in your . For more on monitoring, please see and . metrics monitoring and alerting platform of choice Multi-Cloud Monitoring and Alerting with Prometheus and Grafana Metrics with MinIO using OpenTelemetry, Flask, and Prometheus With , we have your back when it comes to with MinIO. Subscribers gain access to built-in automated troubleshooting tools to keep their clusters running smoothly. They also get unlimited, direct-to-engineer support in real-time via our support portal. We also help you future-proof your object storage investment with an annual architecture review. SUBNET Day 2 operations Migrate and Save It’s far from a secret that the days of writing blank checks to cloud providers are gone. Many businesses are currently evaluating their cloud spend to find potential savings. Now you have everything you need to start your migration from AWS S3 to MinIO, including concrete technical steps and a financial framework. If you get excited about the prospect of repatriation cost savings, then please reach out to us at . hello@min.io Also appears . here

All You Need to Know to Repatriate from AWS S3 to MinIO

AWS S3 থেকে MinIO-তে প্রত্যাবাসন করতে আপনার যা জানা দরকার

Alles, was Sie zur Repatriierung von AWS S3 zu MinIO wissen müssen

Todo lo que necesita saber para repatriar de AWS S3 a MinIO

Tout ce que vous devez savoir pour rapatrier d'AWS S3 vers MinIO

AWS S3 से MinIO में स्थानांतरण के लिए आपको जो कुछ भी जानना चाहिए

AWS S3 から MinIO への復帰に必要なすべての情報

AWS S3에서 MinIO로 송환하기 위해 알아야 할 모든 것

Tudo o que você precisa saber para repatriar do AWS S3 para MinIO

Все, что вам нужно знать для репатриации с AWS S3 на MinIO

AWS S3'ten MinIO'ya Geri Dönmek İçin Bilmeniz Gereken Her Şey

Tất cả những điều bạn cần biết để hồi hương từ AWS S3 về MinIO

从 AWS S3 迁移到 MinIO 所需了解的一切

MINIO

min.io

55 employees

Since 2014

COMPANY RANKING

MINIO

EVERGREEN INDEX #8

Representing this company? Click here to claim and customize this page!

MinIO WIKI

MINIO

min.io

55 employees

Since 2014

COMPANY RANKING

MINIO

EVERGREEN INDEX #8

HACKERNOON STORIES ON MINIO

MINIO NEWS & MENTIONS

Representing this company? Click here to claim and customize this page!

MinIO WIKI