MinIO

minio

The Noonification: Data Migration from Dell ECS to MinIO (9/22/2024)

Using MinIO to Build a Retrieval Augmented Generation Chat Application

Die gebruik van MinIO om 'n herwinningsaansoek vir uitgebreide generasie-klets te bou

استخدام MinIO لبناء تطبيق دردشة يعتمد على الاسترجاع المعزز

Použití MinIO k vytvoření aplikace pro chat s rozšířenou generací načítání

Verwenden von MinIO zum Erstellen einer Retrieval Augmented Generation Chat-Anwendung

Uso de MinIO para crear una aplicación de chat de generación aumentada de recuperación

MinIO を使用して検索拡張生成チャット アプリケーションを構築する

ការប្រើប្រាស់ MinIO ដើម្បីបង្កើតកម្មវិធីជជែកកំសាន្តដែលបង្កើតឡើងវិញ

Mampiasa ny MiniIO hananganana fampiharana amin'ny chat Generation Augmented Retrieval

د ترلاسه کولو وده شوي نسل چیٹ غوښتنلیک جوړولو لپاره MinIO کارول

Použitie MinIO na zostavenie aplikácie na rozhovor s rozšírenou generáciou načítania

MinIO-dan peýdalanyp, gözlegiň giňeldilen nesil söhbetdeşlik programmasyny gurmak

Paggamit ng MiniIO para Bumuo ng Retrieval Augmented Generation Chat Application

Data Migration from Dell ECS to MinIO

نقل البيانات من Dell ECS إلى MinIO

Migración de datos de Dell ECS a MinIO

Dell ECS から MinIO へのデータ移行

ការផ្ទេរទិន្នន័យពី Dell ECS ទៅ MinIO

Migration ya ba données depuis Dell ECS na MinIO

ການຍ້າຍຂໍ້ມູນຈາກ Dell ECS ໄປຫາ MinIO

Миграција на податоци од Dell ECS во MinIO

Migrasi Data daripada Dell ECS kepada MiniIO

Dell ECS kaqmanta MinIO kaqman Willayta Migracion

Migrácia dát z Dell ECS na MinIO

ዳታ ማይግሬሽን ካብ Dell ECS ናብ MinIO

Dell ECS-dan MiniIO-ga ma'lumotlarni ko'chirish

Hands-on with Apache Iceberg on Your Laptop: Deep Dive with Apache Spark, Nessie, Minio and more!

Transitioning from Hitachi Content Platform (HCP) to MinIO has never been easier, thanks to our HCP-to-MinIO tool. Developed to support our customers' evolving storage needs, this tool is freely available on GitHub and greatly simplifies the migration process. Many organizations are transitioning to leverage MinIO's modern, scalable, and high-performance object storage optimized for AI infrastructure. This tutorial provides a comprehensive step-by-step guide to ensure a smooth and efficient transition to MinIO. Why Transition to MinIO? Customers often migrate from HCP to MinIO for several compelling reasons, particularly when optimizing for AI-driven applications: Modern Application Requirements: MinIO is Kubernetes-native and built for the cloud operating model, which means that it inherently supports most modern application stacks for analytics, statistical analysis and AI/ML workloads.


Performance at Scale: MinIO efficiently handles mixed object sizes, from very small to very large, ensuring optimal performance for data-intensive AI operations.


Flexibility: Software-defined, S3-compatible object storage offers greater flexibility and cloud neutrality compared to dedicated storage appliances. HCP-to-MinIO Migration Process During migration, it's recommended to write exclusively to MinIO and not to HCP. This approach ensures you start as you plan to go on: using MinIO as your primary object store. Here’s the process: Initial Read Operations: While migrating, getobject calls are directed to MinIO. If an object isn't found, it's read from HCP. Once all objects are copied, this fallback can be removed.


Preserving Metadata: The tool retains the modified timestamps of objects during the migration. Step 1: Create a Temporary Directory After downloading the HCP-to-MinIO tool the first step is to create a temporary directory for the results of the list command in Step 2: $ mkdir /tmp/data Next, set the following environment variables: $ export MINIO_ACCESS_KEY=<Your-MinIO-Access-Key>
$ export MINIO_SECRET_KEY=<Your-MinIO-Secret-Key>
$ export MINIO_ENDPOINT=https://<Your-MinIO-IP-Address>:9000
$ export MINIO_BUCKET=newbucket  # optional, if unspecified HCP bucket name is used Step 2: List Objects in HCP The list command generates a list of objects stored in HCP and saves it to a local file. This file can be split into smaller lists and processed in parallel by multiple client machines to speed up the migration. Here is the command: hcp-to-minio list --auth-token --namespace-url --host-header --namespace-url --dir And an example of what it might look like: $ hcp-to-minio list --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "HOST:s3testbucket.tenant.hcp.example.com" --namespace-url "https://hcp-vip.example.com" --dir "/tmp/data" Step 3: Migrate Objects to MinIO The migrate command uses the list generated in the first step to transfer objects from HCP to MinIO. The command: hcp-to-minio migrate --namespace-url --auth-token --host-header --data-dir --bucket --input-file The example: $ hcp-to-minio migrate --namespace-url https://finance.europe.hcp.example.com --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "s3testbucket.sandbox.hcp.example.com" --data-dir /mnt/data --bucket s3testbucket --input-file /tmp/data/to-migrate.txt Get Started The HCP-to-MinIO tool is designed to help you move to a modern, scalable object storage solution. Download the tool from GitHub and start your migration today. For any questions or additional support, contact us at hello@min.io. Transitioning from Hitachi Content Platform (HCP) to MinIO has never been easier, thanks to our HCP-to-MinIO tool. Developed to support our customers' evolving storage needs, this tool is freely available on GitHub and greatly simplifies the migration process. Many organizations are transitioning to leverage MinIO's modern, scalable, and high-performance object storage optimized for AI infrastructure. This tutorial provides a comprehensive step-by-step guide to ensure a smooth and efficient transition to MinIO. GitHub GitHub Why Transition to MinIO? Customers often migrate from HCP to MinIO for several compelling reasons, particularly when optimizing for AI-driven applications: Modern Application Requirements: MinIO is Kubernetes-native and built for the cloud operating model, which means that it inherently supports most modern application stacks for analytics, statistical analysis and AI/ML workloads. Performance at Scale: MinIO efficiently handles mixed object sizes, from very small to very large, ensuring optimal performance for data-intensive AI operations. Flexibility: Software-defined, S3-compatible object storage offers greater flexibility and cloud neutrality compared to dedicated storage appliances. Modern Application Requirements: MinIO is Kubernetes-native and built for the cloud operating model, which means that it inherently supports most modern application stacks for analytics, statistical analysis and AI/ML workloads. Modern Application Requirements : MinIO is Kubernetes-native and built for the cloud operating model, which means that it inherently supports most modern application stacks for analytics, statistical analysis and AI/ML workloads. Modern Application Requirements Performance at Scale: MinIO efficiently handles mixed object sizes, from very small to very large, ensuring optimal performance for data-intensive AI operations. Performance at Scale : MinIO efficiently handles mixed object sizes, from very small to very large, ensuring optimal performance for data-intensive AI operations. Performance at Scale Flexibility: Software-defined, S3-compatible object storage offers greater flexibility and cloud neutrality compared to dedicated storage appliances. Flexibility : Software-defined, S3-compatible object storage offers greater flexibility and cloud neutrality compared to dedicated storage appliances. Flexibility HCP-to-MinIO Migration Process During migration, it's recommended to write exclusively to MinIO and not to HCP. This approach ensures you start as you plan to go on: using MinIO as your primary object store. Here’s the process: Initial Read Operations: While migrating, getobject calls are directed to MinIO. If an object isn't found, it's read from HCP. Once all objects are copied, this fallback can be removed. Preserving Metadata: The tool retains the modified timestamps of objects during the migration. Initial Read Operations: While migrating, getobject calls are directed to MinIO. If an object isn't found, it's read from HCP. Once all objects are copied, this fallback can be removed. Initial Read Operations : While migrating, getobject calls are directed to MinIO. If an object isn't found, it's read from HCP. Once all objects are copied, this fallback can be removed. Initial Read Operations getobject Preserving Metadata: The tool retains the modified timestamps of objects during the migration. Preserving Metadata : The tool retains the modified timestamps of objects during the migration. Preserving Metadata Step 1: Create a Temporary Directory After downloading the HCP-to-MinIO tool the first step is to create a temporary directory for the results of the list command in Step 2: $ mkdir /tmp/data $ mkdir /tmp/data Next, set the following environment variables: $ export MINIO_ACCESS_KEY=<Your-MinIO-Access-Key>
$ export MINIO_SECRET_KEY=<Your-MinIO-Secret-Key>
$ export MINIO_ENDPOINT=https://<Your-MinIO-IP-Address>:9000
$ export MINIO_BUCKET=newbucket  # optional, if unspecified HCP bucket name is used $ export MINIO_ACCESS_KEY=<Your-MinIO-Access-Key>
$ export MINIO_SECRET_KEY=<Your-MinIO-Secret-Key>
$ export MINIO_ENDPOINT=https://<Your-MinIO-IP-Address>:9000
$ export MINIO_BUCKET=newbucket  # optional, if unspecified HCP bucket name is used Step 2: List Objects in HCP The list command generates a list of objects stored in HCP and saves it to a local file. This file can be split into smaller lists and processed in parallel by multiple client machines to speed up the migration. list Here is the command: hcp-to-minio list --auth-token --namespace-url --host-header --namespace-url --dir hcp-to-minio list --auth-token --namespace-url --host-header --namespace-url --dir And an example of what it might look like: $ hcp-to-minio list --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "HOST:s3testbucket.tenant.hcp.example.com" --namespace-url "https://hcp-vip.example.com" --dir "/tmp/data" $ hcp-to-minio list --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "HOST:s3testbucket.tenant.hcp.example.com" --namespace-url "https://hcp-vip.example.com" --dir "/tmp/data" Step 3: Migrate Objects to MinIO The migrate command uses the list generated in the first step to transfer objects from HCP to MinIO. migrate The command: hcp-to-minio migrate --namespace-url --auth-token --host-header --data-dir --bucket --input-file hcp-to-minio migrate --namespace-url --auth-token --host-header --data-dir --bucket --input-file The example: $ hcp-to-minio migrate --namespace-url https://finance.europe.hcp.example.com --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "s3testbucket.sandbox.hcp.example.com" --data-dir /mnt/data --bucket s3testbucket --input-file /tmp/data/to-migrate.txt $ hcp-to-minio migrate --namespace-url https://finance.europe.hcp.example.com --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "s3testbucket.sandbox.hcp.example.com" --data-dir /mnt/data --bucket s3testbucket --input-file /tmp/data/to-migrate.txt Get Started The HCP-to-MinIO tool is designed to help you move to a modern, scalable object storage solution. Download the tool from GitHub and start your migration today. For any questions or additional support, contact us at hello@min.io. GitHub GitHub

Migrate to AI-Ready Infrastructure: Hitachi Content Platform to MinIO

Infraestructura AI-Ready ukar jaqukipaña: Plataforma de Contenido Hitachi ukar MinIO ukar sarxataña

Пераход на AI-Ready Infrastructure: Hitachi Content Platform на MinIO

Migrer til AI-Ready Infrastructure: Hitachi Content Platform til MinIO

Μεταφορά στην υποδομή AI-Ready: Πλατφόρμα περιεχομένου Hitachi στο MinIO

Migración a una infraestructura preparada para IA: Hitachi Content Platform a MinIO

مهاجرت به زیرساخت AI-Ready: پلتفرم محتوای هیتاچی به MinIO

AI-रेडी इंफ्रास्ट्रक्चर पर माइग्रेट करें: हिताची कंटेंट प्लेटफॉर्म से मिनियो तक

Migráció az AI-ready infrastruktúrára: a Hitachi Content Platform a MinIO-ra

AI対応インフラストラクチャへの移行：日立コンテンツプラットフォームからMinIOへ

Migrējiet uz AI gatavu infrastruktūru: Hitachi satura platforma uz MinIO

د AI چمتو زیربنا ته مهاجرت: MinIO ته د هیتاچي مینځپانګې پلیټ فارم

AI-Ready انفراسٹرکچر پر منتقل کریں: Hitachi Content Platform MinIO پر

The MinIO DataPod: A Reference Architecture for Exascale Computing

MinIO DataPod: Exascale Computing এর জন্য একটি রেফারেন্স আর্কিটেকচার

Der MinIO DataPod: Eine Referenzarchitektur für Exascale-Computing

MinIO DataPod: una arquitectura de referencia para la computación a escala de exaescala

Le MinIO DataPod : une architecture de référence pour le calcul exascale

मिनियो डेटापॉड: एक्सास्केल कंप्यूटिंग के लिए एक संदर्भ आर्किटेक्चर

MinIO DataPod: エクサスケール コンピューティングのリファレンス アーキテクチャ

MinIO DataPod: 엑사스케일 컴퓨팅을 위한 참조 아키텍처

O MinIO DataPod: Uma Arquitetura de Referência para Computação em Exaescala

MinIO DataPod: эталонная архитектура для эксафлопсных вычислений

MinIO DataPod: Exascale Hesaplama için Bir Referans Mimarisi

MinIO DataPod: Kiến trúc tham chiếu cho điện toán Exascale

MinIO DataPod：百亿亿次计算的参考架构

The Noonification: Limit Cloud Data Costs with MinIO on Equinix (6/30/2024)

In our previous discussion, we introduced the MinIO Object Store Firewall, a novel security component that goes beyond traditional network and application security layers. Neither IP-based firewalls nor application firewalls are designed for data. That is why we built the MinIO Enterprise Object Store Firewall – because in the modern enterprise, data is what must be protected and an S3-aware firewall doesn’t exist. This S3-aware data firewall is crucial for modern data protection, operating at the storage layer to safeguard your data comprehensively. The MinIO Enterprise Object Store Firewall is designed specifically to work with applications using MinIO object store and its API endpoints. The Enterprise Firewall is lightweight, powerful, flexible and extensible. Let’s delve into setting up this advanced firewall, designed to secure your data in today’s increasingly complex digital landscape. Enable and Configure Firewall Let's use the Enterprise Console to set up the firewall. Follow the below steps to enable and configure the Firewall. Configure TLS for Secure Communication As part of the MinIO Enterprise Suite, we’ve always recommended to ensure you enable TLS on MinIO Enterprise Object Store so that even inter cluster communications are encrypted. With that same spirit, we support TLS for the Enterprise Firewall as well. This ensures that any connection to the MinIO Object Store via the Firewall is encrypted end to end. For enhanced security, configure TLS settings when launching the firewall with Let’s Encrypt. Precedence of Anonymous Rule In our MinIO Enterprise Firewall configuration, you’ll encounter two distinct rules for anonymous access: Global Anonymous Setting: At the start when you enable firewall, we specify a global setting that allows anonymous access across all buckets unless a more specific rule denies it. Bucket Specific Anonymous: After configuring the global setting, under each individual rules, we can set a more specific rule that overrides the global setting, effectively denying anonymous access In this case, even though we initially toggled Global Anonymous to Allow for all buckets, because we set another more specific rule below it, which also applies to all buckets, the one we set later takes precedence over the global setting. In other words, anonymous bucket access will be denied for all buckets. Load Balance across MinIO nodes As an added bonus, because we need to define all the MinIO nodes as a backend for the Firewall, the Firewall doubles as a LoadBalancer, negating the need for a separate load balancer between the firewall and MinIO nodes. This eliminates further complexity and ensures there is no single point of failure in case one of the MinIO backend goes offline. Please note, you need to have another level of redundancy to ensure when connecting to Enterprise Firewall the incoming connections are also distributed to multiple Enterprise Firewall instances. This configuration is beyond the scope of this blog but its something to keep in mind. Health Checks and Monitoring The health check enabled you to see if the Firewall is in a healthy state. You can check the health and liveliness as follows. If everything is Green that will indicate that the firewall is functioning properly. Final Thoughts With the MinIO Enterprise Firewall, gone are the days of wrestling with complex IPtables and unclear access policies.Our firewall solution simplifies your security by focusing exclusively on the essential rules required for your object store and API interactions. It is optimized to ensure there are no latency or unforeseen rules, blocking access to your MinIO object store. Moreover, the Enterprise Firewall is fully supported by our awesome team at SUBNET where we can help you by architecting the Enterprise Firewall the right way to work with MinIO object store and troubleshoot any future issues. So what are you waiting for? If you have any questions on MinIO Enterprise Object Store Firewall be sure to reach out to us on Slack or hello@min.io! In our previous discussion , we introduced the MinIO Object Store Firewall, a novel security component that goes beyond traditional network and application security layers. Neither IP-based firewalls nor application firewalls are designed for data. That is why we built the MinIO Enterprise Object Store Firewall – because in the modern enterprise, data is what must be protected and an S3-aware firewall doesn’t exist. This S3-aware data firewall is crucial for modern data protection, operating at the storage layer to safeguard your data comprehensively. previous discussion previous discussion The MinIO Enterprise Object Store Firewall is designed specifically to work with applications using MinIO object store and its API endpoints. The Enterprise Firewall is lightweight, powerful, flexible and extensible. Let’s delve into setting up this advanced firewall, designed to secure your data in today’s increasingly complex digital landscape. Enable and Configure Firewall Let's use the Enterprise Console to set up the firewall. Follow the below steps to enable and configure the Firewall. Configure TLS for Secure Communication As part of the MinIO Enterprise Suite, we’ve always recommended to ensure you enable TLS on MinIO Enterprise Object Store so that even inter cluster communications are encrypted. With that same spirit, we support TLS for the Enterprise Firewall as well. This ensures that any connection to the MinIO Object Store via the Firewall is encrypted end to end. For enhanced security, configure TLS settings when launching the firewall with Let’s Encrypt. Precedence of Anonymous Rule In our MinIO Enterprise Firewall configuration, you’ll encounter two distinct rules for anonymous access: Global Anonymous Setting: At the start when you enable firewall, we specify a global setting that allows anonymous access across all buckets unless a more specific rule denies it. Global Anonymous Setting: Bucket Specific Anonymous: After configuring the global setting, under each individual rules, we can set a more specific rule that overrides the global setting, effectively denying anonymous access Bucket Specific Anonymous: In this case, even though we initially toggled Global Anonymous to Allow for all buckets, because we set another more specific rule below it, which also applies to all buckets, the one we set later takes precedence over the global setting. In other words, anonymous bucket access will be denied for all buckets. Load Balance across MinIO nodes As an added bonus, because we need to define all the MinIO nodes as a backend for the Firewall, the Firewall doubles as a LoadBalancer, negating the need for a separate load balancer between the firewall and MinIO nodes. This eliminates further complexity and ensures there is no single point of failure in case one of the MinIO backend goes offline. Please note, you need to have another level of redundancy to ensure when connecting to Enterprise Firewall the incoming connections are also distributed to multiple Enterprise Firewall instances. This configuration is beyond the scope of this blog but its something to keep in mind. Health Checks and Monitoring The health check enabled you to see if the Firewall is in a healthy state. You can check the health and liveliness as follows. If everything is Green that will indicate that the firewall is functioning properly. Final Thoughts With the MinIO Enterprise Firewall, gone are the days of wrestling with complex IPtables and unclear access policies.Our firewall solution simplifies your security by focusing exclusively on the essential rules required for your object store and API interactions. It is optimized to ensure there are no latency or unforeseen rules, blocking access to your MinIO object store. Moreover, the Enterprise Firewall is fully supported by our awesome team at SUBNET where we can help you by architecting the Enterprise Firewall the right way to work with MinIO object store and troubleshoot any future issues. SUBNET SUBNET So what are you waiting for? If you have any questions on MinIO Enterprise Object Store Firewall be sure to reach out to us on Slack or hello@min.io ! Slack Slack hello@min.io hello@min.io

A Closer Look Into the MinIO Enterprise Object Store Firewall

MinIO এন্টারপ্রাইজ অবজেক্ট স্টোর ফায়ারওয়ালের একটি ঘনিষ্ঠভাবে দেখুন

Ein genauerer Blick auf die MinIO Enterprise Object Store Firewall

Una mirada más cercana al firewall del almacén de objetos empresariales MinIO

Un examen plus approfondi du pare-feu MinIO Enterprise Object Store

मिनियो एंटरप्राइज़ ऑब्जेक्ट स्टोर फ़ायरवॉल पर एक नज़दीकी नज़र

MinIO エンタープライズ オブジェクト ストア ファイアウォールの詳細

MinIO Enterprise 개체 저장소 방화벽 자세히 살펴보기

Uma análise mais detalhada do firewall do MinIO Enterprise Object Store

Более пристальный взгляд на брандмауэр MinIO Enterprise Object Store

MinIO Kurumsal Nesne Deposu Güvenlik Duvarına Daha Yakından Bir Bakış

Xem xét kỹ hơn về Tường lửa lưu trữ đối tượng doanh nghiệp MinIO

深入了解 MinIO 企业对象存储防火墙

Effortlessly Launch LangChain APIs with LangServe and MinIO Integration

অনায়াসে LangServe এবং MinIO ইন্টিগ্রেশন সহ LangChain API চালু করুন

Müheloses Starten von LangChain-APIs mit LangServe- und MinIO-Integration

Lance las API de LangChain sin esfuerzo con la integración de LangServe y MinIO

Lancez sans effort les API LangChain avec l'intégration LangServe et MinIO

LangServe और MinIO एकीकरण के साथ LangChain API को आसानी से लॉन्च करें

LangServe と MinIO の統合により LangChain API を簡単に起動

LangServe 및 MinIO 통합을 통해 LangChain API를 손쉽게 실행

Inicie APIs LangChain sem esforço com integração LangServe e MinIO

Легко запускайте API-интерфейсы LangChain с интеграцией LangServe и MinIO.

LangServe ve MinIO Entegrasyonu ile LangChain API'lerini Zahmetsizce Başlatın

Dễ dàng khởi chạy API LangChain với tích hợp LangServe và MinIO

通过 LangServe 和 MinIO 集成轻松启动 LangChain API

Developing Next-Gen Data Solutions: SingleStore, MinIO, and the Modern Datalake Stack

নেক্সট-জেন ডেটা সলিউশন তৈরি করা: একক স্টোর, মিনিও এবং আধুনিক ডেটালেক স্ট্যাক

Entwicklung von Datenlösungen der nächsten Generation: SingleStore, MinIO und der moderne Datalake-Stack

Desarrollo de soluciones de datos de próxima generación: SingleStore, MinIO y Modern Datalake Stack

Développer des solutions de données de nouvelle génération : SingleStore, MinIO et la pile Datalake moderne

अगली पीढ़ी के डेटा समाधान विकसित करना: सिंगलस्टोर, मिनियो और आधुनिक डेटालेक स्टैक

次世代データソリューションの開発: SingleStore、MinIO、最新のデータレイク スタック

차세대 데이터 솔루션 개발: SingleStore, MinIO 및 최신 Datalake 스택

Desenvolvendo soluções de dados de última geração: SingleStore, MinIO e Modern Datalake Stack

Разработка решений для обработки данных следующего поколения: SingleStore, MinIO и современный стек Datalake

Yeni Nesil Veri Çözümlerinin Geliştirilmesi: SingleStore, MinIO ve Modern Datalake Stack

Phát triển các giải pháp dữ liệu thế hệ tiếp theo: SingleStore, MinIO và Datalake Stack hiện đại

开发下一代数据解决方案：SingleStore、MinIO 和现代 Datalake 堆栈

Stories, News & Mentions about MINIO

Minio’s Stories on HackerNoon