When purchasing storage, the emphasis is usually on media, but it may be even more important to consider access methods too. You will need to take storage protocols into account when designing and procuring infrastructure, especially when you leave legacy storage behind in order to migrate to cloud-native object storage. However, object storage relies on the S3 API for communications, while legacy workloads rely on POSIX, the Portable Operating System Interface that provides a set of standards developed in the 1980s to allow applications to be portable between Unix OSs. Chances are that most enterprises have had applications in service for decades that were developed to run on POSIX. Chances are also that engineers are already aware of POSIX’s poor performance. That being said, when you have legacy systems that can only ingest data in a certain format or from a certain source, your options might be limited and you may have no choice but to implement an outdated protocol or rewrite your code. For example, if you can only ingest from a local disk with a filesystem and not with RESTful APIs accessed over network, then you must first make that data available on disk before your application can use it. However, using an object store as a filesystem has a number of serious negative implications when it comes to performance, compatibility, data integrity and security. Let’s demonstrate  this with some real world testing using a small utility called . This utility allows you to mount an S3 bucket as a local filesystem. It stands for S3 (Simple Storage Service) File System - FUSE (Filesystem in Userspace). It's an open-source project that leverages the FUSE (Filesystem in Userspace) interface to present a filesystem-like interface to S3. s3fs-fuse Once an S3 bucket is mounted using , you can interact with the bucket as if it were a local filesystem. This means you can use regular file operations (like read, write, move, etc.) on files in your bucket. This sounds amazingly convenient and we can argue it simplifies application development. But inherently object storage and file systems have fundamental differences that will affect the s3 bucket mounted as a filesystem. s3fs-fuse Let’s take a moment to step back from the utility to discuss the real reasons why treating object storage as a filesystem is far from optimal. The problem is much bigger than and includes other utilities such as Rust based , a file client that translates local file system API calls to S3 object API calls. The first reason is that all of these utilities rely on POSIX for filesystem operations. POSIX is inefficient and it was never intended for working with very large files over-the-network. s3fs-fuse s3fs-fuse Mountpoint for Amazon S3 The speed of POSIX based systems decreases as the demand, especially concurrent demand, on them increases. When large organizations need to store and access oceans of data for deep learning, AI, and other data intensive use cases, POSIX does not have the ability or scalability to meet these demands. While all-Flash arrays have kept POSIX in the game, scalability and RESTful APIs (the hallmarks of the cloud) are like kryptonite. Because of this, running POSIX on top of an object store is suboptimal. Let’s take a look at some of the reasons why: POSIX FS interface is inherently IOPS centric. They are chatty, expensive and hard to scale. RESTful S3 API addresses this by transforming IOPS into a throughput problem. Throughput is easier and cheaper to scale. This is why object storage is high-performance at a massive scale. Layering POSIX over S3 will not scale because POSIX is too chatty to be performed over an HTTP RESTful interface. Performance: Because object operations are atomic and immutable, there is no way to guarantee consistency correctness. This means you can lose the uncommitted data in case of a crash or run into corruption issues in case of shared mounts. Semantics: : Writes or any mutations to a file won't appear in the namespace until it is committed. This means concurrent access across shared mounts will not see the modifications. It is not useful for shared access. Data Integrity POSIX permissions and ACLs are primitive and incompatible with the S3 API’s way of handling identity and access management policies. It is not possible to safely implement POSIX access management on top S3 APIs. Access Control: POSIX also lacks most of the functionality that developers love about S3, such as  object level encryption, versioning, immutability – these simply have no equivalent in the POSIX world and nothing is able to translate them. POSIX Pain Points These examples illustrate the issue and its implications. To get started, we will use CSV file which is approximately 10GB and has 112 rows. this Note: We will assume the is already installed and you have mounted one of the buckets from the object storage into your filesystem. If not, follow the instructions . s3fs-fuse here In these examples, we will assume the bucket name is test-bucket and the file name is taxi-data.csv is in /home/user/ directory and is mounted at /home/user/test-bucket/ s3fs-fuse bucket Copy Operation We’ll try something simple first and try to copy the CSV file to our test-bucket using mc commands and record the time taken time mc cp /home/user/taxi-data.csv minio/test-bucket/taxi-data.csv This shouldn't take a lot of time and the file should be copied to our bucket. Now let’s try to do the same with s3fs-fuse time cp /home/user/taxi-data.csv /home/user/test-bucket/taxi-data.csv Time it took during the testing real	1m36.056s
user	0m14.507s
sys	0m31.891s In my case I was only able to copy the file partially to the bucket and the operation failed with the following error cp: error writing '/home/user/test-bucket/taxi-data.csv': Input/output error
cp: failed to close '/home/user/test-bucket/taxi-data.csv': Input/output error After multiple tries it succeeded real	5m3.661s
user	0m0.053s
sys	2m35.234s As you can see, because of the amount of API calls the utility needs to make and the general overhead of the operations, the utility becomes unstable and most of the operations do not even finish. Pandas Example We’ve shown you a simple example, which may or may not have been convincing, because let's face it, you might think is quite rudimentary. cp time cp So for the folks who need more empirical evidence let's write a Python snippet to test this. We’ll do a simple Pandas example with both and the package, and see the performance impact s3fs-fuse python s3fs import timeit
import os
import fsspec
import s3fs
import pandas as pd

# Write a dummy CSV file to test-bucket
df = pd.DataFrame({"column1": ["new_value1"], "column2": ["new_value2"]})
df.to_csv("s3://test-bucket/data/test-data.csv", index=False)


def process_s3():
    # add fsspec for pandas to use `s3://` path style and access S3 buckets directly
    fsspec.config.conf = {
      "s3":
      {
        "key": os.getenv("AWS_ACCESS_KEY_ID", "minioadmin"),
        "secret": os.getenv("AWS_SECRET_ACCESS_KEY", "minioadmin"),
        "client_kwargs": {
          "endpoint_url": os.getenv("S3_ENDPOINT", "https://play.min.io")
        }
      }
    }
    s3 = s3fs.S3FileSystem()
    for i in range(100):
        # Read the existing data
        print(i)
        df = pd.read_csv('s3://test-bucket/data/test-data.csv')
        # Append a new row
        new_df = pd.concat([df, pd.DataFrame([{"column1": f"value{i}", "column2": f"value{i}"}])], ignore_index=True)
        # Write the data back to the file
        new_df.to_csv('s3://test-bucket/data/test-data.csv', index=False)


execution_time = timeit.timeit(process_s3, number=1)
print(f"Execution time: {execution_time:.2f} seconds") Time taken during the testing Execution time: 8.54 seconds Now let's try the same for s3fs-fuse import timeit
import pandas as pd

# Write a dummy CSV file to test-bucket
df = pd.DataFrame({"column1": ["new_value1"], "column2": ["new_value2"]})
df.to_csv("s3://test-bucket/data/test-data.csv", index=False)


def process_s3fs():
    for i in range(100):
        # Read the existing data
        print(i)
        df = pd.read_csv('/home/user/test-bucket/data/test-data.csv')
        # Append a new row
        new_df = pd.concat([df, pd.DataFrame([{"column1": f"value{i}", "column2": f"value{i}"}])], ignore_index=True)
        # Write the data back to the file
        new_df.to_csv('/home/user/test-bucket/data/test-data.csv', index=False)


execution_time = timeit.timeit(process_s3fs, number=1)
print(f"Execution time: {execution_time:.2f} seconds") Time taken during the testing Execution time: 9.59 seconds These examples demonstrate constant reads and writes to S3 files. Imagine this performed concurrently by multiple clients – the latency grows dramatically. Remove the Overhead! As you can see, the difference between using POSIX translation to treat objects as files and using the direct API to work with objects is night and day. There is simply no comparison when it comes to security, performance, data integrity and compatibility. MinIO has to integrate with almost any popular programming language, and it can run on almost any platform such as Kubernetes, bare metal Linux, Docker containers – and much more. SDKs MinIO with encryption at rest and in-transit, combined with PBAC to regulate access and to protect data integrity. You will achieve the best performance possible, regardless of where you run MinIO, because it takes advantage of underlying hardware (see ) to deliver the greatest possible performance. We’ve MinIO at 325 GiB/s (349 GB/s) on GETs and 165 GiB/s (177 GB/s) on PUTs with just 32 nodes of off-the-shelf NVMe SSDs. secures objects erasure coding Selecting the Best Hardware for Your MinIO Deployment benchmarked There is simply no need for a filesystem utility in the middle of MinIO and your application! Any advantage legacy apps might receive will be offset by the pain of POSIX. If you have any questions about using POSIX translation for your application, be sure to reach out to us on ! Slack Also published . here

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Why You Shouldn't Put a Filesystem on Top of an Object Store

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Closer Look Into the MinIO Enterprise Object Store Firewall

$DEFI Token Hits 7 Major Exchanges: A Milestone Achievement

$JTC Network To List On BitMart Exchange

$500k Presale: TG.Casino Passes Milestone with Upcoming Telegram-Powered Platform

$3 Million in Seed Funding for Web3 Founders Announced By Necto Labs

A Closer Look Into the MinIO Enterprise Object Store Firewall

$DEFI Token Hits 7 Major Exchanges: A Milestone Achievement

$JTC Network To List On BitMart Exchange

$500k Presale: TG.Casino Passes Milestone with Upcoming Telegram-Powered Platform

$3 Million in Seed Funding for Web3 Founders Announced By Necto Labs

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps