In a story that is not uncommon among SaaS companies, a cloud-native cybersecurity organization was faced with massively escalated cloud costs as its log data expanded to multi-exabytes. Storage costs aside, the cost to simply operate on this data became astronomical. This organization was hit from both sides at the same time because storage costs were only one part of the problem: their cloud storage provider, Amazon’s S3, simply did not have the performance required to handle the log operations they required over this volume of data. This cybersecurity firm would continue to sell its managed service on AWS, but it had to move its internal workloads off the public cloud. This is where an on-prem architecture with MinIO’s AIStor stepped in to solve both of their problems simultaneously: cost and performance. Challenge Challenge Their concerns were multi-faceted: they had to make sure that they had no outages before, during, and after they transferred their log data. Any loss of data or downtime would cost them customers and reputation. AWS’s pricing structure is responsible for the economic side of their difficulty. Generally speaking, while S3 compatible storage costs scale with the volume of data, additional charges accumulate from egress (transfers out of AWS), retrieval fees from speciality storage classes and encryption. More specifically for this cybersecurity company, the cost of GETS and PUTS as they operated over their log data in the public cloud had become intolerable. Anytime this cybersecurity firm wanted to do anything with their own data, it resulted in a new charge. These fees only compounded as their volume of data increased. S3 compatible storage S3 compatible storage Finally, they needed an S3-compatible object store they could implement and manage on their own. They already had an interlocking data architecture that depended on the S3 API. Any solution to their difficulties had to drop into their existing stack and work with everything they had. The S3 API is an integral part of the modern data stack; almost every piece of the software that anyone uses anywhere accommodates the S3 API in some way or another. This is particularly true of this cybersecurity company, whose entire architecture depended on the S3 API. Solution: Adopting MinIO AIStor Solution: Adopting MinIO AIStor After assessing their challenges and researching potential solutions, they realized that only AIStor was able to address their requirements. AIStor’s object storage provided them with: S3-Compatibility: Allowing the cybersecurity company to migrate without disruption to their operations.High Performance: They required faster retrieval times to support their security analytics and incident response workflows. AIStor, with read speeds of up to 2.6 terabits per second (325 GiB/s) and write speeds of up to 1.32 terabits per second (165 GiB/s) in a 32-node cluster configuration, was a natural choice. Availability and Resiliency: Erasure coding is a core component of AIStor and provides resilience during drive or node-level failure events, ensuring continuous uptime for their security operations.Best in Class Support: Being able to leverage the panic button and interact directly with engineers for critical issues became an urgent need as the cybersecurity company continued to scale. S3-Compatibility: Allowing the cybersecurity company to migrate without disruption to their operations. S3-Compatibility: High Performance: They required faster retrieval times to support their security analytics and incident response workflows. AIStor, with read speeds of up to 2.6 terabits per second (325 GiB/s) and write speeds of up to 1.32 terabits per second (165 GiB/s) in a 32-node cluster configuration, was a natural choice. High Performance: read speeds of up to 2.6 terabits read speeds of up to 2.6 terabits write speeds of up to 1.32 terabits per second write speeds of up to 1.32 terabits per second Availability and Resiliency: Erasure coding is a core component of AIStor and provides resilience during drive or node-level failure events, ensuring continuous uptime for their security operations. Availability and Resiliency Availability and Resiliency Availability and Resiliency : Erasure coding Erasure coding Best in Class Support: Being able to leverage the panic button and interact directly with engineers for critical issues became an urgent need as the cybersecurity company continued to scale. Best in Class Support Best in Class Support Best in Class Support panic button panic button Implementation & Architecture Implementation & Architecture This security log data was ingested via a third party as well as a homegrown log management tool which used AIStor as an S3 compatible storage backend. For analytics and security event processing, they leverage a mix of real-time streaming and batch-processing frameworks, ensuring rapid incident detection and response. Additionally, their infrastructure is designed to support high-speed networking, utilizing NVMe storage and 100 GbE networking to handle the demands of their security operations efficiently. S3 compatible storage S3 compatible storage According to Arvind Gupta, Head of Customer Engineering at AIStor, "Given that they are a security company, the data durability and availability is of prime importance. That means they need to ensure the data written to AIStor object store is never lost. Any data loss is business critical for them, and so is AIStor." For most of the organizations, durability and availability of data is the greatest concern. MinIO provides durability with erasure coding across nodes and drives. High availability can be achieved with multi-site deployment across different availability zones and regions. erasure coding erasure coding The CyberSecurity company found that they could use their existing DevOps and IT personnel for the migration. Their experience in cloud technologies was easily transferable to AIStor. This is part of a broader effort of AIStor to perfectly align with S3 API, which includes aligning on error messages. This means the cybersecurity company was able to configure and deploy AIStor on its own. Of course, AIstor was able to help with the planning process and advise on hardware, but the heavy lifting of this migration fell to the cybersecurity company’s own team, which was able to manage it without much difficulty. AIstor runs on commodity hardware, which means this cybersecurity company was able to use an easily accessible and reliable collocation service for its implementation. There was no need to source, maintain, or train for specialized equipment. They are able to swap out hardware as their needs adapt, and because they aren’t tied to any specific vendor, they can negotiate the best prices for their hardware as they add and replace. Proving again that owning your own data can be cost-effective on multiple levels. Related: Object Storage Erasure Coding vs. Block Storage RAID Object Storage Erasure Coding vs. Block Storage RAID Object Storage Erasure Coding vs. Block Storage RAID Dealing with Something Similar? Dealing with Something Similar? Figuring out what to do with log or unstructured data is not a unique problem to cybersecurity. Many organizations struggle with either the large volume of log data under management or how long to keep it. Using a managed service for both storage and the curation of logs can be prohibitively expensive. Disaggregating this approach and using AIStor for log data can be incredibly cost-effective, as it was for this cybersecurity company. Returning to the topic of public cloud costs, skyrocketing and unpredictable cloud bills are certainly not unheard of. The question becomes, when do these expenses no longer make sense for an organization? There is a tipping point where the volume of log data (or any data for that matter) builds to the point where it no longer makes economic sense to store data in the public cloud. Beyond this tipping point, it makes sense to move the data and workload to an on-premises private cloud. Organizations reach that threshold much faster when they need to move large amounts of 'hot' data or due to tiering, API calls, replication, encryption, and similar processes. Great for bursty, experimental and cyclical workloads, the cloud will always have a place in our architectures. But, more and more organizations like this cybersecurity firm are coming to understand the cloud as an operating model, not a specific vendor. In other words, their stack can be optimized for the cloud without having to be in it. Growing Footprint Growing Footprint The cybersecurity company continues to grow its AIStor footprint to support the growing data demands around various new initiatives. Since repatriating from AWS to an on-prem private cloud powered by MinIO AIStor, the cybersecurity company has been consistently expanding its existing cluster and adding more and more clusters. Gupta explained that this footprint is expanding to even more workloads: “They're exploring new use cases and avenues to start using MinIO for all the data needs." The cybersecurity company’s transition from AWS to AIStor exemplifies the power of cloud repatriation in reducing costs and improving performance. By leveraging MinIO AIStor’s high-performance, S3-compatible object storage, they have created a scalable, performant, and cost-effective infrastructure to support their operations. If you’re interested in doing the same, please reach out to us at hello@min.io or on our Slack. hello@min.io hello@min.io Slack Slack

The is an opinion piece based on the author’s POV and does not necessarily reflect the views of HackerNoon.

Escaping AWS S3: How One Cybersecurity Firm Cut Costs With AIStor

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Closer Look Into the MinIO Enterprise Object Store Firewall

$DEFI Token Hits 7 Major Exchanges: A Milestone Achievement

$JTC Network To List On BitMart Exchange

$500k Presale: TG.Casino Passes Milestone with Upcoming Telegram-Powered Platform

$3 Million in Seed Funding for Web3 Founders Announced By Necto Labs

$2M Backing and a Vision: How GAM3S.GG is Reshaping Web3 Gaming

A Closer Look Into the MinIO Enterprise Object Store Firewall

$DEFI Token Hits 7 Major Exchanges: A Milestone Achievement

$JTC Network To List On BitMart Exchange

$500k Presale: TG.Casino Passes Milestone with Upcoming Telegram-Powered Platform

$3 Million in Seed Funding for Web3 Founders Announced By Necto Labs

$2M Backing and a Vision: How GAM3S.GG is Reshaping Web3 Gaming

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps