Canid

canid

Implementing Secure Role-Based Access Control (RBAC) in a Next.js Application Using Permit.io

Breaking Down the Worker Task Execution in Apache DolphinScheduler

Apache DolphinScheduler-এ কর্মী টাস্ক এক্সিকিউশন ব্রেকিং ডাউন

Aufschlüsselung der Worker-Task-Ausführung im Apache DolphinScheduler

Desglose de la ejecución de tareas de trabajo en Apache DolphinScheduler

Décomposer l'exécution des tâches de travail dans Apache DolphinScheduler

अपाचे डॉल्फिन शेड्यूलर में वर्कर टास्क निष्पादन को तोड़ना

Apache DolphinScheduler でのワーカータスク実行の詳細

Apache DolphinScheduler에서 작업자 작업 실행 분석

Quebrando a execução da tarefa do trabalhador no Apache DolphinScheduler

Разбор выполнения рабочей задачи в Apache DolphinScheduler

Apache DolphinScheduler'da İşçi Görevi Yürütme İşleminin Parçalanması

Phân tích quá trình thực thi tác vụ của Worker trong Apache DolphinScheduler

详解 Apache DolphinScheduler 中的 Worker 任务执行

Golden Celestial Ratio (GCR): A Comprehensive Guide to the Divine Ecosystem

গোল্ডেন সেলেস্টিয়াল রেশিও (GCR): ডিভাইন ইকোসিস্টেমের একটি ব্যাপক নির্দেশিকা

Golden Celestial Ratio (GCR): Ein umfassender Leitfaden zum göttlichen Ökosystem

Proporción áurea celestial (GCR): una guía completa del ecosistema divino

Ratio Céleste Doré (GCR) : Un guide complet de l'écosystème divin

स्वर्णिम आकाशीय अनुपात (जीसीआर): दिव्य पारिस्थितिकी तंत्र के लिए एक व्यापक मार्गदर्शिका

黄金比（GCR）：神の生態系への総合ガイド

황금 천체 비율(GCR): 신성한 생태계에 대한 종합 가이드

Proporção Celestial Áurea (GCR): Um Guia Abrangente para o Ecossistema Divino

Золотое небесное сечение (GCR): подробное руководство по божественной экосистеме

Altın Gök Oranı (GCR): İlahi Ekosistem İçin Kapsamlı Bir Kılavuz

Tỷ lệ thiên thể vàng (GCR): Hướng dẫn toàn diện về hệ sinh thái thần thánh

黄金天体比例（GCR）：神圣生态系统的综合指南

How to Create a Server Driven UI Engine for Flutter

ফ্লটারের জন্য কীভাবে একটি সার্ভার চালিত UI ইঞ্জিন তৈরি করবেন

So erstellen Sie eine servergesteuerte UI-Engine für Flutter

Cómo crear un motor de interfaz de usuario controlado por servidor para Flutter

Comment créer un moteur d'interface utilisateur piloté par serveur pour Flutter

फ़्लटर के लिए सर्वर संचालित UI इंजन कैसे बनाएं

Flutter 用のサーバー駆動型 UI エンジンを作成する方法

Flutter용 서버 기반 UI 엔진을 만드는 방법

Como criar um mecanismo de UI orientado a servidor para Flutter

Как создать серверный движок пользовательского интерфейса для Flutter

Flutter için Sunucu Odaklı UI Motoru Nasıl Oluşturulur

Cách tạo Công cụ giao diện người dùng điều khiển máy chủ cho Flutter

如何为 Flutter 创建服务器驱动的 UI 引擎

I'd like to share my experiences using evilginx2 and gophish for red teaming phishing simulations. While there are numerous guides available, I aim to highlight the nuances and practical insights that aren't typically covered in documentation and tutorials. Approaches When setting up a phishing campaign for red team purposes, you have three options: you can clone the page and only collect data
the next level is not only cloning the page but also verifying the captured data against a legitimate page
The last and most sophisticated method involves a man-in-the-middle attack, where your page is a proxy between the victim and the legitimate site. Evilginx2 The third method is done by evilginx2, which, despite its name, does not have nginx built in, but the functionality is the same, except that evilginx2 has built-in filters and functions like inject_js that make the attacker's life more comfortable. Gophish Gophish is a tool that allows you to manage groups and users, dynamically create emails, and send them at will. Integration A few weeks ago, Evilginx got an update 3.3, which introduced integration with Gophish, a framework for phishing campaigns. There have been several such projects in the past, such as gophish[1], but in this text, I will focus on the official evilginx by K. Gretzky. More info about the update with the demo can be found here: https://breakdev.org/evilginx-3-3-go-phish/ Both tools are in Golang, which is a relatively easy-to-understand language. Experience with Evilginx2 Malicious proxy with evilginx has several drawbacks compared to just cloning the page. Firstly, the setup is much more complicated. When you clone a page, you download the necessary files via developer tools and add the password-storing feature via some backend. Evilginx setup is more complicated. Setup needs a configuration file and a phishlet that works as a template for the phishing proxy. Setting up and running evilginx2 in the production version took me over a week compared to just cloning. The advantage is that evilginx scales much better. Once you know how to create them, you can change phishing scenarios at will because adding new phishlet is trivial. I should mention the state of the project in which evilginx is. The documentation is concise and not very useful if you have no experience with evilginx2 already. Some features and behaviors are not documented, and you need to read and debug the source code to find the behavior. The project has special financing where you can pay 400$ for a course of excellent quality, but it replaces the documentation to some extent, which doesn't seem like a good way. Either it's an open-source project with all the parts, or it's not open-source; this way, it's open-code, but you are forced into buying the expensive course. It doesn't help that the project has many open issues and bugs that nobody addresses. If you're serious about phishing, you'll probably have to edit the source code and use your version because the author doesn't put much effort into finding new PRs. For example, the PR request I made to fix the POST of a GET request three weeks[2] ago has still not been merged. The project is plagued by old libraries that are long unused. Of course, I don't want the author to handle everything himself, but sometimes, you have to give up control of the project to benefit the project. And more moderators and maintainers who want to help would certainly be found. But not to be only negative, this project is still the best of its kind. If you learn all its quirks and modify it to your purpose, it will become a tool allowing you to do campaigns of all kinds, with a level of automation that is impossible if you write the tooling yourself. Experience with Gophish Gophish isn't updated very often either, but I haven't encountered any bugs, so it doesn't really bother me. Although gophish has an API, it lacks some functionality that would be useful, so K.Gretzky had to add this API[3] to make the integration with evilginx work, anyway this approach is not sustainable and I think gophish would benefit from an API that would allow event reporting. The Gophish email template community is active, and you don't need to invest much time in a replica of the o365 email because there is a good chance that someone has already created the template. The UI is very good, and many features, like a timeline of events, will be appreciated by clients in the report. Bypassing anti measures If you send emails within a known service such as o365 or Gmail, it is good to first build an instance where you collect crawler IP addresses which you then block on the production instance and domain, thus minimizing the risk of being blacklisted, for example, outlook. Of course, this is the easier part. The harder part is to have an SMPT server with a good reputation that allows you to send phishing emails in bulk, so I do not recommend building your own SMTP server but rather use some email as a service with its own domain that has a good server reputation by default, for example, Gmail business. Summary While evilginx2 offers powerful features and scalability, its complex setup and poor documentation are significant hurdles.
Gophish provides robust campaign management tools but also suffers from limited API functionality.
Both tools require considerable customization and understanding to effectively use in red team operations.
Mastering both tools offers significant rewards. Thank you for reading. Feedback is appreciated! References [1]: https://github.com/fin3ss3g0d/evilgophish [2]: https://github.com/kgretzky/evilginx2/pull/1047 [3]: https://github.com/kgretzky/gophish I'd like to share my experiences using evilginx2 and gophish for red teaming phishing simulations. While there are numerous guides available, I aim to highlight the nuances and practical insights that aren't typically covered in documentation and tutorials. Approaches When setting up a phishing campaign for red team purposes, you have three options: you can clone the page and only collect data the next level is not only cloning the page but also verifying the captured data against a legitimate page The last and most sophisticated method involves a man-in-the-middle attack, where your page is a proxy between the victim and the legitimate site. you can clone the page and only collect data the next level is not only cloning the page but also verifying the captured data against a legitimate page The last and most sophisticated method involves a man-in-the-middle attack, where your page is a proxy between the victim and the legitimate site. Evilginx2 The third method is done by evilginx2 , which, despite its name, does not have nginx built in, but the functionality is the same, except that evilginx2 has built-in filters and functions like inject_js that make the attacker's life more comfortable. evilginx2 evilginx2 nginx evilginx2 inject_js Gophish Gophish is a tool that allows you to manage groups and users, dynamically create emails, and send them at will. Gophish Gophish Integration A few weeks ago, Evilginx got an update 3.3 , which introduced integration with Gophish, a framework for phishing campaigns. There have been several such projects in the past, such as gophish[1], but in this text, I will focus on the official evilginx by K. Gretzky. 3.3 More info about the update with the demo can be found here: https://breakdev.org/evilginx-3-3-go-phish/ https://breakdev.org/evilginx-3-3-go-phish/ Both tools are in Golang, which is a relatively easy-to-understand language. Experience with Evilginx2 Malicious proxy with evilginx has several drawbacks compared to just cloning the page. Firstly, the setup is much more complicated. When you clone a page, you download the necessary files via developer tools and add the password-storing feature via some backend. Evilginx setup is more complicated. Setup needs a configuration file and a phishlet that works as a template for the phishing proxy. Setting up and running evilginx2 in the production version took me over a week compared to just cloning. The advantage is that evilginx scales much better. Once you know how to create them, you can change phishing scenarios at will because adding new phishlet is trivial. I should mention the state of the project in which evilginx is. The documentation is concise and not very useful if you have no experience with evilginx2 already. Some features and behaviors are not documented, and you need to read and debug the source code to find the behavior. The project has special financing where you can pay 400$ for a course of excellent quality, but it replaces the documentation to some extent, which doesn't seem like a good way. Either it's an open-source project with all the parts, or it's not open-source; this way, it's open-code, but you are forced into buying the expensive course. It doesn't help that the project has many open issues and bugs that nobody addresses. If you're serious about phishing, you'll probably have to edit the source code and use your version because the author doesn't put much effort into finding new PRs. For example, the PR request I made to fix the POST of a GET request three weeks[2] ago has still not been merged. The project is plagued by old libraries that are long unused. Of course, I don't want the author to handle everything himself, but sometimes, you have to give up control of the project to benefit the project. And more moderators and maintainers who want to help would certainly be found. But not to be only negative, this project is still the best of its kind. If you learn all its quirks and modify it to your purpose, it will become a tool allowing you to do campaigns of all kinds, with a level of automation that is impossible if you write the tooling yourself. Experience with Gophish Gophish isn't updated very often either, but I haven't encountered any bugs, so it doesn't really bother me. Although gophish has an API, it lacks some functionality that would be useful, so K.Gretzky had to add this API[3] to make the integration with evilginx work, anyway this approach is not sustainable and I think gophish would benefit from an API that would allow event reporting. The Gophish email template community is active, and you don't need to invest much time in a replica of the o365 email because there is a good chance that someone has already created the template. The UI is very good, and many features, like a timeline of events, will be appreciated by clients in the report. Bypassing anti measures If you send emails within a known service such as o365 or Gmail, it is good to first build an instance where you collect crawler IP addresses which you then block on the production instance and domain, thus minimizing the risk of being blacklisted, for example, outlook. Of course, this is the easier part. The harder part is to have an SMPT server with a good reputation that allows you to send phishing emails in bulk, so I do not recommend building your own SMTP server but rather use some email as a service with its own domain that has a good server reputation by default, for example, Gmail business. Summary While evilginx2 offers powerful features and scalability, its complex setup and poor documentation are significant hurdles. Gophish provides robust campaign management tools but also suffers from limited API functionality. Both tools require considerable customization and understanding to effectively use in red team operations. Mastering both tools offers significant rewards. While evilginx2 offers powerful features and scalability, its complex setup and poor documentation are significant hurdles. Gophish provides robust campaign management tools but also suffers from limited API functionality. Both tools require considerable customization and understanding to effectively use in red team operations. Mastering both tools offers significant rewards. Thank you for reading. Feedback is appreciated! References [1]: https://github.com/fin3ss3g0d/evilgophish https://github.com/fin3ss3g0d/evilgophish [2]: https://github.com/kgretzky/evilginx2/pull/1047 https://github.com/kgretzky/evilginx2/pull/1047 [3]: https://github.com/kgretzky/gophish https://github.com/kgretzky/gophish

Red Team Phishing Simulations: Using Evilginx2 and GoPhish

Unlocking IaC Part 5: What Is Git Branching?

The famous cypherpunk mailing list from which Bitcoin (and other privacy tools) came was founded in late 1992 by the activists Eric Hughes, Tim May, and John Gilmore. However, this is far from being the only merit attributed to Gilmore. Indeed, while that specific mailing list isn’t active anymore, other important projects launched or supported by Gilmore are still around, always helping to keep our privacy and digital rights. Born in 1955 in Pennsylvania (US), . Gilmore's journey as a programmer began in 1982 at Sun Microsystems (now Oracle), where he was the fifth employee. John Gilmore is a pioneer in software development and cryptography, who made significant contributions that continue to shape the digital landscape He focused there on further developing Berkeley Software Distribution (BSD), an open-source operating system. In 1985, he also co-authored the Bootstrap Protocol, which evolved into DHCP, a fundamental mechanism for assigning IP addresses on networks. This essentially ensures that devices connected to a network can communicate with each other, forming the backbone of modern Internet connectivity. his efforts Likewise, Gilmore is a frequent collaborator in free software and has been involved in various GNU projects. They include maintaining the GNU Debugger and initiating GNU Radio, demonstrating his commitment to fostering open-source solutions. Besides, his entrepreneurial spirit led him to found Cygnus Solutions in 1989. This was a company dedicated to supporting free software and was sold to Red Hat in 1999. Electronic Frontier Foundation (EFF) In 1990, Gilmore took another step to fight for privacy and freedom online. Along with John Perry Barlow (author of the ), and Mitch Kapor (creator of Firefox), they founded the Electronic Frontier Foundation ( ). Declaration of Independence of Cyberspace EFF This is an NGO with a mission to safeguard online rights, whether it entails combatting mass surveillance or defending freedom of expression worldwide. Not only do they advocate for these causes, but they also equip people with the requisite tools and insights to fortify their online defenses. Through legal advocacy, the EFF takes on cases involving privacy violations, government surveillance, and free speech issues. They also work to shape policies and legislation that protect digital rights, promote Internet freedom, and develop technologies to enhance online privacy and security. One notable example is HTTPS Everywhere, a browser extension created in collaboration with the Tor Project. HTTPS Everywhere automatically switches websites from insecure HTTP to secure HTTPS, enhancing users' security and privacy while browsing the web. Other significant EFF software projects are Privacy Badger, a browser extension designed to block online trackers and advertisements; and Let's Encrypt, a free, automated, and open certificate authority that enables website owners to obtain and install SSL/TLS certificates for secure HTTPS connections. Cypherpunks and Activism Gilmore didn’t create any form of digital money himself, but he supported as a movement from the very beginning. The first physical meetings were held between Eric Hughes’s house and Cygnus Solutions headquarters in California, whereas the mailing list was originally hosted on toad.com, a domain owned and maintained by Gilmore. The mission of cypherpunks is protecting privacy and freedom by using cryptographic tools, something fully shared by : the cypherpunks Gilmore too “I want a guarantee—with physics and mathematics, not with laws—that we can give ourselves things like real privacy of personal communications. Encryption strong enough that even the NSA can’t break it.” Additionally, Gilmore has been a vocal advocate for digital rights and civil liberties. such as the creation of the FreeS/WAN project for IPsec encryption and sponsorship of the EFF's Deep Crack DES cracker. Both projects worked towards making digital communication more secure by either enhancing encryption standards or demonstrating weaknesses in existing ones. He’s also been involved in the development of numerous software projects aimed at enhancing digital security and privacy, Gilmore's commitment to promoting Internet freedom has also led him to support initiatives aimed at increasing transparency and accountability in government surveillance practices. He has been involved in legal battles challenging unconstitutional surveillance and censorship, demonstrating his dedication to defending civil liberties in the digital age. A remarkable case was , where he challenged the requirement to show identification for domestic air travel, arguing it violated his constitutional rights to travel. He also objected to the lack of transparency regarding the directive mandating ID checks, which is fair: although the courts reviewed the Security Directive privately, its contents remain undisclosed to the public. This is deemed an unconstitutional “ .” Sadly, despite Gilmore's claims, the court ruled against him for now. Gilmore v. Gonzales secret law Against Censorship Gilmore, as much as other cypherpunks, has a firm position against censorship. He’s still active in the fight for privacy by like Freedom Box, a small device that works as an independent server to avoid surveillance; and Freedom of Movement, the legal battle against secret laws. , censorship is a social weakness, and it doesn’t offer any advantages. supporting projects For him “Censorship is a counterproductive social policy and weakens the national security, by suppressing the flow of useful information among the honest citizenry. Widespread use of encryption also enhances the national security, by making private information more truly private, and by making systems and networks harder for dishonest people to penetrate.” In these days of censorship and surveillance, we need decentralized and privacy tools more than ever. Luckily, developers and activists like Gilmore keep working to deliver them free to use. One of them is the . Obyte crypto network is structured as a Directed Acyclic Graph (DAG) to achieve consensus and record transactions. This decentralized approach ensures greater transparency, security, and resilience against censorship and surveillance. Its architecture enables peer-to-peer transactions without the need for intermediaries. Moreover, its open and permissionless nature fosters innovation, allowing developers to build decentralized applications (Dapps) and smart contracts on the platform. Obyte Besides, relying on the , users have the option to verify their personal information, such as their real name, while ensuring it remains securely stored in their wallet and nowhere else. This verified information can then be shared selectively with trusted parties, allowing others to check its authenticity using the Obyte DAG. Self-Sovereign ID features Importantly, users retain full control over what information they share. By decentralizing control and empowering users with sovereignty over their data and assets, Obyte paves the way for a more inclusive and equitable digital economy. Just like John Gilmore and other cypherpunks have dreamed of. Read more from series: Cypherpunks Write Code Tim May & Crypto-anarchism Wei Dai & B-money Nick Szabo & Smart Contracts Adam Back & Hashcash Eric Hughes & Remailer St Jude & Community Memory Hal Finney & RPOW Featured Vector Image by Garry Killian / Freepik John Gilmore Photograph by Joi Ito / Flickr

Cypherpunks Write Code: John Gilmore and the Electronic Frontier Foundation

সাইফারপাঙ্কস কোড লিখুন: জন গিলমোর এবং ইলেকট্রনিক ফ্রন্টিয়ার ফাউন্ডেশন

Cypherpunks schreiben Code: John Gilmore und die Electronic Frontier Foundation

Los Cypherpunks escriben código: John Gilmore y la Electronic Frontier Foundation

Cypherpunks écrivent du code : John Gilmore et l'Electronic Frontier Foundation

साइफरपंक्स राइट कोड: जॉन गिलमोर और इलेक्ट्रॉनिक फ्रंटियर फाउंडेशन

サイファーパンクはコードを書く: ジョン・ギルモアと電子フロンティア財団

사이퍼펑크의 코드 작성: 존 길모어(John Gilmore)와 전자 프론티어 재단(Electronic Frontier Foundation)

Cypherpunks escrevem código: John Gilmore e a Electronic Frontier Foundation

Шифропанки пишут код: Джон Гилмор и Electronic Frontier Foundation

Şifrepunklar Kod Yazıyor: John Gilmore ve Electronic Frontier Foundation

Cypherpunks viết mã: John Gilmore và Electronic Frontier Foundation

密码朋克编写代码：约翰·吉尔摩和电子前沿基金会

581 Stories To Learn About Non Fiction

Products like Amazon RDS for are fine for smaller deployments, but scaling PostgreSQL is a different story. Once the project grows to many terabytes, these managed databases get slow and expensive, making data management much more complex. PostgreSQL Performance suffers once tables reach billions of rows, and while , the data will carry on growing. Without proper data management, developers can only watch their disks (and bills) increase in size. it there are ways to improve But not anymore. Today, we’re happy to announce the General Availability of Tiered Storage, a multi-tiered storage architecture engineered to enable infinite, low-cost scalability for your time series and analytical databases in the Timescale platform. You can now store your older, infrequently accessed data in a low-cost storage tier while still being able to access it—without ever sacrificing performance for your frequently accessed data. This is what happens when you insert data into a Timescale time-series database with our new multi-tiered storage backend: Your most recent data will be written into a optimized for fast queries and high ingests. high-performance storage tier Once you don’t access that data frequently, you can automatically tier it to a by setting up a tiering policy. The data in the low-cost storage tier remains fully queryable within your database, and there’s no limit to the amount of data you can store—up to hundreds of TBs or more. lower-cost object storage tier Our low-cost storage tier has a flat price of $0.021 per GB/month for data—cheaper than Amazon S3. Developers need a cheap way to scale their large PostgreSQL databases in without compromising performance. While object stores are amazingly scalable and affordable, they are not the fastest, and developers need to get millisecond query responses for their applications. AWS also However, once data becomes older and rarely accessed, real-time performance is often not as essential. Developers still need to be able to access this historical data for ad hoc queries, data analysis, or regulatory compliance, but they can assume some latency for these types of queries. Now, what developers want is the ability to store this historical data as affordably and efficiently as possible. This new Tiered Storage architecture frees developers from choosing between storage costs and performance trade-offs for real-time applications. By keeping their recent and frequently accessed data in the high-performance tier, they’ll leverage the millisecond query speed and ingestion capabilities of Timescale, and by tiering their older data, they can keep as many TBs as they need in their PostgreSQL databases for less. PostgreSQL Data Management With Tiered Storage: What You Need to Know Timescale’s Tiered Storage architecture leverages the flexibility of PostgreSQL and for effective data management. When creating a hypertable, it can now seamlessly stretch across both storage tiers; , Timescale seamlessly figures out what storage tiers to access to generate the response, boosting performance for your most recent data and lowering storage costs for older data. There’s no extra charge per query or data read and no hidden fees, enabling straightforward, cost-effective data management. hypertables when you run a query This storage architecture also eliminates any storage limitations in Timescale services: since our low-cost storage tier is infinite, you can store as many TBs as you’d like. For example, we’re leveraging Tiered Storage internally to store the huge Timescale database powering our Insights product. This Insights database is constantly collecting and analyzing query statistics from across our customer fleet, and it’s exceeded 350 TB today and is growing quickly. From those 350 TB, 250 TB are tiered to low-cost storage. Let’s do the math: We store 5 TB in our high-performance storage tier Of course, we have compression enabled, and we’re getting compression ratios of 20x—meaning that what was originally 100 TB of Postgres data now fits in a 5 TB disk thanks to compression (!) after compression. The remaining 250 TB of data are stored in the low-cost storage tier. This tiering happens automatically once data hits a certain age, which currently is several weeks old. Our customers with large deployments are also already making use of Tiered Storage: "We perform a lot of analysis on market data, and the sheer volume of data we need to store makes a normal disk-based database solution unfeasible (it's just too expensive). Timescale’s Tiered Storage seamlessly allows us to move large volumes of data to the object storage layer. This is a great solution to store large volumes of historical data and perform post-analysis. Without this, we'd be forced to develop a solution in-house." - Chief Technology Officer at a Proprietary Digital Assets Trading Company Simplicity is the key feature of Tiered Storage. To move your data from the high-performance tier to the low-cost object tier, all you need to do is use our to define policies to tier your data as it ages on a particular hypertable. Data tiering policies work on a chunk basis (the policy will tier complete chunks once they reach the age defined by the policy). There’s no need for any ETL (extract-transform-load) processes or infrastructure changes. simple API Timescale hypertables are PostgreSQL tables automatically partitioned by time. Those partitions are called chunks. Chunks are the unit of user-defined policies in Timescale: for example, when you define a data retention policy, you’ll be deleting entire partitions (chunks), and when you move data between storage tiers, you’ll be moving entire chunks. This is a very convenient way to manage data from a resource-utilization and developer experience perspective. Editor's Note: For example, this policy would move all data older than one month to object storage in your hypertable: events SELECT add_tiering_policy('events', INTERVAL '1 month'); That’s all you need! Everything else happens automatically, including intelligent query planning that only executes any SQL queries on the appropriate tier. To drop data currently stored in the low-cost storage tier, you can define a data retention policy so data is deleted automatically after a certain period (e.g., after five years). . You can also manually delete specific chunks Also, if you want to “move back” a particular chunk from the low-cost storage tier to the high-performance tier ( ), you can easily “untier” it. for example, if you need to backfill or update data -- Untier a particular chunk
CALL untier_chunk('_hyper_1_1_chunk'); You can track how much data has been tiered (and how much it will cost at the end of the month) in the Timescale console: And speaking of billing estimations… How much can I save with Tiered Storage? Our high-performance storage tier has an effective price of $0.177 per GB/month of data (considering the expected compression rate we see across our fleet).  This is now joined by a low-cost storage layer with a flat fee of $0.021 per GB/month of data, with the same price across all cloud regions. after compression When tiering data, you will , not for the queries executed or the amount of data scanned: this is truly a flat price. Our goal with this pricing structure was to provide a transparent, unambiguous, and simple way to calculate the total data storage cost, making it easier to manage your data. only pay for the data you store As a quick example, let’s say you have a hypertable with 5.2 TB of storage, with recent hypertable chunks and other Postgres tables taking up approximately 200 GB and about 5 TB of hypertable data older than a month. You do not access or query this older data frequently, meaning you don’t need it for the daily operations of your application. Still, you’d like to keep it accessible in your database for ad hoc queries or compliance requirements (we see many such cases among our customers). As a cost-effective data management strategy, you can tier all chunks older than a month to the low-cost tier and decrease the cost of storing that 5 TB of data from around $478/month to around $105/month, ( a 78% decrease in your storage bill. For this estimation, we assume you enabled compression for your hypertable and consider the median overall compression in all Timescale services). The savings will increase alongside your data: when moving multiple terabytes to this low-cost tier, your storage bill will decrease from thousands of dollars to a few hundred. The following reference table illustrates how affordable our low-cost storage tier actually is. You get the point! The savings’ multiplication effect of Tiered Storage There is one more thing that makes Tiered Storage even more amazing: when you keep data in the low-cost storage tier, you pay for this data only once, independently of if you have a high-availability replica or read replicas running in your service. The same applies to forks. In Timescale, you can create copies of your primary database (we call them forks) by clicking one button from the UI, for example, for running tests or creating dev environments. When creating one (or more) forks, . If you decide to tier more data that’s not in the primary, you will pay to store it in the low-cost tier, but you will still benefit from substantial savings by moving it from the high-performance tier of the fork to the cheaper tier. you won’t be billed for data shared with the primary in the low-cost storage To make this crystal clear, let’s roll out an example. Imagine that you have a primary service with 6.5 TB of data and that you also have set: A high-availability replica to significantly reduce the risk of downtime and data loss due to failures A read replica to serve your read queries and allow the primary to dedicate itself fully to writes A fork of that service for development and testing purposes From a billing perspective, if you kept the 6.5 TB of data in your primary service in the high-performance storage tier, you would see reflected [6.5 TB x 4] in your storage bill to account for the two replicas, the fork, and the primary service—26 TB in total. Assuming our median compression rate, this would be costly: around $4,602 /month. But what happens if your application needs to actively access only the most recent 500 GB of data? You could tier 6 TB to low-cost storage and only keep 500 GB on your high-performance storage tier. And since you only pay for the data in your low-cost storage tier once, this is how your new storage bill would look like: [500 GB x 4] = 2 TB in high-performance storage (instead of 26 TB) [6 TB x 1] in the low-cost storage tier The above storage bill would come up to approximately $480 /month: This is the savings' multiplication effect of Tiered Storage. Especially if you’re running replicas or forks, it is a great idea to take advantage of the low-cost storage tier—the savings you’ll see in your overall bill will be very significant. you would save over $4,000/month! Our Tiered Storage Journey We released an early version of our Tiered Storage functionality in the Timescale platform as Early Access in March. After many improvements, it has now reached General Availability. Since its first release, we have worked hard to make Tiered Storage stable, reliable, and faster. We also thought long and hard about our pricing model, iterating consistently over multiple ways to price our low-cost storage tier. Finally, we leaned towards the simplest: a flat rate over the data volume pre-compression. We’ve already mentioned how we valued simple and predictable pricing, but it was also important for us to provide a price per GB/month as low as possible. Our price landed at $0.021—as a comparison, storing files on Amazon S3 costs . $0.023 per GB/month Personally, I (Yannis) must admit that after leading teams building cloud-native solutions for more than a decade, I still have to go back and occasionally recheck multiple rate tables on various cloud pricing pages, especially looking for extra fees, every time that I want to compute the total costs for our services accurately. At Timescale, we believe that you should not have to build a complicated spreadsheet to be able to run a database service or make informed choices about storage layers. Our commitment to making developers’ lives easier led us to the flat rate of $0.021 per GB/month—no guesswork, hidden costs, or charges per query or data reads. means the data volume before applying Timescale compression. For example, due to Timescale compression, a 500 GB volume in the high-performance storage tier might end up needing only 50 GB of disk space once compressed. If you decided to tier this data to low-cost storage, your bill would be calculated over the original 500 GB volume, as in [500 GB * $0.021] per month. Data volume pre-compression How Tiered Storage Works: Behind the Scenes All data inserted into Timescale is initially written into our high-performance storage layer. Using faster disks for your most recent data will bring top insert and query performance for your most recent values, a usage pattern that fits the needs of data-intensive applications. On the contrary, the low-cost storage tier is an object store built on Amazon S3. This object store is much more than an external bucket to archive your data, though: it’s an integral part of your database. When you move data to this object storage tier, your database will remain fully aware of all the semantics and metadata, and you can keep querying as usual with standard SQL (albeit with slower performance). Behind the scenes, we’re storing the data in a compressed columnar format (specifically, ). When data is tiered, chunks stored in Timescale’s native internal database format (typically in our ) are asynchronously converted to Parquet format and stored in the underlying S3 object. We’ve built several mechanisms to ensure that the tiered chunks are durably stored in the low-cost storage tier before transactionally removing them from the high-performance tier. Apache Parquet native columnar compression When you run your SQL query, it will transparently pull data from the high-performance storage tier, the object storage tier, or both, as required. When we say transparent, we mean transparent: Timescale supports arbitrarily complex queries across its standard and tiered data, including complex predicates, JOINs, CTEs, windowing, hyperfunctions, and more. In the example query below (with an clause), you can see how the query plan includes a when the database is accessing data from the object storage tier. In this example, and are standard Postgres tables (residing only in high-performance storage), while is a hypertable that stretches across both storage tiers.  When executing this query against the hypertable, three of its chunks were read from standard storage, and five chunks were read from object storage. EXPLAIN Foreign Scan devices sites metrics metrics EXPLAIN 
SELECT time_bucket('1 day', ts) as day,
        max(value) as max_reading, 
        device_id  	
    FROM metrics 
    JOIN devices ON metrics.device_id = devices.id 
    JOIN sites ON devices.site_id = sites.id
WHERE sites.name = 'DC-1b'
GROUP BY day, device_id
ORDER BY day;


QUERY PLAN                                                      
----------------------------------------------------------
GroupAggregate
    Group Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
    -> Sort
        Sort Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
        -> Hash Join
            Hash Cond: (_hyper_5666_706386_chunk.device_id = devices.id)
            -> Append
    -> Seq Scan on _hyper_5666_706386_chunk
                -> Seq Scan on _hyper_5666_706387_chunk
                -> Seq Scan on _hyper_5666_706388_chunk
                -> Foreign Scan on osm_chunk_3334
            -> Hash
                -> Hash Join
                    Hash Cond: (devices.site_id = sites.id)
                    -> Seq Scan on devices
                    -> Hash
                        -> Seq Scan on sites
                           Filter: (name = 'DC-1b'::text) In the above query plan, the corresponds to fetching data from the bottomless object storage tier. Foreign Scan on osm_chunk_3334 To avoid processing chunks falling outside the query’s time window, we perform chunk exclusion to only touch the chunks that are needed to satisfy the query. The database stores various forms of metadata to build a “map” of row groups and columnar offsets within object storage. Furthermore, when a query is run, it is further selective about the data it reads. If your query only touches a range of rows and a few columns, only that subset of the data is read from the S3 object behind the low-cost storage tier. In the above, for example, only the and columns are read from the object storage tier.  If an additional  time-based filter had been included, the database would first use its metadata (stored in high-performance storage and cached in memory) to further reduce which Parquet files and row groups need to be read to execute the query.  All to reduce query latency even when accessing this bottomless storage transparently through PostgreSQL. device_id value WHERE Give Tiered Storage a Try Storage decisions around historical data don’t have to be costly. In Timescale, you now have access to a low-cost, infinite storage tier with no pricing gotchas that allows you to scale your database storage for an affordable price without compromising the performance of your application. If you’re wondering if this is a good solution for your use case, If you’re looking for a more practical perspective, you can also experiment with . check out our Docs for more information and use case fits. Tiered Storage yourself for free in the Timescale platform via our free trial (no credit card required) - Written by Yannis Roussos, Carlota Soto, and Ana Tavares. Also published here.

Scaling PostgreSQL Databases Just Got Cheaper: Timescale's Tiered Storage Hits General Availability

স্কেলিং PostgreSQL ডাটাবেসগুলি সবেমাত্র সস্তা হয়েছে: টাইমস্কেলের টায়ার্ড স্টোরেজ সাধারণ উপলব্ধতাকে আঘাত করে

Die Skalierung von PostgreSQL-Datenbanken ist jetzt noch günstiger: Der mehrstufige Speicher von Timescale ist jetzt allgemein verfügbar

Escalar bases de datos PostgreSQL ahora es más barato: el almacenamiento por niveles de Timescale llega a la disponibilidad general

La mise à l'échelle des bases de données PostgreSQL est devenue moins chère : le stockage hiérarchisé de Timescale arrive en disponibilité générale

PostgreSQL डेटाबेस को स्केल करना अब सस्ता हो गया है: टाइमस्केल का टियर स्टोरेज सामान्य उपलब्धता को प्रभावित करता है

PostgreSQL データベースのスケーリングがさらに安価に: Timescale の階層型ストレージが一般提供開始

PostgreSQL 데이터베이스 확장 비용이 더 저렴해졌습니다. Timescale의 계층형 스토리지가 정식 출시되었습니다.

O dimensionamento de bancos de dados PostgreSQL ficou mais barato: o armazenamento em camadas do Timescale atinge a disponibilidade geral

Масштабирование баз данных PostgreSQL стало дешевле: многоуровневое хранилище Timescale стало общедоступным

PostgreSQL Veritabanlarını Ölçeklendirmek Artık Daha Ucuz: Timescale'in Katmanlı Depolaması Genel Kullanıma Çıktı

Mở rộng cơ sở dữ liệu PostgreSQL vừa rẻ hơn: Bộ lưu trữ theo cấp độ của Timescale đạt mức khả dụng chung

扩展 PostgreSQL 数据库变得更加便宜：Timescale 的分层存储全面上市

PROMISES FULFILLED.

Stories, News & Mentions about CANID

Canid’s Stories on HackerNoon