Elyps is a regulated investment platform giving investors access to digital assets, featuring a Euro current account and an International VISA card

Elyps

elyps

This interview series celebrates all the phenomenal . Congratulations! Well-Deserved, Champion.The HackerNoon community is excited to learn more about you! Startups of the Year Winners Lugano Startup in the ranking. What does it mean for you to win this title? Winning HackerNoon's prestigious Startups of the Year 2023 title is a monumental achievement for us at Elysium, especially as a burgeoning startup from Lugano. This award validates our early efforts and fuels our ambition to revolutionize digital asset management using Lugano as the first market approach and testing pool. Starting our journey in Lugano, where over 400 merchants accept crypto payments, is just the beginning. Our aspirations stretch far beyond, aiming to redefine the self-custody experience for users worldwide. What sets you apart from the competition? What sets Elysium apart is our pioneering Onboarding algorithm and Assets Management system, designed with a user-centric philosophy. Unlike traditional wallets focusing on key storage, Elysium innovates on key creation and reconstruction, ensuring unmatched security and user convenience. Our approach to self-custody, where users' needs shape the technology, establishes Elysium as the first of its kind - a wallet that's as dynamic and flexible as its users. What do you love about your team, and why are you the ones to solve this problem? The strength of Elysium lies in our team's commitment and shared vision. Each professional member brings a unique blend of passion and expertise, united by the goal of building the future, being part of the blockchain space since 2017, and not just following trends. This collective drive is our most incredible people in addressing the complexities of digital asset management. We are proud of everyone on the team who has decided to be part of the journey. If you weren’t building your startup, what would you be doing? Should Elysium not have been our path, our inherent drive to innovate would have found another channel to contribute to the world. Solving humanity's challenges through innovation is not just what we do; it's who we are. At the moment, how do you measure success? What are your core metrics? As an early-stage startup, our success is measured by the positive feedback from professionals, investors, and beta testers. With our official market launch set for 2024, we anticipate this year to be pivotal in achieving real market traction and setting new benchmarks in digital asset management. Tell us how your startup is changing the world. Elysium is not just a wallet; it's a gateway to a more secure, efficient, and user-friendly digital asset ecosystem. Our Algorithms and focus on seamless asset management and payments mark the dawn of a new era in self-custody. What goals are you looking forward to accomplishing in 2024? Our roadmap for 2024 is ambitious, including securing additional funding, launching in Europe and beyond, improving our business model, and exploring the B2B side of our product. Each milestone is a step towards realizing our vision of becoming the "Neo-Bank Like" for Bitcoin and digital assets. Which trend(s) are you most excited about in 2024? Share your reasons. We're particularly excited about the potential of real-world assets on the blockchain, advancements in Bitcoin payments, AI-driven automation and optimization, and the applications of Zero Knowledge proof for digital identity. These trends represent the frontier of innovation in our industry. 2023 had been another crazy year, especially in tech, with layoffs and the Generative AI takeover! Which trend are you most concerned about? Be as brief or as detailed as you like. Indeed, AI has advantages and disadvantages, and we are concerned that every team member uses it to increase the quality of work; for us, AI is not a substitute but an enhancement that will be part of our growth. On the cybersecurity and social intelligence side, we are monitoring developments with our cybersecurity experts and building strategies to mitigate cyber threats related to identity simulation for social attacks and other AI and cybersecurity attacks. Any words of wisdom you’d like to share with us? Reflecting on Theodore Roosevelt's words, we're reminded that true achievement lies in effort, striving towards great goals, and resilience in facing challenges. As we navigate the complexities of the digital asset space, we're committed to daring greatly, driven by the belief that the greatest victories belong to those who dare to venture into the future.

Meet Elysium Lab, Winner of the Startups of the Year in Lugano, CH

Co-authored by Brennon Hampton The ixo protocol defines how data packaged within hashed and signed messages get processed and stored. In the context of ixo, these data resources are described as verifiable impact claims. It turns out that merkelized private data stores are a fundamental piece of the architecture for most Web 3.0 applications that will have uses beyond the ixo protocol. We have ended up building a generic, highly configurable new database utility that implements core new Web 3.0 standards and uses hash-chains to ensure integrity. cryptographically Today we are excited to announce that we will make this available as an open-source release that we call Elysian. The name Elysian derives from the Ancient Greek term for reserving admission in perpetuity to those chosen by the gods, the righteous, and the heroic. The Elysian release of ixo data stores provides a powerful set of tools for controlling access to private data on a permissioned basis to those services that can prove their credentials and capabilities, using the W3C Decentralized Identifier (DID) specification. Only the chosen ones who have proven their credentials can access Elysian data stores, using the powerful capabilities of decentralized authentication and the magic of applied cryptography. For ixo, impact data must be privately owned and access controlled by the creator of an impact project. The claims relating to a project will typically include sensitive information, such as personal identifiable data. The storage of this data must comply with jurisdictional, data protection and privacy regulations. We also want these stores to be censorship-resistant, with as little reliance as possible on central agencies to host project data. To be useful for the many applications of the ixo Protocol, these data stores need to be dynamically customisable, immutable, non-repudiable and only accept requests from verifiable digital identities, driven by their credentials and capabilities. The data stores must also be fast, cheap, secure and make it easy to perform queries on the stored data. It turns out that these are core data storage requirements for many other Web 3.0 applications. ixo How these data stores are deployed in the ixo network, is that each project that is set up on the ixo network points to an independently hosted data store to which the project’s data will be directed. This data remains under the full private control and ownership of the project creator. What goes into the data store and who can write or read the data is set up dynamically and controlled at the level of the data store. This data storage is entirely ‘off-chain’ from the ixo network or any other public . However, the proofs of the data submissions are stored on the ixo public ledger, together with limited metadata that is extremely useful for tracking the performance of projects, indexing and locating the data. We call this the Global Impact Ledger. blockchain As the signed messages containing project data (impact claims) are passed through the ixo blockchain for processing, these are validated by consensus and appended to the hash-chain record within the project’s data store. Effectively, each project has its own project hash-chain/s that have been validated. Refer to the ixo White Paper for more details about how the protocol then enables verification of these claims, tokenizes the proofs of impact and enables trading of these tokens and their underlying data. Elysian data stores are highly customisable On initialising an Elysian Data Store, a dynamically customisable list of identity credentials and capabilities is established to define what actions can be performed, by which of the identified agents that will interact with the data store. These permissions can be updated or revoked at any time. Access control is fine-grained and complies with the consent requirements for data protection and privacy. Data stored within an Elysian data store is immutable as this is an append-only data store (data cannot be deleted or updated). All data operations are authenticated by confirming that the digital identifier relating to the service or agent that signed the incoming request has the capability to perform this action. The data is all verifiable, as this conforms with the W3C standards for verifiable claims and credentials. Only message payloads that match the predefined JSON submission template set up for data store can be submitted. The associated credentials, such as cryptographic signatures, must also verify. The public key associated with the digital identifier that signed the request is retrieved via the identifier blockchain record, from the identity credential provider and validated. Once verified, we insert the record and log the transaction in a timestamped transaction log with a hash that includes the previous transactions hash. To close the loop, for every verified message that passes into the Elysian data store, an event is triggered to the ixo blockchain.This ledgers the transaction and updates the publicly viewable project data. A Web 3.0 data utility for all With Elysian, we believe that we are providing a core building-block for many types of data-intensive Web 3.0 applications that will be fit for the purposes of decentralised architectures. We look forward to taking this into full production as a secure and highly useful piece of the data infrastructure for the ixo network. We encourage others to get involved in the project to help build out the exciting pipeline of features and uses of Elysian, as an open-source utility for all. Join the Community Learn more about ixo and how to get involved at ixo.foundation Follow ixo on Twitter Join the conversation on we’d love to answer your questions! Telegram Like ixo on Facebook See the code and contribute on Github We’re on too Reddit Subscribe to the ixo e-mail newsletter Email the team info@ixo.foundation

Elysian Release of the ixo Data Store

When James Cameron first teased Terminator 2 back in the day, it was through a 90-second teaser that showed an automated robot factory pumping out endless rows of robots, ending with a shot of Arnold as the build process completed. Pure pulp fiction. Or was it? Fast forward to 2024, and we are now entering the era of the hyperfactory, where the main body of cars is autonomously assembled every 76 seconds. No, that’s not a mistype because Xaoimi, the smartphone OEM, has entered the electric vehicle arena with the SU7 and with it, a factory setup that is 91% automated with 400 robots armed with HD cameras and can operate in the dark. It reminds me of the time I spent on stage at the Big Data Expo in China with the chairman of Foxconn, and we debated the merits of running a smartphone factory with the lights off — because who needs lights when it’s just the robots running the show? The factory of the future will have only two employees, a man and a dog. The man will be there to feed the dog. The dog will be there to keep the man from touching the equipment.
— Warren Bennis Watching the keynote reveal is an eye-opening vision of the future of manufacturing but not in the way you think. “We have 269 welding machines to achieve 100% automated production and have the 36 robots to achieve 0.5 millimeters of assembly precision” “…industry leading seven axis coating robots with 70,000 RPM, it can make the paint coverage extremely fine balanced. This produces a natural flat and smooth finish..” “AMR robots that can replace forklifts/internal transport around the factory…” The best part is that the cars can then drive themselves off the factory line with the autopilot software that the company has introduced. In all, it’s a fascinating look at the future of car manufacturing. But there’s something else to consider. We are looking at the precursor of mass-produced, highly automated robot and drone production, something that I can’t help but compare to that dystopian movie. Because, let’s face it, everything is starting to point towards it. We have major AI players all rushing to give their hyperbolic algorithms some form of physical embodiment to contextually understand the world around them in a new dimension. The idea that military, DARPA, or even startup funding won’t find its way to a hyperfactory of this kind is ludicrous because they’ll be eyeing this up as a way to mass-produce killer drones and robotic tech in the blink of an eye if it means flooding the market with robotic slaves. One of the best (read: accurate) dystopian movies of the future to this day is Elysium by Neill Blomkamp, but even then, his vision of a robotic factory is forcing human labor to manufacture the military machines that take the piss on a daily basis. It just won’t be like that. It’s time to pay close attention to other industries for clues as to how stuff like Terminator or The Matrix could feasibly come true — our hubris in automating the human out of every job role and process will ultimately lead us to a path where automation can be easily retooled and reprogrammed to manufacture something else. And it won’t be a Roomba. When James Cameron first teased Terminator 2 back in the day, it was through a 90-second teaser that showed an automated robot factory pumping out endless rows of robots, ending with a shot of Arnold as the build process completed. Pure pulp fiction. Or was it? Fast forward to 2024, and we are now entering the era of the hyperfactory, where the main body of cars is autonomously assembled every 76 seconds. Fast forward to 2024, and we are now entering the era of the hyperfactory , where the main body of cars is autonomously assembled every 76 seconds. hyperfactory No, that’s not a mistype because Xaoimi, the smartphone OEM, has entered the electric vehicle arena with the SU7 and with it, a factory setup that is 91% automated with 400 robots armed with HD cameras and can operate in the dark. It reminds me of the time I spent on stage at the Big Data Expo in China with the chairman of Foxconn, and we debated the merits of running a smartphone factory with the lights off — because who needs lights when it’s just the robots running the show? It reminds me of the time I spent on stage at the Big Data Expo in China with the chairman of Foxconn, The factory of the future will have only two employees, a man and a dog. The man will be there to feed the dog. The dog will be there to keep the man from touching the equipment. — Warren Bennis The factory of the future will have only two employees, a man and a dog. The man will be there to feed the dog. The dog will be there to keep the man from touching the equipment. The factory of the future will have only two employees, a man and a dog. The man will be there to feed the dog. The dog will be there to keep the man from touching the equipment. — Warren Bennis — Warren Bennis Watching the keynote reveal is an eye-opening vision of the future of manufacturing but not in the way you think. keynote reveal keynote reveal “We have 269 welding machines to achieve 100% automated production and have the 36 robots to achieve 0.5 millimeters of assembly precision” “…industry leading seven axis coating robots with 70,000 RPM, it can make the paint coverage extremely fine balanced. This produces a natural flat and smooth finish..” “AMR robots that can replace forklifts/internal transport around the factory…” The best part is that the cars can then drive themselves off the factory line with the autopilot software that the company has introduced. In all, it’s a fascinating look at the future of car manufacturing. But there’s something else to consider. We are looking at the precursor of mass-produced, highly automated robot and drone production, something that I can’t help but compare to that dystopian movie. We are looking at the precursor of mass-produced, highly automated robot and drone production, something that I can’t help but compare to that dystopian movie. that dystopian movie. Because, let’s face it, everything is starting to point towards it. We have major AI players all rushing to give their hyperbolic algorithms some form of physical embodiment to contextually understand the world around them in a new dimension. The idea that military, DARPA, or even startup funding won’t find its way to a hyperfactory of this kind is ludicrous because they’ll be eyeing this up as a way to mass-produce killer drones and robotic tech in the blink of an eye if it means flooding the market with robotic slaves. The idea that military, DARPA, or even startup funding won’t find its way to a hyperfactory of this kind is ludicrous because they’ll be eyeing this up as a way to mass-produce killer drones and robotic tech in the blink of an eye if it means flooding the market with robotic slaves. One of the best ( read: accurate ) dystopian movies of the future to this day is Elysium by Neill Blomkamp, but even then, his vision of a robotic factory is forcing human labor to manufacture the military machines that take the piss on a daily basis. It just won’t be like that. read: accurate It’s time to pay close attention to other industries for clues as to how stuff like Terminator or The Matrix could feasibly come true — our hubris in automating the human out of every job role and process will ultimately lead us to a path where automation can be easily retooled and reprogrammed to manufacture something else. And it won’t be a Roomba.

Skynet Is Not the One You Should Be Fearing - Fear Your Favourite Car Manufacturer Instead

Narrative Resilience in Online Conflict Coverage: Insights from Twitter Analysis

Storing Passwords In A Database: A Better Way to Do So

How Optimistic Rollups Enhance Ethereum's Performance and Security

The article demonstrates the advantages of loop unrolling by iterating over multiple items per cycle while adjusting the loop counter accordingly. Learning Objectives The problem with iterating using traditional loops
Advantages with loop unrolling Prerequisites for Developers Basic understanding of C# programming language.


Familiar with for loops Getting Started The problem with iterating using traditional loops Most developers will iterate over array elements and perform a mathematical operation as shown in the following code snippet. for (int i = 0; i < array.Length; i++)
    {
        array[i] = i * 2;
    } Although the above code snippet is quite clean and straightforward, it incurs a cost due to the loop control structure. Advantages with loop unrolling Please find below the refactored version of the previous code snippet int len = array.Length;
    for (int i = 0; i < len; i += 4)
    {
        array[i] = i * 2;
        if (i + 1 < len) array[i + 1] = (i + 1) * 2;
        if (i + 2 < len) array[i + 2] = (i + 2) * 2;
        if (i + 3 < len) array[i + 3] = (i + 3) * 2;
    } With refactored code, now we have reduced the loop frequency to 4 which leads to faster execution. In addition to that, it will: Reduce loop overheads which in turn decreases any conditional checks inside the loops.
Increase performance as it enhances the execution speed Complete Code Create another class named LoopUnrolling and add the following code snippet public static class LoopUnrolling
    {
        public static void BadWay() {
            const int size = 1024;
            int[] numbers = new int[size];
    
            // Traditional loop
            var watch = System.Diagnostics.Stopwatch.StartNew();
            for (int i = 0; i < numbers.Length; i++)
            {
                numbers[i] = i * 2;
            }
            watch.Stop();
            Console.WriteLine($"Traditional loop time: {watch.ElapsedTicks} ticks");
    
        }
        public static void GoodWay()
        {
            const int size = 1024;
            int[] numbers = new int[size];
            int len = numbers.Length;
            var watch = System.Diagnostics.Stopwatch.StartNew();
            for (int i = 0; i < len; i += 4)
            {
                numbers[i] = i * 2;
                if (i + 1 < len) numbers[i + 1] = (i + 1) * 2;
                if (i + 2 < len) numbers[i + 2] = (i + 2) * 2;
                if (i + 3 < len) numbers[i + 3] = (i + 3) * 2;
            }
            watch.Stop();
            Console.WriteLine($"Unrolled loop time: {watch.ElapsedTicks} ticks");
        }
    } Execute from the main method as follows #region Day 26: Loop Unrolling
    static string ExecuteDay26()
    {
        LoopUnrolling.BadWay();
        LoopUnrolling.GoodWay();
    
        return "Executed Day 26 successfully..!!";
    }
    
    #endregion Console Output Traditional loop time: 20 ticks
    Unrolled loop time: 12 ticks As the output demonstrates loop unrolling took much less time than the traditional loop approach. Complete Code here: GitHub — ssukhpinder/30DayChallenge.Net Thank you for being a part of the C# community! Before you leave: Follow us: Youtube | X | LinkedIn | Dev.to
Visit our other platforms: GitHub
More content at C# Programming The article demonstrates the advantages of loop unrolling by iterating over multiple items per cycle while adjusting the loop counter accordingly. Learning Objectives The problem with iterating using traditional loops Advantages with loop unrolling The problem with iterating using traditional loops Advantages with loop unrolling Prerequisites for Developers Basic understanding of C# programming language. Familiar with for loops Basic understanding of C# programming language. Basic understanding of C# programming language. Familiar with for loops Familiar with for loops Getting Started The problem with iterating using traditional loops Most developers will iterate over array elements and perform a mathematical operation as shown in the following code snippet. for (int i = 0; i < array.Length; i++)
    {
        array[i] = i * 2;
    } for (int i = 0; i < array.Length; i++)
    {
        array[i] = i * 2;
    } Although the above code snippet is quite clean and straightforward, it incurs a cost due to the loop control structure. Advantages with loop unrolling Please find below the refactored version of the previous code snippet int len = array.Length;
    for (int i = 0; i < len; i += 4)
    {
        array[i] = i * 2;
        if (i + 1 < len) array[i + 1] = (i + 1) * 2;
        if (i + 2 < len) array[i + 2] = (i + 2) * 2;
        if (i + 3 < len) array[i + 3] = (i + 3) * 2;
    } int len = array.Length;
    for (int i = 0; i < len; i += 4)
    {
        array[i] = i * 2;
        if (i + 1 < len) array[i + 1] = (i + 1) * 2;
        if (i + 2 < len) array[i + 2] = (i + 2) * 2;
        if (i + 3 < len) array[i + 3] = (i + 3) * 2;
    } With refactored code, now we have reduced the loop frequency to 4 which leads to faster execution. In addition to that, it will: Reduce loop overheads which in turn decreases any conditional checks inside the loops. Increase performance as it enhances the execution speed Reduce loop overheads which in turn decreases any conditional checks inside the loops. Increase performance as it enhances the execution speed Complete Code Create another class named LoopUnrolling and add the following code snippet public static class LoopUnrolling
    {
        public static void BadWay() {
            const int size = 1024;
            int[] numbers = new int[size];
    
            // Traditional loop
            var watch = System.Diagnostics.Stopwatch.StartNew();
            for (int i = 0; i < numbers.Length; i++)
            {
                numbers[i] = i * 2;
            }
            watch.Stop();
            Console.WriteLine($"Traditional loop time: {watch.ElapsedTicks} ticks");
    
        }
        public static void GoodWay()
        {
            const int size = 1024;
            int[] numbers = new int[size];
            int len = numbers.Length;
            var watch = System.Diagnostics.Stopwatch.StartNew();
            for (int i = 0; i < len; i += 4)
            {
                numbers[i] = i * 2;
                if (i + 1 < len) numbers[i + 1] = (i + 1) * 2;
                if (i + 2 < len) numbers[i + 2] = (i + 2) * 2;
                if (i + 3 < len) numbers[i + 3] = (i + 3) * 2;
            }
            watch.Stop();
            Console.WriteLine($"Unrolled loop time: {watch.ElapsedTicks} ticks");
        }
    } public static class LoopUnrolling
    {
        public static void BadWay() {
            const int size = 1024;
            int[] numbers = new int[size];
    
            // Traditional loop
            var watch = System.Diagnostics.Stopwatch.StartNew();
            for (int i = 0; i < numbers.Length; i++)
            {
                numbers[i] = i * 2;
            }
            watch.Stop();
            Console.WriteLine($"Traditional loop time: {watch.ElapsedTicks} ticks");
    
        }
        public static void GoodWay()
        {
            const int size = 1024;
            int[] numbers = new int[size];
            int len = numbers.Length;
            var watch = System.Diagnostics.Stopwatch.StartNew();
            for (int i = 0; i < len; i += 4)
            {
                numbers[i] = i * 2;
                if (i + 1 < len) numbers[i + 1] = (i + 1) * 2;
                if (i + 2 < len) numbers[i + 2] = (i + 2) * 2;
                if (i + 3 < len) numbers[i + 3] = (i + 3) * 2;
            }
            watch.Stop();
            Console.WriteLine($"Unrolled loop time: {watch.ElapsedTicks} ticks");
        }
    } Execute from the main method as follows #region Day 26: Loop Unrolling
    static string ExecuteDay26()
    {
        LoopUnrolling.BadWay();
        LoopUnrolling.GoodWay();
    
        return "Executed Day 26 successfully..!!";
    }
    
    #endregion #region Day 26: Loop Unrolling
    static string ExecuteDay26()
    {
        LoopUnrolling.BadWay();
        LoopUnrolling.GoodWay();
    
        return "Executed Day 26 successfully..!!";
    }
    
    #endregion Console Output Traditional loop time: 20 ticks
    Unrolled loop time: 12 ticks Traditional loop time: 20 ticks
    Unrolled loop time: 12 ticks As the output demonstrates loop unrolling took much less time than the traditional loop approach. Complete Code here: GitHub — ssukhpinder/30DayChallenge.Net Complete Code here: GitHub — ssukhpinder/30DayChallenge.Net GitHub — ssukhpinder/30DayChallenge.Net Thank you for being a part of the C# community! Before you leave: Follow us: Youtube | X | LinkedIn | Dev.to Visit our other platforms: GitHub More content at C# Programming Youtube X LinkedIn Dev.to GitHub C# Programming

30-Day .NET Challenge - Day 26: Loop Unrolling

deployments come in all shapes and sizes. We support bare metal installs on any version of , containerized installs on any version of (including Red Hat OpenShift), and installs just about anywhere you can deploy a small lightweight single binary. But with flexibility comes the inevitability that edge case issues will require debugging. MinIO Linux Kubernetes In this blog post, we’ll show you how to debug a MinIO install running in Kubernetes and also some of the common issues you might encounter when doing bare metal installation and how to rectify them. Kubernetes Debugger Pod There are a few ways to access the MinIO API running inside a Kubernetes cluster. We can use or set up a listening on to be able to access the API. Both of these methods offer a way to access the service from outside the network, but they do come with one major downside: You can only access the Service that the NodePort or Port Forwarding references on an available port (not the usual configuration for the application). For example, you have to access the MinIO API, usually found on port , via a randomly assigned port. kubectl port-forwarding Service NodePort 9000 3xxxx What if I told you there was a better way – and it's not novel? When debugging applications you want to have full access to the native run-time environment so you can use various tools to troubleshoot and debug the cluster. One way to do that is launching a “busybox” style pod and installing all the required tools needed to debug the application. First launch a into the same namespace as your MinIO install. In order to do this create a yaml file called with the following yaml. Pod debugger-pod.yaml apiVersion: v1

kind: Pod

metadata:

  name: mc

  labels:

    app: mc

spec:

  containers:

  - image: minio/mc:latest

    command:

      - "sleep"

      - "604800"

    imagePullPolicy: IfNotPresent

    name: mc

  restartPolicy: Always The above Pod configuration is pulling the image for MinIO utility. In order to ensure the pod doesn’t just launch and then exit, we’ve added a command. mc sleep Once the yaml is saved apply the configuration to the Kubernetes namespace where the MinIO cluster is running kubectl apply -f debugger-pod.yaml Once the pod is up and running access it via shell $ kubectl exec -i -t -n default mc -c mc -- sh -c "(bash || ash || sh)"

[root@mc /]# Then with you can access the MinIO cluster mc [root@mc /]# mc alias set myminio    --insecure

Added `myminio` successfully. Now that we have a debugger pod up and running, you can perform action on the cluster directly within the same network. For example, if the replication was broken due to an site going offline or a hardware failure, you can resync any pending objects to be replicated using the following command [root@mc /]# mc admin replicate resync start minio1 minio2


[root@mc /]# mc admin replicate resync status minio1 minio2


✔ ✔ ✔

ResyncID: 		 2248d1d1-633f-4d61-b938-d8ea0b9b2d31

Status:   		 Completed

Objects:  		 2225

Versions: 		 2225

FailedObjects:     0

Throughput:   	 5.3 MiB/s

IOPs:     		 124.23 objs/s

Transferred:  	 94 MiB

Elapsed:  		 17.909833202s

CurrObjName:  	 testbucket/web-app/tsconfig.json Another reason you would run the debugger pod is if there are some file system permissions or invalid groups configurations in your pod, you can update them using the debugger pod [root@mc /]# chgrp -R 1000780050 .minio.sys/ The above debugging method can also be used in bare metal environments. For instance you can launch a busybox or bastion node with installed and follow the same instructions as above. mc Debugging Bare Metal Bare metal Linux installs are the most straightforward. In fact it just takes a few commands to get MinIO installed and running with SystemD. For details, please see . Configuring MinIO with SystemD Once in a great while, bare metal installs go awry. Here are some of the (not-so-common) pitfalls that we are asked about  in or . These pitfalls are not hardware or operating specific but can be useful to know in any kind of environment. SUBNET Slack File Permission One of the most common pitfalls is the file permissions of the MinIO binary and the configuration file. If this occurs, when you start MinIO using SystemD you will see and here is the full stack trace Assertion failed for MinIO. # systemctl status minio.service

● minio.service - MinIO

     Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: enabled)

     Active: inactive (dead)

     Assert: start assertion failed at Tue 2023-12-26 18:21:38 PST; 8s ago

             AssertFileIsExecutable=/usr/local/bin/minio was not met

       Docs: https://docs.min.io


Dec 26 18:13:37 minio1 systemd[1]: minio.service: Starting requested but asserts failed.

Dec 26 18:17:50 minio1 systemd[1]: Assertion failed for MinIO.

Dec 26 18:21:38 minio1 systemd[1]: minio.service: Starting requested but asserts failed.

Dec 26 18:21:38 minio1 systemd[1]: Assertion failed for MinIO. This could be caused by a number of reasons, let's go down the list and check for each of them. : The binary, in this example located at needs to have permission for user and group, respectively. MinIO Binary /usr/local/bin/minio root:root # ll /usr/local/bin/minio

total 93804

-rwxr-xr-x 1 root    root    96018432 Nov 15 16:35  minio* : The MinIO service needs to run under a unique Linux user and group for security purposes, never run as a user. By default we use for the user and group names. In the SystemD service config file you should see something like this MinIO Service User and Group root minio-user User=minio-user

Group=minio-user : The directory where MinIO data will be stored needs to be owned by or whichever user you decide to run the MinIO service as above. MinIO Data Dir minio-user:minio-user # ls -l /mnt

total 4

drwxrwxr-x 2 minio-user minio-user 4096 Dec 27 09:58 data : Both the config files should have permissions for user and group like so SystemD and MinIO config root:root # ls -l /etc/default/minio

-rw-r--r-- 1 root root 1330 Dec 27 09:52 /etc/default/minio


# ls -l /etc/systemd/system/minio.service

-rw-r--r-- 1 root root 941 Dec 26 17:13 /etc/systemd/system/minio.service : The entire install process should be run as . You can also try if your user has permissions but the recommendation is to run as root as the install needs to place files in a bunch of places that only the user can access. Your bash prompt should have a and not a like so Run as Root root sudo root # $ vs # $ If none of the above work, the best approach is to remove the app, directories and configs and start a fresh install as a root user. Port conflict Another common issue related to deleted files which still hold on to the process, which causes port conflicts. Even when a service is not running, you may be unable to start a new service on the existing port or the service that is running will misbehave (such as not allowing you to login). # lsof -n | grep (deleted)

COMMAND PID USER FD TYPE DEVICE   SIZE 	NODE NAME

nginx   	13423 root 5u  	REG 253,3   42949672960 	17 (deleted)

minio 	13423 minio 6u    REG 253,3 	0         	      18 (deleted)

minio       13423 minio 7u    REG 253,3 	0         	      19 (deleted) You might see errors such as those below on a MinIO install Login Failed net::ERR_FAILED 500 Internal Server Error 401 Unauthorized The screenshot above shows an internal server error and an unauthorized error. While looking at the surface it looks unclear what has caused this error, we can debug with a little linux knowledge what to look for that could cause this, let's take a gander. There are several ways to debug this issue, first lets check to see if multiple MinIO processes are running on the same node # ps aux | grep -i minio

minio-u+    5048  0.3  1.7 1594008 144384 ?      Ssl  11:03   0:01 /usr/local/bin/minio server --console-address :9001 /mnt/data/disk1/minio


minio-u+    9276  0.3  1.7 1594208 144301 ?      Ssl  11:25   0:01 /usr/local/bin/minio server --console-address :9001 /mnt/data/disk1/minio As we can see above there are 2 MinIO processes running. Start by killing the process that is older or has been running the longest, in this case it seems to be the process ID . 5048 kill -9 5048 Sometimes even after killing the process the service might still not start or might still get hung up because it has reserved a process number but not let it go. This can be caused by files that have been deleted but are still being tracked by the operating system. You can find the deleted files via LSOF lsof -n | grep '(deleted)' Last but not least, if there are no deleted files left over or hung processes and if everything looks absolutely clean, the last resort is to quickly restart the node. This is a no-nonsense method that  shuts down and clears any pending files and processes so you start a fresh install. SUBNET to the rescue Although rare, installation edge cases will always exist. MinIO customers know that they have nothing to worry about because they can quickly message our engineers – who have written the code – via the portal. We've seen almost everything under the sun, so while the issue might look cryptic or mind boggling at first glance, we'll put our years of expertise debugging installations in many varied environments to work and help you in a jiffy. SUBNET If you have any questions on troubleshooting and debugging installs be sure to reach out to us on ! MinIO Slack

Pro Tips for Debugging MinIO: Overcoming Common Issues in Kubernetes and Bare Metal Environments

MinIO ডিবাগ করার জন্য প্রো টিপস: কুবারনেটস এবং বেয়ার মেটাল পরিবেশে সাধারণ সমস্যাগুলি অতিক্রম করা

Profi-Tipps zum Debuggen von MinIO: Überwindung häufiger Probleme in Kubernetes- und Bare-Metal-Umgebungen

Consejos profesionales para depurar MinIO: superar problemas comunes en Kubernetes y entornos bare metal

Conseils de pro pour déboguer MinIO : surmonter les problèmes courants dans les environnements Kubernetes et Bare Metal

मिनियो डिबगिंग के लिए प्रो टिप्स: कुबेरनेट्स और बेयर मेटल वातावरण में आम समस्याओं पर काबू पाना

MinIO デバッグのプロのヒント: Kubernetes とベアメタル環境での一般的な問題の克服

MinIO 디버깅을 위한 전문가 팁: Kubernetes 및 베어메탈 환경의 일반적인 문제 극복

Dicas profissionais para depuração do MinIO: superando problemas comuns em ambientes Kubernetes e Bare Metal

Советы для профессионалов по отладке MinIO: решение распространенных проблем в Kubernetes и средах с голым железом

MinIO'da Hata Ayıklamaya Yönelik Profesyonel İpuçları: Kubernetes ve Bare Metal Ortamlarında Yaygın Sorunların Aşılması

Mẹo chuyên nghiệp để gỡ lỗi MinIO: Khắc phục các sự cố thường gặp trong môi trường Kubernetes và Bare Metal

调试 MinIO 的专业技巧：解决 Kubernetes 和裸机环境中的常见问题

The Crash That Brought the Shadow to the Light. On the morning of July 5, 2017, a gray Toyota Camry turned into the cul-de-sac of a neighborhood in Bangkok in a subdivision on the western edge of the city. A driver makes a miscalculation on a 180-degree turn and accidentally causes a collision with a front gate on a model home and real estate office. The driver exits the vehicle obviously upset about the event and needs to speak with the owner of the model house about the accident. After a significant time elapsed, Alexander Caze surfaced from his home to discuss the collision at the front gate of his home. As Alexander Caze made his way to have the conversation with the person in the vehicle, the FBI stormed the property arresting Alexander Caze aka “Alpha02” aka “Admin online” the owner of the Dark Web marketplace AlphaBay. Riches to Wretchedness. Alexander Cazes was just 25 and the CEO of EBX Technologies, a company that offered website creation services for businesses. Or at least that is what Alexandre Cazes said when someone asked about his occupation. According to Cazes’s father, Alexandre was a computer genius who started his own web design business at 17, had a high IQ, AND specialized in everything from web development to encryption services. While still a teenager, Cazes was believed to have started his first company, EBX Technologies, selling computers and helping small businesses set up their websites. His IT presence was clear to see; authorities traced Alexandre as far back as 2008, posting advice on a tech forum on how to remove a virus from a digital photo. Cazes grew up near Montreal and his family said he was a genius. In the shadows, Alexandre was the alleged mastermind of AlphaBay. According to publicly available information on AlphaBay prior to its takedown, one AlphaBay staff member claimed that it serviced over 200,000 users and 40,000 vendors. Around the time of the takedown, there were over 250,000 listings for illegal drugs and toxic chemicals on AlphaBay, and over 100,000 listings for stolen and fraudulent identification documents and access devices, counterfeit goods, malware and other computer hacking tools, firearms, and fraudulent services. Comparatively, the Silk Road dark web marketplace, which was seized by law enforcement in November 2013, had reportedly approximately 14,000 listings for illicit goods and services at the time of seizure and was the largest dark web marketplace at the time. . All Wrongdoing can be Traced to an Excessive Attachment to Material Wealth Alexander Cazes biggest vulnerability turned out to be his penchant for bragging about his newly acquired wealth and status. Alexandre Cazes claimed his worth was about $23 million US, including $12.5 million in properties and vehicles, and the rest in cash and cryptocurrencies. Cazes's assets seized by the U.S. government included $6.6 million Cdn in Bitcoin, $2.4 million in Ethereum, $622,000 in Monero, and $980,512 in Zcash. He had at least 11 bank accounts and cryptocurrency exchange accounts registered in his name or his wife's name in Thailand, Liechtenstein, Switzerland, and St. Vincent and the Grenadines. Beyond cash, Cazes had a taste for luxury vehicles, including a 2013 Lamborghini Aventador, a Porsche Panamera, a Mini Cooper, and a BMW motorcycle. He had three properties in Bangkok, and a vacation property in Phuket, Thailand. Subtle Clues That Were Not So Subtle. Alexandre Cazes lived a seemingly quiet life for more than a year with his Thai girlfriend in a middle-class neighborhood on the outskirts of Bangkok, but the flashy cars he drove stood out. All in an area where people drive pickup trucks and children tool around on plastic tricycles, and there was the nearly $1 million, metallic gray Lamborghini, a Porsche, and then the Mini Cooper for his girlfriend. Nothing except expensive cars, which were completely out of place in the neighborhood where homes cost less than $120,000. The neighbors thought Alexandre Cazes worked in the hotel business. neighbor, who asked not to be named because the case involves crime, said Cazes rarely left the house before noon. She said she got her first good look at him one day when he was outside, trying to photograph a monitor lizard that had crawled out of a deserted field nearby. “We smiled at each other, that’s it,” she said. One Whispers in the Dark That Awoke a Giant. Alexandre Cazes believed that crypto’s anonymity would protect him but just as crypto had opened new markets for the criminal world, it opened up opportunities for law enforcement. According to U.S. government documents, Cazes had an estimated net worth of around $23 million. This wealth was diversified across real estate in Thailand, Cyprus, and Antigua, luxury cars, and liquid assets such as cash and cryptocurrencies. Specifically, $12.5 million was invested in properties and vehicles. At its height, daily sales ranged between $600,000 and $800,000 across 300,000 listings for illicit goods, offered by over 40,000 vendors and viewed by some 200,000 users. Alexadre’s activity with AlphBay led to a police operation shutdown called “Bayonet”, which involved authorities in six countries including the FBI, the Dutch National Police, and the Royal Thai Police. AlphaBay’s Simple Mistakes From a Brilliant Mind. Alexandre Cazes made mistakes and one was with his personal email. Once new users joined the forums and entered their private email accounts, they were greeted with an email directly from AlphaBay welcoming them to the forums. The email address “Pimp_Alex_91@hotmail.com” was included in the header information of the AlphaBay welcome email. Searches for that email led investigators to forum posts by accounts with the same online handle Cazes used on AlphaBay, Alpha02. They eventually tracked down Cazes’ full name and LinkedIn account, which listed him as affiliated with EBX Technology, a company that investigators suspected was a front for AlphaBay. Alexandre Cazes had registered a PayPal account tied to bank accounts held in his name as well as the EBX Technologies business name using the "Pimp_Alex_91@hotmail.com" email account. Also, in 2008 a post to a French-language online tech forum "www.commentcamarche.com" by user "Alpha02" included the name "Alexandre Cazes" and the Hotmail address. The administrator of AlphaBay, notably, had used the username "Alpha02" before changing it to "Admin." To Encrypt or Not to Encrypt…That Is the Question. When the police, aided by the FBI and DEA, raided Alexandre Cazes's home, they said he was using a PC that was logged into AlphaBay as "Admin." The laptop was also found opened and unencrypted - revealing that Alexandre Cazes had been communicating with an AlphaBay data center, revealing the day-to-day operations as well as with AlphaBay users, and numerous passwords to AlphaBay servers and other infrastructure. A Tragic End And a Snake in the Grass. Alexandre Cazes was being held in a cell at Bangkok's Narcotics Suppression Bureau and, just hours before he was due to meet prosecutors for talks over his extradition to the US, was found dead of an apparent suicide. Alexandre Cazes did not run AlphaBay singlehandedly. He worked closely with a “security administrator” and second in command known as , or simply “DS” for short. DeSnake Sometime after the takedown of Alpah Bay and the death of Alexandre Cazes, posted on a popular dark web forum to announce the relaunch of AlphaBay and a dedication to Alexandre Cazes aka “alpha02.” DeSnake

Riches to Wretchedness: The Fall of Alexandre Cazes and the Resurrection of AlphaBay

USA v. Samuel Bankman-Fried Court Filing, retrieved on March 15, 2024 is part of . You can jump to any part in this filing . This part is 10 of 33. HackerNoon’s Legal PDF Series here A. Use of Signal to Auto-Delete Messages At trial, the evidence established that the defendant instructed employees to use Signal, an end-to-end encrypted messaging application, with an auto-delete feature enabled so as to prevent regulators from reviewing communications. Specifically, Adam Yedidia testified that the defendant instructed employees to use Signal with a feature enabled that would automatically delete each message after a certain amount of time had elapsed since the message was sent. (Tr. 175). Bankman-Fried stated that “it was all downside for messages to be kept around” because “if regulators found something they didn’t like in those messages, that could be bad for the company.” (Tr. 176). Similarly, Ellison testified that the defendant “directed [them] to use the disappearing messaging setting on [their] Signal, meaning that all messages [they] sent in these conversations would disappear after seven days.” (Tr. 825). Bankman-Fried said that FTX and Alameda employees “should be very careful about what [they] put in writing” because “putting things in writing was one of the primary ways that financial firms got in legal trouble.” (Id.). After those directives, FTX and Alameda employees increasingly used Signal messages for “sensitive” communications. (Tr. 825-26). Government Exhibit 1083 is a chart of all of the Signal chats on Wang’s device that were set to auto-delete. The chart shows that many of the chats’ auto-delete function was set by the defendant. Continue Reading . Here About HackerNoon Legal PDF Series: We bring you the most important technical and insightful public domain court case filings. This court case retrieved on March 15, 2024, from is part of the public domain. The court-created documents are works of the federal government, and under copyright law, are automatically placed in the public domain and may be shared without legal restriction. storage.courtlistener

SBF Directed Employees to Use the Auto-Delete Signal App Future to Avoid Regulatory Scrutiny

This paper is available on arxiv under CC 4.0 license. Authors: (1) Xiaoyu Ai, School of Electrical Engineering & Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia; (2) Robert Malaney, School of Electrical Engineering & Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia. Table of Links Abstract & Introduction System Overview Overcoming the limitation of key reconciliation Final Key Rate Experimental Results Discussion Conclusion & References Appendix V. EXPERIMENTAL RESULTS A. Decoding Error and Time B. The Optimal NR For this experiment, we pre-built a database to store LDPC codes with their code rates ranging from 0.01 to 0.8 and their block lengths ranging from 106 to 109 . For code rates less than 0.1, Multi-Edge-Type LDPC codes (degree distributions outlined in [22]) were used. These achieve a lower pDecode compared to irregular LDPC codes with the same code rate. For code rates greater than or equal to 0.1, we adopted the irregular LDPC codes (degree distributions outlined in [57]). At such code rates, these latter codes have the same pDecode performance as the Multi-Edge-Type counterparts, but allow for faster code construction. C. Final-key Effect for a Given No [15] Recall, we are particularly interested in the satellite-to-Earth channel. As in other works, we assume losses for this channel are dominated by diffraction effects, and therefore the transmissivity can be held constant. We further assume post-selections, using a bright classical beam sent along with the quantum signals (but different polarisation), remove any significant transmissivity deviations. As discussed elsewhere [17], some receiver/transmitter apertures, coupled to detailed phase-screen simulations of satellite downlink channels, render the constant-transmissivity assumption reasonable [56]. If the transmissivity is highly variable the optimal block length, NR, can be calculated by an expectation over the transmissivity density function. [16] We adopted the following method to determine ch. For m LDPC codes with NR = 106 and T = 0.9, we obtained the total number of arithmetic operations for those codes. Next, we measured the elapsed time to reconcile a block of 106 quadrature values. We then obtained ch by dividing the number of arithmetic operations to this measured elapsed time. [17] These trapping sets are the primary reason that additional decoding iterations are consumed for only a marginal decrease of the decoding error, i.e. the error floor effect [58].

Optimised Multithreaded CV-QKD Reconciliation for Global Quantum Networks: Experimental Results

Now that Super Tuesday has taken place in America, it seems clear that we'll be treated to a rematch of the 2020 presidential duel in America. Donald Trump has been dreaming of this for the past 4 years, and now he'll be able to take on Joe Biden again for revenge. Donald Trump's goal seems obvious: to return to the White House so he can wage his vendetta against all those he deems traitors. , it would seem that Joe Biden has his work cut out for him. With the slogan “It's the economy stupid” seemingly no longer an election staple I'm not for Joe Biden or Donald Trump because I believe that both will pursue disastrous policies that will continue to worsen the situation. It can't be otherwise, because beyond the politicians, the current system is flawed. All these politicians can't change the situation. The system won't be fixed, and the revolution will come from the people. This revolution is embodied by Bitcoin. That said, something is astonishing. While , Donald Trump is now enthusiastic about Bitcoin. Bitcoin has just beaten a new ATH above $72K and entered uncharted territories . He was interviewed a few hours ago on Squawk Box Here's where Donald Trump talks about Bitcoin: “Sometimes we'll let people pay through Bitcoin...it's an additional form of currency.” The most blinkered optimists went straight to ultra-bullish thinking that Donald Trump was now pro-Bitcoin. These people almost forgot what he said a little later about possible competitors to the US dollar: “I would not allow countries to go off the dollar, because when we lose that standard, that will be like losing a revolutionary war.” Donald Trump has become a true politician since he can say two contradictory things in the same televised speech. You can't be pro-Bitcoin and then say that you'll do anything to protect the hegemony of the US dollar over the world. These are two contradictory things. As Hillary Clinton has said in the past, “Bitcoin has the potential of undermining the role of the dollar as a reserve currency.” Bitcoin's potential goes far beyond replacing gold as a reference store of value for the general public. Quite simply, . Bitcoin has the potential to become the world's next reserve currency From now on, you need to learn to read between the lines and understand that Donald Trump was not Bullish on Bitcoin. Far from it! He wants to capitalize on Bitcoin's current popularity to try and make people think he's different from Joe Biden and get more votes for the 2024 presidential election. Remember what Donald Trump said about Bitcoin in 2019: Donald Trump had even tasked some of his staff with finding a way to ban Bitcoin ... LOL ... So, have no illusions. Donald Trump won't do any better than Joe Biden on Bitcoin. In fact, in his 2024 program, Joe Biden is proposing a 30% tax on the electricity consumed by Bitcoin Miners. Something disgraceful, designed to pave the way for the arrival of a Digital Dollar in the future. . This Digital Dollar will be the worst thing for your privacy All these politicians will do nothing but protect the current system, in which the American dollar is king. For those who doubt this, look to the past, and you'll see why Americans have gone to war so often in past decades. The latest example relates to Muammar Gaddafi in 2011. Shortly before Gaddafi's fall, Libya had a plan to create a pan-African currency based on the Libyan dinar backed by its 143 tons of gold reserves, as revealed in an April 2011 email from Sidney Blumenthal, found in the Clinton documents leaked to the Net by Wikileaks. Gaddafi imagined breaking French hegemony over the former African colonies by offering an alternative to the CFA franc and using the weight of his nearly 2 million barrels of daily oil production. At one time, Gaddafi was also talking about denominating his oil exports in euros. In short, Gaddafi's ambition was to undermine the exorbitant privileges that Westerners granted themselves in Africa. It will then come as no surprise that a coalition led by America, France, and the UK took it upon themselves to put an end to Gaddafi's regime in Libya. Of course, I wouldn't say that Gaddafi was a democrat, far from it. He was a dictator, but what must be shocking here is the radical reversal of position on the part of Westerners, who had no hesitation in inviting Gaddafi to their homes in style, as President Sarkozy did in December 2007 at the Elysée Palace. It was later revealed that Gaddafi's Libyan regime had paid 50 million euros to finance Nicolas Sarkozy's 2007 French presidential campaign. The official amount of Sarkozy's 2007 presidential campaign was 20 million euros ... I'll stop there, but you can do your research on these subjects to understand that these politicians will do anything to protect the current system. None is better than the other. The real revolution comes from Bitcoin, and it's by supporting the Bitcoin revolution that the people will be able to regain power. But to do this, we need to put speculation aside and focus on , to take full advantage of . the why of Bitcoin the liberating power of the Bitcoin revolution Quite a program, you may say! But that's what's at stake for the Bitcoin revolution in the future. It's up to you to tell me what you think of Donald Trump's statements on Bitcoin. Also published here

Is Donald Trump Really Bullish on Bitcoin? Read Between the Lines!

When working with local files, you might need to open, create, or update Zip files. In this article, we will learn how to work with Zip files in C#. We will learn how to perform basic operations such as opening, extracting, and creating a Zip file. The main class we will use is named and comes from the namespace. It’s been present in since .NET Framework 4.5, so we can say it’s pretty stable 😉 Nevertheless, . Let’s learn! ZipFile System.IO.Compression C# there are some tricky points that you need to know before using this class Using C# to list all items in a Zip file Once you have a Zip file, you can access the internal items without extracting the whole Zip. You can use the method. ZipFile.Open using ZipArchive archive = ZipFile.Open(zipFilePath, ZipArchiveMode.Read);
System.Collections.ObjectModel.ReadOnlyCollection<ZipArchiveEntry> entries = archive.Entries; Notice that I specified the . This is an Enum whose values are , , and . ZipArchiveMode Read Create Update Using the property of the , you can access the whole list of files stored within the Zip folder, each represented by a instance. Entries ZipArchive ZipArchiveEntry The object contains several fields, like the file’s name and the full path from the root archive. ZipArchiveEntry There are a few about the entries listed in the . key points to remember ZipArchiveEntry It is a : it means that even if you find a way to add or update the items in memory, the changes are not applied to the actual files; ReadOnlyCollection<ZipArchiveEntry> , not only those at the root level. As you can see from the image above, it lists both the files at the root level, like , and those in inner folders, such as ; It lists all files and folders File.txt TestZip/InnerFolder/presentation.pptx Each file is characterized by two similar but different properties: is the actual file name (like ), while contains the path from the root of the archive (e.g. ); Name presentation.pptx FullName TestZip/InnerFolder/presentation.pptx It lists : in the image above, you can see . You can recognize them because their property is empty, and their is 0; folders as if they were files TestZip/InnerFolder Name Length Lastly, remember that returns an , so you should place the operations within a statement. ZipFile.Open IDisposable using ❓❓A question for you! Why do we see an item for the folder, but there is no reference to the folder? Drop a comment below 📩 TestZip/InnerFolder TestZip Using C# to extract the Zip to a local path Extracting a Zip folder is easy but not obvious. We have only one way to do that: by calling the method. ZipFile.ExtractToDirectory It accepts as mandatory parameters the path of the Zip file to be extracted and the path to the destination: var zipPath = @"C:\Users\d.bellone\Desktop\TestZip.zip";
var destinationPath = @"C:\Users\d.bellone\Desktop\MyDestination";
ZipFile.ExtractToDirectory(zipPath, destinationPath); Once you run it, you will see the content of the Zip copied and extracted to the folder. MyDestination Note that if it does not exist. this method creates the destination folder This method accepts two more parameters: , by which you can specify the encoding. . entryNameEncoding The default value is UTF-8 allows you to specify whether it must overwrite existing files. The default value is . If set to false and the destination files already exist, this method throws a saying that the file already exists. overwriteFiles false System.IO.IOException Using C# to create a Zip from a folder The key method here is , which allows you to create Zip files in a flexible way. ZipFile.CreateFromDirectory The first mandatory value is, of course, the . source directory path The second mandatory parameter is the destination of the resulting Zip file. It can be the to the file: local path string sourceFolderPath = @"\Desktop\myFolder";
string destinationZipPath = @"\Desktop\destinationFile.zip";

ZipFile.CreateFromDirectory(sourceFolderPath, destinationZipPath); Or it can be a that you can use later for other operations: Stream using (MemoryStream memStream = new MemoryStream())
{
    string sourceFolderPath = @"\Desktop\myFolder";
    ZipFile.CreateFromDirectory(sourceFolderPath, memStream);

    var lenght = memStream.Length;// here the Stream is populated
} You can finally add some optional parameters: , whose values are , , , . compressionLevel Optimal Fastest NoCompression SmallestSize : a flag that defines if you have to copy only the first-level files or also the root folder. includeBaseDirectory A quick comparison of the four Compression Levels As we just saw, we have four compression levels: , , , and . Optimal Fastest NoCompression SmallestSize What happens if I use the different values to zip all the photos and videos of my latest trip? The source folder’s size is 16.2 GB. Let me zip it with the four compression levels: private long CreateAndTrack(string sourcePath, string destinationPath, CompressionLevel compression)
 {
     Stopwatch stopwatch = Stopwatch.StartNew();

     ZipFile.CreateFromDirectory(
         sourceDirectoryName: sourcePath,
         destinationArchiveFileName: destinationPath,
         compressionLevel: compression,
         includeBaseDirectory: true
         );
     stopwatch.Stop();

     return stopwatch.ElapsedMilliseconds;
 }

// in Main...

var smallestTime = CreateAndTrack(sourceFolderPath,
    Path.Combine(rootFolder, "Smallest.zip"),
    CompressionLevel.SmallestSize);

var noCompressionTime = CreateAndTrack(sourceFolderPath,
    Path.Combine(rootFolder, "NoCompression.zip"),
    CompressionLevel.NoCompression);

var fastestTime = CreateAndTrack(sourceFolderPath,
    Path.Combine(rootFolder, "Fastest.zip"),
    CompressionLevel.Fastest);

var optimalTime = CreateAndTrack(sourceFolderPath,
    Path.Combine(rootFolder, "Optimal.zip"),
    CompressionLevel.Optimal); By executing this operation, we have this table: We can see a bunch of weird things: Fastest compression generates a smaller file than Smallest compression. Fastest compression is way slower than Smallest compression. Optimal lies in the middle. This is to say: don’t trust the names; , even with a test as simple as this. remember to benchmark the parts where you need performance Wrapping up This was a quick article about one specific class in the .NET ecosystem. As we saw, even though the class is simple and it’s all about three methods, there are some things you should keep in mind before using this class in your code. I hope you enjoyed this article! Let’s keep in touch on or ! 🤜🤛 Twitter LinkedIn Happy coding! 🐧 Also published . here

Zip Files in C#: How to Extract, Create, or Update Your Archives

This is the first part in a multi-part series on building Agents with OpenAI's Assistant API using the Python SDK. What Are Agents? The way I like to look at it, an agent is really just a piece of software leveraging an LLM (Large Language Model) and trying to mimic human behavior. That means it can not only converse and understand language, but it can also perform actions that have an impact on the real world. These actions are typically called tools. In this blog post, we will explore how to build an agent using OpenAI's Assistant API using their Python SDK. Part 1 will be just the skeleton of the assistant. That is, just the conversational part. I chose to build a CLI app on purpose to be framework agnostic. We will purposefully call our implementation an Agent and refer to the OpenAI SDK implementation as an Assistant to easily distinguish between the two. I use the terms and interchangeably when it comes to functions that the Agent is able to call. Part 2 will cover function callin in more detail. tools functions Prerequisites To follow along with this tutorial, you will need the following: Python3 installed on your machine An OpenAI API key Basic knowledge of Python programming OpenAI Assistant Concepts : An Assistant in the Assistants API is an entity configured to respond to user messages. It uses instructions, a chosen model, and tools to interact with functions and provide answers. Assistant : A Thread represents a conversation or dialogue in the Assistants API. It is created for each user interaction and can contain multiple Messages, serving as a container for the ongoing conversation. Thread : A Message is a unit of communication in a Thread. It contains text (and potentially files in the future) and is used to convey user queries or assistant responses within a Thread. Message : A Run is an instance of the Assistant processing a Thread. It involves reading the Thread, deciding whether to call tools, and generating responses based on the model's interpretation of the Thread's Messages. Run Setting Up the Development Environment The first step is to create a virtual environment using and activate it. This will ensure that our dependencies are isolated from the system Python installation: venv python3 -m venv venv
source venv/bin/activate Let's install our only dependency: the package: openai pip install openai Create a file. Let's populate with some basic runtime logic for our CLI app: main.py while True:
    user_input = input("User: ")
    if user_input.lower() == 'exit':
        print("Exiting the assistant...")
        break
    print(f"Assistant: You said {user_input}") Try it out by running : python3 main.py python3 main.py
User: hi
Assistant: You said hi As you can see, the CLI accepts a User message as input, and our genius Assistant doesn't have a brain 🧠 yet so he just repeats the message right back. Not so smart yet. The Agent Now, the fun 😁 (or headaches 🤕) begins. I'll provide all the imports needed for the final class right now, so you don't rack your brain on where things are coming from since I kept imports out of code samples for brevity. Let's start by building an class in a new file : Agent agent.py import time
import openai
from openai.types.beta.threads.run import Run

class Agent:
    def __init__(self, name: str, personality: str):
        self.name = name
        self.personality = personality
        self.client = openai.OpenAI(api_key="sk-*****")
        self.assistant = self.client.beta.assistants.create(
            name=self.name,
            model="gpt-4-turbo-preview"
        ) In the class constructor, we initialize the OpenAI client as a class property by passing our OpenAI API key. Next, we create an class property that maps to our newly created Assistant. We store and as class properties for later use. assistant name personality The argument we are passing to the create method is just for identifying the Assistant in the OpenAI dashboard, and the AI is not actually aware of it at this point. You actually have to pass the name to the which we will see later. name instructions You could already set when creating the Assistant, but it will actually make your Assistant less flexible to dynamic changes. instructions You can update an Assistant by calling , but there is a better place to pass in dynamic values that we will see when we get to Runs. client.beta.assistants.update Note that if you pass here and then again when creating a Run, the Assistant's will be overwritten by the of the run. They do not complement each other, so choose one based on your needs: Assistant level for static instructions or Run level for dynamic instructions. instructions instructions instructions For the model, I chose the model so that we can add function calling in part 2 of this series. You could use if you want to save a few fractions of a penny while giving yourself a migraine of pure frustration down the line when we implement tools. gpt-4-turbo-preview gpt-3.5-turbo GPT 3.5 is terrible at calling tools; the hours I've lost trying to deal with it allow me to say that. 😝 I'll leave it at that, and more on this later. Creating a Thread, Adding Messages, and Retrieving the Last Message After we create an agent, we will need to start a conversation thread. class Agent:
    # ... (rest of code)
    def create_thread(self):
        self.thread = self.client.beta.threads.create() And we will want a way to add messages to that thread: class Agent:
    # ... (rest of code)
    def add_message(self, message):
        self.client.beta.threads.messages.create(
            thread_id=self.thread.id,
            role="user",
            content=message
        ) Note that at the moment, it is only possible to add messages with the role . I believe OpenAI plans on changing this in a future release as this is pretty limiting. user Now, we can get the last message in the thread: class Agent:
    # ... (rest of code)
    def get_last_message(self):
        return self.client.beta.threads.messages.list(
            thread_id=self.thread.id
        ).data[0].content[0].text.value Next, we create an entry point method to test out what we have so far. Currently, the method just returns the last message in the thread. It doesn't actually perform a Run. It's still brainless. run_agent run_agent class Agent:
    # ... (rest of code)
    def run_agent(self):
        message = self.get_last_message()
        return message Back in , we create the agent and our first thread. We add a message to the thread. Then return that same message back to the user, but this time, coming from that live thread. main.py from agent import Agent

agent = Agent(name="Bilbo Baggins",
                      personality="You are the accomplished and renowned adventurer from The Hobbit. You act like you are a bit of a homebody, but you are always up for an adventure. You worry a bit too much about breakfast.")

agent.create_thread()

while True:
    user_input = input("User: ")
    if user_input.lower() == 'exit':
        print("Exiting the agent...")
        break
    agent.add_message(user_input)
    answer = agent.run_agent()
    print(f"Assistant: {answer}") Let's run it: python3 main.py
User: hi
Assistant: hi Still not very smart. Closer to a parrot 🦜 than a hobbit. In the next section, the real fun begins. Creating and Polling a Run When you create a run, you need to periodically retrieve the object to check the status of the run. This is called polling, and it sucks. You need to poll in order to determine what your agent should do next. OpenAI plans to add support for streaming to make this simpler. In the meantime, I will show you how to set up polling in this next section. Run Note the on the following method names which is the standard in Python for indicating that the method is intended for internal use and should not be accessed directly by external code. _ First, let's create a helper method for creating a , and update to call this method: _create_run Run run_agent class Agent:
    # ... (rest of code)
    def get_breakfast_count_from_db(self):
        return 1

    def _create_run(self):
        count = self.get_breakfast_count_from_db()
        return self.client.beta.threads.runs.create(
            thread_id=self.thread.id,
            assistant_id=self.assistant.id,
            instructions=f"""
                Your name is: {self.name}
                Your personality is: {self.personality}

                Metadata related to this conversation:
                {{
                    "breakfast_count": {count}
                }}
            """,
        )

    def run_agent(self):
        run = self._create_run() # add this line
        message = self.get_last_message()
        return message Notice how we pass the and to create a run. thread.id assistant.id Remember how I said at the beginning that there was a better place to pass in dynamic instructions and data? That would be the parameter when creating the Run. In our case, we could have the breakfast be fetched from a database. This will allow you to easily pass in different relevant dynamic data every time you want to trigger an answer. instructions count Now, your agent is aware of the world changing around it and can act accordingly. I like to have a metadata JSON object in my instructions that keeps relevant dynamic context. This allows me to pass in data while being less verbose and in a format that the LLM understands really well. Don't run this yet; it won't work because we aren't waiting for the run to complete when we are getting the last message, so it will still be the last user message. Let's solve this by building out our polling mechanism. First, we will need a way to repeatedly and easily retrieve a run, so let's add a method: _retrieve_run class Agent:
    # ... (rest of code)
    def _retrieve_run(self, run: Run):
        return self.client.beta.threads.runs.retrieve(
            run_id=run.id, thread_id=self.thread.id) Notice how we need to pass both the and to find a specific run. run.id thread.id Add a method to our Agent class: _poll_run class Agent:
    # ... (rest of code)
    def _cancel_run(self, run: Run):
        self.client.beta.threads.runs.cancel(
            run_id=run.id, thread_id=self.thread.id)

    def _poll_run(self, run: Run):
        status = run.status
        start_time = time.time()
        while status != "completed":
            if status == 'failed':
                raise Exception(f"Run failed with error: {run.last_error}")
            if status == 'expired':
                raise Exception("Run expired.")

            time.sleep(1)
            run = self._retrieve_run(run)
            status = run.status

            elapsed_time = time.time() - start_time
            if elapsed_time > 120:  # 2 minutes
                self._cancel_run(run)
                raise Exception("Run took longer than 2 minutes.") 🥵 Phew, that's a lot... Let's unpack it. receives a object as an argument and extracts the current Run . All the available statuses can be found in the OpenAI . We'll just use a few that suit our current purpose. _poll_run Run status docs We now run a while loop to check for a completed status while handling a few error scenarios. The actual billing of the Assistant API is a bit murky, so to be on the safe side, I opted to cancel my runs after 2 minutes. Even though there is an status for when OpenAI cancels runs after 10 minutes. If a run takes more than 2 minutes, you probably have a problem anyway. expired Since I also don't want to poll every few milliseconds, I throttle my request by only polling every 1 second until I hit the 2-minute mark and cancel my run. You can adjust this to whatever you see fit. Each iteration after the delay, we fetch the Run status again. Now, let's plug all that into our method. You will notice we first create the run with then we poll with until we get an answer or an error is thrown, and finally when the polling is finished, we retrieve the last message from the thread which will now be from the agent. run_agent _create_run _poll_run We then return the message to our runtime loop, so it can be sent back to the user. class Agent:
    # ... (rest of code)
    def run_agent(self):
        run = self._create_run()
        self._poll_run(run) # add this line
        message = self.get_last_message()
        return message Voilà, now when you run your agent again, you will get a reply from our friendly Agent: python3 main.py
User: hi
Assistant: Hello there! What adventure can we embark on today? Or perhaps, before we set out, we should think about breakfast. Have you had yours yet? I've had mine, of course – can't start the day without a proper breakfast, you know.
User: how many breakfasts have you had?
Assistant: Ah, well, I've had just 1 breakfast today. But the day is still young, and there's always room for a second, isn't there? What about you? How can I assist you on this fine day? In part 2, we will add the ability for our Agent to call tools. You can find the full code on my . GitHub Thank you for your reading. Happy to hear any thoughts and feedback in the comments. Follow me on Linkedin for more content like this: https://www.linkedin.com/in/jean-marie-dalmasso-1b5473141/

How to Build an Agent With an OpenAI Assistant in Python - Part 1: Conversational

পাইথনে ওপেনএআই সহকারী দিয়ে কীভাবে এজেন্ট তৈরি করবেন - অংশ 1: কথোপকথন

So erstellen Sie einen Agenten mit einem OpenAI-Assistenten in Python – Teil 1: Konversational

Cómo crear un agente con un asistente OpenAI en Python - Parte 1: Conversacional

Comment créer un agent avec un assistant OpenAI en Python - Partie 1 : Conversationnel

पायथन में ओपनएआई असिस्टेंट के साथ एक एजेंट कैसे बनाएं - भाग 1: संवादात्मक

Python で OpenAI アシスタントを使用してエージェントを構築する方法 - パート 1: 会話型

Python에서 OpenAI 도우미를 사용하여 에이전트를 구축하는 방법 - 1부: 대화형

Como construir um agente com um assistente OpenAI em Python - Parte 1: Conversacional

Как создать агент с помощью OpenAI Assistant на Python. Часть 1: Диалог

Python'da OpenAI Asistanıyla Aracı Nasıl Oluşturulur - Bölüm 1: Konuşmaya Dayalı

Cách xây dựng một tác nhân với trợ lý OpenAI trong Python - Phần 1: Đàm thoại

如何使用 Python 中的 OpenAI 助手构建代理 - 第 1 部分：对话式

Introduction In developing large-scale systems, it is always essential to think about resilience. The load on an application typically varies over time based on the number of active users or the types of activities they are performing. One of the most common situations you will encounter sooner or later is server overload caused by unanticipated bursts in traffic. It happens because every server has functional limitations. That means the server has limited network and computational resources like CPU and memory, so the server can effectively handle only a certain number of requests. When the number of requests reaches some threshold and continues growing, it increases resource utilization, leading to higher latency and reducing availability. There are many strategies to handle that situation. For instance, you can run several instances of your application and use autoscaling to match the provisioned resources to the users's needs at any time. However, when the amount of traffic is incredibly huge, scaling out may take time, and may be overloaded or fail. I will discuss common strategies to prevent such a situation in this article. servers Causes leading to traffic spikes There may be many reasons leading to the sudden traffic spikes. Let's consider some of the common. An application can experience sudden spikes in traffic due to events like breaking news, product launches, or promotional campaigns. If the servers are not prepared to handle this surge of traffic, they can quickly become overloaded. Also, when the traffic to the application is dropped temporarily for some reason, the application scales down. Then, when the traffic returns, the application can't scale out fast, leading to server overload and even failure. Another common reason is that malicious bots, or malware, can flood a server with requests, effectively overwhelming its resources. It is known as a distributed denial-of-service (DDoS) attack. Strategies to prevent service overload There are two main strategies to prevent server overload: Throttling, Load shedding. Throttling controls the consumption of resources used by an instance of an application, an individual user, or an entire service. For example, a server experiencing high demand may start throttling requests, slowing them down, or rejecting them until the load subsides. It allows the system to handle the incoming traffic without crashing or becoming unresponsive. One common technique for throttling is rate limiting, which defines the maximum number of requests a user, a client, or an entire group can make within a specific time frame, ensuring no single client or user can monopolize system resources. It is a proactive measure that aims to prevent the system from reaching its capacity in the first place. For instance, a rate limiter may allow 100 requests from a given IP address per minute. Exceeding this limit could result in the user being temporarily blocked or having their requests queued until they are within the allowable rate. Load shedding is a more drastic technique. It involves intentionally dropping requests to prevent a server from becoming overloaded. It is only done as a last resort when other measures, such as rate limiting, have proven insufficient. By shedding requests from one application instance, the load can be redistributed to other application instances in the cluster, preventing any single instance from reaching its limits. An example of rate-limiting Several algorithms can be used to implement rate limiters. Here, let's consider a basic implementation of a rate limiter using a token bucket algorithm, where tokens are added to the bucket at a fixed rate. The request is allowed if a token is available when a request is made. Otherwise, it is denied. This implementation can be applied to throttling requests from particular users, IP addresses, API keys, or other groups of users. main.go package main

import (
	"fmt"
	"log"
	"net/http"
	"sync"
	"time"
)

type bucket struct {
	remainingTokens int
	lastRefillTime  time.Time
}

type RateLimiter struct {
	maxTokens      int
	refillInterval time.Duration
	buckets        map[string]*bucket
	mu             sync.Mutex
}

func NewRateLimiter(rate int, perInterval time.Duration) *RateLimiter {
	return &RateLimiter{
		maxTokens:      rate,
		refillInterval: perInterval,
		buckets:        make(map[string]*bucket),
	}
}

func (rl *RateLimiter) IsLimitReached(id string) bool {
	rl.mu.Lock()
	defer rl.mu.Unlock()

	b, ok := rl.buckets[id]

	// If the bucket doesn't exist, it is the first request for this client.
	// Create a new bucket and allow the request.
	if !ok {
		rl.buckets[id] = &bucket{
			remainingTokens: rl.maxTokens - 1,
			lastRefillTime:  time.Now(),
		}
		return false
	}

	// Calculate the number of tokens to add to the bucket since the last
	// request.
	refillInterval := int(time.Since(b.lastRefillTime) / rl.refillInterval)
	tokensAdded := rl.maxTokens * refillInterval
	currentTokens := b.remainingTokens + tokensAdded

	// There is no tokens to serve the request for this client.
	// Reject the request.
	if currentTokens < 1 {
		return true
	}

	if currentTokens > rl.maxTokens {
		// If the number of current tokens is greater than the maximum allowed,
		// then reset the bucket and decrease the number of tokens by 1.
		b.lastRefillTime = time.Now()
		b.remainingTokens = rl.maxTokens - 1
	} else {
		// Otherwise, update the bucket and decrease the number of tokens by 1.
		deltaTokens := currentTokens - b.remainingTokens
		deltaRefills := deltaTokens / rl.maxTokens
		deltaTime := time.Duration(deltaRefills) * rl.refillInterval
		b.lastRefillTime = b.lastRefillTime.Add(deltaTime)
		b.remainingTokens = currentTokens - 1
	}

	// Allow the request.
	return false
}

type Handler struct {
	rl *RateLimiter
}

func NewHandler(rl *RateLimiter) *Handler {
	return &Handler{rl: rl}
}

func (h *Handler) Handler(w http.ResponseWriter, r *http.Request) {
	// Here should be the logic to get the client ID from the request 
	// (it could be a user ID, an IP address, an API key, etc.)
	clientID := "some-client-id"
	if h.rl.IsLimitReached(clientID) {
		w.WriteHeader(http.StatusTooManyRequests)
		fmt.Fprint(w, http.StatusText(http.StatusTooManyRequests))
		return
	}

	w.WriteHeader(http.StatusOK)
	fmt.Fprint(w, http.StatusText(http.StatusOK))
}

func main() {
	// We allow 1000 requests per second per client to our service.
	rl := NewRateLimiter(1000, 1*time.Second)
	h := NewHandler(rl)
	http.HandleFunc("/", h.Handler)
	log.Fatal(http.ListenAndServe(":8080", nil))
} Let's run our example using Docker Compose with configured for a container. Also, we will use , a benchmarking tool, to check how our application works on different loads. We will use the following and : resource constraints bombardier Dockerfile docker-compose.yml Dockerfile FROM golang:1.21 AS build-stage
WORKDIR /code
COPY main.go ./
RUN CGO_ENABLED=0 GOOS=linux go build -o /service main.go
FROM gcr.io/distroless/base AS build-release-stage
WORKDIR /
COPY --from=build-stage /service /service
EXPOSE 8080
ENTRYPOINT ["/service"] docker-compose.yml services:
  rate_limiting:
    build: .
    ports:
      - "8080:8080"
    deploy:
      resources:
        limits:
          cpus: '.20'
          memory: 100M
        reservations:
          cpus: '0.10'
          memory: 50M Let's run our application using the following command: docker compose -f ./cmd/rate_limiting/docker-compose.yml up --build --force-recreate -d You can monitor the container resource usage using the following command: dcocker stats Let's run the bombardier with different settings and check the results in a separate terminal window. First, let's try sending 10000 requests with one concurrent connection. $ bombardier -c 1 -n 10000 http://127.0.0.1:8080/
Bombarding http://127.0.0.1:8080/ with 10000 request(s) using 1 connection(s)
 10000 / 10000 [===============================================================] 100.00% 859/s 11s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec       868.33     833.53    2684.45
  Latency        1.15ms     6.93ms    75.42ms
  HTTP codes:
    1xx - 0, 2xx - 10000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:   151.93KB/s From the results, you can see that the request rate on average is less than 1000 requests per second, which is the maximum allowed rate, and all requests were processed successfully. Next,  let's try sending 10000 requests with 100 concurrent connections. $ bombardier -c 100 -n 10000 http://127.0.0.1:8080/
Bombarding http://127.0.0.1:8080/ with 10000 request(s) using 100 connection(s)
 10000 / 10000 [===============================================================] 100.00% 3320/s 3s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec      3395.87    6984.32   32322.59
  Latency       28.02ms    37.61ms   196.95ms
  HTTP codes:
    1xx - 0, 2xx - 3000, 3xx - 0, 4xx - 7000, 5xx - 0
    others - 0
  Throughput:   675.35KB/s You can see that the server started to return HTTP 429 status code, which means the server is overloaded due to many requests. Clean up your environment using the following command after all tests are done: docker compose -f ./cmd/rate_limiting/docker-compose.yml down --remove-orphans --timeout 1 --volumes An example of load-shedding Depending on your particular case, you can use different approaches to load shedding: Dropping random requests when the system becomes overloaded, Using a load balancer to distribute the load evenly, Monitoring server load continuously and dropping or rejecting requests when a certain load level is reached, Monitoring system metrics like CPU utilization, network I/O, or memory usage and dropping requests when these metrics reach certain thresholds, Set up priority levels for requests and start refusing low-priority requests when the system is under load, Etc. Load shedding can be implemented at various levels, including load balancers, servers, and software libraries. It requires careful planning and implementation, as poorly implemented load shedding can result in important requests being dropped, causing a negative impact on user experience or system functionality. Let's implement a simple load-shedding mechanism by detecting request overload conditions and responding with an HTTP 503 status code. main .go package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"sync/atomic"
	"time"
)

type LoadShedder struct {
	isOverloaded atomic.Bool
}

func NewLoadShedder(ctx context.Context, checkInterval, overloadFactor time.Duration) *LoadShedder {
	ls := LoadShedder{}

	go ls.runOverloadDetector(ctx, checkInterval, overloadFactor)

	return &ls
}

func (ls *LoadShedder) runOverloadDetector(ctx context.Context, checkInterval, overloadFactor time.Duration) {
	ticker := time.NewTicker(checkInterval)
	defer ticker.Stop()

	// Start with a fresh start time.
	startTime := time.Now()
	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			// Check how long it took to process the last batch of requests.
			elapsed := time.Since(startTime)
			if elapsed > overloadFactor {
				// If it took longer than the overload factor, we're overloaded.
				ls.isOverloaded.Store(true)
			} else {
				// Otherwise, we're not overloaded.
				ls.isOverloaded.Store(false)
			}
			// Reset the start time.
			startTime = time.Now()
		}
	}
}

func (ls *LoadShedder) IsOverloaded() bool {
	return ls.isOverloaded.Load()
}

type Handler struct {
	ls *LoadShedder
}

func NewHandler(ls *LoadShedder) *Handler {
	return &Handler{ls: ls}
}

func (h *Handler) Handler(w http.ResponseWriter, r *http.Request) {
	if h.ls.IsOverloaded() {
		w.WriteHeader(http.StatusServiceUnavailable)
		fmt.Fprint(w, http.StatusText(http.StatusServiceUnavailable))
		return
	}

	w.WriteHeader(http.StatusOK)
	fmt.Fprint(w, http.StatusText(http.StatusOK))
}

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// The load shedder will check every 100ms if the last batch of requests
	// took longer than 200ms.
	ls := NewLoadShedder(ctx, 100*time.Millisecond, 200*time.Millisecond)

	h := NewHandler(ls)
	http.HandleFunc("/", h.Handler)

	log.Fatal(http.ListenAndServe(":8080", nil))
} We define the struct to run the load detector used in our request handler to check if we should return the HTTP 503 status code or process the request. The struct has an atomic flag indicating if the system is overloaded. We use atomic boolean value to make it thread-safe. We also have the method that returns the current value of the flag. In the function, we create a new and run the overload detector in a goroutine that checks according to the specified interval if the system is overloaded based on the overload factor. We check how much time has passed since the last check. If it is higher than the overload factor, the system is overloaded. That means that our request handler used resources for too long, and we need more capacity to handle the requests. LoadShedder LoadShedder IsOverloaded NewLoadShedder LoadShedder Let's run our example using. We will use the following and : Dockerfile docker-compose.yml Dockerfile FROM golang:1.21 AS build-stage
WORKDIR /code
COPY main.go ./
RUN CGO_ENABLED=0 GOOS=linux go build -o /service main.go
FROM gcr.io/distroless/base AS build-release-stage
WORKDIR /
COPY --from=build-stage /service /service
EXPOSE 8080
ENTRYPOINT ["/service"] docker-compose.yml services:
  load_shedding:
    build: .
    ports:
      - "8080:8080"
    deploy:
      resources:
        limits:
          cpus: '.20'
          memory: 100M
        reservations:
          cpus: '0.10'
          memory: 50M Let's run our application using the following command: docker compose -f ./cmd/load_shedding/docker-compose.yml up --build --force-recreate -d You can monitor the container resource usage using the following command: dcocker stats Let's run the bombardier with different settings and check the results in a separate terminal window. First, let's try sending 10000 requests with 10 concurrent connections. $ bombardier -c 10 -n 10000 http://127.0.0.1:8080/
Bombarding http://127.0.0.1:8080/ with 10000 request(s) using 10 connection(s)
 10000 / 10000 [===============================================================] 100.00% 1346/s 7s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec      1389.49    1582.00    6284.42
  Latency        7.24ms    22.41ms    98.43ms
  HTTP codes:
    1xx - 0, 2xx - 10000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:   242.78KB/s The results show that the maximum latency is less than 100 ms, and all requests were processed successfully. Next, let's try sending 10000 requests with 1000 concurrent connections. $ bombardier -c 1000 -n 10000 http://127.0.0.1:8080/
Bombarding http://127.0.0.1:8080/ with 10000 request(s) using 1000 connection(s)
 10000 / 10000 [===============================================================] 100.00% 3791/s 2s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec      4242.28   11985.30   58592.64
  Latency      211.67ms   210.89ms      1.54s
  HTTP codes:
    1xx - 0, 2xx - 8823, 3xx - 0, 4xx - 0, 5xx - 1177
    others - 0
  Throughput:   696.96KB/s You can see that the server started to return HTTP 503 status code, which means the server is overloaded due to many requests, and you can see the maximum latency is more than 1 second. Clean up your environment using the following command after all tests are done: docker compose -f ./cmd/load_shedding/docker-compose.yml down --remove-orphans --timeout 1 --volumes Conclusion In this article, we discussed the causes leading to traffic spikes and strategies to prevent server overload. We also implemented a simple rate limiter and load shedder to demonstrate how they work. You can find all the code from the article here . https://github.com/ivanlemeshev/resilience For the production environment, you can use ready-to-use rate limiters like or . https://pkg.go.dev/golang.org/x/time/rate https://github.com/uber-go/ratelimit

How To Prevent Server Overload In Go

This paper is under CC BY 4.0 DEED license. available on arxiv Authors: (1) Ehsan Toreini, University of Surrey, UK; (2) Maryam Mehrnezhad, Royal Holloway University of London; (3) Aad Van Moorsel, Birmingham University. Table of Links Abstract & Introduction Background and Related Work FaaS Architecture Implementation and Performance Analysis Conclusion Acknowledgment and References 4 Implementation and Performance Analysis 4.1 Proof-of-Concept Implementation The back–end is implemented in Python v3.7.1 and the front–end is implemented with Node.js v10.15.3. In our evaluations, the computations required for generation of the cryptogram table (in the ML system) is developed with Python. The elliptic curve operations make use of the Python package tinyec and the conversion of Python classes to a JSON compatible format uses the Python package JSONpickle. All the experiments are conducted on a MacBook pro laptop with the following configurations: CPU 2.7 GHz Quad-Core Intel Core i7 with 16 GB Memory running MacOS Catalina v.10.15.5 for the Operating System Tools and Platform: We use a publicly available dataset from Medical Expenditure Panel Survey (MEPS) [21] that contains 15830 data points about the healthcare utilization of individuals. We developed a model (Logistic Regression) that determines whether a specific patient requires health services, such as additional care. This ML system assigns a score to each patient. If the score is above a preset threshold, then the patient requires extra health services. In the MEPS dataset, the protected attribute is “race”. A fair system provide Case-Study Dataset: Table 4: The required permutations to compute the fairness metrics of an ML system such services fairly independent of the patient’s race. Here, the privileged race group in this dataset is “white ethnicity”. We have used 50% of the dataset as training, 30% as validation and the remaining 20% as test dataset. We set the number of cryptogram table samples to equal the size of test set (N = 3166). In this example we include three attributes in the cryptogram to represent the binary values of A, Y and Yˆ (section 2.1), thus leading to 8 permutations for each data sample. 4.2 Performance This section presents the execution time per data point for each of the main computational tasks, in each protocol stage. Recall that phase I was executed before the ML system’s training and testing. This stage can be developed (and stored separately) in parallel to the implementation of the model in order to mitigate the performance challenge of Phase I. In our implementation, the output of this stage (cryptogram table) is stored in a separate file in JSON format and can be retrieved at the beginning of the phase II Phase II begins after the ML model is trained, tested, and validated. This stage uses the output of the ML model to generate the fairness auditing table from the cryptogram table as well as ZKP for knowledge of the permutation. The output of this phase is transmitted to the Fairness Auditor Service in JSON format for phase III. At this stage, first the ZKPs are verified and then, the summation of the cryptograms determines the number of permutations for each of the sensitive groups. Once the auditing service has these numbers, it can compute the fairness of the ML system In our evaluations (where N = 3166), public/private key pair generation completes in 60 milliseconds (ms) on average with standard deviation of 6ms. The execution time for ZKP of private key was roughly the same (60ms on average with standard deviation of 6ms). The generation of reconstructed public key took around 450ms with standard deviation of 8ms. The most computationally expensive stage in phase I was the 1–out–of–8 ZKP for each of the permutations. This stage took longer than the other ones because first, the algorithm is more complicated and second, it should be repeated 8 times (for each of the permutations separately) for every row in cryptogram table. The computation of 1–out–of–8 ZKPs takes 1.7 seconds for each data sample with STD of 0.1 seconds. Overall, phase I took around 14 seconds with STD of 1 second for each data sample in the test set. In our experiments (where N = 3166 samples), the total execution of phase I took roughly 12 hours and 54 minutes. Phase II consists of creation of the auditing table and generation of the ZKP for knowledge of the permutation. The fairness auditing table is derived from the cryptogram table (as it is mapping the encoding to the corresponding permutation number in the cryptogram table). The elapsed time for such derivation is negligible (total: 1ms). The generation of ZKP for knowledge of the permutation executed less than 60ms on average with standard deviation of 3ms for each data sample. The completion of both stages took less than 3 minutes. The fairness auditing table is sent to the Fairness Auditor Service for Phase III. The verification of ZKPs in the last phase (Phase III) is a computationally expensive operation. The ZKP for the ownership of the private key took around 260ms on average with standard deviation of 2ms. The verification of 1–out– of–8 ZKP for each data point roughly took 2.5 seconds on average with 20ms standard deviation. The verification of the ZKP for knowledge of permutation executed in 100ms with standard deviation of 5ms. The summation of the cryptograms after verification took 450ms overall for N = 3166 items. In our experiment, completion of the stages in phase III took around 2 hours and 30 minutes in total. In summary, the experimental setup for our architecture, where we computed the required cryptograms and ZKPs for N = 3166 data points in a real–world dataset, overall time was around 15 hours on the laptop specification given earlier. The main part of the time is consumed by the computation required for phase I (12 hours and 54 minutes). However, as we noted before, Phase I can be executed before the ML model setup and is stored in a separate JSON file and will be loaded at the beginning of stage II (after the training and validation of the ML model is complete). The other main computational effort, which can only be done after the ML system’s outcomes have been obtained, is in Phase III. For our example, actual computation of fairness takes two and a half hours. In summary, the creation and handling of cryptograms takes considerable computational effort for realistic datasets and for the fairness metrics that require three attributes. In what follows we analyse how performance scales with respect to the number of data points as well as with the number of attributes represented in the cryptograms.

Evaluating the Performance of Fairness Audits in Real-World ML Systems

ClickHouse NewYear Release A new version of ClickHouse was released in December 2023. Let’s see what exciting new features are inside and dive into one of them - new refreshable materialized views! What is on the board? This release contains 21 new features, 18 performance optimizations, and 37 bug fixes. Here are several exciting features I would like to you to know about: New Refreshable Materialized Views. It creates crontab-like refreshing views. New PASTE JOIN clause. Handy 1-to-1 row join of datasets. New ORDER BY clause ALL. Syntax sugar to order query by every column from the SELECT statement. New mutation command ALTER TABLE <table> APPLY DELETED MASK. It applies a mask written by lightweight delete to remove rows marked as deleted from the disk. Ability to refer to ALIAS column in index (non-primary-key) definitions. The complete changelog you can read . here In the meantime, I’ll show an example of the materialized views usage. Refreshable Materialized Views ClickHouse runs the corresponding query and stores its result in a table, replacing the table's previous contents, which differs from regular non-refreshable materialized views fundamentally: it’s not incremental, it’s entire recalculation and reformating data from the source tables. periodically atomically Of course, there are trade-offs: it’s not online anymore, and this is the cost for aggregation queries. Non-refreshable materialized views are limited by using aggregation functions in the query since they apply only to the incremental data. There are no restrictions on the SELECT query. Table functions (e.g. url()), views, UNION, JOIN, are all allowed. Since the feature is experimental, you have to enable it by setting SET allow_experimental_refreshable_materialized_view = 1 Let’s create a sort of synthetic example: crawler traverses urls on the single domain using DFS and stores request headers, response headers, and content to the plain table. CREATE TABLE IF NOT EXISTS crawler.pages (
  url String,
  req_headers Map(String, String),
  resp_headers Map(String, String),
  resp_code Int, 
  content String,
  links Nested (
    uri String,
    titles Array(String)
  )
)
ENGINE MergeTree
PRIMARY KEY url
ORDER BY url This new feature allows me to build handy pipelines. The following example is pretty synthetic, however, it demonstrates The original (“source”) table may contain duplicates for any page, so the first step of data processing would be the selection of the unique pages with the additional condition non-empty etag response header and response code 200 using data transformation like domain extraction and crawling date parsing. CREATE MATERIALIZED VIEW IF NOT EXISTS crawler.uniqpages
REFRESH EVERY 2 MINUTE   --- refresh could be SECOND|MINUTE|HOUR|DAY|WEEK|MONTH|YEAR
RANDOMIZE FOR 1 MINUTE   
ENGINE = Memory
AS SELECT DISTINCT
    domainWithoutWWW(url) as domain, -- extract domain from the url builtin func
    path(url) as uri, -- extract path from the url builtin func
    url as url,
    resp_headers['etag'] as etag,
    parseDateTime32BestEffort(resp_headers['date']) as crawled_at, -- datetime parsing
    req_headers['user-agent'] as user_agent,
    resp_code as resp_code,
    links.uri as nei_uris,
    links.titles as nei_titles
FROM
    crawler.pages
WHERE
    resp_headers['etag'] <> '' AND resp_code = 200 The second step is to precalculate per domain unique valid page stats. This step would be dependent on the uniqpages view refresh completion, without this dependency those both start simultaneously. CREATE MATERIALIZED VIEW IF NOT EXISTS crawler.stats
REFRESH EVERY 2 MINUTE  
DEPENDS ON crawler.uniqpages -- sync with uniqpages view, this ensures refresh to run in sequence
ENGINE = Memory
AS SELECT DISTINCT
    domain,
    count(1) as qnt
FROM
    crawler.uniqpages
GROUP BY
    1
ORDER BY
    ALL Check the for any record to verify results. uniqpages dd4a5ae26651 :) select * from uniqpages limit 1 format Vertical

SELECT *
FROM uniqpages
LIMIT 1
FORMAT Vertical

Query id: ba19be7b-0891-4c07-a707-62e5e3eaa3c4

Row 1:
──────
domain:     levelup.gitconnected.com
uri:        /%EF%B8%8F-how-to-deploy-a-static-website-or-a-server-side-rendered-app-with-aws-amplify-hosting-806b4d620458
url:        https://levelup.gitconnected.com/%EF%B8%8F-how-to-deploy-a-static-website-or-a-server-side-rendered-app-with-aws-amplify-hosting-806b4d620458?source=---------4-----------------------
etag:       W/"1c8ed-8WUFxf2KDh2sBbP4zrGTJCdxo4I"
crawled_at: 2023-12-31 14:20:54
user_agent: Mozilla/5.0 (X11; Linux x86_64; Testbot/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36
resp_code:  200
nei_uris:   ['https://levelup.gitconnected.com/?source=post_page-----806b4d620458--------------------------------']
nei_titles: [['Level Up Coding','Level Up Coding']]

1 row in set. Elapsed: 0.001 sec. Processed 22.91 thousand rows, 209.44 MB (30.17 million rows/s., 275.83 GB/s.)
Peak memory usage: 15.02 KiB. Let’s check the stats view dd4a5ae26651 :) select domain, qnt from stats order by domain

SELECT
  domain,
  qnt
FROM stats
ORDER BY domain ASC

Query id: 9a2975ea-b52a-44ec-acae-698bb51f1973

┌─domain───────────────────┬───qnt─┐
│ habr.com                 │ 12941 │
│ levelup.gitconnected.com │  9295 │
│ w3.org                   │   665 │
│ wikipedia.org            │     6 │
└──────────────────────────┴───────┘

4 rows in set. Elapsed: 0.001 sec. … and in a couple of minutes dd4a5ae26651 :) select domain, qnt from stats order by domain

SELECT
  domain,
  qnt
FROM stats
ORDER BY domain ASC

Query id: 9a2975ea-b52a-44ec-acae-698bb51f1973

┌─domain───────────────────┬───qnt─┐
│ habr.com                 │ 12947 │
│ levelup.gitconnected.com │  9316 │
│ w3.org                   │   665 │
│ wikipedia.org            │     6 │
└──────────────────────────┴───────┘

4 rows in set. Elapsed: 0.001 sec. Conclusion A Refreshable Materialized View is quite handy and the most interesting feature of this release. It allows me to build useful data ETL pipelines in a very simple way just using SQL letting the ClickHouse to do the routine job for you. Let me know in the comments which of the new features excites you the most!

Refreshable Materialized Views and other features in ClickHouse Release 23.12

This paper is available on arxiv under CC 4.0 license. Authors: . Gemini Team, Google Table of Links Abstract and Introduction Model Architecture Training Infrastructure Training Dataset Evaluation Responsible Deployment Discussion and Conclusion, References Contributions and Acknowledgments Appendix 3. Training Infrastructure We trained Gemini models using TPUv5e and TPUv4 (Jouppi et al., 2023), depending on their sizes and configuration. Training Gemini Ultra used a large fleet of TPUv4 accelerators across multiple data centers. This represents a significant increase in scale over our prior flagship model PaLM-2 which presented new infrastructure challenges. Scaling up the number of accelerators results in a proportionate decrease in the mean time between failure of hardware in the overall system. We minimized the rate of planned reschedules and preemptions, but genuine machine failures are commonplace across all hardware accelerators at such large scales. TPUv4 accelerators are deployed in “SuperPods” of 4096 chips, each connected to a dedicated optical switch, which can dynamically reconfigure 4x4x4 chip cubes into arbitrary 3D torus topologies in around 10 seconds (Jouppi et al., 2023). For Gemini Ultra, we decided to retain a small number of cubes per superpod to allow for hot standbys and rolling maintenance. TPU accelerators primarily communicate over the high-speed inter-chip-interconnect, but at the Gemini Ultra scale, we combine SuperPods in multiple data centers using Google’s intra-cluster and inter-cluster network (Poutievski et al., 2022; Wetherall et al., 2023; yao Hong et al., 2018). Google’s network latencies and bandwidths are sufficient to support the commonly used synchronous training paradigm, exploiting model parallelism within superpods and data-parallelism across superpods. The ‘single controller’ programming model of Jax (Bradbury et al., 2018) and Pathways (Barham et al., 2022) allows a single Python process to orchestrate the entire training run, dramatically simplifying the development workflow. The GSPMD partitioner (Xu et al., 2021) in the XLA compiler partitions the training step computation, and the MegaScale XLA compiler (XLA, 2019) pass statically schedules appropriate collectives so that they maximally overlap with the computation with very little variation in step time. Maintaining a high goodput[3] at this scale would have been impossible using the conventional approach of periodic checkpointing of weights to persistent cluster storage. For Gemini, we instead made use of redundant in-memory copies of the model state, and on any unplanned hardware failures, we rapidly recover directly from an intact model replica. Compared to both PaLM and PaLM-2 (Anil et al., 2023), this provided a substantial speedup in recovery time, despite the significantly larger training resources being used. As a result, the overall goodput for the largest-scale training job increased from 85% to 97%. Training at unprecedented scale invariably surfaces new and interesting systems failure modes - and in this instance one of the problems that we needed to address was that of “Silent Data Corruption (SDC)” (Dixit et al., 2021; Hochschild et al., 2021; Vishwanathan et al., 2015). Although these are extremely rare, the scale of Gemini means that we can expect SDC events to impact training every week or two. Rapidly detecting and removing faulty hardware required several new techniques that exploit deterministic replay to isolate incorrect computations, combined with proactive SDC scanners on idle machines and hot standbys. Our fully deterministic infrastructure allowed us to quickly identify root causes (including hardware failures) during the development leading up to the Ultra model, and this was a crucial ingredient towards stable training. [3] We define goodput as the time spent computing useful new steps over the elapsed time of the training job.

Stories, News & Mentions about ELYPS

Elyps’s Stories on HackerNoon