**About Us: **\r\n\r\nLumApps teams across the world all share the same objective

BoostKPI

boostkpi

Boosting Development Efficiency With the Specification-First Principle

As an AI expert, let me be blunt - cyber threats are a massive, unrelenting risk that no business with an online presence can afford to ignore these days. Whether you're running an e-commerce site, logistics operation, or tech firm, phishing, ransomware attacks, or data breaches - these scourges can cripple a company, leading to devastating financial losses, irreparable brand damage, and even regulatory penalties that could bury you. I've personally seen the aftermath of cyber strikes on SMBs with limited security resources and tight budgets. In those cases, traditional cybersecurity just doesn't cut it against today's attack of threats. But that's exactly where artificial intelligence (AI) emerges as a total game-changer. discovered that organizations leveraging AI and automation extensively saved a mind-blowing $1.76 million on average compared to their less AI-savvy counterparts. IBM's 2023 Cost of Data Breach Report And I can testify that AI-powered cyber defense solutions truly earn those savings. By unleashing cutting-edge AI techniques like machine learning, deep learning models, and natural language processing, modern AI cybersecurity acts like an indispensable digital force multiplier reinforcing our defenses against even the wiliest adversaries. As an AI expert, I've deployed these capabilities for clients - it's like having an entire squad of cybersecurity sentries patrolling the digital premises 24/7 to keep the bad actors at bay. Moving forward, I don't believe adopting advanced AI will be a luxury for organizations - it's a flat-out necessity we can't ignore. Whether you're a small business or a giant enterprise, you'd better get proactive and start leveraging AI's offensive cybersecurity potential to safeguard your vital digital assets and ensure continuous operations. In my opinion, AI is quickly separating the cybersecurity haves from the have-nots. AI's Cyber Offense: How It Turbocharges Threat Detection I can't overstate how AI is revolutionizing cybersecurity by unleashing cutting-edge threat detection and response capabilities never before possible. Just look at the rapidly growing market - projects that AI in the cybersecurity sector will explode from $22.4 billion in 2023 to $60.6 billion by 2028. And there are good reasons behind that meteoric growth. MarketsandMarkets Machine learning algorithms excel at quickly identifying patterns and anomalies within massive datasets - making them invaluable assets for recognizing potential cyber threats amid tons of network traffic, user activity logs, and other data. By training these algorithms on historical data, we can have them recognize known attack signatures and deviations signaling new threats before they can wreak havoc. One study I find particularly compelling is from MDPI, showing machine learning models achieved up to 99% malware detection accuracy using techniques like Decision Trees, Convolutional Neural Networks, and Support Vector Machines. That's the level of precision AI can deliver against emerging threats. There are deep learning models that I've deployed to rapidly analyze and classify malware samples with incredible accuracy. The latest deep-learning architectures can see patterns invisible to the naked eye. However, some of the most exciting AI cybersecurity applications my team has tested involve reinforcement learning for adaptive threat response. These self-learning systems can dynamically adjust defensive actions on the fly based on their experiences against evolving attack vectors. Allied Market Research predicts reinforcement learning will see explosive 41.5% annual growth from 2023 to 2032, with cybersecurity as a prime use case. Let's not overlook natural language processing (NLP) for automating incident analysis. I've personally overseen NLP deployments that can rapidly comprehend human language data from reports, intelligence feeds, and more - streamlining our incident response workflows and prioritizing genuine threats over false positives. Combining all these AI disciplines is delivering incredible offensive capabilities to counter even the trickiest cyber adversaries. As an expert and founder in this space, I truly believe AI's ability to autonomously detect and shut down threats before impact will utterly transform cybersecurity over the coming years. Why UEBA and User Behavior Monitoring Matter (And How AI Excels) The cyber risk from malicious insiders or compromised user accounts is one we can't overlook either. That's why I consider user and entity behavior analytics (UEBA) powered by advanced AI/ML models to be paramount for comprehensive, 360-degree security visibility. At a basic level, these AI-driven UEBA solutions leverage machine learning to establish baselines modeling each user's normal behavior based on their activity patterns. Once those baselines are established, the AI can continuously monitor for deviations that may indicate insider threats, privilege abuse, compromised accounts, and other insider risk vectors. What makes this application of AI so powerful is its ability to dynamically score the risk level of unusual user actions based on contextual factors like their role, access privileges, the sensitivity of data/systems involved, and more. This enables real-time flagging of potentially malicious insider activity for further investigation - something that was highly prone to human error with traditional, rigid rule-based approaches. Another area where I've seen UEBA with AI shine is detecting compromised privileged user accounts, which are prime targets for external threat actors seeking a foothold. By understanding the unique behavior patterns of different privileged user roles, AI can rapidly identify the subtle anomalies of a privileged account under hostile control - giving our teams a chance to shut down that breach before catastrophic damage occurs. In my opinion, deploying AI for insider and privileged user threat detection by understanding human behavior is an absolute game-changer. Monitoring our people and accounts is just as vital as external threat monitoring - without it, we're leaving a gaping hole in our defenses that skilled adversaries will inevitably find and exploit. Of course, my teams always emphasize that AI-powered UEBA is just one layer in a comprehensive, defense-in-depth strategy. But it's a crucial layer that lets us get in front of insider incidents before they escalate - buying time that can mean the difference between a contained issue and a full-blown crisis. The Flip Side: Defending Against AI-Powered Threats As amazing as AI's cybersecurity offensive capabilities are, I have to address the flip side too: hackers are actively working on ways to weaponize AI against us. Adversarial attacks designed to bypass, poison, or steal our AI models are a clear and present danger we can't ignore. One disturbing forecast from research analysts indicates: Up to 30% of AI cyberattacks will leverage training data poisoning, model stealing, or adversarial examples to attack AI-powered systems. But we're not defenseless - there are proven ways to harden AI systems against these tactics if we invest in robust AI security controls from the start. Approaches like adversarial training, where models are intentionally exposed to deceptive inputs during the learning process, can improve resilience against evasion. Differential privacy techniques can also obscure model behavior to thwart extraction attacks. We follow best practices around secure AI development Life Cycle, subjecting our models to rigorous adversarial testing and employing defensive AI techniques to stay ahead of emerging threats. I don't mean to sound alarmist, but the cybersecurity industry at large needs to prioritize battling AI-powered threats just as much as leveraging AI for defense. It's a two-way street - we must secure the AI models underpinning our cyber defenses with the same vigor we apply AI offensively. Neglecting AI security WILL be exploited by sophisticated hackers. So while I'm incredibly confident about AI's potential to revolutionize cyber threat detection and response, I'm also pragmatic. We simply can't ignore the dark side and must proactively invest in developing and deploying AI securely to maintain a decisive strategic advantage. Integrating AI Across Security Operations Of course, AI alone is not a silver bullet for cybersecurity. Its true power comes from thoughtful integration across existing processes, teams, and technology stacks. As an enthusiastic AI Expert, I emphasize this force-multiplier effect to clients constantly. AI should augment and empower human expertise, not replace it entirely. That's why the most successful deployments I've seen use AI to offload tedious, repetitive tasks to machines - freeing up skilled personnel to focus on higher-level analysis, investigation, and decision-making that requires human judgment. That's also why AI is rapidly becoming essential for SOC analysts and incident response teams: by automating alert triage, log analysis and basic response workflows, AI drastically reduces burnout from low-value "grunt work" while improving overall efficiency. Research from Capgemini revealed organizations using AI for security operations saw a 12% increase in breach detection rates and an impressive 19% reduction in time-to-respond. Those are major gains I've validated through my own startup's AI deployments. AI and machine learning are also driving huge advances in security orchestration, automation, and response (SOAR) tools that streamline entire incident response processes. With SOAR, AI handles tasks like correlating disparate security signals, surfacing relevant threat intelligence, and even executing semi-automated response playbooks. It's a huge step towards smarter, more autonomous incident handling. Ultimately though, AI is a complement to skilled human teams, not a wholesale replacement. Cybersecurity operations will always require human expertise, analysis, and judgment. But by intelligently augmenting our people with AI automation and assistance, we can concentrate their talents on higher-impact duties while optimizing productivity and efficiency. I already see this human-machine symbiosis enhancing the industry's ability to outpace threats. As AI integration continues maturing, I'm confident embracing this collaborative model is the only way we'll realistically close the cybersecurity skills gap plaguing our field. AI's Cybersecurity Superpowers By rapidly analyzing network data, user activities, system logs, and more, AI can identify malicious behavior patterns and potential threats with superhuman speed - giving us an early warning to shut down attacks before escalation. Threat Detection & Prevention: While phishing lures and malware grow more devious, AI algorithms can precisely distinguish malicious files and emails from legitimate ones by evaluating content, sender metadata, behavioral signals, and other artifacts that often evade human analysis. Phishing and Malware Detection: Rather than rigid rules, AI lets us dynamically model normal user activity baselines. It can then rapidly detect anomalies indicative of insider threats, compromised accounts, data exfiltration attempts, and more - all in real-time. User Behavior Analytics: I've deployed AI to prioritize vulnerabilities based on their risk and exploit potential, automating the triaging process. AI can also correlate vulnerabilities with active threats, showing us which exposures need patching first. Automated Vulnerability Management: Modern AI-powered SIEM solutions can correlate security events across networks, endpoints, and other sources to rapidly identify genuine incidents amidst the noise. AI also streamlines response workflows based on alert contexts. Smart SIEM & Incident Response: Across incident triage, alert analysis, threat hunting, and more, AI is automating repetitive, labor-intensive security tasks - amplifying the impact of human personnel while reducing burnout and fatigue. Security Ops Automation: I could go on, but you get the idea. As an AI expert and entrepreneur, I've seen these capabilities utterly transform cybersecurity operations into an amazingly proactive, autonomous counter-threat force. And AI's offensive cyber capabilities are still rapidly evolving. The Way Forward: Practical AI Cyber Defense But when I gaze into the future of AI-powered cybersecurity, I see incredible potential beyond just better threat detection and response. Used wisely, I believe AI can fundamentally reshape and elevate our entire profession into a truly proactive, predictive security posture - finally outpacing even the wiliest cyber adversaries. However, realizing this AI-driven future won't happen automatically. It requires us as an industry to thoughtfully and securely integrate AI across our people, processes, and technology stacks with a keen focus on augmenting human expertise and capabilities rather than wholesale automation. At an operational level, this will mean embracing increasingly autonomous security workflows offloaded to AI systems - handling tedious triaging and basic response while escalating only the most critical issues to skilled analysts for analysis and key decisions. It's a pragmatic balance of human and machine capabilities. We'll also need a new class of cybersecurity professionals fluent in interpreting AI system outputs and translating them into strategic guidance. These AI-savvy analysts must be adept at not just using but explaining and justifying AI system behaviors and decisions to leadership stakeholders. It plays a crucial role in building organizational trust and accountability around AI systems. Additionally, organizations must adopt secure AI system development lifecycles, proactively battling adversarial AI threats through approaches like robust model training, adversarial testing, privacy-preserving analytics, and defensive AI architectures. We simply can't leave our AI systems vulnerable to adversarial subversion. Thankfully, I see positive momentum towards these goals across the AI security startup ecosystem and cybersecurity community. But we must keep pressing forward holistically - investing in skilling our people, refining our processes, and pioneering new AI-centric technologies securely and ethically. Only then can we responsibly wield AI's incredible cyber defense potential without enabling worse nightmares. Despite all the work ahead, I'm hugely optimistic as an AI expert and founder. I've already deployed these transformative AI capabilities in the field, experiencing their game-changing impact firsthand. AI gives us a truly decisive advantage in combating even the most sophisticated threat actors - if we successfully fuse machine intelligence and human expertise. It won't be easy, but I'm convinced the payoff will be a safer, more secure digital world actively hardening itself against cyber threats in real-time. A world where novel attacks are neutralized before impact, where vulnerabilities are patched before exploitation, and where malicious insiders and compromised accounts are stopped in their tracks. Best of all, AI cyber defense done right should free up our human personnel for higher-impact strategic work, not tedious data grunt work. AI should empower us to be the cybersecurity experts, creative problem-solvers, and strategic leaders we always aspired to be - not menial attack analysts. So to my fellow AI professionals, founders, and innovators - the path ahead is clear. Let's blaze this trail TOGETHER, pioneering the new era of practical, intelligent, and ethical AI-powered cyber defense. The future of digital safety depends on us doing this right. And with our unique combination of human ingenuity and machine intelligence, I'm confident we're up to the immense challenge. Let's get to work.

Applications of Artificial Intelligence in Cybersecurity: Boosting Threat Defense System

As , we are always striving for high performance and efficiency in our code. Whether it’s optimizing algorithms or fine-tuning data structures, every decision we make can have a significant impact on the overall performance of our applications. One powerful way that can help us accurately measure the performance of our code is a process called benchmarking and we’ll look at how to use BenchmarkDotNet with our C# code. software engineers In this article, we’ll explore the ins and outs of using BenchmarkDotNet to benchmark C# code efficiently. We’ll be starting with the setup and I’ll guide you through examples of how to write benchmarks and run them as well. By the end, you’ll be able to write and run benchmarks on your C# code using BenchmarkDotNet effectively. Let’s jump into it! 1 – Installing and Setting Up BenchmarkDotNet The first step is to in your project. You can do this by opening the NuGet Package Manager in Visual Studio and searching for “BenchmarkDotNet.” Select the latest version and click on the Install button to add it to your project. There are no other external dependencies or anything fancy that you need to do to get going. install the BenchmarkDotNet package As you’ll see in the following sections, it’s purely code configuration after the NuGet package is installed! It’s worth mentioning that BenchmarkDotNet automatically performs optimizations during benchmarking C# code, such as JIT warm-up and running benchmarks in a random order, to provide accurate and unbiased results. But this all happens without you having to do anything special. Keep in mind, that just like with writing tests, you’ll very likely want to have your benchmark code in a dedicated project. This will allow you to release your core code separately from your benchmarks. There are probably not a lot of great reasons to deploy your benchmark code with your service or ship your benchmark code to your customers. 2 – Getting Started With Benchmark Methods in BenchmarkDotNet When it comes to writing benchmark methods using the BenchmarkDotNet framework, there are a few things we need to configure properly. In this section, I’ll guide you on how to write effective benchmark methods that accurately measure the performance of your C# code. This video on getting started with BenchmarkDotNet is a helpful guide as well: https://youtu.be/Wa3sdKGp3wE?embedable=true Structure Your Benchmark Code It’s important to structure your benchmark code properly to ensure accurate and efficient benchmarks. Follow these guidelines: : Start by creating a dedicated class for your benchmarks. This keeps your benchmark code organized and separate from the rest of your application code. Create a separate class for benchmarks : Select the appropriate type of benchmark job you’d like to run by marking your benchmark class with one of the following attributes , , , or . Apply the correct [XXXRunJob] attribute [ShortRunJob] [MediumRunJob] [LongRunJob] [VeryLongRunJob] attribute: To enable memory measurements during the benchmark, apply the attribute to your benchmark class. This attribute allows you to gather memory-related information along with execution time. If you’re strictly concerned about runtime and not memory, you can omit this. Apply the [MemoryDiagnoser] [MemoryDiagnoser] Note that your class cannot be sealed. I’d recommend just sticking to a standard definition with the appropriate attributes from above. You can have your benchmark class inherit from something else though, so if you find there’s a benefit to using some inheritance here for reusability, that might be a viable option. public class Writing Benchmark Methods Benchmark methods are where you define the code that you want to measure. Here are some tips for writing benchmark methods using BenchmarkDotNet: Apply the attribute: Each benchmark method needs the attribute. This attribute tells BenchmarkDotNet that this method should be treated as a benchmark. [Benchmark] [Benchmark] Avoid setup costs in benchmark methods: Benchmark methods should focus on measuring the performance of the code itself, not the cost of initializing your benchmark scenario. Avoid allocations and overuse of memory: Benchmark methods should not be concerned with the overhead caused by memory allocations. Minimize allocations and reduce memory usage within your benchmark methods to get accurate performance measurements. Note that BenchmarkDotNet handles all of the warmup for you – You don’t need to go out of your way to manually code in doing iterations ahead of time to get things to a steady state before benchmarking C# code. Example Benchmark Method: Here’s an example of a benchmark method in BenchmarkDotNet: [Benchmark]
public void SimpleMethodBenchmark()
{
    for (int i = 0; i < 1000; i++)
    {
        // Execute the code to be measured
        MyClass.SimpleMethod();
    }
} In this example, the attribute is applied to the method, indicating that it should be treated as a benchmark. Suppose we were using an instance method instead of a static method as illustrated. In that case, we’d want to instantiate the class OUTSIDE of the benchmark method — especially if we need to create, configure, and pass in dependencies. Minimize (read: eliminate) the amount of work done in the method that isn’t what you are trying to benchmark. [Benchmark] SimpleMethodBenchmark Remember, if you’re interested in memory in addition to the runtime characteristics, make sure to apply the attribute to the benchmark class — not the method that is the benchmark. [MemoryDiagnoser] 3 – Running Benchmarks with BenchmarkDotNet When it comes to running benchmarks using BenchmarkDotNet in C#, there are a few important considerations to keep in mind. In this section, I’ll explain how to run benchmarks, discuss different scenarios and options, and provide code examples to help you get started. You can follow along with this video on how to run your BenchmarkDotNet benchmarks as well: https://youtu.be/Hqeq9ycqteQ?embedable=true Running Benchmarks With BenchmarkRunner The BenchmarkRunner class provides a very simple mechanism for running all of the Benchmarks that you provide it from a type, list of types, or an assembly. The benefit of this is in the simplicity as you can just run the executable and it will go immediately run all of the benchmarks that you’ve configured the code to run. You can or below: check out some example code on GitHub using BenchmarkDotNet.Running;

var assembly = typeof(Benchmarking.BenchmarkDotNet.BenchmarkBaseClass.Benchmarks).Assembly;

BenchmarkRunner.Run(
    assembly,
    args: args); In the code above, we are simply specifying an assembly for one of our benchmarks. However, you could use or find other ways to list the types you’re interested in. Assembly.GetExecutingAssembly Running Benchmarks With BenchmarkSwitcher The BenchmarkSwitcher class is very similar – but the behavior is different in that you can filter which benchmarks you’d like to run. There are some slight API differences in that the BenchmarkRunner allows you to specify a collection of assemblies where the BenchmarkRunner (at the time of writing) does not. Here’s a code example, or you can : check out the GitHub page using BenchmarkDotNet.Running;

var assembly = typeof(Benchmarking.BenchmarkDotNet.BenchmarkBaseClass.Benchmarks).Assembly;


BenchmarkSwitcher
    // used to load all benchmarks from an assembly
    .FromAssembly(assembly)
    // OR... if there are multiple assemblies,
    // you can use this instead
    //.FromAssemblies
    // OR... if you'd rather specify the benchmark
    // types directly, you can use this
    ///.FromTypes(new[]
    ///{
    ///    typeof(MyBenchmark1),
    ///    typeof(MyBenchmark2),
    ///})
    .Run(args); As you will note in the code comments above, there are several ways that we can call the BenchmarkSwitcher. The primary difference between these two methods is that the switcher will allow you to provide user input or filter on the commandline. 4 – Customizing Benchmark Execution BenchmarkDotNet provides various options to customize the execution of your benchmarks, allowing you to fine-tune the benchmarking process according to your needs. Two important options to consider are the iteration count and warm-up iterations. Configuring Parameters for Benchmarks If we want to have variety in our benchmark runs, we can use the attribute on a public field. This is akin to using the xUnit Theory for parameterized tests, if you’re familiar with that. For each field you have marked with this attribute, you’ll essentially be building a matrix of benchmark scenarios to go run. [Params] Let’s check out some example code: [MemoryDiagnoser]
[ShortRunJob]
public class OurBenchmarks
{
    List<int>? _list;

    [Params(1_000, 10_000, 100_000, 1_000_000)]
    public int ListSize;

    [GlobalSetup]
    public void Setup()
    {
        _list = new List<int>();
        for (int i = 0; i < ListSize; i++)
        {
            _list.Add(i);
        }
    }

    [Benchmark]
    public void OurBenchmark()
    {
        _list!.Sort();
    }
} In the code above, we have a field which is marked with . This means that in our method, we’re able to get a new value for each variation of the benchmarking matrix we want to run. In this case, since there’s only one parameter, there will only be a benchmark for each value of ListSize — so 4 different benchmarks based on the 4 different values specified. ListSize [Params] [GlobalSetup] Adjusting Per Benchmark Iteration Count BenchmarkDotNet allows you to control the number of method. By default, each benchmark is executed a reasonable number of times to obtain reliable measurements. However, you can adjust the iteration count by applying the attribute to individual benchmark methods, specifying the desired number of iterations. iterations for each benchmark [IterationCount] [Benchmark]
[IterationCount(10)] // Custom iteration count
public void MyBenchmarkMethod()
{
    // Benchmark code here
} I should note that in practice, I have not personally had to configure the iteration count manually. Custom Per Benchmark Warm-up Iterations Warm-up iterations are executed before the actual benchmark starts, allowing the JIT compiler and CPU caches to warm up. This helps eliminate any performance inconsistencies caused by the initial compilation of the benchmark method. You can customize the number of warm-up iterations using the attribute. [WarmupCount] [Benchmark]
[WarmupCount(5)] // Custom warm-up count
public void MyBenchmarkMethod()
{
    // Benchmark code here
} Like the iteration count, I have also not had an issue with the default warmup. I’d advise that you leave these as defaults unless you know what you’re doing to tune these accordingly. 5: Analyzing and Interpreting Benchmark Results from BenchmarkDotNet Benchmarking is an important part of the software development process when it comes to optimizing performance. After running benchmarks using BenchmarkDotNet, it’s important to be able to accurately analyze and interpret the results. In this section, I’ll guide you through the process of analyzing benchmark results and identifying potential performance improvements. Establishing a Baseline Benchmark in BenchmarkDotNet When analyzing benchmark results, several key metrics can provide valuable insights into the performance of your code. But what can be challenging is understanding what you’re trying to compare against. If you simply have two implementations you are trying to compare against, it may not feel that challenging. However, when you’re comparing many you likely want to establish a baseline to see your improvements (or regressions). Here’s how we can make a benchmark method as a “baseline”: [Benchmark(Baseline = true)]
public void Baseline()
{
    // Code under benchmark
} Interpreting Benchmark Results Once you have the benchmark results and understand the key metrics, it’s important to interpret the findings to identify potential performance improvements. Here are a few examples of how to interpret the benchmark results: Identifying Bottlenecks: Look for outliers or significantly higher execution times in certain benchmarks. This could indicate potential bottlenecks in your code that need to be optimized. Comparing Performance: Compare the mean or median execution times of different benchmark methods or different versions of your code. This can help you identify which approach or version is more efficient. Spotting Variability: Pay attention to the standard deviation and percentiles to identify any significant variability in the benchmark results. This can help you identify areas where performance optimizations could make a difference. I find that I am spending most of my time looking at the median/mean to see what stands out as faster and slower. If you need to dive deeper into the statistical analysis and you just need a primer, I have found even . Wikipedia provided a reasonable starting point 6: Optimizing C# Code with BenchmarkDotNet As software engineers, we constantly strive to write code that is not only functional but also performs efficiently. BenchmarkDotNet is a powerful tool that helps us measure and compare the performance of our code. By analyzing the results provided by BenchmarkDotNet, we can identify areas where our code may be underperforming and optimize it to achieve better results. In this section, I’ll share some techniques for optimizing C# code based on the results we’d gather from running our benchmarks. You can check out this video on measuring and tuning iterator performance for a practical example: https://youtu.be/G2-d7kZFlRA?embedable=true Identifying Performance Bottlenecks One of the first steps in optimizing C# code is identifying the performance bottlenecks. BenchmarkDotNet allows us to measure the execution time of our code and compare different implementations. By analyzing the benchmark results, we can pinpoint the areas that are taking up the most time. Let’s consider an example where we have a loop that performs a computation on a large array. We can use BenchmarkDotNet to measure the execution time of different implementations of this loop and identify any potential bottlenecks. [ShortRunJob]
public class ArrayComputation
{
    private readonly int[] array = new int[1000000];

    [GlobalSetup]
    public void Setup()
    {
        // TODO: decide how you want to fill the array :)
    }

    [Benchmark]
    public void LoopWithMultipleOperations()
    {
        for (int i = 0; i < array.Length; i++)
        {
            array[i] += 1;
            array[i] *= 2;
            array[i] -= 1;
        }
    }

    [Benchmark]
    public void LoopWithSingleOperation()
    {
        for (int i = 0; i < array.Length; i++)
        {
            array[i] = (array[i] + 1) * 2 - 1;
        }
    }
} In this example, we have two benchmark methods and . The first method performs multiple operations on each element of the array, while the second method combines the operations into a single computation. By comparing the execution times of these two methods using BenchmarkDotNet, we can determine which implementation is more efficient. LoopWithMultipleOperations LoopWithSingleOperation Recall from the earlier sections to we could parameterize this for different sizes! Sometimes this is necessary to see if we have different behaviors under different scenarios, so it’s worth exploring beyond just the surface. Optimizing Loops and Reducing Memory Allocations Loops are often an area where we can optimize our code for better performance. Inefficient loops can result in unnecessary memory allocations or redundant computations. BenchmarkDotNet can help us identify such issues and guide us in optimizing our code. Consider the following example where we have a loop that concatenates strings, and we’re making sure to use a as well: [MemoryDiagnoser] [MemoryDiagnoser]
[ShortRunJob]
public class StringConcatenation
{
    private readonly string[] strings = new string[1000];

    [GlobalSetup]
    public void Setup()
    {
        // TODO: decide how you want to fill the array :)
    }

    [Benchmark]
    public string ConcatenateStrings()
    {
        string result = "";
        for (int i = 0; i < strings.Length; i++)
        {
            result += strings[i];
        }

        return result;
    }

    [Benchmark]
    public string StringBuilderConcatenation()
    {
        StringBuilder stringBuilder = new StringBuilder();
        for (int i = 0; i < strings.Length; i++)
        {
            stringBuilder.Append(strings[i]);
        }

        return stringBuilder.ToString();
    }
} In this example, we have two benchmark methods: and . The first method uses string concatenation inside a loop, which can result in frequent memory allocations and poor performance. The second method uses a to efficiently concatenate the strings. By comparing the execution times of these two methods using BenchmarkDotNet, we can observe the performance difference and validate the effectiveness of using a for string concatenation. ConcatenateStrings StringBuilderConcatenation StringBuilder StringBuilder Now You Know How to Use BenchmarkDotNet! BenchmarkDotNet is a valuable tool for accurately and efficiently benchmarking C# code. Throughout this article, we explored tips for using BenchmarkDotNet effectively. By leveraging these, you can accurately measure and optimize the performance of your C# code. Improved performance can lead to more efficient applications, better user experiences, and overall enhanced software quality! Remember, benchmarking is an iterative process, and there are other resources and tools available that can further assist you in optimizing your C# code. Consider exploring profiling tools, performance counters, and other performance analysis techniques to gain deeper insights into your application’s performance. If you found this useful and you’re looking for more learning opportunities, consider and check out my ! subscribing to my free weekly software engineering newsletter free videos on YouTube Also published . here

6 Simple Performance-Boosting Tips to Get Started With BenchmarkDotNet

While the United States continued its reactionary posture toward the crypto space throughout 2023, last year saw many countries take more hardline approaches toward industry oversight and enforcement. The European Union formally its long-awaited Markets in Crypto-Assets legislation, or MiCA, which set in motion the implementation of a legal framework for key dimensions of the crypto ecosystem. Much like the workplace safety standards that helped correct cruelties of the , MiCA’s standardized language establishes ecosystem conduct with the aim of increasing stability and participant trust. passed Industrial Revolution In similar moves, Hong Kong, South Korea, and Singapore all passed sweeping policies to define how cryptocurrency companies can conduct business in their regions. In Hong Kong, a went live in June 2023 that imposed financial and regulatory restrictions on exchanges, while greenlighting a limited menu of services for retail customers. While some critics bemoaned the high application costs as a for aspiring companies, the new process would officially legalize basic retail trading. In turn, greater due diligence was implemented for builders, business, and tokens alike, requiring background checks and clean bills of financial health to qualify. Prior to this decision, Hong Kong-based participants relied on unregulated platforms to access the digital economy, often at their own . new licensing regime barrier to entry peril Also in June, South Korea accelerated its oversight of the crypto space by passing that combined 19 pending bills aimed at ramping up participant protections. The Act on the Protection of Virtual Asset Users tightened restrictions for virtual asset service providers’ (VASPs) handling of customer funds and information and mandated oversight through regular reporting. Rules for “self-issued virtual assets” appear in response to the shell game accounting practices that caused countless participants hardship. Perhaps more importantly, the law confers power on the Financial Services Commission (FSC) to enforce these regulations through imprisonment, sanctions, and fines for bad actors. After successful passage, these policy changes are now set to take effect in July 2024. omnibus legislation Later in August, the Monetary Authority of Singapore (MAS) announced of its legal framework for stablecoins, and how they can interact with the traditional economy. In addition to clearly defining stablecoins, the body outlined its regulatory reach as pertaining to “single-currency stablecoins (SCS) pegged to the Singapore Dollar or any G10 currency, that are issued in Singapore.” This allowed the MAS to set for SCS transactions and redemption that placed the onus of compliance on token issuers while establishing clear mechanisms for enforcement. By dispelling another legal gray area, retail and institutional participants alike can transact with greater certainty and peace of mind after . a finalized version broader rules high-profile collapses Back in London, the Financial Conduct Authority (FCA) is cracking down on crypto companies in an effort to root out scams and potential fraud. In addition to protecting consumers by maintaining the integrity of financial markets, the FCA holds service providers accountable for adhering to regional standards. While the regulator imposed a host of requirements to ensure crypto companies are operating above-board, many of the latest policy updates have concerned messaging. Centered primarily around the of crypto products and services, the FCA’s intersectional guidelines propose a unique topographical environment that appears to disqualify the bulk of common industry practice. This includes the use of promotional incentives, affiliate program activities, and “refer a friend” bonuses, all of which must pass review by an . marketing approved third-party It’s amidst this climate that the U.S. opened the floodgates to Bitcoin ETFs, as outlined above, despite maintaining an overall standoffish posture toward the rest of the ecosystem. The institutional of SEC precedent, after causing uncertainty and litigation, now appears to be setting the stage for greater cross-pollination between digital assets and financial behemoths. Those playing close attention to the of asset managers and regulators leading up to the approval saw how raw financial will can influence the wheels of bureaucracy. However, it’s crucial to note that the embrace of specific financial instruments, in the case of Bitcoin ETFs, does little to resolve the absence of legal foundations necessary to adequately regulate the crypto ecosystem. flip-flopping choreography From a bird’s eye view, this realignment toward greater regulatory policy is a dominant, worldwide trend. According to a by TRM Labs examining “21 jurisdictions representing approximately 70% of global crypto exposure,” governments are beginning to turn a critical eye toward the digital economy. The crypto analytics firm discovered that “[i]n 2023, 80% of these jurisdictions moved to tighten crypto regulations, and almost half specifically progressed consumer protection measures.” If these percentages denoted the chance of precipitation, it would be foolish to not carry an umbrella. In light of such a seismic change in climate, industry leaders are being left with little choice but to start dressing for the weather. Looking ahead in 2024, TRM Labs drew up key dates and landmark decisions to illustrate how the regulatory landscape will continue to evolve throughout the year. While an element of uncertainty remains in the air due numerous election cycles, it should be clear that crypto is becoming more regulated across major jurisdictions. report While this timeline highlights the electoral vulnerabilities that could serve to undermine efforts toward regulation, industry leaders must embrace and support these measures. To that end, it’s also crucial to question those who see any interference by the government as a detriment to innovation. There’s a growing movement in tech circles for that encourage developers to reach further in their renderings of products and systems. This attitude will be essential going forward as crypto platforms decide how to operate in the increasingly defined world of digital asset regulation. As more markets have clear rules and penalties in place, companies will be forced to tailor their services by region or comply with the steepest guidelines platform-wide. Barring a step backward to legal gray zones and black swan events that could (and should) have been prevented, these are the paths forward for ethical crypto providers. generative regulations Better yet, these provisions should be celebrated as concrete solutions to the rampant mismanagement of customer funds and outright fraud that’s occurred in recent years. Most jurisdictions currently exploring heightened regulations have experienced the implosion of a homespun crypto hero, and are not in any rush to re-live such trauma. That’s why many of these legislative packages set their sights on common sense measures that demarcate official oversight and lean heavily on consumer protections. It’s at this crossroads that crypto leaders could choose to send a strong message by working with officials to ensure a smooth transition. Rather than just asserting user-centric values in mission statements, the industry should embrace and be receptive to regulation if it means protecting and growing the global crypto community. As we’ve seen, words can only go so far to remedy actions. Therefore, leaders must refrain from pushing empty platitudes, and instead focus on concrete ways to boost participant confidence. These changes are inevitable, and the crypto ecosystem would be wise to anticipate and work to accommodate their implementation. Undoubtedly, there will be some that choose to leapfrog expectations, forever seeking the next regulatory lillipad amenable to their unique situation. Where fringe services may continue to operate outside official oversight, retail exchange is gaining legal architecture at a rate that will outpace even the nimblest of entities. With each legislative victory portending greater control in an official capacity, eventually, there’ll be nowhere left to hop. But then again, it’s a tough sell to lobby for customer trust after fleeing regulation to the point of running out of road. To avoid such a humbling and exhaustive process, companies should strive to ensure their services meet all jurisdictional requirements, or risk the consequences.

In 2024, Crypto Regulation is Key to Boosting Participant Confidence

In a rapidly evolving world towards sustainable and environmentally conscious solutions, the (EV) industry has emerged as a formidable force, promising a cleaner, greener future. At the heart of this global shift stands India, the fastest-growing economy in the world with immense potential to lead the electric mobility revolution. As a result, the demand for electric battery packs has increased substantially. electric vehicle According to a by , a Bengaluru-based corporate and commercial law firm, and , a seed-stage VC firm, India has huge potential for in manufacturing EV battery packs as the country is heavily dependent on importing 60-65% of the total component requirement for battery packs. report GameChanger Law Advisors Speciale Invest The Indian EV battery market size is expected to grow from USD 16.77 billion in 2023 to USD 27.70 billion by 2028, at a CAGR of 10.56% between 2023 and 2028, indicating that India’s EV battery industry landscape is at the cusp of a remarkable expansion. While a majority of EV Original Equipment Manufacturers (OEMs) have developed assembly facilities in-house by using imported lithium-ion cells or purchasing imported battery packs, VC-funded startups such as Ather, Ola Electric, Cosbike, and Pure EV continue to supply their battery packs to most two and three-wheeler OEMs, and this is a reflection of recent market trends. battery pack Recently, , an EV charging station aggregation platform, installed smart EV charging stations at World Trade Centre, Bengaluru. Located at the ground floor of the multi-level car parking, the dedicated area comprises four fast charging stations for cars and two-wheelers, and they are easily accessible for employees and visitors alike. Charzer The company also introduced introduced a new Whatsapp feature in October to offer hassle free charging for EV users. This new feature aims to make EV charging more accessible and user-friendly, particularly for individuals who may find mobile applications challenging to operate. Current Indian EV Market Trends The Indian EV market was valued at $.3.21 billion in 2022 and is expected to grow to $113.99 billion by 2029 at a compound annual growth rate of 66.52%. In 2022, 25% of EVs sold in India were purchased by fleet operators, including for taxis. In recent years, EV sales in India have significantly increased because it is a cleaner and more efficient alternative to gasoline-powered vehicles. With the growth of sales of EVs in India, the Indian government has set a target to achieve 30% electrification of India’s vehicle fleet by 2030. The predicts that India’s domestic EV market will witness a 49% compound annual growth rate between 2022 and 2030, and the number of EV units sold annually will touch 10 million by 2030. Economic Survey 2023 The Report outlines a set of suggestions for Indian policymakers on a host of concepts, including but not limited to Carbon Footprint Declaration (CFD), Battery Swapping Policy, Battery Passport, Second-Life Battery Policy and Safety Standards for Second-Life Batteries. Interestingly, the Report describes the Guidelines for Battery Safety Tests issued by the Ministry of Heavy Industries in November 2022 to enhance human safety and maps it with safety regulations prescribed in other jurisdictions. Charging Networks are Important for EV Adoption A Juniper Research found that fragmentation in charging networks is restricting EV adoption. study The fact that chargers have a concentration in urban areas is leading to widespread range anxiety among potential drivers. This is coupled with the difficulty of accessing charging points via different apps and cards, as well as the lack of standards for charging vehicles at the same rate. As such, EV charging networks must simplify access and work with local authorities to roll out chargers to a wider range of locations, or the EV market will struggle to accelerate. Also, recently, have made a groundbreaking discovery that could enhance the safety and efficiency of lithium-metal batteries. By preventing corrosion during the deposition of lithium, the researchers found that lithium atoms form a unique 12-sided shape, reducing the risk of explosions. This innovation could potentially revolutionize lithium battery technology, leading to improved safety and performance. innovations for UCLA researchers Rechargeable lithium-ion batteries power smartphones, electric vehicles, and storage for solar and wind energy, among other technologies. Automakers have touted plans to when they lose power, but competition for battery packs and cell materials, and the appetite for affordable cars cast doubt on this part of the circular economy. An array of startups offers second-life energy storage using old EV batteries. re-use electric vehicle batteries This article was originally published by on Navanwita Bora Sachdev The Tech Panda.

The Crucial Role of Batteries in Boosting the Growth of Electric Vehicles in India

is a seasoned entrepreneur and fintech specialist and is here to shed light on the technicalities surrounding international bank transfers and their impact on businesses, especially SMEs. Iurii Gugnin With over a decade and a half of experience in the field, Gugnin is currently spearheading the growth of Evita, an American that is revolutionizing the way companies handle international transfers. startup In a short span of six months, Evita has achieved a remarkable turnover of $15 million per month, proving its profitability and effectiveness. The startup is instrumental in facilitating overseas purchases of real estate, automobiles, and equipment for foreign clients while also aiding American small and medium-sized businesses in importing goods from abroad at a fraction of the cost. Unlike traditional banking institutions such as Bank of America and Chase, which levy a 3-5% charge on transfers, Evita operates on a minimal commission of 0.4-1.5%. This approach not only fuels the growth of the US economy but also aids in re-establishing the country's dominance in . global exports Join us as Iurii Gugnin delves into the challenges faced by the U.S. in expanding its exports, the reasons behind the high costs of international wire transfers, and potential solutions to these issues. Q: With a wealth of experience in fintech since 2009, could you share some insights about your journey with international payments? A: My journey with international payments has been quite diverse, encompassing roles as an entrepreneur, consultant, and individual user. I've had the privilege of traveling to over 100 countries, living in several for extended periods, and consistently encountered a common issue: the incompatibility of bank cards when moving to a new location. For instance, a Thai card was unusable in the States, and a Dubai card was ineffective in the UK. Despite Visa and MasterCard's global presence, their usability is conditional. In my entrepreneurial journey, I've faced similar challenges. I established an international business, secured licenses in Europe and the CIS countries, and am currently in the process of obtaining licenses in America. One of my notable ventures is the Karma investment platform, which has successfully built a community of over 8,000 investors, attracted more than 500 business borrowers, and structured a $250m deal with a major bank. However, the process of making international transfers with currency exchange was often marred by high fees and lengthy waiting periods. In addition to this, I've advised five countries on digital asset legalization, engaging with central banks to discuss payment structuring and the integration of digital money to enhance transfer transparency. Drawing from these experiences, I founded Evita, a US-based company, in 2021. After a year of product-market fit exploration and several pivots, we found our niche in making international money transfers faster and cheaper, leading to rapid growth. Q: The Federal Reserve reports that the average cost of money transfers from the States to other countries is now about 5.6%, even higher than in 2022. Why are these fees so high? A: The high fees can be attributed to what I call the 'imperial syndrome' prevalent in the States. There's a common belief that dollars are universally accepted, negating the need for other currencies. Mainstream banks like Wells Fargo and Chase primarily operate in dollars in the United States, and the SWIFT system also predominantly uses USD. Growing up in Asia, I was accustomed to carrying different currencies and exchanging them with ease, a practice not common in the US. Despite my knowledge and net worth, it took me three months to open a EUR account in the US, and even then, there were significant usage restrictions. This is a challenge that many local entrepreneurs face. There is a niche segment of VIP banks, such as HSBC, Morgan Stanley, and JP Morgan, where corporations or wealthy individuals can open accounts in different currencies. These banks offer competitive exchange rates and reasonable transfer fees, but they are not designed for the mass market. In essence, there's an oligopoly of 3-4 leaders in the US banking sector who set their own prices. They've decided to charge 3-5% for international transfers, and most customers, unaware of how exchange rates are determined or where to find current information, have no choice but to comply. Q: In many Asian countries, the transfer fee is significantly lower than in the United States. For instance, in South Korea, it's only 3.55%. Can you shed some light on why this is the case? A: Asia, with its multitude of island nations, has a unique culture of multi-currency transactions. However, even a seemingly small fee of 3.5% can add up quickly. Consider purchasing a Mercedes-Benz G-Class abroad for 300,000 EUR. The commission alone would amount to a staggering 10,500 EUR. There are companies like Revolut or Wise that cater to smaller transactions, such as a $1,000 payment for freelance services. In these cases, an additional percentage doesn't seem as daunting. But in the B2B sector, the stakes are much higher. Transactions often involve tens or even hundreds of thousands for equipment, material supplies, and furniture. On our platform, we've even facilitated the purchase of a thoroughbred horse from Germany with an impressive pedigree. The price tag was 120,000 EUR! In such scenarios, even half a percent can make a significant difference. Q: The process of cross-border transfers often results in high costs and extended durations. Would you provide some insight into the underlying reasons for this? A: Certainly, let's consider a hypothetical scenario. Imagine two individuals who wish to open dollar accounts in a bank in Mongolia and another in Korea. The question arises, how can this be achieved? The challenge lies in the fact that to handle non-cash dollars, both banks require a license from the United States. Acquiring such a license is a daunting task for foreign financial institutions. As a workaround, these Mongolian and Korean banks would typically open correspondent accounts in U.S. banks, say Wells Fargo and Citi, respectively. Within these accounts, virtual folders are created for specific users. Now, let's assume a client from Mongolia wishes to send dollars to a friend in Korea via SWIFT. As I mentioned earlier, the currency is held in American banks. The initiating bank, Wells Fargo, would first verify the sender and recipient of the transfer. Following this, it would execute the transfer within the U.S., to Citi. The receiving bank, Citi, would then confirm that the Korean bank has an account with them and that the intended recipient is indeed a client of theirs. This process of verification and confirmation takes time, and during this period, transaction fees accumulate. Hence, the perceived delay and cost associated with cross-border transfers. Q: So, if I understand correctly, Mongolian and Korean banks might not directly interact with US dollars, but rather through an intermediary. Does this additional layer extend the transfer time? A: Yes, this is a common practice in many countries. A local bank might open an account with a Korean bank, which in turn has an account with a US bank. Consequently, the money passes through several "hands" of correspondent banks, each taking a slice of the transaction as commission. Moreover, when funds are sent through this complex chain, the transaction details can be extensive, often filling an entire page of text. This complexity significantly increases the likelihood of errors, which further extends the transfer time. But that's not all. Customers have come to expect SWIFT transfers to be a lengthy process. For instance, when money is sent from the US to Latin America, the CIS, or Asia, local banks might reflect the transfer in the customer's account a day or even a week after the funds have arrived. During this period, the banks essentially use the customer's money for free, earning interest on the deposits. Q: Would it be accurate to say the complexities associated with international transfers contribute significantly to the U.S. losing its leadership in global exports? A: I believe it's a crucial factor. The U.S.'s reliance on the dollar has had unintended consequences. In 2012, the U.S. was surpassed by China in terms of global exports, and by 2022, the Chinese yuan had become in transaction volume. one of the top five currencies While the dollar still leads, the use of the yuan from 2019 to 2022. nearly doubled Despite the slowdown in China's economy due to low birth rates, it continues to progress, driven by momentum. Furthermore, China frequently devalues its currency, which benefits the country as expenses are nullified while income remains in dollars. Consequently, Chinese goods remain competitive in the global market. In contrast, the U.S. faces high costs in various sectors, including real estate, transportation, internet, and food. These costs inflate the price of goods, making them less attractive overseas. Unique products like Boeing airplanes or Nvidia chips are exceptions. Additionally, businesses bear the burden of a 5%+ commission for payment transfers and interest rates on loans. Q: What solution does Evita offer? A: Fast and cheap international B2B transfers. We have a large community of exporters and importers who need to regularly transfer large sums to other countries. We also have our accounts in the States, in the UK, in the EU, and soon to be in China. This allows us to offset counter-flows of money. Q: Could you provide some insight into how the counterflow system operates? A: Imagine a scenario where one company needs to transfer $1 million from the US to Europe today, and another company needs to send the same amount from Europe to the US. In this abstract situation, we at Evita can accept the dollars into our American account and simultaneously transfer euros from our local European account and vice versa. This makes all transactions swift as they are conducted within the US and EU respectively. Moreover, we save on currency conversion because we don't convert the currencies. Our counterflow system is supported by a community of importers, exporters, logistics operators, and entrepreneurs from various countries. My extensive experience in fintech has been instrumental in bringing this community together. Large banks also have systems that allow for such internal clearing, which significantly increases the profitability of transactions. However, clients are often not informed about this and are charged hefty commissions for transfers. In reality, banks do not change the currency but simply adjust the figures on the accounts. At Evita, we choose not to keep all the profit for ourselves. Instead, we share a portion of it with our clients in the form of a discount. As a result, the user will pay a commission of only 0.4-1.5%, depending on the transaction volume, instead of the usual 5% or even 3%. Essentially, we democratize access to clearing, which is currently monopolized by banks, and save up to 90% of money on currency exchange. According to the , U.S. small businesses export goods and services worth $341 billion annually. We estimate that products like Evita could save Americans between $7-16 billion a year. SBA Q: What happens if the counterflows don't align? A: In such cases, we resort to transferring money through digital wallets using stablecoins. We maintain legal and institutional accounts on regulated exchanges like Kraken. Cryptocurrency transfers offer a faster and more cost-effective alternative to traditional bank wires, taking just a minute instead of two days and costing 50 cents instead of $45. Q: Your primary focus is on small businesses, correct? A: Our focus is on small and medium-sized businesses. However, it's important to note that these businesses often engage in quite large transactions. While we don't cater to corporations on the scale of Apple, our clients are primarily involved in buying and selling cars. Currently, we're exploring the luxury furniture niche. Our services are also frequently used for payments related to equipment and charter flights, which can cost up to $200,000. Q: Is it accurate to say Evita also assists businesses in addressing their working capital issues? A: That's correct. It's a common scenario where a business has paid for a product and is simply waiting for its delivery. Many suppliers only ship products after receiving payment. If the transfer is made via SWIFT, the customer may have to wait an additional two weeks. During this period, the money is essentially "blocked" in the system. However, if the funds are transferred within a day, businesses can turn over their working capital more frequently, leading to increased earnings at the end of the year. This is often more significant than the profitability of a single transaction. We aim to shorten the cycle and increase the number of "turnovers.” Moreover, businesses can avoid taking out loans and paying high-interest rates. Currently, the interest rate on a small business loan . Not all businesses can secure these loans from banks due to stringent requirements, leading to the emergence of loan sharks who offer loans at annual rates of 100-200%. ranges from 6% to 45% By freeing up working capital, Evita could potentially save US exporters around $20 billion a year at the very least. Given that in the U.S., it's crucial to help them save money. small businesses are the backbone of job creation Q: Rounding off the interview, what do you think the future holds for the international payments market? A: The future of international payments, in my view, may present more challenges. Despite the technological capability for instant and affordable global transfers existing for over two decades, two major obstacles persist. Secondly, there is a global trend of increasing governmental control over business operations. European banks, for instance, annually on Anti-Money Laundering (AML) and related activities. These costs are inevitably passed on to consumers through service fees. spend an average of 14.25 million euros Collectively, Europeans per year on AML measures. However, these efforts only uncover less than 1% of actual criminal activity, while 99.99% of individuals and businesses bear the brunt of compliance requirements. spend 140 billion euros As regulatory measures continue to intensify, domestic and ecosystem-specific money transfers will become the norm. Consequently, services like ours that facilitate these transactions are poised to gain popularity.

Interview With Iurii Gugnin: Boosting U.S. Export Competitiveness With Cheaper Remittances

Unlocking peak performance from your C++ code can be daunting, demanding meticulous profiling, intricate memory access adjustments, and cache optimization. Is there a trick to simplify this a bit?? Fortunately, there is a shortcut to achieving remarkable performance gains with minimal effort — provided you have the right insights and know what you’re doing. Enter compiler optimizations that can significantly elevate your code’s performance. Modern compilers serve as indispensable allies in this journey toward optimal performance, particularly in automatic parallelization. These sophisticated tools possess the prowess to scrutinize intricate code patterns, especially within loops, and execute optimizations seamlessly. This article aims to spotlight the potency of compiler optimizations, focusing on the — renowned for their popularity and widespread usage. Intel C++ compilers In this story, we unravel the layers of compiler magic that can transform your code into a high-performance masterpiece, requiring less manual intervention than you might think. What are compiler optimizations? | -On | Architecture targeted | Interprocedural Optimization | -fno-aliasing | Compiler Optimization reports Highlights: What are compiler optimizations? Compiler optimizations encompass various techniques and transformations a compiler applies to the source code during compilation. But why? To enhance the performance, efficiency, and, in some instances, the size of the resulting machine code. These optimizations are pivotal in influencing various aspects of code execution, including speed, memory usage, and energy consumption. Any compiler executes a series of steps for converting the high-level source code to the low-level machine code. These involve lexical analysis, syntax analysis, semantic analysis, intermediate code generation (or IR), optimization, and code generation. During the optimization phase, the compiler meticulously seeks ways to transform a program, aiming for a semantically equivalent output that utilizes fewer resources or executes more rapidly. Techniques employed in this process encompass but are not limited to . constant folding, loop optimization, function inlining, and dead code elimination I’m not going to discuss all the available options, but how we can instruct the compiler to do specific optimization that might improve code performance. So, the solution???? Compiler Flags. Developers can specify a set of compiler flags during the compilation process, a practice familiar to those using options like “ or with GCC for debugging and profiling information. As we go ahead, we’ll discuss similar compiler flags we can use while compiling our application with the Intel C++ compiler. These might help you improve your code’s efficiency and performance. -g” “-pg” So, what are we working with? I won’t delve into dry theory or inundate you with tedious documentation listing every compiler flag. Instead, let’s try to understand why and how these flags work. How do we accomplish this??? We’ll take an unoptimized C++ function responsible for calculating a iteration, and step by step, we’ll unravel the impact of each compiler flag. Along this exploration, we’ll measure the speedup by systematically comparing each iteration with the base version — starting with no optimization flags (-O0). Jacobi The speedups (or execution time) were measured on an machine. Here, the Jacobi method solves a 2D partial differential equation (Poisson equation) for modeling the heat distribution on a rectangular grid. Intel® Xeon® Platinum 8174 Processor u(x,y,t) is the temperature at point (x,y) at time t. We solve the stable state when the distribution isn’t changing anymore: A set of Dirichlet boundary conditions have been applied at the boundary. We essentially have a C++ coding performing the Jacobi iterations on grids of variable sizes (that we call resolutions). Basically, a grid size of 500 means solving a matrix of size 500x500, and so on. The function for performing one Jacobi iteration is as follows: /*
 * One Jacobi iteration step
 */
void jacobi(double *u, double *unew, unsigned sizex, unsigned sizey) {
 int i, j;

 for (j = 1; j < sizex - 1; j++) {
  for (i = 1; i < sizey - 1; i++) {
   unew[i * sizex + j] = 0.25 * (u[i * sizex + (j - 1)] +  // left
      u[i * sizex + (j + 1)] +  // right
      u[(i - 1) * sizex + j] +  // top
      u[(i + 1) * sizex + j]); // bottom
  }
 }

 for (j = 1; j < sizex - 1; j++) {
  for (i = 1; i < sizey - 1; i++) {
   u[i * sizex + j] = unew[i * sizex + j];
  }
 }
} We keep performing Jacobi iteration until the residual reaches a threshold value (inside a loop). The residual calculation and threshold evaluation are done outside this function and are not of concern here. So, let’s talk about the elephant in the room now! How does the base code perform? With no optimizations (-O0), we get the following results: Here, we measure the performance in terms of the MFLOP/s. This will be the basis of our comparison. MFLOP/s stands for “Million Floating Point Operations Per Second.” It is a unit of measurement used to quantify the performance of a computer or processor in terms of floating-point operations. Floating-point operations involve mathematical calculations with decimal or real numbers represented in a floating-point format. MFLOP/s is often used as a benchmark or performance metric, especially in scientific and engineering applications where complex mathematical calculations are prevalent. The higher the MFLOP/s value, the faster the system or processor performs floating-point operations. To provide a stable result, I run the executable 5 times for each resolution and take the average value of the MFLOP/s values. Note 1: It’s important to note that the default optimization on Intel C++ compiler is -O2. So, it is important to specify -O0 while compiling the source code. Note 2: Let’s go ahead and see how these run times will vary as we try different compiler flags! The most common ones: -O1,-O2,-O3 and -Ofast These are some of the most commonly used compiler flags when one begins with compiler optimizations. In an ideal case, the performance of . However, this doesn’t necessarily happen. The critical points of these options are as follows: Ofast > O3 > O2 > O1 > O0 -O1: Optimize for speed while avoiding code size increase. Goal: Suitable for applications with large code sizes, many branches, and where execution time isn’t dominated by code within loops. Key Features: -O2: Enhancements over -O1: Enables vectorization. Allows inlining of intrinsics and intra-file interprocedural optimization. -O3: Enhancements over -O2: Enables more aggressive loop transformations (Fusion, Block-Unroll-and-Jam). Optimizations may only consistently outperform -O2 if loop and memory access transformations occur. It can even slow down the code. Recommended For: Applications with loop-heavy floating-point calculations and large data sets. -Ofast: Sets the following flags: “-O3” “- : It enables optimizations that give fast and slightly less precise results than full IEEE division. For example, A/B is computed as A * (1/B) to improve the computation speed. no-prec-div” “ : enables more aggressive floating-point optimizations. -fp-model fast=2" The talks in detail about exactly which optimizations these options offer. official guide When using these options on our Jacobi code, we obtain these execution run times: It is clearly evident that all these optimizations are much faster than our base code (with “-O0”). The execution run time is 2–3x lower than the base case. What about MFLOP/s?? Well, that’s something!!! There is a big difference between the MFLOP/s of the base case and those with the optimization. Overall, though only slightly, “-O3” performs the best. The extra flags used by “- ” (“ ”) aren’t giving any additional speedup. Ofast -no-prec-div -fp-model fast=2 Architecture targeted (-xHost,-xCORE-AVX512) The machine’s architecture stands out as a pivotal factor influencing compiler optimizations. It can significantly enhance performance when the compiler knows the available instruction sets and the optimizations supported by the hardware (like vectorization and SIMD). For instance, my Skylake machine has 3 SIMD units: 1 AVX 512 and 2 AVX-2 units. Can I really do something with this knowledge??? The answer lies in strategic compiler flags. Experimenting with options such as “ ” and, more precisely, “ ” may allow us to harness the full potential of the machine’s capabilities and tailor optimizations for optimal performance. -xHost -xCORE-AVX512 Here is a quick description of what these flags are all about: -xHost: Specifies that the compiler should generate code optimized for the host machine’s highest instruction set. Goal: Takes advantage of the latest features and capabilities available on the hardware. It can give an amazing speedup on the target system. Key Features: While this flag optimizes for the host architecture, it might result in binaries that are not portable across different machines with varying instruction set architectures. Considerations: -xCORE-AVX512: Explicitly instruct the compiler to generate code that utilizes the Intel Advanced Vector Extensions 512 (AVX-512) instruction set. Goal: AVX-512 is an advanced SIMD (Single Instruction, Multiple Data) instruction set that offers wider vector registers and additional operations compared to previous versions like AVX2. Enabling this flag allows the compiler to leverage these advanced features for optimized performance. Key Features: Portability is again the culprit here. The binaries generated with AVX-512 instructions may not run optimally on processors that do not support this instruction set. They may not work at all! Considerations: AVX-512 set instructions use Zmm registers, which are a set of 512-bit wide registers. These registers serve as the foundation for vector processing. By default, “ ” assumes that the program will unlikely benefit from zmm registers usage. The compiler avoids using zmm registers unless a performance gain is guaranteed. -xCORE-AVX512 If one plans to use the zmm registers without restrictions, “ ” can be set to high. That’s what we’ll be doing as well. -qopt-zmm-usage Don’t forget to check the for detailed instructions. official guide Let’s see how these flags work for our code: Woohoo! We now cross the 1200 MFLOP/s mark for the smallest resolution. The MFLOP/s values for other resolutions have also increased. The remarkable part is that we achieved these results without any substantial manual interventions — simply by incorporating a handful of compiler flags during the application compilation process. However, it is essential to highlight that the compiled executable will only be compatible with a machine using the same instruction set. The optimization-versus-portability trade-off is evident, as code optimized for a particular instruction set may sacrifice portability across different hardware configurations. So, make sure you know what you’re doing!! Don’t worry if your hardware doesn’t support AVX-512. Intel C++ Compiler supports optimizations for AVX, AVX-2 and even SSE. The has everything you need to know! Note: documentation Interprocedural Optimization (IPO) Interprocedural Optimization involves analyzing and transforming code across multiple functions or procedures, looking beyond the scope of individual functions. IPO is a multi-step process focusing on the interactions between different functions or procedures within a program. IPO can include many different kinds of optimizations, including Forward substitution, Indirect call conversion, and Inlining. Intel Compiler supports two common types of IPO: Single-file compilation and multi-file compilation (Whole Program Optimization) [ ]. There are two common compiler flags performing each of them: 3 -ipo: Enables interprocedural optimization, allowing the compiler to analyze and optimize the entire program, beyond individual source files, during compilation. Goal: Whole Program Optimization: “ ” performs analysis and optimization across all source files, considering the interactions between functions and procedures throughout the entire program.- Cross-function and cross-module optimization: The flag facilitates inlining functions, synchronization of optimizations, and data flow analysis across different program parts. Key Features:- -ipo It requires a separate link step. After compiling with “ ”, a particular link step is needed to generate the final executable. The compiler performs additional optimizations based on the whole program view during linking. Considerations: -ipo -ip: Enables interprocedural analysis-propagation, allowing the compiler to perform some interprocedural optimizations without requiring a separate link step. Goal: Analysis and propagation: “ ” enables the compiler to perform research and data propagation across different functions and modules during compilation. However, it does not perform all optimizations that require the full program view.- Faster compilation: Unlike “ ”, “ ” doesn’t necessitate a separate linking step, resulting in speedier compilation times. This can be beneficial during development when quick feedback is essential. Key Features:- -ip -ipo -ip Only some limited interprocedural optimizations occur, including function inlining. Considerations: -ipo generally provides more extensive interprocedural optimization capabilities as it involves a separate link step but comes at the cost of longer compilation times. [ ] 4 -ip is a quicker alternative that performs some interprocedural optimizations without requiring a separate link step, making it suitable for development and testing phases.[ ] 5 Since we’re only talking about performance and different optimizations, compile times, or size of the executable not being our concern, we’ll focus on “ ”. -ipo -fno-alias All the above optimizations depend on how well you know your hardware and how much you would experiment. But that’s not all. If we try to identify how the compiler would see our code, we may identify other potential optimizations. Let’s again have a look at our code: /*
 * One Jacobi iteration step
 */
void jacobi(double *u, double *unew, unsigned sizex, unsigned sizey) {
 int i, j;

 for (j = 1; j < sizex - 1; j++) {
  for (i = 1; i < sizey - 1; i++) {
   unew[i * sizex + j] = 0.25 * (u[i * sizex + (j - 1)] +  // left
      u[i * sizex + (j + 1)] +  // right
      u[(i - 1) * sizex + j] +  // top
      u[(i + 1) * sizex + j]); // bottom
  }
 }

 for (j = 1; j < sizex - 1; j++) {
  for (i = 1; i < sizey - 1; i++) {
   u[i * sizex + j] = unew[i * sizex + j];
  }
 }
} jacobi() function takes a couple of pointers to double as parameters and then does something inside the nested for loops. When any compiler sees this function in the source file, it has to be very careful. Why?? The expression to calculate using involves the average of 4 neighboring values. What if both and point to the same location? This would become the classical problem of [ ]. unew u u u unew aliased pointers 7 Modern compilers are very smart and to ensure safety, they assume that aliasing could be possible. And for scenarios like this, they avoid any optimizations that may impact the semantics and the output of the code. In our case, we know that and are different memory locations and are meant to store different values. So, we can easily let the compiler know there won’t be any aliasing here. u unew How do we do that? There are two methods. First is the C “ . But it requires changing the code. We don’t want that for now. ” keyword restrict Anything simple? Let’s try “ ”. -fno-alias -fno-alias: Instruct the compiler to not assume aliasing in the program. Goal: Assuming no aliasing, the compiler can more freely optimize the code, potentially improving the performance. Key Features: The developer has to be careful in using this flag as in case of any unwarranted aliasing, the program may give unexpected outputs. Considerations: More details can be found in the . official documentation How does this perform for our code? Well, now we have something!!! We’ve achieved a remarkable speedup here, nearly 3x of the previous optimizations. What’s the secret behind this boost? By instructing the compiler not to assume aliasing, we’ve given it the freedom to unleash powerful loop optimizations. A closer examination of the assembly code (though not shared here) and the generated compile optimization report (see ) reveals the compiler’s savvy application of and . These transformations contribute to a highly optimized performance, showcasing the significant impact of compiler directives on code efficiency. below loop interchange loop unrolling Final graphs This is how all the optimizations perform against each other: Compiler Optimization report (-qopt-report) The Intel C++ compiler provides a valuable feature that allows users to generate an optimization report summarizing all the adjustments made for optimization purposes [ ]. This comprehensive report is saved in the YAML file format, presenting a detailed list of optimizations applied by the compiler within the code. For a detailed description, see the official documentation on “ ”. 8 -qopt-report What next? We discussed a handful of compiler flags that can drastically improve the performance of our code without us actually doing much. The only prerequisite: don’t do anything blindly; make sure you know what you’re doing!! There are hundreds of such compiler flags, and this story talks about a handful. So, it is worth looking at your preferred compiler’s official compiler guide (especially the documentation related to optimization). Apart from these compiler flags, there are a whole bunch of techniques like Vectorization, SIMD intrinsics, , and , which can amazingly improve the performance of your code. Profile Guided Optimization Guided Auto Parallelism Similarly, Intel C++ compilers (and all the popular ones) also support pragma directives, which are very nice features. It’s worth checking some of the pragmas like etc., on the . ivdep, parallel, simd, vector, Intel-Specific Pragma Reference Suggested reads [1] Optimization and Programming (intel.com) [2] High Performance Computing with “Elwetritsch” at the University of Kaiserslautern-Landau (rptu.de) [3] Interprocedural Optimization (intel.com) [4] ipo, Qipo (intel.com) [5] ip, Qip (intel.com) [6] Intel Compiler, Optimization and Other flags for use by SPEChpc [7] Aliasing — IBM Documentation [8] Intel® Compiler Optimization Reports Featured Photo by on . Igor Omilaev Unsplash Also published . here

Compiler Optimizations: Boosting Code Performance With Minimum Tweaks!

কম্পাইলার অপ্টিমাইজেশান: ন্যূনতম টুইক সহ কোড পারফরম্যান্স বুস্টিং!

Compiler-Optimierungen: Steigern Sie die Code-Leistung mit minimalen Optimierungen!

Optimizaciones del compilador: ¡aumento del rendimiento del código con ajustes mínimos!

Optimisations du compilateur : améliorer les performances du code avec un minimum de modifications !

कंपाइलर अनुकूलन: न्यूनतम बदलाव के साथ कोड प्रदर्शन को बढ़ावा देना!

コンパイラーの最適化: 最小限の調整でコードのパフォーマンスを向上!

컴파일러 최적화: 최소한의 조정으로 코드 성능 향상!

Otimizações do compilador: aumentando o desempenho do código com ajustes mínimos!

Оптимизация компилятора: повышение производительности кода с минимальными изменениями!

Derleyici Optimizasyonları: Minimum Düzenlemelerle Kod Performansını Artırma!

Tối ưu hóa trình biên dịch: Tăng hiệu suất mã với các chỉnh sửa tối thiểu!

编译器优化：以最少的调整提高代码性能！

There are a sufficient number of ways to increase the speed of a project at runtime. Let’s look at the most popular of them: Limiting dynamic dispatch in classes Under the hood, any new class will use dynamic (table) dispatch. This is a price for the ability to be inherited, but if the class is not inherited, you can save a significant amount of resources. The prefix will turn your class into a statically dispatched object. final Before optimization: // parent class with dynamic table dispatch
class A {}
// child class with dynamic table dispatch
class B: A {} After optimization: // child class with static dispatch
final class B: A {} Examples of speed boost Parent class A: Subclass B: Final class B: Limitation of dynamic dispatch at methods A similar approach is used for class methods, but there are several options available: If your method is not used outside the class, feel free to use . private If you are going to protect your method from the possibility of override, use . final Both prefixes will turn your method into a statically dispatched method but with different scopes. Before optimization: // method with dynamic table dispatch
func method() {} After optimization: // private method with static dispatch
private func method() {}

// method with static dispatch cannot be override
final func method() {} Examples of speed boost Nonfinal method: Final or private method: Inline optimization One of the most curious and rarely used types of optimization. Inline optimization allows you to copy code calculations directly to the place where a given function is called without calling a method marked with inline, which reduces the number of method calls and works faster than static dispatch. Among the pitfalls, in order to force the compiler to call a method with inline optimization — it is necessary to mark the method not only but also @inlineable @inline(__always) not to mark methods that spend a large amount of resources on copying with these prefixes; there is a risk of filling the cache of all processor levels with calculations of these methods and, as a result, getting slower instead of speeding up. It is important Before optimization: // test method with multiple calling
func testMethod() {} After optimization: // inlinable test method with multiple calling
@inlinable
@inline(__always)
func testMethod() {} Examples of speed boost regular calling: inlined calling: Optimization in protocols Under the hood, protocols in Swift can be implemented for reference types and value types. Due to the fact that the memory management mechanism for reference types and value types are different, you can significantly speed up the compiler by limiting protocol adoption to reference types only. Before optimization: // can be adopted to reference types and value types,
// but value types adoption is unnecessarily
protocol Implementable {} After optimization: // can be adopted to reference types only
protocol Implementable: AnyObject {} Examples of speed boost creation array elements with a protocol for reference and value type creation array elements with protocol for reference type only Arrays was originally intended not only as a collection type for Swift, allowing elements of the same type to be stored sequentially, but also as a collection type providing compatibility with . Array NSArray In case the array is intended to store non-ObjC data types, it is recommended to use for more efficient operation of the array. ContagiousArray Before optimization: // creation array with NSArray interop
let array: Array<Int> = [1, 2, 3] After optimization: // creation array without NSArray interop
let fastArray: ContiguousArray<Int> = [1, 2, 3] Examples of speed boost: creation of : Array creation of : ContagiousArray Also published . here Don’t hesitate to contact me on if you have any questions. Also, you can always . Twitter buy me a coffee

Swift Runtime Optimization: Boosting Project Speed with Simple Techniques

Decorators are an extremely useful feature in Python language that allows programmers to change the behaviour of a function without changing the function. Sounds strange? Well, don't worry because In this post, We are going to: Understand what decorators are and how they work. How to create your own decorators See the practical use case of decorators. So, let’s jump into it. Understanding Functions in Python! We all have used functions in Python. The most basic example, print(), is a function that lets us see the output on the screen. Functions are basically a reusable block of code that performs a specific task or set of tasks. Functions are a fundamental concept in programming and are used to organize and modularize code, making it more manageable, readable, and efficient. The basic structure of a Python function is shown below. All Python functions start with the keyword "def" followed by the function name and parentheses. For example def hello(name):
    print(f"Hello, {name}")

hello("Shekhar") Output Hello, Shekhar This function simply prints a hello message with the given argument. But what if we simply call the function like this, what will happen now? print(hello) We will get an output which will look something like this <function hello at 0x000001EC6A3F8CC0> As with most of the things in Python, functions are also an object, and the function name is just a reference to that object. The above output tells us that there is a function object at the memory location "0x000001EC6A3F8CC0". That means that you can use functions as arguments to other functions, store functions as dictionary values, or return a function from another function. This leads to many powerful ways to use functions. 😲 Don't get confused; let's start one by one. Function names are just references to the function Let's look at this baby example to understand the concept that function names are just references. print("Shekhar")

myfunction = print

myfunction("Hello")

print(print)
print(myfunction) Output Shekhar
Hello
<built-in function print>
<built-in function print> When we write myfunction = print, we are referencing the print object through myfunction so myfunction acts as a print function. Now, let's look at another example. We can do something like this def add_one(number):
    return number + 1

def multiply_10(number):
    return number * 10

function_list = [add_one, multiply_10]

print(add_one(10))
print(multiply_10(10))

print(function_list[0](10))
print(function_list[1](10)) We get this output 11
100
11
100 Nested Functions Now that we have understood the basics of functions in Python, let's dive a little deeper. Take this function as an example. def prefix(prefix):
    def hello(name):
        print(f"{prefix} Hello, {name}!")
    return hello

print_debug = prefix("DEBUG :")
print_debug("shekhar")

print_warning = prefix("Warning :")
print_warning("Opps!") Output DEBUG : Hello, shekhar
Warning :Hello, Opps! Here, prefix("DEBUG:") returns the with prefix argument as "DEBUG :" reference to function hello Creating Function Decorators Let's understand decorators through an example. Consider this code below. def reverse(func):
    def reverse_caller(text):
        func(text[::-1])

    return reverse_caller

rev_print = reverse(print)
rev_print("Hello Shekhar!")

rev_print = reverse(print_warning)
rev_print("Shekhar!") Output !rahkehS olleH
Warning : Hello, !rahkehS! Now, this function reverse() is taking a function reference as a parameter and returning a function 😯 This is what we call a Decorator. It's a function that takes a function as a parameter, modifies it, and returns the function. Another way to do that is as follows. @reverse
def greet(name):
    print(f"Hello {name}")

greet("Shekhar") Hello rahkehS This is exactly the same as greet=reverse(greet). The @reverse is just a syntax to make things tidy and easier to read The general template to make a decorator is as follows def decorator_name(func):
    def wrapper(func):
        #DO SOMETHING
    return wrapper Exercise Now that we have understood the basics let's write a decorator that will Print "BEFORE" runs the function and print "AFTER". I'd highly recommend you try it on your own before beginning further. Following our decorator template, we have the following def before_after(func):
    def wrapper(name):
        print("Before")
        func(name)
        print("After")
    return wrapper

@before_after
def greet(name):
    print(f"Hello {name}")


greet("shekhar") Output And it works as expected 😃 Before
Hello shekhar
After Handling arguments in Decorators Now, take our previous example greet and try to add a default argument to it and see what happens. @before_after
def greet(name="Shekhar"):
    print(f"Hello {name}") What do you think? Will it work or not? Let's see the output. TypeError: before_after.<locals>.wrapper() missing 1 required positional argument: 'name' It failed because when we decorate a function and use it, then we are not actually using our original function. We are using the nested wrapper function. If we pass greet() with no arguments, then our wrapper function will also receive no arguments, but it requires one "name" argument to function properly. That's why the error says that "wapper() missing 1 required positional argument: name" Let's take another example to understand this concept more. def greet(name):
    print(f"Hello {name}")

greet("Shekhar")

print(greet.__name__)

@before_after
def greet(name):
    print(f"Hello {name}")

greet("Shekhar")

print(greet.__name__) Output Hello Shekhar
greet
Hello rahkehS
wrapper The is an in-built property in Python that returns the name of the object, and as I mentioned before, function names are references to the function objects we can get their names (or you can say the name of the function object). name One to notice here is that when we add our decorator to the function, the name of the function changes to the wrapper (because that was the name of the wrapper function in our decorator). This shows that when we use the decorated function, we are actually using the wrapper function from the decorator. interesting thing How to improve our decorators? As we saw, it is really important to make our decorators in such a way that they can handle these kinds of cases. We will face a similar error if we try to pass more than one argument to our decorated function. To solve these kinds of issues, we use *args and **kwargs, they translate to argument and keyword arguments, respectively. I will be explaining these in much more detail in an upcoming blog in which I will share how Python actually runs and how functions are executed and actually hold meaning in Python; these are called unpacking operators. Expands any variable as List * Expands any variable as Dictionary ** For example a = [1,2,3]
b = [3,4,5]
c1 = [*a,*b]
c2 = [a,b]

dict1 = {'a':1,'b':2}
dict2 = {'b':2,'c':3}
dict3 = {**dict1,**dict2}

print(f"c1:{c1}\n c2:{c2}")
print(dict3) c1:[1, 2, 3, 3, 4, 5]
c2:[[1, 2, 3], [3, 4, 5]]
{'a': 1, 'b': 2, 'c': 3} The *args enables us to send any number of arguments to the function, and **kwargs enables us to pass in any number of keyworded arguments when calling the function. For example def testing(*args,**kwargs):
    print(f"Args = {args}")
    print(f"KWargs = {kwargs}")

testing("hello",1,name="shekhar") Args = ('hello', 1)
KWargs = {'name': 'shekhar'} "args" and "kwargs" is the general naming convention. You can use any other name following the same * and ** pattern, and it will work the same way. Using Args and Kwargs in Decorators def before_after(func):
    def wrapper(*args, **kwargs):
        print("Before")
        func(*args, **kwargs)
        print("After")

    return wrapper


@before_after
def greet(name="Shekhar"):
    print(f"Hello {name}")


greet()
greet("joey") Before
Hello Shekhar
After
Before
Hello joey
After Another Exercise Let's write a decorator to run the decorated function thrice and return a 3-value tuple with results. import random

def do_thrice(func):
    print(f"Adding decorator to {func.__name__}")

    def wrapper():
        return (func(), func(), func())

    return wrapper


@do_thrice
def roll_dice():
    return random.randint(1, 6)

print(roll_dice()) Output Adding decorator to roll_dice
(2, 2, 1) Real use Cases Now that we have seen enough examples of creating decorators, let's see what are some real use cases in which decorators are extremely useful. Measuring Performance of Other Functions Suppose you have many functions in your code, and you want to optimize your code. Now, instead of writing the code for getting the runtime and memory usage for each function, we can simply write a decorator and apply it to all the functions we want to test. Let's take this example in which we are calculating function runtime using perf_counter (It's just like a stopwatch) and calculating memory usage using tracemalloc function. (It's part of the standard library, I will be making a dedicated blog on this topic). from functools import wraps
import tracemalloc
from time import perf_counter 

def measure_performance(func):
    @wraps(func)  # It is a inbuilt decorator in python
    # When used, the decorated function name remains the same

    def wrapper(*args, **kwargs):
        tracemalloc.start()
        start_time = perf_counter()
        func(*args, **kwargs)
        current, peak = tracemalloc.get_traced_memory()
        finish_time = perf_counter()
        print(f"Function: {func.__name__}")
        print(
            f"Memory usage:\t\t {current / 10**6:.6f} MB \n"
            f"Peak memory usage:\t {peak / 10**6:.6f} MB "
        )
        print(f"Time elapsed is seconds: {finish_time - start_time:.6f}")
        print(f'{"-"*40}')
        tracemalloc.stop()

    return wrapper


@measure_performance
def function1():
    lis = []
    for a in range(1000000):
        if a % 2 == 0:
            lis.append(1)
        else:
            lis.append(0)


@measure_performance
def function2():
    lis = [1 if a % 2 == 0 else 0 for a in range(1000000)]


function1()
function2() Output Function: function1
Memory usage:            0.000000 MB
Peak memory usage:       8.448768 MB
Time elapsed is seconds: 0.329556
----------------------------------------
Function: function2
Memory usage:            0.000000 MB
Peak memory usage:       8.448920 MB
Time elapsed is seconds: 0.294334
---------------------------------------- We simply added two decorators to our functions instead of adding the whole code to each function. This makes our code much more concise and readable. Dynamically adding functionality Many Libraries, such as Flask or FastAPI, use decorators extensively to create API routes. For example, in FastAPI, we use something like this: @app.get("/helloWorld", tags=["Testing"])
def read_root():
    return {"Hello": "World"} This creates a get endpoint /helloworld which triggers the read_root() function whenever called. Wrap-up That was it for this blog. I hope you learned something new today. If you did, please like/share so that it reaches others as well. Connect with me on where I write regularly about software development and solopreneurship. @twitter ✅ Here are some more posts you might be interested in. How I improved my python code Performance by 371%! RaspberryPi PICO and PYUSB: Unlocking the Power of Custom USB Endpoints Why I Started This Blog? Thank you for reading 😄 Also published . here

Mastering Python Function Decorators: Boosting Functionality and Code Efficiency

How are you, hacker?
🪐 has got you covered with fresh content from our trending stories of the day! Set email preference . Want to know what's trending right now?: The Techbeat by HackerNoon here 7 Ways CMOs Can Cut Costs While Boosting Growth and Efficiency By [ 4 Min read ]
CMOs can embrace the power of SEO and create content that connects with their target audience, reduces costs, and helps drive significant business growth. @lomitpatel Read More. 834 Stories To Learn About Decentralized Internet By [ 118 Min read ]
Learn everything you need to know about Decentralized Internet via these 834 free HackerNoon stories. @learn Read More. Leveraging ML Algorithms for Business Growth Optimization By [ 11 Min read ]
Explore the transformative impact of machine learning on modern business growth strategies. @dkorotenko Read More. Will DevOps Cease to Exist? My Honest Opinion By [ 5 Min read ]
Debunking the rumor of DevOps' demise in the face of no-code platforms. @maksimmuravev Read More. Stateful vs. Stateless Applications in Containerized Environments By [ 5 Min read ]
This article focuses on the nuances and distinction between the  two fundamental types of applications in containerized environments: stateful and stateless. @feranmiadeyinka Read More. The 8 Most Dangerous Cash App Scams (with Screenshots) By [ 7 Min read ]
If you're a Cash App user, here are the eight most dangerous scams to look out for. @marcusleary Read More. Analyzing Chiliz for the Sports and Media Industries By [ 7 Min read ]
Chiliz (CHZ), the purpose-built, EVM-compatible layer 1 providing blockchain infrastructure for the sport and media industry, will get a SWOT. @andreydidovskiy Read More. NoOps: The Future of DevOps or a Mythical Utopia? By [ 6 Min read ]
Explore the evolving landscape of NoOps in DevOps. Dive into the debate to uncover the truth about this game-changing concept. @ezikielemmanuel Read More. GPT-4V Unveiled: From Detecting Emotions to Ordering Food - You Won't Believe What Else It Can Do! By [ 3 Min read ]
GPT-4V Unveiled: From Detecting Emotions to Ordering Food - You Won't Believe What Else It Can Do! @sergey-baloyan Read More. Researchers Decode Speech With Brain Waves: Can AI Read Minds Now? By [ 4 Min read ]
Researchers from Meta have discovered how to turn brain waves into speech using noninvasive methods like EEG and MEG @mikeyoung44 Read More. Unlocking Structured JSON Data with LangChain and GPT: A Step-by-Step Tutorial By [ 8 Min read ]
Manipulating Structured Data (from PDFs) with the Model behind ChatGPT, LangChain, and Python for Powerful AI-driven Applications. @horosin Read More. How This CEO Tripled Revenue in 6 Months With a Winning Culture By [ 4 Min read ]
Kashish Gupta, CEO of Hightouch, has a winning formula for building a high-growth startup. The key is how he cultivated a high performance culture. @rustygaillard Read More. The Best Practices For DevOps Pipelines By [ 6 Min read ]
Learn about DevOps Pipelines and how to implement it @abrahamdahunsi Read More. Unpopular Opinion: It’s Harder Than Ever to Be a Good Software Engineer By [ 5 Min read ]
Surviving tech's fast lane: A story of engineers, hype, and staying focused on what truly matters in a changing world @jurajmalenica Read More. AI Self-Improvement: How PIT Revolutionizes LLM Enhancement By [ 5 Min read ]
Rather than manually distilling criteria into prompts, implicit information in preference data can be leveraged. @mikeyoung44 Read More. Hybrid Cloud Strategies: Integrating On-Premises Infrastructure with IaaS By [ 5 Min read ]
Explore Hybrid Cloud Strategies: Integrate on-premises systems with IaaS for enhanced efficiency and scalability. Optimize your business for the future. @emmanuelohaba Read More. The System Design Cheat Sheet: Networks By [ 10 Min read ]
Networking Fundamentals for Systems Design. @gavr Read More. Front-End Optimization: My Journey to Accelerate Load Times in Heavy Front-End By [ 8 Min read ]
How I was able to improve the app's load time by implementing techniques such as import and render on visibility, image proxy, proficient work with SVG images @dsitdikov Read More. Complex Document Recognition: OCR Doesn’t Work and Here’s How You Fix It By [ 6 Min read ]
OCR solutions don't work — at least when it comes to complex documents. Learn how you can supercharge OCR tools wqith AI to handle any document @olegkokorin Read More. AI Safety is Moving Faster Than You Think By [ 2 Min read ]
Don't believe the AI doom hype - researchers understand and steer models more than ever before using techniques like reinforcement learning from human feedback. 🧑‍💻 What happened in your world this week?It's been said that , , . Feeling stuck? We got you covered ⬇️⬇️⬇️ We hope you enjoy this worth of free reading material. Feel free to forward this email to a nerdy friend who'll love you for it.See you on Planet Internet! With love,
The Hacker Noon Team ✌️ @smwii Read More. writing can help consolidate technical knowledge establish credibility and contribute to emerging community standards ANSWER THESE GREATEST INTERVIEW QUESTIONS OF ALL TIME

The TechBeat: 7 Ways CMOs Can Cut Costs While Boosting Growth and Efficiency (10/16/2023)

How are you, hacker?
🪐 has got you covered with fresh content from our trending stories of the day! Set email preference . Want to know what's trending right now?: The Techbeat by HackerNoon here 7 Ways CMOs Can Cut Costs While Boosting Growth and Efficiency By [ 4 Min read ]
CMOs can embrace the power of SEO and create content that connects with their target audience, reduces costs, and helps drive significant business growth. @lomitpatel Read More. Leveraging ML Algorithms for Business Growth Optimization By [ 11 Min read ]
Explore the transformative impact of machine learning on modern business growth strategies. @dkorotenko Read More. 834 Stories To Learn About Decentralized Internet By [ 118 Min read ]
Learn everything you need to know about Decentralized Internet via these 834 free HackerNoon stories. @learn Read More. Will DevOps Cease to Exist? My Honest Opinion By [ 5 Min read ]
Debunking the rumor of DevOps' demise in the face of no-code platforms. @maksimmuravev Read More. GPT-4V Unveiled: From Detecting Emotions to Ordering Food - You Won't Believe What Else It Can Do! By [ 3 Min read ]
GPT-4V Unveiled: From Detecting Emotions to Ordering Food - You Won't Believe What Else It Can Do! @sergey-baloyan Read More. Analyzing Chiliz for the Sports and Media Industries By [ 7 Min read ]
Chiliz (CHZ), the purpose-built, EVM-compatible layer 1 providing blockchain infrastructure for the sport and media industry, will get a SWOT. @andreydidovskiy Read More. The 8 Most Dangerous Cash App Scams (with Screenshots) By [ 7 Min read ]
If you're a Cash App user, here are the eight most dangerous scams to look out for. @marcusleary Read More. NoOps: The Future of DevOps or a Mythical Utopia? By [ 6 Min read ]
Explore the evolving landscape of NoOps in DevOps. Dive into the debate to uncover the truth about this game-changing concept. @ezikielemmanuel Read More. The System Design Cheat Sheet: Networks By [ 10 Min read ]
Networking Fundamentals for Systems Design. @gavr Read More. Stateful vs. Stateless Applications in Containerized Environments By [ 5 Min read ]
This article focuses on the nuances and distinction between the  two fundamental types of applications in containerized environments: stateful and stateless. @feranmiadeyinka Read More. Unlocking Structured JSON Data with LangChain and GPT: A Step-by-Step Tutorial By [ 8 Min read ]
Manipulating Structured Data (from PDFs) with the Model behind ChatGPT, LangChain, and Python for Powerful AI-driven Applications. @horosin Read More. The Best Practices For DevOps Pipelines By [ 6 Min read ]
Learn about DevOps Pipelines and how to implement it @abrahamdahunsi Read More. Researchers Decode Speech With Brain Waves: Can AI Read Minds Now? By [ 4 Min read ]
Researchers from Meta have discovered how to turn brain waves into speech using noninvasive methods like EEG and MEG @mikeyoung44 Read More. The Bloated State of Web Development By [ 4 Min read ]
The web development landscape keeps getting crowded and crowded but there are no signs of improvement, only of heavy and bloated web apps. @techroasts Read More. 600 Stories To Learn About Crypto By [ 107 Min read ]
Learn everything you need to know about Crypto via these 600 free HackerNoon stories. @learn Read More. AI Self-Improvement: How PIT Revolutionizes LLM Enhancement By [ 5 Min read ]
Rather than manually distilling criteria into prompts, implicit information in preference data can be leveraged. @mikeyoung44 Read More. Prompt Engineering: Understanding the Potential of Large Language Models By [ 4 Min read ]
Whether you're a developer integrating AI into your software or a no-coder, marketer, or business analyst adopting AI, prompt engineering is a MUST-HAVE skill t @youraimarketer Read More. Front-End Optimization: My Journey to Accelerate Load Times in Heavy Front-End By [ 8 Min read ]
How I was able to improve the app's load time by implementing techniques such as import and render on visibility, image proxy, proficient work with SVG images @dsitdikov Read More. AI Safety is Moving Faster Than You Think By [ 2 Min read ]
Don't believe the AI doom hype - researchers understand and steer models more than ever before using techniques like reinforcement learning from human feedback. @smwii Read More. 5 Lessons From Crossing The Chasm That Every B2B Startup Needs to Revisit By [ 2 Min read ]
First published in 1991, Crossing the Chasm outlines the harrowing journey of taking a technology product from early adoption to mainstream success. 🧑‍💻 What happened in your world this week?It's been said that , , . Feeling stuck? We got you covered ⬇️⬇️⬇️ We hope you enjoy this worth of free reading material. Feel free to forward this email to a nerdy friend who'll love you for it.See you on Planet Internet! With love,
The Hacker Noon Team ✌️ @vvmrk Read More. writing can help consolidate technical knowledge establish credibility and contribute to emerging community standards ANSWER THESE GREATEST INTERVIEW QUESTIONS OF ALL TIME

The TechBeat: 7 Ways CMOs Can Cut Costs While Boosting Growth and Efficiency (10/14/2023)

As marketing rapidly evolves, CMOs face a formidable challenge in finding effective ways to reduce costs (CAC). With escalating economic uncertainty, the pressure to cut costs has reached unprecedented levels. Whether you helm a Fortune 500 company or a nimble startup, lowering your CAC is a paramount objective. customer acquisition While paid media spending continues to soar as a viable response, resulting in soaring customer acquisition costs, another simple yet powerful solution exists: Search Engine Optimization (SEO). You can dramatically slash your CAC by implementing a meticulously designed SEO strategy. , CEO and Co-Founder of estimates that a robust SEO strategy can reduce customer acquisition costs by . Elsewhere, a recent study from Econsultancy and DeepCrawl found that of top performers strongly agreed that search results performance is a key indicator of business health, while 89% of marketing leaders agreed that organic search will drive more revenue for their business. Prashant Puri AdLift 60% 42% To conquer the challenge of minimizing customer acquisition costs, a comprehensive SEO strategy can serve as your guiding star. Here are seven powerful SEO strategies to significantly reduce your CAC. 1. Unleash the power of People Also Ask (PAA) A People Also Ask (PAA) Box can be a treasure trove of question-and-answer goodness to help users discover further information about their initial search query. Leaders should craft highly targeted content that enhances organic search visibility by diligently analyzing and addressing user questions. Incorporating relevant keywords and providing comprehensive answers to PAA queries will elevate your SEO rankings and better attract qualified traffic. 2. Invest in User-Generated Content (UGC) In the era of social proof, user-generated content has emerged as an invaluable tool. By actively encouraging customers to leave reviews, provide feedback, and share their experiences, you can amplify brand credibility, foster engagement, and boost performance. Implementing creative strategies such as contests, social media campaigns, and incentivized reviews will inspire users to generate valuable content to entice new customers and drive down acquisition costs. SEO 3. Create compelling content The significance of high-quality, informative, and engaging content cannot be overstated in any effective SEO strategy. Captivating content lies at the heart and soul of capturing the attention and loyalty of your target audience. Brands can bolster their content through meticulous keyword research and seamless integration to ensure prominent placement in search engine results. By consistently updating websites or blogs with fresh, captivating content, you can enhance organic rankings, drive website traffic, and attract potential customers. 4. Elevate your brand with Digital PR Digital PR and SEO go hand in hand, creating a match made in marketing heaven. While not strictly an SEO strategy, Digital PR complements your SEO efforts by enhancing your brand's E-A-T, or Expertise, Authority, and Trustworthiness. Increase organic visibility through collaborative efforts such as guest blogs, expert interviews, and share-worthy content. Earn high-quality backlinks with compelling content that establishes your brand's authority in the industry. 5. Leverage synergies between paid media and SEO Seamlessly integrating your paid media and SEO strategies optimizes your online presence, drives targeted traffic to your website, and boosts search engine rankings, all while keeping costs to a minimum. The synergy between paid media and SEO is a potent combination that unlocks tremendous potential for cost-effective customer acquisition. By aligning messaging keywords and targeting strategies across both channels, you create a unified and powerful marketing force that maximizes online visibility and drives qualified leads. 6. Optimize for featured snippets Like PAA, optimizing your content for featured snippets can be a game-changer in reducing customer acquisition costs. Providing clear and concise answers to specific queries increases the likelihood of your content being featured prominently in search results. Appearing in featured snippets enhances online visibility and drives targeted traffic to your website, reducing reliance on paid media channels. By claiming those coveted snippet spots, you establish your brand as a trusted and authoritative source of information, organically attracting potential customers. 7. Target your high-revenue products Focus on your most profitable products or services to achieve optimal ROI and reduce overall marketing spend. By directing resources towards high-revenue offerings, you can maximize the impact of your marketing campaigns and attract valuable customers who are more likely to convert. This targeted approach allows you to strategically allocate your budget and generate greater returns while keeping acquisition costs in check. A people-first approach is critical to growth and cost reduction As business leaders, we must understand the importance of a robust marketing strategy that delivers results while minimizing costs. To achieve this, leaders should prioritize creating content that resonates with their target audience. Building content for people is the cornerstone of any successful marketing strategy. We must invest in understanding our customers' unique needs, preferences, and pain points to build strong relationships and drive engagement. Many leaders opt to work with an SEO agency that can perform a comprehensive SEO audit to accomplish this most effectively. This step can pave the way for all future SEO strategies. With the right strategies and agency as a foundation, you can better focus on innovatively and engagingly telling your brand's story. I hope this piece provides some valuable takeaways for CMOs seeking to embrace the power of SEO and create content that connects with their target audience, reduces costs, and helps drive significant business growth. Also published . here About the Author is the Chief Growth Officer of Tynker, a leading K-12 edtech platform that has helped over 100 million kids learn to code. He has previously scaled startups like Roku, TrustedID, Texture, and IMVU. He is a public speaker, advisor, and author of , part of Eric Ries' bestselling "The Lean Startup" series. Lomit Patel Lean AI

7 Ways CMOs Can Cut Costs While Boosting Growth and Efficiency

Migrating from one OLAP database to another is huge. Even if you're unhappy with your current and have found some promising candidates, you might still hesitate to do the big surgery on your data architecture because you're uncertain about how things are going to work. So you need experience shared by someone who has walked the path. data tool Luckily, a user of has written down their migration process from ClickHouse to Doris, including why they need the change, what needs to be taken care of, and how they compare the performance of the two databases in their environment. Apache Doris To decide whether you want to continue reading, check if you tick one of the following boxes: You need your join queries to be executed faster. You need flexible data updates. You need real-time data analysis. You need to minimize your components. If you do, this post might be of some help to you. Replacing Kylin, ClickHouse, and Druid with Apache Doris The user undergoing this change is an e-commerce SaaS provider. Its data system serves real-time and offline reporting, customer segmentation, and log analysis. Initially, they used different OLAP engines for these various purposes: : The system provides offline reporting services for over 5 million sellers. The big ones among them have more than 10 million registered members and 100,000 SKUs, and the detailed information is put into over 400 data cubes on the platform. Apache Kylin for offline reporting : This entails high-frequency updates, high QPS, and complicated SQL. ClickHouse for customer segmentation and Top-N log queries : Sellers extract data they need by combining different dimensions, and such real-time reporting requires quick data updates, quick query response, and strong stability of the system. Apache Druid for real-time reporting The three components have their own sore spots. runs well with a fixed table schema, but every time you want to add a dimension, you need to create a new data cube and refill the historical data in it. Apache Kylin is not designed for multi-table processing, so you might need an extra solution for federated queries and multi-table join queries. And in this case, it was below expectation in high-concurrency scenarios. ClickHouse implements idempotent writing, so it does not support data updating or deletion itself. That means when there is something wrong at the upstream, you will need a full data replacement. And such data fixing is a multi-step process if you think it all the way through because of all the data backups and movements. Plus, newly ingested data will not be accessible for queries until it is put in segments in Druid. That means a longer window such that data inconsistency between upstream and downstream. Apache Druid As they work together, this architecture might be too demanding to navigate because it requires knowledge of all these components in terms of development, monitoring, and maintenance. Also, every time the user scales a cluster, they must stop the current cluster and migrate all databases and tables, which is not only a big undertaking but also a huge interruption to business. fills these gaps. Apache Doris : Doris is good at high-concurrency queries and joins queries, and it is now equipped with an inverted index to speed up searches in logs. Query performance : The Unique Key model of Doris supports both large-volume update and high-frequency real-time writing, and the Duplicate Key model and Unique Key model supports partial column update. It also provides exactly-once guarantee in data writing and ensures consistency between base tables, materialized views, and replicas. Data update : Doris is MySQL-compatible. It supports easy scaling and light schema change. It comes with its own integration tools, such as Flink-Doris-Connector and Spark-Doris-Connector. Maintenance So they plan on the migration. The Replacement Surgery ClickHouse was the main performance bottleneck in the old data architecture, and why the user wanted the change in the first place, so they started with the ClickHouse. Changes in SQL statements Table creation statements The user-built their own SQL rewriting tool that can convert a ClickHouse table creation statement into a Doris table creation statement. The tool can automate the following changes: : It converts ClickHouse field types into the corresponding ones in Doris. For example, it converts String as a Key into Varchar and String as a partitioning field into Date V2. Mapping the field types : Some tables have historical partitions, and the number of partitions should be specified upon table creation in Doris. Otherwise, a "No Partition" error will be thrown. Setting the number of historical partitions in dynamic partitioning tables : It decides the number of buckets based on the data volume of historical partitions; for non-partitioned tables, it decides the bucketing configurations based on the historical data volume. Determining the number of buckets : It decides the time to live of partitions in dynamic partitioning tables. Determining TTL : For the Unique Key model of Doris, it can specify the data import order based on the Sequence column to ensure orderliness in data ingestion. Setting the import sequence Query statements Similarly, they have their own tool to transform the ClickHouse query statements into Doris query statements. This is to prepare for the comparison test between ClickHouse and Doris. The key considerations in the conversions include: : This is simple, given the mapping rules in table creation statements. Conversion of table names : For example, the function in ClickHouse is equivalent to , is equivalent to and , and and should be converted to window functions. Conversion of functions COUNTIF SUM(CASE WHEN_THEN 1 ELSE 0) Array Join Explode Lateral View ORDER BY GROUP BY : ClickHouse goes by its own protocol, while Doris is MySQL-compatible, so there needs to be an alias for subqueries. In this use case, subqueries are common in customer segmentation, so they use Difference in semantics sqlparse Changes in data ingestion methods Apache Doris provides broad options for data writing methods. For the real-time link, the user adopts Stream Load to ingest data from NSQ and Kafka. For the sizable offline data, the user tested different methods, and here are the takeouts: Insert Into Using Multi-Catalog to read external data sources and ingesting with Insert Into can serve most needs in this use case. Stream Load The Spark-Doris-Connector is a more general method. It can handle large data volumes and ensure writing stability. The key is to find the right writing pace and parallelism. The Spark-Doris-Connector also supports Bitmap. It allows you to move the computation workload of Bitmap data in Spark clusters. Both the Spark-Doris-Connector and the Flink-Doris-Connector rely on Stream Load. CSV is the recommended format choice. Tests on the user's billions of rows showed that CSV was 40% faster than JSON. Spark Load The Spark Load method utilizes Spark resources for data shuffling and ranking. The computation results are put in HDFS, and then Doris reads the files from HDFS directly (via Broker Load). This approach is ideal for huge data ingestion. The more data there is, the faster and more resource-efficient the ingestion is. Pressure Test The user compared the performance of the two components on their SQL and join query scenarios, and calculated the of Apache Doris. CPU and memory consumption SQL query performance Apache Doris outperformed ClickHouse in 10 of the 16 SQL queries, and the biggest performance gap was a ratio of almost 30. Overall, Apache Doris was 2~3 times faster than ClickHouse. Join query performance For join query tests, the user used different sizes of main tables and dimension tables. : user activity table (4 billion rows), user attribute table (25 billion rows), and user attribute table (96 billion rows) Primary tables : 1 million rows, 10 million rows, 50 million rows, 100 million rows, 500 million rows, 1 billion rows, and 2.5 billion rows. Dimension tables The tests include and . Full join queries join all rows of the primary table and dimension tables while filtering join queries retrieve data of a certain seller ID with a filter. The results are concluded as follows: full join queries, filtering join queries WHERE Primary table (4 billion rows): Full join queries: Doris outperforms ClickHouse in full join queries with all dimension tables. The performance gap widens as the dimension tables get larger. The largest difference is a ratio of 5. Filtering join queries: Based on the seller ID, the filter screened out 41 million rows from the primary table. With small dimension tables, Doris was 2~3 times faster than ClickHouse; with large dimension tables, Doris was over ten times faster; with dimension tables larger than 100 million rows, ClickHouse threw an OOM error, and Doris functions normally. Primary table (25 billion rows): Full join queries: Doris outperforms ClickHouse in full join queries with all dimension tables. ClickHouse produced an OOM error with dimension tables larger than 50 million rows. Filtering join queries: The filter screened out 570 million rows from the primary table. Doris responded within seconds, and ClickHouse finished within minutes and broke down when joining large dimension tables. Primary table (96 billion rows): Doris delivered relatively quick performance in all queries, and ClickHouse was unable to execute all of them. In terms of CPU and memory consumption, Apache Doris maintained stable cluster loads in all sizes of join queries. Future Directions As the migration goes on, the user works closely with the , and their feedback has contributed to the making of . We will continue assisting them in their migration from Kylin and Druid to Doris, and we look forward to seeing their Doris-based unified data platform come into being. Doris community Apache Doris 2.0.0 Also published . here

Migrating from ClickHouse to Apache Doris: Boosting OLAP Performance

because it’s awesome: its rock-solid reliability, rich feature set, and open-source nature make it a popular choice for businesses of all sizes. But as your data grows, you may encounter certain challenges, especially when dealing with large tables. Anyone struggling with query performance, maintenance overhead, or managing the data lifecycle? 🙋 PostgreSQL is the most popular relational database on the planet One popular strategy to tackle these issues is . In this article, we'll explore when and why you should consider partitioning your large tables, sharing some best practices and common pitfalls in case you decide to move on with partitioning. table partitioning PostgreSQL What Is Table Partitioning? Table partitioning is a database design technique that divides a large table into smaller, more manageable pieces known as partitions. Each partition is a sub-table that holds a subset of the data, with each row existing in exactly one partition. Partitions can have their own indexes, but any unique indexes (including primary keys) must include the column that was used to partition the table. This technique can offer significant advantages when managing large tables. By breaking down a large table into smaller partitions, you can benefit from improved query performance, optimized index sizes, and more efficient data maintenance operations, keeping your large-scale databases more agile and responsive overall. But partitioning is not a one-size-fits-all solution: sometimes, partitioning won't benefit you. It could even harm your performance, as we’ll see later on. In terms of the types of partitioning you could implement, : PostgreSQL supports three partitioning strategies : this strategy is ideal for time-series data or incrementing sequences (maybe a BIGINT primary key), where you partition data based on a range of values (e.g.,g: by day or by a number of keys). Range partitioning : you would use list partitioning when you want to partition data based on specific values in a column (e.g., country or department). List partitioning : hash partitioning is suitable when there is no clear partitioning key, as it distributes data evenly across partitions based on a hash function. Hash partitioning One important characteristic of (independent of which strategy you follow) is that it does not support “global indexes”: there is no way to create an index across partitions. Instead, when querying across partitions, PostgreSQL will evaluate the clause, and if there are constraints on the partition column, it will use those to exclude partitions that don’t hold data relevant to the query. partitioning in PostgreSQL WHERE Consider the following example: Imagine we took a table storing sensor metrics and partitioned it by the column representing the ingest time. The table could now be broken down into multiple partitions, each of which could store the metrics ingested in a single day.  Once the table is partitioned by day (if we query the table for a range of days), PostgreSQL would exclude the partitions outside the range—meaning that only the relevant data would be scanned. Compare this to querying the original table, where we would have to scan all the data (or one large index). One can clearly see how partitioning could help you reduce CPU, disk activity, and query time. It’s probably clear to you at this point that if you were to implement partitioning in a production setup, you'd need automation in place to create and maintain partitions, especially if the partitioned table expects to receive data continuously. There are multiple ways to do this, the most common being the following: You can write SQL procedures/functions to check if the necessary partitions for upcoming days/weeks/months exist, using cron (on Unix-like systems) or Task Scheduler (on Windows) to run the script regularly. You can use a PostgreSQL extension dedicated to this, the most popular being (which will probably need to be paired with a scheduler). pg_partman Determining which methodology to follow for table partitioning —but this is something you can think about later. For now, let’s help you figure out if partitioning would be beneficial for you. deserves its own article When Should You Consider Partitioning? As we said above, partitioning can be very powerful, but it is certainly not a good idea for every single use case. Contrary to what people often think, the decision to partition a PostgreSQL table is not strictly based on an absolute table size but rather on various factors that interact with the table's size. It's essential to evaluate your database's characteristics and requirements before implementing partitioning. Generally speaking, you should start thinking about partitioning if you identify with one (or more) of the following: As we said above, the size of your tables is not the only thing that determines if you may see benefits from partitioning; this said, if you have large tables (this is from tens of millions of rows to billions of rows), you would probably benefit from partitioning. You have large tables. Even if the current table size isn't massive, a high data ingestion rate can indicate that the table will grow significantly in the near future. To implement a partitioning strategy, it might be beneficial to preemptively manage this growth before it starts affecting your performance and maintenance operations. Your ingestion rate is very high. Partitioning may also be beneficial if your queries are starting to slow down, especially those that should only touch a subset of your data. This could be true even when your tables are smaller due to the complexity of the data and queries. For example, partitioning can significantly enhance query performance when your daily queries include searches based on a specific range or criteria. Let's say you're dealing with time-series data: partitioning by date can help you quickly retrieve records within a particular time frame without scanning the entire table. You’re beginning to notice query performance degradation. As a table grows, maintenance operations like , , and indexing can take longer and might start impacting your operational efficiency. Partitioning can simplify these operations because you can focus on maintaining smaller partitions independently, reducing the impact on your database's overall performance. You’re dealing with maintenance overhead. VACUUM ANALYZE If your dataset has built-in obsolescence, where older data is periodically purged, partitioning can make these operations much more efficient. Dropping an old partition is much faster and less resource-intensive than deleting rows. You’re managing data retention policies. If you want to operate with limited memory, you might benefit from partitioning, as smaller indexes and data chunks fit better in memory and improve cache hit rates. In most cases, this will also improve performance. You want to use less memory. When Not to Use Partitioning On the contrary, there are situations where introducing partitioning may be counterproductive, and you may want to look at other optimizations instead, such as tuning your indexes or queries: If your tables are not large and are unlikely to grow significantly, partitioning would add unnecessary complexity without delivering noticeable benefits. Your tables are small, and you’re not ingesting at high rates. Implementing and maintaining partitioned tables introduces a certain level of operational and architectural complexity: managing such tables would become more challenging, so make sure you feel ready to go there— You’re just starting with PostgreSQL. or use a hosted service to manage this for you. If your queries usually access data uniformly across the table rather than focusing on specific subsets (like recent data, ranges of data, specific categories, and so on), then your query performance could actually become worse. You have uniform data access patterns. If most of your queries don’t use your partition key in the clause, you’ll scan every partition every time. This will be slow and will get slower the more partitions you have. Frequent full scans. WHERE Partitioning Best Practices If, after reading this article, , make sure to follow this advice—it will help you set up things right from the start: you’ve decided to give partitioning a go . This is one of the first design questions that will come to your mind when implementing partitioning: what’s your ideal partition size? The answer is it depends—you should aim for a balance between too large and too small. While PostgreSQL can handle a high number of partitions, having too many will increase planning time and could negatively affect query time. At the same time, if your partitions are too large, then you won’t be able to use ranges to exclude data, and the effectiveness of partition pruning will be minimized. Choose the right partition size Aim to keep partitions relatively uniform in size, ensuring that maintenance tasks and query performances are consistent across partitions. Keep your partition size consistent. . Opt for a key that aligns with your query patterns. For instance, if most of your queries filter by date, a timestamp or date column would be an ideal partitioning key. Choose the right partitioning key . Ensure that partitions for upcoming periods (e.g., future days or months) are created ahead of time so there's no interruption in data ingestion. While you can use a default partition to catch orphaned records, in practice, this introduces a maintenance burden and does not perform well. Create partitions in advance . For example, if you're partitioning by time and data has a limited useful life, schedule regular tasks to drop or archive old partitions. Take advantage of data retention policies to maintain old partitions If you’re especially interested in optimizing query performance, make sure to analyze and understand the query execution plan to validate that only necessary partitions are being scanned. Optimize your queries. If you're using tablespaces to place partitions on different storage engines (e.g., EBS or S3), ensure that frequently accessed partitions are on faster storage and older or less accessed partitions can be on slower, cheaper storage. Properly place partitions across different storage mediums. Pitfalls to Avoid Similarly, make sure to avoid these common mistakes: . It's tempting to create a high number of small partitions, but as we mentioned earlier, this won’t work well—you’ll get into query planning and management challenges. Over-partitioning . Avoid creating unnecessary indexes on your partitions. Only index the columns that are frequently filtered or joined on. Inefficient indexing Queries spanning multiple partitions or not using the partition key in the clause might suffer in performance. Ensure that the majority of your queries are optimized for the partitioning scheme. Unoptimized query pattern. WHERE . If you insert data with no partition to live in, it will either be rejected or stored in a partition. Ensure you either pre-create partitions at a quiet time (as this will lock your table) or use an extension that creates new partitions on the fly. Running out of partitions DEFAULT If you're creating many partitions, especially on different tablespaces or disks, monitor disk usage to avoid out-of-space issues. Monitor disk usage (partitions need extra space). Wrap Up Partitioning can be a powerful ally in your production database, especially if your tables are starting to get big, your ingestion is growing, or your query performance is starting to degrade. Still, partitioning is not a one-size-fits-all solution: consider if it’s the right fit for your use case before deciding to implement it, and if we decide to do so, make sure to follow best practices to get the most out of it. PostgreSQL tables Also published . here

PostgreSQL Table Partitioning: Boosting Performance and Management

In modern web development, optimizing the performance of our applications is crucial to provide a smooth user experience. An effective technique to achieve this is lazy loading, which involves loading only the necessary components and resources when they are actually required. In this blog, we will explore how to implement lazy loading in React using the function and the component, and we will see the benefits it brings to our . lazy Suspense applications Understanding Lazy Loading Lazy loading is a strategy that defers the loading of non-essential resources until they are needed, reducing the initial loading time and improving the overall performance of the application. In React, we can achieve lazy loading using the function and the component, introduced in React 16.6. lazy Suspense Using the Function lazy The function allows us to dynamically import a component, which is especially useful for code splitting and optimizing package size. lazy Here's how you can use it: const MyLazyComponent = React.lazy(() => import('./MyComponent')); In this example, will only be loaded when is rendered in the application. MyComponent MyLazyComponent Implementing Lazy Loading with the Component. The component is used to wrap components that are lazily loaded. It allows us to specify a fallback content that will be displayed while the lazy component is loading. Suspense Suspense Here's how it's used: import React, { lazy, Suspense } from 'react';

const MyLazyComponent = lazy(() => import('./MyComponent'));

function App() {
  return (
    <div>
      <h1>Lazy Loading Example</h1>
      <Suspense fallback={<div>Loading...</div>}>
        <MyLazyComponent />
      </Suspense>
    </div>
  );
} In this example, while is being fetched, the fallback content ("Loading...") will be shown, providing a smooth transition for the user. MyLazyComponent Benefits of Lazy Loading By loading only the necessary components, the initial loading time of the application is reduced, enhancing the user experience. Faster Initial Loading: Lazy loading avoids unnecessary loading of resources, which can lead to faster navigation and interactions within the application. Performance Improvement: Lazy loading encourages code splitting, resulting in smaller package sizes that load more efficiently. Code Splitting: Users can interact with visible components more quickly, improving engagement and interaction.más rápidamente, mejorando la participación y la interacción. Increased User Engagement: Let’s Code a Real Example Imagine we are working on a web application containing several pages or routes, each represented by a different component. We want to optimize the application’s performance by ensuring that components are loaded only when the user needs them. https://codesandbox.io/s/priceless-pond-38qfks?from-embed=&embedable=true In the previous example, , , and are components created using and the function. This allows the components to be loaded lazily, meaning only when needed. HomePage AboutPage ContactPage lazy import() The function receives a function that returns a promise that loads the component. lazy() Within , routes are defined using the component. This component contains several elements that define the different routes of the application. Suspense Routes Route Each has a corresponding to the URL and representing the component to be rendered when the URL matches the path. Route path element If you want to learn more about you can check out my other article: React Router https://hackernoon.com/2-2-using-react-router-v6-for-private-and-public-routes-with-access-validation?embedable=true Conclusion Lazy loading is a powerful technique in React to enhance performance and user experience in our applications. By utilizing the function and the component, we can efficiently load components on demand, leading to faster loading times and more responsive applications. lazy Suspense Read more: Supercharge Your React Applications: 7 Best Practices and Techniques Protect Your Application by Hiding API Keys and Tokens in React - Beginners guide Want to connect with the Author? Love connecting with friends all around the world on . Twitter Also published . here

Boosting React App Performance: A Guide to Lazy Loading and Suspense

Hi there! I'm thrilled to share how I managed to improve our release flow by 20% through the implementation of a system QA role. Given that my company is a typical product company, the teams are divided not by product but by components. Thanks to this, on the one hand, we managed to form powerful teams with "holders" of knowledge. But on the other hand, the roles within the units are relatively isolated, and different sets of hard skills and expertise impose their limitations. For example, I sometimes need to move a tester from the backend team to the frontend or vice versa. This led to the fact that there was a question of effective orchestration within QA teams and management of cross-team interaction. Which, of course, ultimately affected the release flow. Release flow before changes Before the changes, our release flow looked like this: A is a document that outlines the detailed needs, expectations, and specifications for a specific feature, system, product, or project within a business. It serves as a bridge between business stakeholders and the development or implementation teams, ensuring a clear understanding of what is to be achieved and how it aligns with the overall business should contain detailed scenarios that describe how users will interact with the feature. The Feature Owner (FO) is tasked with the duty of formulating business requirements. Business Requirements Specification (BRS) goals.It A is a comprehensive document that outlines the detailed description of what a software system is expected to accomplish.
For example, how and which commands interact, by which protocol, and what data will be transmitted. If the team working on the feature utilizes a graphical interface, the SRS should include layouts of the feature being developed. The feature Architect is responsible for writing SRS. Software Requirements Specification (SRS) – a set of cases that a feature owner checks before accepting a feature. After the documents are described, we proceed to implement the feature. When the first team finishes developing and testing its part of the feature, it's transferred to the second team. The second team performs integration/development/testing and passes it to the next unit. And so on until all component teams participating in the feature development pass through this flow. FO validates the feature, and it's sent to the release. Quality Action Plan (QAP) Release flow problems So, everything seems to be okay – documents, applications, acceptance cases. However, we encountered the following difficulties in this process: Problems on the part of QA: Requirements testing begins only after development. Tasks that have already been implemented by developers, along with their requirements, are handed over to the QA team. But, as we know, there may be errors in the requirements. Finding the team responsible for a bug takes quite a bit of time because it's not always clear which cases have already been tested by other teams. No reuse of test artifacts. As part of testing one feature, QA teams prepare similar test data sets. But due to the isolation and narrow specialization of the teams, they could not transmit this data to each other. Problems on the part of FO: Due to the many features, writing QAP takes a lot of time. Validation of the feature occurs after development by all teams. In our case, this significantly lengthens the release flow due to many integration issues. The preparation of the test environment is also precise due to the complexity of the product and the volume of integrations between teams. Different teams are testing their components simultaneously, increasing the risks of mashing, changing, or deleting data. System QA in the updated release flow To facilitate effective cross-team interaction among QA teams and reduce the release flow, we introduced the role of system QA. This helped alleviate the workload in the form of writing acceptance cases with FO and speed up the writing of test scenarios, introduce intermediate testing of the feature component before passing it to the next team, and also shift the time-consuming work of preparing the test environment to the system QA, taking into account all the nuances and requirements of the teams for integrations and test data. System QA has become a link between the technical and business requirements for each feature and the product as a whole. Onboarding for system QA To understand the entire release cycle, System QAs need to understand how a specific release cycle works in each team. Onboarding typically lasts about three months as the system QA spends 2-3 weeks in each team, understanding their specific release cycles. Results of the new process We are now testing the BRS/SRS requirements from feature owners and architects. Early bug detection leads to cost savings for the business. We've established a cross-team QA space, where test artifacts are attached to each feature – business requirements, technical requirements, acceptance cases, cases of other teams, test data. This significantly helped all QA teams to be in a single context and effectively reuse data. Accelerated the process of localization of bugs because the system QA has sets of test cases from all teams. Since the system QA is writing acceptance cases for each team, this is an excellent hint for speeding up and improving testing quality. The integration process has become painless since the feature is validated by acceptance cases after each command. Having removed a significant part of the load from the FO, the acceptance of features and the preparation of an integration stand with test data accelerated. Overall, accelerated the release flow by 15-20% and reduced the number of integration bugs by almost half since now we catch them both at the stage of writing BRS and SRS requirements and during team integrations within the framework of feature development. Happy and productive testing!

Boosting Productivity: A Guide to Implementing a New QA Role for Faster Releases

উত্পাদনশীলতা বৃদ্ধি করা: দ্রুত প্রকাশের জন্য একটি নতুন QA ভূমিকা বাস্তবায়নের জন্য একটি নির্দেশিকা৷

Produktivität steigern: Ein Leitfaden zur Implementierung einer neuen QS-Rolle für schnellere Releases

Impulsar la productividad: una guía para implementar un nuevo rol de control de calidad para lanzamientos más rápidos

Booster la productivité : un guide pour la mise en œuvre d'un nouveau rôle d'assurance qualité pour des versions plus rapides

उत्पादकता बढ़ाना: तेजी से रिलीज के लिए एक नई क्यूए भूमिका लागू करने के लिए एक गाइड

生産性の向上: より迅速なリリースのための新しい QA ロールの実装ガイド

생산성 향상: 더 빠른 릴리스를 위한 새로운 QA 역할 구현 가이드

Aumentando a produtividade: um guia para implementar uma nova função de controle de qualidade para lançamentos mais rápidos

Üretkenliği Artırma: Daha Hızlı Sürümler için Yeni Bir Kalite Güvence Rolü Uygulama Kılavuzu

Tăng năng suất: Hướng dẫn triển khai vai trò QA mới để phát hành nhanh hơn

提高生产力：实施新的 QA 角色以加快发布速度的指南

The possibility of working remotely marked a new chapter in the lives of many. You don’t have to go to the office or be confined to a specific location anymore. All a remote professional needs is a laptop and smartphone, and the world is open for travel and work. Thousands have already tried workationing. But it can be challenging to be as effective as you are in the office when you’re on workation. So, here are five apps that help me stay as productive as possible while working and traveling. Evernote When working remotely, you may use notebooks, sticky notes, a whiteboard, and other tools, in addition to your laptop and smartphone, to keep track of important tasks. But the fewer belongings you take with you on workation, the better. Because, for one thing, they make your luggage heavier, and for another, you may lose them while you’re moving around. An app like is the most effective solution to this, as it lets you organize all your notes in one place and is available anywhere in the world with Internet access. You can take down ideas in the format of searchable notes, attach various types of media to them, and collect them in digital notebooks. Evernote iScanner Imagine that you need to scan a document (such as a contract or an ID) but can’t find a computer scanner in your hotel or even anywhere nearby. In such situations, the app comes in handy. With it, all you need to get a quality scan is your smartphone. The app removes any type of distortions, like folds or crumples, makes the document clearer, and even erases captured fingers. Besides, you can edit and sign your scans on the go, which saves you a lot of time. iScanner Any.do One of the cornerstones of a productive day is listing your tasks and prioritizing them. I recommend planning your day with the app. It allows you to create a to-do list and visualize your tasks. When you see that you’ve planned ten projects for the day but then only manage to accomplish three by lunch, this makes you rearrange your priorities for the rest of the day and reassess how you spend your time. Plus, moving cards from the ‘To-do’ section to ‘Done’ is so gratifying! Any.do Wifi Analyzer A quality internet connection is key for remote work. A poor connection may not only prevent you from taking an active part in a video call, it may not even let you work with your files in the cloud. To avoid such situations, I recommend using . It measures your real-time network speed and traffic in seconds. This app will help you check the quality of the connection before you choose a spot to work, giving you confidence in your accessibility and effectiveness during the day. Wifi Analyzer Yelp When you arrive at a new place, you may want to find good cafés and restaurants to work from. And, if something happens in the apartment you're renting, you also may need to contact some local businesses, such as plumbing or electrical services. To find the most trusted local services, use —an app with the necessary contacts and user reviews. Finding a great professional right away will save you time and leave you satisfied with the results of their work. Yelp I hope that these five apps that always make my workation a little easier and more enjoyable will come in handy to you as well. Of course, these are just a few of the many apps out there, and I could have mentioned apps for flight tickets, hotel bookings, sightseeing, and others, but that’s another story. And even though a workation is an opportunity to have a change of scenery, meet new people, and discover new places, work itself should always remain a priority.

5 Handy Apps for Boosting Productivity During a Workation

To ensure a good user experience in web development, prioritize web application performance. File compression is a powerful optimization tool that reduces data size and leads to faster page loading times. Gzip and Brotli are effective compression tools that can be used with Webpack Compression Plugin to achieve fast loading speeds on all devices and network conditions. Here's a little about me: I am skilled in improving performance and enjoy enhancing applications to save time and resources. I always strive to optimize everything I work on. I organized my approach by topics. Here are some suggestions: Optimize Your Web Experience: Master the Art of Lazy Loading Imports Boost Your JavaScript Dockerfile for Top-Notch Production Deployments Today, you will get an in-depth understanding of webpack and some valuable tips. By going deeper, you will understand what you are doing, with answers to questions like "Why?" and "How does it work?" Let's analyze the events that we will influence when building the application. Research Scope Before we start, let us quickly remind you what comes next after your code passes the PR. Building project on the server. The builder creates a bundle of files the user will open in its browsers. The customer opens the address of your application, and the browser sends HTTP requests. In response browser downloads bundle files. Browser run bundle file. All code of your application is accessible for the browser. DOM is creating and displaying and interacting with the site. What is a bundle file? A bundle file(-s) is a collection of resources such as JavaScript code, CSS styles, and images combined and optimized to make loading and running on the client side more convenient. Thus browsers have tools for demodulating the code into digestible user form. The browser understands what needs to be done to turn the code into a website or application. The speed of demodulation and the construction of the DOM depends on the computing power of the user's device. As a developer, you may not be able to control the performance of the browser or the power of the user’s device. However, there is a crucial aspect where you and/or the DevOps engineer can significantly impact the bundle size. This is the focus of our analysis, and the solution lies in compression. What is Compression? Theoretical diving Compression of bundle files during Webpack build is the process of using compression algorithms, such as Gzip and Brotli, to reduce the size of the final bundle files containing all the code and resources of your web application. This improves website loading performance, reduces network traffic, and allows your application to run faster on the client side. Compression algorithms identify repeating data and replace it with shorter symbols, reducing the amount of transmitted information and file size. Gzip and Brotl are safe for compressing text and code but may cause a quality loss in images and videos. To use the compression algorithm when building a project, or a web application, you have to support it: Server Configuration: Ensure that your web server supports gzip/brotli compression. Usually, this can be set up through the web server's configuration files. Compression Configuration: Configure the server to compress specific types of files when transmitting them to clients. Typically, text files like HTML, CSS, JavaScript, JSON, XML, and others, such as SVG, are compressed. File Compression: During the project build, you can use compression tools like Gulp or Webpack to automatically compress and optimize your files before hosting them on the server. Setting Headers: After compressing the files, the server sends the appropriate HTTP headers using Gzip or Brotli compression. The «Content-Encoding: gzip» and «Content-Type» headers help clients understand the compressed data and how to unpack it correctly. A small schema for representation is below. The good news is that most browsers support “under the hood” for decompressing Data from the server and do decompression “on the fly.” If you have specific browser support, please, check compatibility here on this . link Compression — First Steps So, now we need to comprehend how to Compress data. I am using my well-known from a previous topic about lazy imports and will now focus on applying the gzip algorithm. Angular pet application I will break down the steps into smaller subtopics to make them easier to follow. Switch from the standard builder to a custom one. This will help us better understand its benefits. It is necessary to generate a specific configuration file for webpack. To prepare for work, it is necessary to install additional libraries to facilitate the workflow. Create an application using the updated configuration. Run and compare performance. Standard Builder vs. Custom Builder Angular Approach The process of building the app is outlined in the configuration file. Within the configuration, a package offers a standard build setup for most applications. angular.json build Angular "projects": {
  ...
  "[project]": {
    ...
    "architect": {
      "builder": "@angular-devkit/build-angular:browser", We require something else. We aim to adjust the settings and enhance the functionality of the standard build. We need greater flexibility in managing the build process and Webpack configuration. We strive for excellence in our standards. The good news is that a package already exists for non-standard developers. Install and change the builder string. custom-webpack npm i @angular-builders/custom-webpack "architect": {
  ...
  "build": {
    "builder": "@angular-builders/custom-webpack:browser",
    "options": {
      "customWebpackConfig": {
         "path": "./custom-webpack.config.js",
         "replaceDuplicatePlugins": true
      },
      ...
    }
  }, The place where we change the webpack configuration is a particular JavaScript file. Here, we have the options and sections. To specify the path to the new webpack configuration file, you should create a new file with the same name or a different one and link it to the path. customWebpackConfig React Approach For React developers, you can make the same things without a custom builder and use this command to configure your build. {
  "scripts": {
    "build": "webpack --config webpack.config.ts"
  }
} Webpack Configuration File We need to install the special CompressionPlugin (the main star of our topic) before we can proceed with configuring the file. custom-webpack.config npm install compression-webpack-plugin --save-dev Add configuration to custom-webpack.config.js is: const CompressionPlugin = require("compression-webpack-plugin");

module.exports = {
  mode: "production",
  plugins: [
    new CompressionPlugin({
      // Be very carefully with 'true', sometimes bug happens
      deleteOriginalAssets: false,
      algorithm: 'gzip',
      test: /\.(js|css|html)$/,
    }),
  ]
}; “deleteOriginalAssets: true.” We will address this further in a later discussion. We have installed the CompressionPlugin dependency and set it up. We aim to use the gzip algorithm for compression, and we have specified the file extensions we want to compress in the Regular Expression format. This library supports algorithms such as Gzip and Google’s Brotli. Brotli provides better compression than Gzip, especially for text-based files such as HTML, CSS, and JavaScript. It can achieve compression rates 20-30% higher than Gzip. For instance, JavaScript and CSS files are typically compressed to 30-40% of their original size with Gzip, whereas Brotli can achieve 50% or more compression. Build Process Our upcoming project will be built using custom-webpack and its new configuration file, custom-webpack.config.js. Let’s build: ng build Let's take a look at the dist/lazy-import folder. Here, you will find both the original and … the compressed gzip files. What? Copies? Wait, wait, wait. An inquiry that DevOps engineers may have is whether keeping both clean and compressed application files will increase the server's application size. The answer is yes- the size will be 🔼X plus (compression ratio multiplied by X). Prior to this, only clean application files with X megabytes were retained. 💾 Can I only save compressed files and securely use the application? , the answer is affirmative. Yes Compress and delete Overall, there are libraries available to remove non-compressed files automatically. The CompressionPlugin has a setting called deleteOriginalAssets set to true, which you may want to check. However, some forums report that this feature can accidentally delete important files, causing the application to fail. However, in this tutorial, we will create a script for this purpose and integrate it into the process. This will help you better understand how these libraries function behind the scenes. To begin, create a new JavaScript file in the project's root, like , and add the script. remove-non-compressed-files.js const fs = require('fs');
const path = require('path');

const directoryPath = './dist/lazy-import';

fs.readdir(directoryPath, (err, files) => {
  if (err) {
    console.error('Error reading directory contents:', err);
    return;
  }

  // Filter files to keep only those with the .js extension
  const jsFiles = files.filter((file) => path.extname(file) === '.js');

  // Delete each file
  jsFiles.forEach((file) => {
    const filePath = path.join(directoryPath, file);
    fs.unlinkSync(filePath);
    console.log(`Deleted file: ${filePath}`);
  });

  console.log('File deletion completed!');
}); If we add this script to the build process after the build, this script will delete all files with the given condition. ng build && node remove-non-compressed-files.js The result is more respectable now. Fantastic! But storing only compressed files can be helpful but requires a and proper integration with your actual server and build process. careful approach Our upcoming article will discuss Docker Example and cover essential topics such as server Dynamic Compression, Gzip Support, and Caching. A comprehensive understanding of these aspects is crucial; we'll provide the necessary information. Run and compare performance. It is time to make sure that our approach is correct and efficient. Let's set up a local server to handle isolated and built applications using only gzip files. Do the command and read my notes. npm install express compression express-static-gzip A popular web framework for Node.js used to build web applications. It provides features to handle routes, requests, and responses. express: : An express middleware that enables HTTP response compression on the server side using methods like Gzip and Brotli. It reduces the size of data sent to the client, improving application performance. compression — is a middleware that provides static file serving with built-in gzip compression support. It enables to serve static files, such as JavaScript, CSS, and images, while automatically compressing them using gzip or Brotli, if available. This helps to reduce the size of transferred files, leading to faster loading times and improved performance for web applications. expressStaticGzip express express Let's create a server configuration file called. server.js. const express = require('express');
const expressStaticGzip = require('express-static-gzip');
const path = require('path');

const app = express();

// Enable gzip compression for server responses
app.use(expressStaticGzip(path.join(__dirname, 'dist', 'lazy-import'), {
  enableBrotli: true, // Enable Brotli compression
  orderPreference: ['br', 'gz'], // Compression preference order (Brotli will be preferred if available)
  setHeaders: (res) => {
    // Set correct Content-Encoding and Content-Type headers
    res.setHeader('Cache-Control', 'public, max-age=31536000');
  },
}));

// Start the server on port 8080
const PORT = 8080;
app.listen(PORT, () => {
  console.log(`Server running at http://localhost:${PORT}/`);
}); Let’s run the server and open the application. node server.js Performance Tests Prerequisites: I often employ network speed throttling, such as simulating a slow 3G network, to test the performance of applications under real-world conditions that may have limited network bandwidth and higher latency. This helps developers understand how their application behaves and performs when network speed is not optimal. By simulating a slow network connection, they can gain insights into the application's performance in such situations. After each test, kindly delete the dist/[app-name] folder, restart the server following the build, and clear the cache in your browser. This feature only works in browsers that support gzip compression - all modern browsers. IE bye. The initial picture represents the uncompressed version of the application. It resulted in a transfer of . 106 kB The second picture is intended for the compressed application, resulting in a transfer of . 284 kB The results show that around 50% of the 📊 transfer wins and occurs during the ⌛️ DOMContentLoaded event. Scale the application and see more fascinating results. Last look on headers. Our compressed application has Content-Encoding: gzip instead of standard application/javascript. The browser decompresses the data and uses the resulting JavaScript to process and render the page. Decompression Proccess We see that our full-time uploading included a decompression process. The legitimate question is - " ". How long does the decompression process take, and could it potentially affect the user's experience? Gzip and Brotli decompression usually happens very quickly in modern browsers and computers. Compression algorithms are designed to provide high-speed decompression with minimal computational overhead. The speed of decompression depends on several factors: The processing power and available memory impact the decompression speed. Modern computers and mobile devices typically have sufficient resources for rapid gzip data processing. Hardware: Browser developers continuously optimize decompression algorithms to improve performance. This helps speed up the unpacking of data in the browser. Also, one example is that some compression libs have compressed levels for ours. Add and check the balance between bundle size and decompression time. new CompressionPlugin({ ... //for gzip max compression level is 9. Algorithm Optimization: new CompressionPlugin({
    ...
   //for gzip max compression level is 9.
      compressionOptions: { level: 9 },
    ...
    }), : Smaller files decompress faster than larger ones. Therefore, optimizing the size of files and combining multiple small files into one can reduce decompression time. Size of Source Data : If files have been previously decompressed and cached in the browser, it contributes to faster page loading. Caching Overall, gzip decompression is a relatively fast operation and is usually not a performance bottleneck when working with web applications. However, like any performance optimization, monitoring file sizes and ensuring proper caching is essential to maximize the user experience. Conclusion. In summary, by harnessing the full potential of compression technologies and staying dedicated to continuous improvement, web developers can pave the way toward a faster, more sustainable, and more engaging web experience for users worldwide. Before you leave: Please, follow me: LinkedIn Hackernoon

Stories, News & Mentions about BOOSTKPI

Boostkpi’s Stories on HackerNoon