Microsoft will be both competitor and customer of the AI cloud provider

Microsoft to invest $10bn in CoreWeave by end of decade - DCD

CoreWeave Inc., a cloud-computing provider that’s among the most valuable artificial intelligence startups, has closed a $650 million credit facility led by JPMorgan, Goldman Sachs, and Morgan Stanley that is intended to be used to support growth.

CoreWeave Closes $650 Million Credit Facility for AI Cloud-Computing Push - Bloomberg

Google Cloud executive hires and leadership departures in 2024 include new Microsoft VP, AWS AI leader and Google’s Kubernetes general manager.

6 Big Google Cloud Exec Hires And Exits From AWS, Microsoft And CoreWeave

Amid the chip shortage, CoreWeave&#x27;s hoard of high-powered graphics processing units made it one of the largest GPU-specialized cloud-service providers. &quot;If you’re using AI in any part of your life—whether it’s ChatGPT or Microsoft Copilot—there’s a good chance you’re using CoreWeave’s ...

CoreWeave: 2024 TIME100 Most Influential Companies | TIME

Cloud AI startup CoreWeave, funded by Nvidia and private equity firms, is seeking an initial public offering in 2025 to drive its AI and GPU portfolio.

Nvidia-Backed Cloud AI Startup CoreWeave Seeks IPO: Report

CoreWeave

coreweave

How Mixtral 8x7B Sets New Standards in Open-Source AI with Innovative Design

In June, the young family of generative AI (gen AI) unicorns suddenly got a new member. Inflection AI ripped the chart suggested by CBInsights just a month before and got onto the third position, dethroning Cohere AI. So we have to oblige, and after covering two Gen AI leaders OpenAI and Anthropic, we dive into the precocious startup, Inflection AI. Having introduced its core product, an AI assistant named Pi, in May and published a memo about Inflection-1, a Large Language Model ( ) behind Pi in June, Inflection AI took the industry by storm the same month with an astonishing $1.3 billion in funding and a $4 billion valuation – all within its first two years of existence! LLM The company's sudden surge owes much to the attraction of high-profile investors, including technology luminaries such as Bill Gates, Eric Schmidt, and Reid Hoffman (also a co-founder), along with tech giants and . These financial ties extend beyond mere investments. Microsoft is providing its cloud computing infrastructure to strengthen Inflection's capabilities, while Nvidia, in partnership with CoreWeave, is integrating 20,000 of its H100 GPUs for LLM training. Microsoft Nvidia Let's dive deeper and discover more about the founders, their vision, the development behind the model powering Pi, and other crucial details. Finding them was not that easy! Content Overview The starting point of Inflection Financial and legal situation Mission of Inflection and its founders First product Tech behind Pi (LLM development facts) How Inflection is going to make money? Founders’ attitude toward AI risks A few not widely known facts about Inflection's founders 1. The starting point of Inflection According to the official information, Inflection AI has three founders: Karén Simonyan (Chief Scientist and Co-founder) Reid Hoffman (Co-founder) Mustafa Suleyman (CEO and Co-founder) , Inflection hasn't been very transparent with independent media. Some information available that extends beyond mere press releases or PR work stems from the podcast hosted by Reid Hoffman for Greylock Partners (a VC fund where Hoffman has been a partner since 2009). In January 2022, announcing Mustafa Suleyman as a new partner in Greylock Partners, Reid Hoffman : “I’ve known Mustafa personally for more than a decade. We first met at a pub in London. What struck me most was our inspiring and provocative discussion on the power and potential of AI to help solve some of humanity’s most urgent challenges. It marked the beginning of a deep and lasting friendship between Mustafa and I.” Operating with a low profile said More than a decade ago from 2022 was about a time when Mustafa Suleyman (along with Demis Hassabis and Shane Legg) co-founded DeepMind, one of the first AI research labs. In 2014, it was acquired by Google ( ). In January 2022, Suleyman left Google, joined Greylock Partners, and now the main AI R&D is led by the new structure called Google DeepMind in March 2022, Inflection was born (incubated by Greylock). Located in Palo Alto, California. Introducing Inflection AI on the Greylock website, Suleyman called it an "AI-first consumer products company." However, at that moment he didn't clarify exactly what that meant. 2. Financial situation and legal situation In May 2022, in equity financing, though the source of the capital wasn't immediately clear as Inflection didn't respond to the request for more information. Subsequently, when Inflection announced the $1.3 billion funding round in 2023, they clarified in that the first round was raised from Greylock, Microsoft, Reid Hoffman, Bill Gates, Eric Schmidt, Mike Schroepfer, Demis Hassabis, Will.i.am, Horizons Ventures, and Dragoneer. TechCrunch that Inflection had secured $225 million reported the official press release At some point, the information on Inflection's website changed to describe the company as an "AI studio creating a personal AI for everyone." They chose the same legal structure as did Anthropic: , a new type of corporation to balance the interests of shareholders, materially affected individuals, and the public good. public benefit corporation 3. Mission of Inflection and its founders Introducing Inflection, as always, on the Greylock podcast, , "Our mission was to build a tool that is designed to understand the user, rather than the other way around. We wanted it – whether that be connecting with other human beings, making decisions, figuring out how to navigate a problem at work, or more.” Reid Hoffman said to be more than an answerer of questions but a participant in the dialogue that helps you along your life's path Throughout other interviews about Inflection and its first product Pi, they used words such as 'reliable,' 'ever-present,' 'non-judgmental,' 'kind,' and 'supportive’. To get to the point where a chatbot can be all this sounded like a mission. 4. First product In May 2022, Inflection AI shipped its first core product – Pi, a chatbot with the primary goal to serve as a 'Personal AI'. It’s said that Pi uses emotional intelligence to aim for human-like conversations. While its use case is narrower than other services like ChatGPT, Google’s Bard, or Microsoft’s Bing chatbot, it strives to mimic human conversation patterns. Pi's training incorporates data up to November 2022, and it is committed to user privacy, refraining from using identifiable information in training. WhatsApp, Instagram PMs, Facebook Messenger, iMessage, or via the website – the ways to talk to Pi. The main difference from the most commonly used LLM products is the Pi remembers communications with you. The chatbot is also made to resist starting to flirt with a user. Named for "personal intelligence", Pi functions in several roles like teacher, coach, and confidante, designed to align with individual interests and needs. 5. Tech behind Pi Not much is known about the technology behind Pi. However, a month after its launch, the company revealing its foundation on the proprietary LLM, Inflection-1. According to the website, Pi doesn't utilize any third-party language model APIs. released a memo In the memo, the company stated: "The Inflection-1 architecture, dataset, and training procedure are proprietary, and we omit their details in this memo." What we do know is that Inflection-1 demonstrated impressive performance on the Massive Multitask Language Understanding (MMLU) benchmark, surpassing other well-known LLMs such as Meta's LLaMA, OpenAI's GPT 3.5, and Google's PaLM 540B, particularly in answering trivia questions. Additional information from Inflection's partner, CoreWeave's official website, that Inflection AI utilized 3,500 NVIDIA H100 Tensor Core GPUs and a cluster comprising over 40,000 cables and 500 miles of InfiniBand fiber cables for MLPerf Training benchmark competition submission, achieving an impressive 11-minute training benchmark. confirms , that Inflection AI is working on a cutting-edge supercomputer equipped with 22,000 NVIDIA H100 GPUs that is expected to be one of the largest In the industry, right behind AMD's Frontier. However, there's some debate regarding the true computational power of this setup. While Inflection AI claims that these GPUs could provide 22 exaflops in 16-bit precision mode, suggest that the actual performance might not meet these expectations. There are rumors calculations by The Next Platform Though the model is proprietary, the references that are mentioned in the do not list any in-house research papers. official memo 6. How Inflection is going to make money? There are many unknowns at the moment. The developers are exploring different monetization strategies, including the possibility of adding subscription options. Another option is to keep it free but use ads to earn revenue. Currently, Inflection AI is a free platform available to everyone worldwide. On our request to learn more about the business model, neither Reid Hoffman nor Mustafa Suleyman replied. In another podcast with Reid Hoffman, Mustafa Suleyman stated, “ And likewise, we sort of don’t want to drive the kind of addictive dopamine hits of outrage or anger or despair or viral misinformation. These kinds of things have really incentivized people to spend longer on the platform, and sometimes that’s because they’re getting paid for it. We are very, very conscious that this is a moment that needs to change with AI, and that new business models need to emerge to address this." It’s really important to us that we figure out a monetization mechanic that doesn’t incentivize us to keep you engaged or maybe distracted on the platform for as long as possible. 7. Founders’ attitude toward AI risks Reid Hoffman intentionally chose to be positive about AI. , “So in recent months, He has done so in blog posts, television interviews, and fireside chats. He has spoken to government officials around the world. He hosts three podcasts and a YouTube channel. And in March, he published a book, “Impromptu,” co-written with the A.I. tool . The New York Times reports Mr. Hoffman has engaged in an aggressive thought-leadership regimen to extol the virtues of A.I. GPT-4 “I’m beating the positive drum very loudly, and I’m doing so deliberately,” he said. The 55-year-old sits on the boards of 11 tech companies including , which has gone all in on AI, and eight nonprofits. His venture capital firm, Greylock Partners, has backed at least 37 AI companies.” Microsoft , Reid Hoffman said that the , He thinks that since technology 'always puts tools in the hands of good and bad actors,' throughout our history, we have learned how to figure this out. In the same interview, he said, ' . In an interview to Washington Post conversations about superintelligence divert us from the real risk which is AI in the hands of bad actors. Part of the reason why I beat the drum so firmly in the positive camp is not that I'm risk unaware, and I think we need to navigate them, but oh, my gosh, every month we delay an AI tutor and doctor, for everyone who has a smartphone or has a friend who has a smartphone, just think about the human suffering that is caused by that delay, and that is super important This story was at Turing Post. Consider to receive new episodes straight to your inbox. originally published ing subscrib 8. A few not widely known facts about Inflection Founders Reid Hoffman Reid Hoffman ( ) is among a small group of interconnected tech executives leading the A.I. charge, many of whom also led the last internet boom. on Twitter He is part of the "PayPal Mafia," a group of former PayPal employees who went on to become successful entrepreneurs and investors in the tech industry. Reid Hoffman's entrepreneurial journey has been marked by resilience and determination. After his failure with SocialNet, he became obsessed with scaling and invested heavily in the social networking phenomenon, which eventually led to the co-founding of LinkedIn, the world's largest professional network with a diversified revenue model that includes subscriptions, advertising, and software licensing. He also led the company to its successful acquisition by Microsoft. One of the first investors in OpenAI, he was sitting on its board until March 2023, stepping down to avoid any conflicts of interest with Inflection AI. He still sits on the Board of directors at Microsoft. Reid Hoffman is a proponent of effective altruism (founders of OpenAI and Anthropic have strong ties with EA also), which advocates for using wealth and resources to create the best for the most people. In 2017, he : “If you have wealth, and at least a reasonable moral compass, to ask, “How do you create the most good for the most people?” I’m not saying this is the perfect system. But what’s the better system?” said to the Atlantic Reid Hoffman is a prominent philanthropist and has invested in impact investments through charitable entities. over $1.5 billion Hoffman is an avid fan of Dungeons & Dragons, and he attributes its influence to his thinking about life as a heroic quest with various players coming together to accomplish something. He believes in the power of friendship and considers it the true center of his life. Reid views friendship as a skill that can be learned and improved through asking questions and interacting with others. Mustafa Suleyman ( ) on Twitter He dropped out of Oxford University at 19 to start the Muslim Youth Helpline, a telephone counseling service for Muslims in the UK. He worked as a policy officer on human rights for Ken Livingstone, the Mayor of London, before founding Reos Partners, a ‘systemic change’ consultancy that uses methods from conflict resolution to navigate social problems. In February 2016, Suleyman launched DeepMind Health at the Royal Society of Medicine, a project that builds clinician-led technology for the NHS and other partners to improve frontline healthcare services He received the Commander of the Most Excellent Order of the British Empire (CBE) in 2019 for his influence in the UK technology sector and also accepted the Silicon Valley Visionary Award the same year Within DeepMind, he established the DeepMind Ethics & Society research unit, aiming to study the real-world implications of AI and promote ethical practices in the technology industry. One of the founding members of the Partnership on AI — an organization set up in September 2016 to ensure that AI is developed safely, ethically, and transparently — along with Facebook's AI head Yann LeCun, Microsoft Research director Eric Horvitz, and several others. Suleyman recently proposed a modern Turing test, , “to measure whether a machine has achieved ACI, you give an AI $100,000 and see if it can turn the seed investment into $1 million. To do so, the bot must research an e-commerce business opportunity, generate blueprints for a product, find a manufacturer on a site like Alibaba and then sell the item (complete with a written listing description) on Amazon or Walmart.com.” Suleyman expects AI will pass this more practical threshold sometime in the next two years. according to Bloomberg Karén Simonyan There is much less information about Karén Simonyan. Karén is not active on social media, he has no Twitter, a barely filled , and a well-developed . LinkedIn account Google Scholar page Education & Career History: 🔬 Chief Scientist, Inflection AI (2022 – Present) 🔬 Research Scientist, DeepMind (2014 – 2022) 🔬 Postdoc, University of Oxford (2013 – 2014) 🔬 Ph.D. student, University of Oxford (2009 – 2013) While doing his postdoc in 2013, in Oxford, Karén Simonyan along with Andrew Zisserman created the VGGNet image processing framework. VGGNet won the prestigious ImageNet Challenge. To commercialize it, they created a company Vision Factory AI, which was soon acquired by a not-very-known at that time startup DeepMind Technologies, where in 2014 Simonyan became a Principal Research Scientist. There, he played a key role in various AI breakthroughs, such as AlphaZero, AlphaFold, WaveNet, BigGAN, and Flamingo. While in DeepMind, Karen authored and co-authored numerous publications, including five papers in Nature and Science, accumulating an impressive 180,000 citations. Since he co-founded Inflection AI, only one paper was published with Karén Simonyan as a co-author . HiP: Hierarchical Perceiver That’s it for now. The information about Inflection AI will be constantly updated… Thank you for reading, please feel free to share with your friends and colleagues. This story was at Turing Post. Consider subscribing to receive new episodes straight to your inbox. originally published

The Inflection AI Story: How a Secretive Unicorn Reached a $4 Billion Valuation

Reducing AWS S3 .NET client LOH allocations by 98% Contents Problem discovery Why is it a problem? Introducing the best magic number — 81,920 Idle hands Just one more thing TLDR — Give me the good stuff Footnotes Problem discovery One of the things we do at is help people find their next vehicle. That usually involves customers seeing what vehicle they are buying — I mean, would you buy a car without seeing what it looks like? The application that holds this responsibility is worst offender for obscene amounts of allocations, time spent in GC, and generally eating RAM like the Cookie Monster eats well…cookies. Codeweavers the Every now and then we like to take a memory dump of this application from our production environment. We have done this enough times that we have automated the most common diagnostics steps we take and bundled them into a little tool¹ called ADA (Automated Dump Analysis). If you are interested you can find the tool and all the code talked about in this article . here here One of the analysers that we run is to dump all the byte[] arrays found on the Large Object Heap (LOH). After running that analyser against our eight gigabyte memory dump, we found several hundred byte[] arrays with a length of 131,096 or 131,186. Well that is pretty odd. Opening some of the files in just presented us with lots of random characters. Notepad++ Throwing the out of the window for a second, I decided to mass rename all the dumped byte[] arrays to *.jpg - hey presto some of the files were now displaying thumbnails! On closer inspection, around 50% of the files were images. The other 50% failed to open as an image at all. Opening a handful of the non-image files in Notepad++ showed that they all had a line similar to this right at the beginning of the file:- scientific method 0;chunk-signature=48ebf1394fcc452801d4ccebf0598177c7b31876e3fbcb7f6156213f931b261d Okay, this is beginning to make a little more sense. The byte[] arrays that have a length of 131,096 are pure images. The byte[] arrays that are not images have a length of 131,186 and have a chunk-signature line before the rest of the contents. I would the signature is a SHA256 hash of the contents. guess Before we go any further, it is worth establishing how busy this application is with image processing. All of our image processing is distributed across our farm using AWS and . Using we can see that easily:- SNS SQS CloudWatch Metrics Okay, so busy. It is worth noting that before performance centric work is carried out, always establish how often the code is hit and the current costs. If a code path has a high cost (e.g. takes twenty seconds) but is only hit once a day, then it is not worth investigating. However, if the same code path is hit a lot (e.g. a million times a day) then it is definitely worth investigating. fairly any At this point I had two culprits in mind. We have already established the application in question does a lot of image processing. But there are a few moving parts and two ways of kicking off the image processing:- Images are pushed to us We pull images from a SFTP After that we transform the image and then upload it to . At this stage I was leaning towards the SFTP, as it probably needed to verify each chunk it received from the server. But following my hunches has lead me on a wild goose chase before so ignoring my hunch I plugged chunk-signature into Google and smashed enter. Google pointed to AWS S3 as . But that is just theory, we need to prove that. AWS S3 the culprit <a href="https://medium.com/media/f15095d201b5b069bd5fcbd95eeca6b2/href">https://medium.com/media/f15095d201b5b069bd5fcbd95eeca6b2/href</a> If we upload the ten times and use to view the LOH we see an interesting pattern:- same image dotTrace It looks like every time we call PutObject on the AWS S3 .NET client there is a fixed cost of 0.3 MB. This is a problem because it means every time you use PutObject, you are paying a high cost of 0.3 MB per upload. Just to make sure; what happens if we increase the number of times we upload from ten times to one hundred times? Yes, we can definitely say that for every invocation of PutObject a costly allocation of 0.3 MB is made. Going one step further and dumping the process using :- ProcDump procdump64.exe -ma -64 AWS-S3.exe Running the dump file through ADA we see the exact same characteristics of there being two groups of byte[] arrays; 50% have a length 131,096 and the other 50% have a length of 131,186. Half the files are images when renamed, and half the files have the chunk-signature starting line. At this point we are certain that the AWS S3 .NET client is allocating byte[] arrays directly onto the LOH; and that a problem. is Why is it a problem? The LOH is a region of memory that is but never — though as of .NET v4.5.1 — word of warning compaction of the LOH is expensive; around . A good rule of thumb is that short-lived objects should make it onto the LOH. collected compacted compaction is now possible 2.3 milliseconds per megabyte never Objects that are equal to or greater than bytes go straight onto the LOH. The LOH operates very differently from other regions of memory. Other regions of memory get collected and compacted regularly meaning that you can just add new objects to the end after the garbage collector runs. Whereas the LOH tries to fit newly allocated objects in free space left after dead objects are discarded. This works fine if the newly allocated object is the exact same size or smaller as the free space. If a space can not be found then the LOH has to grow to accommodate that object. 85,000 It helps to think of it like a bookshelf; in other regions of memory, books that are no longer used are simply thrown away and the remaining books are pushed together and any new books go at the end of the book shelf. Within the LOH that is not possible, instead books (objects) are thrown away, and the number of pages that used to be in that space (bytes) is recorded and the next time a book gets allocated to the that shelf (the LOH) it attempts to find an empty space that can hold that many pages (bytes). If the shelf can not accommodate the newly allocated book (object) then the shelf must be extended to hold that new book (object). The garbage collector will collect dead objects from the LOH, and in the mean time new objects are being allocated to the LOH. This can lead to a situation over a lifetime of a long running application where the LOH size has grown to a few gigabytes (because new objects did not fit into existing empty space) but actually only contains a few alive objects. This is known as LOH fragmentation. We were lucky in this situation as the byte[] arrays that made it onto the LOH had two sizes; 131,186 and 131,096. This means that as old objects of either size died and were collected, newly allocated objects were just the right size to slot right into the empty space. extremely Okay, back to the fun stuff. Introducing the best magic number — 81,920 Thanks to dotTrace we were able to establish exactly what was causing the LOH fragmentation. It also showed us that the fixed cost of 0.3 MB per invocation of PutObject happened inside of the constructor for ChunkedUploadWrapperStream:- A quick visit to in aws-sdk-net repository. Shows that two byte[] arrays are created with a length of at least 131,072:- that file <a href="https://medium.com/media/2309e8b4dbb4e731387d7636c5ea73d4/href">https://medium.com/media/2309e8b4dbb4e731387d7636c5ea73d4/href</a> This is exactly why these byte[] arrays are allocated directly to the LOH, they are above the LOH threshold (85,000 bytes). At this point there are a few possible solutions:- Use to rent two byte[] arrays from a pool of byte[] arrays System.Buffers Use and operate directly on the incoming stream using a pool of Stream's Microsoft.IO.RecycableMemoryStream Expose DefaultChunkSize so that consumers of the API can set it themselves Lower DefaultChunkSize to a number that is below LOH threshold (85,000 bytes) The first and second solutions are probably the ones with the biggest wins to be had, but that would require a large pull request and introducing a dependency that the library maintainers might not want². The third solution means that the consumers of the library have to know about the problem and set it to a reasonable number to avoid LOH allocations. No, it seems like the fourth solution is the most likely to get accepted and has the least possibility of breaking existing functionality. All we need is a number that is lower than 85,000, normally something like 84,000 would have been perfectly suitable. However, a few weeks prior to discovering this problem I was poking around (investigating a different issue) when I stumbled across :- Reference Source this gem <a href="https://medium.com/media/5bc56da17854a3be9aa0d5e37336a6cf/href">https://medium.com/media/5bc56da17854a3be9aa0d5e37336a6cf/href</a> Windows memory pages are , so picking a multiple of that which falls under the LOH threshold (85,000 bytes) makes complete sense. Time to fork, branch, , and make a . 4,096 bytes in size create an issue pull request Luckily, we can make the change locally³ and see what the benefits are. Statistics for one hundred uploads of the same image via PutObject:- <a href="https://medium.com/media/411e40a3134f9170f6a45a69e31296ac/href">https://medium.com/media/411e40a3134f9170f6a45a69e31296ac/href</a> Idle hands Whilst waiting for my pull request to be reviewed I decided to poke around the AWS S3 documentation and I stumbled across the concept of . That sounds interesting! Creating V2 of the uploader:- pre-signed URLs <a href="https://medium.com/media/9ab7508ba03a714cfe3e1150fef36913/href">https://medium.com/media/9ab7508ba03a714cfe3e1150fef36913/href</a> We see it has the following statistics when uploading the same file one hundred times:- <a href="https://medium.com/media/98f289c65d76d26f1b56c0e45edd7f12/href">https://medium.com/media/98f289c65d76d26f1b56c0e45edd7f12/href</a> That is pretty awesome, and all we actually had to do to achieve that gain was read the documentation! Well, not true, you have the benefit of reading a summarised article with all the juicy bits. The work you see here took place over the course of a week, slotted in between client work. There is a small downside to using GetPreSignedURL in that if the GetPreSignedUrlRequest is modified and the WebRequest is not modified accordingly then AWS will return (e.g. removing the XAmzAclHeader on the WebRequest). This is because the client-side hash and the server-side hashes no longer match. HTTP 403 Forbidden Just one more thing Thanks to my I have learnt what is — something I do to myself quite a lot. At this stage I was feeling that giddiness about what else could be shaved off, I was wholly looking at the 0.4 MB remaining on the LOH. Again, dotTrace points us in the direction of code path causing that 0.4 MB allocation to the LOH:- last article nerd sniping Yikes, . Quietly backing away and trying a different tact; we know a pre-signed URL looks something like this:- that looks fairly serious https://##bucket_name##.s3.##region_name##.amazonaws.com/##path##/##file_name##?X-Amz-Expires=300&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=##access_key##/20180613/##region_name##/s3/aws4_request&X-Amz-Date=20180613T233349Z&X-Amz-SignedHeaders=host;x-amz-acl&X-Amz-Signature=6bbcb0f802ad86022674e827d574b7a34a00ba76cd1411016c3581ba27fa5450 We should be able to generate that URL ourselves as AWS has very kindly published their . At this point I will admit that I was ready to accept defeat and just leave the 0.4 MB kicking about on the LOH. I really did not feel like the reams of code I was about to write to eliminate that remaining 0.4 MB was going to be worth it. signing process possibly That was until I spotted of what I wanted. With considerably less effort required on my part; V3 was born:- an example <a href="https://medium.com/media/c4e3c000a9cd56ef26b0ac0739234bbf/href">https://medium.com/media/c4e3c000a9cd56ef26b0ac0739234bbf/href</a> V3 was just an experiment to see what was possible, given how small the gains and how much code there is to maintain it is not something we would actually use in production code. The discovery of pre-signed URLs is the main win here:- <a href="https://medium.com/media/30bffa1808cbbb00853f1f420941a214/href">https://medium.com/media/30bffa1808cbbb00853f1f420941a214/href</a> Meanwhile my pull request had been and released in version 3.3.21.19 of . Quick overview of the timeline:- merged AWSSDK.Core 2018–03–07 — Issue created on the aws-sdk-net repository 2018–03–13 — Pull request sent in 2018–03–29 — Pull request merged 2018–03–29 — New version of AWSSDK.Core released to NuGet I love open source. TLDR — Give me the good stuff Versions of AWSSDK.Core below 3.3.21.19 caused a fixed cost of 0.3 MB per invocation of PutObject on the AWS S3 .NET client. This was rectified in versions 3.3.21.19 and above. For particularly hot code paths, it is worth exploring the use of GetPreSignedURL on the AWS S3 .NET client as that dropped LOH allocations by 98% in our context and use case. Find me , , or . Twitter LinkedIn GitHub Footnotes ¹ Another reason be that WinDbg still scares me. may ² That being said a to take advantage of .NET Core goodness recent conversation has been started ³ Make sure to build in release unlike a someone — okay it was me certain Originally published at dev.to .

AWS S3 .NET Client High Memory Usage

Reducing memory allocations from 7.5GB to 32KB Contents Context of the problem Establishing a baseline Easy win 1 Easy win 2 Splits are never cool Lists are not always nice Pooling byte arrays Goodbye StringBuilder Skipping commas The war between classes and structs Goodbye StreamReader TLDR — Give me a table Context of the problem is a financial services software company, part of what we do is to enable our customers to bulk import their data into our platform. For our services we require up-to-date information from all our clients, which includes lenders and manufacturers across the UK. Each of those imports can contain several hundred megabytes uncompressed data, which will often be imported on a daily basis. Codeweavers This data is then used to power our real-time calculations. Currently this import process has to take place outside of business hours because of the impact it has on memory usage. In this article we will explore potential optimisations to the import process specifically within the context of reducing memory during the import process. If you want to have a go yourself, you can use to generate a sample input file and you can find all of the code talked about . this code here Establishing a baseline The current implementation uses StreamReader and passes each line to the lineParser. <a href="https://medium.com/media/9e6d1b9676d1508fada0eed416cf5d04/href">https://medium.com/media/9e6d1b9676d1508fada0eed416cf5d04/href</a> The most naive implementation of a line parser that we originally had looked something like this:- <a href="https://medium.com/media/7efd374838b638e413fef1a61815da58/href">https://medium.com/media/7efd374838b638e413fef1a61815da58/href</a> The ValueHolder class is used later on in the import process to insert information into the database:- <a href="https://medium.com/media/afcb770d65486f2cdff212f82f202126/href">https://medium.com/media/afcb770d65486f2cdff212f82f202126/href</a> Running this example as a command line application and enabling monitoring:- <a href="https://medium.com/media/b18f5d9ef4c9b008ba847df8f1522b37/href">https://medium.com/media/b18f5d9ef4c9b008ba847df8f1522b37/href</a> Our main goal today is to reduce . In short, the less memory we allocate, the less work the garbage collector has to do. There are that garbage collector operates against, we will also be monitoring those. Garbage collection is a complex topic and outside of the scope of this article; but a good rule of thumb is that short-lived objects should never be promoted past generation 0. allocated memory three generations We can see V01 has the following statistics:- Took: 8,750 ms
Allocated: 7,412,303 kb
Peak Working Set: 16,720 kb
Gen 0 collections: 1809
Gen 1 collections: 0
Gen 2 collections: 0 Almost 7.5 GB of memory allocations to parse a three hundred megabyte file is less than ideal. Now that we have established the baseline, let us find some easy wins… Easy win 1 Eagle-eyed readers will have spotted that we string.Split(',') twice; once in the line parser and again in the constructor of ValueHolder. This is wasteful, we can overload the constructor of ValueHolder to accept a string[] array and split the line once in the parser. After that simple change the statistics for V02 are now:- Took: 6,922 ms
Allocated: 4,288,289 kb
Peak Working Set: 16,716 kb
Gen 0 collections: 1046
Gen 1 collections: 0
Gen 2 collections: 0 Great! We are down from 7.5GB to 4.2GB. But that is still a lot of memory allocations for processing a three hundred megabyte file. Easy win 2 Quick analysis of the input file reveals that there are 10,047,435 lines, we are only interested in lines that are prefixed with MNO of which there are 10,036,466 lines. That means we are unnecessarily processing an additional 10,969 lines. A quick change to V03 to only parse lines prefixed with MNO:- <a href="https://medium.com/media/a8b5adb1b6348a4a8e06f75819431027/href">https://medium.com/media/a8b5adb1b6348a4a8e06f75819431027/href</a> This means we defer splitting the entire line until we know it is a line we are interested in. Unfortunately this did not save us much memory. Mainly because we are interested in 99.89% of the lines in the file. The statistics for V03:- Took: 8,375 ms
Allocated: 4,284,873 kb
Peak Working Set: 16,744 kb
Gen 0 collections: 1046
Gen 1 collections: 0
Gen 2 collections: 0 It is time to break out the trusty profiler, in this case :- dotTrace Strings in the .NET ecosystem are immutable. Meaning that anything we do to a string returns a brand new copy. Therefore calling string.Split(',') on every line (remember there are 10,036,466 lines we are interested in) returns that line split into several smaller strings. Each line at minimum has five sections we want to process. That means in the lifetime of the import process we create at least 50,182,330 strings..! Next we will explore what we can do to eliminate the use of string.Split(','). always Splits are never cool A typical line we are interested in looks something like this:- MNO,3,813496,36,30000,78.19,, Calling string.Split(',') on the above line will return a string[] containing:- 'MNO'
'3'
'813496'
'36'
'30000'
'78.19'
''
'' Now at this point we can make some guarantees about the file we are importing:- The length of each line is not fixed The number of sections that are delimited by a comma are fixed We only use the first three characters of each line to determine our interest in the line This means there are five sections we are interested in but the section length is unknown Sections do not change locations (e.g MNO is always the first section) Guarantees established, we can now build a short lived index of the positions of all the commas for a given line:- <a href="https://medium.com/media/0aecfff5995cd2eb17c6aa7c675096d7/href">https://medium.com/media/0aecfff5995cd2eb17c6aa7c675096d7/href</a> Once we know the position of each comma, we can directly access the section we care about and manually parse that section. <a href="https://medium.com/media/8f55a2295ff28a6bed29cb9e23cd5773/href">https://medium.com/media/8f55a2295ff28a6bed29cb9e23cd5773/href</a> Putting it all together:- <a href="https://medium.com/media/f30a9957615f738c0e88487e3afe6e74/href">https://medium.com/media/f30a9957615f738c0e88487e3afe6e74/href</a> Running V04 reveals this statistics:- Took: 9,813 ms
Allocated: 6,727,664 kb
Peak Working Set: 16,872 kb
Gen 0 collections: 1642
Gen 1 collections: 0
Gen 2 collections: 0 Whoops, that is worse than expected. It is an easy mistake to make but dotTrace can help us here… Constructing a StringBuilder for every section in every line is incredibly expensive. Luckily it is a quick fix, we constructor a single StringBuilder on the construction of V05 and clear it before each usage. V05 now has the following statistics:- Took: 9,125 ms
Allocated: 3,199,195 kb
Peak Working Set: 16,636 kb
Gen 0 collections: 781
Gen 1 collections: 0
Gen 2 collections: 0 Phew we are back on the downwards trends. We started at 7.5GB and now we are down to 3.2GB. Lists are not always nice At this point dotTrace becomes an essential part of the optimisation process. Looking at V05 dotTrace output:- Building the short lived index of commas positions is expensive. As underneath any List<T> is just a standard T[] array. The framework takes care of re-sizing the underlying array when elements are added. This is useful and very handy in typical scenarios. However, we know that there are six sections we need to process (but we are only interested in five of those sections), ergo there are at least seven commas we want indexes for. We can optimise for that:- <a href="https://medium.com/media/b40fecfcde0bcb437c158c3e141fcf86/href">https://medium.com/media/b40fecfcde0bcb437c158c3e141fcf86/href</a> V06 statistics:- Took: 8,047 ms
Allocated: 2,650,318 kb
Peak Working Set: 16,560 kb
Gen 0 collections: 647
Gen 1 collections: 0
Gen 2 collections: 0 2.6GB is pretty good, but what happens if we force the compiler to use byte for this method instead of the compiler defaulting to use int:- <a href="https://medium.com/media/c2f9b937862d6531f3ed8701844beb95/href">https://medium.com/media/c2f9b937862d6531f3ed8701844beb95/href</a> Re-running V06:- Took: 8,078 ms
Allocated: 2,454,297 kb
Peak Working Set: 16,548 kb
Gen 0 collections: 599
Gen 1 collections: 0
Gen 2 collections: 0 2.6GB was pretty good, 2.4GB is even better. This is because an int has a much than a byte. larger range Pooling byte arrays V06 now has a byte[] array that holds the index of each comma for each line. It is a short lived array, but it is created many times. We can eliminate the cost of creating a new byte[] for each line by using a recent addition to the .NET ecosystem; Systems.Buffers. Adam Sitnik has a on using it and why you should. The important thing to remember when using ArrayPool<T>.Shared is you must return the rented buffer after you are done using it otherwise you introduce a memory leak into your application. great breakdown always will This is what V07 looks like:- <a href="https://medium.com/media/351c630ba9f6052517a870d21e059baf/href">https://medium.com/media/351c630ba9f6052517a870d21e059baf/href</a> And V07 has the following statistics:- Took: 8,891 ms
Allocated: 2,258,272 kb
Peak Working Set: 16,752 kb
Gen 0 collections: 551
Gen 1 collections: 0
Gen 2 collections: 0 Down to 2.2GB, having started at 7.5GB. It is pretty good, but we are not done yet. Goodbye StringBuilder Profiling V07 reveals the next problem:- Calling StringBuilder.ToString() inside of the decimal and int parsers is incredibly expensive. It is time to deprecate StringBuilder and write our own¹ int and decimal parsers without relying on strings and calling int.parse() / decimal.parse(). According to the profiler this should shave off around 1GB. After writing our own int and decimal parsers V08 now clocks in at:- Took: 6,047 ms
Allocated: 1,160,856 kb
Peak Working Set: 16,816 kb
Gen 0 collections: 283
Gen 1 collections: 0
Gen 2 collections: 0 1.1GB is a huge improvement from where we were last (2.2GB) and even better than the baseline (7.5GB). ¹ Code can be found here Skipping commas Until V08 our strategy has been to find the index of every comma on each line and then use that information to create a sub-string which is then parsed by calling int.parse() / decimal.parse(). V08 deprecates the use of sub-strings but still uses the short lived index of comma positions. An alternative strategy would be to skip to the section we are interested in by counting the number of preceding commas then parse anything after the required number of commas and return when we hit the next comma. We have previously guaranteed that:- Each section is preceded by a comma. And that the location of each section within a line does not change. This would also means we can deprecate the rented byte[] array because we are no longer building a short lived index:- <a href="https://medium.com/media/ca545e99215aa810eb48de7d61bd8880/href">https://medium.com/media/ca545e99215aa810eb48de7d61bd8880/href</a> Unfortunately V09 does not save us any memory, it does however reduce the time taken:- Took: 5,703 ms
Allocated: 1,160,856 kb
Peak Working Set: 16,572 kb
Gen 0 collections: 283
Gen 1 collections: 0
Gen 2 collections: 0 Another benefit of V09 is that it reads much more closer to the original implementation. The war between classes and structs This blog post is not going to cover the difference or the pros/cons of classes vs structs. That topic has been . In this particular context, it is beneficial to use a struct. Changing ValueHolder to a struct in V10 has the following statistics:- covered many times Took: 5,594 ms
Allocated: 768,803 kb
Peak Working Set: 16,512 kb
Gen 0 collections: 187
Gen 1 collections: 0
Gen 2 collections: 0 Finally, we are below the 1GB barrier. Also, word of warning please do not use a struct blindly, always test your code and make sure the use case is correct. Goodbye StreamReader As of V10 the line parser itself is virtually allocation free. dotTrace reveals where the remaining allocations occur:- Well this is awkward, the framework is costing us memory allocations. We can interact with the file at a lower-level than a StreamReader:- <a href="https://medium.com/media/7c7ef9b0bb85fc189f46247b705165ff/href">https://medium.com/media/7c7ef9b0bb85fc189f46247b705165ff/href</a> V11 statistics:- Took: 5,594 ms
Allocated: 695,545 kb
Peak Working Set: 16,452 kb
Gen 0 collections: 169
Gen 1 collections: 0
Gen 2 collections: 0 Well, 695MB is still better than 768MB. Okay, that was not the improvement I was expecting (and rather anti-climatic). Until, we remember we have previously seen and solved this problem before. In V07 we used ArrayPool<T>.Shared to prevent lots of small byte[]. We can do the same here:- <a href="https://medium.com/media/0778f743114d116c4db278c2918ffca0/href">https://medium.com/media/0778f743114d116c4db278c2918ffca0/href</a> The final version of V11 has the following statistics:- Took: 6,781 ms
Allocated: 32 kb
Peak Working Set: 12,620 kb
Gen 0 collections: 0
Gen 1 collections: 0
Gen 2 collections: 0 Yes, only 32kb of memory allocations. That the climax I was looking for. is TLDR — Give me a table <a href="https://medium.com/media/1ed6a5446d0c70436aaf813054c6ec15/href">https://medium.com/media/1ed6a5446d0c70436aaf813054c6ec15/href</a>

Strings Are Evil

367 Stories To Learn About Web Design

Let's learn about via these 52 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Image Processing /Learn Repo 1. How to Print images with TSPL and JavaScript How to Print images with TSPL and JavaScript 2. How Computer Vision Turns Images Into Arrays How Images are turned into arrays in Computer Vision 3. How To Convert WebP Images Into Other Image Formats If you are having trouble loading an image on the web, do not worry because you always have the option of converting it into another format that is more widely 4. What Is Digital Image Processing? With digital image processing, users can enhance digital pictures, extract useful information, and save them in different formats. 5. 5 Best Online Tools to Compress and Optimize Your Images Here are the 5 Best Online Tools to Compress and Optimize Your Images. 6. This Is How Easy it Is to Read and Display Images Working with images is more and more popularized; we are going to take dive into this field, first with the basics and starting with an open CV library. 7. JPEG Sucks but Deep Learning Can Help It Improve In this post, I’ll show how you can reduce image sizes by an additional 20–50% with a single line of code. 8. How Image to Text Technology Can Ease Your Writing Process To collect the information from the images, you do not have to write it down manually.  Instead of that, you can use image-to-text technology. 9. How to Create a High-Performing QR Code Scanner in Flutter In this article, I will take you on a journey through the process of creating a scanner that is not only functional but also visually stunning. 10. Implementing Handwritten Text Recognition to Automate Clinical Trials Manual processing of data (overviews, summaries,study reports) is significantly slowed down and inefficient.AI-driven handwritten text recognition is a solution 11. How To Optimize Images in Django Using ImageEngine Learn how to optimize images in Django using an easy-to-use ImageCDN. ImageEngine automatically serves your images in next-gen formats on the global CDN. 12. 3 Simple Ways to Optimize Your Images and Improve Core Web Vitals Image performance plays a huge role in #CoreWebVitals, which in turn play a huge role in SEO. We'll show 3 quick and simple ways to optimize images. #webperf 13. AVIF Is the Next-Gen Image Format You Will Hear a Lot More About The AVIF Format is the next-gen image format based on the AV1 video encoding format, that reduces file size, while improving #CoreWebVitals. Learn how. 14. New Gatsby Plugin Image: One Of The Coolest Innovations At GatsbyConf 2021 March is off to a great start with the coming & going of GatsbyConf 2021 and the big announcement of Gatsby v3.0! Let's talk about the new Gatsby Image Plugin! 15. 5 Ways to Load Less Imagery to Increase Ranking and Client Happiness Improve your ranking by both search engines and humans by not forcing the client to download unnecessary imagery. 16. Building a FastAPI OCR Microservice In this article, I will try to explain basic ideas on how to create your own OCR service for free, using python, fastAPI, tesseract, redis, celery and docker. 17. Speed up AMP pages with an Image CDN It takes a lot of time and effort to build a profitable website that offers great user experience on any given device type. When you start from a clean slate, you have to take care of everything on your own: browser compatibility, performance, responsive design, discoverability and a whole slew of other factors. 18. Will Generative Models Be The Next Machine Learning Boom? Machine Learning is a rapidly growing and very complex field of study. Generative Models might prove to be a new breakthrough that will make a new boom. 19. How to Integrate an Image CDN in Your React Application One of the most crucial maxims that I internalized during my career in web engineering is that every technical decision embodies tradeoffs. No matter the benefits, you’ll always face certain costs and restrictions: be it creating an abstraction in your code, scaling a distributed data store or choosing between web and mobile app format for the initial product launch. 20. Introducing CropScore: An Image Cropper for Content Creators ✂✨ If you are a content creator, the chances are you have come across stretched, shrank, non-centered or pixelated images after the posting of your material. 21. Improve Core Web Vitals Using Lazy Loading and BlurHash Lazy loading is an important way to improve your PageSpeed Insights. Combined with #Blurhash, you can improve load times. #lazyloading #corewebvitals #SEO 22. Automatic Image Optimization Using the Nuxt Image Component with imgix See how you can use imgix and the Nuxt Image Component to accelerate page speed, simplify image workflow, and transform images at scale. 23. 5 Common Mistakes to Avoid When Adding Images to Your Website We spend a lot of time crafting the perfect pitch in our blog posts and web copies. However, when it comes to visuals, many skimp out and add just about any relevant image available – without getting into the details. 24. How to Implement Gaussian Blurs A Gaussian blur is applied by convolving the image with a Gaussian function. We’ll take the Gaussian function and we’ll generate an n x m matrix. 25. Feature/Permission Policies Overview It's already challenging enough to create a website that is blazing fast, adheres to all the latest best practices and offers optimal experience for various network conditions and device types. As the website and its codebase evolves over time, it can become even harder to maintain the desired user experience over time. To prevent UX deterioration, it’s often a good idea to place guardrails which ensure that crucial website elements, such as images, hold to a certain criteria. In this article, we are going to look at a robust mechanism which helps to steer your website in the right direction - an HTTP header called Feature Policies. 26. Canva Business Model Breakdown: Product, Revenue, Competition, and The Big Question What is Canva? Canva is a software product that helps designers of any skill level to create media 10x faster. 27. Improve the User Experience in Your Mobile App by Using Low-light Enhancement Tech This article introduces how to use AI technology to solve the image quality of Low-Light Images and improve user experience. 28. A Python Library for Face Detection and Extraction with OpenCV Using HOG/Neural Network Many people, including me, use a combination of libraries to work on the images, such as: OpenCV itself, Dlib, Pillow etc. But this is a very confusing and problematic process. Dlib installation, for example, can be extremely complex and frustrating. 29. How Online Classifieds Can Innovate With Immersive Technologies What solutions allow app developers to embed immersive content displaying into a mobile app. 30. Making a Music Visualization with P5JS I had the immense pleasure of attending several creative coding workshops on April 4th. They were streamed live on the SpacyCloud Twitch channel. There were additional sessions involving Hydra, Raspberry Pi, Haskell, and more. However for this post I want to focus on the first session which was a P5JS tutorial. In this post I hope to translate the P5JS tutorial into a written format, for posterity and to share what I learned. I'm going to review what was taught in the live session. Hopefully SpacyCloud will have another live stream in the future so I can catch up on what I missed. Here is the landing page for the event schedule. 31. What it Means to be a Creative Coder Programmers rarely agree on whether or not coding is a creative profession. My interest in coding always stemmed from what I could create with the code. Seeing an interesting visual result from my efforts is usually the most satisfying part. Most programmers are less concerned with how their app looks and more concerned with the functionality. Usually, as long as the app works the way it is supposed to, most programmers are satisfied. 32. PULSE: Photo Upsampling Makes Blurry Faces 60 Times Sharper The new PULSE: Photo Upsampling algorithm transforms a blurry image into a high-resolution image. 33. Dall-E May Be Awesome, but It Still Can't Count. OpenAI's "Dall-E" artificial intelligence can be very frustrating for some professional uses. Here are a few things that Dall-E just can't seem to do. 34. Obtaining Data From Images Using Exif: How To Automate The Process A how-to guide for automating the EXIF data from batch image files. 35. I Found The One Tool To Sort All Your Pictures Like A Boss For years I have been frustrated and I wanted to find a proper way of sorting my pictures. I had photos coming from several devices, and made manual copies on my laptop and on my external hard drive. I ended up with tons of folders and a lot of duplicates. 36. How is Artificial Intelligence Transforming Life Sciences? 4 Main Ways AI has been revolutionizing the life sciences by using its ability to process data quickly to assist with diagnoses and other important medical procedures. 37. Essential Guide to Image Processing with WebAssembly Simple image manipulations with WebAssembly and AssemblyScript in the web browser. 38. Filestack And Other OCR APIs OCR software enables the digitization of essential data in paper documents and photographs. 39. Image Optimization and Transformation with Cloudinary The web is now dominated with more visual content than ever hence the need to consider ways of delivering these images with the smallest possible file size while maintaining optimal visual quality. 40. How to Keep Your e-Store Core Web Vitals Healthy with an Image CDN As we now know, how users interact with your page is just as influential as what is on your pages when it comes to user satisfaction. Search engines, like Google, have wised up to this dynamic by introducing user experience as a significant component in their ranking factors. 41. The Best Ways to Improve Your NFT Marketplace Performance In NFT marketplaces,  page experience, SEO, and credibility can all determine success. Here are 7 ways to improve performance through visual media optimization. 42. Understanding Basic Image Processing Algorithms [A Hands-on JavaScript Tutorial] We've had a lot of digital image processing tools for a long time: Photoshop, Lightroom, GIMP, PhotoScape, and many more. However, in the past few years, one became popular among non-expert users due to its easiness of use and social features: Instagram. Have you ever wondered how Instagram filters work? It is actually pretty simple matrix operations! So simple we can build our own without any external library, just pure and simple HTML + JS. Let's build one now. 43. Use Cases and Roll-Out Tips for Image Recognition in Retail Heavily shattered by the pandemic, the retail sector is on the lookout for innovation. 44. An In-Depth Look at Image Scaling Concepts Considerations upon image sizing for web display 45. Image Processing Algorithms: Adjusting Contrast And Image Brightness Let's take a look at the common approaches for implementing image contrast adjustments. We'll go over histogram stretching and histogram equalization. 46. How to Build an Image Search Engine to Find Similar Images After reading this article, you will be able to create a search engine for similar images for your objective from scratch 47. Intro to Image Processing in Python with Pillow Pillow is Python Imaging Library that is free and open-source an additional library for the Python programming language that adds support for opening, manipulating, and saving in a variety of extension. 48. ColorDetection Module: Python  Color Detection Algorithms Images. That's it. Images. As a point of practicality, take a fashion designer (as a forum member vividly described to me at one point). You are given an image or have an image at your disposal that simply tickles your curiosity and want to incorporate it in one of your new lines. Let's swerve a little into the genetics section. Given a petri dish image for instance, with pigmented bacteria or similar organisms, and you would like to find the abundance of that organism or organisms in this specific image. Get the gist? 49. A Quick Guide to Image Processing in Computer Vision Using OpenCV The image processing library which stands for Open-Source Computer Vision Library was invented by intel in 1999 and written in C/C++ 50. High Quality 8x Upscaling with AI! Have you ever had an image you really liked and could only manage to find a small version of it that looked like this image above on the left? How cool would it be if you could take this image and make it twice look as good? It’s great, but what if you could make it even four or eight times more high definition? Now we’re talking, just look at that. 51. How To Identify Trees with Deep Learning Idea / inspiration 52. How To Resize an Image Correctly in JavaScript JavaScript implementation of so-called Seam Carving algorithm for the content-aware image resizing and objects removal. Dynamic programming approach is applied Thank you for checking out the 52 most read stories about Image Processing on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

52 Stories To Learn About Image Processing

299 Stories To Learn About Seo

302 Stories To Learn About Front End Development

Rotate Expiring Spring Cloud Vault Database Credentials Without Downtime TL;DR The first episode of this series of blog posts can be found here: Hashicorp Vault max_ttl Killed My Spring App It is possible to rotate the Spring Cloud Vault database credentials at runtime for relational databases if you use HikariCP. To do so add a via which LeaseListenener addLeaseListener() calls on the when the dynamic database lease expires requestRotatingSecret() SecretLeaseContainer reacts on a with mode ) by: SecretLeaseCreatedEvent ROTATE setting username and password on the of the HikariConfigMXBean HikariDataSource call softEvictConnections() on the to use the new credentials HikariPoolMXBean New Spring Cloud Vault database secrets without downtime This is the second episode in a series of blog post about how to handle the expiration of Hashicorp Vault generated dynamic database credentials in a Spring application. Spring leaves your application without a database connection when these credentials expire. For more context and some general solutions please check the . first post This time I would like to show you how to renew the database credentials at runtime. So, this time you neither need to nor to nor do you have to , which could potentially result in downtime or not met . regularly restart or redeploy your application use a (probably too) long maximum time-to-live for the credentials programmatically restart the application SLAs As we all know . The costs for the approach I am presenting you this time are: there ain’t no such thing as a free lunch more implementation effort stricter prerequisites (only relational databases supported) The first bullet point is addressed because this post should help you with the implementation. This leaves us with the…​ Prerequisites This approach is only applicable for Spring applications which use . HikariCP Luckily the usual way to store and retrieve data in a relational database with is to use . In Spring Boot 2, Hikari is the default DataSource implementation, which makes it typical setup for using relational databases. Spring Boot Spring Data JPA Show me the code To fulfill the prerequisites, it is enough to depend on the Spring Boot JPA Starter: plugins {
 id( ) version id( ) version }

dependencies {
 implementation( ) runtimeOnly( ) } "org.springframework.boot" "2.2.4.RELEASE" // <1> "io.spring.dependency-management" "1.0.9.RELEASE" // <2> "org.springframework.boot:spring-boot-starter-data-jpa" // <3> "org.postgresql:postgresql" // <4> <1> You don’t have to use the Spring Boot Gradle plugin, but it makes your live easier <2> The Spring dependency-management plugin together with Spring Boot Gradle plugin ensures that all Spring related dependencies have the version being compatible with the Spring Boot version <3> By adding the dependency together with Spring Boot 2.x you automatically get HikariCP <4> In my example I use PostgreSQL but also most other relational databases, like MySQL, would work spring-boot-starter-data-jpa These few lines are basically enough to meet the requirements of using HikariCP, in this case with PostgreSQL. Rotating the expiring database credentials at runtime (Rotation at runtime - Image by Peter H from pixabay ) To rotate the database credentials, which are dynamic secret from Hashicorp Vaults point of view, we have to do following steps: Detect when the database credentials are expiring Get new dynamic database credentials from Hashicorp Vault Refresh the database connections to use the new credentials Detect when the database credentials are expiring To detect when the database credentials are expiring we can use the same approach like we did to . Let’s again autowire the and the database role which is configured as the property to the configuration class: restart the application when credentials expire in the first blog post SecretLeaseContainer spring.cloud.vault.database.role VaultConfig ( leaseContainer: SecretLeaseContainer, databaseRole: String
) { @Configuration class VaultConfig private val @Value( ) "\${spring.cloud.vault.database.role}" private val As before in a method you can then add the additional which does the lease rotation: @PostConstruct LeaseListenener { vaultCredsPath = leaseContainer.addLeaseListener { event -> (event.path == vaultCredsPath) {
 log.info { } (event.isLeaseExpired && event.mode == RENEW) { }
 }
 }
} @PostConstruct private fun postConstruct () val "database/creds/ " $databaseRole if "Lease change for DB: ( ) : ( )" $event ${event.lease} if // TODO Rotate the credentials here <1> <1> When this code path is reached, the database secret expired Next step is to…​ Get new dynamic database credentials from Hashicorp Vault (The credentials should be renewed - Image by pasja1000 from pixabay ) When the lease for the database credentials expire we have to request a new secret. (event.isLeaseExpired && event.mode == RENEW) {
 log.info { }
 leaseContainer.requestRotatingSecret(vaultCredsPath) } if "Replace RENEW for expired credential with ROTATE" // <1> <1> Tells Spring Vault to request a new rotating database secret The returned value of is of type : requestRotatingSecret() RequestedSecret Represents a requested secret from a specific Vault path associated with a lease . RequestedSecret.Mode A can be renewing or rotating. RequestedSecret — Spring Vault Javadoc As mentioned in the Javadoc, the contains the path and the mode of the secret, but it does not contain the secret itself. So how do we get the requested credentials? RequestedSecret We have just requested a new rotating database secret within our own . This listener receives s which are also created, when a new rotating secret is received. This is exactly what we need! So, let’s also react on this kind of event. LeaseListener SecretLeaseEvent (event.isLeaseExpired && event.mode == RENEW) {
 log.info { }
 leaseContainer.requestRotatingSecret(vaultCredsPath) } (event SecretLeaseCreatedEvent && event.mode == ROTATE) { credentials = event.credentials } if "Replace RENEW for expired credential with ROTATE" // <1> else if is // <2> val // <3> // TODO Update database connection <1> The rotating secret is requested <2> The new secret event is a rotating <3> The event contains the new database credentials SecretLeaseCreatedEvent The contains the new credentials requested from Hashicorp Vault. The is an (see code below). SecretLeaseCreatedEvent event.credentials property extension property Details of extracting the secrets safely The contains a with the secrets, so there is no typesafe option to get the database credentials. If for some reason the event does not contain the credentials we are again in the situation, that we cannot contact the database anymore. In that case I would prefer to shut down the application. That’s why we need the to shut down the Spring application. SecretLeaseCreatedEvent Map<String, Object> ConfigurableApplicationContext Let’s add this as another autowired dependency to this class: ( leaseContainer: SecretLeaseContainer, databaseRole: String
) { @Configuration class VaultConfig private val @Value( ) "\${spring.cloud.vault.database.role}" private val Now we can extract the credentials from the event. The extension property returns if the credentials cannot be received. With the we can handle this error case: event.credentials null ConfigurableApplicationContext (credentials == ) {
 log.error { }
 applicationContext.close() }
refreshDatabaseConnection(credentials) if null "Cannot get updated DB credentials. Shutting down." // <1> return @addLeaseListener // <2> // <3> <1> If we cannot get the renewed credentials shutdown the application <2> because of the return from the lambda, is smart casted to a non-nullable value after the block. Kotlin is awesome! <3> here the cannot be and can be used to credentials if credentials null refresh the database connection Now let’s see how the credentials are retrieved from the event within the extension property: SecretLeaseCreatedEvent.credentials: Credential? () { username = ( ) ?: password = ( ) ?: Credential(username, password)
 } : String? { secrets[param] ? String } ( username: String, password: String) private val get val get "username" return null // <1> val get "password" return null // <1> return private SecretLeaseCreatedEvent. fun get (param: ) String return as // <2> private data class Credential val val <1> username and password are extracted using the extension method . If one of the calls return then is returned instead of a <2> the secret is read out of the map and with to a . If the entry does not exist in the map or is not a then is returned get() get() null null Credential safe casted as? String String null Refresh the database connection (Refreshed version of access restriction - Image by Nenad Maric from pixabay ) Now that we know the new credentials we have to ensure that these fresh secrets are used instead of the old ones. {
 updateDbProperties(credential) updateDataSource(credential) } private fun refreshDatabaseConnection (credential: ) Credential // <1> // <2> <1> first update the database system properties <2> finally update the datasource to use the newly created credentials To update the datasource credentials we need the . So, let’s add this also to the constructor: HikariDataSource ( applicationContext: ConfigurableApplicationContext, hikariDataSource: HikariDataSource, leaseContainer: SecretLeaseContainer, databaseRole: String
) { @Configuration class VaultConfig private val private val private val @Value( ) "\${spring.cloud.vault.database.role}" private val Utilizing the we can update the database credentials used by the Spring application: HikariDataSource { (username, password) = credential
 System.setProperty( , username)
 System.setProperty( , password)
} { (username, password) = credential
 log.info { }
 hikariDataSource.hikariConfigMXBean.apply { setUsername(username)
 setPassword(password)
 }
 hikariDataSource.hikariPoolMXBean?.softEvictConnections() ?.also { log.info { } }
 ?: log.warn { }
} private fun updateDbProperties (credential: ) Credential // <1> val "spring.datasource.username" "spring.datasource.password" private fun updateDataSource (credential: ) Credential val "==> Update database credentials" // <2> // <3> "Soft Evict Hikari Data Source Connections" "CANNOT Soft Evict Hikari Data Source Connections" <1> Updating the database system properties is technically not mandatory but ensures consistency, if other parts of the system rely these properties being accurate <2> From the we can get the which allows setting the new credentials <3> As the final step, all connections have to be evicted to use the new credentials HikariDataSource HikariConfigMXBean Summary With these steps the PostgreSQL or other relational database credentials can be rotated, when there Hashicorp Vault leases expire. This works at runtime and without downtime. The logs will look something like this: Lease change for DB: (org.springframework.vault.core.lease.event.SecretLeaseExpiredEvent[source=RequestedSecret [path='database/creds/readonly', mode=RENEW]]) : (Lease [leaseId='database/creds/readonly/wzUQ81Ng4YQcBwdAyLrSZSvd', leaseDuration=PT10S, renewable=true])
Replace RENEW for expired credential with ROTATE
Lease change for DB: (org.springframework.vault.core.lease.event.SecretLeaseCreatedEvent[source=RequestedSecret [path='database/creds/readonly', mode=ROTATE]]) : (Lease [leaseId='database/creds/readonly/ur8C5V1wJMSAdiatwkWXCi03', leaseDuration=PT30S, renewable=true])
==> Update database credentials
Soft Evict Hikari Data Source Connections The complete repository can be found . on GitHub Finally the handling of expiring Hashicorp Vault database secrets in a Spring application is production-ready. Originally published at https://secrets-as-a-service.com on February 18, 2020 Image Source

Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring

the global IT spending is expected to grow to a staggering $3.8 trillions in 2019. So, it’s no wonder if your great idea of an app or a software product is a part of this statistics. And if want to execute it quickly, efficiently, you’re probably looking into IT outsourcing companies to find your perfect match. According to Gartner, And from here the world is your oyster! The options to choose from are more than enough. In fact, the choice is so vast that you might feel overwhelmed. But don’t worry. Essentially, the are two main questions you need to answer — where? and who? Let’s start from where. Today, you can find a web developer virtually in any country or a part of the world. Let’s compare the main destinations for software outsourcing in terms of billing rates and the ease of cooperation. And see why you should consider Eastern European partner for your next project. Top destinations for IT outsourcing companies The image above shows top IT destinations and key countries in each region as well as a rough hourly rate. Now let’s look more closely into global software outsourcing rates of the last year for each job title and how the prices compare to the United States. Eastern Europe here looks like a golden middle. It has been a popular IT outsourcing destination for many years. Which is not surprising, it has a rich talent pool, an impressive engineering expertise and a solid educational background. On top of that, this region seems appealing because of its proximity to Western Europe, decent level of English and a good cultural compatibility. This is a sound choice for companies that are after a good quality of work rather than looking for the cheapest solution. Now that we’ve established the “Where”, let’s look into the ‘Who’ Top 10 IT Outsourcing companies in Eastern Europe Below is the list of reliable software development partners that have developing centers in Eastern Europe and have been recognized by as top firms in Eastern Europe. Clutch.co Perfectial https://perfectial.com/ Lviv, Kyiv, Ivano-Frankivsk, Ukraine Location: Cloud & SaaS applications, Android & iOS mobile applications, Artificial Intelligence & Machine Learning, R&D services, Blockchain development, Enterprise development Services: Perfectial is a business-oriented product design and development company committed to helping businesses kickstart, develop, and succeed. Perfectial has implemented innovative ideas that allowed businesses — large and small — to scale, evolve, and prosper in a range of business domains. Perfectial’s competence in working with modern platforms and frameworks is recognized and highly valued within the U.S. and European markets. The company has over 200 professionals in-house, including project managers, solutions architects, .Net, Java, Ruby on Rails, Python and Node.js developers, UI/UX design and Quality Assurance teams. 2. Ingemark https://www.ingemark.com/ Zagreb, Croatia Location: Software development, Cloud consulting Services: Ingemark is a custom software development agency trusted by some of the largest companies in Southeast Europe, as well as a number of companies from Western Europe, Middle East and the U.S. Founded in 1990., custom software development has been company’s core business since 2006.Ingenmark can help with everything from software design and development to post-launch support and maintenance. 3. SoftwareMill https://softwaremill.com/ Warsaw, Poland Location: Big Data & Machine Learning, Web development, Enterprise solutions, UI/UX design Services: SoftwareMill is a Poland-based software and consultancy company that develops customized IT systems, especially big data, business intelligence, and machine learning solutions. The company specializes in Java, Scala, Akka, the Lightbend Reactive Platform, big data / ML frameworks such as Kafka, Spark, and TensorFlow, SQL & NoSQLdatabases: PostgreSQL, Cassandra, Mongo. 4. CodeWave https://codewave.eu/ Wroclaw, Poland Location: Solutions for creative agencies, Web-based systems and mobile apps, Cloud solutions, Services: CodeWave has been trading since 2008, at first delivering web development and server administration services to local advertising agencies in Poland, later expanding to global markets (EU & US) and delivering complete IT project outsourcing services — providing roles of dev-ops, database administrators, backend and frontend web and mobile applications developers. 5. Sloboda Studio https://sloboda-studio.com/ : Kharkiv, Ukraine Location Back-end development, Front-end development, Project management, Quality assurance Services: Sloboda Studio is a company providing custom web development services. Founded in 2010, they’ve completed 92 projects for the clients from Western European countries. Among company’s clients are Marketplaces, Fintech and Adtech products. Working with them, they apply Agile methodologies in order to ensure the timely delivery and the high quality of the product. 6. GP Solutions https://gpsolutions.com/ Minsk, Belarus Location: Dedicated development team, Web application development, Enterprise software development, Mobile application development, UI/UX design Services: GP solutions is an established European software development house with a proven track record of delivering exceptional quality in custom software development, IT consulting and support. The Headquarters of GP Solutions is located in Munich, Germany, the Sales Office is in Miami Beach, USA and the Development Center is in Minsk, Belarus. GP Solutions has earned the trust of some of the world’s leading brands and most discerning clients in the , , . e-banking travel & hospitality information technology 7. Code Runners https://www.code-runners.com/ Sofia, Bulgaria Location: Software development, Data management, Product development, Machine Learning & AI Services: Code Runners build online products and help implement digital transformation projects around the globe. The company has been in cooperation with many small startups and large business clients. They have experience in web design, web and mobile applications, database management systems and various specific projects aligned with our customers’ wishes. The company is using the newest technologies on the market, which guarantee that the products they build will look and feel equally good on different devices and will remain useful in years to come, besides being easy to maintain and configure. 8. Pulilab http://www.pulilab.com/ Budapest, Hungary Location: Hardware prototyping, Ionic App Development, iOT integration, Drone Technology, App design & development, Digital product creation & conceptualization, Tech specification, Architecture & software design, Web & mobile application development, Big-data API integrations Services: Pulilab is an international creative software and digital media agency that offers innovative solutions crafted by talented developers and designers in Budapest. Their mission is to help companies design and build software products for web, mobile and connected devices. 9. Vega IT Sourcing https://www.vegaitsourcing.rs/ Novi Sad, Serbia Location: Software product development, Website development, Dedicated development team, Embedded software development, Mobile apps development, Front-end development Services: Vega IT is a company that has been developing software solutions for 10 years for foreign markets. The company was founded in 2008 and is still led by its original founders. Besides Software Product Development, their team of more than 150 experienced developers can help you build Enterprise software solutions, CMS Websites, Mobile applications. Moreover, they can help with maintenance and support, Salesforce and Research and Development. 10. Itera https://itera.no/en/ Kyiv, Ukraine, Bratislava, Slovakia Location: Services: Application management, UX design, Business design, DevOps Itera is a specialist in creating digital business, with communication, technology and innovation as core competency tools. Itera delivers projects and services in cross-functional teams to Nordic organisations that see the instrumental contribution that innovation, efficient communication and smart utilization of technology can make to achieve their goals. Itera’s core sectors are banking and insurance, public, healthcare, the service industry, energy and utility. Itera is headquartered in Oslo, and has about 500 employees across the Nordics and in Ukraine and Slovakia. All things considered, Eastern Europe does look like a golden mean. You can find a reliable and transparent partner in any and every country of the region. The billing rates are quite sensible while companies offer a wide range of software development services that will suit everyone’s taste.

Stories, News & Mentions about COREWEAVE

Coreweave’s Stories on HackerNoon

Coreweave's News & Mentions