MLabs

mlabs

Recently, I had been working with a database I had created several months before. For that particular project, I had created a separate database collection for each group of items. At the time, I had a collection of books and articles, sorted by topic, and for some reason, had put all books and articles in separate collections based on topic. Then, I created endpoints for each topic. That way, I can switch out the url collection for the topic the user searched for, like this: In this case, the user searched for “how to put on pants”. But having a collection with the topic of “how to put on pants”, I now can fetch just those 4 results, instead of fetching the entire library. Another component I used to make this work was a word map object. I won’t get into that here, but if you’re interested, here’s where you can check that out: https://hackernoon.com/creating-more-accurate-search-results-for-small-sites-436e64da79b6 Although this works great, I recently wanted to be able to search all of the collections at once, which wasn’t easy with so many separate collections. Basically I wanted to combine all the collections into one single collection. I started off by doing a fetch which listed all of the collections in the database. I’m using mlab, and the way to do that if you’re using mlab is as follows: To get the collections in the specified database: GET /databases/{database}/collectionsExample:https://api.mlab.com/api/1/databases/my-db/collections?apiKey=myAPIKey So your code in implementation would look something like this: fetch(' {if (response.status != 200) {window.alert("Oopsie daisy");return ;}response.json().then(function(data) {let api = data; https://api.mlab.com/api/1/databases/my-db/collections?apiKey=myAPIKey ').then(function(response) If you console.log your api at this point, you’ll have an array of each and every one of your collection names as strings. Perfect. Now I’m going to do a for loop, and loop through those collection names, and do basically the same thing I did way up at the top, fetching in each loop iteration, and replacing the fetch collection name with the current api array item. let arr = [];for (var i = 0; i < api.length; i++) {fetch(' {if (response.status != 200) {window.alert("dag nabbit");return ;}response.json().then(function(data) {let test = data;for (var k = 0; k < test.length; k++) {arr.push(test[k])}});});} https://api.mlab.com/api/1/databases/my-db/collections/api[i]?apiKey=myAPIKey ').then(function(response) Here, we have our databases name in the fetch url (my-db) and in each loop, our collections name changes (api[i]). We fetch the data there and save the data as “test” . then, we have a nested loop, where we loop through test, and push each array item into our array called “arr”. Now, if we console.log “arr” we should have a consolidated array of every single one of our collections in one place, instead of in multiple collections. Next, we’ll need to reformat all the data into a new array of JSON objects. setTimeout(function(){for (var i = 0; i < arr.length; i++) {let text = arr[i].article_text;let text2 = text.replace(/"/g, "");let text3 = text2.replace(/'/g, "");arr[i] = `<p>{"articleAuthor": "${arr[i].article_author}","articleAuthorTitle": "${arr[i].article_author_title}","articleName": "${arr[i].article_name}","articleDate": "${arr[i].article_date}","articleText": "${text3}"}</p>`;document.getElementById('verseText').innerHTML += arr[i];}}, 50000); I used a set timeout function here, to allow all of the fetches in the previous step to complete. Otherwise, I may begin building the JSON data before the fetches complete. I loop through the data contained in the array “arr” and slap it in a paragraph with key:value pairs inside curly braces, then render each object in the DOM. Then I just copy and pasted the output into a json file. As long as there are no errors, then I should be good to go. With mlabs at this point I can just save the file and then export it to an mlabs collection using the command line prompt: mongoexport -h ds#####.mlab.com:#### -d <database name> -c <collection name> -u <user> -p <password> -o filename.json And we’re done. Now, I can access all of my data from a separate database, but I haven’t altered the original structure with separate collections, in case I still want to use that format later. Now, before you comment and say how dumb this method is, don’t worry, I know. But, if you’re like me, and are more comfortable with the front end, this is one possible way to do this. I’m not saying it’s the best, but it’s one way, and it works.

Combining MongoDB Collections

Trump Said WHAT?!?! (Parsing and Searching JSON data)

Previous installments: Day 0: the ground rules Day 1: idea number one Today’s going to be on the shorter side as I’m still busy emailing folks. Also, I’d love to do a comprehensive post on my early validation attempts in a few days, after I’ve tried a few more channels. The start of validation I have three primary goals with this early stage of validation: Find which group of people have the best response rate Find which channel of sourcing these folks yields the best response rate Of course, find a real interest in my idea #1 and #2 are to let me find one or two specific channels that work in terms of response rate, and double down in what works. Yesterday, I mentioned there were 5 different groups of people I might sell this to. Today, I’ve focused my efforts on the two groups I assume will have the best response: . UX leadership/seniors and Product Managers I hit my goal of reaching out to 10 targeted prospects, and even this small number turned out to be a great exercise in finding my audience. For the first few rounds, : I’ll be testing two different cold audience channels (you can search for tweets containing images which is super handy in my case: ) Using this, I found companies who were having a recent influx of customers pinging support with errors Bystander.io could help fix. Leveraging Twitter search filter:images “error message” This helped in solidifying who’s thinking about these user-facing error messages the most — authors were exclusively UX folks. I followed this breadcrumb trail to see what either the authors or interested commenters belonged to. Blog posts about error messages business So for my cold outreach, I had a blog post to discuss or a real example use-case from their customers. Here’s a pixelated view of my prospecting spreadsheet: There’s better ways to track this, but at this early scale a Google doc will do just fine! Tomorrow, now that I have the beginnings of a method to prospect these two channels, I’ll push out messages to another group of folks. And I’ll message 10 people in a new channel, to begin opening up testing response rates from different sources in parallel to these two. Scaffolding the actual app. Yesterday I also mentioned I’d like to get a seed going for the app itself. A this point we want the pipes that can leveraged in any idea we might pivot into, so we scrap as little development time as possible if we kill this first idea. I started out as a front end dev, and I’m most comfortable with . So we’ll be going fullstack JS to make sure I can move fast, using technology I’m most familiar with (there is one exception, we’ll get to that!) JavaScript The (MEAN+extras) stack: on the front end (a frontend that requires Typescript will be a nice help while we’re building) Angular 5+ : it looks like we’ll go with the as a starting point. Bootstrap Nebular theme on the backend with Node Express + (hosted on MLabs for the free development tier while we hack things together) MongoDB mongoose for authentication, this will save a ton of time, and their technical content team is amazing. Auth0 Now, for the unfamiliar, something I’ve really wanted to put to use in a project is . I know it’s only recommended for larger apps as there’s a lot of boilerplate and a bit of a learning curve. But, this inclusion is to sprinkle in a fun challenge. And the debugging options with a single source of state are awesome. We’ll use . Redux NgRx Tomorrow, day 3! More validation, of course. I think I’ll pull the trigger on . I don’t like putting up a full landing page until I have static mockups for screenshots, so I’d like to test Ship as a pre-official-landing-page solution. Product Hunt’s Ship I’ll set up my authentication service in Angular to wire up , using some of to help speed this along. Auth0 Kim Maida’s awesome posts And I’ll try to fix my woodstove, ‘cause it’s cold in here. As always if you have any feedback or want to chat or drop a comment here :-) Thanks for reading! my email is on my website Back to day 1. Forward to day 3.

[Day 2] Zero to MVP in 30 Days: Beginning validation and app buildout

The digital age has transformed how we choose where to eat, stay, and play, with online reviews playing a critical role in our decisions. Yet, this reliance on online recommendations has led to a growing problem: fake reviews. In 2023, research found that up to 30% of online reviews were inauthentic, leading to widespread distrust among consumers and eroding confidence in businesses that once seemed reliable. The Problem of Fake Reviews The prevalence of fake reviews is a symptom of deeper issues. Intense competition among businesses, combined with the ease of manipulating online reviews, has created an environment where unethical practices thrive. Companies can easily purchase positive reviews—or negative ones for their competitors—distorting the true picture for consumers. This not only harms businesses that rely on genuine customer feedback but also misleads consumers, leading to disappointing experiences based on fabricated information. Recent surveys reveal that online reviews influence the purchasing decisions of 93% of users. However, trust in these reviews is steadily declining, with 67% of consumers now acknowledging fake reviews as a significant problem. The impact is substantial: consumers waste time and money on inferior products and services, legitimate businesses are overshadowed by those willing to engage in dishonest tactics, and overall confidence in online reviews continues to erode. Current Solutions Overview Several approaches have been tried to combat fake reviews, each with its own limitations: Human Moderation: Traditional moderation systems rely on human reviewers to detect and remove fraudulent content. While effective to some extent, this method is time-consuming, expensive, and often fails to keep pace with the sheer volume of reviews being posted. Automated Systems: Recently, review platforms have started deploying machine learning-based systems to identify suspicious patterns in reviews. These systems can catch many fake reviews but are not foolproof, often struggling with nuanced or sophisticated fake reviews. Additionally, automated systems risk flagging legitimate feedback as fraudulent, which can frustrate honest users and damage the platform’s credibility. Verified Purchase Badges: Some platforms have introduced verified purchase badges or similar markers to indicate that a reviewer actually used the product or service. While helpful, this approach is not universally adopted and doesn’t fully address the problem—especially in industries like hospitality, where experiences are subjective and not always tied to a simple transaction. Moreover, the verification process can sometimes be circumvented, allowing fake reviews to slip through. Reputation-Based Systems: By assigning reputation scores to users based on the quality and consistency of their reviews, this approach aims to encourage honest feedback. The idea is that users with a history of trustworthy reviews will be more reliable, helping to filter out fake content. However, building a strong reputation takes time, during which fake reviews can still be posted. Additionally, these systems are not immune to manipulation; coordinated groups can work together to inflate or deflate reputation scores, undermining their overall effectiveness Blockchain and AI: Innovative Approaches to Combat Fake Reviews Blockchain and AI provide powerful tools in the battle against fake reviews, offering complementary methods to enhance trust and authenticity. Blockchain’s decentralized and immutable ledger ensures that once reviews and recommendations are recorded, they cannot be altered or deleted without network consensus, safeguarding the integrity of the content and preventing tampering. This technology also enables identity verification, tying reviews to verified users and making it more difficult for fake accounts to flood platforms with misleading feedback. Additionally, blockchain tech can support token-based incentive systems, rewarding users with cryptocurrency or tokens for posting honest reviews, which further discourages dishonest submissions. AI complements blockchain by adding an extra layer of protection through content analysis and filtering before it reaches the public. AI algorithms can detect patterns, language, and behavior that indicate fake reviews, flagging suspicious content in real-time. Together, these technologies help ensure that the information consumers rely on is both genuine and trustworthy. However, despite the significant advancements in the crypto, Web3, and DeFi sectors, the full potential of blockchain and AI in combating fake reviews has yet to be fully realized. Case Study: daGama’s Multi Level Anti Fake System(MLAFS) daGama is pioneering the use of blockchain and AI to tackle fake recommendations. Its Multi-Level Anti-Fake System (MLAFS) effectively integrates blockchain’s secure ledger with AI's detailed analysis, creating a robust barrier against fraudulent content. This dual-layered approach ensures that only genuine recommendations make it through. In addition, it introduced the “Review and Earn” concept to tackles a critical issue in current systems: the lack of incentives for users to provide honest feedback, ensuring that users are rewarded with tokens for valuable recommendations that are upvoted and tipped by other community members. The platform’s RWL-Map (Real World Locations Map) is a dynamic, user-driven ecosystem that helps users discover the best locations, restaurants, and venues based on real, verified recommendations. With a global launch slated for 2024, daGama is set to transform the review industry by restoring trust and simplifying the process of discovering new places to visit. Conclusion As the online review industry grapples with the persistent challenge of fake reviews, traditional methods—such as human moderation, automated systems, verified purchase badges, and reputation-based systems—have proven to be only partially effective. Each approach has its limitations, often addressing only part of the problem while leaving significant gaps that can be exploited by those intent on manipulating recommendations. However, innovative solutions leveraging blockchain and AI present a promising way forward. By combining the immutability and transparency of blockchain with the analytical power of AI, these technologies offer a more robust defense against fraudulent content. The potential impact of blockchain and AI on the review industry goes beyond just preventing fake recommendations; it also fosters a culture of trust and transparency that benefits consumers and businesses alike. By restoring confidence in the accuracy of online reviews, these technologies make it easier for consumers to make informed decisions and for businesses to build genuine reputations based on authentic feedback. Looking ahead, as blockchain and AI technologies continue to mature, we can expect even more sophisticated systems to emerge, potentially incorporating other technologies like natural language processing, sentiment analysis, and decentralized identity verification. These advancements will make it increasingly difficult for fraudulent actors to manipulate review systems and will help build a more reliable, user-centric digital landscape. The digital age has transformed how we choose where to eat, stay, and play, with online reviews playing a critical role in our decisions. Yet, this reliance on online recommendations has led to a growing problem: fake reviews. In 2023, research found that up to 30% of online reviews were inauthentic, leading to widespread distrust among consumers and eroding confidence in businesses that once seemed reliable. research found research found The Problem of Fake Reviews The prevalence of fake reviews is a symptom of deeper issues. Intense competition among businesses, combined with the ease of manipulating online reviews, has created an environment where unethical practices thrive. Companies can easily purchase positive reviews—or negative ones for their competitors—distorting the true picture for consumers. This not only harms businesses that rely on genuine customer feedback but also misleads consumers, leading to disappointing experiences based on fabricated information. Recent surveys reveal that online reviews influence the purchasing decisions of 93% of users. However, trust in these reviews is steadily declining, with 67% of consumers now acknowledging fake reviews as a significant problem. The impact is substantial: consumers waste time and money on inferior products and services, legitimate businesses are overshadowed by those willing to engage in dishonest tactics, and overall confidence in online reviews continues to erode. Recent surveys Recent surveys Current Solutions Overview Several approaches have been tried to combat fake reviews, each with its own limitations: Human Moderation: Traditional moderation systems rely on human reviewers to detect and remove fraudulent content. While effective to some extent, this method is time-consuming, expensive, and often fails to keep pace with the sheer volume of reviews being posted. Automated Systems: Recently, review platforms have started deploying machine learning-based systems to identify suspicious patterns in reviews. These systems can catch many fake reviews but are not foolproof, often struggling with nuanced or sophisticated fake reviews. Additionally, automated systems risk flagging legitimate feedback as fraudulent, which can frustrate honest users and damage the platform’s credibility. Verified Purchase Badges: Some platforms have introduced verified purchase badges or similar markers to indicate that a reviewer actually used the product or service. While helpful, this approach is not universally adopted and doesn’t fully address the problem—especially in industries like hospitality, where experiences are subjective and not always tied to a simple transaction. Moreover, the verification process can sometimes be circumvented, allowing fake reviews to slip through. Reputation-Based Systems: By assigning reputation scores to users based on the quality and consistency of their reviews, this approach aims to encourage honest feedback. The idea is that users with a history of trustworthy reviews will be more reliable, helping to filter out fake content. However, building a strong reputation takes time, during which fake reviews can still be posted. Additionally, these systems are not immune to manipulation; coordinated groups can work together to inflate or deflate reputation scores, undermining their overall effectiveness Blockchain and AI: Innovative Approaches to Combat Fake Reviews Blockchain and AI provide powerful tools in the battle against fake reviews, offering complementary methods to enhance trust and authenticity. Blockchain’s decentralized and immutable ledger ensures that once reviews and recommendations are recorded, they cannot be altered or deleted without network consensus, safeguarding the integrity of the content and preventing tampering. This technology also enables identity verification, tying reviews to verified users and making it more difficult for fake accounts to flood platforms with misleading feedback. Additionally, blockchain tech can support token-based incentive systems, rewarding users with cryptocurrency or tokens for posting honest reviews, which further discourages dishonest submissions. AI complements blockchain by adding an extra layer of protection through content analysis and filtering before it reaches the public. AI algorithms can detect patterns, language, and behavior that indicate fake reviews, flagging suspicious content in real-time. Together, these technologies help ensure that the information consumers rely on is both genuine and trustworthy. However, despite the significant advancements in the crypto, Web3, and DeFi sectors, the full potential of blockchain and AI in combating fake reviews has yet to be fully realized. Case Study: daGama’s Multi Level Anti Fake System(MLAFS) daGama is pioneering the use of blockchain and AI to tackle fake recommendations. Its Multi-Level Anti-Fake System (MLAFS) effectively integrates blockchain’s secure ledger with AI's detailed analysis, creating a robust barrier against fraudulent content. This dual-layered approach ensures that only genuine recommendations make it through. In addition, it introduced the “Review and Earn” concept to tackles a critical issue in current systems: the lack of incentives for users to provide honest feedback, ensuring that users are rewarded with tokens for valuable recommendations that are upvoted and tipped by other community members. The platform’s RWL-Map (Real World Locations Map) is a dynamic, user-driven ecosystem that helps users discover the best locations, restaurants, and venues based on real, verified recommendations. With a global launch slated for 2024, daGama is set to transform the review industry by restoring trust and simplifying the process of discovering new places to visit. Conclusion As the online review industry grapples with the persistent challenge of fake reviews, traditional methods—such as human moderation, automated systems, verified purchase badges, and reputation-based systems—have proven to be only partially effective. Each approach has its limitations, often addressing only part of the problem while leaving significant gaps that can be exploited by those intent on manipulating recommendations. However, innovative solutions leveraging blockchain and AI present a promising way forward. By combining the immutability and transparency of blockchain with the analytical power of AI, these technologies offer a more robust defense against fraudulent content. The potential impact of blockchain and AI on the review industry goes beyond just preventing fake recommendations; it also fosters a culture of trust and transparency that benefits consumers and businesses alike. By restoring confidence in the accuracy of online reviews, these technologies make it easier for consumers to make informed decisions and for businesses to build genuine reputations based on authentic feedback. Looking ahead, as blockchain and AI technologies continue to mature, we can expect even more sophisticated systems to emerge, potentially incorporating other technologies like natural language processing, sentiment analysis, and decentralized identity verification. These advancements will make it increasingly difficult for fraudulent actors to manipulate review systems and will help build a more reliable, user-centric digital landscape.

Leveraging Blockchain And AI To Eliminate Fake Reviews

When we first came up with our “big idea” back in 2017, my partner Mike Wagner and I didn’t know how much impact it would have. The idea was simple enough: We wanted to develop a virtual appliance that could discover, provision, and maintain servers from any manufacturer. By 2020, that vision became a reality with the launch of our flagship platform, which we built entirely on open-source technologies. That strategy was working well from the start. Well enough, I should say. We knew we could improve our metal-as-a-service software appliance if we could just find a way to boost performance, ensure higher system uptime, and streamline the architecture. The only way to do that, we realized, was to replace the Linux operating system and KVM virtual machine solution with something that would give us those benefits. What we did know was that we wanted to continue with our strategy of utilizing open source—and ideally tied to a very permissive license and open governance model. That would give us more flexibility as our roadmap unfolded and enable extensive use and modification without stringent (or murky) conditions. Taking the plunge After our due diligence, we transitioned to the Unix-like open source FreeBSD operating system, along with the BSD hypervisor Bhyve. We liked that the FreeBSD project had off-the-shelf capabilities within a single distribution, and had plenty of mission-critical enterprise use cases (Netflix and Sony being two of the most well-known). Together with existing technologies like the Redfish standard, the new and improved platform we’d built (called Mojo) was ready to really take off. With FreeBSD and bhyve fully in place, we were now able to add edge data center provisioning to the platform, which helps discover, configure, and provision bare metal servers and storage within a data center and at the edge. It works really well with another new technology (PhotoISP, which I’ll discuss in a moment) to manage geographically distributed bare metal assets. These open-source changes to our infrastructure proved to be the silver bullet we were looking for. More organizations began to contact us, and we started landing some really interesting use cases, including Major League Baseball. Before the 2021 season, MLB was looking for a way to move from VMware to a containerized environment across 30 data centers. In a nutshell, they needed a way to quickly process 15 different input sources to track all the player and team statistics generated during a game. By more quickly provisioning servers, that data could be quickly gathered at the edge and uploaded into Google Cloud. The Mojo platform, backed by an evolved open-source strategy, helped streamline MLB’s deployment pipelines (cutting deployment time in half). It also simplified the provisioning and management of MLB’s bare metal edge sites and made it really easy to provision Anthos, BIOS, firmware, OS patches, and upgrades. They also saw big performance gains for their GPU-enabled apps and more optimized CI/CD pipelines. Expanding a vision backed by open source Albert Einstein once said that necessity is the mother of all invention. My family lives in the Blue Ridge Mountains, and when COVID hit, word spread quickly that several teachers and students were unable to stream their classes because they had insufficient rural broadband access. Mike and I brainstormed about how we could help, and we immediately recognized how Mojo, paired with a reliable and secure CPE, could solve the problem. That led development of the Photon ISP platform, which delivers routing, firewall, and QoS through the open-source FreeBSD operating system. It also monitors IDS/IPS for wireless and fixed fiber networks and ensures uninterrupted 4G/5G and Starlink failover as a hedge to wireless ISP services. Today, the Photon ISP platform has 50 rural customers, and that number continues to grow. And its uptime is unparalleled; we have FreeBSD CPEs that have been running for more than a year without issue. Speeds have been very consistent, streaming content has been reliable, and there has been almost zero latency. We’re still pushing the limits of what we can do with our optimized open-source infrastructure. Next, we plan to finish moving the remainder of the stack to FreeBSD once a viable OCI-compliant stack becomes available. In the meantime, we still use Linux within Bhyve virtual machines for a few of our services. Simply put, the strategic shift to open-source FreeBSD and Bhyve for our platforms has catalyzed innovation and customer growth. The success with MLB—cutting deployment times and improving performance—is a great example of how the right open-source pieces can have an outsized impact while being far more flexible (not to mention cost-efficient!) than proprietary alternatives. When we first came up with our “big idea” back in 2017, my partner Mike Wagner and I didn’t know how much impact it would have. The idea was simple enough: We wanted to develop a virtual appliance that could discover, provision, and maintain servers from any manufacturer. By 2020, that vision became a reality with the launch of our flagship platform, which we built entirely on open-source technologies. That strategy was working well from the start. Well enough , I should say. enough We knew we could improve our metal-as-a-service software appliance if we could just find a way to boost performance, ensure higher system uptime, and streamline the architecture. The only way to do that, we realized, was to replace the Linux operating system and KVM virtual machine solution with something that would give us those benefits. What we did know was that we wanted to continue with our strategy of utilizing open source—and ideally tied to a very permissive license and open governance model. That would give us more flexibility as our roadmap unfolded and enable extensive use and modification without stringent (or murky) conditions. Taking the plunge After our due diligence, we transitioned to the Unix-like open source FreeBSD operating system, along with the BSD hypervisor Bhyve . We liked that the FreeBSD project had off-the-shelf capabilities within a single distribution, and had plenty of mission-critical enterprise use cases (Netflix and Sony being two of the most well-known). Together with existing technologies like the Redfish standard, the new and improved platform we’d built (called Mojo) was ready to really take off. open source FreeBSD Bhyve Redfish With FreeBSD and bhyve fully in place, we were now able to add edge data center provisioning to the platform, which helps discover, configure, and provision bare metal servers and storage within a data center and at the edge. It works really well with another new technology (PhotoISP, which I’ll discuss in a moment) to manage geographically distributed bare metal assets. These open-source changes to our infrastructure proved to be the silver bullet we were looking for. More organizations began to contact us, and we started landing some really interesting use cases, including Major League Baseball. Before the 2021 season, MLB was looking for a way to move from VMware to a containerized environment across 30 data centers. In a nutshell, they needed a way to quickly process 15 different input sources to track all the player and team statistics generated during a game. By more quickly provisioning servers, that data could be quickly gathered at the edge and uploaded into Google Cloud. The Mojo platform, backed by an evolved open-source strategy, helped streamline MLB’s deployment pipelines (cutting deployment time in half). It also simplified the provisioning and management of MLB’s bare metal edge sites and made it really easy to provision Anthos, BIOS, firmware, OS patches, and upgrades. They also saw big performance gains for their GPU-enabled apps and more optimized CI/CD pipelines. Expanding a vision backed by open source Albert Einstein once said that necessity is the mother of all invention. My family lives in the Blue Ridge Mountains, and when COVID hit, word spread quickly that several teachers and students were unable to stream their classes because they had insufficient rural broadband access. Mike and I brainstormed about how we could help, and we immediately recognized how Mojo, paired with a reliable and secure CPE, could solve the problem. That led development of the Photon ISP platform, which delivers routing, firewall, and QoS through the open-source FreeBSD operating system. It also monitors IDS/IPS for wireless and fixed fiber networks and ensures uninterrupted 4G/5G and Starlink failover as a hedge to wireless ISP services. Today, the Photon ISP platform has 50 rural customers, and that number continues to grow. And its uptime is unparalleled; we have FreeBSD CPEs that have been running for more than a year without issue. Speeds have been very consistent, streaming content has been reliable, and there has been almost zero latency. We’re still pushing the limits of what we can do with our optimized open-source infrastructure. Next, we plan to finish moving the remainder of the stack to FreeBSD once a viable OCI-compliant stack becomes available. In the meantime, we still use Linux within Bhyve virtual machines for a few of our services. Simply put, the strategic shift to open-source FreeBSD and Bhyve for our platforms has catalyzed innovation and customer growth. The success with MLB—cutting deployment times and improving performance—is a great example of how the right open-source pieces can have an outsized impact while being far more flexible (not to mention cost-efficient!) than proprietary alternatives.

How Our Open Source OS Pivot Accelerated Product Innovation

Authors:
(1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz;
(2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Threat Model Background and Related Work Methodology Experiments Setup Results Discussion Conclusion, Acknowledgments, and References Appendix 3 Background and Related Work This section provides background information and summarizes the related work in machine learning malware evasion and model extraction. It also presents some relevant background information on reinforcement learning and the environments used for training agents for malware evasion. 3.1 Reinforcement Learning Reinforcement Learning (RL) is a sub-field of machine learning where the optimization of a reward function is done by agents that interact with an environment. The agent performs actions in the environment and receives back observations (view of the new state) and a reward. The goal of RL is to train a policy, which instructs the agent to take certain actions to maximize the rewards received over time [40]. More formally, the agent and the environment interact over a sequence of time steps t. At each step, the agent receives a state from the environment st and performs an action at from a predefined set of actions A. The environment produces the new state st+1 and the reward that the agent receives due to its action. Through this interaction with the environment, the agent tries to learn a policy π(a|s) that maps the probability of selecting an action a given a state s, so that the expected reward is maximized. One of the essential aspects of RL is the Markov property, which assumes that future states of the process only depend on the current state. Some of the most successful RL algorithms are model-free approaches, i.e., they do not try to build a model of the environment. A downside of model-free algorithms is that they tend to require a lot of data and work under the assumption that the interactions with the environment are not costly. In contrast, model-based reinforcement learning algorithms learn a model of the environment and use it to improve the policy learning. A fairly standard approach for modelbased RL is to alternate between policy optimization and model learning. During the model learning phase, data is gathered from interacting with the environment, and a supervised learning technique is employed to learn the model of the environment. Subsequently, in the policy optimization phase, the trained model explores methods for enhancing the policy [21].. In terms of security applications, reinforcement learning has been proposed for use in several applications such as honeypots [11,17], IoT and cyber-physical systems [26,42], network security [44], malware detection [13,43], and malware evasion [2,14,22,23,29,30,38]. Most malware evasion papers use Malware-gym or extensions of it to test their proposed algorithms, therefore we present the basic concepts of the environment in more detail below. Malware-Gym The Malware-Gym environment is a reinforcement learning environment based on OpenAI Gym [5], and it was first introduced in [2]. Figure 1 shows the internal architecture of the environment. The environment encapsulates a target model, which is considered a black box. The target takes as input extracted features from a binary file and outputs a label or a prediction score. The binary file bytes are the state st and the extracted features is the observation ot, which the agent receives from the environment at time t. The observation is also fed into the target model that outputs a score or a label. The score is transformed into a reward rt that is returned to the agent that has to decide which action to take, at. The action is fed back into the environment, which modifies the binary through the transition function (modifier) to produce the next binary state st+1, observation ot+1, and reward rt+1. The list of available actions in the latest version of the Malware-Gym can be found in Table 1. The actions listed aim to preserve functionality, and ideally, the modified binary should still be valid. Some of the actions require a benign dataset in order to use strings and sections from benign files. The reward is calculated based on the output of the classifier/target. If the score is below a predetermined decision threshold, the binary is considered benign, i.e., it evaded the target, and the returned reward equals 10. If the score is higher than the threshold, the binary is considered malicious, and the reward is calculated as the difference between the previous and current scores. The process continues until the binary is evasive or a predetermined number of maximum iterations is reached. The agent receives the current observation (features) and the reward and decides on the next action unless the malware is evasive, where it moves to the next binary. The agent’s goal is to learn a policy π that generalizes well and can be applied to future malicious binaries. Table 1. Action space in latest Malware-Gym 3.2 Malware Evasion The main goal of malware evasion is to make the malware undetectable by security systems so that it can perform its malicious activities unnoticed. Malware creators continuously develop new techniques to evade detection, which makes it difficult for security experts to keep up with the evolving threat landscape. The concept has evolved to include the evasion of machine learning classifiers and also to use machine learning techniques to evade antivirus (AV) systems. For the purpose of this work, we are mainly interested in machine learning approaches that attackers may use to generate evasive fully functional Windows PE malware so that they can not be detected by malware classifiers and AVs. There have been several works on this topic, and detailed taxonomies and definitions can be found in the following surveys [10,24]. Anderson et al. [2] created the Malware-Gym and were the first work that generated adversarial malware that aimed to be functionality preserving. They used the Actor-Critic model with Experience Replay (ACER) algorithm and tested against the original Ember model. Other works extended the Malware-Gym and tested different algorithms such as Double Deep Q-Network(DQN) [14], Distributional Double DQN [22], and REINFORCE [30]. Apart from testing different algorithms, several authors proposed changes to the reward function [12,22]. MAB-malware [38] is another RL framework that is based on multi-armed bandits (MAB). MAB and AIMED [22] are the only works that assume that the target model provides only a hard label instead of a prediction score. The different works propose different numbers of permitted modifications, varying from as low as 5 to as high as 80. However, none of the prior work puts a constraint on the number of queries to the target or discusses whether unlimited query access is a realistic scenario. Other black-box attacks for the evasion of malware classifiers proposed different approaches such as genetic algorithms [9], explainability [33], or using simpler but effective techniques such as packing [6]. 3.3 Model Extraction Model extraction or model stealing is a family of attacks that aim to retrieve a model’s parameters, such as neural network weights or a functional approximation, using a limited query budget. These types of attacks are usually tested against Machine Learning as a Service (MLaaS) applications. The majority of the model extraction attacks use learning-based approaches, where they query the target model with multiple data samples X and retrieve labels or prediction scores y to create a thief dataset. Using the thief dataset, they train a surrogate model that behaves similarly to the target model in the given task. Model extraction attacks may have different goals [19]. In a fidelity attack, the adversary aims to create a surrogate model that learns the decision boundary of the target as faithfully as possible, including the errors that the target makes. These surrogate models can be used later in other tasks, such as generating adversarial samples. In a task accuracy attack, the adversary aims to construct a surrogate model that performs equally well or better than the target model in a specific task such as image or malware classification. The attacker’s ultimate goal affects the selection of the thief dataset, the metrics for a successful attack, and the attack strategy itself. Learning-based model extraction attacks use different strategies to select the samples they use for querying the target, such as active learning [7,28], reinforcement learning [27], generative models [34], or just selecting random data [8]. In the security domain, model extraction has been proposed as the first step of an attack that aims to generate evasive malware [33,16,32]. Both [33] and [32] generate evasive malware in a two-step approach: first creating a surrogate and then using it to evade the target. Hu et al. [16] use a Generative Adversarial Network (GAN) and a surrogate detector to bypass a malware detector that works with API calls. Neither [16] and [33] are concerned with the number of queries they are making to the target, while in [32] the authors measured only the query efficiency of the model extraction and not that of the subsequent task of the malware evasion. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Authors: Authors: (1) Maria Rigaki, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and maria.rigaki@fel.cvut.cz; (2) Sebastian Garcia, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic and sebastian.garcia@agents.fel.cvut.cz. Table of Links Abstract & Introduction Abstract & Introduction Threat Model Threat Model Background and Related Work Background and Related Work Methodology Methodology Experiments Setup Experiments Setup Results Results Discussion Discussion Conclusion, Acknowledgments, and References Conclusion, Acknowledgments, and References Appendix Appendix 3 Background and Related Work This section provides background information and summarizes the related work in machine learning malware evasion and model extraction. It also presents some relevant background information on reinforcement learning and the environments used for training agents for malware evasion. 3.1 Reinforcement Learning Reinforcement Learning (RL) is a sub-field of machine learning where the optimization of a reward function is done by agents that interact with an environment. The agent performs actions in the environment and receives back observations (view of the new state) and a reward. The goal of RL is to train a policy, which instructs the agent to take certain actions to maximize the rewards received over time [40]. More formally, the agent and the environment interact over a sequence of time steps t. At each step, the agent receives a state from the environment st and performs an action at from a predefined set of actions A. The environment produces the new state st+1 and the reward that the agent receives due to its action. Through this interaction with the environment, the agent tries to learn a policy π(a|s) that maps the probability of selecting an action a given a state s, so that the expected reward is maximized. One of the essential aspects of RL is the Markov property, which assumes that future states of the process only depend on the current state. Some of the most successful RL algorithms are model-free approaches, i.e., they do not try to build a model of the environment. A downside of model-free algorithms is that they tend to require a lot of data and work under the assumption that the interactions with the environment are not costly. In contrast, model-based reinforcement learning algorithms learn a model of the environment and use it to improve the policy learning. A fairly standard approach for modelbased RL is to alternate between policy optimization and model learning. During the model learning phase, data is gathered from interacting with the environment, and a supervised learning technique is employed to learn the model of the environment. Subsequently, in the policy optimization phase, the trained model explores methods for enhancing the policy [21].. In terms of security applications, reinforcement learning has been proposed for use in several applications such as honeypots [11,17], IoT and cyber-physical systems [26,42], network security [44], malware detection [13,43], and malware evasion [2,14,22,23,29,30,38]. Most malware evasion papers use Malware-gym or extensions of it to test their proposed algorithms, therefore we present the basic concepts of the environment in more detail below. Malware-Gym The Malware-Gym environment is a reinforcement learning environment based on OpenAI Gym [5], and it was first introduced in [2]. Figure 1 shows the internal architecture of the environment. Malware-Gym The environment encapsulates a target model, which is considered a black box. The target takes as input extracted features from a binary file and outputs a label or a prediction score. The binary file bytes are the state st and the extracted features is the observation ot, which the agent receives from the environment at time t. The observation is also fed into the target model that outputs a score or a label. The score is transformed into a reward rt that is returned to the agent that has to decide which action to take, at. The action is fed back into the environment, which modifies the binary through the transition function (modifier) to produce the next binary state st+1, observation ot+1, and reward rt+1. The list of available actions in the latest version of the Malware-Gym can be found in Table 1. The actions listed aim to preserve functionality, and ideally, the modified binary should still be valid. Some of the actions require a benign dataset in order to use strings and sections from benign files. The reward is calculated based on the output of the classifier/target. If the score is below a predetermined decision threshold, the binary is considered benign, i.e., it evaded the target, and the returned reward equals 10. If the score is higher than the threshold, the binary is considered malicious, and the reward is calculated as the difference between the previous and current scores. The process continues until the binary is evasive or a predetermined number of maximum iterations is reached. The agent receives the current observation (features) and the reward and decides on the next action unless the malware is evasive, where it moves to the next binary. The agent’s goal is to learn a policy π that generalizes well and can be applied to future malicious binaries. Table 1. Action space in latest Malware-Gym Table 1. 3.2 Malware Evasion The main goal of malware evasion is to make the malware undetectable by security systems so that it can perform its malicious activities unnoticed. Malware creators continuously develop new techniques to evade detection, which makes it difficult for security experts to keep up with the evolving threat landscape. The concept has evolved to include the evasion of machine learning classifiers and also to use machine learning techniques to evade antivirus (AV) systems. For the purpose of this work, we are mainly interested in machine learning approaches that attackers may use to generate evasive fully functional Windows PE malware so that they can not be detected by malware classifiers and AVs. There have been several works on this topic, and detailed taxonomies and definitions can be found in the following surveys [10,24]. Anderson et al. [2] created the Malware-Gym and were the first work that generated adversarial malware that aimed to be functionality preserving. They used the Actor-Critic model with Experience Replay (ACER) algorithm and tested against the original Ember model. Other works extended the Malware-Gym and tested different algorithms such as Double Deep Q-Network(DQN) [14], Distributional Double DQN [22], and REINFORCE [30]. Apart from testing different algorithms, several authors proposed changes to the reward function [12,22]. MAB-malware [38] is another RL framework that is based on multi-armed bandits (MAB). MAB and AIMED [22] are the only works that assume that the target model provides only a hard label instead of a prediction score. The different works propose different numbers of permitted modifications, varying from as low as 5 to as high as 80. However, none of the prior work puts a constraint on the number of queries to the target or discusses whether unlimited query access is a realistic scenario. Other black-box attacks for the evasion of malware classifiers proposed different approaches such as genetic algorithms [9], explainability [33], or using simpler but effective techniques such as packing [6]. 3.3 Model Extraction Model extraction or model stealing is a family of attacks that aim to retrieve a model’s parameters, such as neural network weights or a functional approximation, using a limited query budget. These types of attacks are usually tested against Machine Learning as a Service (MLaaS) applications. The majority of the model extraction attacks use learning-based approaches, where they query the target model with multiple data samples X and retrieve labels or prediction scores y to create a thief dataset. Using the thief dataset, they train a surrogate model that behaves similarly to the target model in the given task. Model extraction attacks may have different goals [19]. In a fidelity attack, the adversary aims to create a surrogate model that learns the decision boundary of the target as faithfully as possible, including the errors that the target makes. These surrogate models can be used later in other tasks, such as generating adversarial samples. In a task accuracy attack, the adversary aims to construct a surrogate model that performs equally well or better than the target model in a specific task such as image or malware classification. The attacker’s ultimate goal affects the selection of the thief dataset, the metrics for a successful attack, and the attack strategy itself. Learning-based model extraction attacks use different strategies to select the samples they use for querying the target, such as active learning [7,28], reinforcement learning [27], generative models [34], or just selecting random data [8]. In the security domain, model extraction has been proposed as the first step of an attack that aims to generate evasive malware [33,16,32]. Both [33] and [32] generate evasive malware in a two-step approach: first creating a surrogate and then using it to evade the target. Hu et al. [16] use a Generative Adversarial Network (GAN) and a surrogate detector to bypass a malware detector that works with API calls. Neither [16] and [33] are concerned with the number of queries they are making to the target, while in [32] the authors measured only the query efficiency of the model extraction and not that of the subsequent task of the malware evasion. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

The Intersection of Reinforcement Learning and Model Extraction in Cybersecurity

USA v. Samuel Bankman-Fried Court Filing, retrieved on March 15, 2024 is part of . You can jump to any part in this filing . This part is 32 of 33. HackerNoon’s Legal PDF Series here III. The Court Should Order Victim Compensation Through Remission as an Alternative to Restitution The Government seeks to return all stolen assets that are finally forfeited to victims of the defendant’s crimes in the most efficient manner possible. Because of the complexity of the case and the number of victims involved, awarding restitution to victims in accordance with 18 U.S.C. § 3663A would be extremely costly and administratively impractical. The cost and time associated with calculating each victim’s loss, determining whether the victim has already been compensated through the pending bankruptcy, and then paying out a percentage of the victim’s losses, will delay payment and diminish the amount of money actually paid to victims. Therefore, consistent with the Government’s frequent approach in complex cases with numerous victims, the Government moves for entry of an order authorizing the United States to compensate victims with finally forfeited assets through a remission process, as restitution would be impracticable in this case. 18 U.S.C. § 3663A(c)(3). As noted in the defendant’s submission, he consents to this request. Restitution to persons “directly and proximately harmed” by fraud and money laundering is ordinarily mandatory. See 18 U.S.C. §§ 3663A(a)(1), (c)(1)(A)(ii). In such a case, the Court is authorized to determine a restitution amount and a schedule of victims and payments. However, an order of restitution is not required when “the number of identifiable victims is so large as to make restitution impracticable,” or when imposing restitution would require “determining complex issues of fact related to the cause or amount of the victim’s losses” which “would complicate or prolong the sentencing process.” Id. § 3663A(c)(3). In those situations, instead of ordering restitution, courts have authorized the Government to compensate victims through the process of remission under the forfeiture statutes and related regulations. See, e.g., United States v. Madoff, 09 Cr. 213 (DC), Dkt. 106 & United States v. Bonventre, 10 Cr. 228, Dkt. 318 (Madoff Ponzi scheme); United States v. Sharma, 18 Cr. 340, Dkt. 407 (multi-million dollar cryptocurrency scheme); United States v. Dos Santos, 20 Cr. 398, Dkt. 283 (multi-million dollar cryptocurrency scheme). The forfeiture statutes authorize the Attorney General to “grant petitions for mitigation or remission of forfeiture, restore forfeited property to victims of a violation of this subchapter, or take any other action to protect the rights of innocent persons which is in the interest of justice and which is not inconsistent with the provisions of this section.” 21 U.S.C. § 853(i)(1). The applicable regulations are set forth in 28 C.F.R. Part 9. In order to qualify for remission, a crime victim must provide information to support specific elements, including that they suffered a specific pecuniary loss directly caused by the criminal offense, that the victim did not contribute to the offense, that the victim has not been in compensated for the loss from another means, and that the victim has no other recourse available. See generally 28 C.F.R. § 9.8(b). It is the policy of the Department of Justice, consistent with the Crime Victims’ Rights Act, to ensure that crime victims receive “full and timely restitution as provided in law.” See 18 U.S.C. § 3771(c)(1). Accordingly, when the Government seizes property in connection with a criminal case, the Government’s goal is to forfeit the property and then, in remission proceedings administered by the Attorney General through his delegee, the Chief of the Money Laundering and Asset Recovery Section of the Department of Justice’s Criminal Division (“MLARS”), to distribute funds to victims. Frequently MLARS will distribute those assets through a remission program by which victims may petition for portions of forfeited property. E.g. United States v. Sharma, 18 Cr. 340 (LGS), 2022 WL 1910026, at *2 (S.D.N.Y. June 3, 2022). In a multi-victim or complex case, the process of notifying potential victims, processing petitions, verifying losses, and recommending a distribution of available funds may be managed on behalf of the Department of Justice by a Special Master or trustee, as authorized by 28 C.F.R. § 9.9(c). While use of such means is common, there are other methods of distribution that the Department has used. In cases that also involve bankruptcy proceedings, the Department has at times worked with the bankruptcy estate to distribute forfeited assets. E.g. United States v. $7,206,157,717 on Deposit at JP Morgan Chase Bank, N.A., 10 Civ. 9398, Dkt. 2 (S.D.N.Y. December 17, 2010) (stipulation and court ordered settlement whereby the estate of Jeffry Picower, who had received billions of dollars from Bernie Madoff, forfeited more than $7 billion; the Government agreed to credit $5 billion in payment to the bankruptcy estate to be allocated under bankruptcy rules, and then named the trustee as administrator for the remaining funds). Victims’ interests will be best served in this case by ordering remission of finally forfeited funds and by using the remission process. The Government has been in frequent communication with MLARS about the most efficient way to do this, including by retaining a claims administrator or by transferring funds into the FTX bankruptcy estate upon specified conditions.[16] An alternative arrangement whereby restitution is calculated and distributed to victims by the Government and the Court would be nearly impossible administratively, and would come at great expense to victims’ recovery. In order to enter an order of restitution, the Court would need to determine and corroborate each victim’s losses. That would require determining and corroborating for each customer victim both the amount of funds he or she transferred to FTX, and any funds withdrawn. That task, challenging as it would ever be for the global fraud which the defendant led, is compounded here further for victims who transferred funds via cryptocurrencies, given their ever-fluctuating value. There are millions of potential victims and substantial complexity in computing each victim’s investments and withdrawals. Restitution in this case is even more complicated by the different classes of victims: FTX’s dozens of equity investors, and the Alameda lenders who often lent or received funds via cryptocurrency. Instead, a process whereby the Government distributes finally forfeited funds, either through a claims administrator or through a distribution to the bankruptcy estate, will more be efficient, and not “would complicate or prolong the sentencing process.” 18 U.S.C. § 3663A(c)(3). Continue Reading . Here [16] The creditor class in the bankruptcy overlaps substantially but is not a perfect match with the victims of the defendant’s crimes. For example, customers of FTX’s US branch are not victims of the defendant’s crimes as charged in the Indictment, but are expected to recover in the FTX bankruptcy; conversely, equity investors are victims of the charged offenses but may not recover their losses in the bankruptcy. About HackerNoon Legal PDF Series: We bring you the most important technical and insightful public domain court case filings. This court case retrieved on March 15, 2024, from is part of the public domain. The court-created documents are works of the federal government, and under copyright law, are automatically placed in the public domain and may be shared without legal restriction. storage.courtlistener

Proposed Strategy for Compensating Bankman-Fried's Victims: Remission Over Restitution

If we talk about software development and operations, the convergence of cutting-edge technologies such as Artificial Intelligence (AI) with DevOps methodologies has assisted in forming a remarkable shift in architectural paradigms. The journey from monolithic applications to microservices architecture ( with AI-powered DevOps practices) represents an important evolution in software engineering. Monolithic vs. Microservices Architecture Traditional Approach - Monolithic Architecture: Traditionally, monolithic architectures have traditionally stood as the foundation. In this paradigm, applications are constructed as a singular, tightly integrated unit, wherein all components function within a unified codebase and execution environment. Although monolithic architectures simplify development and deployment, they also present certain challenges. Scaling particular functionalities becomes difficult. Implementing updates requires redeploying the entire application which causes downtime and potential disruptions in user experience. Modern Approach - Microservices Architecture: The emergence of microservices architecture addresses the limitations of monolithic structures. Microservices decompose applications into smaller and autonomous services. These services function independently and communicate via APIs. So, the collective approach offers compelling advantages: Microservices architecture breaks down big software applications into smaller and independent parts that work together. This sort of setup helps in a few ways. First, each part can grow or shrink on its own when needed. Second, different parts can use different programming languages by making the whole system more adaptable. Third, if one part fails, it won't affect the whole system. Fourth, changes can happen faster because the pieces are smaller and easier to work with. Lastly, it makes it simpler for different teams to work on separate parts without disturbing the whole system. So, it is easier to manage. This is where DevOps steps in. We can check the following comparison chart for the detailed insights: The Intersection of AI and DevOps in Software Evolution AI in DevOps: When AI technologies merge with DevOps practices, they introduce automation, data-driven decision-making, and predictive insights. It effectively alters various phases of the software development lifecycle. The AI integration into DevOps can have an important significance like tasks that were previously manual or required human intervention can now be automated. It consists of various processes like testing, deployment, and monitoring. So, it makes them more efficient and less prone to errors. Additionally, AI algorithms analyze vast amounts of data generated in the development pipeline. So, it enables teams to make smarter decisions based on information derived from this data. Essentially, AI in DevOps contributes to optimizing workflows, identifying potential issues before they occur, and predicting outcomes. Hence, the AI-based development process could improve overall software quality. The integration of AI within DevOps revolutionizes the conception, construction, and delivery of software. It accelerates development cycles and enhances the reliability, efficiency, and adaptability of software products by setting new standards for the industry. Important Aspects of AI Integration in DevOps: The diagram illustrates essential components highlighting the integration of Artificial Intelligence (AI) in DevOps practices. It describes five key parts of this combination. First, it shows automated testing and quality assurance tools driven by AI. These tools analyze code, find bugs, and predict issues by confirming the better software quality. Secondly, it shows Continuous Integration/Continuous Deployment (CI/CD). It demonstrates how AI assists and automates build, test, and deployment tasks by making the release process much faster. The third part highlights performance monitoring and anomaly detection which shows  AI-powered tools that analyze real-time metrics, quickly spotting irregularities, and optimizing system performance. Moving on, the fourth section addresses security and compliance with a highlight of AI's ability to proactively identify threats, vulnerabilities, and maintain adherence to regulations. The last part underlines the importance of cost optimization and resource management. This showcases how AI plays a crucial role in enhancing resource allocation, reducing expenses, and improving overall efficiency. Each segment of the diagram exhibits the integral role played by AI in revolutionizing DevOps practices, delivering efficiency, reliability, and security throughout the software development lifecycle. Strategies for Transitioning From Monolith to Microservices Moving from a monolithic setup to microservices necessitates a step-by-step approach. The strategies for this transition involve crucial steps such as identifying service boundaries, gradual decomposition, implementing API gateways, managing data, adopting DevOps practices, monitoring, team structuring, resilience, security, and testing. These strategies facilitate a seamless transition, enhancing scalability, agility, and overall software development practices. Analyze the existing monolith to identify logical boundaries for breaking functionalities into smaller and independent services. Adopt a phased approach to decompose the monolith by gradually extracting smaller and less complex components into microservices. Introduce an API gateway to centralize communication, manage requests, and provide a unified interface for clients. Adjust data management strategies to suit microservices (choosing between database per service or shared databases with caution) ensuring data consistency. Perform DevOps practices and set up CI/CD pipelines for automated deployment, testing, and monitoring. Implement best-suit monitoring tools to track microservices' performance and quickly identify and resolve issues. Encourage a culture where teams take ownership of their microservices. This sort of practice assists in having a cross-functional collaboration. Design microservices resiliently with fault-tolerant patterns to handle failures gracefully and prevent cascading issues. Apply stringent security measures like authentication, authorization, and encryption at different levels to safeguard microservices. Establish comprehensive testing strategies encompassing unit tests, integration tests, and end-to-end tests to validate microservices' functionality, performance, and interactions. Benefits and Challenges of Microservices and AI-Driven DevOps Microservices architecture with AI-driven DevOps brings many advantages to modern software development. Microservices offer scalability, flexibility, and agile deployment, while AI integration automates processes throughout the development cycle. However, this transition presents challenges. Organizations need to balance these benefits and challenges when adopting this innovative software development approach. Benefits: Microservices enable flexible scaling of individual components to manage varying workloads efficiently. Diverse programming languages in a single application enhance adaptability to different technologies. Isolated services prevent failures from affecting the entire system. Smaller codebases allow faster development iterations which assist in having a quicker response to market demands. AI-driven tools automate testing, integration, performance monitoring, and security. Challenges: Coordinating and managing multiple services introduces complexities in orchestration and maintenance. Distributed systems present challenges in ensuring strong and dependable security measures across all services. Teams need varied skill sets to handle various technologies. Effective monitoring tools and governance mechanisms are essential to manage the complex environment. Smooth integration and efficient orchestration of microservices present technical challenges during deployment and upkeep. Some Real-World Examples: Here are a few real-world examples that exhibit the remarkable shift from monolithic architectures to microservices empowered by AI-driven DevOps: 📥Netflix: Netflix transitioned from a monolithic architecture to a microservices-based infrastructure to improve scalability and resilience. They enhanced their platform's agility with the help of AI-driven DevOps tools for continuous integration and deployment. You can check the details . here By breaking down their system into small, manageable services, they achieved faster feature deployment and greater resilience during unexpected traffic spikes. For detailed information, check their technical articles and . documentation 📥Spotify: adopted a microservices architecture facilitated by AI-powered DevOps practices. They combined machine learning algorithms (MLAs) to optimize music recommendations and user experiences. Spotify's music streaming platform Utilizing microservices, Spotify successfully scaled its services. It assisted them in enabling seamless updates and new feature releases while maintaining system stability. 📥Amazon: shifted from monolithic architecture to microservices-based infrastructure which assisted them in transforming the cloud computing capabilities to another height. AI-driven DevOps practices enabled AWS to automate deployment, manage resources efficiently, and improve fault tolerance. Amazon Web Services (AWS) This architectural shift allowed them to introduce new services rapidly and maintain high availability for their global customer base. 📥Uber: transition from a monolithic application to a microservices architecture with a combination of AI, revolutionized its ride-sharing platform. With microservices, they could independently scale different components of their system which improves reliability and resilience. Uber's AI tools supported their DevOps practices by automating testing, deployment, and monitoring processes. 📥Microsoft: is more than a cloud computing giant; it's a breeding ground for AI-driven DevOps innovations. They have infrastructure that dynamically adjusts its scale, much like a self-driving car navigating rush-hour traffic. Microsoft's Azure platform Azure achieves this by integrating AI, analyzing resource usage, and predicting demand to optimize resource allocation. This not only keeps costs in check but also ensures soaring performance. Beyond effective cost savings, Azure's AI-driven infrastructure management liberates DevOps engineers to focus on their strengths. So, with the of automate routine tasks like resource scaling and anomaly detection, AI empowers teams to unleash creativity, boost productivity, and expedite software delivery while maintaining high quality. 📥Google: Google takes an all-encompassing AI-driven approach to revolutionize DevOps at every stage of the pipeline. Their vision involves continuous code scrutiny for potential issues, automatic anomaly detection, and resolution, and combining past incidents to prevent future ones. From continuous integration and delivery (CI/CD) to automated incident remediation, Google's AI tools transform software development and deployment. by taking care of repetitive tasks and providing immediate insights. This means developers can concentrate on inventing new things and solving problems. As a result, the software they create becomes more dependable and secure. AI helps developers The Future: With the rapid advancements in technology, the future of AI-driven DevOps and microservices architecture appears boundless. Advanced language models and cutting-edge AI technologies are expected to redefine the landscape of software development and operational efficiency. These progressive tools enable not only automation but also predictive analysis, enhancing decision-making and accelerating the development life cycle. The combination of AI-driven DevOps and microservices architecture is expected to push the boundaries of innovation which assists in facilitating more sophisticated, autonomous, and scalable systems. Moreover, the evolving capabilities of AI promise to optimize resource utilization, mitigate risks, and deliver software solutions that are more adaptive and responsive to changing demands and complexities in the tech industry. As these technologies evolve, they are anticipated to transform the way organizations build, deploy, and manage software applications. Conclusion: Switching from monolithic software to microservices, powered by AI-driven DevOps, marks a major turning point in software development. While microservices bring many benefits, such as faster development, easier scaling, and more innovation, they also require careful planning, smart implementation, and ongoing adaptation to overcome the challenges they bring. By combining microservices architecture with AI-driven DevOps, organizations can create a future where innovation is fast, systems are scalable and resilient, and they can thrive in the ever-changing tech world. Ultimately, this transformation is a harmonious blend of technology and methodology. It creates a symphony of software evolution that leads to new levels of efficiency and excellence.

Remarkable Shift From Monolith to Microservices - Empowered by AI-Driven DevOps

মনোলিথ থেকে মাইক্রোসার্ভিসেসের উল্লেখযোগ্য স্থানান্তর - AI-চালিত DevOps দ্বারা ক্ষমতাপ্রাপ্ত

Bemerkenswerter Wandel vom Monolithen zu Microservices – unterstützt durch KI-gesteuertes DevOps

Cambio notable del monolito a los microservicios: potenciado por DevOps impulsado por IA

Passage remarquable du monolithe aux microservices - optimisé par un DevOps piloté par l'IA

मोनोलिथ से माइक्रोसर्विसेज में उल्लेखनीय बदलाव - एआई-संचालित डेवऑप्स द्वारा सशक्त

モノリスからマイクロサービスへの顕著な移行 - AI 主導の DevOps による強化

모놀리스에서 마이크로서비스로의 놀라운 변화 - AI 기반 DevOps로 강화

Mudança notável de monólito para microsserviços – capacitada por DevOps baseado em IA

Замечательный переход от монолита к микросервисам – благодаря DevOps на основе искусственного интеллекта

Monolitten Mikro Hizmetlere Olağanüstü Geçiş - Yapay Zeka Odaklı DevOps ile Destekleniyor

Sự thay đổi đáng chú ý từ Monolith sang Microservices - Được hỗ trợ bởi DevOps do AI điều khiển

从整体架构到微服务的显着转变 - 由 AI 驱动的 DevOps 提供支持

ZKP Meets ML in Verifiability: Existing Scheme & Experimental Analysis

Microsoft v. United States (2016) Court Filing, retrieved on July 14, 2016, is part of . You can jump to any part in this filing . This part is 18 of 22. HackerNoon’s Legal PDF Series here E. Extraterritoriality of the Warrant Having thus determined that the Act focuses on user privacy, we have little trouble concluding that execution of the Warrant would constitute an unlawful extraterritorial application of the Act.  See Morrison, 561 U.S. at 266–67; RJR Nabisco, 579 U.S. at __, 2016 WL 3369423, at *9. The information sought in this case is the content of the electronic communications of a Microsoft customer. The content to be seized is stored in Dublin. J.A. at 38.  The record is silent regarding the citizenship and location of the customer.   Although the Act’s focus on the customer’s privacy might suggest that the customer’s actual location or citizenship would be important to the extraterritoriality analysis, it is our view that the invasion of the customer’s privacy takes place under the SCA where the customer’s protected content is accessed—here, where it is seized by Microsoft, acting as an agent of the government.[27] Because the content subject to the Warrant is located in, and would be seized from, the Dublin datacenter, the conduct that falls within the focus of the SCA would occur outside the United States, regardless of the customer’s location and regardless of Microsoft’s home in the United States.[28]  Cf. Riley v. California, 134 S. Ct. 2473, 2491 (2014) (noting privacy concern triggered by possibility that search of arrestee’s cell phone may inadvertently access data stored on the “cloud,” thus extending “well beyond papers and effects in the physical proximity” of the arrestee). The magistrate judge suggested that the proposed execution of the Warrant is not extraterritorial because “an SCA Warrant does not criminalize conduct taking place in a foreign country; it does not involve the deployment of American law enforcement personnel abroad; it does not require even the physical presence of service provider employees at the location where data are stored. . . .  [I]t places obligations only on the service provider to act within the United States.” In re Warrant, 15 F. Supp. 3d at 475– 76.  We disagree. First, his narrative affords inadequate weight to the facts that the data is stored in Dublin, that Microsoft will necessarily interact with the Dublin datacenter in order to retrieve the information for the government’s benefit, and that the data lies within the jurisdiction of a foreign sovereign. Second, the magistrate judge’s observations overlook the SCA’s formal recognition of the special role of the service provider vis‐à‐vis the content that its customers entrust to it. In that respect, Microsoft is unlike the defendant in Marc Rich and other subpoena recipients who are asked to turn over records in which only they have a protectable privacy interest. The government voices concerns that, as the magistrate judge found, preventing SCA warrants from reaching data stored abroad would place a “substantial” burden on the government and would “seriously impede[]” law enforcement efforts. Id. at 474. The magistrate judge noted the ease with which a wrongdoer can mislead a service provider that has overseas storage facilities into storing content outside the United States.  He further noted that the current process for obtaining foreign‐stored data is cumbersome.  That process is governed by a series of Mutual Legal Assistance Treaties (“MLATs”) between the United States and other countries, which allow signatory states to request one another’s assistance with ongoing criminal investigations, including issuance and execution of search warrants. See U.S. Dep’t of State, 7 Foreign Affairs Manual (FAM) § 962.1 (2013), available at fam.state.gov/FAM/07FAM/07FAM0960.html (last visited May 12, 2016) (discussing and listing MLATs).[29] And he observed that, for countries with which it has not signed an MLAT, the United States has no formal tools with which to obtain assistance in conducting law enforcement searches abroad.[30] These practical considerations cannot, however, overcome the powerful clues in the text of the statute, its other aspects, legislative history, and use of the term of art “warrant,” all of which lead us to conclude that an SCA warrant may reach only data stored within United States boundaries. Our conclusion today also serves the interests of comity that, as the MLAT process reflects, ordinarily govern the conduct of cross‐ boundary criminal investigations. Admittedly, we cannot be certain of the scope of the obligations that the laws of a foreign sovereign—and in particular, here, of Ireland or the E.U.—place on a service provider storing digital data or otherwise conducting business within its territory.  But we find it difficult to dismiss those interests out of hand on the theory that the foreign sovereign’s interests are unaffected when a United States judge issues an order requiring a service provider to “collect” from servers located overseas and “import” into the United States data, possibly belonging to a foreign citizen, simply because the service provider has a base of operations within the United States. Thus, to enforce the Warrant, insofar as it directs Microsoft to seize the contents of its customer’s communications stored in Ireland, constitutes an unlawful extraterritorial application of the Act. [27] We thus disagree with the magistrate judge that all of the relevant conduct occurred in the United States. See In re Warrant, 15 F. Supp. 3d at 475–76. [28] The concurring opinion suggests that the privacy interest that is the focus of the statute may not be intrinsically related to the place where the private content is stored, and that an emphasis on place is “suspect when the content consists of emails stored in the ‘cloud.’” Concurring Op. at 14 n.7. But even messages stored in the “cloud” have a discernible physical location. Here, we know that the relevant data is stored at a datacenter in Dublin, Ireland.  In contrast, it is possible that the identity, citizenship, and location of the user of an online communication account could be unknown to the service provider, the government, and the official issuing the warrant, even when the government can show probable cause that a particular account contains evidence of a crime. [29] The United States has entered into an MLAT with all member states of the European Union, including Ireland. See Agreement on Mutual Legal Assistance Between the European Union and the United States of America, June 25, 2003, T.I.A.S. No. 10‐201.1. 30 In addition, with regard to the foreign sovereign’s interest, the District Court described § 442 (1)(a) of the Restatement of Foreign Relations Law as “dispositive.” Tr. of Oral Arg., supra note 25, at 69. That section provides: A court or agency in the United States, when authorized by statute or rule of court, [is empowered to] order a person subject to its jurisdiction to produce documents, objects, or other information relevant to an action or investigation, even if the information or the person in possession of the information is outside the United States. Restatement of Foreign Relations Law (3d) § 442(1)(a) (1987).  We are not persuaded. The predicate for the Restatement’s conclusion is that the court ordering production of materials located outside the United States is “authorized by statute or rule of court” to do so. Whether such a statute―the SCA―can fairly be read to authorize the production sought is precisely the question before us. Continue Reading . Here About HackerNoon Legal PDF Series: We bring you the most important technical and insightful public domain court case filings. This court case No. 15–777 retrieved on September 27, 2023, from is part of the public domain. The court-created documents are works of the federal government, and under copyright law, are automatically placed in the public domain and may be shared without legal restriction. cases.justia.com

Stories, News & Mentions about MLABS

Mlabs’s Stories on HackerNoon