neLia

nelia

Time estimation is an important skill for software developers, a nightmare for project managers, and just as opaque for customers. Why such a situation? Let's take a look! What is time estimation? is an estimate of the number of hours required to complete a task or series of tasks. The better you understand the features and nuances of the work, the more accurate estimates of time you can get. Time estimation 3 main stages of software evaluation: Estimation of product size for development Estimation of effort in man-months/man-hours and schedule in calendar months Estimation of the project cost in money Time estimation plays a very important role in software development, because without it you will not be able to estimate the duration and cost of a software development project. Why you need time estimation in software development If you want to correctly calculate the time and material costs of the project, not lose money and time during its implementation, you need to know how much time and how much effort you need to put into this project. Let's take a look at why proper a time estimate is so important: Underestimation of time and effort can lead to the fact that you will offer a job at a price that will only bring you losses. If you overestimate the time and effort, it can lead to the fact that you will not get the project. One of your competitors who calculated more accurately can receive this project. As a result, try to objectively and realistically assess your abilities. It will help you to determine as accurately as possible the time and material costs required to develop the project. Why time estimations go wrong Let's look at the main reasons why software development time estimations can go wrong: Not enough attention was paid to the project estimation Negligence. . It is impossible to anticipate all the problems and obstacles in the development and estimate the time frame. However, you should include extra time for problem-solving into account Difficulties are underestimated Project managers and project staff are pressured by specifications like requirements, goals and time frames. Sometimes it can lead to polite estimates of time and costs that later cannot be made. Customers’ restrictions and unrealistic time frame characteristics. "Pink glasses" combined with avoidance of conflict often lead to unrealistically optimistic assessments Too much optimism. Tips based on our experience: projects can often take longer and can be more expensive than planned, so estimate your time correctly. How to avoid incorrect time estimations You need to make the most accurate and reliable estimate of time and material costs. But the more precisely you want to do it, the more effort is required, as well as extra time. Prepare well, use the most qualified evaluators and act systematically. Best practices for time estimation In order to achieve reliable time estimates, you need to remember these two things: Your experience is your best friend who will tell you what to do best and what mistakes to avoid. Personal experience. If you or your competitors have done similar projects, you need to rely on time and material costs for similar projects. Analogy. Also, there are many practices and methods that are actively used in time estimation. The most important factors that affect the accuracy of time estimates are: Accurate evaluation of all input data; Accuracy of all calculations; Maximum proximity of historical data or industry data; Predictability of the software development process; Stability of product requirements. The best 4 methods for time estimation in software development Method of comparison: Use the experience of previous projects The method of comparison or analogy is based on the systematic recording and evaluation of the experience of previous projects. The quality of experience-based evaluation is better than the quality of purely intuitive evaluation. Key figure method: Create your own system There are certain key indicators available for different types of projects. They have been identified many years ago by specialists with a lot of experience in this industry. The main advantage of this method is that with the help of such key indicators it is possible to quickly and easily estimate the expected overall effort. Using the bottom-up method for a detailed assessment The bottom-up method takes a relatively long time but provides a good assessment of quality. Due to the high level of detail on every stage, you can create a detailed project plan, which you can use at the implementation stage to manage projects. This approach requires a work division structure with the required level of detail and appropriate workflow descriptions. Team evaluation: Difficult but high-quality evaluation Practical experience shows that the quality of group assessment is much better than the individual. Therefore, when possible, you should involve your employees and colleagues. "In particular, for large and risky projects, you should pass the evaluation test according to the . In order to be able to evaluate bottom-up, you must have a work-sharing structure with detailed descriptions of work packages based on clear project objectives", - comments Nelia Kovbasa that helps startups with their projects and software development in Delphi method GTM Plus. Conclusion: why it is important to estimate time correctly Time estimation plays a particularly important role in the project planning process. Remember that you should: Allocate sufficient resources to properly manage the time estimate and obtain all the data needed for the most accurate forecasting Structurize the work unit and project scope information. Analyze external risks that may prevent the desired results from being achieved in a timely manner. Analyze internal risks, including potentially inaccurate budget estimates and their possible impact on the project schedule. The keys to estimating time correctly are using strong evidence, avoiding overly optimistic assumptions, and taking into account uncertainty. Also published on: https://dzone.com/articles/time-estimation-in-software-development-what-shoul

What you Need to Know About Time Estimation in Software Development

are everywhere, but what’s behind their rapid growth? Let’s consider some aspects of to find out more about its power. IoT and Big Data analytics IoT Big Data What Is IoT? The Internet of Things, or IoT, basically means integrating the Internet into anything beyond technology and devices to make them smarter. Putting it differently, IoT makes everything in the world function through the Internet, such as through information processing or data management. The Importance of IoT You may wonder why make things even more complicated with ? They don’t actually get more difficult to use; instead, they become smarter. And when things become smarter, they can perform the tasks they’re supposed without any human interaction. IoT and Big Data analytics That’s how our lives become more convenient. Since are used in farming, lighting, and numerous other industries, you may not even know that IoT has been used to simplify a process. Still, IoT plays a vital role in the fast and effective performance of various systems. IoT and Big Data projects And that’s why IoT matters. IoT and Big Data Trends IoT keeps advancing, and new approaches to using it have been occurring. Among such, there are several most popular trends. IoT Big Data Edge Computing Connecting data to powerful devices, as opposed to clouds, helps speed the processing. That’s what edge computing does because of being powered by . Instead of storing raw data, which is what cloud technology aims to do, edge computing now allows gathering and filtering information fast and leave you with the most meaningful parts of it to store. — these are what offers with edge computing. IoT and Big Data analytics Less space, more value IoT Big Data A New Level of Retail Business Consumer behavior is becoming much easier and more oriented on details. With access to diverse data channels, Big Data helps in predictive analytics a lot. Retailers can now collect information on customers’ previous purchases and create an in-depth portfolio of the target audience. Can you imagine visiting a store, both online and on-premises, and seeing everything you’ve been planning to buy over the previous year? Well, it’s exactly what’s coming with online, through apps, or even in kiosks IoT Big Data. Quantum Computing help process data at the most rapid pace than ever before. Quantum computing handles believed to be impossible. In fact, processing loads of information that are these huge doesn’t take forever since it’s a completely new way of dealing with this kind of data. Although it’s still not fully developed, the area has already gained much interest and become one of the trends. IoT and Big Data analytics theoretical and complex calculations in physics IoT Big Data Cybersecurity Using the perks of to enhance cybersecurity is one of the trends as well. Integrating into the breach and threat analysis allows generating and spreading response faster than a human would react. In addition, storing information on previous incidents with technology simplifies predicting the potential ones to develop strong and secure protection against them in the future. IoT and Big Data projects Machine Learning and Artificial Intelligence IoT Big Data What’s Next? Since is a relatively new area, it has a lot to face ahead. The predictions on what to expect in imply an increase in devices, data, and investments in the field, along with their growing role in various industries. IoT Big Data IoT and Big Data analytics At the same time, the skill shortage is a significant challenge to overcome in IoT and Big Data, which is predicted to continue. Another challenge that the area has been facing is the lack of governmental regulations to guide the integration into industries. Lastly, updating IoT hardware can be difficult and lead to delays in making things smart. The rise of interest to IoT started a series of events devoted to the future applications of the technologies available and seeking solutions to the issues in the industry. Among such, AI & Big Data Expo Europe 2019 gathers leaders in the tech industry to discuss tech-savvy solutions and the future of . The conference will take place in Amsterdam on 19–20 June. IoT Big Data

The Power of IoT and Big Data

Top 10 Skills of Developers in 2019 A decent developer is a huge rarity. The problem is not even in the poor knowledge of programming language, but in an integrated approach. The modern developer should not only comprehend, but also integrate modern technologies into their work and be aware of the correlation between innovation and high demand of the computer software. Only this developer will be the most sought after in any company, no matter how many programming languages ​​he/she knows. Today more than 50 programming languages ​​are being actively used, but the most significant ones can be counted on the fingers of one hand. They constitute the great bulk of the market and bring income to programmers. Following this introduction, we approach the first and the most important skill of any developer. Skill 1 — Programming Languages In order to be a popular and promising developer with a good monthly salary, it’s enough to be fluent in only two or three common programming languages ​​from the listed below: The Ranking is based on the polling data among programmers from spectrum.ieee.org The table shows the programming languages, their popularity and the platforms on which they are used. If you are going to study the first or the new programming language, it is worth starting not from its prevalence, but from the goals you are pursuing and the areas that interest you. Let’s address the first 5 programming languages. Python The language is common in various areas of development, but it is a serious rival of the other languages in the following directions: In recent years, developers of the websites and apps, based on the client-server model structure, widely use popular Python frameworks: Django (completed version with administrator bar and interface, speed and minimal need of skills are its advantages) and Flask (a shell that may be customized for specific programmer needs, it is the best option for creating a REST API — client-server architecture style for building scalable distributed web-site systems and applications, as an example, the social network API) Web-development. Python has two popular and frequently used libraries: Scikit-learn (for more primitive solutions) and TensorFlow (for major and high-performance solutions), which can be used for neural networks creation, deep machine learning, as well as analytics and forecasting. One of the most primitive ways of solving learning tasks is the “smart” news feeds in social networks. Machine learning. С++, C, C# C language is the ancestor of C # and C ++. This programming language, like its successors, is used to create software. The main designation of these languages ​​is the development of system software. The main advantage of it is the high processing speed, which allows you to create modern, resource-intensive games and apps in C ++ and C #. If you plan to develop software or games (game engines like Unreal Engine, Frostbite, Unity), then this programming language should be learnt in first place, since in the upcoming years it certainly will not be losing its ground in this sphere. This is evidenced by the positive rating dynamics: none of these programming languages have left the Top 7 over the past 10 years. However, it is disturbing that this language has recently paused to develop. Except the minimal adjustment and interface elaboration, there have been no breakthrough innovations in code and algorithms improvement for a long time. Java Until 2006 nobody could imagine that Java language would be used not only in the household equipment programming. Nowadays it is the most popular language for creating cross-platform software with compatibility support for older code versions. It is used to create applications, games, to synchronize devices, to develop web services, banking and financial systems. One of the downsides is the lack of flexibility, hence it is currently reminding Lego. All that you can create or write in Java is already scattered throughout the internet. To create a mid-level Java app, you need only a minimal understanding of programming principles and basic engine skills search in order to find patterns of completed projects on the web. This is what made Java one of the most popular programming languages ​​for beginners. As you can see, each language is designed for its own sphere, where it looks more promising and more convenient. Although in the light of modern realities, it is not enough for programmers to know one language, it is much more important to have additional skills that increase the level of the specialist’s demand. Skill 2 — Client-server architecture Keeping in mind that recent developments are in 80 % of cases based on the client-server model, its understanding is rather mandatory than desirable for the developer. Poorly designed architecture may lead to the dozens of typical problems linked with processing and distribution of http-requests. Therefore, the ignorance of this model may be the reason of the employers’ denial. Client-server architecture is a system of interaction between user’s device and server host. The device is a client that connects to the server and delivers POST requests via Internet. The server, in turn, is engaged in data processing and it sends the completed solution back to the device. Thus, it minimizes the risks of client’s software being hacked and also it reduces the level of strain on the device, because the main calculations are made at server capacities. Besides, the client-server architecture enables to save and restore client data on a new device, if the old one failed or was formatted. First of all, it is essential for the developer to understand the REST philosophy, which was originally conceived as a system of request management for interaction between the server and the client: Get, Post, Put, Delete with a data relevance handler. Client-server architecture is used in: business automation; online games; commercial applications; website development; cloud computing. Skill 3 — Systems engineering Without implementing the basis of systems engineering, it will be extremely difficult for a programmer to create a unique and competitive product. Systems engineering is a big course, which, unfortunately, is taught in an extremely compressed form by the majority of educational institutions where they address the next main points only in a shallow way: . Competence in this field will help you to analyze and examine volatility of the market, its movement toward technological (integration of modern technologies, development prospects), social (wishes, interest of users, value for the consumer), functional (maximum of possibilities, without sacrificing UI convenience) aspects. This approach to the development of the final product will allow you to see the flaws and deficiencies of the idea that is always imperfect at the initial stage and requires a more accurate modeling, a clear strategy. Market analysis It implies a strong prioritization and time control over tasks performance. It is a condition that directly affects the development deadlines. Breaks or delays may lead to the release of the product irrelevant to the market. The project managers too often don’t fully understand the scope of the tasks to be done and set deadlines based on their own ideas. This leads to the discrepancies between planned deadlines and real time frame. Therefore, the developers should be involved in plan timing, but not only managers. Planning. The life cycle is a continuous process, starting from the moment a decision is made to develop a product and ending at the moment of its complete decommissioning. The developer needs to know all the stages of the product life cycle, as well as the tasks and requirements typical for each stage. It will help the developer to create a high-quality product that will be relevant for a very long time. Life cycle. These three key points are closely related. Ignoring one of them may pull the rest of them into the abyss, and vice versa. However, even a well-mastered system engineering course will not be productive if a non-tested product with a large number of bugs is released. Skill 4 — Basics of software testing A completed software shouldn’t be released without pre-testing. At present, there are 10 methods and kinds of product testing, but the programmer may only master three of them: This type implies testing each individual unit (module, class) for proper operation. It is relevant and correct to use the TDD methodology (Test-driven development). In some languages, there are different frameworks for testing. For example, there are NUnit, Rhino Mocks, etc. in Visual Studio. Unit testing. The most difficult type of testing that involves testing the interaction between blocks (modules, classes). This method is time-consuming, but it will allow you to exclude private incorrect processing of data. Integration testing. This testing method involves the study of an already completed product. Testing of small projects may be done on your own, but the large and resource-intensive ones require using automation tools. System testing. Skill 5 — Databases Creating any software is impossible without implementing a database where all of its data is stored. It is a group of files that contain records in a template structure. If it is enough to have a simple database connection system for a regular client program, then a cloud or client-server application needs a more serious remote database access scheme. Understanding the principles of working with the database and basics of building connections in client-server solutions is a promising and extremely essential skill for the developer, along with knowledge of the security principles, especially if the programmer plans to work on serious business projects. Skill 6 — Encryption and cryptography Many programmers who keep up with modern technologies and cyber-security have already come across the concept of cryptography or, in other words, data encryption. Today, two cryptography methods are used: . The method was relevant even in the time of Julius Caesar, when the letters were shifted a certain number of places down the alphabet. But over time, encryption technologies became more complicated and eliminated the ability to decrypt a character set without a specific key. The method is used nowadays, but it is almost irrelevant. Private key The method was first invented in 1975, however, it gained the maximum spread and popularity only in 2009, when the popular Bitcoin cryptocurrency was built on its basis. From that time, encryption has been actively introduced into any software and has undergone many modifications. Public key. Nowadays there are 4 cryptographic primitives in cryptography: symmetric-key; asymmetric-key; hashing; digital signature. Each primitive has its own appliance and usage features. Therefore, if you are determined to develop business apps, work with personal data or plan to transfer voice/text/video messages among individuals or legal entities, be sure to learn the principles of cryptography. This knowledge will be especially vital if you plan to use modern technology — Blockchain, on which there are already many interesting projects. Skill 7 — Blockchain Blockchain technology is used to maintain a high level of protection for transactions and personal data of users, as well as for the reliability of the wallet and the entire system. The principle of blockchain work may be compared to torrent trackers, where the files or, in our case, the transaction log of the running client is compared to other clients and replicated to new ones, thus copying the log data to a new device. All devices (transmitters and receivers) are anonymous spots that do not transmit any identification data, but only exchange blocks (groups of facts). The main advantage of the blockchain in the modern world is the impossibility of replacing events that have already occurred. And this is currently the main advantage of its widespread implementation, because it is impossible to deceive this system. Therefore, if you think about your future and want to be a sought-after programmer, we advise you to explore this promising technology. Skill 8 — Artificial intelligence Artificial intelligence, like blockchain, is a rapidly developing technology, the huge potential of which we will observe in the future. According to the definition given by Andreas Kaplan and Michael Henlein, artificial intelligence is “system’s ability to correctly interpret external data, to learn from such data, and to use this learning to achieve specific goals and tasks through flexible adaptation.” Globally, AI mimics the model of human thinking. If the program simply retrieves values ​​from the database according to certain rules, it is not endowed with AI. If the system after training creates programs, methods and documents, solving certain tasks, it possesses AI. A developer who is competent in this sphere will definitely be in demand both today and in the near future. Skill 9 — Cross-platform software The cross-platform application allows you to produce a final product, written in a “universal” form for several operating systems. This is a big benefit for distributing and increasing the popularity of software. Virtually all the development environments for creating projects in high-level programming languages ​​are cross-platform and easily compile code to different devices without the need for recompilation. Thus, you can double or even triple the customers amount of your software. Many people currently abandon the use of PCs and switch to mobile devices: smartphones, laptops, tablets. Therefore, it is important to adapt the developed software for the maximum number of systems before release. The only disadvantage here is that a cross-platform application will never be “perfect” for a specific individual OS. Skill 10 — Operating Systems That is why it is essential to not only build cross-platform apps but also to optimize it further for a specific system, taking into account the interaction peculiarities. Learn the new features of operating systems, analyze the way you can apply them in the completed software in order to improve interaction with a consumer, improve usability, speed of data processing, etc. In conclusion, it should be added that it’s not obligatory to possess all the 10 skills listed above. It is better to possess the perfect knowledge in something than to be able to do a bit of everything. Choose what you want to do, that is interesting for you, set a goal and stick to it. Do not forget about trends in your area and new solutions that will improve professional skills and allow the developer to be in high demand for many years. <a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a>

Top 10 skills of developers in 2019

This interview series celebrates the remarkable achievements of the Hackernoon . Startups of the Year winners Meet , a content management system (CMS) for academic journals that put up impressive numbers in 2023. We sat down with Neliti CEO Anton Lucanus, whose transition from research scientist to tech entrepreneur has enabled millions across Southeast Asia to access the region's rich trove of academic content. Neliti https://hackernoon.com/startups/asia/asia-singapore-singapore?embedable=true What does it mean for you to win this title? Amidst the buzz around Web 3.0 and AI, it's nice to see our Web 2.0 SaaS product among the winners. We’re not fancy, but we address a very real problem: making academic research more easily accessible in developing countries. It’s affirming to know that this problem and our solution to it are receiving recognition. The entire team at Neliti is deeply committed to this mission, and the Hackernoon Startups of the Year title provides us with a healthy dose of rocket fuel! Tell us how your startup is changing the world. Academic publishing is the editing and publication of academic journals, which are publications containing all the scientific papers in a certain field. It is the main method for communicating the results of recent research. Humanity relies on this method of communication to solve the world’s biggest scientific challenges, such as human disease, climate change, and poverty. A significant obstacle within the academic publishing system is the limited access to vital scientific papers for researchers in developing countries, hindering their ability to contribute solutions to the world’s most critical scientific challenges. Neliti bridges this gap by enabling academic content providers, such as academic publishers, to make their research openly and readily available. The impact of our solution is evident in the millions of emerging researchers who rely on Neliti monthly to access the essential data they need for their work. What sets you apart from the competition? Our holistic understanding of the unique challenges and needs within the academic publishing sector, particularly in the developing world. Academic articles, the vessels of important scientific discoveries, are disseminated via three primary channels: academic journals, academic conferences, and institutional repositories. The latter serve as open-access digital libraries hosting a diverse range of academic content, from scholarly articles and books to scientific datasets. There are two main types of existing solutions in our industry: free open source solutions (such as OJS for journals or ePrints for repositories) and paid solutions (such as Silverchair for journals or Pure for repositories). The challenge with the free open source options is their complexity and lack of user-friendliness, requiring substantial technical know-how to deploy and maintain. This poses a significant barrier for their primary users—academics and librarians—who are experts in their fields but not in web development, often leading to subpar and bug-prone websites. On the flip side, the premium solutions, while robust, are prohibitively expensive for all except the most affluent institutions in developed nations. Neliti aims to disrupt this space by offering a solution that is as user-friendly as it is powerful, all while being free of charge. Our vision is to enable even the smallest universities in developing countries to create academic journals and institutional repositories that surpass the aesthetics and functionality of those developed by wealthy institutions with extensive budgets. This ambition is at the heart of our UI/UX team's efforts. Beyond aesthetics, we are committed to alleviating the technical burdens our users face. By managing the entire tech stack, including web hosting and content indexing, we allow our users to concentrate on what they’re good at: publishing impactful research. What do you love about your team, and why are you the ones to solve this problem? I hold immense admiration for every member of our team. One thing I love is our celebration of each individual's uniqueness—their personalities, skills, and respective contributions to our overarching mission. Another thing, and probably what I treasure most, is the depth of our relationships. I see each team member as a friend first, which naturally enriches our work. As a remote team, our annual in-person meetups are particularly special. These occasions allow us to share our passion for the Neliti mission, creating a buzz of shared purpose and enthusiasm. This, coupled with our unique team dynamics, positions us ideally to address the complexities of open access in the academic world, especially within developing regions. At the moment, how do you measure success? What are your core metrics? Neliti’s success is quantitatively gauged by a set of core metrics that reflect our widespread adoption and impact: number of institutions publishing content using our platform (currently 1,200+), number of articles published (400,000+) number of researchers accessing that content each month (2 million+). What goals are you looking forward to accomplishing in 2024? Currently, Neliti is mostly popular in Indonesia. But over the last few years, we’ve seen sporadic usership from dozens of other developing countries. In 2024, we hope to go deep into at least one of those markets. We’ll keep it a surprise as to which one 😉 If you weren’t building your startup, what would you be doing? Writing a book about the history of the academic publishing system, how it evolved to where it is today, and how we can improve the industry. The book would shed light on how the industry can be reformed and improved for the betterment of scientific discovery and human advancement more generally. I find the academic publishing industry fascinating. In 1655, the academic journal emerged. At the time, it was a revolutionary development in scientific communication. It provided a transparent and open exchange of evidence-based findings, while also indicating clear provenance of scientific discoveries. The current system hasn’t change too much since 1655, relatively speaking. In the current status quo, a team of researchers spends months (even years) getting funding and performing rigorous research, all of which culminates in a paper that is submitted to an academic publisher. The paper is then evaluated by peer reviewers, who are other researchers in the field. This is the main quality-assurance mechanism in scientific research. If accepted by the publisher, the paper is published. Libraries then pay up to millions of dollars annually for journal subscriptions to the publisher to access the research. Academic publishing is an odd system — the authors are not paid for their writing, nor are the peer reviewers (they’re just more unpaid academics), and in some fields even the journal editors are unpaid. The authors sign away their intellectual property rights to publishers for free, and sometimes they must even pay the publishers. And yet scientific publications are some of the most expensive pieces of literature you can buy. It is such an odd system that those who fund and perform research must actually pay to read the research they create, while taxpayers must pay to access publicly funded research. This scenario underscores why many within the academic community view the system as “broken”. Yet, there's a general lack of understanding regarding how the industry reached its current state and, more importantly, what measures can be taken to rectify it. My book would delve into these issues, offering a comprehensive exploration of the academic publishing landscape and proposing pathways for its improvement. Which trend(s) are you most excited about in 2024? Share your reasons. OK, let’s take a break from academic publishing for a minute! I’m most excited about scientific advancement in the ageing and longevity space. There have been major breakthroughs in understanding human ageing over the last decade, and 2024 is a big year for the field. I’m excited for this, because there is a growing consensus among scientists that ageing is a “disease” that can be solved. 2023 had been another crazy year, especially in tech, with layoffs and the Generative AI takeover! Which trend are you most concerned about? Be as brief or as detailed as you like. AI. We didn’t know smoking was dangerous until it was. We didn’t know asbestos was dangerous until it was. We didn’t know nuclear power was dangerous until it was. AI is like that, on steroids, and irreversible. Political intervention needed. Even more so than the great Partial Nuclear Test Ban Treaty of 1963. We can learn from history, but the economic and national security benefits of AI currently pose too much of a benefit to any country considering its regulation. This is at the core of its threat. History has repeatedly shown us that the allure of such benefits can make it difficult to regulate powerful technologies like AI. Such a tension between benefits and risks is not new. Unfortunately, it often takes a disaster to catalyze the implementation of necessary regulations. I hope we can learn from our past to proactively manage the risks of AI, rather than waiting for adverse outcomes to force our hand. We would love your feedback on HackerNoon as a tech publication! How has your experience been with us? Hackernoon is awesome. It’s a jungle of varied perspectives on all tech trends, which makes for some very unfiltered reading. I find it important to read opinions I disagree with, because the truth often lies somewhere in the middle. Any words of wisdom you’d like to share with us? Every groundbreaking discovery was discovered by a human like yourself. Every groundbreaking innovation was ideated by a human like yourself. The human brain is phenomenal, what will you do with yours? Startups of The Year is HackerNoon’s Flagship community-driven event celebrating startups that survived and thrived in 2023. 30,000 startups across 4200+ cities and six continents participated this year to be crowned the best startup in their city. The 2023 Startups Of The Year is sponsored by: . . The winners will get a free .Tech Domain, a HackerNoon NFT, and a Tech Company News Page. TECH Domains

Meet Neliti, Winner of the Startups of the Year in Singapore

This tutorial shows you how to deploy and automate a Docker container using Pulumi in Google Cloud Platform's Cloud Run with minimum permissions. Here's the GitHub Repository for reference: https://github.com/superjose/deploy-to-cloud-run-go Get the CLIs: . Create a Google Cloud Account Install Google Cloud CLI Install Pulumi CLI - We need it to build the Docker container. Install Docker Bootstrap the project: Create a directory: pulumi mkdir pulumi Run to initialize a pulumi project with Go (It can be your language of choice pulumi new go Navigate to the directory. pulumi cd pulumi Navigate to and create a new project. cloud.google.com Take note of the (Usually the project's name). project-id Generate the necessary permissions using gcloud CLI Login with the Google Auth CLI: gcloud auth login (Optional) Set the project in google cloud CLI (Can be changed anytime). This saves you from passing into every command. --project [PROJECT-ID] gcloud If your machine has multiple GCP projects, skip this step and pass the flag into every command. --project gcloud Create a service account (The account that Pulumi will connect to): gcloud iam service-accounts create pulumi-gcp --description="Pulumi GCP" Download the credentials for the service accounts and store them locally (Remember to replace with your GCP Project Id): [PROJECT-ID] gcloud iam service-accounts keys create ~/keys/gcp/pulumi-service-account-key-file.json --iam-account=pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com Set Pulumi's gcp credentials config path:
(This will connect the service account with Pulumi) pulumi config set gcp:credentials ~/keys/gcp/pulumi-service-account-key-file.json Set the GCP Project by doing: pulumi config set gcp:project [PROJECT-ID] Create a file (Inside the dir) and add the required permissions in : roles.gcp.yml pulumi includedPermissions # https://cloud.google.com/iam/docs/creating-custom-roles#creating
# Yaml to define the Pulumi GCP Roles that need to be created with gcloud CLI
title: Pulumi GCP Roles
description: |
  This policy ensures that all GCP roles are created using Pulumi.
stage: GA
# https://cloud.google.com/iam/docs/permissions-reference
includedPermissions:
  - serviceusage.services.list
  - serviceusage.services.enable
  - serviceusage.services.disable
  - serviceusage.services.get
  - serviceusage.services.use
    # Permissions for GCR
  - storage.objects.create
  - storage.objects.delete # Optional: only include if you need to delete images
  - storage.objects.get
  # Permissions for Google Artifact Registry
  - artifactregistry.repositories.create
  - artifactregistry.repositories.delete
  - artifactregistry.repositories.get
  - artifactregistry.repositories.list
  - artifactregistry.repositories.update
  - artifactregistry.repositories.downloadArtifacts
  - artifactregistry.repositories.uploadArtifacts
  - artifactregistry.repositories.deleteArtifacts
  # Permissions
  # Permissions
  - run.services.create
  - run.services.get
  - run.services.list
  - run.services.update
  - run.services.delete
  - run.services.getIamPolicy
  - run.services.setIamPolicy
  - iam.serviceAccounts.actAs
# NOTE: This should be removed the first time you're creating a role.
# This etag is to update the current active role (As GCP lets you manage multiple roles)
# I'm commenting it out so I can always replace the role
# etag: BwYS74Xx5y4= Create the with the file above:
(We assume we're running this code from the directory) pulumi_admin_role pulumi gcloud iam roles create pulumi_admin_role --project=[PROJECT-ID] --file='./roles.gcp.yml' In case you need to make edits, change the file and use: gcloud iam roles update pulumi_admin_role --project=[PROJECT-ID] --file='./roles.gcp.yml' We're also adding the role (I haven't found a better way) (Otherwise we'd get 403 errors when refreshing and updating in Pulumi)<sup>1</sup> serviceAccountAdmin gcloud projects add-iam-policy-binding [PROJECT-ID] --role roles/iam.serviceAccountAdmin   --member serviceAccount:pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com (The following will add the new role to the service account) gcloud projects add-iam-policy-binding [PROJECT-ID] --role projects/[PROJECT-ID]/roles/pulumi_admin_role   --member serviceAccount:pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com Our in the directory:
(Check the code for comments!) main.go pulumi We enable the required services (Artifact Registry, and Cloud Run). Artifact Registry is used to host the docker container image. Cloud Runner will launch the Docker image from Artifact Registry. We build the docker image locally (We specify the platform in case you're using an ARM chip like M1, M2, Snapdragon SQ, X Elite, etc.) We create a chain of "DependsOn" to notify Pulumi:
5.1 Services need to be enabled first
5.2 We create the Artifact Repository
5.3 We build the docker image and push it to Artifact Registry.
5.4 We pull the Docker Image from Artifact Registry and run it.
5.5 We add IAM permissions so it can be accessed from anywhere. package main

import (
	"errors"
	"log"
	"os"

	"github.com/joho/godotenv"
	"github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/artifactregistry"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/cloudrun"
	"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/projects"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

// This is the name you created in Google Cloud Platform (GCP).
const gcpProjectId = "deploy-to-cloud-run-go"

// The Docker Image Name
const dockerImageName = "my-app-docker"
const artifactRegistryServiceName = "artifact-registry-api"
const artifactRegistryRepoName = "my-app-artifact-repo"
const artifactRegistryRepoLocation = "us-east1"
const cloudRunAdminServiceName = "cloud-run-admin-service"
const cloudRunServiceName = "cloud-run-service"

// For more info: https://cloud.google.com/run/docs/locations
const cloudRunLocation = "us-east1"

// The tag for the Docker image
const imageTag = "latest"

// This is a url like: us-east1-docker.pkg.dev
// It is used to push the Docker image to Google Container Registry
// For more info: https://cloud.google.com/container-registry/docs/pushing-and-pulling
// The format is: <region>-docker.pkg.dev
var dockerGCPServer = cloudRunLocation + "-docker.pkg.dev"

// The full path to the Docker image
// It is used to deploy the Docker image to Google Cloud Run
// The format is: <region>-docker.pkg.dev/<project-id>/<repo-name>/<image-name>:<tag>
// For more info: https://cloud.google.com/run/docs/deploying
// Example: us-east1-docker.pkg.dev/deploy-to-cloud-run-go/my-app--artifact-repo/my-app-docker:latest
var dockerImageWithPath = dockerGCPServer + "/" + gcpProjectId + "/" + artifactRegistryRepoName + "/" + dockerImageName + ":" + imageTag

func main() {

	// Load the .env file
	err := godotenv.Load()

	if err != nil {
		log.Fatal("Error loading .env file")
	}

	pulumi.Run(func(ctx *pulumi.Context) error {

		enabledServices, serviceResultErr := enableServices(ctx)

		if serviceResultErr != nil {
			return serviceResultErr
		}

		artifactRegistryRepo, createArtifactErr := createArtifactRegistryNewRepository(ctx, &enabledServices)
		if createArtifactErr != nil {
			return createArtifactErr
		}

		dockerImage, buildAndPushErr := buildAndPushToContainerRegistry(ctx, &enabledServices, artifactRegistryRepo)

		if buildAndPushErr != nil {
			return buildAndPushErr
		}

		deployContainerErr := deployContainerToCloudRun(ctx, &enabledServices, dockerImage)

		if deployContainerErr != nil {
			return deployContainerErr
		}

		return nil
	})
}

type EnabledServices struct {
	CloudRunService         *projects.Service `pulumi:"cloudRunService"`
	ArtifactRegistryService *projects.Service `pulumi:"artifactRegistryService"`
}

func enableServices(ctx *pulumi.Context) (EnabledServices, error) {

	cloudRunService, cloudRunAdminErr := projects.NewService(ctx, cloudRunAdminServiceName, &projects.ServiceArgs{
		Service: pulumi.String("run.googleapis.com"),
		Project: pulumi.String(gcpProjectId),
	})

	if cloudRunAdminErr != nil {
		return EnabledServices{}, cloudRunAdminErr
	}

	artifactRegistryService, err := projects.NewService(ctx, artifactRegistryServiceName, &projects.ServiceArgs{
		Service: pulumi.String("artifactregistry.googleapis.com"),
	})

	if err != nil {
		return EnabledServices{}, err
	}
	return EnabledServices{
		CloudRunService:         cloudRunService,
		ArtifactRegistryService: artifactRegistryService,
	}, nil
}

func createArtifactRegistryNewRepository(ctx *pulumi.Context, enabledServices *EnabledServices) (*artifactregistry.Repository, error) {

	if enabledServices == nil || enabledServices.ArtifactRegistryService == nil {
		return nil, errors.New("enabledServices cannot be nil")
	}

	dependingResources := []pulumi.Resource{
		enabledServices.ArtifactRegistryService,
	}

	repo, err := artifactregistry.NewRepository(ctx, artifactRegistryRepoName, &artifactregistry.RepositoryArgs{
		Location:     pulumi.String(artifactRegistryRepoLocation),
		RepositoryId: pulumi.String(artifactRegistryRepoName),
		Format:       pulumi.String("DOCKER"),
		Description:  pulumi.String("The repository that will hold social-log Docker images."),
	}, pulumi.DependsOn(dependingResources))

	if err != nil {
		return nil, err
	}

	return repo, nil
}

func buildAndPushToContainerRegistry(ctx *pulumi.Context, enabledServices *EnabledServices, artifactRegistryRepo *artifactregistry.Repository) (*docker.Image, error) {

	if enabledServices == nil || enabledServices.ArtifactRegistryService == nil {
		return nil, errors.New("enabledServices cannot be nil")
	}

	if artifactRegistryRepo == nil {
		return nil, errors.New("artifactRegistryRepo cannot be nil")
	}

	// Lookup GOOGLE_CREDENTIALS environment variable which should hold the path to the JSON key file
	jsonKeyPath, present := os.LookupEnv("GOOGLE_CREDENTIALS_FILE_PATH")
	if !present {
		return nil, errors.New("GOOGLE_CREDENTIALS_FILE_PATH environment variable is not set")
	}

	// Read the JSON key file
	jsonKey, err := os.ReadFile(jsonKeyPath)
	if err != nil {
		return nil, err
	}

	dependingSources := []pulumi.Resource{
		enabledServices.ArtifactRegistryService,
		artifactRegistryRepo,
	}

	// Build and push Docker image to Google Container Registry using the JSON key
	image, err := docker.NewImage(ctx, dockerImageName, &docker.ImageArgs{
		Build: &docker.DockerBuildArgs{
			Context: pulumi.String("../"), // Adjust the context according to your project structure

			ExtraOptions: pulumi.StringArray{
				// This option is needed for devices running on ARM architecture, such as Apple M1/M2/MX CPUs
				pulumi.String("--platform=linux/amd64"),
			},
		},
		ImageName: pulumi.String(dockerImageWithPath),
		Registry: &docker.ImageRegistryArgs{
			Server:   pulumi.String(dockerGCPServer),
			Username: pulumi.String("_json_key"),     // Special username for GCP
			Password: pulumi.String(string(jsonKey)), // Provide the contents of the key file
		},
	}, pulumi.DependsOn(dependingSources))
	if err != nil {
		return nil, err
	}

	return image, nil
}

func deployContainerToCloudRun(ctx *pulumi.Context, enabledServices *EnabledServices, dockerImage *docker.Image) error {

	if enabledServices == nil || enabledServices.CloudRunService == nil {
		return errors.New("enabledServices cannot be nil")
	}

	if dockerImage == nil {
		return errors.New("dockerImage cannot be nil")
	}

	dependingSources := []pulumi.Resource{
		enabledServices.CloudRunService,
		dockerImage,
	}

	appService, err := cloudrun.NewService(ctx, cloudRunServiceName, &cloudrun.ServiceArgs{
		Location: pulumi.String(cloudRunLocation), // Choose the appropriate region for your service
		Template: &cloudrun.ServiceTemplateArgs{
			Spec: &cloudrun.ServiceTemplateSpecArgs{
				Containers: cloudrun.ServiceTemplateSpecContainerArray{
					&cloudrun.ServiceTemplateSpecContainerArgs{
						Image: dockerImage.ImageName,
						Resources: &cloudrun.ServiceTemplateSpecContainerResourcesArgs{
							Limits: pulumi.StringMap{
								"memory": pulumi.String("256Mi"), // Adjust the memory limit as needed
							},
						},
					},
				},
			},
		},
		Traffics: cloudrun.ServiceTrafficArray{
			&cloudrun.ServiceTrafficArgs{
				Percent:        pulumi.Int(100),
				LatestRevision: pulumi.Bool(true),
			},
		},
	}, pulumi.DependsOn(dependingSources))

	if err != nil {
		return err
	}

	_, iamErr := cloudrun.NewIamMember(ctx, "invoker", &cloudrun.IamMemberArgs{
		Service:  appService.Name,
		Location: appService.Location,
		Role:     pulumi.String("roles/run.invoker"),
		Member:   pulumi.String("allUsers"),
	})

	if iamErr != nil {
		return iamErr
	}

	ctx.Export("containerUrl", appService.Statuses.Index(pulumi.Int(0)).Url().ToOutput(ctx.Context()))
	return nil
} Update the in your Dockerfile:
There's a , and in which you need to export your environment variable to ENV known issue here HOME /root # Set the ENV HOME before your ENTRYPOINT.
ENV HOME=/root

# This is specific to your project. 
ENTRYPOINT ["/whatever-is-your-entrypoint"] Create a .env file in the pulumi directory: Set the * to the one you saved on full path 11) GOOGLE_CREDENTIALS_FILE_PATH="/Users/myusername/keys/gcp/pulumi-service-account-key-file.json" Run pulumi up You should be up and running! Extras The go.mod module social-log-go

go 1.21

toolchain go1.22.0

require (
	github.com/pulumi/pulumi-gcp/sdk/v7 v7.11.2
	github.com/pulumi/pulumi/sdk/v3 v3.108.1
)

require (
	dario.cat/mergo v1.0.0 // indirect
	github.com/Masterminds/semver v1.5.0 // indirect
	github.com/Microsoft/go-winio v0.6.1 // indirect
	github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 // indirect
	github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da // indirect
	github.com/agext/levenshtein v1.2.3 // indirect
	github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
	github.com/atotto/clipboard v0.1.4 // indirect
	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
	github.com/blang/semver v3.5.1+incompatible // indirect
	github.com/charmbracelet/bubbles v0.16.1 // indirect
	github.com/charmbracelet/bubbletea v0.24.2 // indirect
	github.com/charmbracelet/lipgloss v0.7.1 // indirect
	github.com/cheggaaa/pb v1.0.29 // indirect
	github.com/cloudflare/circl v1.3.7 // indirect
	github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 // indirect
	github.com/cyphar/filepath-securejoin v0.2.4 // indirect
	github.com/djherbis/times v1.5.0 // indirect
	github.com/emirpasic/gods v1.18.1 // indirect
	github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
	github.com/go-git/go-billy/v5 v5.5.0 // indirect
	github.com/go-git/go-git/v5 v5.11.0 // indirect
	github.com/gogo/protobuf v1.3.2 // indirect
	github.com/golang/glog v1.1.0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.3 // indirect
	github.com/google/uuid v1.3.0 // indirect
	github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645 // indirect
	github.com/hashicorp/errwrap v1.1.0 // indirect
	github.com/hashicorp/go-multierror v1.1.1 // indirect
	github.com/hashicorp/hcl/v2 v2.17.0 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
	github.com/joho/godotenv v1.5.1 // indirect
	github.com/kevinburke/ssh_config v1.2.0 // indirect
	github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
	github.com/mattn/go-isatty v0.0.19 // indirect
	github.com/mattn/go-localereader v0.0.1 // indirect
	github.com/mattn/go-runewidth v0.0.15 // indirect
	github.com/mitchellh/go-ps v1.0.0 // indirect
	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
	github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
	github.com/muesli/cancelreader v0.2.2 // indirect
	github.com/muesli/reflow v0.3.0 // indirect
	github.com/muesli/termenv v0.15.2 // indirect
	github.com/opentracing/basictracer-go v1.1.0 // indirect
	github.com/opentracing/opentracing-go v1.2.0 // indirect
	github.com/pgavlin/fx v0.1.6 // indirect
	github.com/pjbgf/sha1cd v0.3.0 // indirect
	github.com/pkg/errors v0.9.1 // indirect
	github.com/pkg/term v1.1.0 // indirect
	github.com/pulumi/appdash v0.0.0-20231130102222-75f619a67231 // indirect
	github.com/pulumi/esc v0.6.2 // indirect
	github.com/pulumi/pulumi-docker/sdk/v3 v3.6.1 // indirect
	github.com/rivo/uniseg v0.4.4 // indirect
	github.com/rogpeppe/go-internal v1.11.0 // indirect
	github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
	github.com/santhosh-tekuri/jsonschema/v5 v5.0.0 // indirect
	github.com/sergi/go-diff v1.3.1 // indirect
	github.com/skeema/knownhosts v1.2.1 // indirect
	github.com/spf13/cobra v1.7.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/texttheater/golang-levenshtein v1.0.1 // indirect
	github.com/tweekmonster/luser v0.0.0-20161003172636-3fa38070dbd7 // indirect
	github.com/uber/jaeger-client-go v2.30.0+incompatible // indirect
	github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
	github.com/xanzy/ssh-agent v0.3.3 // indirect
	github.com/zclconf/go-cty v1.13.2 // indirect
	go.uber.org/atomic v1.9.0 // indirect
	golang.org/x/crypto v0.17.0 // indirect
	golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
	golang.org/x/mod v0.14.0 // indirect
	golang.org/x/net v0.19.0 // indirect
	golang.org/x/sync v0.5.0 // indirect
	golang.org/x/sys v0.15.0 // indirect
	golang.org/x/term v0.15.0 // indirect
	golang.org/x/text v0.14.0 // indirect
	golang.org/x/tools v0.15.0 // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20230706204954-ccb25ca9f130 // indirect
	google.golang.org/grpc v1.57.1 // indirect
	google.golang.org/protobuf v1.31.0 // indirect
	gopkg.in/warnings.v0 v0.1.2 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	lukechampine.com/frand v1.4.2 // indirect
) If you include this Run , and this will fetch all the packages for you go.mod go tidy Footnotes I fought against permissions for 5 days. The predefined GCP role brought in the additional permissions needed. 1* serviceAccountAdmin

Automate and Deploy a Docker Container to Google Cloud Run from Scratch Using Pulumi and Go

This paper is available on arxiv under CC 4.0 license. Authors: (1) Eleonora Alei, ETH Zurich, Institute for Particle Physics & Astrophysics & National Center of Competence in Research PlanetS; (2) Björn S. Konrad, ETH Zurich, Institute for Particle Physics & Astrophysics & National Center of Competence in Research PlanetS; (3) Daniel Angerhausen, ETH Zurich, Institute for Particle Physics & Astrophysics, National Center of Competence in Research PlanetS & Blue Marble Space Institute of Science; (4) John Lee Grenfell, Department of Extrasolar Planets and Atmospheres (EPA), Institute for Planetary Research (PF), German Aerospace Centre (DLR) (5) Paul Mollière, Max-Planck-Institut für Astronomie; (6) Sascha P. Quanz, ETH Zurich, Institute for Particle Physics & Astrophysics & National Center of Competence in Research PlanetS; (7) Sarah Rugheimer, Department of Physics, University of Oxford; (8) Fabian Wunderlich, Department of Extrasolar Planets and Atmospheres (EPA), Institute for Planetary Research (PF), German Aerospace Centre (DLR); (9) LIFE collaboration, www.life-space-mission.com. Table of Links Abstract & Introduction Methods Results Discussion Conclusions Next Steps & References Appendix A: Scattering of terrestrial exoplanets Appendix B: Corner Plots Appendix C: Bayes’ factor analysis: other epochs Appendix D: Cloudy scenarios: additional figures 6. Next Steps Several new interesting questions and opportunities for more detailed studies arise from this work. First of all, we plan to work on a study that will take advantage of the model selection potential that Bayesian retrievals have to offer, for example by comparing retrievals including and excluding non-retrieved parameters (e.g. the CO abundance). We are also performing retrievals assuming various cloud models (Konrad et al., in prep.). Retrievals of hazy planets (see e.g. Arney et al. 2016), as well as ocean worlds, might also help us further quantify the science potential of LIFE for a variety of different planet types. Another interesting study would be to increase S/N and R to even higher values. This will not only evaluate the extreme limits of a concept like LIFE, but also help us better understand if retrievals are limited by R rather than S/N (e.g. due to unresolved narrow features at low R). It would also be useful to compare different R-S/N combinations, this time fixing the observing time. This would help us quantify the best R-S/N combination needed to optimize the characterization of a terrestrial atmosphere. Further work is needed to optimize the yield in the characterization phase of the LIFE mission concept. The estimates of the observation time needed to establish knowledge about the habitability and the presence of biologically relevant molecules in the atmosphere that we derived here are a crucial piece of information for these follow-up studies. In this work, we only used simulated data obtained with the LIFE mission. However, in the future there will likely be more information available to each system and planet. Therefore, it will be important to put this study in context with other observations. For instance, joint retrievals of reflected light data obtained with LUVOIR/HabEx at optical/near-infrared wavelengths and thermal emission spectra as obtained by LIFE would provide useful insight on the synergies between the various missions. One of the most important open questions regarding the ultimate goal of detecting extrasolar life will require to put our results in context with life detection frameworks (e.g. Green et al. 2021; Catling et al. 2018; Walker et al. 2018). Our ongoing retrieval efforts could be useful for the fine-tuning of such frameworks. These, in turn, would provide insight on the meaning and the likelihood of a potential biosignature detection, which would allow us to infer and justify the presence of life forms on another planet. This work has been carried out within the framework of the National Center of Competence in Research PlanetS supported by the Swiss National Science Foundation. S.P.Q. and E.A. acknowledge the financial support from the SNSF. P.M. acknowledges support from the European Research Council under the European Union’s Horizon 2020 research and innovation program under grant agreement No. 832428. J.L.G. thanks ISSI Team 464 for useful discussions. Acknowledgements. E.A. carried out the analyses, created the figures, and wrote the bulk part of the manuscript. B.S.K. and D.A. wrote part of the manuscript. S.P.Q. initiated the project, guided the project and wrote part of the manuscript. Author contributions. All authors discussed the results and commented on the manuscript. This research made use of: Astropy[7] , a community-developed core Python package for Astronomy (Astropy Collaboration et al. 2013, 2018); Matplotlib[8] (Hunter 2007); pandas (pandas development team 2020); seaborn [9] . Software. References Arney, G., Domagal-Goldman, S. D., Meadows, V. S., et al. 2016, Astrobiology, 16, 873 Astropy Collaboration, Price-Whelan, A. M., Sipocz, B. M., et al. 2018, AJ, 156, ˝ 123 Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33 Barstow, J. K., Changeat, Q., Garland, R., et al. 2020, Mon. Not. R. Astron. Soc., 493, 4884 Baudino, J.-L., Mollière, P., Venot, O., et al. 2017, ApJ, 850, 150 Bryson, S., Kunimoto, M., Kopparapu, R. K., et al. 2021, AJ, 161, 36 Buchner, J., Georgakakis, A., Nandra, K., et al. 2014, Astronomy & Astrophysics, 564, A125 Burch, D. E., Gryvnak, D. A., Patty, R. R., & Bartky, C. E. 1969, J. Opt. Soc. Am., 59, 267 Campbell, I. H. & Squire, R. J. 2010, Geochim. Cosmochim. Acta, 74, 4187 Catling, D. C., Krissansen-Totton, J., Kiang, N. Y., et al. 2018, Astrobiology, 18, 709–738 Catling, D. C., Krissansen-Totton, J., Kiang, N. Y., et al. 2018, Astrobiology, 18, 709 Chen, J. & Kipping, D. 2016, ApJ, 834, 17 Chubb, K. L., Rocchetto, M., Yurchenko, S. N., et al. 2021, A&A, 646, A21 Cobb, A. D., Himes, M. D., Soboczenski, F., et al. 2019, AJ, 158, 33 Dannert, F., Ottiger, M., Quanz, S. P., et al. 2022, arXiv e-prints, arXiv:2203.00471 Ertel, S., Defrère, D., Hinz, P., et al. 2020, AJ, 159, 177 Feautrier, P. 1964, Comptes Rendus Academie des Sciences (serie non specifiee), 258, 3189 Feng, Y. K., Robinson, T. D., Fortney, J. J., et al. 2018, AJ, 155, 200 Feroz, F., Hobson, M. P., & Bridges, M. 2009, MNRAS, 398, 1601 Feulner, G. 2012, Reviews of Geophysics, 50, RG2006 Gaudi, B. S., Seager, S., Mennesson, B., et al. 2020, arXiv e-prints, arXiv:2001.06683 Gharib-Nezhad, E. & Line, M. R. 2019, ApJ, 872, 27 Gordon, I. E., Rothman, L. S., Hill, C., et al. 2017, J. Quant. Spectr. Rad. Transf., 203, 3 Graham, R. J. 2021, Astrobiology, 21, 1406, pMID: 34375145 Green, J., Hoehler, T., Neveu, M., et al. 2021, Nature, 598, 575 Gregory, B. S., Claire, M. W., & Rugheimer, S. 2021, Earth and Planetary Science Letters, 561, 116818 Hartmann, J. M., Boulet, C., Brodbeck, C., et al. 2002, J. Quant. Spectr. Rad. Transf., 72, 117 Harvey, A. H., Gallagher, J. S., & Levelt Sengers, J. M. H. 1998, Journal of Physical and Chemical Reference Data, 27, 761 Heller, R., Duda, J.-P., Winkler, M., Reitner, J., & Gizon, L. 2021, PalZ, 95, 563 Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90 Jeffreys, H. 1998, The Theory of Probability, Oxford Classic Texts in the Physical Sciences (OUP Oxford), 432–441 Kaltenegger, L. & Traub, W. A. 2009, ApJ, 698, 519 Kaltenegger, L., Traub, W. A., & Jucks, K. W. 2007, The Astrophysical Journal, 658, 598 Karman, T., Gordon, I. E., van der Avoird, A., et al. 2019, Icarus, 328, 160 Kawashima, Y. & Rugheimer, S. 2019, AJ, 157, 213 Kolmogorov, A. 1933, Inst. Ital. Attuari, Giorn., 4, 83 Konrad, B. S., Alei, E., Angerhausen, D., et al. 2021 [arXiv:2112.02054] Krissansen-Totton, J., Thompson, M., Galloway, M. L., & Fortney, J. J. 2022, Nature Astronomy, 6, 189–198 Lederberg, J. 1965, Nature, 207, 9 Lee, E., Taylor, J., Grimm, S. L., et al. 2019, MNRAS, 487, 2082 Lovelock, J. E. 1965, Nature, 207, 568 Luo, G., Ono, S., Beukes, N. J., et al. 2016, Science Advances, 2, e1600134 Lyons, T. W., Diamond, C. W., Planavsky, N. J., Reinhard, C. T., & Li, C. 2021, Astrobiology, 21, 906 Lyons, T. W., Reinhard, C. T., & Planavsky, N. J. 2014, Nature, 506, 307 Madhusudhan, N. 2018, Handbook of Exoplanets, 2153–2182 Marais, D. J. D. 2000, Science, 289, 1703 Márquez-Neila, P., Fisher, C., Sznitman, R., & Heng, K. 2018, Nature Astronomy, 2, 719 Meadows, V. S., Reinhard, C. T., Arney, G. N., et al. 2018, Astrobiology, 18, 630 Mollière, P., Stolker, T., Lacour, S., et al. 2020, A&A, 640, A131 Mollière, P., van Boekel, R., Bouwman, J., et al. 2017, A&A, 600, A10 Mollière, P., Wardenier, J. P., van Boekel, R., et al. 2019, A&A, 627, A67 Mollière, P., Boekel, R. v., Dullemond, C., Henning, T., & Mordasini, C. 2015, ApJ, 813, 47 National Academies of Sciences, Engineering, and Medicine. 2021, Pathways to Discovery in Astronomy and Astrophysics for the 2020s (Washington, DC: The National Academies Press) Olson, S. L., Schwieterman, E. W., Reinhard, C. T., & Lyons, T. W. 2018, Earth: Atmospheric Evolution of a Habitable Planet, ed. H. J. Deeg & J. A. Belmonte (Cham: Springer International Publishing), 2817–2853 pandas development team, T. 2020, pandas-dev/pandas: Pandas Peterson, B. M., Fischer, D., & LUVOIR Science and Technology Definition Team. 2017, in American Astronomical Society Meeting Abstracts, Vol. 229, American Astronomical Society Meeting Abstracts #229, 405.04 Quanz, S. P., Absil, O., Angerhausen, D., et al. 2021, Atmospheric characterization of terrestrial exoplanets in the mid-infrared: biosignatures, habitability & diversity Quanz, S. P., Ottiger, M., Fontanet, E., et al. 2021, arXiv e-prints, arXiv:2101.07500 Rothman, L., Gordon, I., Barber, R., et al. 2010, Journal of Quantitative Spectroscopy and Radiative Transfer, 111, 2139, xVIth Symposium on High Resolution Molecular Spectroscopy (HighRus-2009) Rothman, L. S., Gordon, I. E., Babikov, Y., et al. 2013, J. Quant. Spectr. Rad. Transf., 130, 4 Rugheimer, S. & Kaltenegger, L. 2018, The Astrophysical Journal, 854, 19 Rugheimer, S., Kaltenegger, L., Zsom, A., Segura, A., & Sasselov, D. 2013, Astrobiology, 13, 251 Sharp, C. M. & Burrows, A. 2007, ApJS, 168, 140 Shields-Zhou, G. & Och, L. 2011, GSA Today, 21, 4 Skilling, J. 2006, Bayesian Anal., 1, 833 Smirnov, N. V. 1939, Bull. Math. Univ. Moscou, 2, 3 Sneep, M. & Ubachs, W. 2005, J. Quant. Spectr. Rad. Transf., 92, 293 Thalman, R., Zarzana, K. J., Tolbert, M. A., & Volkamer, R. 2014, J. Quant. Spectr. Rad. Transf., 147, 171 Thalman, R., Zarzana, K. J., Tolbert, M. A., & Volkamer, R. 2017, J. Quant. Spectr. Rad. Transf., 189, 281 Waldmann, I. P. 2016, ApJ, 820, 107 Walker, S. I., Bains, W., Cronin, L., et al. 2018, Astrobiology, 18, 779–824 Walker, S. I., Bains, W., Cronin, L., et al. 2018, Astrobiology, 18, 779 Wolfe, J. M. & Fournier, G. P. 2018, Nat. Ecol. Evol., 2, 897 Yurchenko, S. N. & Tennyson, J. 2014, MNRAS, 440, 1649 [7] http://www.astropy.org [8] https://matplotlib.org/3.1.1/index.html [9] https://seaborn.pydata.org This paper is available on under CC 4.0 license. arxiv

What Are the Next Steps in Exoplanet Research?

Discussing the Impact of Our Exoplanet Study: How This Impacts Everything

Technology is relentless in forcing industries to evolve. Regarding big data, machine learning, and artificial intelligence, processing and analyzing massive volumes of real-time data has become a critical competitive advantage for all companies. However, building and maintaining reliable data platforms capable of handling such scale presents formidable challenges. Now more than ever, software engineers must be creative thinkers and innovators. Rahul Chaturvedi, a Staff Software Engineer at Uber Technologies Inc., is an excellent example of the problem-solver these challenges require. His innovative approaches to data infrastructure are meeting every future-forward standard in the tech industry today. With nearly two decades of experience in software engineering, Chaturvedi has established himself as an authority in big data, streaming technologies, and cloud infrastructure. His expertise, honed through roles at industry giants and bolstered by his academic achievements at IIT Kharagpur and the University of Washington, has been instrumental in tackling some of the most complex data challenges modern tech companies face. At Uber, Chaturvedi has been at the forefront of optimizing one of the world's largest Kafka deployments, a system that processes trillions of messages and petabytes of data daily. His work addresses the immediate needs of real-time data processing and lays the groundwork for robust AI ecosystems that can scale efficiently in cloud environments. The Scale of the Challenge: Uber's Data Ecosystem To truly appreciate the significance of Rahul Chaturvedi's contributions, one must first grasp the sheer scale of Uber's data ecosystem. As one of the world's leading transportation platforms, Uber's operations generate an astronomical volume of data: trillions of messages daily, amounting to petabytes of information that must be processed, analyzed, and acted upon in real-time. This data deluge stems from various sources: ride requests, driver locations, traffic conditions, payment transactions, and countless other data points from Uber's global network of services. Each data stream is critical for the company's operations, feeding into systems that power everything from dynamic pricing algorithms to route optimization and fraud detection. The complexity of Uber's data landscape is further compounded by several factors, including real-time processing requirements, the global distribution of data, seamless scalability without compromising performance or reliability, and cost efficiency. Chaturvedi's challenge was maintaining this system and enhancing its performance while preparing it for significant cloud migration. This required innovative approaches to long-standing problems in distributed systems and data engineering - solutions that would need to work at an unprecedented scale. Innovating on Solutions: Kafka Optimization Strategies Much of Rahul Chaturvedi's work at Uber was defined by his innovative approach to optimizing Kafka, a distributed streaming platform that forms the backbone of Uber's real-time data processing infrastructure. One of Chaturvedi's most significant contributions was leading the effort to co-locate Kafka with other technologies on the same host. Due to the critical nature of the technology, this move had never been attempted before at Uber. This strategy was crucial for enabling Uber's cloud migration while optimizing costs. Co-location introduced new challenges, particularly the "noisy neighbor" problem, where one service could potentially impact the performance of others on the same host. Other challenges included resource constraints and kernel version challenges, but Chaturvedi worked with various teams to solve these problems. The results of these optimization efforts were significant. Not only did they pave the way for a smooth cloud migration, but they also led to substantial improvements in system efficiency and reliability. The co-location strategy alone is estimated to save Uber millions of dollars annually in infrastructure costs. Future-Forward Outcomes: Building Robust AI Ecosystems Rahul Chaturvedi's work at Uber extends beyond optimizing Kafka and managing cloud migration. His efforts have been instrumental in building a robust ecosystem that supports Uber's extensive AI and machine learning initiatives. The optimized Kafka infrastructure allows for the ingestion and distribution of massive data streams necessary for training and feeding live data to AI models, and it can handle the varying and often unpredictable data volume requirements of different ML models and training processes. Besides Kafka, Chaturvedi's work involved integrating Redis into Uber's data infrastructure. This allows ML/AI teams to cache frequently accessed data or pre-processed features, significantly improving AI models' efficiency during training and inference. By strategically using Redis alongside Kafka, Chaturvedi's team created a system that provides ultra-low latency access to critical data essential for real-time AI applications. The smooth combination of Redis with an efficiently co-located Kafka infrastructure is just the kind of future-focused innovation that engineers should aspire to. Chaturvedi's work lays the foundation for future advancements in Uber's AI capabilities thanks to a data platform designed with the flexibility to incorporate new AI technologies and methodologies as they emerge. Chaturvedi has played a pivotal role in creating an ecosystem where AI and ML can flourish at Uber's massive scale through his work on Kafka, Redis, and the overall data infrastructure. Industry Impact: The Engineering of the Future Rahul Chaturvedi's contributions to Uber's data infrastructure have had far-reaching implications for the company and the broader tech industry. The co-location of Kafka with other technologies is projected to save Uber millions of dollars annually in infrastructure costs. Latency reduction in critical Kafka clusters has improved overall system responsiveness, directly impacting user experience. The optimized infrastructure can now handle Uber's massive daily data volumes - trillions of messages - more efficiently. The significance of Chaturvedi's work has not gone unnoticed. He has received three coveted "Search Storage Data (SSD) Impact Awards" in 2023 at Uber Technologies Inc., and his innovative approaches have garnered attention in the tech community, positioning him as a thought leader in big data and infrastructure. Rahul Chaturvedi's work at Uber is an excellent example of the critical role that innovative data engineering plays in modern technology companies. His strategies for building reliable, scalable, and efficient data platforms have solved immediate challenges and positioned Uber at the forefront of big data and AI technologies. Solutions like these, whether conducted on a massive scale like Uber or smaller scales by ambitious startups and small businesses, will inspire new technology and new strategies in future decades. Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Technology is relentless in forcing industries to evolve. Regarding big data, machine learning, and artificial intelligence, processing and analyzing massive volumes of real-time data has become a critical competitive advantage for all companies. However, building and maintaining reliable data platforms capable of handling such scale presents formidable challenges. Now more than ever, software engineers must be creative thinkers and innovators. Rahul Chaturvedi , a Staff Software Engineer at Uber Technologies Inc., is an excellent example of the problem-solver these challenges require. His innovative approaches to data infrastructure are meeting every future-forward standard in the tech industry today. Rahul Chaturvedi Rahul Chaturvedi With nearly two decades of experience in software engineering, Chaturvedi has established himself as an authority in big data, streaming technologies, and cloud infrastructure. His expertise, honed through roles at industry giants and bolstered by his academic achievements at IIT Kharagpur and the University of Washington, has been instrumental in tackling some of the most complex data challenges modern tech companies face. At Uber, Chaturvedi has been at the forefront of optimizing one of the world's largest Kafka deployments, a system that processes trillions of messages and petabytes of data daily. His work addresses the immediate needs of real-time data processing and lays the groundwork for robust AI ecosystems that can scale efficiently in cloud environments. The Scale of the Challenge: Uber's Data Ecosystem To truly appreciate the significance of Rahul Chaturvedi's contributions, one must first grasp the sheer scale of Uber's data ecosystem. As one of the world's leading transportation platforms, Uber's operations generate an astronomical volume of data: trillions of messages daily, amounting to petabytes of information that must be processed, analyzed, and acted upon in real-time. This data deluge stems from various sources: ride requests, driver locations, traffic conditions, payment transactions, and countless other data points from Uber's global network of services. Each data stream is critical for the company's operations, feeding into systems that power everything from dynamic pricing algorithms to route optimization and fraud detection. The complexity of Uber's data landscape is further compounded by several factors, including real-time processing requirements, the global distribution of data, seamless scalability without compromising performance or reliability, and cost efficiency. Chaturvedi's challenge was maintaining this system and enhancing its performance while preparing it for significant cloud migration. This required innovative approaches to long-standing problems in distributed systems and data engineering - solutions that would need to work at an unprecedented scale. Innovating on Solutions: Kafka Optimization Strategies Much of Rahul Chaturvedi's work at Uber was defined by his innovative approach to optimizing Kafka, a distributed streaming platform that forms the backbone of Uber's real-time data processing infrastructure. One of Chaturvedi's most significant contributions was leading the effort to co-locate Kafka with other technologies on the same host. Due to the critical nature of the technology, this move had never been attempted before at Uber. This strategy was crucial for enabling Uber's cloud migration while optimizing costs. Co-location introduced new challenges, particularly the "noisy neighbor" problem, where one service could potentially impact the performance of others on the same host. Other challenges included resource constraints and kernel version challenges, but Chaturvedi worked with various teams to solve these problems. The results of these optimization efforts were significant. Not only did they pave the way for a smooth cloud migration, but they also led to substantial improvements in system efficiency and reliability. The co-location strategy alone is estimated to save Uber millions of dollars annually in infrastructure costs. Future-Forward Outcomes: Building Robust AI Ecosystems Rahul Chaturvedi's work at Uber extends beyond optimizing Kafka and managing cloud migration. His efforts have been instrumental in building a robust ecosystem that supports Uber's extensive AI and machine learning initiatives. The optimized Kafka infrastructure allows for the ingestion and distribution of massive data streams necessary for training and feeding live data to AI models, and it can handle the varying and often unpredictable data volume requirements of different ML models and training processes. Besides Kafka, Chaturvedi's work involved integrating Redis into Uber's data infrastructure. This allows ML/AI teams to cache frequently accessed data or pre-processed features, significantly improving AI models' efficiency during training and inference. By strategically using Redis alongside Kafka, Chaturvedi's team created a system that provides ultra-low latency access to critical data essential for real-time AI applications. The smooth combination of Redis with an efficiently co-located Kafka infrastructure is just the kind of future-focused innovation that engineers should aspire to. Chaturvedi's work lays the foundation for future advancements in Uber's AI capabilities thanks to a data platform designed with the flexibility to incorporate new AI technologies and methodologies as they emerge. Chaturvedi has played a pivotal role in creating an ecosystem where AI and ML can flourish at Uber's massive scale through his work on Kafka, Redis, and the overall data infrastructure. Industry Impact: The Engineering of the Future Rahul Chaturvedi's contributions to Uber's data infrastructure have had far-reaching implications for the company and the broader tech industry. The co-location of Kafka with other technologies is projected to save Uber millions of dollars annually in infrastructure costs. Latency reduction in critical Kafka clusters has improved overall system responsiveness, directly impacting user experience. The optimized infrastructure can now handle Uber's massive daily data volumes - trillions of messages - more efficiently. Rahul Chaturvedi's contributions Rahul Chaturvedi's contributions The significance of Chaturvedi's work has not gone unnoticed. He has received three coveted "Search Storage Data (SSD) Impact Awards" in 2023 at Uber Technologies Inc., and his innovative approaches have garnered attention in the tech community, positioning him as a thought leader in big data and infrastructure. Chaturvedi's work Chaturvedi's work Rahul Chaturvedi's work at Uber is an excellent example of the critical role that innovative data engineering plays in modern technology companies. His strategies for building reliable, scalable, and efficient data platforms have solved immediate challenges and positioned Uber at the forefront of big data and AI technologies. Solutions like these, whether conducted on a massive scale like Uber or smaller scales by ambitious startups and small businesses, will inspire new technology and new strategies in future decades. Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Vested Interest Disclosure : This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Vested Interest Disclosure

Mastering Real-Time Data: Rahul Chaturvedi's Strategies for Building Reliable Data Platforms

রিয়েল-টাইম ডেটা আয়ত্ত করা: নির্ভরযোগ্য ডেটা প্ল্যাটফর্ম তৈরির জন্য রাহুল চতুর্বেদীর কৌশলগুলি

Echtzeitdaten beherrschen: Rahul Chaturvedis Strategien zum Aufbau zuverlässiger Datenplattformen

Dominar los datos en tiempo real: estrategias de Rahul Chaturvedi para crear plataformas de datos confiables

Maîtriser les données en temps réel : les stratégies de Rahul Chaturvedi pour créer des plates-formes de données fiables

रियल-टाइम डेटा में महारत हासिल करना: विश्वसनीय डेटा प्लेटफ़ॉर्म बनाने के लिए राहुल चतुर्वेदी की रणनीतियाँ

リアルタイム データの習得: 信頼性の高いデータ プラットフォームを構築するための Rahul Chaturvedi の戦略

실시간 데이터 마스터하기: 안정적인 데이터 플랫폼 구축을 위한 Rahul Chaturvedi의 전략

Dominando dados em tempo real: estratégias de Rahul Chaturvedi para construir plataformas de dados confiáveis

Освоение данных в реальном времени: стратегии Рахула Чатурведи по созданию надежных платформ данных

Gerçek Zamanlı Verilerde Uzmanlaşmak: Rahul Chaturvedi'nin Güvenilir Veri Platformları Oluşturma Stratejileri

Làm chủ dữ liệu thời gian thực: Chiến lược xây dựng nền tảng dữ liệu đáng tin cậy của Rahul Chaturvedi

掌握实时数据：Rahul Chaturvedi 构建可靠数据平台的策略

Reliable Testing with Kafka: Isolation Techniques

Hi There! So, I believe each of us has encountered a situation when, after modifying an entity in the database, it's necessary to notify other systems about this change. For example, upon accepting an order, we need to inform all related services and teams about the arrival of a new order. The first solution that comes to mind is straightforward: have each service or database polled at regular intervals to check for new orders or status updates. But what if the logic becomes too complex and needs to be segmented? This is where we turn to the event-driven approach, where we broadcast what happened with our objects. is an architectural pattern that allows systems to asynchronously exchange messages or events. The main advantage of EDA is its ability to increase system scalability and flexibility by minimizing direct dependencies between its components. Event-driven architecture (EDA) This facilitates easier adaptation to changes and growth, ensures higher resilience, and simplifies the integration of diverse services. However, a challenge arises in synchronizing data between the database and the messaging system. If we first record data about a new order in the database and then attempt to send a message to Kafka, we cannot guarantee that the message will be sent. A potential producer error could lead to the loss of critical notifications. actually Attempting to wrap these operations in a single transaction (writing to the database and sending to Kafka, followed by a transaction commit) faces fundamental difficulties, as Kafka operates outside the transaction context of databases. This necessitates an alternative solution that ensures atomicity and delivery guarantees for messages without compromising the system's performance and reliability. Here is where the comes to the rescue. Outbox pattern https://microservices.io/patterns/data/transactional-outbox.html?embedable=true The Outbox pattern involves creating a special table (outbox) in the database, where an event intended for the messaging system is recorded within the same transaction as the main data changes. Thus, both operations - changing the state in the main table and creating an event in the outbox table - are guaranteed to be executed together. Subsequently, a dedicated connector (e.g., Kafka Connect) asynchronously reads data from the outbox table and sends it to Kafka, thereby ensuring the delivery of events to the end recipients. This approach achieves high reliability and data consistency without adding undue complexity to the system's architecture or compromising its performance. Let’s review it. The Problem ACID is an acronym describing key transaction properties in databases: A transaction is either executed in full or not executed at all. Atomicity: A transaction transitions the database from one consistent state to another. Consistency: The execution of one transaction doesn't impact the execution of others. Isolation: Completed transactions persist in the database, even in case of system failures. Durability: In distributed systems, the requirement to synchronize data across multiple services is a common challenge. The conventional Two-Phase Commit (2PC) approach is often deemed suboptimal due to its sluggish and blocking nature. Moreover, dealing with data integrity recovery in the event of a system failure poses significant difficulties. The Outbox Pattern emerges as a remedy to address the need for atomicity in writing operations to the database and dispatching messages to a message queue. The core of this solution involves incorporating an additional table within the service's database, referred to as the "Outbox." This table acts as a buffer, storing messages destined for dispatch to a message queue. How the Outbox Pattern Works: Local Transaction: When performing a business operation, data changes are written to the respective database tables. A message intended for other services is placed in the Outbox table within the same transaction. Message Publication: Pollers periodically scan the Outbox table for new messages. Each message from the Outbox table is published to the corresponding message queue. After successful publication, the message is either deleted from the Outbox table or marked as sent. Let’s Code Let's break down this process step by step with a simple example, utilizing Go. You can find the source code and additional materials at the following . link Step 1: Setting Up the Order Service We'll start by creating a service for handling orders. This service will be responsible for receiving orders and recording them in the database. A key component will be implementing the "Outbox" pattern to ensure that every change in order status is reliably transmitted to Kafka. For our case, we need to create only one method: curl -X POST
  http://localhost:8080/api/v1/create-order
  -H 'Content-Type: application/json'
  -d '{
    "name": "John",
    "surname": "Doe",
    "details": {
      "code": 123,
      "operation_code": 456,
      "transaction_code": "sdfsdfsdfs"
    }
  }' The core concept of this service involves writing to both the orders table and the outbox table within a single transaction. For instance, in Go, this would be implemented as follows: func (r *OrderRepository) CreateOrder(userUUID string, order domain.Order) (*int32, error) {
	det := order.Details
	var id int32

	tx, err := r.db.Begin()
	if err != nil {
		return nil, err
	}

	err = tx.QueryRow(
		query,
		order.Name, order.Surname, det.Code, det.OperationCode, det.TransactionCode, StatusCreated).Scan(&id)

	if err != nil {
		tx.Rollback()
		return nil, err
	}

	// Save to outbox
	err = r.createOutbox(userUUID, order)
	if err != nil {
		tx.Rollback()
		return nil, err
	}

	return &id, tx.Commit()
} As a result, we’ll expect messages like this: Step 2: Configuring Kafka JDBC Connector Next, we'll set up the Kafka JDBC Connector to automatically transfer messages from the outbox table to Kafka. This allows other services to subscribe to these events and react to changes in order statuses in real-time. For the Kafka connector, I'll be using the JDBC connector. Find a good way for you to install it: https://docs.confluent.io/kafka-connectors/jdbc/current/source-connector/overview.html?embedable=true#install-the-jdbc-sink-connector After installation, just call connector API: curl http://localhost:8083/connector-plugins We’ll expect to see: [{"class":"io.confluent.connect.jdbc.JdbcSinkConnector","type":"sink","version":"10.7.5"},
{"class":"io.confluent.connect.jdbc.JdbcSourceConnector","type":"source","version":"10.7.5"},
{"class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector","type":"source","version":"7.6.0-ccs"},
{"class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector","type":"source","version":"7.6.0-ccs"},
{"class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","type":"source","version":"7.6.0-ccs"}] Next, we need to create settings for the source. Be careful at this stage, as we are providing credentials for our connector. I recommend creating separate credentials with read-only permissions for the outbox table so that this service can only observe changes. Overall, there are plenty of best practices in this regard, but we won't delve into them here. {
  "name": "jdbc-source-connector",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:postgresql://postgres:5432/postgres",
    "connection.user": "postgres",
    "connection.password": "postgres",
    "topic.prefix": "microshop.order-",
    "mode": "incrementing",
    "incrementing.column.name": "id",
    "table.whitelist": "outbox",
    "poll.interval.ms": "5000",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "key.converter.schemas.enable": "false",
    "transforms": "createKey",
    "transforms.createKey.type": "org.apache.kafka.connect.transforms.ValueToKey",
    "transforms.createKey.fields": "user_id"
  }
} Set - our table. "table.whitelist": "outbox" Set - our topic. "topic.prefix": "microshop.order-" Set - our message key. "transforms.createKey.fields": "user_id" More about settings and fields: … here And send it. curl -X POST -H "Content-Type: application/json" --data @jdbc-source-config.json http://localhost:8083/connectors And that's it! Now, you just need to create an order, and check if everything is working and if messages are being recorded in the tables. To view what we have in Kafka, I use Kafka UI for such tasks, which allows us to quickly see what we wrote there and if it's working. Step 3: Creating the Background Processing Service We'll create a background service that listens for messages from Kafka and executes the appropriate processing logic. For instance, this service might be responsible for updating order statuses or initiating delivery processes. Full schema: The last service is a background (order-background-service). In our case, it’s just a simple service that can consume messages from Kafka. ...
	err = c.SubscribeTopics([]string{"microshop.order-outbox"}, nil)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Failed to subscribe to topics: %s\n", err)
		os.Exit(1)
	}

...

	ev := c.Poll(cfg.Poll)
	if ev == nil {
	  continue
	}

	switch e := ev.(type) {
		case *kafka.Message:

		// Create an instance of Outbox to store the unmarshaled data
		var outbox domain.Outbox

		// Unmarshal the JSON message into the Outbox struct
		err := json.Unmarshal(e.Value, &outbox)
		if err != nil {
			fmt.Println("Error decoding message:", err)
			return
		}
		fmt.Println("Payload:", outbox.Payload)

		_, err = c.CommitMessage(e)
		if err != nil {	
			fmt.Printf("Failed to commit message: %v\n", err)
		} Be Careful! It is essential to pause here and reflect on the following: as we can observe, entries are now being recorded in tables atomically, ensuring ACID compliance. However, our pipeline has been expanded to include both a Kafka connector and Kafka itself. It is crucial to remember the delivery guarantees that Kafka promises. Apache Kafka offers developers various levels of message delivery guarantees, enabling a balance between performance and reliability according to application requirements. These guarantees are based on the concepts of atomicity, durability, and isolation. Here are the primary delivery guarantees provided by Kafka: : This foundational guarantee ensures a message is delivered at least once. It implies that, in the event of failures or retries, a message may be delivered or processed more than once. This guarantee is facilitated by retry mechanisms in case of errors and acknowledgments from consumers. At Least Once : In this mode, messages can be lost but will not be delivered more than once. This is the fastest and least reliable mode, as it does not account for retries in case of errors. This approach may be acceptable for scenarios where the loss of some messages is not critical. At Most Once : This is the most complex and desirable guarantee, ensuring that each message is delivered and processed “exactly once”, eliminating duplication or loss. Kafka has supported "exactly once" delivery since version 0.11, using a combination of techniques such as idempotent producers and transactions to ensure atomicity across various topics and consumers. Exactly Once The choice among these guarantees depends on your application's requirements for reliability and performance. If you need more information about it just let me know. In conclusion, implementing an Outbox pattern alongside a reliable Kafka connector like the JDBC connector offers a robust solution for orchestrating data changes across microservices. This approach ensures atomicity and consistency when updating multiple systems, significantly reducing the chances of data inconsistencies. This method serves as a crucial first step toward an Event-Driven Architecture (EDA). By introducing a reliable messaging system, like Kafka, and seamlessly integrating it with a transactional database, we establish a foundation for distributing events across the microservices landscape. This approach becomes particularly valuable when considering the need for analytics, as all relevant events are readily available for consumption. Analytics teams can effortlessly retrieve and process these events, contributing to the generation of valuable insights and reports. In essence, the Outbox pattern, coupled with Kafka integration, not only addresses immediate concerns of data consistency but also lays the groundwork for a scalable, event-driven ecosystem that is essential for modern microservices architectures. This facilitates the seamless flow of information between services, unlocking the potential for enhanced analytics and real-time decision-making. If you're interested in learning more about advanced topics such as managing deletions from the Outbox table and further insights into working with this setup, consider subscribing for future updates. Your subscription will help gauge the interest in this topic and guide the direction of upcoming articles. Stay tuned for more in-depth explorations into the intricacies of microservices architectures and event-driven systems! Useful Links Kafka UI: click here Kafka JDBC connector: click here Outbox pattern: click here

Unlocking Microservices Reliability With ACID and the Outbox Pattern

Reliable Messaging in Distributed Systems

Authors: (1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China; (2) Peng Liang, School of Computer Science, Wuhan University, Wuhan, China; (3) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China; (4) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany; (4) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia; (4) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland. V. THREATS TO VALIDITY refers to the extent to which a research tool or method can accurately assess the variable or concept being measured, and whether the results obtained are consistent with the RQs, which is a critical aspect of any research study that involves the measurement of variables or concept. As the processes of data labelling, data extraction, and data analysis in this study were conducted manually, there is a risk of introducing personal bias. Therefore, we implemented various strategies to enhance construct validity. Before each formal step of the research process, we conducted pilot experiments to test the validity of the methods and the standards between different researchers. The first author and the third author would engage in discussions after each step of the pilot experiment and formal research procedures to evaluate the results and ensure consistency with the RQs. If any disagreements arose, the second author was consulted to facilitate reaching a consensus and to further improve the validity of the study. Construct validity refers to the generalization of the findings in this study. For our research, the primary threat to external validity is the selection of data sources. To maximize external validity, we chose GitHub Issues, GitHub Discussions, and SO posts as data sources. GitHub Issues is a tool used to report and track software issues, allowing users to report errors, request features, and raise questions to developers. While GitHub Discussion is a new feature on GitHub that aims to provide a more open and organized platform for users to communicate and share insights with other community members. As a popular Q&A community, Stack Overflow is also a platform for many developers to engage in discussions and share insights regarding Copilot usage. These platforms contain a diverse and substantial amount of relevant data, and their data are complementary to each other. Consequently, we were able to collect diverse usage-related data of Copilot from a large number of developers and projects from these three data sources. However, despite all these efforts, we admitted that there may still be relevant data that we missed. External validity refers to the degree to which a research method can consistently produce reliable and reproducible results. To minimize potential uncertainties arising from the research methodology, we have implemented multiple measures to maximize the reliability of our study. We conducted a pilot labelling to assess the consistency of the two authors prior to the formal data labelling process. The Cohen’s Kappa coefficients of the three pilot labelling processes are 0.824, 0.834, and 0.806, indicating good agreement between the authors. Throughout the data labelling, extraction, and analysis process, we thoroughly discussed and resolved any inconsistencies within the team to ensure the consistency and accuracy of the result. Additionally, we have made available the dataset of the study [13] to enable other researchers to validate our findings. Reliability This paper is under CC 4.0 license. available on arxiv

Reliability in Copilot Research: Measures for Consistent and Reproducible Results

Welcome back to this series where we have been learning how to build web applications with . AI So far in this series, we’ve created a working app that uses AI to determine who would win in a fight between two user-provided opponents and generates text responses and images. It’s working, but we’ve been following the happy path. Intro & Setup Your First AI Prompt Streaming Responses How Does AI Work Prompt Engineering AI-Generated Images Security & Reliability Deploying In this post, we’re going to talk about what happens when things don’t follow the happy path by accounting for error handling and security concerns. https://www.youtube.com/watch?v=DpOTSKtpEHI&embedable=true Dealing With Bad HTTP Requests The first issue to deal with is around our requests. So far, we’ve just been assuming that the requests from the client to the server will just work. HTTP const response = await jsFormSubmit(form)

// Do something with response This is a mistake. We need to account for situations where the server experiences an error or returns a bad status code. In a real-world application, we would want a sophisticated notification service to communicate to users in the event of different errors (server error, validation error, authorization error, not found error, etc.). It would also be good to tie in an error and bug-tracking software so you get notified of any issues. For today’s example, the discount brand will have to do. We’ll check the response property, and in case of a bad response, we’ll just the user that there was an error. ok alert const response = await jsFormSubmit(form)

if (!response.ok) {
  state.isLoading = false
  alert("The request experienced an issue.")
  return
} The code above only accounts for the HTTP request between the client and the server. Don’t forget, we have another request between the server and OpenAI. Consider the scenario where OpenAI returns a bad status code. How should we communicate that to the end user on the client? This is also tricky and unique to each app. For the sake of convenience, we can do a similar check on the property. response.ok In the event of a bad request, you’ll once again want to report on the error, and maybe respond to the user with the same status code. I would recommend against passing the response message to the client in case it contains sensitive data. const response = await fetch('https://api.openai.com/v1/chat/completions', {
  // ... fetch options
})
if (!response.ok) {
  reportError(response)
  throw error(response.status, 'ERROR: Service unavailable');
} This error handling is very rudimentary, and I’ll leave it like that because unfortunately, I’ve never seen two apps that handle errors the same way. It’s highly subjective. Suffice it to say that you should spend time thinking about how app should behave in the event of an error. How do you report it internally, and how do you communicate it to users? your And what happens when users deliberately try to break something…? Dealing With Bad User Input In addition to following the happy path where we assumed every HTTP request would always work, we assumed every user was benevolent. This is another mistake. Sometimes, users are malicious. Oftentimes, they are just plain silly. We should account for both. Any time you receive user-submitted data, you have to validate it. In our app, we expect the user to submit two opponents. What happens if they submit just one, or none, or empty strings? We probably should catch that early before sending the malstructured prompt to OpenAI. We can add the HTML attribute to the textareas to tell the form that both inputs need to be filled before the form can be submitted. If the user tries to submit the form without the controls filled in, the browser will prevent the submission, focus on the first invalid input, and provide a little error message telling the user what the problem is. required This is good for the user experience because it provides some early feedback, but client-side validation is easily bypassed, so we have to also validate data on the server. Fortunately, there are some very good validation libraries available that can help with this. My favorite is called . We can install it with . Zod npm install zod Zod allows us to define a schema that will be used to validate input data. If the input doesn’t match the schema, Zod can either throw an error or report it. In our app, we are receiving the user input through the method, which returns the submitted form data as an object containing and properties. So, what we need to do is create a validation schema, and then pass the form data into one of the schema validation methods. requestEvent.parseBody() opponent1 opponent2 I prefer throwing an error, and instead getting an object back with the validation information. That way, I can add the logic myself to deal with bad data. not Inside my , before doing too much work, let’s make sure we have the right data: onPost middleware import { z } from 'zod'
// ...

export const onPost: RequestHandler = async (requestEvent) => {
  const formData = await requestEvent.parseBody()

  const schema = z.object({
    opponent1: z.string().min(1),
    opponent2: z.string().min(1),
  })
  const validation = schema.safeParse(formData)

  if (!validation.success) {
    requestEvent.json(400, {
      errors: validation.error.issues
    })
    return 
  }

  // Continue with OpenAI API request and response
} In the code above, I create an Object schema that should have two properties, and . Both properties are required, must be strings, and cannot be empty. Passing the form data into the schema’s method will return an object that can tell me if the validation was successful, what the error was, if any, and the validated data. opponent1 opponent2 safeParse() In the event of invalid data, I return early from the request handler with an error response explaining the errors. 400 is the Bad Request status code. HTTP 400 One other thing I like to change is how I use the form data once it’s been validated. Zod also provides a property on the returned object from . data safeParse const prompt = await promptTemplate.format({
  opponent1: validation.data.opponent1,
  opponent2: validation.data.opponent2
}) In our example, it doesn’t make too much of a difference whether we use this or the form data directly, but it’s nice to get in the habit of using the data property because Zod will coerce the data to the appropriate format. Form data and query parameters are almost always received as strings, but if your Zod schema was expecting a number, it will try to coerce it for you, turning something like the string into the number . "420" 420 So that covers users sending missing data or not enough data, but what about sending too much? By giving users unbounded input length that gets injected directly into our prompt, we are opening the gates for users to create massive prompts that would require a lot of tokens and cost us money. Why don’t we add a maximum length to the inputs to something more appropriate for this app? We can add a maximum length to both the server-side validation schema and the client-side validation attributes. // Reusable constant
const MAX_INPUT_LENGTH = 50

// In our schema
const schema = z.object({
  opponent1: z.string().min(1).max(MAX_INPUT_LENGTH),
  opponent2: z.string().min(1).max(MAX_INPUT_LENGTH),
})

// In our template
<Input
  maxLength={MAX_INPUT_LENGTH}
/> By reducing the amount of data a user can provide, we are reducing the amount of tokens that our API request could potentially use. This step also limits the amount of flexibility a user has to manipulate our prompt. Consider the fact that a user could provide an “opponent” that actually contains malicious instructions for the app. This segues nicely to a very important security concern for AI applications specifically. Dealing With Injection Attacks We’re doing some basic validation that the data we get from the user is the right type, but we aren’t checking the content that they send us. We’re just blindly sticking it into our prompt and sending it off, and this opens us up to a very interesting kind of attack called a prompt injection attack. If you’ve done any work building applications with SQL, this may sound similar to an , and that’s because it is. An SQL injection attack is when a user submits some data of the right type, but containing SQL commands that could be harmful if run. SQL injection attack Here’s an example. Let’s say our app had some SQL logic to select a user by ID based on the input provided: const query = "SELECT * FROM Users WHERE UserId = " + inputId; An attacker could provide the string as the input and would return the information of all the users. This is bad, but you can avoid it by using parameterized queries, stored procedures, or escaping user input. Unless you’re writing raw SQL queries, most tools protect against injection. If you’re interested, here’s . '1 OR 1=1' more on prevention from OWASP Prompt injection is a bit different because prompts don’t have a structured language with specific keywords you can search for. Literally anything you (or the user) provide is a valid prompt, and there’s no easy delineation between what you write and what a user writes. To their credit, OpenAI does include tactics for prompt engineering that . This way, you can help the AI identify the different identities, like the system and the user. encourage using delimiters to indicate separate parts of the input It might look like this: Translate the text after the delimiting characters "~~~~~":

~~~~~

[text to be translated] This is a big improvement as it provides a clearer separation between the system and the user input. Still, it’s not without gaps. has pointed out several examples of prompt injection attacks and why it may never be a solved problem: Simon Willison Prompt injection attacks against GPT-3 I don’t know how to solve prompt injection It’s kind of scary and should make you think twice about building AI-powered apps. But this is a bit of a . Even with its inherent vulnerabilities, AI is here to stay. My suggestion is to keep yourself up to date on attack vectors and incorporate several layers of security. Pandora’s box Conclusion Building applications with the happy path in mind is great, but we must also address the not-so-happy path. It’s important to familiarize ourselves with points of failure and vulnerabilities and address them appropriately. This makes our applications more secure for our users and more reliable. In this post, we discussed: What happens when our app experiences HTTP errors? How do we validate user input? What are some security concerns specifically for AI apps? This is not a comprehensive list, but I hope it serves as a good starting point. With this work out of the way, I think we are ready to launch our app to the world. We’ll do that in the next post. Intro & Setup Your First AI Prompt Streaming Responses How Does AI Work Prompt Engineering AI-Generated Images Security & Reliability Deploying Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to , , and . share it sign up for my newsletter follow me on Twitter Originally published on . austingil.com

How to Address Bugs, Security, & Reliability: AI for Web Devs

These 13 Hidden Open-Source Libraries Will Help You Become an AI Wizard 🧙‍♂️🪄 

এই 13টি লুকানো ওপেন-সোর্স লাইব্রেরি আপনাকে এআই উইজার্ড হতে সাহায্য করবে 🧙‍♂️🪄 

Diese 13 versteckten Open-Source-Bibliotheken helfen Ihnen, ein KI-Zauberer zu werden 🧙‍♂️🪄

Estas 13 bibliotecas ocultas de código abierto te ayudarán a convertirte en un mago de la IA 🧙‍♂️🪄 

Ces 13 bibliothèques open source cachées vous aideront à devenir un assistant d'IA 🧙‍♂️🪄 

ये 13 छुपी हुई ओपन-सोर्स लाइब्रेरीज़ आपको AI जादूगर बनने में मदद करेंगी 🧙‍♂️🪄

これら 13 個の隠れたオープンソース ライブラリは、AI ウィザードになるために役立ちます 🧙‍♂️🪄

이 13개의 숨겨진 오픈 소스 라이브러리는 당신이 AI 마법사가 되는 데 도움이 될 것입니다 🧙‍♂️🪄 

Estas 13 bibliotecas ocultas de código aberto ajudarão você a se tornar um assistente de IA 🧙‍♂️🪄 

Эти 13 скрытых библиотек с открытым исходным кодом помогут вам стать волшебником искусственного интеллекта 🧙‍♂️🪄 

Bu 13 Gizli Açık Kaynak Kitaplığı Bir Yapay Zeka Sihirbazı Olmanıza Yardımcı Olacak 🧙‍♂️🪄 

13 thư viện mã nguồn mở ẩn này sẽ giúp bạn trở thành phù thủy AI 🧙‍♂️🪄 

这 13 个隐藏的开源库将帮助你成为 AI 奇才🧙‍♂️🪄

The Hypersonic Drone Market: How the Axis of Innovation Is Forming Around It

My Testing Journey With Jasmine and Mocha: What I've Learned

A Newbie Developer's Guide to Bringing AI Into Blockchain dApps

ব্লকচেইন dApps-এ AI আনার জন্য একজন নবাগত বিকাশকারীর গাইড

Ein Leitfaden für Entwicklerneulinge zum Integrieren von KI in Blockchain-dApps

Una guía para desarrolladores novatos para incorporar la IA a las dApps de Blockchain

Guide du développeur débutant pour intégrer l'IA dans les dApps Blockchain

ब्लॉकचेन dApps में AI लाने के लिए एक नौसिखिया डेवलपर की मार्गदर्शिका

ブロックチェーン dApps に AI を導入するための初心者開発者向けガイド

블록체인 dApp에 AI를 도입하기 위한 초보자 개발자 가이드

Guia do desenvolvedor novato para trazer IA para dApps Blockchain

Руководство для новичков по внедрению искусственного интеллекта в децентрализованные приложения на блокчейне

Yapay Zekayı Blockchain dApp'lerine Getirmeye Yönelik Yeni Başlayan Geliştirici Kılavuzu

Hướng dẫn dành cho nhà phát triển mới về việc đưa AI vào các dApps trên Blockchain

将人工智能引入区块链 dApp 的新手开发者指南

GRAND CAYMAN, Cayman Islands, July 26th, 2024/Chainwire/--Kwenta, the leading onchain perpetuals exchange on the Optimism network, has partnered with Perennial to launch a new joint product and incentive program on the Arbitrum network. This initiative, backed by a substantial 1.9 million ARB grant, aims to attract new users and liquidity providers to Arbitrum and spark renewed interest in onchain perpetuals trading. Kwenta And Perennial’s Expansion Strategy Kwenta, known for being a UX layer for perps trading on Optimism and Base, is expanding its reach to Arbitrum in a bid to build a comprehensive marketplace for onchain perpetuals. This move is facilitated by Perennial V2, an Arbitrum-native protocol designed to be a hyper-efficient liquidity layer for derivatives trading. The joint initiative is designed to revitalize the popular AMM model for leveraged trading by providing traders with a powerful, familiar interface to interact with smart contract liquidity. Perennial’s vision is to create a single, global liquidity layer where all markets, all chains, and all liquidity converge into one decentralized nexus. By integrating Kwenta’s premier trading interface with Perennial’s advanced perps infrastructure, users on Arbitrum will benefit from an optimized trading experience. Trade orders on selected Kwenta markets will be seamlessly routed through Perennial, ensuring efficiency and reliability. This collaboration combines Kwenta's top-tier trading platform with Perennial’s hyper-efficient trading infrastructure, delivering the best possible experience for traders. Incentive Program Details The newly launched incentive program focuses on two main areas: Trading Rebates: Active traders will benefit from substantial fee rebates, making it more cost-effective to trade on the platform. This initiative is expected to drive trading volume and retain users. Liquidity Boosting: Liquidity providers will receive incentives to supply liquidity, ensuring deep and efficient markets. For more details on how to potentially earn rewards, users can visit Kwenta's Rewards Blog for trading rewards details, or visit Perennial’s Discord server for information about rewards for providing liquidity. Team Insights Burt Rock, Marketing Lead at Kwenta, highlighted how far onchain perps have come, stating, “We’ve seen a lot of new perpetuals markets focused on improving the user experience, but most are making huge sacrifices in terms of decentralization. We want to show people you really can build decentralized apps which are incredible to use, and even better than offchain versions.”Kevin Britz, CEO & Co-Founder at Perennial, added, "Kwenta has been a market leader in the perp vertical, regularly pushing over $100M in OI. We're excited to welcome them to our home on Arbitrum and expand the number of teams leveraging Perennial infrastructure. In a saturated perps landscape, combining our protocol with Kwenta’s interface will lead to increased volume, deeper liquidity, and a CEX-level user experience." Future Outlook The introduction of this incentive program represents a crucial step in the evolution of Kwenta and Perennial. With Kwenta concentrating on the UX and frontend aspects of trading and Perennial focusing on providing a robust liquidity layer, the collaboration is set to offer a decentralized trading experience that is both highly efficient and user-friendly. This model of specialization showcases how projects can synergize to deliver superior services in the DeFi space. The combined expertise of Kwenta and Perennial promises to elevate the standard of onchain perpetual trading, making it more accessible and appealing to a wider audience. For more information, users can visit Kwenta’s website and Perennial’s website. Users can also read Perennial’s launch announcement for additional details. About Kwenta And Perennial Kwenta: As a leading platform for derivatives trading, Kwenta has facilitated over $50 billion in volume on Optimism and Base. The platform is expanding to Arbitrum to launch its perpetual marketplace, a single app to compare and trade on multiple onchain venues. Perennial: Perennial is a pioneering DeFi liquidity layer that offers a hyper-efficient infrastructure for a variety of trading applications. Its modular and efficient design supports customized markets, leveraging fast execution and capital efficiency to power DeFi apps. Contacts Burt Rock Kwenta burt@kwenta.io Lucas Terry Perennial lucas@perennial.finance This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here. GRAND CAYMAN, Cayman Islands, July 26th, 2024/Chainwire/--Kwenta, the leading onchain perpetuals exchange on the Optimism network, has partnered with Perennial to launch a new joint product and incentive program on the Arbitrum network. This initiative, backed by a substantial 1.9 million ARB grant, aims to attract new users and liquidity providers to Arbitrum and spark renewed interest in onchain perpetuals trading. Kwenta And Perennial’s Expansion Strategy Kwenta, known for being a UX layer for perps trading on Optimism and Base, is expanding its reach to Arbitrum in a bid to build a comprehensive marketplace for onchain perpetuals. This move is facilitated by Perennial V2, an Arbitrum-native protocol designed to be a hyper-efficient liquidity layer for derivatives trading. The joint initiative is designed to revitalize the popular AMM model for leveraged trading by providing traders with a powerful, familiar interface to interact with smart contract liquidity. Perennial’s vision is to create a single, global liquidity layer where all markets, all chains, and all liquidity converge into one decentralized nexus. By integrating Kwenta’s premier trading interface with Perennial’s advanced perps infrastructure, users on Arbitrum will benefit from an optimized trading experience. Trade orders on selected Kwenta markets will be seamlessly routed through Perennial, ensuring efficiency and reliability. This collaboration combines Kwenta's top-tier trading platform with Perennial’s hyper-efficient trading infrastructure, delivering the best possible experience for traders. Incentive Program Details The newly launched incentive program focuses on two main areas: Trading Rebates: Active traders will benefit from substantial fee rebates, making it more cost-effective to trade on the platform. This initiative is expected to drive trading volume and retain users. Trading Rebates: Active traders will benefit from substantial fee rebates, making it more cost-effective to trade on the platform. This initiative is expected to drive trading volume and retain users. Liquidity Boosting: Liquidity providers will receive incentives to supply liquidity, ensuring deep and efficient markets. Liquidity Boosting: Liquidity providers will receive incentives to supply liquidity, ensuring deep and efficient markets. For more details on how to potentially earn rewards, users can visit Kwenta's Rewards Blog for trading rewards details, or visit Perennial’s Discord server for information about rewards for providing liquidity. Kwenta's Rewards Blog Kwenta's Rewards Blog Perennial’s Discord server Perennial’s Discord server Team Insights Burt Rock, Marketing Lead at Kwenta, highlighted how far onchain perps have come, stating, “We’ve seen a lot of new perpetuals markets focused on improving the user experience, but most are making huge sacrifices in terms of decentralization. Burt Rock, Marketing Lead at Kwenta, highlighted how far onchain perps have come, stating, “We’ve seen a lot of new perpetuals markets focused on improving the user experience, but most are making huge sacrifices in terms of decentralization. We want to show people you really can build decentralized apps which are incredible to use, and even better than offchain versions.”Kevin Britz, CEO & Co-Founder at Perennial, added, "Kwenta has been a market leader in the perp vertical, regularly pushing over $100M in OI. We're excited to welcome them to our home on Arbitrum and expand the number of teams leveraging Perennial infrastructure. In a saturated perps landscape, combining our protocol with Kwenta’s interface will lead to increased volume, deeper liquidity, and a CEX-level user experience." We want to show people you really can build decentralized apps which are incredible to use, and even better than offchain versions.”Kevin Britz, CEO & Co-Founder at Perennial, added, "Kwenta has been a market leader in the perp vertical, regularly pushing over $100M in OI. We're excited to welcome them to our home on Arbitrum and expand the number of teams leveraging Perennial infrastructure. In a saturated perps landscape, combining our protocol with Kwenta’s interface will lead to increased volume, deeper liquidity, and a CEX-level user experience." Future Outlook The introduction of this incentive program represents a crucial step in the evolution of Kwenta and Perennial. With Kwenta concentrating on the UX and frontend aspects of trading and Perennial focusing on providing a robust liquidity layer, the collaboration is set to offer a decentralized trading experience that is both highly efficient and user-friendly. This model of specialization showcases how projects can synergize to deliver superior services in the DeFi space. The combined expertise of Kwenta and Perennial promises to elevate the standard of onchain perpetual trading, making it more accessible and appealing to a wider audience. For more information, users can visit Kwenta’s website and Perennial’s website . Users can also read Perennial’s launch announcement for additional details. Kwenta’s website Kwenta’s website Perennial’s website Perennial’s website Perennial’s launch announcement Perennial’s launch announcement About Kwenta And Perennial Kwenta : As a leading platform for derivatives trading, Kwenta has facilitated over $50 billion in volume on Optimism and Base. The platform is expanding to Arbitrum to launch its perpetual marketplace, a single app to compare and trade on multiple onchain venues. Kwenta Kwenta Perennial : Perennial is a pioneering DeFi liquidity layer that offers a hyper-efficient infrastructure for a variety of trading applications. Its modular and efficient design supports customized markets, leveraging fast execution and capital efficiency to power DeFi apps. Perennial Perennial Contacts Burt Rock Kwenta burt@kwenta.io Lucas Terry Perennial lucas@perennial.finance This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here. This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here . here here

Kwenta And Perennial: How They Are Kickstarting the Arbitrum Expansion With 1.9M ARB

Kwenta এবং বহুবর্ষজীবী: কিভাবে তারা 1.9M ARB দিয়ে আরবিট্রাম সম্প্রসারণ শুরু করছে

Kwenta und Perennial: Wie sie die Arbitrum-Expansion mit 1,9 Mio. ARB ankurbeln

Kwenta y Perennial: cómo están impulsando la expansión de Arbitrum con 1,9 millones de ARB

Kwenta et Perennial : comment ils lancent l'expansion d'Arbitrum avec 1,9 million d'ARB

क्वेंटा और पेरेनियल: कैसे वे 1.9M ARB के साथ आर्बिट्रम विस्तार को गति दे रहे हैं

Kwenta と Perennial: 190 万 ARB で Arbitrum の拡大を加速させる方法

Kwenta와 Perennial: 190만 ARB로 Arbitrum 확장을 시작하는 방법

Kwenta e Perennial: como estão iniciando a expansão da Arbitrum com 1,9 milhão de ARB

Kwenta и Perennial: как они запускают расширение Arbitrum с помощью 1,9 млн ARB

Kwenta ve Perennial: 1,9 Milyon ARB ile Arbitrum Genişlemesini Nasıl Başlatıyorlar?

Kwenta và Perennial: Họ đang khởi động việc mở rộng Arbitrum với 1,9 triệu ARB như thế nào

Kwenta 和 Perennial：如何利用 190 万 ARB 启动 Arbitrum 扩张

Stories, News & Mentions about NELIA

Nelia’s Stories on HackerNoon