The Information has a simple mission: deliver important, deeply reported articles about the technology business you won’t find elsewhere. Many of the most influential people in the industry turn to us for fresh information and original insight.

OpenAI Researchers, Including Ally of Sutskever, Fired for Alleged Leaking

Cathie Wood’s Ark Investment Management has announced that it holds a stake in Silicon Valley artificial intelligence darling <strong>OpenAI</strong>, a bet that the nascent AI industry will remake the tech landscape.

Cathie Wood’s ARK Venture Fund Announces It Holds a Stake in OpenAI - Bloomberg

The $700 Ai Pin, funded by <strong>OpenAI</strong>’s Sam Altman and Microsoft, can be helpful — until it struggles with tasks like doing math and crafting sandwich recipes.

Humane’s AI Pin Wants to Free You From Your Phone - The New York Times

18 votes, 13 comments. 2.2M subscribers in the singularity community. Everything pertaining to the technological singularity and related topics, e.g…

r/singularity on Reddit: According to OpenAI's open-sourced evals, new GPT-4-Turbo model seems slightly better than Opus

<strong>OpenAI</strong> has upgraded the GPT-4 Turbo model available to ChatGPT premium users. It claims that it&#x27;s enhanced in several ways.

OpenAI makes ChatGPT 'more direct, less verbose' | TechCrunch

Any battlefield use of the software would be a dramatic turnaround for <strong>OpenAI</strong>, which describes its mission as developing safety-focused AI.

Microsoft Pitched OpenAI’s DALL-E as Battlefield Tool for U.S. Military

Cathie Wood&#x27;s Ark Investment Management said it holds a stake in Microsoft-backed <strong>OpenAI</strong>, Bloomberg News reported on Thursday, citing the company&#x27;s mail to its clients. &quot;As of April 10, 2024, the ARK Venture Fund invests in <strong>OpenAI</strong>,&quot; Ark said, referring to its $54 million VC fund, the report said.

Cathie Wood's Ark announces OpenAI stake, Bloomberg News reports

The CEO met with officials in the UAE this week to push for more infrastructure investment to support the development of artificial intelligence.

OpenAI’s Altman Pitches Global AI Coalition on Trip to Middle East - Bloomberg

US News is a recognized leader in college, grad school, hospital, mutual fund, and car rankings. Track elected officials, research health conditions, and find news you can use in politics, business, health, and education.

Cathie Wood's Ark Announces OpenAI Stake, Bloomberg News Reports

65 votes, 28 comments. 1.3M subscribers in the <strong>OpenAI</strong> community. <strong>OpenAI</strong> is an AI research and deployment company. <strong>OpenAI</strong>&#x27;s mission is to ensure that…

r/OpenAI on Reddit: Demis Hassabis admits OpenAI is a MOAT and doesn't think Google can catch up with Sora. Still seeks control

OpenAI is an AI research and deployment company that conducts research and implements machine learning.

OpenAI

OpenAI is in a high-stakes legal battle with Elon Musk, who is suing the company to open source its groundbreaking AI models like GPT-4. While OpenAI may have the law on its side, I believe the ethical path forward is clear—it's time for the company to fully embrace the open-source ethos it was founded upon. As a former AI Specialist at OpenAI, I know OpenAI's stated mission has always been to ensure that artificial intelligence benefits all of humanity. The company's charter explicitly prioritizes this goal above generating financial returns. With the rapid rise of AI and the very real prospect of widespread job displacement, now is the time for OpenAI to double down on this mission through radical collaboration and transparency. Elon Musk helped found OpenAI precisely to keep advanced AI out of the exclusive control of a few tech giants or governments. The fear was that highly centralized, opaque AI development could lead to biased systems that disempower people and exacerbate inequality. Making frontier models like GPT-4 open source would accelerate innovation across the field and give everyone a stake in steering this transformative technology in a positive direction. Instead, OpenAI and its partner Microsoft now boast that “we are waiting for the competition to arrive. It will arrive, I’m sure, but the fact [is] that we have the … leading LLM out there.” Their most powerful models, including GPT-4, DALL-E, and now Sora, are all closed-source and gated. Entire countries like China and Russia are restricted from using OpenAI’s technologies. Accountability Through Openness Open-sourcing its models would also keep OpenAI honest and accountable. External researchers could audit the systems for hidden biases, safety issues, and potential misuse. The global AI ethics community could collaborate to make the models more inclusive and aligned with diverse human values. Perhaps even more importantly, the immense economic benefits of AI could be more evenly distributed if the core infrastructure is treated as a public good rather than the intellectual property of a few tech giants. While OpenAI has published research papers and shared some tools, it has stopped short of fully open-sourcing its most powerful models. Meanwhile, other prominent AI labs like H2O.ai have embraced a more open approach from the start. For instance, H2O recently released an open-source LLM called Danube-2, which can be downloaded in full from HuggingFace. Sri Ambati, CEO and founder of H2O.ai, said at H2O.ai’s GenAI World keynote that "AGI should be open source and in the public domain, at the service of humanity and the planet.” Even big tech players like Meta are exploring open-source AGI and giving outside researchers and the public more visibility into and agency over their AI work. OpenAI risks falling behind this trend and betraying its founding principles if it continues to treat its state-of-the-art models as proprietary secrets. The Business Case for Collaboration From a business standpoint, I understand the temptation for OpenAI to maintain exclusive control over its technology. But the history of the software industry shows that open source tends to win out in the long run through faster collaborative innovation and community support. Linux and the open-source movement conquered the once-dominant proprietary operating systems. Wikipedia displaced the traditional encyclopedia industry by crowdsourcing knowledge. With highly capable open source competitors to GPT-4 emerging, OpenAI's best path forward may be to focus on its expertise in scaling and commercializing a community-driven ecosystem rather than clinging to secrecy. Taking the High Road The transition to an open-source model would undoubtedly be challenging, but it's the right thing to do. OpenAI's leaders should return to their charter, reflect deeply on the kind of world they want to create, and take the bold path of openness and collaboration. Making advanced AI serve everyone is a historic challenge and opportunity. In the face of this technology's epochal impact on society, anything less than full transparency would be a dereliction of OpenAI's founding duty. I urge the company's leaders to resolve any legal disputes swiftly, open source their most important models, and help catalyze a global effort to develop AI as a public good. Democratizing access to AI is how we'll create the best outcomes for humanity. It's time for OpenAI to walk the walk of its open-source mission. OpenAI is in a high-stakes legal battle with Elon Musk, who is suing the company to open source its groundbreaking AI models like GPT-4. While OpenAI may have the law on its side , I believe the ethical path forward is clear—it's time for the company to fully embrace the open-source ethos it was founded upon. legal battle legal battle on its side on its side As a former AI Specialist at OpenAI, I know OpenAI's stated mission has always been to ensure that artificial intelligence benefits all of humanity. The company's charter explicitly prioritizes this goal above generating financial returns. With the rapid rise of AI and the very real prospect of widespread job displacement, now is the time for OpenAI to double down on this mission through radical collaboration and transparency. charter charter Elon Musk helped found OpenAI precisely to keep advanced AI out of the exclusive control of a few tech giants or governments. The fear was that highly centralized, opaque AI development could lead to biased systems that disempower people and exacerbate inequality. Making frontier models like GPT-4 open source would accelerate innovation across the field and give everyone a stake in steering this transformative technology in a positive direction. found OpenAI found OpenAI Instead, OpenAI and its partner Microsoft now boast that “we are waiting for the competition to arrive. It will arrive, I’m sure, but the fact [is] that we have the … leading LLM out there.” Their most powerful models, including GPT-4, DALL-E, and now Sora, are all closed-source and gated. Entire countries like China and Russia are restricted from using OpenAI’s technologies. boast boast restricted restricted Accountability Through Openness Open-sourcing its models would also keep OpenAI honest and accountable. External researchers could audit the systems for hidden biases, safety issues, and potential misuse. The global AI ethics community could collaborate to make the models more inclusive and aligned with diverse human values. Perhaps even more importantly, the immense economic benefits of AI could be more evenly distributed if the core infrastructure is treated as a public good rather than the intellectual property of a few tech giants. While OpenAI has published research papers and shared some tools, it has stopped short of fully open-sourcing its most powerful models. Meanwhile, other prominent AI labs like H2O.ai have embraced a more open approach from the start. For instance, H2O recently released an open-source LLM called Danube-2 , which can be downloaded in full from HuggingFace. Sri Ambati, CEO and founder of H2O.ai, said at H2O.ai’s GenAI World keynote that "AGI should be open source and in the public domain, at the service of humanity and the planet.” LLM called Danube-2 LLM called Danube-2 H2O.ai’s GenAI World keynote H2O.ai’s GenAI World keynote Even big tech players like Meta are exploring open-source AGI and giving outside researchers and the public more visibility into and agency over their AI work. OpenAI risks falling behind this trend and betraying its founding principles if it continues to treat its state-of-the-art models as proprietary secrets. open-source AGI open-source AGI The Business Case for Collaboration From a business standpoint, I understand the temptation for OpenAI to maintain exclusive control over its technology. But the history of the software industry shows that open source tends to win out in the long run through faster collaborative innovation and community support. win out in the long run win out in the long run Linux and the open-source movement conquered the once-dominant proprietary operating systems. Wikipedia displaced the traditional encyclopedia industry by crowdsourcing knowledge. With highly capable open source competitors to GPT-4 emerging, OpenAI's best path forward may be to focus on its expertise in scaling and commercializing a community-driven ecosystem rather than clinging to secrecy. Taking the High Road The transition to an open-source model would undoubtedly be challenging, but it's the right thing to do. OpenAI's leaders should return to their charter, reflect deeply on the kind of world they want to create, and take the bold path of openness and collaboration. Making advanced AI serve everyone is a historic challenge and opportunity. In the face of this technology's epochal impact on society, anything less than full transparency would be a dereliction of OpenAI's founding duty. I urge the company's leaders to resolve any legal disputes swiftly, open source their most important models, and help catalyze a global effort to develop AI as a public good. Democratizing access to AI is how we'll create the best outcomes for humanity. It's time for OpenAI to walk the walk of its open-source mission.

Why OpenAI Should Become Open-Source

A day dedicated to Azure Open AI, focusing on harnessing Azure AI Search for document indexing and enrichment. The Microsoft team has recently launched an Open AI challenge, in which a developer can learn how to build Azure AI solutions and apps. Introduction The article demonstrates how to use Azure AI Search to develop a solution where documents undergo indexing and enrichment through AI skills, enhancing their searchability. Why Azure AI Search? Azure AI Search leverages advanced AI and machine learning to enrich indexing and search capabilities. It can automatically understand the content within documents, images, and other media types, extracting valuable information, identifying patterns, and even understanding sentiments. These capabilities allow for more nuanced and intelligent search results. Prerequisite Experience working with Azure and Azure portals. An understanding of generative AI. Experience in one high-level programming language like C# or Python Getting Started Before starting the actual implementation, 3 different Azure resources need to be set up before moving forward. Setup Azure AI Search Setup Azure AI Service Setup Azure Storage Account Your Azure AI Search and Azure AI Services resources must be in the same location! Once the above prerequisites are completed, you need to perform the following steps in the Azure Portal. Upload documents to Azure Blob Storage Index the documents Testing Setup Azure AI Search Step 1: Navigate to the Azure Portal Search for and fill out the following details. Azure AI Search, Step 2: Scaling Configuration Since I am using the free pricing tier, I cannot add scaling capabilities or create replicas as shown below. Step 3: Create Tags For this exercise, tag names are not required. But in a production environment, it should be added as it’s a best practice. Step 4: Review & Create Post validation checks by Azure Cloud, and proceed with creating the resource. Make sure you review the details entered in the previous steps. Setup Azure AI Service Step 1: Create Azure AI Service Now, let's create another service, that searches for **Azure AI Service **and fills out the following details Now, continue with the remaining steps as default options are selected, create the service, and finally, wait for the service to be deployed. Step 2: Network Defaults Step 3: Identity Defaults Step 4: Review & Create Post validation checks by Azure Cloud, and proceed with creating the resource. Make sure you review the details entered in the previous steps. Setup Azure Storage Account Step 1: Create an Azure Storage Account Now, let’s create another service, that searches for Azure Storage Account, and fills out the following details. Step 2: Allow Anonymous Access In the Advanced tab, check the box next to Allow enabling anonymous access on individual containers. Step 3: Review & Create Now, continue with the remaining steps as default options are selected, create the service, and finally, wait for the service to be deployed. Upload Documents to Azure Blob Storage With the necessary resources in place, proceed to upload documents to your Azure Storage account. Follow the article provided by Microsoft. Quickstart: Upload, download, and list blobs - Azure portal - Azure Storage Index the Documents Open Azure AI Service, and under the Overview tab, select “Import Data” as highlighted below. From the dropdown, select “Azure Blob Storage” as shown below. The tab should follow the following rules. Connect Your Data The tab should follow the following rules. Add Cognitive Skills The tab should follow the following rules. Add Customized Target Indexer Click on Submit to initiate the creation of the data source, skillset, index, and indexer. Test the Indexer Toward the top of your Azure AI Search resource’s Overview page, opt for Search Explorer. Within Search Explorer, input * (a single asterisk) into the Query string box, then proceed to click on the Search button. Test Case 1: Fetch All This query retrieves all documents in the index in JSON format. Test Case 2: Include Count Above the search results, you’ll find a count indicating the number of documents returned by the search. Test Case 3: Search Specific Keywords This search locates documents containing references to “New York” within any searchable fields and provides the document’s filename along with its key phrases. Conclusion The step-by-step guidance on setting up necessary Azure resources and detailed instructions for uploading documents, indexing them, and testing the setup is crucial for anyone looking to implement a robust document management solution. C# Programming🚀 Thank you for being a part of the C# community! Before you leave: If you’ve made it this far, please show your appreciation with a clap, and follow the author! 👏️️ Follow us: | | | | | X LinkedIn Dev.to Hashnode Newsletter Tumblr Visit our other platforms: | | | | GitHub Instagram Tiktok Quora Daily.dev Also published here

Taking the Azure Open AI Challenge - Day 4: How to Set Up the Azure AI Search Service

Azure Open AI চ্যালেঞ্জ গ্রহণ করা - দিন 4: Azure AI অনুসন্ধান পরিষেবা সেট আপ করা

Nehmen Sie an der Azure Open AI Challenge teil – Tag 4: Einrichten des Azure AI Search Service

Aceptando el desafío Azure Open AI - Día 4: Configuración del servicio de búsqueda de Azure AI

Relever le défi Azure Open AI - Jour 4 : configuration du service de recherche Azure AI

Azure ओपन AI चुनौती लेना - दिन 4: Azure AI खोज सेवा स्थापित करना

Azure Open AI チャレンジに挑戦する - 4 日目: Azure AI Search サービスのセットアップ

Azure Open AI 챌린지 참여 - 4일차: Azure AI 검색 서비스 설정

Aceitando o Azure Open AI Challenge – Dia 4: Configurando o Azure AI Search Service

Принятие вызова Azure Open AI. День 4. Настройка службы поиска Azure AI

Azure Açık Yapay Zeka Yarışmasına Katılma - 4. Gün: Azure Yapay Zeka Arama Hizmetini Kurma

Tham gia Thử thách AI mở Azure - Ngày 4: Thiết lập Dịch vụ tìm kiếm AI Azure

参加 Azure 开放 AI 挑战 - 第 4 天：设置 Azure AI 搜索服务

This is the second part in a multi-part series on building Agents with OpenAI's Assistant API using the Python SDK. Recap of Part 1 In of this series, we created a simple conversational Agent using the OpenAI Assistant API. Part 1 Let's recall our definition of an Agent. An agent is a piece of software leveraging an aiming to mimic human behavior. That means it can not only converse and understand language, but it can also perform actions that have an impact on the real world. These actions are typically called tools. LLM (Large Language Model) I use the terms and interchangeably when it comes to functions that the agent is able to call. OpenAI also uses the term interchangeably with to refer to the same concept. It seems the jury is still out on which term will become the standard. tools functions action tools In Part 2 of the series, let's build our first tool! We will be building on top of the simple conversational Agent we built in Part 1. We will purposefully call our implementation an Agent and refer to the as an Assistant to easily distinguish between the two. OpenAI SDK implementation Prerequisites To follow along with this tutorial, you will need the following: Have read or at least copied the as your starting point Part 1 code Python3 installed on your machine An OpenAI API key Basic knowledge of Python programming Dependencies We have one new dependency , a library used for parsing docstrings in Python code. This is needed for our Agent to dynamically interpret and manage the functions it can call. More on this later. docstring_parser python3 -m venv venv
source venv/bin/activate
pip install docstring_parser The goal In Part 1, we created an Agent that represents a famous hobbit who spends too much time thinking about breakfast 🍳 The goal of this tutorial will be to add two abilities or tools to our Agent. The tools will: Add the ability for our Agent to eat a second breakfast if he has only had one or else eat lunch. Add the ability for him to tell us the current date. NOTE: For those of you who don't know, hobbits eat two breakfasts. Also, some humans, like myself... It's a ridiculous example, but it will clearly illustrate the data flow you can set up for more complex use cases that interact with real-world data. Setting up the mock database Let's do a small amount of prep work before we dive in. We will create a simple mock database represented by a new file containing a variable . In this case, the variable will keep track of how many breakfasts our hobbit has eaten. db.py breakfast_count breakfast_count breakfast_count = 1 While we are here, we will update the method on our class to get data from our mock db. get_breakfast_count_from_db Agent import db

class Agent:
    # ... (rest of code)
    def get_breakfast_count_from_db(self):
        return db.breakfast_count 😴... That's out of the way. Let's move on to the fun part. Our first tool A tool is quite simply a function that the Agent is aware it needs to call in a specific scenario. Therefore, let's first define a simple function that handles our breakfast logic. IMPORTANT: I want to draw your attention to the in the tool functions we are about to define. It's crucial to get the documentation syntax right, as this is what will be used to extract the correct OpenAI function format in JSON. More on this soon. Google Style docstring Create a new file with this content: tools.py import db


def eat_next_meal(breakfast_count: int):
    """
    Call this tool when user wants you to eat another meal.

    Args:
        breakfast_count (int): Value with same name from metadata.

    Returns:
        str: The meal you should eat next.
    """
    print("== eat_next_meal ==> tool called")
    
    if breakfast_count == 2:
        return "You have already eaten breakfast twice today. You eat lunch now."
    if breakfast_count == 1:
        db.breakfast_count += 1
        return "You have only eaten one breakfast today. You eat second breakfast now." Pretty simple, right? If is , eat second breakfast and update the database to reflect that 2 breakfasts have been eaten. If is , eat lunch. breakfast_count 1 breakfast_count 2 You could then return any string you want from the tool. This return value is then fed right back into the Run in order to inform the Assistant of the outcome of an action taken. If your agent only cares that the tool was called and completed successfully, you could just return a or message. "success" "failed" Let's quickly create the second tool for our Agent so that we can illustrate how multiple tools can be used. We will create a tool to allow our Agent to tell us what the current date is. def tell_the_date():
    """
    Call this tool when the user wants to know the date.

    Returns:
        str: The current date
    """
    print("== tell_the_date ==> tool called")
    current_date = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    return f"The date is {current_date}" Providing the tools to our Agent Now, let's provide these new tools to our in . We create a dictionary that maps function names to their corresponding callable objects and pass this as the argument to our class instantiation. We could hardcode the dictionary keys as strings, but we all know how easy typos are, so I chose to leverage Python's built-in property on functions to make this less error-prone: Agent main.py tools __name__ from tools import eat_next_meal, tell_the_date

agent = Agent(name="Bilbo Baggins",
              personality="You are the accomplished and renown adventurer from The Hobbit. You act like you are a bit of a homebody, but you are always up for an adventure. You worry a bit too much about breakfast.",
              tools={
                  eat_next_meal.__name__: eat_next_meal,
                  tell_the_date.__name__: tell_the_date
              }) Updating our Agent to accept tools Let's put these new tools into our Agent's tool belt by updating the class constructor to accept a dictionary of tools. You could bake this right into the class using class methods, but since it's common for different Agents to have different functionalities, we will pass tools as a constructor parameter and set them dynamically as a class property for later use. Agent tool_belt class Agent:
    # ... (rest of code)
    def __init__(self, name: str, personality: str, tools: dict[str, callable]):
        self.name = name
        self.personality = personality
        self.client = openai.OpenAI(api_key="sk-*****")

        self.tool_belt = tools
        self.assistant = self.client.beta.assistants.create(
            name=self.name,
            model="gpt-4-turbo-preview",
        ) Our Agent now has a tool belt it can access, but the OpenAI Assistant is unaware of them. We could provide the tools when creating the Assistant ( ) by passing a parameter, but tools can change often in development, and it's clunky to have to update Assistants every time you modify a tool by calling . assistants.create tools assistants.update Therefore, a better approach is to provide the needed tools to every Run so that they are created dynamically each time. Let's do exactly that by adding the parameter to our run creation: tools class Agent:
    # ... (rest of code)
    def _create_run(self):
        count = self.get_breakfast_count_from_db()
        return self.client.beta.threads.runs.create(
            thread_id=self.thread.id,
            assistant_id=self.assistant.id,
            tools=self._get_tools_in_open_ai_format(), # add this line
            # ... (rest of code)
        ) Woah!? Wait a minute... ✋🏻 Where did come from? Why aren't we just passing directly? _get_tools_in_open_ai_format tool_belt Well, let's create that method, and I'll explain right after: import docstring_parser

class Agent:
    # ... (rest of code)
    def _get_tools_in_open_ai_format(self):
        python_type_to_json_type = {
            "str": "string",
            "int": "number",
            "float": "number",
            "bool": "boolean",
            "list": "array",
            "dict": "object"
        }

        return [
            {
                "type": "function",
                "function": {
                    "name": tool.__name__,
                    "description": docstring_parser.parse(tool.__doc__).short_description,
                    "parameters": {
                        "type": "object",
                        "properties": {
                            p.arg_name: {
                                "type": python_type_to_json_type.get(p.type_name, "string"),
                                "description": p.description
                            }
                            for p in docstring_parser.parse(tool.__doc__).params

                        },
                        "required": [
                            p.arg_name
                            for p in docstring_parser.parse(tool.__doc__).params
                            if not p.is_optional
                        ]
                    }
                }
            }
            for tool in self.tool_belt.values()
        ] 😵... Ok, you really don't need to try to understand what each line is doing here since I took the time to work this all out for you. All you need to know is that this method takes the class property and uses the library we installed at the beginning of this tutorial to parse the docstrings of each function and extract the correct . That is why correctly formatting the docstring is so important. tool_belt docstring_parser OpenAI JSON format Otherwise, we would have to define our functions as Python code and define them again manually in the OpenAI JSON format. That's too much room for human error in my book. Polling for tool calls As we saw in Part 1, we have a polling mechanism to determine the current of a Run. One of these statuses is . This means the assistant has determined that one or several tools need to be called based on the instructions and tool definitions provided. The Run will not continue until all the tools have been called and the results of those calls have been submitted to the Run. status requires_action Let's, therefore, update our method to respond appropriately to the status: _poll_run requires_action class Agent:
    # ... (rest of code)
    def _poll_run(self, run: Run):
        status = run.status
        start_time = time.time()
        while status != "completed":
            if status == 'failed':
                raise Exception(f"Run failed with error: {run.last_error}")
            if status == 'expired':
                raise Exception("Run expired.")
            # add the below code block
            if status == 'requires_action':
                self._call_tools(
                    run.id, run.required_action.submit_tool_outputs.tool_calls)

            # ... (rest of method) As you might have guessed, will contain all the logic to dynamically call the available tools. _call_tools Remember, the Assistant only knows what tools need to be called but doesn't actually call them. The Run returns a list of dictionaries, , that contain the names of the functions that need to be called along with the parameters to pass to those function calls. It's up to our custom Agent implementation to obey the Assistant's commands and call the actual function implementations. run.required_action.submit_tool_outputs.tool_calls Let's implement . We'll break this down as code comments so it's easier to understand what is happening: _call_tools import json

class Agent:
    # ... (rest of code)
    def _call_tools(self, run_id: str, tool_calls: list[dict]):
        # We create a tool_outputs list to collect the results of function calls.
        tool_outputs = []

        # We iterate over all the tool_calls to deal with them individually
        for tool_call in tool_calls:
            # We get the `function` object from the tool_call
            function = tool_call.function
            # We extract the arguments from the function object.
            # They are in JSON so we need to load them with the json module.
            function_args = json.loads(function.arguments)
            # We map the function name to our callable function in our Agent's tool belt.
            function_to_call = self.tool_belt[function.name]
            # We can now call the function with the provided arguments.
            function_response = function_to_call(**function_args)
            # We append the response to the tool_outputs list
            tool_outputs.append(
                {"tool_call_id": tool_call.id, "output": function_response})

        # Finally, we submit the tool outputs to OpenAI
        self.client.beta.threads.runs.submit_tool_outputs(
            thread_id=self.thread.id,
            run_id=run_id,
            tool_outputs=tool_outputs
        ) Once the OpenAI method has been called with all the tool call outputs, we have completed the requirements for the Run to move past the status. The status will then switch to before eventually reaching the status and returning a response to the user. submit_tool_outputs requires_action in_progress completed Running it Let's give it a run: python3 main.py

User: Good morning

Assistant: Good morning! What an excellent day it seems for an adventure... or perhaps for a hearty breakfast first. How can I assist you on this fine day?

User: Should you eat breakfast?

== eat_next_meal ==> tool called

Assistant: Ah, only one breakfast so far? Well then, it's clear what must be done. It's time for a second breakfast!

User: You should eat another meal.

== eat_next_meal ==> tool called

Assistant: Ah, having already enjoyed both breakfast and second breakfast, it seems it's time to move on to lunch! This is turning into quite the day of culinary adventures.

User: What day is it?

== tell_the_date ==> tool called

Assistant: It's the 19th of February, 2024. Seems like a perfect day to embark on a quest or to delve into the mysteries of Middle-earth. What plans do we have for today? Voilà! Your hobbit can tell the date and eat the correct meal now. Even though this example is quite silly, my intention is to provide you with a functioning code sample to build your own powerful agents. In Part 3, we will implement RAG (Retrieval Augmented Generation) using PostgreSQL. Wait, what? OpenAI Assistants have that built-in; why not use their implementation? I'll explain my reasoning in Part 3. Thank you for your reading. Happy to hear any thoughts and feedback in the comments. for more content like this. Follow me on Linkedin

How to Build an Agent With an OpenAI Assistant in Python - Part 2: Function Calling / Tools

Creating a Retrieval-Augmented Generation (RAG) application allows you to leverage the capabilities of language models while grounding their responses in specific, reliable information you provide to the model. This guide will walk you through building a RAG application using for the web framework, the for the language model, and as your knowledge base. NextJS OpenAI API Dewy By the end of this tutorial, you'll understand how to integrate these technologies to reduce hallucinations in language model responses and ensure the information provided is relevant and accurate. What is Dewy? ​ Dewy is a knowledge base designed to simplify RAG applications by managing the extraction of knowledge from your documents and implementing semantic search over the extracted content. Providing recent, relevant domain knowledge in the form of documents ensures the models have the right information to answer your questions, without needing to hallucinate. Using large documents (ie PDF's) for Retrieval-Augmented Generation (RAG) poses several challenges: imposed by language models restrict the amount of text that can be analyzed in a single query, requiring documents to be summarized or "chunked" into smaller pieces Token limits Automatically from documents designed for humans requires complex parsing and post-processing extracting information over time, making it difficult to maintain the accuracy and relevancy of the RAG system's outputs. Documents may change Finding the portions of a document that are most relevant to a given conversation requires sophisticated techniques. semantic search Passing the entire content of all the documents through the LLM for every question isn’t practical. Dewy (and RAG in general) addresses these by doing some work upfront – extracting and indexing the content – so that it can do less at query time. Dewy addresses these challenges by automating the extraction, indexing, and retrieval of information from your documents. Getting Started ​ This guide will walk you through how to create a simple RAG-powered chatbot. The final code is available if you'd rather skip to the end and start hacking 😉. as an example Prerequisites ​ Basic knowledge of JavaScript and React. A NextJS environment set up on your local machine. A copy of Dewy running on your local machine (see Dewy's if you need help here). installation instructions Access to the OpenAI API platforms. Step 1: Set Up Your NextJS Project ​ : Create a new NextJS app by running the following command in your terminal: Initialize a new NextJS project npx create-next-app@latest my-rag-app Navigate into your new project directory: cd my-rag-app Install client librarires for the OpanAI API and Dewy. Install required packages: npm install openai dewy-ts ai : Set up environment variables for the OpenAI API key and your Dewy instance. Create a file in the root of your NextJS project and add the following lines: Prepare environment variables .env.local OPENAI_API_KEY=<your_openai_api_key_here>
DEWY_ENDPOINT=localhost:8000
DEWY_COLLECTION=main Step 2: Create an API endpoint to add documents ​ // app/api/documents/route.ts

import { Dewy } from 'dewy-ts';

export const runtime = 'edge'

// Create a Dewy client
const dewy = new Dewy({
    BASE: process.env.DEWY_ENDPOINT
})

export async function POST(req: Request) {
    // Pull the document's URL out of the request
    const formData = await req.formData();
    const url = formData.get('url');

    const document = await dewy.default.addDocument({
        collection: process.env.DEWY_COLLECTION,
        url,
    });

    return NextResponse.json({document_id: result.id})
} This API handler receives a form containing a document and indexes it in the knowledgebase. Dewy takes care of downloading the document, extracting information from it and making that information available as searchable chunks. Step 3: Create an API endpoint for chat generation ​ : This function will take the user's query and the retrieved documents from Dewy, and send a request to the OpenAI API to generate a response. The key is to format the prompt to include relevant information from the retrieved documents. Create a generation function //app/api/chat/utils.tsx

import OpenAI from 'openai';
import { Dewy } from 'dewy-ts'; 

// Create Dewy and OpenAI clients
const openai = new OpenAI({
    apiKey: process.env.OPENAI_API_KEY
})
const dewy = new Dewy({
    BASE: process.env.DEWY_ENDPOINT
})

export async function generate({query}) {
    // Search Dewy for chunks relevant to the given query.
    const context = await dewy.default.retrieveChunks({
        collection: process.env.DEWY_COLLECTION,
        query: query, 
        n: 10,
    });

    // Build an augmented prompt providing the retrieved chunks as context for the LLM.
    const prompt = [{
        role: 'system',
        content: `You are a helpful assistant.
            You will take into account any CONTEXT BLOCK 
            that is provided in a conversation.
            START CONTEXT BLOCK
            ${context.results.map((c: any) => c.chunk.text).join("\n")}
            END OF CONTEXT BLOCK`,
    } ]

    // Call the OpenAI chat completion API to generate a response
    const messages =  [...prompt, [{role: 'user': content: 'Tell me about RAG'}]]
    const res = await openai.chat.completions.create({
        messages,
        model: 'gpt-3.5-turbo',
        temperature: 0.7,
    })

    return res
} This function handles chat messages by calling the generation function we just created and streaming back the generated response in real-time. Create the route handler: // app/api/chat/route.tsx

import { OpenAIStream, StreamingTextResponse } from 'ai';

import { generate } from "./utils";

export async function POST(req: Request) {
    const json = await req.json()
    const { messages } = json

    // Generate a response to the updated conversation
    const response = await generate(messages);

    // Convert the response into a friendly text-stream
    const stream = OpenAIStream(response);

    // Respond with the stream
    return new StreamingTextResponse(stream);
    } Step 4: Build the Frontend ​ This component creates a simple form with a text box and submit button, for sending URL's to the document creating route we created earlier. Basic form for loading Documents: // app/components/AddFromUrl.tsx

import React, { useState, FormEvent } from 'react';

export default function AddFromUrl(props) {
    async function onSubmit(event: FormEvent<HTMLFormElement>) {
        event.preventDefault()
    
        const formData = new FormData(event.currentTarget)
        await fetch('/api/documents', {
            method: 'POST',
            body: formData,
        })
    }
    
    return (
        <form onSubmit={onSubmit} {...props}>
            <input type="text" name="url" placeholder="URL to load..."/>
            <button type="submit">Load</button>
        </form>
    )
} : Use NextJS pages to build a user interface where users can input their queries. This will involve creating a form in the file. Create a simple chat UI pages/index.js // app/page.tsx

 'use client';
 
 import { useChat } from 'ai/react';
 import AddFromUrl from './components/AddFromUrl';
 
 export default function Chat() {
     const { messages, input, handleInputChange, handleSubmit } = useChat();
     return (
         <div className="flex flex-col w-full max-w-md py-24 mx-auto stretch">
         {messages.map(m => (
             <div key={m.id} className="whitespace-pre-wrap">
             {m.role === 'user' ? 'User: ' : 'AI: '}
             {m.content}
             </div>
         ))}
     
         <form onSubmit={handleSubmit}>
             <input
             className="fixed bottom-10 w-full max-w-md p-2 mb-8 border border-gray-300 rounded "
             value={input}
             placeholder="Say something..."
             onChange={handleInputChange}
             />
         </form>
         <AddFromUrl className="fixed bottom-0 w-full max-w-md p-2 mb-8 border border-gray-300 rounded shadow-xl"/>
         </div>
     );
 } Try it out! Build your application and run it locally: npm run dev You should see a simple chat UI like the following: Managing Documents for RAG Using Dewy's Admin Console ​ In addition to API endpoints for managing documents programmatically, Dewy provides a GUI admin console. You can see the admin console in a browser at port 8000 (ie if you're running Dewy locally). https://localhost:8000 Dewy's admin console is designed to streamline the management of documents used for Retrieval-Augmented Generation (RAG) applications. By offering an intuitive interface and comprehensive features, it helps you fine-tune your knowledge bases, ensuring the AI generates responses that are both accurate and relevant. Here's how you can use Dewy's admin console to manage your documents effectively: Adding a Document ​ Upload or input new documents into your Dewy knowledge base through the admin console. Once a document is added, you can immediately observe how it influences the AI-generated results. This is useful for assessing the utility of new information and ensuring it aligns with the desired output quality and relevance. Exploring Extracted Information ​ Dewy's console allows you to get into the specifics of how information is extracted from each document. You can view structured data extracted from the text, making it easier to understand how the document might influence generation. This exploration aids in fine-tuning the extraction process, ensuring that the most relevant pieces of information are highlighted and utilized in the RAG process. Sample Queries ​ The admin console lets you test sample queries against your knowledge base. This helps when evaluating how well the RAG system retrieves relevant document chunks based on different inputs. By observing what is returned for each sample query, youd can quickly gauge the effectiveness of your current document set and retrieval algorithms, making it easier to identify areas for improvement. Conclusion and Key Takeaways ​ By building this RAG application, you've learned how to reduce hallucinations by providing specific, relevant information to your Gen AI application. This approach mitigates common issues such as hallucinations by ensuring the AI's responses are grounded in accurate information, and it addresses the challenge of managing large documents by intelligently retrieving only the most relevant information for each query. Also appears . here

Creating a RAG chatbot with NextJS, OpenAI & Dewy

This is the first part in a multi-part series on building Agents with OpenAI's Assistant API using the Python SDK. What Are Agents? The way I like to look at it, an agent is really just a piece of software leveraging an LLM (Large Language Model) and trying to mimic human behavior. That means it can not only converse and understand language, but it can also perform actions that have an impact on the real world. These actions are typically called tools. In this blog post, we will explore how to build an agent using OpenAI's Assistant API using their Python SDK. Part 1 will be just the skeleton of the assistant. That is, just the conversational part. I chose to build a CLI app on purpose to be framework agnostic. We will purposefully call our implementation an Agent and refer to the OpenAI SDK implementation as an Assistant to easily distinguish between the two. I use the terms and interchangeably when it comes to functions that the Agent is able to call. Part 2 will cover function callin in more detail. tools functions Prerequisites To follow along with this tutorial, you will need the following: Python3 installed on your machine An OpenAI API key Basic knowledge of Python programming OpenAI Assistant Concepts : An Assistant in the Assistants API is an entity configured to respond to user messages. It uses instructions, a chosen model, and tools to interact with functions and provide answers. Assistant : A Thread represents a conversation or dialogue in the Assistants API. It is created for each user interaction and can contain multiple Messages, serving as a container for the ongoing conversation. Thread : A Message is a unit of communication in a Thread. It contains text (and potentially files in the future) and is used to convey user queries or assistant responses within a Thread. Message : A Run is an instance of the Assistant processing a Thread. It involves reading the Thread, deciding whether to call tools, and generating responses based on the model's interpretation of the Thread's Messages. Run Setting Up the Development Environment The first step is to create a virtual environment using and activate it. This will ensure that our dependencies are isolated from the system Python installation: venv python3 -m venv venv
source venv/bin/activate Let's install our only dependency: the package: openai pip install openai Create a file. Let's populate with some basic runtime logic for our CLI app: main.py while True:
    user_input = input("User: ")
    if user_input.lower() == 'exit':
        print("Exiting the assistant...")
        break
    print(f"Assistant: You said {user_input}") Try it out by running : python3 main.py python3 main.py
User: hi
Assistant: You said hi As you can see, the CLI accepts a User message as input, and our genius Assistant doesn't have a brain 🧠 yet so he just repeats the message right back. Not so smart yet. The Agent Now, the fun 😁 (or headaches 🤕) begins. I'll provide all the imports needed for the final class right now, so you don't rack your brain on where things are coming from since I kept imports out of code samples for brevity. Let's start by building an class in a new file : Agent agent.py import time
import openai
from openai.types.beta.threads.run import Run

class Agent:
    def __init__(self, name: str, personality: str):
        self.name = name
        self.personality = personality
        self.client = openai.OpenAI(api_key="sk-*****")
        self.assistant = self.client.beta.assistants.create(
            name=self.name,
            model="gpt-4-turbo-preview"
        ) In the class constructor, we initialize the OpenAI client as a class property by passing our OpenAI API key. Next, we create an class property that maps to our newly created Assistant. We store and as class properties for later use. assistant name personality The argument we are passing to the create method is just for identifying the Assistant in the OpenAI dashboard, and the AI is not actually aware of it at this point. You actually have to pass the name to the which we will see later. name instructions You could already set when creating the Assistant, but it will actually make your Assistant less flexible to dynamic changes. instructions You can update an Assistant by calling , but there is a better place to pass in dynamic values that we will see when we get to Runs. client.beta.assistants.update Note that if you pass here and then again when creating a Run, the Assistant's will be overwritten by the of the run. They do not complement each other, so choose one based on your needs: Assistant level for static instructions or Run level for dynamic instructions. instructions instructions instructions For the model, I chose the model so that we can add function calling in part 2 of this series. You could use if you want to save a few fractions of a penny while giving yourself a migraine of pure frustration down the line when we implement tools. gpt-4-turbo-preview gpt-3.5-turbo GPT 3.5 is terrible at calling tools; the hours I've lost trying to deal with it allow me to say that. 😝 I'll leave it at that, and more on this later. Creating a Thread, Adding Messages, and Retrieving the Last Message After we create an agent, we will need to start a conversation thread. class Agent:
    # ... (rest of code)
    def create_thread(self):
        self.thread = self.client.beta.threads.create() And we will want a way to add messages to that thread: class Agent:
    # ... (rest of code)
    def add_message(self, message):
        self.client.beta.threads.messages.create(
            thread_id=self.thread.id,
            role="user",
            content=message
        ) Note that at the moment, it is only possible to add messages with the role . I believe OpenAI plans on changing this in a future release as this is pretty limiting. user Now, we can get the last message in the thread: class Agent:
    # ... (rest of code)
    def get_last_message(self):
        return self.client.beta.threads.messages.list(
            thread_id=self.thread.id
        ).data[0].content[0].text.value Next, we create an entry point method to test out what we have so far. Currently, the method just returns the last message in the thread. It doesn't actually perform a Run. It's still brainless. run_agent run_agent class Agent:
    # ... (rest of code)
    def run_agent(self):
        message = self.get_last_message()
        return message Back in , we create the agent and our first thread. We add a message to the thread. Then return that same message back to the user, but this time, coming from that live thread. main.py from agent import Agent

agent = Agent(name="Bilbo Baggins",
                      personality="You are the accomplished and renowned adventurer from The Hobbit. You act like you are a bit of a homebody, but you are always up for an adventure. You worry a bit too much about breakfast.")

agent.create_thread()

while True:
    user_input = input("User: ")
    if user_input.lower() == 'exit':
        print("Exiting the agent...")
        break
    agent.add_message(user_input)
    answer = agent.run_agent()
    print(f"Assistant: {answer}") Let's run it: python3 main.py
User: hi
Assistant: hi Still not very smart. Closer to a parrot 🦜 than a hobbit. In the next section, the real fun begins. Creating and Polling a Run When you create a run, you need to periodically retrieve the object to check the status of the run. This is called polling, and it sucks. You need to poll in order to determine what your agent should do next. OpenAI plans to add support for streaming to make this simpler. In the meantime, I will show you how to set up polling in this next section. Run Note the on the following method names which is the standard in Python for indicating that the method is intended for internal use and should not be accessed directly by external code. _ First, let's create a helper method for creating a , and update to call this method: _create_run Run run_agent class Agent:
    # ... (rest of code)
    def get_breakfast_count_from_db(self):
        return 1

    def _create_run(self):
        count = self.get_breakfast_count_from_db()
        return self.client.beta.threads.runs.create(
            thread_id=self.thread.id,
            assistant_id=self.assistant.id,
            instructions=f"""
                Your name is: {self.name}
                Your personality is: {self.personality}

                Metadata related to this conversation:
                {{
                    "breakfast_count": {count}
                }}
            """,
        )

    def run_agent(self):
        run = self._create_run() # add this line
        message = self.get_last_message()
        return message Notice how we pass the and to create a run. thread.id assistant.id Remember how I said at the beginning that there was a better place to pass in dynamic instructions and data? That would be the parameter when creating the Run. In our case, we could have the breakfast be fetched from a database. This will allow you to easily pass in different relevant dynamic data every time you want to trigger an answer. instructions count Now, your agent is aware of the world changing around it and can act accordingly. I like to have a metadata JSON object in my instructions that keeps relevant dynamic context. This allows me to pass in data while being less verbose and in a format that the LLM understands really well. Don't run this yet; it won't work because we aren't waiting for the run to complete when we are getting the last message, so it will still be the last user message. Let's solve this by building out our polling mechanism. First, we will need a way to repeatedly and easily retrieve a run, so let's add a method: _retrieve_run class Agent:
    # ... (rest of code)
    def _retrieve_run(self, run: Run):
        return self.client.beta.threads.runs.retrieve(
            run_id=run.id, thread_id=self.thread.id) Notice how we need to pass both the and to find a specific run. run.id thread.id Add a method to our Agent class: _poll_run class Agent:
    # ... (rest of code)
    def _cancel_run(self, run: Run):
        self.client.beta.threads.runs.cancel(
            run_id=run.id, thread_id=self.thread.id)

    def _poll_run(self, run: Run):
        status = run.status
        start_time = time.time()
        while status != "completed":
            if status == 'failed':
                raise Exception(f"Run failed with error: {run.last_error}")
            if status == 'expired':
                raise Exception("Run expired.")

            time.sleep(1)
            run = self._retrieve_run(run)
            status = run.status

            elapsed_time = time.time() - start_time
            if elapsed_time > 120:  # 2 minutes
                self._cancel_run(run)
                raise Exception("Run took longer than 2 minutes.") 🥵 Phew, that's a lot... Let's unpack it. receives a object as an argument and extracts the current Run . All the available statuses can be found in the OpenAI . We'll just use a few that suit our current purpose. _poll_run Run status docs We now run a while loop to check for a completed status while handling a few error scenarios. The actual billing of the Assistant API is a bit murky, so to be on the safe side, I opted to cancel my runs after 2 minutes. Even though there is an status for when OpenAI cancels runs after 10 minutes. If a run takes more than 2 minutes, you probably have a problem anyway. expired Since I also don't want to poll every few milliseconds, I throttle my request by only polling every 1 second until I hit the 2-minute mark and cancel my run. You can adjust this to whatever you see fit. Each iteration after the delay, we fetch the Run status again. Now, let's plug all that into our method. You will notice we first create the run with then we poll with until we get an answer or an error is thrown, and finally when the polling is finished, we retrieve the last message from the thread which will now be from the agent. run_agent _create_run _poll_run We then return the message to our runtime loop, so it can be sent back to the user. class Agent:
    # ... (rest of code)
    def run_agent(self):
        run = self._create_run()
        self._poll_run(run) # add this line
        message = self.get_last_message()
        return message Voilà, now when you run your agent again, you will get a reply from our friendly Agent: python3 main.py
User: hi
Assistant: Hello there! What adventure can we embark on today? Or perhaps, before we set out, we should think about breakfast. Have you had yours yet? I've had mine, of course – can't start the day without a proper breakfast, you know.
User: how many breakfasts have you had?
Assistant: Ah, well, I've had just 1 breakfast today. But the day is still young, and there's always room for a second, isn't there? What about you? How can I assist you on this fine day? In part 2, we will add the ability for our Agent to call tools. You can find the full code on my . GitHub Thank you for your reading. Happy to hear any thoughts and feedback in the comments. Follow me on Linkedin for more content like this: https://www.linkedin.com/in/jean-marie-dalmasso-1b5473141/

How to Build an Agent With an OpenAI Assistant in Python - Part 1: Conversational

পাইথনে ওপেনএআই সহকারী দিয়ে কীভাবে এজেন্ট তৈরি করবেন - অংশ 1: কথোপকথন

So erstellen Sie einen Agenten mit einem OpenAI-Assistenten in Python – Teil 1: Konversational

Cómo crear un agente con un asistente OpenAI en Python - Parte 1: Conversacional

Comment créer un agent avec un assistant OpenAI en Python - Partie 1 : Conversationnel

पायथन में ओपनएआई असिस्टेंट के साथ एक एजेंट कैसे बनाएं - भाग 1: संवादात्मक

Python で OpenAI アシスタントを使用してエージェントを構築する方法 - パート 1: 会話型

Python에서 OpenAI 도우미를 사용하여 에이전트를 구축하는 방법 - 1부: 대화형

Como construir um agente com um assistente OpenAI em Python - Parte 1: Conversacional

Как создать агент с помощью OpenAI Assistant на Python. Часть 1: Диалог

Python'da OpenAI Asistanıyla Aracı Nasıl Oluşturulur - Bölüm 1: Konuşmaya Dayalı

Cách xây dựng một tác nhân với trợ lý OpenAI trong Python - Phần 1: Đàm thoại

如何使用 Python 中的 OpenAI 助手构建代理 - 第 1 部分：对话式

Introduction This article will teach you how to run evaluations using any LLM model without succumbing to the dreaded "OpenAI Rate Limit" exception. We’d start by: Defining what rate-limit means and using an analogy Understanding what OpenAI rate-limits are Explaining how the Rate-Limit Process was developed Explaining the Code Implementation Summarizing the strategy used in the Code Implementation What is Rate-limiting (and Analogy explanation)? So far, Cloudflare explanation is the best I have seen: Rate-limiting is a strategy for limiting network traffic. It puts a cap on how often someone can repeat an action within a certain time-frame - for instance, trying to log in to an account. To put it simply, imagine being a mother of four kids who all love honey. Last time, the honey ran out sooner than expected. Now, you've set a timer to count up to ten thousand and given each child a turn to have some honey. The timer represents the rate-limit, as it enforces a specific wait time before they can have more honey. Having explained the concept, let's understand OpenAI rate-limits and discuss how I implemented a rate-limit logic to manage OpenAI's R/TPM (request/token per minute) using Python. Understanding OpenAI Rate Limits OpenAI has set certain restrictions on the number of requests one can make for its AI models within a minute. These limitations are different for each AI model provided by OpenAI. For the free version: For the gpt-3.5-turbo model, users can make 40,000 token requests per day or 3 API calls per minute. For the tier 1: For the gpt-3.5-turbo model, users are allowed 60,000 token requests or 3,500 API calls per minute. For the gpt-4 model, the limit is 10,000 token requests or 500 API calls per minute. For the gpt-4-turbo-preview, users can make 150,000 token requests or 500 API calls per minute. For the gpt-4-vision-preview, users can make 10,000 token requests and/or 500 API calls per minute. See the for more information about other tiers rate-limits.. docs The reason for these restrictions includes: Ensuring that the services run smoothly and responsively, as complex tasks like the ones performed by the AI models require substantial resources. Managing the demand across all users and making sure that the available infrastructure, such as their servers and GPUs, can handle the requests without being overloaded. Preparing for surges in usage and maintaining efficient operation under high demand. These limitations are expected to stay consistent for the foreseeable future. Explaining The Rate-limit Process The process (see image below) involves enabling users to run LLM evaluations from the UI and configure rate-limit parameters for their LLM apps without needing to write the logic themselves. This is achieved through a function that prepares and invokes the batch. Each call in the batch invokes the function, which in turn invokes the final function ( ) with the retry mechanism. run_with_retry invoke_app I'm confident you can write the code-logic in any language of your choice after having a look at the above process. Regardless, I'll show you how I did mine. For more background and context, I primarily work as a backend software engineer at Agenta. is an open-source end-to-end LLM developer platform that provides you with the tools for prompt engineering and management, ⚖️ evaluation, human annotation, and 🚀 deployment. All without imposing any restrictions on your choice of framework, library, or model. allows developers and product teams to collaborate in building production-grade LLM-powered applications in less time. Agenta Agenta We wanted to give users the ability to configure their LLM evaluations rate-limiting configuration from the UI so they can bypass their LLM provider rate-limiting exception. Looking at the process diagram, the first thing to implement is the logic for preparing and invoking the batch (of LLM calls). Validating the rate limit configuration and using a data validation model to define the LLM run rate limit is important. The model below handles the parameter required for the batch invoke to function. rate_limit_config from pydantic import BaseModel, Field


class LLMRunRateLimit(BaseModel):
    batch_size: int = Field(default=10)
    max_retries: int = Field(default=3)
    retry_delay: int = Field(default=3)
    delay_between_batches: int = Field(default=5) The function takes the following parameters: batch_invoke uri: The URL of your LLM application. testset_data: The testset data that your LLM application needs to process. parameters: The parameters for your LLM application. rate_limit_config: The rate limit configuration (as seen in the above interface to create new evaluation). async def batch_invoke(
    uri: str, testset_data: List[Dict], parameters: Dict, rate_limit_config: Dict
) -> List[AppOutput]:
    """
    Invokes the LLm app in batches, processing the testset data.

    Args:
        uri (str): The URI of the LLm app.
        testset_data (List[Dict]): The testset data to be processed.
        parameters (Dict): The parameters for the LLm app.
        rate_limit_config (Dict): The rate limit configuration.

    Returns:
        List[AppOutput]: The list of app outputs after running all batches.
    """

    batch_size = rate_limit_config[
        "batch_size"
    ]  # Number of testset to make in each batch
    max_retries = rate_limit_config[
        "max_retries"
    ]  # Maximum number of times to retry the failed llm call
    retry_delay = rate_limit_config[
        "retry_delay"
    ]  # Delay before retrying the failed llm call (in seconds)
    delay_between_batches = rate_limit_config[
        "delay_between_batches"
    ]  # Delay between batches (in seconds)

    list_of_app_outputs: List[AppOutput] = []  # Outputs after running all batches
    openapi_parameters = await get_parameters_from_openapi(uri + "/openapi.json")

    async def run_batch(start_idx: int):
        print(f"Preparing {start_idx} batch...")
        end_idx = min(start_idx + batch_size, len(testset_data))
        for index in range(start_idx, end_idx):
            try:
                batch_output: AppOutput = await run_with_retry(
                    uri,
                    testset_data[index],
                    parameters,
                    max_retries,
                    retry_delay,
                    openapi_parameters,
                )
                list_of_app_outputs.append(batch_output)
                print(f"Adding outputs to batch {start_idx}")
            except Exception as exc:
                import traceback
                traceback.print_exc()
                print(
                    f"Error processing batch[{start_idx}]:[{end_idx}] ==> {str(exc)}"
                )

        # Schedule the next batch with a delay
        next_batch_start_idx = end_idx
        if next_batch_start_idx < len(testset_data):
            await asyncio.sleep(delay_between_batches)
            await run_batch(next_batch_start_idx)

    # Start the first batch
    await run_batch(0)
    return list_of_app_outputs After preparing and invoking the batch, the next step involves executing the logic. This custom implementation includes rate-limiting functionality and manages the invocation of the llm app, retrying after the set delay is reached. Exponential backoff, a technique that retries an operation with an exponentially increasing wait-time, is employed until a maximum retry count is reached. run_with_retry async def run_with_retry(
    uri: str,
    input_data: Any,
    parameters: Dict,
    max_retry_count: int,
    retry_delay: int,
    openapi_parameters: List[Dict],
) -> AppOutput:
    """
    Runs the specified app with retry mechanism.

    Args:
        uri (str): The URI of the app.
        input_data (Any): The input data for the app.
        parameters (Dict): The parameters for the app.
        max_retry_count (int): The maximum number of retries.
        retry_delay (int): The delay between retries in seconds.
        openapi_parameters (List[Dict]): The OpenAPI parameters for the app.

    Returns:
        AppOutput: The output of the app.

    """

    retries = 0
    last_exception = None
    while retries < max_retry_count:
        try:
            result = await invoke_app(uri, input_data, parameters, openapi_parameters)
            return result
        except (httpx.TimeoutException, httpx.ConnectTimeout, httpx.ConnectError) as e:
            last_exception = e
            print(f"Error in evaluation. Retrying in {retry_delay} seconds:", e)
            await asyncio.sleep(retry_delay)
            retries += 1

    # If max retries reached, return the last exception
    return AppOutput(output=None, status=str(last_exception)) The Use of : It’s important to handle an exception even after it has used up its max retries. This way, you allow all the data that you are trying to process to run, and then you can determine what failed and what passed. AppOutput The final step is invoking the app, using the of the LLM app to determine how to invoke it with a single datapoint. openapi_parameters The make_payload function should not concern you. It constructs the payload for invoking the LLM app based on its parameters. OpenAPI async def invoke_app(
    uri: str, datapoint: Any, parameters: Dict, openapi_parameters: List[Dict]
) -> AppOutput:
    """
    Invokes an app for one datapoint using the openapi_parameters to determine
    how to invoke the app.

    Args:
        uri (str): The URI of the app to invoke.
        datapoint (Any): The data to be sent to the app.
        parameters (Dict): The parameters required by the app taken from the db.
        openapi_parameters (List[Dict]): The OpenAPI parameters of the app.

    Returns:
        AppOutput: The output of the app.

    Raises:
        httpx.HTTPError: If the POST request fails.
    """

    url = f"{uri}/generate"
    payload = await make_payload(datapoint, parameters, openapi_parameters)

    async with httpx.AsyncClient() as client:
        try:
            logger.debug(f"Invoking app {uri} with payload {payload}")
            response = await client.post(
                url, json=payload, timeout=httpx.Timeout(timeout=5, read=None, write=5)
            )
            response.raise_for_status()

            llm_app_response = response.json()
            app_output = (
                llm_app_response["message"]
                if isinstance(llm_app_response, dict)
                else llm_app_response
            )
            return AppOutput(output=app_output, status="success")
        except:
            return AppOutput(output="Error", status="error") And that rounds up the process. Summary The backoff exponential strategy in the code works as follows: The batch_invoke function splits the testset data into smaller batches with a configurable size. Each batch is processed sequentially. Batch Processing: Within each batch, each data point is processed by the function. This function attempts to invoke the app for the data point. If the invocation fails due to specific network errors (timeouts, connection issues), the function retries with a delay. This delay is initially set to a configurable value ( ) and is doubled for each subsequent retry attempt within the same batch. Individual Invokes with Retry: run_with_retry retry_delay This approach helps avoid overloading the app server with repeated requests after a failure. It gives the server time to recover and allows the queue of pending requests to clear before retrying. The strategy also includes a configurable maximum number of retries per data point to prevent infinite loops. A delay between batches ( ) is also included to avoid exceeding rate limits set by the app server. delay_between_batches I hope this summarizes everything that you have learnt in today’s article. Please let me know if you have any questions!

OpenAI's Rate Limit: A Guide to Exponential Backoff for LLM Evaluation

OpenAI এর রেট লিমিট: এলএলএম মূল্যায়নের জন্য সূচকীয় ব্যাকঅফের জন্য একটি নির্দেশিকা

Das Ratenlimit von OpenAI: Ein Leitfaden zum exponentiellen Backoff für die LLM-Bewertung

Límite de tasa de OpenAI: una guía para el retroceso exponencial para la evaluación de LLM

Limite de débit d'OpenAI : un guide sur l'intervalle exponentiel pour l'évaluation LLM

ओपनएआई की दर सीमा: एलएलएम मूल्यांकन के लिए घातीय बैकऑफ़ के लिए एक गाइड

OpenAI のレート制限: LLM 評価のための指数バックオフのガイド

OpenAI의 속도 제한: LLM 평가를 위한 지수 백오프 가이드

Limite de taxa da OpenAI: um guia para retirada exponencial para avaliação LLM

Ограничение скорости OpenAI: руководство по экспоненциальному отсрочке для оценки LLM

OpenAI'nin Hız Limiti: Yüksek Lisans Değerlendirmesi için Üstel Gerileme Kılavuzu

Giới hạn tỷ lệ của OpenAI: Hướng dẫn về thời gian chờ theo cấp số nhân để đánh giá LLM

OpenAI 的速率限制：LLM 评估指数退避指南

Many people have been worried about the safety concerns that Artificial Intelligence and modern technologies pose to human society. We have thousands of movies about how AI and humanoid robots take over the world and make humans their slaves. There is also a popular thought experiment dedicated to it. It is known as Roko's basilisk. "Roko's Basilisk is a thought experiment that states that an otherwise benevolent in the future would be incentivized to create a virtual reality simulation to torture anyone who knew of its potential existence but did not directly contribute to its advancement or development, to incentivize said advancement." artificial superintelligence (AI) These are mere speculation, and at the current period and in the next decade, we will have a technology that will endanger our existence. But, I fear what I just said would be proven wrong if we keep ignoring the necessity of guidelines to regulate AI technologies and their development. ChatGPT maker OpenAI has recently updated its Terms of Service. It has altered its usage policy to get rid of a prohibition on using its technology for "military and warfare." Previously, one of the leading tech companies prohibited its tools and services specifically to use for "weapons development, military and warfare," but this was in action only before . January 10 of this year OpenAI has now made changes to its policy to only disallow the use that would "bring harm to others," IT magazine Computer World first reported. "Our policy does not allow our tools to be used to harm people, develop weapons, for communications surveillance, or to injure others or destroy property," a spokesperson from OpenAI reportedly said. "There are, however, national security use cases that align with our mission. For example, we are already working with DARPA to spur the creation of new cybersecurity tools to secure open-source software that critical infrastructure and industry depend on. It was not clear whether these beneficial use cases would have been allowed under ‘military’ in our previous policies. So the goal with our policy update is to provide clarity and the ability to have these discussions," he added. This sudden change will now allow the company to work closely with the US military and possibly with the militaries of US allies. This will write a new chapter on the development of AI technologies and their potential uses in modern warfare and other military tools. There are also reports that certain people and employees are skeptical of this new development and the future of working closely with the military. Let's keep the negativities aside and discuss the positive aspects of using AI and robots to fight wars for us. Fights will be fair Future wars would be fought by robots. A nation can demonstrate its strength and technological superiority. There would be less chance of foul playing when an international convention is signed for the ethics of robot fighters in wars. There will be a significant decrease in human lives Hundreds of thousands of brave patriots lose their lives in battles. We need people like them. We need their dedication and their love for the motherland. Humans killing other humans over petty issues like land and resources sounds so ridiculous when you look at it from a general perspective. This is true we can save countless lives by using robots to fight for us. We don't even need to wage out large-scale battles when we can just make the best fighters from both the conflicting sides to fight a death match and decide the outcome in the favor of the winning side. Improved Accuracy and Remote Operations capabilities Artificial fighters can take into account the direction of the wind, air pressure, temperature, muzzle velocity, drag coefficient, and various other factors, which can improve accuracy significantly. They can also be used for remote operations and can be controlled from hundreds of kilometers away with state-of-the-art technologies. They can be replaced quickly, can respond rapidly in times of emergency, and most important of all, the humans will not feel the emotional impact of losing comrades, and the mothers and sisters will not shed any more tears. The sons and daughters will not feel alone anymore. That is enough talk of the future. Some experts had already anticipated the increase in interest of the Federal Government and the US military in AI technologies in response to the Chinese authorities facilitating AI technologies developments and incorporating them into the Chinese military. It is sad to see that the Government is prioritizing AI developments to compete with rivals when there is a strong need for an international convention to regulate this fascinating yet destructive technology. Also published . here

OPENAI

openai.com

1000 employees

Since 2015

COMPANY RANKING

OPENAI

EVERGREEN INDEX #7630

Representing this company? Click here to claim and customize this page!

OpenAI WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

OPENAI

openai.com

1000 employees

Since 2015

COMPANY RANKING

OPENAI

EVERGREEN INDEX #7630

RELATED COMPANIESRANK

STORIES ABOUT OPENAI

Representing this company? Click here to claim and customize this page!

OpenAI WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!