Agentero helps independent insurance agents grow their business by providing them with access to modern carriers and a technology platform to identify the needs of their customers, quote and bind

Agentero

This tutorial is part of this book and through it, we are going to discover how to use Rancher in order to create a scalable application. Wordpress Rancher Architecture In , everything (like containers, networks or images ) is an resource with a process lifecycle. Containers, images, networks, and accounts are all resources with their own process lifecycles. Rancher API API Rancher is built on the top of containers: A web UI An API A server that manages Rancher Agents A database A machine microservice The binary docker-machine Rancher Server Architecture When you run using the + + The Database + are processes that live inside this container. Rancher docker run rancher/server ... Rancher API Rancher Process Server Machine Microservice Note that the docker-machine binary is also living in the same container but only runs when it is called by the . API has also an Agent part that manages the life cycle of containers. Rancher If creates a machine successfully, some events are exchanged between the and the microservice. A event is created and a command is executed to get the details needed to connect to the machine’s Docker daemon. docker-machine docker-machine bootstrap docker-machine config If everything run without problems, the service fires up a on the machine via docker Rancher Agent run rancher/agent … open a connection to the server in order to establish a 2-way communication. The manage its containers and reports every change using the Docker . Rancher Agents WebSocket Rancher Agent API During this tutorial, we are going to use an machine and this is how a different view of the layers of our installation: EC2 Rancher RancherOS Architecture RancherOS is a small distribution released by team. It is an easy way to run containers at scale in production and includes only the services needed to run Docker. RancherOS Rancher It only includes the latest version of Docker and removes any unneeded library that a “normal” distribution could have. Linux In , everything is a container, the traditional system is replaced so that Docker run directly on the . RancherOS init Kernel A special component in this system is called which is the daemon that allow a user (a non-system user) runs its containers. User Docker RancherOS Architecture We are going to run in an machine using : RancherOS EC2 AWS CLI aws ec2 run-instances --image-id ami-ID --count 1 --instance-type t2.micro --key-name MySSHKeyName --security-groups sg-name This is the list of AMI by region: Now you can login to your machine using the common command : AWS ssh ssh -i "MySSHKeyName" rancher@xxxx.xx-xxxx-x.compute.amazonaws.com Running Rancher Since Docker is installed by default, we can start using it directly. Let’s start a server then use it with . MariaDB Rancher Server docker run --name mariadb -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=cattle -e MYSQL_USER=cattle -e MYSQL_PASSWORD=cattle -p 3306:3306 -d mariadb Run the and change 172.31.0.190 by your : Rancher Server MariaDB IP docker run --name rancher_server -p 8080:8080 -e CATTLE_DB_CATTLE_MYSQL_HOST=172.31.0.190 -e CATTLE_DB_CATTLE_MYSQL_PORT=3306 -e CATTLE_DB_CATTLE_MYSQL_NAME=cattle -e CATTLE_DB_CATTLE_USERNAME=cattle -e CATTLE_DB_CATTLE_PASSWORD=cattle -v /var/run/docker.sock:/var/run/docker.sock -d rancher/server Now you can go to your and add a host with a supported version of Docker: <host_ip>:8080 Linux Adding A Rancher Host We are not going to use the public address but the private ( ) one. IP eth0 Adding A Rancher Host Let’s add a custom host. A command is given to run on any reachable host in order to let it join the server.At this step, I already created an machines (with as an operating system) and I am going to use it and run this command: EC2 RancherOS sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.1 http://172.31.0.190:8080/v1/scripts/02048221239BE78134BB:1483142400000:i5yrQRE4kFo7eIKz9n1o5VFTew Make sure any or allow traffic from and to all other hosts on ports 500 and 4500. Security Groups firewalls UDP After the installation, we can notice that some containers are running in each machine like the , , ..: Agent Agent dns healthcheck network-manager CONTAINER ID        IMAGE                            COMMAND                  CREATED              STATUS              PORTS               NAMES8339a8af3bb8        rancher/net:v0.9.4               "/rancher-entrypoint."   About a minute ago   Up About a minute                       r-ipsec-ipsec-router-1-19bd25e395f3d3afdb7c        rancher/net:holder               "/.r/r /rancher-entry"   About a minute ago   Up About a minute                       r-ipsec-ipsec-1-90844ad45643089ca485        rancher/healthcheck:v0.2.3       "/.r/r /rancher-entry"   2 minutes ago        Up 2 minutes                            r-healthcheck-healthcheck-1-91ae658dfec564c53f13        rancher/dns:v0.14.1              "/rancher-entrypoint."   2 minutes ago        Up 2 minutes                            r-network-services-metadata-dns-1-471d9deac162c07d07f6        rancher/net:v0.9.4               "/rancher-entrypoint."   2 minutes ago        Up 2 minutes                            r-ipsec-ipsec-cni-driver-1-e02f6a010313ff5cb812        rancher/scheduler:v0.7.5         "/.r/r /rancher-entry"   2 minutes ago        Up 2 minutes                            r-scheduler-scheduler-1-5ca518560f6f62a190c7        rancher/network-manager:v0.5.3   "/rancher-entrypoint."   2 minutes ago        Up 2 minutes                            r-network-services-network-manager-1-03d3f619ec270864ff07        rancher/metadata:v0.8.11         "/rancher-entrypoint."   2 minutes ago        Up 2 minutes                            r-network-services-metadata-1-4be6ebc698a78058bc3c        rancher/agent:v1.2.1             "/run.sh run"            2 minutes ago        Up 2 minutes                            rancher-agent Running A Wordpress Service If everything is ok, we are going to create a service. Go to menu and click . Wordpress Stack User Create the database, name it (we are going to use it in order to link it to the container) MariaDB wordpressdb Wordpress Then add your environment variables: Now, using the same way, create the container, map host port 80 to the container port 80 and link the container as a instance. Wordpress mariadb mysql Then add your environment variables: At this step, we can check the running services, use -> then to see the containers: Stack User default What we started is the equivalent of 2 docker commands, one to start the container and the other one to start the container. mariadb wordpress You can visit the fresh installation using its address. Wordpress IP We can see the details of the different configurations a running container could have if you choose the container from the same view then use the “ ” menu. View in API This is an example of the container: wordpress {"id":"1s7","type":"service","links":{"self":"…/v2-beta/projects/1a5/services/1s7","account":"…/v2-beta/projects/1a5/services/1s7/account","consumedbyservices":"…/v2-beta/projects/1a5/services/1s7/consumedbyservices","consumedservices":"…/v2-beta/projects/1a5/services/1s7/consumedservices","instances":"…/v2-beta/projects/1a5/services/1s7/instances","networkDrivers":"…/v2-beta/projects/1a5/services/1s7/networkdrivers","serviceExposeMaps":"…/v2-beta/projects/1a5/services/1s7/serviceexposemaps","serviceLogs":"…/v2-beta/projects/1a5/services/1s7/servicelogs","stack":"…/v2-beta/projects/1a5/services/1s7/stack","storageDrivers":"…/v2-beta/projects/1a5/services/1s7/storagedrivers","containerStats":"…/v2-beta/projects/1a5/services/1s7/containerstats"},"actions":{"upgrade":"…/v2-beta/projects/1a5/services/1s7/?action=upgrade","restart":"…/v2-beta/projects/1a5/services/1s7/?action=restart","update":"…/v2-beta/projects/1a5/services/1s7/?action=update","remove":"…/v2-beta/projects/1a5/services/1s7/?action=remove","deactivate":"…/v2-beta/projects/1a5/services/1s7/?action=deactivate","removeservicelink":"…/v2-beta/projects/1a5/services/1s7/?action=removeservicelink","addservicelink":"…/v2-beta/projects/1a5/services/1s7/?action=addservicelink","setservicelinks":"…/v2-beta/projects/1a5/services/1s7/?action=setservicelinks"},"baseType":"service","name":"wordpress","state":"active","accountId":"1a5","assignServiceIpAddress":false,"createIndex":1,"created":"2017-04-04T21:05:53Z","createdTS":1491339953000,"currentScale":1,"description":"wordpress files","externalId":null,"fqdn":null,"healthState":"healthy","instanceIds":["1i14"],"kind":"service","launchConfig":{"type":"launchConfig","capAdd":[ ],"capDrop":[ ],"dataVolumes":[ ],"dataVolumesFrom":[ ],"devices":[ ],"dns":[ ],"dnsSearch":[ ],"environment":{"WORDPRESS_DB_NAME":"wordpress","WORDPRESS_DB_USER":"wordpress","WORDPRESS_DB_PASSWORD":"wordpress"},"imageUuid":"docker:wordpress","instanceTriggeredStop":"stop","kind":"container","labels":{"io.rancher.container.pull_image":"always"},"logConfig":{"type":"logConfig","config":{ },"driver":""},"networkMode":"managed","ports":["80:80/tcp"],"privileged":false,"publishAllPorts":false,"readOnly":false,"secrets":[ ],"startOnCreate":true,"stdinOpen":true,"system":false,"tty":true,"version":"0","dataVolumesFromLaunchConfigs":[ ],"vcpu":1},"lbConfig":null,"linkedServices":{"mysql":"1s6"},"metadata":null,"publicEndpoints":[{"type":"publicEndpoint","hostId":"1h1","instanceId":"1i14","ipAddress":"54.246.163.197","port":80,"serviceId":"1s7"}],"removed":null,"retainIp":null,"scale":1,"scalePolicy":null,"secondaryLaunchConfigs":[ ],"selectorContainer":null,"selectorLink":null,"stackId":"1st5","startOnCreate":true,"system":false,"transitioning":"no","transitioningMessage":null,"transitioningProgress":null,"upgrade":null,"uuid":"0b8a8290-0d10-47be-b182-45f1e57e80bc","vip":null} We can also inspect other services and hosts configurations in the same way. Cattle: The Rancher Container Orchestrator What we started below is powered by its own orchestration tool called . Rancher offers the possibility to use other orchestration tools like , or . Rancher Cattle Kubernetes Docker Swarm Mesos is a container orchestration and scheduling framework, in its beginning, it was designed as an extension to but since continues to develop, and started to diverge. is used extensively by itself to orchestrate infrastructure services as well as setting up, managing, and upgrading , , and clusters. Cattle Docker Swarm Docker Swarm Cattle Swarm Cattle Rancher Swarm Kubernetes Mesos application deployments are organized into which can be used to or stacks. A is a collection of and the latter is primarily a Docker image with its networking, scalability, storage, health checks, service discovery links, environment and all of the other configurations. Cattle stacks User Infrastructure Cattle Stack Services A could be a load balancer or an external service. A can be launched using a or file or just start containers like we did for the stack. Cattle Service Stack docker-compose.yml rancher-compose.yml Wordpress In addition to this, you can start several applications using an application catalog. More than 50 different apps are available in the app catalog. An application is defined by a and a file and can be deployed easily with the default configurations. docker-compose rancher-compose Let’s take the example of : Portainer Portainer is a lightweight management UI which allows you to easily manage your Docker host or Swarm cluster.Portainer is meant to be as simple to deploy as it is to use. It consists of a single container that can run on any Docker engine (Docker for Linux and Docker for Windows are supported).Portainer allows you to manage your Docker containers, images, volumes, networks and more ! It is compatible with the standalone Docker engine and with Docker Swarm. This is the file of : docker-compose.yml Portainer portainer:  labels:    io.rancher.sidekicks: ui    io.rancher.container.create_agent: true    io.rancher.container.agent.role: environment  image: rancher/portainer-agent:v0.1.0  volumes:  - /config

ui:  image: portainer/portainer:pr572  command: --no-auth --external-endpoints=/config/config.json --sync-interval=5s -p :80  volumes_from:  - portainer  net: container:portainer This is its file: rancher-compose.yml .catalog:  name: "Portainer"  version: "1.11.4"  description: Open-source lightweight management UI for a Docker host or Swarm cluster  minimum_rancher_version: v1.5.0-rc1 Starting Portainer Rancher offers the possibility to start infrastructure . If you go to -> , you can see a catalog of tools like , , ..etc Stacks Stacks Infrastructure Bind9 Portainer Rancher vxlan Infrastructure Catalog Scaling Wordpress Using Rancher Now we want to scale our frontal app in order to handle more traffic. In this case, you should click -> then the name of the (it should be the default one if you followed this tutorial as it is). Now click on and scale your app to 3 containers: Wordpress Stack User Stack Wordpress At this step, you should see how won’t scale the right way. Wordpress This is normal since all of our containers are mapped to the same port 80 (that can handle only one container). Wordpress In order to scale the right way, we should tell Rancher to create a load balancer service with auto-discovery and let choose the container port automatically. Wordpress Rancher Wordpress This part is detailed in . Painless Docker Book Connect Deeper This article is part of . Painless Docker Book: Unlock The Power Of Docker & Its Ecosystem is a practical guide to master and based . Painless Docker Docker its ecosystem on real-world examples Painless Docker tends to be a complete and detailed guide to create, deploy, optimize, secure, trace, debug, log, orchestrate & monitor Docker and Docker clusters. Through this book, you will learn how to use Docker in development and production environments and the DevOps pipeline between them in order to build modern microservices applications. If you resonated with this article, please subscribe to : An Online Community Of Diverse & Passionate DevOps, SysAdmins & Developers From All Over The World. DevOpsLinks You can find me on and you can also check my books: & . Twitter SaltStack For DevOps Painless Docker If you liked this post, please recommend and share it with your followers.

Creating A Scalable Application Using Docker + Rancher + RancherOs + Cattle

If you’ve ever worked with complex high-load cloud infrastructure in Microsoft Azure, you probably understand how important it is to have an appropriate monitoring system. Unfortunately, there aren’t always ready-made solutions that can guarantee the proper coverage of all required metrics and provide the option for extra customization. Eventually, if you find them, they will likely cost your company a fortune, particularly if you have a complex multicomponent infrastructure. First, here’s a few words about the company and why we’ve implemented this solution, which will give you a better understanding of what we’re dealing with. Wirex's entire platform is hosted in the Microsoft Azure cloud. We have a complex multicomponent infrastructure with thousands of transactions and a million events each day. As a fintech product with 3.5 million clients, our platform uses cryptocurrencies in addition to traditional currencies, so we can't allow customers to experience any difficulties during transactions or any delays when exchanging currencies. This imposes certain risks and challenges which means it requires specific monitoring systems that need to be more advanced and dynamic. What problems did we try to solve with the implementation of the monitoring system in Microsoft Azure? Here’s the list of our main targets: Multicomponent monitoring, including metric collection and coverage of the maximum possible number of components in our cloud. Scale Set monitoring (part of the Service Fabric cluster). Without properly monitoring Scale Sets, we can't get full tracking of our cluster. Monitoring of microservices. Our platform itself and our entire product consist of many separate services that are interconnected and work in a single cluster. Unfortunately, Microsoft Azure doesn't provide the ability to monitor microservices. Continuous monitoring deployment. We want to simplify our monitoring system's deployment process, as the monitoring infrastructure's failing would lead to blind spots. Continuous monitoring upgrade. We want to be able to release updates as quickly as possible, often in less than a minute; that's why it should be continuous. Alerting. Our goal is to use alerts to cover every Scale Set instance and each service. For example, we want to have thresholds set up on 90% of using compute resources; in this case, we want to determine the particular microservices on a development stage and react immediately if there begins to be a high consumption. We can't afford this service to be deployed to the production because it will significantly impact the product. Before going any further, let's take a brief look at what we’re dealing with: More than 150 Virtual Machines Scale Sets (This is not just a Virtual Machine, this is a set of instances, which, in our case, can be up to 25 Virtual Machines) Service Fabriс clusters include about 12 active environments on which development is carried out, and this includes production. There are about 50 applications with up to 370 microservices in each environment. Our goal is to monitor every single Virtual Machine instance, each microservice, and receive metrics for every single environment. What tools can be used for infrastructure monitoring in Microsoft Azure? Let's start at the beginning with tools Microsoft offers from the box. Azure has a standard , a great instrument that allows you to get different insights on various services such as Key Vaults, Storage Accounts, VM, Service Health Statuses among all resources. This tool also gives the ability to set up auto-scale for scaling resources, which is really useful to have all in one place. What’s more, you can configure alerts. Azure Monitor has convenient settings for sending notifications regarding service health statuses to any preferred messenger. An added bonus is that it can be easily integrated with our 3rd party products. Azure Monitor Another essential tool from Microsoft is , and I think it’s worth mentioning as it's a unique tool with a unique script language. It provides a large number of modules to work with Azure, and you can even cover aspects and metrics of resources that are sometimes hidden and hard to find, such as various different customization tools. PowerShell Of course, Microsoft provides a lot more, but we are not going to cover them all. The main point to understand is that we’re still not able to get all the information we need. In addition to Microsoft tools, we have third-party products that we use to monitor our infrastructure. . This is our main tool for visualization. It provides dashboards that display the monitoring of specific resources. Grafana . We'll talk about this product later. Prometheus is used for log analytics and for working with Big Data on applications. Splunk provides custom scripts for alerting. PowerShell . With the help of Grafana, we can partially get metrics using this integration. Azure Monitoring Integration is a stack for additional integrations and manipulations. ELK What metrics did we collect when implementing our additional monitoring system? First of all, we had all of our Application Insights metrics in one place. Azure cache, databases, Virtual Machines, Application Services, Key Vaults, Storage Accounts, and many more. Below is just a brief look at what we have visualized and customized in Grafana. Unfortunately, we were unable to display information on Scale Sets and microservices and this information is not only fundamental but it’s also a goal target of our cluster. The issue is that the integration with Azure Monitor does not provide metrics on output from Virtual Machine Scale Sets. We also didn't know what was happening with our microservices on each instance; so, we tried to solve this problem. What kind of solution did we come up with? Our first answer is . It's an Open Source product, and, in the world of Open Source projects, it’s quite popular in the monitoring industry, almost as popular as  Kubernetes is for cluster orchestration. Prometheus How does the Prometheus work? According to the official documentation, Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts. or other API consumers can then be used to visualize the collected data. Grafana Source: https://prometheus.io/docs/introduction/overview/ What’s useful is that Prometheus includes a local on-disk time-series database and optionally integrates with remote storage systems. It might sound like a disadvantage initially, but in general, it’s good to have the extra security in regards to the setup process, as it means there will be fewer points for database failure as well as a direct read/write operation. Of course, a big plus is that Prometheus can discover Scale Sets automatically with Azure configurations. Why is this a plus for us? Usually, when you write a Prometheus configuration, you have to set each target individually, using the target IP address with its name. But when you have an infrastructure with more than 150 Virtual Machine Scale Sets with more than 20 instances in each set, that's not so convenient. The question is how to discover each Scale Set, especially the Scale Sets that appear after auto-scaling? Furthermore, would you have the ability to monitor these new, extra Scale Set? The answer is “yes” because it will be automatically discovered, and named as it is in Grafana, which is extremely useful. Prometheus also has a customization tool called PromQL, which allows you to customize your queries and get the information that you need. So, here’s the list of main Prometheus benefits: It’s an Open Source system monitoring and alerting toolkit. It supports fast and reliable deployment through containerized infrastructure. Azure SD configurations allow you to discover and retrieve scrape targets from Azure. It includes a local on-disk time-series database. Grafana supports querying Prometheus (PromQL). How does Prometheus know what’s happening in each particular instance? And how do we connect Prometheus with our instances? The answer is with Windows Exporter. Windows Exporter is an extra tool developed by the Prometheus community. It's deployed on each instance of the Virtual Machine Scale Set. And this service collects metrics that were previously set in a pre-deploy. Here's a brief view of available collectors. The most important thing here is that we’re able to display information on processes. Our microservice exists on an instance as a process, so getting metrics for processes allows us to monitor them. We get information specifically for services such as its CPU time, working set, virtual bites, i/o, display of when the process was started, etc. Also, if you have extra deployed services on your instance that need to be monitored, this isn’t a problem. How Windows Exporter and configuration can be deployed in every instance In these situations, we use Terraform, DSC configurations, and Chocolatey. Terraform is our main "infrastructure as a code" tool for managing our infrastructure's deployment in the cloud. DSC is a powerful PowerShell extension and a very flexible solution from Microsoft that allows you to manage server configurations. Chocolatey or Choco is probably the most popular package manager for downloading additional applications for Windows. We deploy DSC configuration with the described procedure to install an exporter as a Virtual Machine extension in Azure. What do these tools give us? It allows us to manage the infrastructure and control our resources. We can deploy our exporter on every instance and configure continuous configuration deployment. It’s a good advantage as we get rid of all drift configurations, so all our instances are consistent with each other, and that's really remarkable. How can we visualize the monitoring process? Here is a monitoring scheme. On the left, you can see a Service Fabric cluster which consists of several Scale Sets; it could even be adopted for Kubernetes. Each Scale Set has its own installed Exporter and, of course, deployed application microservices. Then we have our Prometheus with TSDB and Grafana, which use Prometheus as a data source. What does Exporter do? It runs on localhost on a specific port (9128, by default) and collects all predefined metrics, and posts them. Prometheus then scrapes metrics with configured scrape intervals through HTTP, which could be 5-10-15 seconds, so you can get instant updates. For example, you could configure your entire monitoring infrastructure as a separate local network and whitelist required connections through firewall rules without exposing resources to the public. Lastly, you need to configure Grafana to use Prometheus as a data source and customize output through PromQL in order to display all the data on the beautiful dashboards. How to automate deployment? Here’s how we configured the monitoring system in Continuous mode. We have a BitBucket repository with Ansible playbooks, which describes the installation process and configuration for Prometheus. We also have TeamCity as our main Continuous Integration (CI) tool that we use for our deployment project builds, allowing them to deploy and update. We have Ansible as a configuration management tool for the deployment of Prometheus of Grafana as Docker containers. Here, Ansible could be a separate instance, or it could be configured as a TeamCity runner; either option is possible. If you select a runner, it means it will be adapted as a TeamCity plugin. How does the pipeline work? It’s actually quite simple. TeamCity pulls configuration for our Prometheus configuration and then it triggers Ansible, which is deploying Prometheus as a Docker container. All this running through TeamCity agents. This is how you can implement your automation pipeline. It could take less than 20 seconds to deploy a new container or update it with our pre-deployed configuration. What have we got as a result? Thanks to Grafana and Prometheus, we've got all the metrics we need, and more for all our Virtual Machine Scale Sets instances, plus we’ve got information for the instances themselves and for each microservice in every instance. We monitor about 370 microservices in every environment, from development to production and we’ve got metrics for all Scale Sets. We have a better view of what is happening on production, and, as a bonus, we understand what's going on with each service on development before it’s deployed on the production. This means we can be notified and react quickly. We’re not worried about targeting and auto-naming if auto-scaling goes up or down on our Virtual Machine Scale Sets, we’re still able to monitor each service and instance. Also, we've got configured thresholds alerts, and we are able to receive alerts for a huge number of different events. We also get great customization options. Grafana provides us with the ability to build dashboards as we want, writing the queries to receive all the needed metrics. And thanks to these tools, we can continuously deploy, re-work, re-design and re-upgrade our infrastructure.

Implementing MS Azure Monitoring System for Resources and Microservices

In 2024, there is no shortage of LLM-based applications. Major tech companies like Microsoft and Google are pushing forward ever-more-powerful versions of their flagship ChatGPT and Gemini models, and specialist players like Anthropic are pushing forward competing offerings with additional integrations. In the applied LLM world, companies and even governments are experimenting with chat applications for a variety of contexts. Despite the enormous entrepreneurial energy poured into LLMs, most high-profile applications are still limited by their focus on chat-like interfaces that take in and return human-readable text. This is understandable - after all, one of the most exciting developments of LLMs is the ability to comprehend and generate human language, allowing for a conversational user interface. However, chat interfaces overlook another very important use case for LLMs - text processing as an intermediate step in a larger data pipeline. Today, we will explore this use case and see how LLMs can be useful as part of a data pipeline, not just as a natural language generator. Function Calling LLMs In order to use LLMs as part of a data pipeline, we need to change the output format of the LLM - instead of generating paragraphs for humans to read, the model has to generate something that computers can read. Usually, this means structured output in a data format like JSON or instructions in  a programming language like Python or SQL. These output formats are a lot less forgiving than natural language, since a missing quotation mark or bracket can crash the entire pipeline. Therefore, we will need to rely on specialized methods and features that have been developed to support this use case. These features are collectively referred to as function calling, since the output tends to call functions or be consumed in function calls. Before we delve into these methods, though, let’s first explore a little more in depth why function calling features were developed in the first place. Use Cases Connecting to external services The original and most straightforward use case for function calling is connecting to external services. When OpenAI originally released GPT-4, it also updated the API with a feature that allowed the user to provide the model a set of python-style functions that the model could call to accomplish its goals. The model could, for example, be told that it can call an external function to calculate the price of a complex security. Using this information, the model would be able to write code that calculates the value of a portfolio made up of these securities, without having to possess specialized knowledge on securities pricing. Calling python-style functions is only the tip of the iceberg. Once the importance of function-calling was proven in the market, OpenAI and other LLM providers started to support other output formats, such as JSON or SQL. The important thing was that these models were outputting machine-readable outputs, which could be reliably read by other processes. Astute readers might recognize that we have done something similar in previous articles, using LLM to generate , , or . In these , we accomplished this using prompt engineering and output-checking, but now that function-calling is a widely-supported feature, we can do this more easily by relying on these model-level features. training data SQL JSON articles Chaining LLMs Function calling created new possibilities for multi-LLM applications, and soon developers were experimenting with chaining LLMs to create sophisticated systems. Some of these systems became known as , which could independently search the web, gather new data, and call another LLM with the new information. These pipelines are capable of surprising amounts of autonomy and are capable of solving sophisticated problems with very little input, though there are still limitations, such as API costs and behavioral safeguards, that prevent agents from gaining widespread adoption. agents Using LLM as input and intermediate processing Another use case for function-calling LLMs is input and intermediate data processing. LLMs can be used to parse unstructured inputs into structured data that can be used for downstream processing. While many of these tasks tend to be relegated to “traditional” NLP technologies, the flexible nature of transformer-based models means that a specifically trained model can perform much better on these tasks than other NLP technologies. As a result, many developers are trying to leverage these specialized models in their data pipelines. Now that we’re familiar with the theory behind function-calling, let’s look at the application we will be building today. Demo Application CB Insights Entity Relationship Analyzer In a , I created a simple RAG application to answer natural language questions about CB Insights popular newsletter. In today’s article, we will build a similar question-answering application, but instead of relying on embedding search and GPT3.5, we will be using entity recognition as the primary indexing method. Additionally, we will use DRAGON as the summarization engine so we can run the entire application on our laptop, no cloud services needed. previous article Why Use Entity Recognition? Before we delve into the implementation, let’s first explore the advantages of using NER as the retrieval technology instead of embedding search. While in production we will want to use each technique according to the demands of the specific situation, NER offers several advantages over a system that relies on embedding search. Debuggability: Because entity recognition is an easily verifiable task, the entire pipeline is much more debuggable. It is easy to check if the model correctly identifies all of the entities, and it is easy to build and improve a matching algorithm based on these entities. Comparatively, it is much more difficult to verify that the embedding algorithm is properly identifying similarities and differences in passages. Flexibility: By separating recognition and filtering, we make the pipeline much more flexible than an embedding-based search algorithm. We can add additional metadata, change search algorithms based on entity type, and even use embedding algorithms on top of the NER-based search results. This allows for much more powerful features to be developed based on combinations of technologies. For example, an NER-based pipeline can use entity-based search to narrow down a set of documents, then use an embedding algorithm to narrow down further within the search space. This tends to make the search much faster and more efficient. Identification Power: Due to the level of manual control that is possible, NER pipelines are much more suited for certain search and retrieval tasks. Embedding searches are based on the proximity of passages within the training data, which calculates the probability that two documents come from the same larger document. In some use cases, this will cause embedding search to miss important documents. For example, if we were trying to understand Microsoft’s AI strategy, we might not only want to retrieve information about Microsoft’s activities within the AI software space, but also information about investments in chipmaking, data centers, and robotics. A general purpose embedding algorithm will not be able to identify these connections, whereas an NER-based algorithm will be able to draw these connections. Now that we understand the benefits of using NER within a RAG pipeline, let’s get a deeper look at the specific technologies we’re using in our application. Technologies Used LLM Entity Recognition Entity Recognition is a “traditional” NLP technology for extracting structured data from unstructured language data. The extracted data can then be used in down-line processing or as metadata for the passage being analyzed. Historically, this was accomplished using small, purpose-built machine learning algorithms that first tags the parts of speech, followed by a second pass that determines if the proper nouns identified are named entities. With function-calling LLMs, we can accomplish the same task, with several added advantages. Because transformer architecture is able to understand the language structure better than smaller NLP models, the performance is likely to be more robust, for example in cases where the passage is improperly formatted. Because LLMs are being updated with the most recent training data, the entity recognition system is likely to stay more up-to-date than other open-source entity recognition models. Prompt engineering techniques can be used to embed additional data and give additional instructions, allowing for more flexible behavior even in off-the-shelf models. We have covered Dragon in a previous article where we highlighted its impressive performance in summarizing natural-language data. We will be using Dragon to perform the final step of analysis, where we will summarize all of the articles relevant to the selected entities. DRAGON SLIMs are a new family of miniaturized, locally-runnable models from - it’s no secret I am a fan of their miniaturization work - that specializes in function calling. These models have been fine-tuned specifically to generate outputs that can be interpreted by machines, allowing users to take advantage of the recent developments in function calling architecture without having to rely on external APIs. SLIMs LLMWare Today we will be using the SLIMs-NER model, which will perform the Named Entity Recognition on the newsletter articles. In addition to NER, there are also SLIMs models for sentiment analysis, SQL generation, and multi-step agents. Now we understand the technology, let’s implement the application! Implementation Data Downloading and Processing Let’s begin by downloading the CB Insights articles. Let’s import the dependencies: import requests
from bs4 import BeautifulSoup
import os
import pandas as pd
import json
import re And now the code to download the newsletter archive: res = requests.get('https://www.cbinsights.com/newsletter/')

soup = BeautifulSoup(res.text)
article_links = [[i.text, i['href']] for i in soup.find_all('a') if 'campaign-archive' in i['href'] ]
article_soups = [BeautifulSoup(requests.get(link).text) for title, link in article_links] Now that we have downloaded the newsletter archive, let’s process it into JSON format result_json = {}
for soup_meta, soup_art in zip(article_links, article_soups):
    article_tables = []
    cur_article = []
    for table in soup_art.find_all('table'):
        if table.attrs.get('mc:variant') == 'Section_Divider':
            article_tables.append(get_deduped_article_tables(cur_article))
            cur_article = []
        else:
            cur_article.append(table.text)
    article_tables.append(get_deduped_article_tables(cur_article))
    result_json[soup_meta[0]] = article_tables

articles_with_meta = []
for name, tables in result_json.items():
      print(name, pd.to_datetime(tables[0][1].strip()))
      articles_with_meta.append({
            'name': name, 'date':  pd.to_datetime(tables[0][1].strip()).strftime('%Y-%m-%d'), 'tables': tables
      })
df = pd.DataFrame(articles_with_meta) Now we are at the same place we were at in the previous article. Instead of going right to the RAG construction by creating embeddings, however, we will instead run an entity-recognition step. First, let’s import the dependencies for running SLIMs, a new set of models from LLMWare that allows us to perform entity recognition locally on our machine: from llmware.agents import LLMfx
from llmware.parsers import WikiParser
from collections import defaultdict

# define a function for identifying all the named entities
def run_ner(text):
    agent = LLMfx()
    agent.load_work(text)
    agent.load_tool("ner")
    named_entities = agent.ner()
    ner_dict= named_entities["llm_response"]
    return ner_dict Now we can run the named entity recognition on all of the articles: date_entities = defaultdict(dict)
for _, row in df.iterrows():
    for idx, t in enumerate(row['tables'][1:]):
        if 'Start Your Free Trial' in t:
            t = t[:t.index('Start Your Free Trial')]
        date_entities[row['date']][idx] = run_ner('\n'.join(t)) It might take a few minutes for the NER pipeline to complete, but this is all you need to recognize entities using a state-of-the-art miniaturized LLM. You can check the date_entities dictionary by printing some test items. For example, the code: date_entities[list(date_entities.keys())[0]] Should produce the output below: {0: {'people': ['Yahoo!'],
  'place': [],
  'company': ['Databricks', 'MosaicML'],
  'misc': []},
 1: {'people': [],
  'place': ['New York'],
  'company': ['CB Insights'],
  'misc': []}} Which shows all of the various entities that have been detected by the SLIMs model. Question-Answering with DRAGON Now that the entities have been detected, let’s build a question-answering workflow that demonstrates the power of this technique. For our example, we will use the test question: What role does OpenAI play in Microsoft's AI strategy? Let’s begin by importing the proper packages for running DRAGON: from llmware.prompts import Prompt
query = "What role does OpenAI play in Microsoft's AI strategy?"

model_name = "llmware/dragon-llama-7b-gguf"
prompter = Prompt().load_model(model_name) Now, we can build the function that answers the question using entity recognition. In order to leverage our NER data, we will want to implement the following workflow: Identify the entities in the user query. We can use the same SLIMs function we used in the previous step. Perform a lookup based on the results of the lookup, that is, we search for articles in the corpus that talk about the same entities. Once we find these articles, we can supply them to the question-answering model to extract the relevant information. Here is what the workflow looks like in code form: def answer_question_with_ner(query):
    ner_results = run_ner(query) # run NER on the user query
    search_entities = []
    for ent_type, ent_list in ner_results.items():
        search_entities.extend(ent_list)
    # create a list of entities to search for
    search_entities = list(set(search_entities))

    # now perform a lookup for articles that mention the identified entities, using set arithmetic. 
    articles = set(entity_to_place_map[search_entities[0]])
    for se in search_entities:
        articles &= set(entity_to_place_map[se])

    # now process the corpus into a prompt and feed it to the question-answering LLM.
    article_content = []
    for article in articles:
        article_content.extend(df[df['date'] == article[0]].iloc[0]['tables'][article[1]+1])
    response = prompter.prompt_main(query, 
        context='\n'.join(article_content),
        prompt_name="default_with_context", temperature=0.3)
    return response # return the response. Run the function, with the query from the first code block, and you should see the following result: Microsoft has poured billions of dollars into ChatGPT developer OpenAI. However, it's also placed a number of bets on other GenAI startups across computing, mapping, and gaming. You can also see in the “evidence” section of the response object that, indeed the two articles from the CB Insights archive that mention Microsoft and OpenAI have been retrieved, and that the LLM’s answer is based directly on the evidence. Note that because we are doing explicit entity recognition, the retrieval process is extremely transparent and debuggable. Should a question ever arise as to why the model retrieved these specific pieces of information from the corpus - all it takes is a simple print statement to see that the model chose the article because the query mentions “Microsoft” and “OpenAI”, and the two retrieved newsletter sections are the only sections that mention both entities. Additionally, when compared with embedding-based vector search, the NER lookup method provides much more precise answers. In my tests, the selected articles were in the top 10 most relevant articles ranked by OpenAI’s ada algorithm, but was not the one identified as having the closest proximity to the question at hand. Therefore, it is possible an application using embedding search may not have correctly answered the question at all, and the lack of debuggability would only contribute to the confusion. And with that, we have built a question-answering application using NER lookups, improving the quality of the pipeline in the process! Conclusion Today, we built an application using function calling, an exciting new LLM feature. Miniaturized, locally-runnable function-calling models is a revolutionary development that unlocks a new class of AI applications, and we are only seeing the first iteration of these technologies. It will be exciting to see what applications developers will create with these technologies in the coming months. If you have an idea in the AI space that you want to see implemented or simply want to chat about technology, please do not hesitate to reach out on or . Github LinkedIn If you want to learn more about LLMWare, the company behind SLIMs and DRAGON, you can find them on or . HugginFace Github

Function Calling LLMs: Combining Multiple Models for Better RAG Performance

ফাংশন কলিং এলএলএম: আরও ভাল RAG পারফরম্যান্সের জন্য স্লিম এবং ড্রাগনকে একত্রিত করা

Funktionsaufruf-LLMs: Kombination von SLIMs und DRAGON für eine bessere RAG-Leistung

LLM de llamadas a funciones: combinación de SLIM y DRAGON para un mejor rendimiento de RAG

Appel de fonctions LLM : combiner SLIM et DRAGON pour de meilleures performances RAG

फंक्शन कॉलिंग एलएलएम: बेहतर आरएजी प्रदर्शन के लिए एसएलआईएम और ड्रैगन का संयोजन

関数呼び出し LLM: SLIM と DRAGON を組み合わせて RAG パフォーマンスを向上させる

함수 호출 LLM: 더 나은 RAG 성능을 위해 SLIM과 DRAGON 결합

LLMs de chamada de função: combinando SLIMs e DRAGON para melhor desempenho de RAG

Вызов функций LLM: объединение SLIM и DRAGON для повышения производительности RAG

İşlev Çağırma Yüksek Lisansı: Daha İyi RAG Performansı için SLIM'leri ve DRAGON'u Birleştirme

Gọi chức năng LLM: Kết hợp SLIM và DRAGON để có hiệu suất RAG tốt hơn

函数调用 LLM：结合 SLIM 和 DRAGON 实现更好的 RAG 性能

Playing Checkers With Genetic Algorithms: Evolutionary Learning in Traditional Games

জেনেটিক অ্যালগরিদম দিয়ে চেকার খেলা: ঐতিহ্যগত গেমে বিবর্তনীয় শিক্ষা

Dame spielen mit genetischen Algorithmen: Evolutionäres Lernen in traditionellen Spielen

Jugar a las damas con algoritmos genéticos: aprendizaje evolutivo en juegos tradicionales

Jouer aux dames avec des algorithmes génétiques : apprentissage évolutif dans les jeux traditionnels

जेनेटिक एल्गोरिदम के साथ चेकर्स खेलना: पारंपरिक खेलों में विकासवादी शिक्षा

遺伝的アルゴリズムを使用したチェッカーのプレイ: 従来のゲームにおける進化的学習

유전 알고리즘으로 체커 게임하기: 전통 게임의 진화 학습

Jogando Damas com Algoritmos Genéticos: Aprendizagem Evolutiva em Jogos Tradicionais

Игра в шашки с использованием генетических алгоритмов: эволюционное обучение в традиционных играх

Genetik Algoritmalarla Dama Oynamak: Geleneksel Oyunlarda Evrimsel Öğrenme

Chơi cờ đam bằng thuật toán di truyền: Học tập tiến hóa trong các trò chơi truyền thống

用遗传算法玩西洋跳棋：传统游戏中的进化学习

Are we on the brink of debunking Moravec’s Paradox? In the 1980s, AI and robotics researcher Hans Moravec highlighted a counterintuitive aspect of AI: tasks requiring high-level reasoning — like chess or Go — are easier for to master than basic sensory and motor skills — such as walking or identifying your mom’s face — which humans find instinctive. Adding complexity, these “simpler” skills actually demand much more computational power. This insight sheds light on the complexity of replicating human-like perception and dexterity, outcomes of millions of years of evolution, as opposed to logical reasoning, a more recent development. In today’s AI and ML landscape, this paradox underscores the challenges in creating robots and AI systems capable of seamlessly navigating and interacting with the physical world. AI However, last week, Bernt Bornich, CEO and founder of 1x, a humanoid robotics company, , “ .” I suspect that this has something to do with the advancements in foundation models. Originally known for their ability to perform a wide range of tasks based on a single type of data (like text for language models), these models become “multimodal” when integrating and interpreting information across different sensory inputs, closely mirroring human-like understanding. wrote New progress update on the droids dropping in 4 weeks, looks like Moravec’s paradox might be debunked, and we just didn’t have the data Could the embodiment of AI, with all its sensory inputs, plus reasoning-imitation algorithms like LLMs, be the pool of data that disproves Moravec’s paradox? Another intriguing development caught my attention. Huang Jensen, Nvidia’s CEO, to a question from Wired about what current development could change everything. Jensen replied, “ ” responded There are a couple of things. One doesn’t really have a name, but it’s part of the work we’re doing in foundational robotics. If you can generate text and images, can you also generate motion? The answer is probably yes. And if you can generate motion, you can understand the intent and generate a generalized version of articulation. Therefore, humanoid robotics should be right around the corner. Something to observe in the coming weeks! In related news from the robotics universe, Figure AI, a humanoid robotics startup, made headlines by raising approximately in funding. What’s more impressive is the list of backers: Amazon, NVIDIA, Microsoft, OpenAI, Intel, LG, and Samsung. This indicates a strong belief in the potential of humanoid robotics to disrupt various sectors. $675 million Yet, there are skeptical voices. Rodney Brooks, who coined Nouvelle AI*, last week: “ ” posted Tele-op robots presented as autonomous, like the Tesla Optimus humanoid folding a shirt, and 1X humanoid robots, are misrepresentations of what robots are actually doing, which can also be called LIES. Note that the Stanford robot cooking and cleaning videos are also tele-operated. If 2023 was the year of LLMs, are we ready to evolve to an embodied AI and make 2024 the year of robots? 🎁 Bonus: The freshest research papers from the week of Feb 19 — Feb 25 Enhancing Large Language Models (LLMs) : Expands the processing capability of LLMs to handle over 2 million tokens, pushing the boundaries of context window sizes for more comprehensive understanding and generation tasks. LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Read the paper. : Demonstrates the versatility of LLMs in performing numerical regression tasks, suggesting their potential as universal tools for predictive modeling across a variety of domains. . OmniPred: Language Models as Universal Regressors Read the paper : Investigates efficient strategies for distilling large models into smaller, more manageable ones, particularly for reasoning tasks, emphasizing the importance of decomposition over problem-solving. . Divide-or-Conquer? Which Part Should You Distill Your LLM? Read the paper : Proposes a method to enhance the efficiency of self-attention mechanisms in LLMs, crucial for improving performance and reducing resource consumption. . ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition Read the paper : Introduces a framework for personalizing LLM interactions using user embeddings, enhancing the model’s responsiveness to individual user preferences and histories. USER-LLM: Efficient LLM Contextualization with User Embeddings Multimodal and Multi-Agent Systems : Explores integrating video and language for advanced AI understanding and interaction, leveraging a novel RingAttention mechanism for efficient multimodal learning. . World Model on Million-Length Video and Language with RingAttention Read the paper : Develops a multi-agent platform that enhances cooperation and flexibility among agents, addressing the complexity of multi-agent systems and their practical applications. . AgentScope: A Flexible yet Robust Multi-Agent Platform Read the paper : Focuses on the design and analysis of small-scale multimodal models, proving that with strategic optimizations, smaller models can achieve or surpass the performance of larger counterparts. . TinyLLaVA: A Framework of Small-scale Large Multimodal Models Read the paper : Introduces a versatile multimodal language model that processes speech, text, images, and music, demonstrating the power of discrete representations in unifying various data modalities within a single framework. This approach simplifies the integration of new modalities without needing to modify the underlying architecture or training methodologies. AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling read the paper : Presents a novel dataset that enhances multimodal understanding by incorporating touch with vision and language, aiming to advance touch-vision-language alignment and understanding through a tactile encoder and text generation model. A Touch, Vision, and Language Dataset for Multimodal Alignment read the paper Advancements in Specific Domains : Introduces a novel object detection model that leverages programmable gradient information for enhanced accuracy and efficiency in learning. . YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information Read the paper : Demonstrates the application of Transformers in complex planning tasks, offering a method that surpasses traditional search algorithms in efficiency and effectiveness. . : Better Planning with Transformers via Search Dynamics Bootstrapping* Beyond A Read the paper Developer Tools and APIs : Presents a vast dataset designed for training LLMs to interact with APIs, addressing the challenge of creating effective models for API usage and integration. . API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs Read the paper : Develops an open-source system for code generation, execution, and refinement, facilitated by a dataset of multi-turn interactions, aiming to bridge the gap between code generation models and practical coding tasks. . OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement Read the paper Security and Adversarial Research : Explores the susceptibility of LLMs to a wide range of adversarial attacks, highlighting the need for comprehensive security measures to protect against unintended behaviors and data extraction. . Coercing LLMs to do and reveal (almost) anything Read the paper Model Efficiency and Quantization : Discusses a novel framework for quantizing LLM weight matrices to 1-bit to drastically reduce storage and computational demands while maintaining performance, enabling efficient deployment of LLMs on resource-constrained devices. OneBit: Towards Extremely Low-bit Large Language Models read the paper Instruction Tuning and Data Quality : Introduces REALIGN, a method for refining instruction data quality for LLMs to better align with human values, emphasizing the importance of instruction data quality in model alignment and suggesting areas for further exploration in LLM science. Reformatted Alignment read the paper : Proposes a novel method for instruction tuning that generates synthetic instruction data across all disciplines, showcasing a scalable and customizable approach to instruction tuning without relying on specific training data. Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models read the paper : Proposes a pre-instruction-tuning method to enhance LLMs’ knowledge updating capabilities, demonstrating significant improvements in factual knowledge absorption and cross-domain generalization. Instruction-tuned Language Models are Better Knowledge Learners read the paper *Also published . * here

Is 2024 The Year of Advanced Robotics?

It was Friday evening and I needed to make myself lunch after work. I found myself staring into my fridge wondering what to cook. I had leftovers from last night, however, I needed to make something fresh and exciting. I thought to myself, why not build a personal AI kitchen assistant to analyze the ingredients I have and suggests delicious dishes tailored precisely to my pantry? This is where Large Language Models (LLMs) stepped into the kitchen with me. LLMs are a type of artificial intelligence that have been trained on a massive dataset of text (and code), including a vast trove of recipes; meaning that they can do so much more than just finding a suitable recipe. They can become your culinary genius on demand. literally So, how does it all come together? In this article, I'll detail how I: Developed an LLM app utilizing GPT-3.5 turbo LLM to assess available ingredients and recommend recipes Deploy the LLM app on for production, enabling integration into a user interface for seamless communication via my mobile device Agenta Developing The LLM App I’ll be using OpenAI gpt-3.5-turbo in this application, however, I’d also be including various directives of the turbo model and also gpt-4. Use whichever model you think is appropriate for the task. Additionally, I will make use of to deploy my LLM application, as it is my preferred open-source platform for creating robust AI applications with LLMs. Agenta provides the necessary tools for managing and evaluating prompts. Agenta Here are the capabilities of Agenta: Quickly experiment with and compare prompts in any LLM workflow (chain-of-prompts, Retrieval Augmented Generation (RAG), LLM agents...) Swiftly generate test sets and golden datasets for evaluation Evaluate your application using pre-existing or custom evaluators Annotate and A/B test your applications with human feedback Collaborate with product teams on prompt engineering and evaluation Deploy your application with a single click through the UI, CLI, or github workflows By using Agenta, I am able to concentrate on constructing my LLM app and streamline the entire development cycle of the application, thereby enhancing the speed of experimentation. I’d be using Python to build the kitchen assistant. Creating LLM app directory and dependency file Create a directory named and a file named and add the following packages inside the file: and kitchen_assistant requirements.txt agenta openai If you have something like the image attached below, you’re good to go. Environment variable & LLM app development Now that you have the assistant folder and have included the packages inside the file, proceed to create a new file named and include your OPENAI secret key to make the LLM app work. requirements.txt .env Ensure that the file is inside the folder. You should have something like this: .env kitchen_assistant OPENAI_API_KEY=sk-vxxxxxxxxxxxxx # replace with your secret key Next would be to create a file named , and copy the following code into it. I’ll explain what the code is doing in a minute. app.py # Stdlib Imports
from typing import Dict, Any

# Third Party Imports
import agenta as ag # type: ignore
from openai import AsyncOpenAI


SYSTEM_PROMPT = "You are my AI kitchen assistant. Help me find creative recipes based on my ingredients, provide step-by-step cooking instructions, and advise me on food safety practices."
CHAT_LLM_GPT = [
    "gpt-3.5-turbo-16k",
    "gpt-3.5-turbo-0301",
    "gpt-3.5-turbo-0613",
    "gpt-3.5-turbo-16k-0613",
    "gpt-4",
]

# Initialize agenta sdk and update the configuration parameters
ag.init()
ag.config.default(
    temperature=ag.FloatParam(0.2),
    max_tokens=ag.IntParam(-1, -1, 4000),
    prompt_system=ag.TextParam(SYSTEM_PROMPT),
    model=ag.MultipleChoiceParam("gpt-3.5-turbo", CHAT_LLM_GPT),
)

# Initialize async openai sdk
client = AsyncOpenAI()


@ag.entrypoint
async def suggest(inputs: ag.MessagesInput = ag.MessagesInput()) -> Dict[str, Any]:
    """
    Suggest  delicious dish tailored precisely based on the ingredients (inputs).

    Args:
        inputs (ag.MessagesInput): messages input for chat completion

    Returns:
        dict: The llm response (along with additional information).
    """

    messages = [{"role": "system", "content": ag.config.prompt_system}] + inputs
    max_tokens = ag.config.max_tokens if ag.config.max_tokens != -1 else None
    chat_completion = await client.chat.completions.create(
        model=ag.config.model,
        messages=messages,
        temperature=ag.config.temperature,
        max_tokens=max_tokens,
    )
    token_usage = chat_completion.usage.dict() # type: ignore
    return {
        "message": chat_completion.choices[0].message.content,
        **{"usage": token_usage},
        "cost": ag.calculate_token_usage(ag.config.model, token_usage),
    } The imports are essentially importing the required modules that are needed to ensure the code works correctly. System prompt sets the context for every conversation that happens between the user (you and I) and the LLM application. You’d be pissed at your friend who in the middle of a finance conversation with you starts talking about something unrelated, right? Imagine a Large Language Model (LLM) as a super-powered reader. To really grasp what it's reading, it needs context – the words and sentences surrounding whatever it's focused on. This 'context window' is like a spotlight: the bigger the window, the more information the LLM can use to understand and generate meaningful responses. Next to the system prompt is a list of defined models that can be used when testing the user prompt for the LLM app in Agenta. You get to choose whichever model to see how it performs. The command is used to initialize the Agenta SDK. Subsequently, is where you define crucial parameters that dictate the behavior of the kitchen AI assistant. These parameters include: ag.init() ag.config.default(...) Regulates the creativity of responses (lower temperature = more predictable, higher = more unexpected). temperature: Determines the maximum length of the AI's response (measured in tokens, which roughly correspond to words). max_tokens: Sets the initial instructions or context provided to the LLM, specifying it as your kitchen assistant. prompt_system: Specifies the LLM to be used (with a range of GPT model versions as options). model: The line initializes the OpenAI Async SDK, enabling your code to establish a connection and interact with OpenAI's language models. client = AsyncOpenAI() To register your LLM app with Agenta and use it in their playground, you need to define an entry point. I used the decorator on my function. This tells Agenta that this function is where the action starts. Agenta then automatically converts this function into an API endpoint (using FastAPI), making it accessible through their web interface. @ag.entrypoint suggest Publishing The LLM App Now that your AI kitchen assistant is ready, we should publish it to Agenta for experimentation. To do this, you can install the Agenta CLI in a virtual environment using your preferred Python package installer or globally on your system. However, I advise against the latter to avoid potential conflicts with your system packages. Once the installation is complete, make sure to activate the virtual environment. I will proceed with publishing the LLM app to Agenta Cloud, enabling me to deploy the LLM application to a production environment and later, create a UI application for mobile use. If you wish to publish to the cloud version of Agenta as well, I suggest creating an first and then returning to follow the steps below. account If you prefer not to deploy the app to the cloud version of agenta, you can the open-source version on your local machine by following these steps. It will be a slightly different process, but not overly complex. run Create a new project for the LLM app To publish the application in Agenta, we must first create an empty project. Execute the following command in the directory where your application code is located: (.venv) [kitchen_assistant]$ agenta init This will prompt you to enter the project name, Agenta host, API key (if using the cloud or enterprise version), organization, and how you want to initialize the app. Once the app initialization is successfully done, a blank project will be generated in Agenta along with a config.toml file in the same directory, containing all project information. The config.toml file and the empty project in Agenta Cloud will look like the following. Publishing/Serving the first app variant With the project created in Agenta cloud, we need to add the first app variant to it. This can be done by running the following command: agenta variant serve app.py This will create a new app variant in Agenta under the name . Here, is the name of the codebase containing the LLM app logic, while is a default configuration created for that codebase. Each new app variant created from the web interface or from the CLI will always have the name format . app.default app default <codebase_name>.<configuration_name> An is a distinct version of an LLM app, featuring different parameters or code but keeping the same inputs and outputs. This feature enables you to experiment with alternative methods within a single app. For instance, you could have two variants of the kitchen assistant app, each using a different system prompt. app variant Executing the serve command above will create a container for the application featuring a REST API endpoint. This endpoint will be used in Agenta when interacting with the application. The CLI will also display the URL of the endpoint, which can be used to navigate to the application API documentation or playground in Agenta. Copy the playground URI and paste it into your browser. After doing so, you will see the following user interface. It may take 10-20 seconds to load as it sets up the playground for you. You have successfully published your LLM app to Agenta and would need to experiment with it to ensure that it functions as intended. In the chat section right after “User”, write the ingredients and click on the “Run” button. You’ll see that the AI kitchen assistant is considering what you can make with the ingredients: When it’s done processing the ingredients that you inputted, you’ll get suggestion on what you can make. And so, you own a Kitchen AI assistant that suggests recipes based on your ingredients. Did you observe something after each use? You view the total tokens used, the run's cost, and the time taken! Don't hold back now, keep experimenting with the LLM app. Customize the system prompt to match a particular culture that you want, adjust the temperature, and switch the model if needed. Deploying the LLM App To An Environment After using the playground to find a good configuration for your AI Kitchen assistant, it’s time to deploy the application. By deploying the application, you can integrate it to a UI (web and/or mobile) application or anywhere since the deployed app will be an endpoint. You can also later change the configuration from the playground without having to update the code. Clicking on the “Publish” button will bring up a modal with environments you want to deploy the LLM application to. Agenta provides a way to deploy an application to multiple environments: development, staging, and production. Each environment having its own unique configuration. Since the LLM app works and it’s the weekend, we can deploy to production :-). Click on the "production" checkbox and then the "Publish" button. You will receive a success notification informing you that the LLM app variant has been deployed. Look at the side navbar, you’re going to see different sections. I’ll keep what each of them do: The playground is a user-friendly environment in Agenta where you can create and test new app variants. Here, you can adjust parameters, input data, and observe the outputs from your app. Playground: The test set is another user-friendly environment where you can create high-quality datasets. A golden test set, also known as a ground truth test set, consists of various inputs and their expected correct answers. Test Sets: Another environment that enables you to generate automated assessments using various evaluators: , , , , , , and . The purpose of conducting evaluations is to methodically evaluate the outcomes of the LLM application and compare various versions to determine the most optimal one. Another objective is to set a benchmark for the application and evaluate any potential risks. Evaluations: exact match similarity match regex test JSON field match AI critique code evaluation webhook test At times, you might require assessing your models' performance through human judgement. This is where the Human Evaluation environment becomes valuable. It enables you to carry out and to gauge your models' performance using human judgement. Annotations: A/B tests single model tests Endpoints are where you can access the deployed application code. You can view the code only after configuring your application successfully and deploying it to a specific environment. Currently, Agenta supports just three languages to integrate the deployed LLM app: Python, cURL, and TypeScript. If you’d like to add more languages or a specific language, feel free to contribute by raising an and tackling it. ;-) issue To ensure the deployed LLM app works in production, choose cURL in the languages drop-down, click "Copy," update the content value with the ingredients you have, and run it in your terminal. You should receive a result similar to the one below. Conclusion How does it feel to have a Personal Kitchen AI assistant? My partner feels like she has a sous chef that handles recipe making for her. Quite amusing, isn't it? XD In this article, you learnt what LLMs are, how to create a kitchen assistant app using the GPT-3.5-turbo model to analyze your available ingredients and offer recipe suggestions. You also learned how to deploy the LLM app on Agenta for practical use. As an extra tip, I will be a user interface using Svelte and deploy it on Github Pages. I will design it to allow you to input the URI of your LLM app, enabling personal interaction. In the upcoming article, I will demonstrate how to generate test sets for conducting human evaluations to ensure that the LLM app's output aligns with your expectations. building

Your AI-Powered Kitchen Assistant: Building a Recipe Genius With GPT3.5-turbo and Agenta

আপনার AI-চালিত রান্নাঘর সহকারী: GPT3.5-turbo এবং Agenta দিয়ে একটি রেসিপি জিনিয়াস তৈরি করা

Ihr KI-gestützter Küchenassistent: Erstellen Sie ein Rezeptgenie mit GPT3.5-turbo und Agenta

Su asistente de cocina con tecnología de inteligencia artificial: creación de recetas geniales con GPT3.5-turbo y Agenta

Votre assistant de cuisine alimenté par l'IA : créer un génie des recettes avec GPT3.5-turbo et Agenta

आपका एआई-संचालित रसोई सहायक: जीपीटी3.5-टर्बो और एजेंटा के साथ एक रेसिपी जीनियस का निर्माण

AI を活用したキッチン アシスタント: GPT3.5 ターボと Agenta を使用して天才レシピを構築する

AI 기반 주방 도우미: GPT3.5 터보 및 Agenta를 사용하여 천재적인 레시피 구축

Seu assistente de cozinha com tecnologia de IA: construindo um gênio em receitas com GPT3.5-turbo e Agenta

Ваш кухонный помощник на базе искусственного интеллекта: создайте гения рецептов с помощью GPT3.5-turbo и Agenta

Yapay Zeka Destekli Mutfak Asistanınız: GPT3.5-turbo ve Agenta ile Bir Tarif Dahisi Oluşturmak

Trợ lý nhà bếp được hỗ trợ bởi AI của bạn: Xây dựng một thiên tài về công thức nấu ăn với GPT3.5-turbo và Agenta

您的人工智能厨房助手：使用 GPT3.5-turbo 和 Agenta 打造食谱天才

This is the first part in a multi-part series on building Agents with OpenAI's Assistant API using the Python SDK. What Are Agents? The way I like to look at it, an agent is really just a piece of software leveraging an LLM (Large Language Model) and trying to mimic human behavior. That means it can not only converse and understand language, but it can also perform actions that have an impact on the real world. These actions are typically called tools. In this blog post, we will explore how to build an agent using OpenAI's Assistant API using their Python SDK. Part 1 will be just the skeleton of the assistant. That is, just the conversational part. I chose to build a CLI app on purpose to be framework agnostic. We will purposefully call our implementation an Agent and refer to the OpenAI SDK implementation as an Assistant to easily distinguish between the two. I use the terms and interchangeably when it comes to functions that the Agent is able to call. Part 2 will cover function callin in more detail. tools functions Prerequisites To follow along with this tutorial, you will need the following: Python3 installed on your machine An OpenAI API key Basic knowledge of Python programming OpenAI Assistant Concepts : An Assistant in the Assistants API is an entity configured to respond to user messages. It uses instructions, a chosen model, and tools to interact with functions and provide answers. Assistant : A Thread represents a conversation or dialogue in the Assistants API. It is created for each user interaction and can contain multiple Messages, serving as a container for the ongoing conversation. Thread : A Message is a unit of communication in a Thread. It contains text (and potentially files in the future) and is used to convey user queries or assistant responses within a Thread. Message : A Run is an instance of the Assistant processing a Thread. It involves reading the Thread, deciding whether to call tools, and generating responses based on the model's interpretation of the Thread's Messages. Run Setting Up the Development Environment The first step is to create a virtual environment using and activate it. This will ensure that our dependencies are isolated from the system Python installation: venv python3 -m venv venv
source venv/bin/activate Let's install our only dependency: the package: openai pip install openai Create a file. Let's populate with some basic runtime logic for our CLI app: main.py while True:
    user_input = input("User: ")
    if user_input.lower() == 'exit':
        print("Exiting the assistant...")
        break
    print(f"Assistant: You said {user_input}") Try it out by running : python3 main.py python3 main.py
User: hi
Assistant: You said hi As you can see, the CLI accepts a User message as input, and our genius Assistant doesn't have a brain 🧠 yet so he just repeats the message right back. Not so smart yet. The Agent Now, the fun 😁 (or headaches 🤕) begins. I'll provide all the imports needed for the final class right now, so you don't rack your brain on where things are coming from since I kept imports out of code samples for brevity. Let's start by building an class in a new file : Agent agent.py import time
import openai
from openai.types.beta.threads.run import Run

class Agent:
    def __init__(self, name: str, personality: str):
        self.name = name
        self.personality = personality
        self.client = openai.OpenAI(api_key="sk-*****")
        self.assistant = self.client.beta.assistants.create(
            name=self.name,
            model="gpt-4-turbo-preview"
        ) In the class constructor, we initialize the OpenAI client as a class property by passing our OpenAI API key. Next, we create an class property that maps to our newly created Assistant. We store and as class properties for later use. assistant name personality The argument we are passing to the create method is just for identifying the Assistant in the OpenAI dashboard, and the AI is not actually aware of it at this point. You actually have to pass the name to the which we will see later. name instructions You could already set when creating the Assistant, but it will actually make your Assistant less flexible to dynamic changes. instructions You can update an Assistant by calling , but there is a better place to pass in dynamic values that we will see when we get to Runs. client.beta.assistants.update Note that if you pass here and then again when creating a Run, the Assistant's will be overwritten by the of the run. They do not complement each other, so choose one based on your needs: Assistant level for static instructions or Run level for dynamic instructions. instructions instructions instructions For the model, I chose the model so that we can add function calling in part 2 of this series. You could use if you want to save a few fractions of a penny while giving yourself a migraine of pure frustration down the line when we implement tools. gpt-4-turbo-preview gpt-3.5-turbo GPT 3.5 is terrible at calling tools; the hours I've lost trying to deal with it allow me to say that. 😝 I'll leave it at that, and more on this later. Creating a Thread, Adding Messages, and Retrieving the Last Message After we create an agent, we will need to start a conversation thread. class Agent:
    # ... (rest of code)
    def create_thread(self):
        self.thread = self.client.beta.threads.create() And we will want a way to add messages to that thread: class Agent:
    # ... (rest of code)
    def add_message(self, message):
        self.client.beta.threads.messages.create(
            thread_id=self.thread.id,
            role="user",
            content=message
        ) Note that at the moment, it is only possible to add messages with the role . I believe OpenAI plans on changing this in a future release as this is pretty limiting. user Now, we can get the last message in the thread: class Agent:
    # ... (rest of code)
    def get_last_message(self):
        return self.client.beta.threads.messages.list(
            thread_id=self.thread.id
        ).data[0].content[0].text.value Next, we create an entry point method to test out what we have so far. Currently, the method just returns the last message in the thread. It doesn't actually perform a Run. It's still brainless. run_agent run_agent class Agent:
    # ... (rest of code)
    def run_agent(self):
        message = self.get_last_message()
        return message Back in , we create the agent and our first thread. We add a message to the thread. Then return that same message back to the user, but this time, coming from that live thread. main.py from agent import Agent

agent = Agent(name="Bilbo Baggins",
                      personality="You are the accomplished and renowned adventurer from The Hobbit. You act like you are a bit of a homebody, but you are always up for an adventure. You worry a bit too much about breakfast.")

agent.create_thread()

while True:
    user_input = input("User: ")
    if user_input.lower() == 'exit':
        print("Exiting the agent...")
        break
    agent.add_message(user_input)
    answer = agent.run_agent()
    print(f"Assistant: {answer}") Let's run it: python3 main.py
User: hi
Assistant: hi Still not very smart. Closer to a parrot 🦜 than a hobbit. In the next section, the real fun begins. Creating and Polling a Run When you create a run, you need to periodically retrieve the object to check the status of the run. This is called polling, and it sucks. You need to poll in order to determine what your agent should do next. OpenAI plans to add support for streaming to make this simpler. In the meantime, I will show you how to set up polling in this next section. Run Note the on the following method names which is the standard in Python for indicating that the method is intended for internal use and should not be accessed directly by external code. _ First, let's create a helper method for creating a , and update to call this method: _create_run Run run_agent class Agent:
    # ... (rest of code)
    def get_breakfast_count_from_db(self):
        return 1

    def _create_run(self):
        count = self.get_breakfast_count_from_db()
        return self.client.beta.threads.runs.create(
            thread_id=self.thread.id,
            assistant_id=self.assistant.id,
            instructions=f"""
                Your name is: {self.name}
                Your personality is: {self.personality}

                Metadata related to this conversation:
                {{
                    "breakfast_count": {count}
                }}
            """,
        )

    def run_agent(self):
        run = self._create_run() # add this line
        message = self.get_last_message()
        return message Notice how we pass the and to create a run. thread.id assistant.id Remember how I said at the beginning that there was a better place to pass in dynamic instructions and data? That would be the parameter when creating the Run. In our case, we could have the breakfast be fetched from a database. This will allow you to easily pass in different relevant dynamic data every time you want to trigger an answer. instructions count Now, your agent is aware of the world changing around it and can act accordingly. I like to have a metadata JSON object in my instructions that keeps relevant dynamic context. This allows me to pass in data while being less verbose and in a format that the LLM understands really well. Don't run this yet; it won't work because we aren't waiting for the run to complete when we are getting the last message, so it will still be the last user message. Let's solve this by building out our polling mechanism. First, we will need a way to repeatedly and easily retrieve a run, so let's add a method: _retrieve_run class Agent:
    # ... (rest of code)
    def _retrieve_run(self, run: Run):
        return self.client.beta.threads.runs.retrieve(
            run_id=run.id, thread_id=self.thread.id) Notice how we need to pass both the and to find a specific run. run.id thread.id Add a method to our Agent class: _poll_run class Agent:
    # ... (rest of code)
    def _cancel_run(self, run: Run):
        self.client.beta.threads.runs.cancel(
            run_id=run.id, thread_id=self.thread.id)

    def _poll_run(self, run: Run):
        status = run.status
        start_time = time.time()
        while status != "completed":
            if status == 'failed':
                raise Exception(f"Run failed with error: {run.last_error}")
            if status == 'expired':
                raise Exception("Run expired.")

            time.sleep(1)
            run = self._retrieve_run(run)
            status = run.status

            elapsed_time = time.time() - start_time
            if elapsed_time > 120:  # 2 minutes
                self._cancel_run(run)
                raise Exception("Run took longer than 2 minutes.") 🥵 Phew, that's a lot... Let's unpack it. receives a object as an argument and extracts the current Run . All the available statuses can be found in the OpenAI . We'll just use a few that suit our current purpose. _poll_run Run status docs We now run a while loop to check for a completed status while handling a few error scenarios. The actual billing of the Assistant API is a bit murky, so to be on the safe side, I opted to cancel my runs after 2 minutes. Even though there is an status for when OpenAI cancels runs after 10 minutes. If a run takes more than 2 minutes, you probably have a problem anyway. expired Since I also don't want to poll every few milliseconds, I throttle my request by only polling every 1 second until I hit the 2-minute mark and cancel my run. You can adjust this to whatever you see fit. Each iteration after the delay, we fetch the Run status again. Now, let's plug all that into our method. You will notice we first create the run with then we poll with until we get an answer or an error is thrown, and finally when the polling is finished, we retrieve the last message from the thread which will now be from the agent. run_agent _create_run _poll_run We then return the message to our runtime loop, so it can be sent back to the user. class Agent:
    # ... (rest of code)
    def run_agent(self):
        run = self._create_run()
        self._poll_run(run) # add this line
        message = self.get_last_message()
        return message Voilà, now when you run your agent again, you will get a reply from our friendly Agent: python3 main.py
User: hi
Assistant: Hello there! What adventure can we embark on today? Or perhaps, before we set out, we should think about breakfast. Have you had yours yet? I've had mine, of course – can't start the day without a proper breakfast, you know.
User: how many breakfasts have you had?
Assistant: Ah, well, I've had just 1 breakfast today. But the day is still young, and there's always room for a second, isn't there? What about you? How can I assist you on this fine day? In part 2, we will add the ability for our Agent to call tools. You can find the full code on my . GitHub Thank you for your reading. Happy to hear any thoughts and feedback in the comments. Follow me on Linkedin for more content like this: https://www.linkedin.com/in/jean-marie-dalmasso-1b5473141/

How to Build an Agent With an OpenAI Assistant in Python - Part 1: Conversational

পাইথনে ওপেনএআই সহকারী দিয়ে কীভাবে এজেন্ট তৈরি করবেন - অংশ 1: কথোপকথন

So erstellen Sie einen Agenten mit einem OpenAI-Assistenten in Python – Teil 1: Konversational

Cómo crear un agente con un asistente OpenAI en Python - Parte 1: Conversacional

Comment créer un agent avec un assistant OpenAI en Python - Partie 1 : Conversationnel

पायथन में ओपनएआई असिस्टेंट के साथ एक एजेंट कैसे बनाएं - भाग 1: संवादात्मक

Python で OpenAI アシスタントを使用してエージェントを構築する方法 - パート 1: 会話型

Python에서 OpenAI 도우미를 사용하여 에이전트를 구축하는 방법 - 1부: 대화형

Como construir um agente com um assistente OpenAI em Python - Parte 1: Conversacional

Как создать агент с помощью OpenAI Assistant на Python. Часть 1: Диалог

Python'da OpenAI Asistanıyla Aracı Nasıl Oluşturulur - Bölüm 1: Konuşmaya Dayalı

Cách xây dựng một tác nhân với trợ lý OpenAI trong Python - Phần 1: Đàm thoại

如何使用 Python 中的 OpenAI 助手构建代理 - 第 1 部分：对话式

Effective metrics and monitoring play a key role in high-quality development, bug fixes, and handling of user requests and incidents. At Social Discovery Group, we employ a diverse range of tools to evaluate the performance of our products. In this article, we will share some hacks of how we managed to build a stable application monitoring and metrics system and reduced bug detection time on our products with the help of Datadog. SDG products are helping more than 250 mln people connect and build relationships globally, and our user base is constantly growing. The main factor in this success is our ability to promptly respond to user needs. We have a lot of experience working with various monitoring systems, including Datadog. Here's why: Currently, Datadog offers a variety of integration options with systems at different levels. You can explore these capabilities . here Datadog provides comprehensive documentation for configuring integrations with various services. We managed to build strong partnership relations. Here, we're sharing our experience in setting up and configuring APM metrics for applications within our Kubernetes cluster for Datadog. The article won't cover the deployment of projects in AKS, the CI/CD process, and other DevOps details. Instead, we will focus on the finer points of configuring Datadog monitoring for APM metrics. The technology stack used: Azure Services, Azure Kubernetes Service (AKS), ASP .Net Core 7, Datadog. For monitoring applications and services, we utilize the Datadog agent deployed within the cluster through a Helm chart with additional parameters from the values.yaml file. In the main agent configuration file, you'll need to enable the DogStatsD module and specify the port (default is 8125). For the data sent from an external host, the Datadog agent requires the following configuration: dogstatsd_non_local_traffic: true and apm_non_local_traffic: true. Here is the values.yaml file for one of the clusters, with some variables being passed at the deployment stage. The deployment is produced with Azure Devops. " 
datadog:
  apiKey: #{apiKey}#
  appKey: #{appKey}#
  clusterName:  #{ClusterName}#
  kubeStateMetricsEnabled: true
  clusterChecks:
    enabled: true
  dogstatsd:
    useSocketVolume: false
    nonLocalTraffic: true
  collectEvents: false
  apm:
    portEnabled: true #important to include
  env:
    - name: "DD_KUBELET_TLS_VERIFY"
      value: "false"
  systemProbe:
    collectDNSStats: false
  orchestratorExplorer:
    enabled: false
clusterAgent:
  image:
    repository: public.ecr.aws/datadog/cluster-agent
    tag: #{tag}#
  admissionController:
    enabled: false
agents:
  image:
    repository: public.ecr.aws/datadog/agent
    tag: #{tag}#
    doNotCheckTag:  true
clusterChecksRunner:
  image:
    repository: public.ecr.aws/datadog/agent
    tag: #{tag}#
" Then, you need to specify the agent's address for transmitting metrics to Datadog in the application settings. The dashboards for app monitoring are based on the internally used metrics; they had to be created independently. To set up monitoring of ASP.Net services, we used the official documentation that can be found by the . link Since the agent was already configured, one of the methods was to add the necessary lines to the image build and pass variables in the CI/CD system: DD_ENV, DD_SERVICE, DD_AGENT_HOST, to specify the environment, service name, and agent address, respectively. We also need to add the following to the dockerfiles for the services: "
RUN TRACER_VERSION=$(curl -s \https://api.github.com/repos/DataDog/dd-trace-dotnet/releases/latest | grep tag_name | cut -d '"' -f 4 | cut -c2-) \
    && curl -Lo /tmp/datadog-dotnet-apm.deb https://github.com/DataDog/dd-trace-dotnet/releases/download/v${TRACER_VERSION}/datadog-dotnet-apm_${TRACER_VERSION}_amd64.deb

# Copy the tracer from build target
COPY --from=build /tmp/datadog-dotnet-apm.deb /tmp/datadog-dotnet-apm.deb
# Install the tracer
RUN mkdir -p /opt/datadog \
    && mkdir -p /var/log/datadog \
    && dpkg -i /tmp/datadog-dotnet-apm.deb \
    && rm /tmp/datadog-dotnet-apm.deb
 
# Enable the tracer
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
ENV CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so
ENV DD_DOTNET_TRACER_HOME=/opt/datadog
ENV DD_INTEGRATIONS=/opt/datadog/integrations.json

" This method works, but it didn't seem like the most optimal solution. We decided to take it a step further and added the following to the deployments of our services: "
metadata.labels:
    tags.datadoghq.com/env: feature
    tags.datadoghq.com/service: service_name
    tags.datadoghq.com/version: '1552'

spec.template.metadata.labels:
    admission.datadoghq.com/config.mode: service
    admission.datadoghq.com/enabled: 'true'
    tags.datadoghq.com/env: feature
    tags.datadoghq.com/service: service_name
    tags.datadoghq.com/version: '1111'

spec.template.metadata.annotations:
    admission.datadoghq.com/dotnet-lib.version: v2.38.0

spec.template.spec.containers.name.env:
    - name: DD_TRACE_AGENT_URL
      value: datadog-agent.monitoring
    - name: DD_TRACE_STARTUP_LOGS
      value: 'true'
    - name: DD_LOGS_INJECTION
      value: 'true'
    - name: DD_RUNTIME_METRICS_ENABLED
      value: 'true'
    - name: DD_PROFILING_ENABLED
      value: 'true'
    - name: DD_APPSEC_ENABLED
      value: 'true'
" That’s what was changed in the agents: "
datadog:
  apm:
    socketEnabled: true
    portEnabled: true
    enabled: true
clusterAgent:
  admissionController:
    enabled: true
    mutateUnlabelled: false
providers:
  aks:
    enabled: true
" After all these steps, the data with highly detailed metrics for all services started flowing in Datadog under the APM -> Services section, and the graphs were automatically displayed. We had to tinker with the annotations settings for the second method; not everything started working smoothly right away. Regarding the notification system, it's worth mentioning that it is user-friendly and intuitive in Datadog. Notifications are created in the "Monitors -> Manage Monitors" section. The improvements we described above yielded several valuable outcomes. We now have a deeper understanding of how our system operates and adapts to various changes. Additionally, we've established a stable application monitoring and metrics system that operates independently of service builds, helping to reduce bug detection times. This, in turn, has allowed us to optimize our services and improve development speed and overall system quality.

How to Use Datadog for the APM Metrics Application

এপিএম মেট্রিক্স অ্যাপ্লিকেশনের জন্য ডেটাডগ কীভাবে ব্যবহার করবেন

So verwenden Sie Datadog für die APM Metrics-Anwendung

Cómo utilizar Datadog para la aplicación de métricas APM

Comment utiliser Datadog pour l'application APM Metrics

एपीएम मेट्रिक्स एप्लिकेशन के लिए डेटाडॉग का उपयोग कैसे करें

AGENTERO

agentero.com

-- employees

Since n.d.

COMPANY RANKING

AGENTERO

EVERGREEN INDEX #1068

Representing this company? Click here to claim and customize this page!

Agentero WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

AGENTERO

agentero.com

-- employees

Since n.d.

COMPANY RANKING

AGENTERO

EVERGREEN INDEX #1068

RELATED COMPANIESRANK

STORIES ABOUT AGENTERO

Representing this company? Click here to claim and customize this page!

Agentero WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!