5,537 reads

Real-life LLM Implementation: A Backender’s Perspective

by Filipp ShcherbanichDecember 14th, 2023

Too Long; Didn't Read

Our modern world is becoming increasingly difficult to imagine without Large Language Models (LLMs). This technology is quickly becoming an essential part of our daily lives, exerting a noticeable influence, including backend development. The impact of LLMs is felt not only in streamlining work processes, such as assisting in code writing.

Company Mentioned

featured image - Real-life LLM Implementation: A Backender’s Perspective

‘a red robot and a blue robot’ Image created by HackerNoon AI Image Generator

While many of us apply these technologies to optimize daily tasks, we are also becoming more actively confronted with the need to incorporate LLMs into the products we develop.

For instance, we may need to enhance them with new exciting features or improve existing functionality using new text processing capabilities.

In this article, I would like to explore the advantages and disadvantages of various large language models; share thoughts on potential issues that backend developers may encounter while employing this technology; and discuss what has to be done to ensure that LLM integration into your product does not lead to financial losses and customer attrition.

Additionally, I will try and focus on the less obvious issues and provide guidance on how to address them.

Open Source or Closed Source?

When you face a task that involves a large language model, you have to choose from a vast array of existing solutions. These solutions can be divided into two main types: open-source and closed-source options. This choice is determined by numerous factors, and it inevitably leads to potential problems and complexities that you will encounter.

Different issues require unique optimizations for their resolution, and the cost and time of the final result can vary significantly if something among these problems is overlooked. Let's consider the key advantages and disadvantages of each approach.

Open-Source LLM

When using open-source solutions, a deeper understanding of model functioning is required for effective configuration and usage. Working with closed models through an API is like communicating with a "black box."

Open-source models are a different story: with them, you have to handle all the configuration and integration problems yourself. This introduces certain risks:

Pros:	Cons:
- No need to pay a third-partyservice provider to obtain results. -No issues with being dependent on a specific vendor; for example, there is no risk that the API version suddenly changes and your system ceases to function. -Full control over your data. While organizations like OpenAI can be trusted not to use private data for model fine-tuning or store it anywhere, there are certain risks of potential leaks. Ultimately, every API request with sensitive data poses a potential danger, so when we control the data ourselves, we mitigate these risks. -Scalabilitywithout additional costs: since open models don't require licensing fees for scaling, they allow increasing the workload without extra spending. -Flexibility and adaptability: Open models can be modified and adapted without limitations, enabling flexible integration, and/or meeting industry-specific requirements.	-Incorrect software configurationcan lead to security vulnerabilities, especially when the model interacts with sensitive data. Imagine training a model on sensitive corporate data, and due to misconfigurations, it becomes accessible to the entire internet. Or using a model version with a known vulnerability that you forgot to update. In reality, there are numerous security challenges with such solutions, and by deploying such software, you assume full responsibility for addressing them. -In-depth technical work with the model requireshighly qualified specialists, and even minor errors can result in failures or incorrect model operations. -Configuring and optimizing open LLMs may takea lot of time. LLMs typically require substantial computational resources, and this is a crucial thing to consider. In some cases, the inability to scale or the excessively high cost of maintaining such solutions may render them impractical. -Lack of centralized supportmeans all updates and fixes need to be implemented independently, potentially leading to falling behind on the latest developments and standards. -Some open models may have licenses imposing specific restrictions on their use and modification, which could be overlooked without a deep understanding of the licensing policy.

Pros:

Cons:

- No need to pay a third-partyservice provider to obtain results.

-No issues with being dependent on a specific vendor; for example, there is no risk that the API version suddenly changes and your system ceases to function.

-Full control over your data. While organizations like OpenAI can be trusted not to use private data for model fine-tuning or store it anywhere, there are certain risks of potential leaks. Ultimately, every API request with sensitive data poses a potential danger, so when we control the data ourselves, we mitigate these risks.

-Scalabilitywithout additional costs: since open models don't require licensing fees for scaling, they allow increasing the workload without extra spending.

-Flexibility and adaptability: Open models can be modified and adapted without limitations, enabling flexible integration, and/or meeting industry-specific requirements.

-Incorrect software configurationcan lead to security vulnerabilities, especially when the model interacts with sensitive data. Imagine training a model on sensitive corporate data, and due to misconfigurations, it becomes accessible to the entire internet. Or using a model version with a known vulnerability that you forgot to update. In reality, there are numerous security challenges with such solutions, and by deploying such software, you assume full responsibility for addressing them.

-In-depth technical work with the model requireshighly qualified specialists, and even minor errors can result in failures or incorrect model operations.

-Configuring and optimizing open LLMs may takea lot of time. LLMs typically require substantial computational resources, and this is a crucial thing to consider. In some cases, the inability to scale or the excessively high cost of maintaining such solutions may render them impractical.

-Lack of centralized supportmeans all updates and fixes need to be implemented independently, potentially leading to falling behind on the latest developments and standards.

-Some open models may have licenses imposing specific restrictions on their use and modification, which could be overlooked without a deep understanding of the licensing policy.

Fortunately, some of these issues can be resolved without too much effort and without involving expensive specialists. If you're a backend developer, you can tackle certain security issues, optimize backend interaction with LLMs, address software update challenges, and partially automate license checks. Let's explore the specific steps you can take today if you are using open-source solutions:

Secure access to your LLM through a VPN. This will address some information protection issues, making it much more challenging, if not impossible, for malicious actors to exploit potential vulnerabilities.

When training or fine-tuning a model, remember to anonymize sensitive data. Do not store any confidential data in the model if individuals with insufficient viewing rights have access to it.

Configure monitoring and logging systems to track suspicious activity and respond promptly to incidents.

Reduce the number of requests to the language model to decrease resource consumption. Apply caching where possible. This recommendation is relevant in any situation where the backend interacts with a resource-intensive system.

Ensure that the LLM license you want to use aligns with your purposes. Also, exercise caution with content generated by the language model, as it may violate someone's rights. As a developer, you are obliged to be mindful of such nuances in using this technology.

Closed-Source LLM

When discussing closed-source LLM, the first thing that comes to mind is API-accessible language models, such as ChatGPT. In this section, we'll focus on such options, given their immense popularity and seamless integration into various systems. Many drawbacks and advantages of open-source and closed-source solutions are exactly opposite:

Pros:	Cons:
-Closed-source LLMs involve fewer technical complexitiesas they are provided as ready-to-use products. This is one of the significant advantages, as no configuration or support for the LLM is required, saving time and allowing a focus on developing the product side of the project. -Closed-source LLMs are typically developed withscalabilityin mind and can handle large volumes of requests without additional setup costs. -Access to unique capabilitiesand features. For example, well-known models like ChatGPT offer the ability to invoke functions, enabling the creation of products at a different level. There are also plugin systems that significantly expand the capabilities of these language models without substantial developer effort. -Paid solutions currently providemore powerful models compared to open-source alternatives. In many cases, this can be critically important.	-You need to pay a third-party service provider toobtain results. Working with such systems requires extreme caution because the cost of each subsequent request can snowball, especially if we want to convey maximum context. -Using such solutionsties us to a specific vendor, especially if we employ unique features like ChatGPT functions or plugins. Thus, the life of our product could come to an abrupt end one day; for instance, the recent scandal at OpenAI may have prompted many developers to rethink their dependence on this service provider. -Lack of control over data. Every API request with sensitive data is a potential risk because there is no guarantee that it won't be used for model fine-tuning or simply stolen or logged somewhere along the way to the model. -The API can change at any moment,and your application will stop working until you fix errors related to these changes. For example, a recent update of the ChatGPT version was associated with API changes, causing the code of several popular libraries to stop working correctly. Some companies likely experienced financial losses due to this. -If a third-party solution stops working, without preparation, there's little you can do except wait for the service provider to resolve the issue.

Pros:

Cons:

-Closed-source LLMs involve fewer technical complexitiesas they are provided as ready-to-use products. This is one of the significant advantages, as no configuration or support for the LLM is required, saving time and allowing a focus on developing the product side of the project.

-Closed-source LLMs are typically developed withscalabilityin mind and can handle large volumes of requests without additional setup costs.

-Access to unique capabilitiesand features. For example, well-known models like ChatGPT offer the ability to invoke functions, enabling the creation of products at a different level. There are also plugin systems that significantly expand the capabilities of these language models without substantial developer effort.

-Paid solutions currently providemore powerful models compared to open-source alternatives. In many cases, this can be critically important.

-You need to pay a third-party service provider toobtain results. Working with such systems requires extreme caution because the cost of each subsequent request can snowball, especially if we want to convey maximum context.

-Using such solutionsties us to a specific vendor, especially if we employ unique features like ChatGPT functions or plugins. Thus, the life of our product could come to an abrupt end one day; for instance, the recent scandal at OpenAI may have prompted many developers to rethink their dependence on this service provider.

-Lack of control over data. Every API request with sensitive data is a potential risk because there is no guarantee that it won't be used for model fine-tuning or simply stolen or logged somewhere along the way to the model.

-**The API can change at any moment,**and your application will stop working until you fix errors related to these changes. For example, a recent update of the ChatGPT version was associated with API changes, causing the code of several popular libraries to stop working correctly. Some companies likely experienced financial losses due to this.

-If a third-party solution stops working, without preparation, there's little you can do except wait for the service provider to resolve the issue.

When working with third-party systems via API, possible solutions to problems are generally quite similar and not particularly unique, but discussing them additionally is always important to avoid overlooking anything.

Use monitoring tools, and set up alerts in case the API of the closed-source solution becomes unavailable to react quickly to such an event.

If possible, prepare your Plan B in advance. For instance, identify a second vendor whose solution can be switched to handle at least part of the traffic or functionality.

Anonymize sensitive data before sending it to the API to minimize the risk of leaks. Use data generated by the language model with extreme care, being prepared for potential legal complexities.

Control the size of the context sent with the main request to save costs. Estimate tokens in advance, and use information about your pricing plan to predict expenses and manage your balance.

Whenever possible, try to use fewer unique features of the vendor to avoid becoming too dependent on them. This way, you can safeguard your project.

Conclusion

Choosing between two such important options when marrying your product to an LLM may indeed bring you to a philosophical state of mind. Pessimists might quote Søren Kierkegaard, a Danish philosopher: his idea was that, whatever you choose, you are going to regret it anyway.

But I prefer to stay on the bright side and rephrase Socrates, the Ancient Greek: “By all means, marry. If you get a good match, you will be happy. If you get a bad one, you will become a philosopher an expert in LLM”.

Just try to be practical and, well, follow your gut. Good luck, and feel free to share your experience in the comments!