Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Basic premises for our models
2.1. How LLMs are charged
2.2 An anedotal model from industry
2.3 Choice of Costs in the Model
2.4 Why multiple scenarios


A Decision Theoretic Model
3.1 How to model success


A Model for Commercial Operations Based on a Single Transaction
4.1 Example of Using the Model
4.2 Model Analysis
4.3 Discussion of the RoI vs. Earning dillema


Modelling of a Binary Classification Problem
5.1 Local Sensitivity Analysis
5.2 Global Sensitivity Analysis by Sobol Method


Related Work


Future work


Conclusion, References, and Acknowledgements Basic premises for our models
2.1. How LLMs are charged
2.2 An anedotal model from industry
2.3 Choice of Costs in the Model
2.4 Why multiple scenarios Basic premises for our models Basic premises for our models 2.1. How LLMs are charged 2.1. How LLMs are charged 2.2 An anedotal model from industry 2.2 An anedotal model from industry 2.3 Choice of Costs in the Model 2.3 Choice of Costs in the Model 2.4 Why multiple scenarios 2.4 Why multiple scenarios A Decision Theoretic Model
3.1 How to model success A Decision Theoretic Model A Decision Theoretic Model 3.1 How to model success 3.1 How to model success A Model for Commercial Operations Based on a Single Transaction
4.1 Example of Using the Model
4.2 Model Analysis
4.3 Discussion of the RoI vs. Earning dillema A Model for Commercial Operations Based on a Single Transaction A Model for Commercial Operations Based on a Single Transaction 4.1 Example of Using the Model 4.1 Example of Using the Model 4.2 Model Analysis 4.2 Model Analysis 4.3 Discussion of the RoI vs. Earning dillema 4.3 Discussion of the RoI vs. Earning dillema Modelling of a Binary Classification Problem
5.1 Local Sensitivity Analysis
5.2 Global Sensitivity Analysis by Sobol Method Modelling of a Binary Classification Problem Modelling of a Binary Classification Problem 5.1 Local Sensitivity Analysis 5.1 Local Sensitivity Analysis 5.2 Global Sensitivity Analysis by Sobol Method 5.2 Global Sensitivity Analysis by Sobol Method Related Work Related Work Related Work Future work Future work Future work Conclusion, References, and Acknowledgements Conclusion, References, and Acknowledgements Conclusion, References, and Acknowledgements 2 Basic premises for our models According to the PMBoK, 7th edition, “value is the ultimate indicator of project success”. For every planned project, it is important to analyze the value of the project in different qualitative and quantitative terms. Among other indicators, financial results are key performance measures for project success. When establishing a business case for a project, it is crucial to understand its costs, revenues, and return on investment [Project Management Institute, 2021]. Costs are the total amount of money spent on a project, while benefits are the total amount of money gained from a new project. Earnings are the difference between benefits and costs [1], the total amount of money the project will bring to the company, and return of investment (RoI), is the relation between earnings and costs [Project Management Institute, 2021]. 2.1 How LLMs are charged There are different ways in which LLMs are charged. Our models assume that they are charged by token, as OpenAI does [Shekhar et al., 2024]. However, AWS charges specific engines by the hour of the instance used [Kodali et al.,2024]. Moreover, the costs of the use of proprietary machines can be calculated using traditional capital and operating expenses methods [Park, 2013]. 2.2 An anedotal model from industry In this subsection, we use our experience to discuss how most companies try to calculate the expected earnings from using an LLM in a project. We bring attention to the fact that although one is actually making a prediction, probabilistic terms such as “expected value” and “probability of success” are not commonly used in a business case for a project. This model represents the idea that the introduction of the LLM allows the company to convert some lost revenue into new revenue; however, it disregards the fact that the result of an operation using the LLM can be bad for the business. Moreover, usually it does not consider business tasks, that is, it considers that if the LLM task is a success, then the business task will also be a success. A better model should include losses from the overall results of using LLMs, defined as L and the earnings would be the following: Where Q, Q ≪ M, is the number of transactions that would result in a negative result. With the same reasoning, RoI can be recalculated as: This model is yet too simplistic, and we improve it in the next subsections, showing how it can be made more representative of the problem. Authors:
(1) Geraldo Xexéo, Programa de Engenharia de Sistemas e Computação – COPPE, Universidade Federal do Rio de Janeiro, Brasil;
(2) Filipe Braida, Departamento de Ciência da Computação, Universidade Federal Rural do Rio de Janeiro;
(3) Marcus Parreiras, Programa de Engenharia de Sistemas e Computação – COPPE, Universidade Federal do Rio de Janeiro, Brasil and Coordenadoria de Engenharia de Produção - COENP, CEFET/RJ, Unidade Nova Iguaçu;
(4) Paulo Xavier, Programa de Engenharia de Sistemas e Computação – COPPE, Universidade Federal do Rio de Janeiro, Brasil. Authors: Authors: (1) Geraldo Xexéo, Programa de Engenharia de Sistemas e Computação – COPPE, Universidade Federal do Rio de Janeiro, Brasil; (2) Filipe Braida, Departamento de Ciência da Computação, Universidade Federal Rural do Rio de Janeiro; (3) Marcus Parreiras, Programa de Engenharia de Sistemas e Computação – COPPE, Universidade Federal do Rio de Janeiro, Brasil and Coordenadoria de Engenharia de Produção - COENP, CEFET/RJ, Unidade Nova Iguaçu; (4) Paulo Xavier, Programa de Engenharia de Sistemas e Computação – COPPE, Universidade Federal do Rio de Janeiro, Brasil. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv available on arxiv [1] We will use earnings E in this paper to avoid using P for profits, using P for probability

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

LLM Pricing Explained: Tokens, Embeddings, and More

New Research Breaks Down LLM ROI Across Business Use Cases

How the Most Important Models Actually Work

Too Long; Didn't Read

Make resilience your competitive advantage

Understanding the Financial Implications of LLM Deployment

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Formalization of the SylloBio-NLI Resource Generation Process

New Research Breaks Down LLM ROI Across Business Use Cases

LLM Pricing Explained: Tokens, Embeddings, and More

How to Measure the Business Impact of LLMs

Measuring LLM ROI from a Single Transaction Using Real-World Costs

A Formalization of the SylloBio-NLI Resource Generation Process

New Research Breaks Down LLM ROI Across Business Use Cases

LLM Pricing Explained: Tokens, Embeddings, and More

How to Measure the Business Impact of LLMs

Measuring LLM ROI from a Single Transaction Using Real-World Costs

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps