Measuring LLM ROI from a Single Transaction Using Real-World Costs

Table of Links

Abstract and 1. Introduction

2. Basic premises for our models

   2.1. How LLMs are charged

   2.2 An anedotal model from industry

   2.3 Choice of Costs in the Model

   2.4 Why multiple scenarios

3. A Decision Theoretic Model

   3.1 How to model success

4. A Model for Commercial Operations Based on a Single Transaction

   4.1 Example of Using the Model

   4.2 Model Analysis

   4.3 Discussion of the RoI vs. Earning dillema

5. Modelling of a Binary Classification Problem

   5.1 Local Sensitivity Analysis

   5.2 Global Sensitivity Analysis by Sobol Method

6. Related Work

7. Future work

8. Conclusion, References, and Acknowledgements

4 A Model for Commercial Operations Based on a Single Transaction

This scenario supposes that a company evaluates different LLMs for the same task, based on the probability of success of each LLM in the business. The analytical steps include building a decision-theoretic RoI model, performing a qualitative sensitivity analysis based on first-degree partial derivatives, and performing a Sokol evaluation of the first and second degree. In this scenario, we also show a numerical example based on real prices from the market in May 2024, and an arbitrarily supposed performance that is compatible with LLMs with the specified cost, to show how RoI formulae can be used in a real case.

In this first analysis, we consider only the (final) probability of success, modeled as P.

To quantify the economic impact of choosing an LLM, it is assumed that success in the task generates a gain G and an error implies a cost L. Gains and losses are inherent to the task and not to the LLM.

Thus, the expected value of earnings, E[E], and RoI, E[R], per transaction using an LLM, can be calculated with a decision-theoretical model where the earnings are calculated from the gain per transaction that generates a positive result multiplied by the probability of the result being positive, minus the loss per transaction that generates a negative impact, multiplied by the probability of a negative impact occurring and minus the total cost of the transaction, as in Equation 18. The expected RoI is the expected earnings divided by the total cost of the transaction, as in Equation 19.

These expressions facilitate a comparative analysis of earnings and RoI, taking into account not only the direct costs of using LLMs but also the economic consequences of the results produced by these technologies.

4.1 Example of Using the Model

For classification tasks, the cost can be completely dominated by the input, as the output can only be 1 token. Under the assumption of those specific conditions in a classification task, the earnings (E) and the RoI (R) for each LLM would be:

This example demonstrates that, despite llm-1 costing 20 times more and presenting a higher income, it also presents a significantly lower return on investment. This is a typical result of comparing projects with a large difference in investment [Project Management Institute, 2021].

However, if we consider a very large transaction, we can change the value of T to 128,000 and maintain the other values. With these data, it is possible to find an approximate value of 84% as the minimum performance required for llm-2 to become more economical than llm-1.

The graph in Figure 1 shows the evolution of the earnings for two competitive models according to the values used in the previous example as a function of the average transaction size.