Large language models (LLMs) are a category of deep learning models trained on immense amounts of data, making them capable of understanding and generating natural language and other types of content to perform a wide range of tasks. LLMs are built on a type of neural network architecture called a transformer which excels at handling sequences of words and capturing patterns in text. 

LLMs work as giant statistical prediction machines that repeatedly predict the next word in a sequence. They learn patterns in their text and generate language that follows those patterns. 

LLMs represent a major leap in how humans interact with technology because they are the first AI system that can handle unstructured human language on scale, allowing for natural communication with machines. 

2. How big are LLMs? 

An important characteristic of an LLM is its size. This refers to the number of parameters the model has and is usually specified in its name. For example, the model “Llama-2-13b-chat-hf” has a model size of 13 billion parameters (indicated by “13b”). 

The size of an LLM, i.e., the number of parameters it contains, determines how complex the model is and how much data it can process. A larger model has more detailed knowledge and is usually trained in more areas. This provides users with satisfactory answers to more questions. Regardless of their size, all LLMs can understand, process, and respond to human language (known as natural language processing and NLP capabilities). 

2.1 Overview of LLMs of different sizes and with different purposes 

As shown in the table, there are significant variations in the size of LLMs. In 2019, the number of parameters was small. For example, the LLM RoBERTa, released in 2019, has 340 million parameters. Starting in 2022, more LLMs with more than 60 billion parameters were produced (Llama 70B models, GPT4, GPT5, etc.). These are mostly designed for multiple/general purposes and represent a large knowledge base. However, due to their considerable size, fine-tuning is very time-consuming and unattractive. Model hosting is also very expensive. For this reason, in addition to very large LLMs, “small” LLMs, known as SLMs, are increasingly being produced. 

2.2 Small Language Models (SLM) 

Small Language Models (SLMs) are lightweight versions of traditional language models designed to operate efficiently on resource-constrained environments such as smartphones, embedded systems, or low-power computers. They typically range from 1 million to 10 billion parameters. The SLMs are significantly smaller, but they still retain core NLP capabilities like text generation, summarization, translation, and question-answering. 

While the term "Small Language Model" (SLM) is widely used, some practitioners find it misleading, as models with a billion parameters are objectively large. An alternative, "small Large Language Model (sLLM)”, was proposed but is considered too convoluted. As a result, the community has largely adopted the "Small Language Model," with the understanding that the term "small" is used only in comparison to the much larger language models. 

SLMs differ from conventional LLMs not only in their smaller size, but also in their design as a specialized and optimized language model. They should be used for a single task, a specific purpose, or a specific application. Fine-tuning is an integral part of the adaptation process. The pre-trained models provide all the necessary skills for communication in natural language and understanding prompts, while fine-tuning enables the SLM to acquire domain-specific understanding and/or task-specific capabilities. You will read more about fine-tuning in the last chapter (LLM Integration Patterns) of this article. 

You may integrate multiple specialized SLMs into a more comprehensive application or platform to meet the requirements of complex use cases while ensuring high accuracy and reliability in a resource-efficient manner. 

3. Memory demand for training and inference (including formula) 

3.1 Introduction 

If you want to host or fine-tune large language models yourself, it is essential to know the memory requirements of a model. The total memory requirement consists of the memory needed to load the model into the server's working memory and the additional memory needed for hosting for inference (also known as model serving) or fine-tuning. In fact, loading the model is often very demanding and severely limits the remaining available memory for training and inference or requires special servers/VMs with a lot of (video) memory.  

The parameter count and the desired accuracy of the model play a decisive role in loading the model. The desired precision determines the data type used for each parameter: 

Of these data types, BF16 is most used because it offers a good compromise between precision and memory requirements. BF16 was developed as an improved version of the FP16 data type by Google Brain (hence the “b”). FP32 offers the best precision but also requires the most memory. INT8 and INT4 are data types used in techniques such as quantization. In quantization, a model with parameters in BF16 is often loaded, and its parameters are then reduced to INT8 or INT4 using special algorithms, halving or quartering the memory requirements. 

3.2 Calculating base memory demand 

If you know a model parameter count and the desired numerical precision, you can calculate the memory required to load it. The chosen precision determines how much memory each parameter occupies. You can calculate the total required memory using the following formula: 

Kindly note that the above formula applies to all ML models and that the number of parameters in the classical ML world is typically expressed in millions. For models that use parameters in the billions, the formula can be simplified: 

Alternatively, instead of multiplying by 0.93, you can multiply by 1 (one), which allows you to perform an approximate calculation in your head. 

Taking the “Llama-3-8B” model with a parameter data type of BF 16 as an example, the exact memory requirements for loading the model can be calculated as follows: 

3.3 Inference memory demand 

Inference (model serving) requires additional memory. For each request, memory corresponding to the size of the input token and the text generated so far must be available. Consequently, the amount of total available memory limits the maximum number of requests that can be processed simultaneously. When calculating memory requirements, it is recommended to use the simplified variant, which involves multiplying by one. This immediately provides a small buffer that can be used for incoming inference requests. 

3.4 Training and fine-tuning memory demand 

Calculating the storage requirements for training and fine-tuning is much more complicated and depends heavily on the algorithms and training settings used. In any case, it is significantly higher than for inference. As a rough guideline, you can assume four to six times the memory requirements of inference. For the “Llama-3-8B” model using the BF16 data type, that would be approximately 60 GB to 90 GB.  

In terms of memory requirements, it does not matter whether training is carried out from scratch or as fine-tuning – in both cases, the memory requirements are identical. The usually smaller amount of training data required for fine-tuning compared to training from scratch only affects the duration and utilization of the GPU; memory requirements are not affected. Only with memory-optimized algorithms such as parameter-efficient fine-tuning (abbreviated: PEFT) and (quantized) low-rank adaptation (abbreviated: LoRA and QLoRA) is the memory requirement significantly reduced. 

4. Choosing the right hardware for training and inference

For training and inference of LLMs, you must decide which hardware to choose. For this, special resource constraints must be considered: algorithms and technologies/tools used in training and inference. They determine what type of hardware can and should be acquired. For example, GPU-based algorithms cannot usually be run on CPUs. And although commonly integrated tools/technologies, such as vLLM in inference, often support a variety of hardware (CPUs, GPUs, TPUs, etc.) they have much better performance when using hardware other than CPUs. 

5. The VRAM Bottleneck: Why memory is the core challenge in LLM infrastructure

5.1 Limited VRAM capacity in infrastructure offerings 

When searching for suitable GPUs offered by cloud providers, you will quickly encounter a significant limitation: the size of the available video memory (VRAM). Important context: GPU cloud instances or VMs usually have a large amount of regular RAM combined with CPUs and only a small portion is the actual GPU with the included VRAM. Only the smaller VRAM portion (which is often difficult to find on the provider's website) is suitable for the high-performance processing of LLM tasks. CPUs and regular RAM perform a key supporting role: it preprocesses and loads constantly fresh batches of data into the GPU, acts as overflow buffer when using special algorithms, and runs all other necessary tasks of the cloud instance/VM like the operating system. Nevertheless, VRAM capacity, which is often difficult to find, should be your main concern when choosing a GPU. 

Even in professional data center GPUs, video memory is small compared to CPU-based servers. For example, the Nvidia A100 GPU, often listed in benchmarks and released in 2020, has a variant with 40 GB VRAM and a variant with 80 GB VRAM. The Nvidia H200 GPU released in 2024 offers a maximum of 141 GB VRAM. This is more than the A100 variant, but not enough to run large models such as GPT-3-175B or Falcon 180B on a single GPU without potential accuracy losses. Only the newest Nvidia B200 datacenter GPU with 192 GB of VRAM would be able to fulfil the memory requirements without reducing precision but at extreme costs. However, constantly evolving techniques such as quantization can be used to greatly reduce the memory requirements for training and inference at the expense of slightly lower precision, thereby making more efficient use of available resources. You can also use a technique called model parallelism (or tensor parallelism), to shard or split a model over multiple GPUs, but this comes with a slight overall performance loss and network overhead. 

The table shows the minimum VRAM capacity required for standard precision (BF16) and when using quantization (INT8/INT4). The requirements for training and inference are added on top of this. When selecting a model, also pay attention to the most common VRAM sizes of data center GPUs: 24 GB (budget-oriented), 40 or 48 GB (general purpose), 80 GB (new standard for training) and high-end GPUs (128 GB, 141 GB, 192 GB).

5.2 The VRAM bottleneck explained 
To use a GPU for LLM tasks, it is essential that all data required to perform a process or task is available in the GPU's main video memory. The high number of parameters in LLMs means that a large portion of the available video memory is used solely for loading the model. For example, Microsoft’s Phi-3-medium open model with 14 billion parameters using a 16-bit data type takes up approximately 26 GB of VRAM to load (see formula described in previous sections). Memory requirements for fine-tuning or processing requests/prompts during inference are added on top of this. 

Even with the use of memory optimization for inference techniques like request batching, KV-batching and paged attention, the remaining video memory of a GPU after loading a single model is very limited. In practice, only a fraction of the GPU's available computing power is used in inference, making the size of the video memory a limiting factor in inference (memory-bound). Hitting the upper limit of the VRAM is responsible for a large part of latency.  

For this reason, when selecting GPUs for inference of LLMs, the size of the video memory should always be prioritized over the computing power of the GPU. Only for pre-training or fine-tuning a LLM the computing power really matters. But don't forget that in this cases, VRAM requirements are much higher than for inference when memory-efficient fine-tuning algorithms such as PEFT, LoRA and QLoRA are not used. 

Understanding how a model's parameter count translates into gigabytes of VRAM is not just an engineering exercise; it is the foundation of your budget and the key to your application's performance. The trade-offs between a general-purpose GPU and a specialized cloud accelerator are strategic decisions about vendor lock-in, scalability, and long-term operational cost. 

So, how do you translate these fundamentals into a competitive advantage? 

This is precisely the challenge we solve. For years, Liquid Reply has specialized in building robust, scalable Internal Developer Platforms (IDPs) on Kubernetes to master complex infrastructure. We now apply this deep expertise to the next frontier: helping organizations build sovereign, production-grade AI and LLMOps platforms.