What is the difference between "prompt tuning" and "fine-tuning"?

Introduction: the same word pointed at two different things

This was the first naive question I hit when I stepped into the world of generative AI.

“Prompt tuning” and “fine-tuning” — what actually makes them different?

The more I read, the more confused I got. The reason only made sense later: the phrase “prompt tuning” itself points at opposite things depending on where you see it. The “prompt tuning” listed in job postings and the “Prompt Tuning” in research papers disagree on the one thing that matters most — whether you train the model’s weights. Collect information without noticing this, and the lines never connect in your head.

In this article I line up these scattered terms again on a single axis: do you touch the model’s weights, or not? The conclusion up front is that this axis is the only thing you need to remember. When the next buzzword floats by, you will know exactly where to put it.

The bottom line first: the axis is “touch the weights or not”

An LLM is, at its core, a collection of billions of numbers (weights). The values of those numbers are the model’s “intelligence, habits, and knowledge.” So the difference between methods boils down to one thing: do you change those numbers, or leave them alone?

	Touch the weights?	What it includes
Help from outside	✕ No	Prompt engineering (the colloquial “prompt tuning”) / RAG
Rewrite the inside	⭕ Yes	Fine-tuning (full / LoRA, QLoRA, soft prompts)

That’s it. Everything else is detail. As a single diagram, it looks like this.

                An LLM = billions of weights
                (behavior lives in those numbers)
                            │
            ┌───────────────┴────────────────┐
            ▼                                 ▼
  Don't touch the weights          Touch the weights = Fine-tuning
  (help from outside)              (rewrite the model itself)
            │                                 │
      ┌─────┴──────┐               ┌──────────┴───────────┐
      ▼            ▼               ▼                      ▼
 Prompt Eng.      RAG        Full fine-tuning      PEFT (train a small part)
                                                   LoRA / QLoRA / Soft-prompt

An analogy: how to get the best out of a talented new hire

Since this is abstract, picture a sharp new employee.

Improve how you give instructions → prompt engineering
Hand them documents, then ask → RAG
Send them to training so they grow → fine-tuning

The key point: only fine-tuning rebuilds what is inside the newcomer’s head (= the weights). Prompt engineering and RAG both leave the person unchanged and lend a hand from outside. Neither is “better” — they have different purposes.

The terminology trap: two meanings of “prompt tuning”

This is the epicenter of the confusion, so let me pin it down in a table.

Term	What it does	Touch the weights?	What it really is
Prompt tuning (colloquial)	Crafting and improving instructions, examples, and outputs	✕	The same thing as prompt engineering
Prompt Tuning (academic)	Training only a “sequence of learnable vectors” prepended to the input	⭕ (lightweight training)	A member of the fine-tuning family (PEFT)

The same words, “prompt tuning,” mean “crafting the instruction” in practitioner slang but “a weight-training method” in papers. When a conversation feels like it isn’t connecting, this mismatch is usually the cause. Once you see it, it stops throwing you off.

What each one is (just the essentials)

1. Prompt engineering (weights unchanged)

Craft the instructions and few-shot examples, check the output, fix it — repeat. Its greatest weapon is being fast, cheap, and easy to throw away. The flip side: you have to keep explaining things in the prompt every time, and it is bad at producing “always the same behavior” reliably.

2. RAG (weights unchanged)

For each question, search external documents and hand them over as context. Its strength is freshness of knowledge — you can reflect the latest information or your own materials without retraining. But the model itself does not get smarter; quality is decided by how well you build the retrieval (“what to pick up, and how”).

3. Fine-tuning (training the weights)

Show many examples of “input → the output you want,” update the weights little by little, and burn the behavior itself into the model. Concretely, you endlessly repeat “predict (forward) → how far off were you (loss) → which weights to move and how (gradient/backprop) → nudge a little (update)” — the loop in the figure below.

   ┌──────────────┐   predict    ┌──────────────┐
   │ 1. Forward   │ ───────────► │ 2. Loss      │
   │  (predict)   │              │ (how wrong)  │
   └──────────────┘              └──────────────┘
          ▲                              │
   update │                              │ gradient
          │                              ▼
   ┌──────────────┐ nudge weights ┌──────────────┐
   │ 4. Update    │ ◄──────────── │ 3. Backprop  │
   └──────────────┘               └──────────────┘
      repeat over data → loss goes down → behavior is learned

Full fine-tuning: update every weight. Highly expressive, but very heavy and it needs a big GPU.
PEFT (the lightweight camp): freeze the base and train only a small added part. Cost-effective, and the realistic choice for an individual today.
- LoRA / QLoRA: add small adapter matrices and train those (QLoRA also compresses the base to 4-bit for even less memory).
- Soft prompts (= the academic “Prompt Tuning”) / Prefix / P-Tuning: train only the learnable vectors prepended to the input.

The strength of fine-tuning is producing a fixed format, tone, or classification stably and cheaply, with a short instruction. You can build a small, dedicated model. The cost is that you need labeled data and a GPU, and when the world changes you have to retrain.

The most important skill is “choosing”

Knowing the names of the tools matters less than which one you pick for the problem in front of you. That is the real skill.

What you want to do	Pick first	Why
Answer with the latest facts or in-house knowledge / content changes often	RAG	Reflects instantly without training; strong on freshness
Just want to try / requirements not settled	Prompt engineering	Fast, cheap, easy to throw away
Mass-produce a fixed format, classification, or tone, reliably	Fine-tuning (start with LoRA/QLoRA)	Burns the behavior into the weights
Rebuild a giant model wholesale	Full fine-tuning	But assumes a big GPU; PEFT is usually enough

The standard play in practice is “1) prompt → if that’s not enough, 2) RAG → if that’s still not enough, 3) fine-tuning.” Jumping straight to 3 is usually the long way around.

A note: #3 trips you on “environment,” not “code”

One last thing from experience. With fine-tuning (#3), the hard part was not the algorithm but the environment setup. The steps are in the official docs, but the moment you run them on your own machine, ground-level walls appear one after another: mismatched library generations, character encodings, differences between GPU generations. You can copy the code, but you cannot copy the judgment of “where it got stuck and why I fixed it that way.” That is where the value always remains.

The “I actually tried it” side — the full record of getting stuck while fine-tuning locally on a single Turing-generation GPU (RTX 2070) — is in a separate article.

🔗 Hands-on edition: Taking on local QLoRA fine-tuning with a single Turing-generation GPU, the RTX 2070 (in progress)

Summary (in one place)

There is only one axis of difference: touch the weights or not.
Don’t touch … prompt engineering (the colloquial “prompt tuning”) / RAG.
Touch … fine-tuning (full / PEFT: LoRA, QLoRA, soft prompts).
“Prompt tuning” is a different thing in slang versus academia. It is the number-one reason conversations fail to connect.
And — more than memorizing names, being able to judge “which one for this problem” is the real skill.

References (where the terms come from)

Soft-prompt-style Prompt Tuning (Lester et al., 2021) / Prefix Tuning / P-Tuning
QLoRA (4-bit quantization + LoRA, Dettmers et al., 2023)

Glossary (words used in this article)

Term	Meaning
Prompt tuning (two senses: colloquial / academic)	The subject of this article. In slang it means “prompt engineering”; in academia it means PEFT’s “Prompt Tuning” (different things)
Weights	The vast set of numbers inside the model. The substance of its intelligence and knowledge
Prompt engineering	Crafting instructions and few-shot examples to shape the output (weights unchanged)
Few-shot (examples)	Putting a few correct examples in the prompt to steer the output
RAG	Search relevant documents at question time and pass them as context (weights unchanged)
Fine-tuning	Update the weights with examples to burn the behavior into the model
Full fine-tuning	The heavy method that updates every weight
PEFT	Umbrella term for lightweight methods that freeze the base and train only a part
LoRA / QLoRA	Add small adapter matrices and train those (QLoRA quantizes the base to 4-bit to save memory)
Soft prompts (academic Prompt Tuning) / Prefix / P-Tuning	A PEFT family that trains only the learnable vectors prepended to the input
forward (predict) / loss / backprop / update	The four steps of the training loop
Gradient	A quantity that tells you which way and how far to move each weight to reduce the loss