How to adapt and assure AI behaviour, and why that demands a new approach to quality.
More and more software contains AI models tailored to a specific organisation or domain. That tailoring process is called fine-tuning. But once you change a model’s behaviour, the quality question changes too: no longer “does the feature work?”, but “does the model behave consistently, predictably, and safely?”. In this article, you will learn what fine-tuning is, how it differs from prompt engineering and RAG, and, perhaps most importantly, how to test and assure the result in a risk-driven way with Evaluaite, QA Company’s quality framework for AI systems.
What is fine-tuning?
Fine-tuning is the process of taking a powerful, pre-trained AI model, one that has already learned general patterns from enormous datasets, and making it perform better on a specific task, domain, or style.
Rather than training a model from scratch, you essentially complete the model’s education with a carefully curated dataset that reflects your goal: customer service tailored to one organisation, classifying support tickets, summarising documents in a fixed format, or maintaining a recognisable brand voice.
General-purpose models are excellent all-rounders, but they are not automatically good at any one specific task. A model trained on the broad internet does not know your compliance policies, internal knowledge base, or the exact tone your organisation uses. Fine-tuning bridges that gap: it adjusts the model’s internals so that it learns the style, vocabulary, and decision boundaries that matter for your organisation.
A fine-tuned model behaves differently from the original, and that new behaviour must be tested, measured, and assured.
Types of fine-tuning (and what that means for quality)
There are several ways to fine-tune a model, each with its own trade-offs in compute, flexibility, and behaviour. For quality purposes, what matters most is how significantly a method changes the model’s behaviour, because that determines how much regression risk is involved.
Supervised fine-tuning SFT
The most straightforward approach: you train a model on labelled input-output pairs. A common variant is instruction fine-tuning, where examples are formatted as instructions with corresponding answers. Many chat models (such as ChatGPT or Claude) have undergone this process. SFT works well when you have clear examples of correct behaviour and want the model to reproduce that pattern consistently. A word of caution: skewed or unbalanced examples in the training data carry just as much influence. Bias that is already present in the training data is simply carried over by the model.
Parameter-efficient fine-tuning LoRA & QLoRA
With Parameter-Efficient Fine-Tuning (PEFT), you leave the bulk of the model unchanged and only train a small additional component, a kind of add-on module. The rest of the model remains untouched; only that module learns the new task. This is far cheaper than retraining the entire model, and in practice it works surprisingly well. Low-Rank Adaptation (LoRA) is the most popular approach; Quantised Low-Rank Adaptation (QLoRA) extends this by compressing the base model further, making fine-tuning feasible on standard, less powerful hardware.
There is another advantage: because the base model stays intact, you can create multiple small add-on modules for different tasks and swap them in and out as needed, without retraining a full model each time. That also makes regression more manageable: you know exactly which small component has changed.
Full fine-tuning
With full fine-tuning, you are not adjusting a small add-on, but the model itself: a large proportion, or all, of what the model has learned internally is updated. This offers the greatest control over the outcome, but it is also the most disruptive approach, carrying the greatest risks. Because you are making changes so deep within the model, you risk it forgetting things it previously did well (catastrophic forgetting), or its behaviour gradually shifting over time relative to what was observed during initial testing (output drift).
A model fine-tuned aggressively on legal documents may suddenly struggle with everyday conversation, or forget how to format code. Reserve full fine-tuning for situations where the lighter PEFT methods genuinely cannot meet your objectives, and test broadly and repeatedly on what must remain stable.
PEFT (LoRA / QLoRA)
Full fine-tuning
Fine-tuning vs. prompt engineering vs. RAG
Fine-tuning is often confused with prompt engineering and RAG. They are different techniques, and each fails in a different way, which determines where you need to focus your testing.
Prompt engineering steers behaviour through instructions within the prompt itself. Quick and cost-free, but rigid when strict output formats are required and sensitive to variations in input. The model has not actually learned the requirements; it is simply following instructions.
RAG (retrieval-augmented generation) retrieves relevant documents from a knowledge source at query time and appends them to the prompt. Ideal when facts change frequently. But quality stands or falls on the retrieval: does the system fetch the right documents, and does the model use them correctly? When it does not, hallucination becomes a risk: the model presents information that cannot be found in the retrieved sources.
Fine-tuning embeds behaviour within the model itself. Well suited to a consistent tone, a fixed output format such as JSON, domain-specific reasoning, and predictable behaviour. But the knowledge is static: updating it means retraining.
In practice, teams combine all three: RAG for up-to-date knowledge, fine-tuning for behavioural alignment, and prompts for per-request instructions. This also means an AI feature is rarely testable in just one way; you need to understand which layers are involved.
Quick decision guide
- Fine-tuningfor a consistent tone, domain reasoning, or predictable product behaviour.
- RAGwhen knowledge changes regularly.
- Prompt engineeringfor exploration and prototyping.
Why testing AI differs from classical testing
Whether you are fine-tuning, using RAG, or writing carefully crafted prompts: you are changing the behaviour of a system that is non-deterministic. The same input can produce different output. This renders the classical binary pass/fail approach, on which traditional testing relies, insufficient.
AI also introduces new risks, such as stochastic behaviour, retrieval drift, output drift, hallucination, and inconsistent agent output. And those risks rarely manifest in a single place. A small variation in input can lead to unstable output, which in turn reinforces bias, supports incorrect decisions, or causes downstream systems to fail.
Organisations therefore need a framework that makes risks explicit, keeps quality measurable, and enables teams to build reliable AI functionality. That is what QA Company built Evaluaite for.
Evaluaite: a risk-driven quality framework for AI
Evaluaite makes AI quality measurable, discussable, and manageable. It does not start from isolated issues, but from their interconnections, translating risks into concrete evaluations via eight quality pillars.
Input Robustness
Predictable behaviour when input varies: consistency, intent stability, context adherence, and retrieval reliability.
Output Robustness
Reliable, consistent output: groundedness, factual correctness, stability, completeness, and interoperability.
Bias & Fairness
Fair, comparable quality across groups, languages, and contexts.
Data Quality & Drift
Current, complete, and stable data over time.
Security & Privacy
Safe and compliant behaviour within clear boundaries: input control, data protection, access control, and policy compliance.
Explainability & Transparency
Understandable, traceable, and explainable behaviour.
Integration & System Quality
Robust collaboration with other systems, APIs, and workflows.
Performance & Reliability
Predictable performance and availability under load.
The risks discussed in this article map almost directly onto those pillars.
| Risk | Pillar / quality dimension |
|---|---|
| Output drift, catastrophic forgetting | Output Robustness (stability) |
| Retrieval drift | Input Robustness (retrieval reliability) and Output Robustness (groundedness) |
| Hallucination | Output Robustness (groundedness) |
| Fixed output format (e.g. JSON), consistent tone | Output Robustness (interoperability and stability) |
| Bias in training data | Bias & Fairness |
| Outdated or skewed data | Data Quality & Drift |
| Prompt injection, data leakage | Security & Privacy |
From risk to decision: how Evaluaite works
Within each pillar, Evaluaite links a specific risk to a quality dimension and to a measurable metric. This turns behaviour that previously only “felt” a certain way into something you can actually steer.
Take Output Robustness. The risk of hallucination maps to the groundedness dimension, measured by a Groundedness Score that shows whether output can be traced back to available context or sources. Not to prove that something is good, but to make unreliable behaviour visible before it causes problems in production.
Metrics reveal behaviour; thresholds make quality steerable. A threshold defines what behaviour is acceptable and when intervention is needed. This shifts AI quality from interpretation to decision-making: teams make informed acceptance decisions, deploy human-in-the-loop controls more deliberately, and can automate and monitor quality assurance.
The Risk Strip In just a few minutes
The starting point is the Risk Strip, a lightweight analysis that helps a team identify the relevant risks in just a few minutes:
Teams select the risks. Evaluaite takes it from there.
Evaluation-Driven Development
Because AI systems are probabilistic, the focus shifts from purely deterministic testing to evaluating stability, robustness, and predictability.
- Hard asserts
- Absolute truths
- Binary: pass or fail
- Steering by evaluations
- Working with thresholds
- Measuring how consistently and reliably a model performs
Because Evaluaite fits into modern development processes, those evaluations can run continuously in the CI/CD pipeline, so quality grows alongside the system as it evolves.
What this means for your team
Without an explicit quality framework for AI, teams often end up in discussions based on gut feeling. Evaluaite breaks that pattern.
- Discussions based on gut feeling
- Uncertainty about what acceptable AI behaviour actually looks like
- Testing effort without clear priorities
- Decisions made too late
- Better design choices
- A prioritised QA and test strategy
- Clearer decisions around acceptance, monitoring, and further development
- Quality steered from the outset
In short: insight and control over AI quality.
From “can it work?” to “can we trust it?”
This shift is not conjecture; we see it happening with our clients every day. Research firm Gartner expects that by the end of 2026, approximately 40% of all enterprise applications will contain an AI agent, up from less than 5% a year earlier.[1]
Yet research from Forrester[2] and MIT[3] shows that a significant proportion of AI pilots never make the transition to production, often precisely because reliability and governance are not in place. The question is shifting from “can it work?” to “can we trust it?”. That is fundamentally a quality question. Whether you are embedding a fine-tuned model in your software or using AI to accelerate your work: consistent, predictable, and safe behaviour requires a structured, risk-driven approach.
Evaluaite is QA Company’s risk-driven quality framework for AI systems. We support implementation across four phases:
Laying the quality foundation
Embedding it into existing workflows
Continuous evaluations in the CI/CD pipeline
Coaching towards a lasting quality mindset
Want to know more about Evaluaite?
Get in touch to find out how to make AI quality measurable and manageable within your team.
Sources
- Gartner, “Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026, Up From Less Than 5% in 2025”, August 2025: gartner.com/en/newsroom/press-releases
- Forrester, cited in: ITPro, “IT leaders are bullish on agents, but keep falling at the final hurdle”, 2026: itpro.com
- MIT NANDA (Project NANDA), “The GenAI Divide: State of AI in Business 2025”, July 2025, as reported by Fortune: fortune.com
