Case Study · Model Training & Fine-Tuning
Fine-Tuning an Open-Source LLM for Domain-Specific Service Triage
A privately hosted, fine-tuned language model that understands product-specific terminology to classify, prioritise, summarise and draft replies for service tickets, at a fraction of the cost of a general-purpose API.
The problem
Service tickets are full of model numbers, fault codes and technician shorthand. General-purpose LLMs misread this language and send tickets to the wrong queue. Sending customer and equipment data to a public API was also ruled out by the client's data policy, and per-token costs rose with ticket volume.
Custom training and domain adaptation
Data preparation
Collect historical tickets with their final category, priority and resolution notes. Redact personal data, remove duplicates and balance rare but critical categories such as safety issues.
Baseline first
Measure a prompted general model on a held-out test set, so every improvement from fine-tuning is measured against a real baseline.
Fine-tuning
Fine-tune a 7–8B-parameter open model with LoRA/QLoRA on instruction pairs covering classification, priority, a structured summary and a suggested first reply.
Evaluation harness
Measure accuracy and F1 per category, how often priority is under-estimated, summary faithfulness (human review plus automated hallucination checks), latency and cost per ticket. Regression tests run on every new checkpoint.
Optimisation
Quantise the model and serve it with a high-throughput inference server so it runs on modest GPUs with low latency.
Production deployment
Deploy inside the client's own cloud account, integrated with the helpdesk. Agents see suggestions directly in their ticket view.
Continuous improvement
Every agent correction is logged as new training data. Drift monitoring flags new products or fault types and triggers scheduled retraining.
What the client measures
Technology
Have a similar challenge?
Talk to our team about what this could look like for your business.