Comparing Open-Source LLMs for Customer-Service Conversation Summaries

Background

#

Testing whether locally-hosted, open-source models are good enough to take a customer-service conversation — after it’s been converted from speech to text — and produce a usable summary. All of this ran on consumer-grade GPU hardware, since that’s what’s actually practical for a setup like this.

Models tested

#

Ten models pulled via Ollama:

Evaluation approach

#

Each model was scored across three dimensions: how accurately it captured the customer’s underlying intent, how well it documented the steps the agent actually took, and how precisely it identified the core issue. Scores were gathered from a mix of perspectives — customer-service staff, ML practitioners, and end users — and averaged into a final result.