What happens when you take a chatbot that already serves thousands of customers per day in four different languages and try to deliver an even better experience using broad language models? Good question.
It’s well known that evaluating and comparing LLMs is complicated. Reference data sets can be difficult to obtain, and metrics like BLEU are imperfect. But these are largely academic concerns: how do industry data teams address these issues when incorporating LLMs into production projects?
In my job as a conversational ai engineer, I do exactly that. And that’s how I ended up center stage at a recent data science conference, giving the (optimistically titled) talk “No baseline? No benchmarks? No big deal!” Today’s post is a summary of this, with:
- The challenges of evaluating an evolving LLM-driven PoC versus a working chatbot
- How we use different types of tests at different stages of the PoC to production process
- Practical pros and cons of different types of tests