GetYourGuide prioritizes LLM opportunities where they can improve travel discovery, supplier onboarding, or slow operational work.
2
The team uses human review, automated checks, documentation, and model observability to reduce the risks of inaccurate or misleading output.
3
For free-text evaluation, GetYourGuide uses GPT-4 as a scoring model, then sends threshold violations to humans for inspection and iterates on either model or prompts.
Summary
Tina Treimane and Meghana Satish describe how GetYourGuide decides where LLMs fit and how it puts them into production. Tina frames the opportunity around conversational search, better understanding of the company's growing inventory, and faster supplier onboarding. She also explains the risks, including misleading descriptions, unclear sources of truth, and loss of customer trust. GetYourGuide's principles include tight scoping, reliability checks, transparency about model limits, human oversight, and ongoing monitoring. Meghana explains why the team uses LLMs when labeled data is hard to obtain: they allow fast experiments and can help generate data for later in-house models. For a supplier onboarding project, the team evaluates generated text with a GPT-4-based scoring system, custom prompts, thresholds, and human inspection. Arize monitors predictions and metrics after launch. The approach is useful, although Meghana says it still produces false positives and needs continued prompt work.
GetYourGuide looks for LLM opportunities where existing work is slow or limited
Tina says the team asks which tasks were previously blocked by technical, budget, or time constraints. It also asks how the product and operations would be designed if they started in the era of LLMs, and which parts of the business depend on people in ways that will not scale. At GetYourGuide, the main areas include conversational search and trip planning, understanding a rapidly growing inventory well enough to shape user interfaces and advertising, and speeding up operational work such as supplier onboarding. The team wants LLMs to make onboarding smoother and accelerate it substantially.
A plausible generated description can still damage customer trust
Tina describes an internal GPT experiment that summarized activities and identified their selling points. The results looked promising overall, but one summary of the Changing of the Guard in London said customers could get "up close and personal with the guards." That was not what the experience offered. Putting that description in front of customers could lead them to book the wrong experience and harm GetYourGuide's brand. Tina also warns that interactions between LLMs and supervised models can make it difficult to identify the source of truth or understand the final customer experience.
GetYourGuide combines product controls with human oversight
The team's principles begin with tight product scoping, data product practices, and guardrails that show whether a project is on track. Tina says reliability and safety require pre-launch checks and bug bashes that combine human feedback with automated checks. The team documents model limitations, biases, and possible errors so that other teams can use the model with context. Human reviewers help detect unusual outputs and patterns. The team also plans automated checks, evaluations, and observability to catch regressions after launch.
LLMs help the team experiment before it has a large labeled dataset
Meghana says the largest bottleneck in GetYourGuide's data science pipeline is obtaining a curated, high-quality dataset of sufficient size. LLMs let the team iterate and launch experiments quickly, fail faster, and generate learnings sooner. If GetYourGuide later decides to build an in-house model, those experiments can also help collect data over time. The company has seen more teams use LLMs, with people from different backgrounds discussing their experiences, and people with coding experience can deploy data products to production.
LLM use brings cost, API, and output-control problems
Meghana says fine-tuning OpenAI models can become expensive when costs accumulate across teams. Fine-tuning jobs may also wait in a queue for a considerable time, sometimes days. API changes can alter model behavior and the user experience, and the team cannot fully control the output. Hallucinations are another problem. Standard metrics such as F1 score and accuracy work when labeled data exists, but they do not solve the problem of evaluating free text when a suitable dataset is unavailable.
The supplier onboarding evaluation uses one model to score another
For AI-based supplier onboarding, GetYourGuide wants suppliers to submit activities without a GetYourGuide employee acting as an intermediary. The project uses historical pairs of raw supplier text and freelancer-refined text to fine-tune a model. To evaluate new refined text, the team moved beyond human evaluation, which was too slow to scale. Its evaluation framework uses prompts that define how scoring should work and GPT-4 to assess the main model's output. The scoring model can explain its reasoning and sometimes catches mistakes that a human evaluator missed, although it can also produce unnecessary false positives.
Thresholds turn model evaluation into an operating loop
The evaluation produces a score from 1 to 10. When a live data point falls below a configured threshold, it is sent for manual inspection. The team can then change the main generation model or the scoring model, and repeat the cycle until the performance is acceptable. Meghana says the setup needs substantial prompt iteration. Arize is used to monitor data and predictions, configure custom metrics, set thresholds and alerts, and route flagged examples to human reviewers.
The team controls production cost through estimates, monitoring, and caching
Tina says GetYourGuide estimates how costs might scale before launch. The team can use caching and other optimizations to reduce the cost of requests. It also monitors the cost of different services and investigates spikes as they appear. That review can reveal unnecessary use cases that should be removed. Tina describes this as a way to keep costs within the normal range rather than allowing production usage to grow without review.
"It is just one LLM basically evaluating the performance of another LLM, but more structured and in a way that follows best practices around LLM benchmarking."Meghana Satish14:35
Who should watch
You are deciding whether an LLM is appropriate when labeled training data is limited or expensive to create.
Your team is shipping generated text to customers and needs practical controls for human review, evaluation, and monitoring.
You need to estimate LLM costs and handle model changes before an experiment becomes a production service.
