Thomas Capelle improved a function-calling assistant from 79% accuracy with prompting to 98% after fine-tuning 600 high-quality examples with parameter-efficient methods.
2
Boris Dayma argues that teams should avoid training large models from scratch when an API, an open-source model, retrieval, or fine-tuning can meet the need, then prototype with small models and short feedback loops if training is necessary.
3
Jonathan Whitaker found that retrieval and careful data inspection mattered more than simply scaling model size in the Kaggle LLM Science Exam competition, while Robbie McCorkell showed how model-generated prototypes can expose hidden biases and unsafe behavior.
Summary
This meetup contains four short sessions about building and inspecting LLM systems. Thomas Capelle walks through a local voice assistant that calls tools. Prompt format and model choice raised accuracy substantially, while 600 carefully filtered examples and parameter-efficient fine-tuning raised it further. Boris Dayma gives a practical recipe for large-model training. He recommends using existing APIs or open-source models first, then trying retrieval or fine-tuning before considering training from scratch. When training is unavoidable, he recommends small prototypes, logged experiments, application-specific benchmarks, and inference testing from the start. Robbie McCorkell presents Leap Labs' prototype-based interpretability work. Generating inputs that strongly trigger a model's output can reveal spurious features, class entanglement, and unsafe associations. Jonathan Whitaker uses the Kaggle LLM Science Exam to discuss prompting, retrieval, synthetic data, fine-tuning, and evaluation. His examples show that retrieval quality and data inspection often require more work than the model training itself.
Prompt format and model choice determine whether tool calling works
Thomas Capelle starts with Llama 2 and a test set of 60 queries, where an unstructured prompt produces about 30% accuracy. Listing available functions and showing the desired weather call helps, but the model still adds unwanted text. Switching to a chat-tuned model and using its required system and end-of-instruction tokens raises exact accuracy to 11% in one intermediate setup. Adding the expected function-call format to the prompt then raises accuracy to 75%. The lesson is practical: the model must match the task, and the prompt must match the model's training format.
Small, high-quality fine-tuning data can close the remaining tool-calling gap
Thomas creates examples of user requests and the exact assistant function calls they should produce. He uses ChatGPT to generate more examples from a schema, then filters them because fewer high-quality examples are better than many bad ones. About 600 examples bring Mistral to 98% accuracy on his test. He uses parameter-efficient fine-tuning with LoRA, which adds trainable adapters rather than updating every model parameter. He describes LoRA as the default choice for its performance and hardware trade-off, with QLoRA useful when GPU memory is especially limited.
Teams should exhaust simpler options before training a large model
Boris Dayma says training a large model brings higher compute costs, longer iteration times, extra instability, and no guarantee of success. He recommends starting with an API when possible, then checking open-source models such as Stable Diffusion, Llama, or Mistral. If an existing model is close to the need, prompt engineering, few-shot examples, retrieval-augmented generation, and fine-tuning should come before training from scratch. Fine-tuning can improve retrieval of knowledge already present in the base model, but it cannot easily add knowledge the base model never learned.
Large-model work should begin with small prototypes and logged experiments
When training is unavoidable, Boris recommends reproducing a simple public baseline before trying new architecture ideas. He scales model size in roughly three-to-ten-times steps, tests data loading, validation, checkpointing, and inference early, and adds a parameter for every experimental choice. He records conclusions with the graphs that support them in Weights & Biases reports. His advice is specific to experimental work: use an application-specific benchmark, test the model as it will be used in production, and reserve creative changes for small runs where failures are cheap to understand.
Generated prototypes can expose features and biases a test set misses
Robbie McCorkell describes generating an input that would strongly produce a chosen output. For an image classifier, the resulting prototype can reveal what the model treats as important. In one example, the classifier associated running with grass, and it was less likely to identify people running indoors or on asphalt. Other prototypes suggested that darker skin tones were associated with dancing or fighting, and testing confirmed a 10% higher likelihood of those classifications. Leap Labs uses prototypes, entanglement charts, and isolation images to help users form hypotheses about model behavior before deployment.
Prototype-based interpretability is still exploratory for language models
For language models, Leap Labs is exploring prototypes for both single-token and multi-token outputs. Robbie shows GPT-2 inputs that make the model produce the token 'boy', then compares the associations found through generation with those found from embedding distance. 'Basketball' appears as an association despite being far away in embedding space, suggesting that embeddings alone do not show the whole picture. For Llama 2, generated prompts can push the model toward unsafe outputs such as racist jokes or instructions for wrongdoing. Robbie says the open research problem is finding more naturalistic inputs that could plausibly occur in the real world.
Retrieval quality drove results in the Kaggle LLM Science Exam
Jonathan Whitaker explains that the competition used science questions generated from Wikipedia and required offline models. Basic prompting produced useful results, but adding retrieved context raised a local model from about 70% to near 80% accuracy. The strongest submissions used several retrieval methods, reranking, and sometimes multiple models. The competition showed that retrieval is more than adding a vector search step. Participants had to inspect the retrieved passages, match the validation distribution to the test set, and account for answer wording that often came directly from the source article.
Data inspection and short feedback loops matter more than elaborate training
Jonathan repeatedly recommends decoding batches immediately before they enter the model and manually inspecting them. This catches missing end-of-sequence tokens, incorrect prompt formats, omitted context, truncation errors, and accidental inclusion of the answer. He also found public web datasets and Wikipedia dumps full of parsing problems, including missing values and formulas. His broader workflow is to generate a small amount of synthetic data, review and filter it, fine-tune a small model, and repeat. Large models should come only after the idea works in a fast local loop.
