Promptimize applies test-driven development ideas to prompt engineering by running fixed prompt cases against evaluation functions.
2
A stable test set lets teams compare prompts, models, parameters, cost, speed, and accuracy for their own product use cases.
3
Maxime Beauchemin argues that prompt engineering and vector databases are currently the pragmatic way for small teams to work with language models, while fine-tuning remains expensive and difficult.
Summary
Maxime Beauchemin explains why language model APIs need a testing discipline closer to software engineering. Traditional APIs have documented schemas and mostly deterministic outputs. Prompts are open-ended, probabilistic, and sensitive to context, model choice, and parameters. His open-source tool, Promptimize, lets developers define prompt cases and evaluation functions as code, run them as a suite, and compare reports across prompts, models, and settings. His main example is text-to-SQL, where generated queries can be tested against expected answers more easily than open-ended writing. The conversation also covers vector databases, fine-tuning, model costs, privacy, and the business question of whether an AI feature creates enough user value to justify its API cost. Beauchemin is pragmatic about current limitations. He recommends keeping a representative, deterministic test set as an anchor while models, context windows, prices, and tooling change.
Language model APIs make consistency and measurement difficult
Maxime Beauchemin contrasts ordinary REST APIs with language model interfaces. Traditional APIs have clear input and output schemas, strong documentation, and deterministic behavior. A language model prompt can include whatever context the developer thinks will help, in whatever structure seems intelligible, and the output remains probabilistic. Developers can request formats or word limits, but they still need to determine whether a prompt is better than its predecessor and whether it works across specific sub-use cases.
Text-to-SQL provides a practical evaluation problem
Beauchemin describes his own use case, text-to-SQL. A user asks a natural-language question such as which countries have the fastest sales growth, and the model generates SQL against a database schema. The question space is open-ended, but the result can often be checked more directly than an essay. The query can run, return the relevant answer, mostly succeed, or fail. That makes it possible to build a test set across databases with different naming conventions, schemas, and levels of complexity.
Prompt cases bring test-driven development into prompt work
Promptimize lets a developer define a collection of input prompts and evaluation functions, then run them systematically. A test suite might contain hundreds or thousands of questions with expected results. The developer can change the prompt, run the suite again, and obtain a success report. The same cases can also be run against different models, such as different GPT versions or open-source models, so the comparison is tied to the product's own use case rather than a general benchmark.
The toolkit records model settings and operational costs
The conversation describes a matrix of prompts, models, and parameters such as temperature. Promptimize can report how each combination performs, where one model succeeds while another fails, and how much each run costs. It records information such as response time, token usage, and OpenAI-style parameters. Reports can be written to YAML, summarized through the command line, or loaded into a database for more detailed analysis. Developers can add categories and weights when some prompt cases matter more than others.
Vector retrieval can be tested against simpler prompt designs
Beauchemin says vector databases may help when the available context exceeds the model's prompt window. A team might embed documents, retrieve nearby content, and construct a prompt from that context. Promptimize can compare this design with a simpler prompt and measure whether retrieval improves the team's actual use case. The same approach applies while changing embeddings or document formats. The test suite provides a way to evaluate retrieval decisions instead of assuming that a vector database is automatically better.
A stable test set is an anchor while the technology changes
Models, context limits, prices, open-source options, and vector databases are changing quickly. Beauchemin argues that a carefully built collection of prompt cases may be the most durable asset in this environment. The test cases describe the product's real requirements, so they can be reused as models and prompts change. A team can add cases as it learns more about users, while keeping the evaluations written as code and therefore more predictable.
Fine-tuning can add complexity before it adds value
Beauchemin discusses fine-tuning as another path beyond prompt engineering. A model could potentially absorb a company's schemas, documentation, pipelines, and other context, but training order matters. Adding new material to an already trained model can cause catastrophic forgetting or disrupt existing behavior. Training from scratch may require substantial GPU time and cost. For a small organization moving quickly, he recommends starting with prompt engineering and vector databases, then watching for better fine-tuning methods and economics.
The discussion moves from technical quality to product economics. Teams need to measure speed, cost, and the usefulness of generated output for their own users. Beauchemin suggests using feature flags, private betas, user interviews, and product logs such as prompts, generated SQL, and thumbs-up or thumbs-down feedback. An AI feature still needs to create enough perceived value to justify its API cost. A successful demo does not answer that business question.
"Promptimize is less about the tool, the Python little toolkit and CLI, but more the idea of how do we bring a more deterministic approach to interacting with these LLMs."Maxime Beauchemin1:10:47
Who should watch
You are building a product feature on top of a language model and need repeatable checks for prompt changes.
Your team is comparing models, retrieval setups, or parameters and wants results tied to its own use cases.
You are considering fine-tuning but need a pragmatic way to evaluate whether prompt engineering is enough for now.
