Prompt engineering for products needs tests because LLMs have probabilistic behavior, changing models, and no fixed input or output schema.
2
Promptomize lets teams define prompt cases as code, attach evaluators, run test suites, compare results, and inspect detailed reports.
3
The text-to-SQL use case in Apache Superset shows how teams can measure whether a prompt meets its success criteria before putting it into production.
Summary
Maxime Beauchemin argues that teams building AI features should treat prompts more like product code than casual chat instructions. LLMs differ from ordinary APIs because their behavior is probabilistic, their capabilities change quickly, and their input and output formats need to be defined by the application. His proposed workflow starts with the desired behavior, then defines test cases before writing the prompt. Teams run those cases, evaluate the results, refine the prompt and tests, and only release the prompt when it meets measurable criteria. Promptomize is his open source toolkit for expressing prompt cases as code, attaching evaluation functions, running tests across engines, supporting human review, and producing reports. He explains the approach through Preset's text-to-SQL work for Apache Superset. The talk is brief, but Beauchemin is clear that prompt quality needs ongoing measurement after release, including accuracy, runtime, cost, and comparisons between prompt versions.
LLMs make product development less deterministic than ordinary API work
Beauchemin says product builders are used to APIs with defined schemas for inputs and outputs. LLMs behave differently because they are probabilistic, have an effectively broad API surface, and change quickly as models, context-window limits, and other constraints evolve. A prompt can request structure, such as a JSON object containing a confidence score and SQL, but the application team has to specify that structure and add limits and guard rails.
Prompt engineering means building an LLM into a product
He separates prompt crafting from prompt engineering. Crafting is the everyday practice of asking an LLM questions. Engineering begins when a team uses an LLM as part of an application. It includes adding product context, specifying the answer format, limiting scope, setting guard rails, and measuring whether the result works. His example asks for SQL and a confidence score instead of an essay.
Prompt tests should be written before the first production prompt
The proposed lifecycle starts by defining the use case and desired AI behavior, including the input, context, and output. Before writing the first prompt, the team defines test cases. For text-to-SQL, a case might provide a database schema and question, then expect SQL containing particular columns or tables. The team runs the tests, evaluates results, refines the tests and prompt, and repeats until the success criteria are met.
LLM evaluations can be graded rather than treated as simple pass or fail
Beauchemin says traditional test-driven development often assumes a Boolean result, while an LLM result may be 50 percent or 80 percent right. Prompt testing therefore needs evaluation functions that can judge degrees of success. The test suite measures whether a result is useful for the product instead of assuming that every output fits a simple correct-or-incorrect boundary.
Promptomize is Beauchemin's open source testing toolkit for prompts. It lets developers define prompt cases as code, attach evaluation functions, generate prompt variations, run tests across different engines, and rank the results. He compares it with a testing library because it provides ways to express tests, run them, and evaluate their outputs.
A prompt test suite can validate generated code and text-to-SQL behavior
One example creates a prompt asking for a Python function that determines whether a number is prime and returns a Boolean. Evaluators then check cases such as whether 2, 4, and 7 are prime. Beauchemin also describes using the same style for Superset's text-to-SQL feature, where a larger suite checks generated SQL against expected properties.
Promptomize can run a suite, support human review, and produce a YAML report with the inputs, outputs, and details of each test. Teams can inspect success rates by category and calculate execution statistics such as average duration and percentile timings. They can also compare two test runs to find cases that changed from success to failure or the reverse.
The test loop continues after a prompt reaches production. A team may change the model, use a different vector database, or adopt a new prompt-generation technique. Beauchemin says those changes need to be tested against the earlier version. The team should also track how long the tests take and how much they cost when calling an API.
"Prompt engineering to me is like adding the proper context from your product, your application, specifying an answer format, bringing structure to its output, limiting scope, setting guard rails and measuring success."09:09
Who should watch
You are adding an LLM feature to an existing product and need a repeatable way to decide whether a prompt is good enough to ship.
Your team is building text-to-SQL or another structured-output feature and needs evaluations that inspect generated results against application-specific conditions.
You already change prompts or models after release and want reports that compare accuracy, runtime, and cost across versions.
