Prompt engineering often relies on repeated manual testing, personal experience, and informal rules such as telling a model not to be lazy.
2
AI-assisted prompt editors can inspect prompts and their outputs, then suggest changes such as removing negation, splitting compound instructions, or shortening the prompt.
3
Synthetic test cases and formal evaluation can support more advanced prompt optimization, but many production teams lack large labeled datasets and reliable metrics.
Summary
Alex Cabrera argues that prompt engineering still depends too much on guess-and-check. Developers write a prompt, try it on a few examples, inspect the output, and revise it based on experience. He compares this with writing and programming, where tools such as Grammarly and Copilot provide direct assistance. His demo shows a prompt editor that generates example inputs, runs a model, identifies possible improvements, applies edits, and produces new outputs. The suggestions include replacing negation with a positive instruction, returning structured JSON, splitting compound instructions, and shortening prompts to reduce token costs. The tool can also generate additional examples and add labels for deeper evaluation. Cabrera is careful about the limits of automated optimization. Prompt evolution works best with large labeled test sets and useful metrics, which many teams do not have. His current design therefore focuses on lightweight guidance for the early, creative stage of prompt development, with more advanced optimization as teams collect better test data.
Prompt engineering is still a manual loop inside complex AI systems
Cabrera says that even sophisticated text chains and agentic graphs eventually reduce to individual prompts connected to models, function calls, or databases. Developers usually start with a prompt, test a few cases, add failures from production, and revise the text. Experienced prompt engineers improve through repeated practice and by learning model-specific habits, such as adding emphatic wording or changing how JSON instructions are written. The process depends heavily on tribal knowledge and repeated guess-and-check.
Prompt editors could provide the assistance that code and writing already have
Cabrera compares prompt engineering with writing and programming. Grammarly can point out writing problems, and Copilot can help with code, but prompt work generally happens in a text editor or playground without comparable guidance. His project explores whether an AI assistant can identify common prompt patterns and suggest changes while the developer is working. He presents the project as an early experiment and asks the audience to consider what useful assistance should look like.
The demo starts by generating examples and inspecting live model outputs
The demo creates a prompt that extracts numeric scores from movie reviews. The system generates possible example inputs, then runs a selected model, initially GPT-3.5 Turbo, on those inputs. The developer can edit the prompt and immediately see new outputs. Cabrera notes that users can paste in their own domain-specific data when automatically generated examples are not useful enough.
Editors can turn common prompt heuristics into direct edits
The interface displays suggestions beside the prompt and its outputs. One editor detects negation in the instruction "do not return anything but the number" and recommends telling the model what to do instead. Applying the suggestion rewrites the instruction. Other editors suggest returning JSON with an output key or using separate sentences instead of compound instructions. The system saves each version and reruns it so the user can inspect the effect of the change.
Synthetic examples can expose cases that the initial test set misses
After the initial examples become too easy, Cabrera adds a review saying that the movie was awful but contains no numeric score. The model returns zero, while the intended result is null. He updates the prompt to specify what should happen when no number is present. The tool can also generate similar reviews, including examples without numbers, so the test set can grow around observed failure cases.
Evaluation can be added after the initial prompt exploration
Users can attach labels to examples and calculate formal metrics for comparing models or prompt versions. Cabrera connects this feature to his work on Zeno, an evaluation platform for detailed analysis of model behavior. He presents formal evaluation as a later stage that can follow quick interactive testing, rather than requiring every prompt-development session to begin with a large evaluation framework.
Prompt suggestions need to respond to new versions and changing model behavior
The system saves each prompt version and updates its suggestions using the current prompt, inputs, and outputs. Some editors inspect whether the output matches what the prompt requested, so adding new inputs can change the recommendations. Cabrera says changing underlying models is harder to handle. Even temperature zero does not guarantee deterministic OpenAI outputs because of mixture-of-experts behavior, so a future system might track how often outputs change for the same model.
Advanced prompt evolution needs data that many production teams do not have
Cabrera discusses research using genetic algorithms to mutate prompts and test the variants. He finds the approach promising, but says it assumes a large test set with ground-truth labels and a metric that reflects the quality people care about. Many teams instead begin with an idea, implement a prompt, and refine their mental model by looking at outputs. His proposed path is to offer simple heuristics first, then use the accumulated examples and iterations for more advanced optimization.
Human judgment may be a better default than a metric
Cabrera says he and his collaborator disagree about whether metrics and labels should be enabled by default. His preference is to begin with human judgment while the developer iterates, because many applications lack a metric that tracks the desired quality. The current system supports one evaluation metric, character overlap, or CHRF, between a label and an output. He also says a metric can be used when the specification and test cases are defined well enough.
"When you ask really good prompt engineers how they got good at prompt engineering, it's a lot of just practice, a lot of just trying it over and over again and learning the nuances of exactly how these models perform."02:30
Who should watch
You are building LLM chains or agents and currently revise prompts by hand after checking a small set of examples.
You want prompt suggestions inside a development workflow, especially for output formatting, instruction wording, and test-case creation.
Your team is considering automated prompt optimization but needs to understand why labeled data and a useful quality metric may be missing.
