AI safety depends on the application, its users, and the harm that would follow from failure.
2
Teams should define unacceptable behavior before building, then test the system on broad, randomly selected data rather than a few impressive examples.
3
Speed is easier to manage when teams set product constraints, collect human feedback, and involve product, engineering, legal, risk, compliance, and domain specialists.
Summary
This panel treats AI safety as a product and company risk question rather than a distant discussion about machines taking over the world. Erica Greene describes risks in media, such as inaccurate election information and comment moderation that silences certain voices. Shreya Rajpal says the right safety definition changes with the application, from a news summarizer to an internal employee bot or a financial customer-support system. Both panelists argue that teams must define unacceptable outcomes before building, then evaluate systems on real, varied data. Greene warns against demos built from hand-picked examples and describes killing a headline-generation project after random testing found serious errors. Rajpal discusses human feedback, guardrails, changing model behavior, and the limits of fixed metrics. The panel also recommends involving product owners, engineers, risk and compliance teams, lawyers, journalists, and other domain specialists when deciding whether a system is safe enough to ship.
AI safety is defined by the harm an application can cause
Erica Greene says safety is not mainly about robots taking over the world. It is about the risks created by increasingly complex products, and those risks depend on the industry. In media, an automatically generated headline or summary could give people false information about election polling times. Shreya Rajpal adds that model-level safety can concern weapons or cyber risks, while application-level safety includes factuality, bias, and the responsibility of a company whose users may trust generated content. The relevant definition changes with the product and its domain.
Generative AI has widened the task and made correctness harder to define
Rajpal contrasts older recommendation systems with systems that receive abstract instructions and act across a codebase or other broad environment. A recommendation task can be narrowly described as ranking a list. An autonomous software system can inspect a repository and create a pull request, so the number of things that can go wrong is much larger. This makes it harder to define what correct behavior means and harder to determine whether the system is doing the right thing. Greene also notes that calling a model API has opened AI development to people without machine learning training.
Teams do not need to follow every new AI development
Greene describes the fear of missing out created by rapid changes in foundation models, hardware, and companies. She separates keeping up with research from building applied systems, since much of the activity around scaling models is unrelated to her daily work. Her practical example is a machine learning engineer who reads one newsletter each Friday and posts two items to the company Slack. Rajpal recommends following a small number of high-signal communities, newsletters, company blogs, and accounts, including the r/LocalLLaMA subreddit and MLOps communities. Neither panelist offers a complete information source; both favor focused attention.
Premortems and diverse review expose risks before launch
Greene recommends defining possible harms and conducting a premortem. Teams should imagine an investigative reporter trying to expose how the feature fails, then look for edge cases and worst-case outcomes. They also need people with different backgrounds in the room and an environment where concerns can be raised during any stage of the project. Her New York Times comment-moderation example shows why mechanism matters. The team built an AI-augmented system that could automatically accept comments, while rejection remained with people. This reduced the chance that biased data would silently suppress comments about race or homosexuality.
Greene says companies should consider whether AI adds risk or removes it. At Etsy, image models could help reviewers find prohibited products such as children's toys and bottles that created choking hazards. Rajpal says deployed language-model applications often have a narrow scope or a protective layer, such as a human who makes the final decision in a support chatbot. Internal employee question-answering systems are another constrained use case. These examples limit what the model can directly do and give people a role in handling uncertain or harmful outputs.
Evaluation needs human feedback and a clear account of unacceptable failures
Rajpal calls evaluation an unsolved problem for language-model applications. Traditional static metrics are difficult to apply when tasks are open-ended and even comparison with a ground-truth answer can be hard. She recommends collecting human feedback accurately and using it to improve parts of the pipeline, such as retrieval, generation, or prompts. Greene advises building a rubric with tiers for easy cases, ambiguous cases, and severe failures. A demo should be tested on a broad set of examples, with engineers and other staff reviewing outputs rather than admiring a handful of selected successes.
The speed and safety trade-off starts with product constraints
Rajpal says teams should define their inviolable conditions during product scoping. They should state what behavior they need from the system, ground that requirement in data, and measure how often it is violated. There will be failures in a nondeterministic system, so the acceptable level depends on the product and its users. A hallucination rate that might be acceptable for one application may be unacceptable for a financial institution. Greene adds that reputation matters. A company whose brand depends on accuracy may decide that a language-model feature is unsuitable, as in the example of a government chatbot that gave incorrect business-permit advice.
Hosted model APIs make system behavior harder to control
Rajpal says teams lose some control when they rely on a model hidden behind another company's API. Repeated runs of an LLM-as-judge evaluation can produce different results, and a provider can change a model without the application owner controlling the update. Earlier machine learning systems offered more consistency because teams owned the training data and could control settings such as the seed and inference stack. Greene suggests using language models to annotate data and then training a traditional neural network when a task can support that approach. Rajpal cautions that smaller models may struggle more with out-of-distribution examples.
Safety decisions require product, technical, legal, and domain voices
Rajpal names the product owner, risk and compliance teams, and the developers who implement or evaluate mitigations as participants in risk decisions. Greene adds that teams need to understand the regulations affecting their industry and consult lawyers. In media, she includes journalists and editorial partners because they are good at defining precise questions and finding holes in a system. She describes writing product constitutions with specific language. The panel's view is that safety cannot be left to the engineers who build the model or to a single generic metric.
