AI safety has expanded beyond accuracy because models must keep working when deployment data and environments change.
2
Data quality, model validation, and model improvement are tightly connected, so teams need to inspect the full AI lifecycle.
3
Standards and regulation can set a baseline for trust, but companies first need to make their models work in real applications.
Summary
Petar Tsankov describes how AI safety grew out of a decade of research focused on model robustness and reliability. Earlier machine learning work optimized accuracy against fixed datasets. Production systems face changing environments, biased or incorrect labels, missing data, and adversarial inputs. Petar argues that teams cannot assess a model separately from its data because model failures often reveal gaps in data coverage or curation. He explains how LatticeFlow AI developed from research on mathematically verifying properties of neural networks into a broader product for checking data quality, model performance, and blind spots. The conversation also covers independent assessments for third-party models, since vendors may not share training data with customers. Petar sees standards such as ISO 5259 and ISO 42001 as useful ways to define practices and communicate across engineering, risk, and executive teams. Regulation will set limits for high-risk uses, but business needs and the cost of model failures will drive adoption.
AI safety grew because accuracy on fixed datasets was not enough
Petar Tsankov says AI research spent roughly a decade optimizing accuracy against fixed datasets such as ImageNet. That approach produced highly accurate models in the lab, but deployment introduces changing environments and mission-critical consequences. A model must keep working when its surroundings change, and its decisions must not harm people in areas such as business, medicine, and financial services. He uses trust as the broader term because the problem includes reliability, ethics, and bias. The gap between a model that works on laboratory data and one that works reliably in production is where AI safety becomes necessary.
Third-party models make transparency part of the trust problem
Petar says machine learning is shifting from teams building every model themselves to applications built on models supplied by other organizations. Customers may not know how those models were trained, what data was used, or which biases were introduced. This became visible to a broad audience when generative models produced unexpected or offensive results. The same issue applies to specialized models used in serious business processes. A customer needs to know whether a third-party model performs reliably for its intended use, especially when the model affects consumers or manages large financial exposures.
Traditional supervised models still carry much of the industrial risk
The attention around generative AI can make older machine learning systems seem less important, but Petar says many valuable applications still depend on supervised models built for specific tasks. These models can run on autopilot and may support business-critical operations in banks, medical systems, and other enterprises. He says a large share of such models fail to reach production because of reliability problems. The attraction of generative AI does not remove the need to make conventional models work reliably. Petar expects companies to return attention to these systems as they focus on the value that machine learning delivers.
LatticeFlow AI began with attempts to prove neural-network properties
Petar traces LatticeFlow AI back to research on neural-network verification. Researchers found that small changes to a model's inputs could cause unexpectedly large changes in its outputs, even when the model had high accuracy. His team worked on expressing properties that a neural network should satisfy and verifying, in some cases mathematically, whether it satisfied them. Petar is careful about the limits of this approach. Machine learning handles tasks that cannot be fully specified in code, so no one can completely describe everything a model should do. Verification became one part of a wider effort to understand what is needed for reliable industrial systems.
Data problems and model problems have to be investigated together
Petar says industry initially approached research teams asking for model verification, only to find that production data differed from the data used during development. Labels could be wrong, the data could contain bias, or an important part of the deployment distribution could be missing. LatticeFlow AI therefore expanded from robustness checks toward data quality, model validation, and iterative improvement across the AI lifecycle. Its model blind-spot analysis looks for parts of the data distribution where performance is poor. The result can point teams toward missing data, data curation work, or synthetic data generation. Petar treats data work and model work as connected activities.
Independent assessments can protect vendor data while giving customers evidence
Petar explains that model vendors may not share their training data or code with the customer because the data is part of their intellectual property. In the Swiss bank example, an independent provider could access the vendor's data, training code, and model, run the checks, and report the results to the bank. This arrangement gives the bank evidence about the model without handing over the vendor's private data. Petar says black-box testing can check some properties, such as aspects of robustness, but it cannot establish data relevance or representativeness when the evaluator has no information about the training data.
Standards turn broad trust requirements into checks teams can apply
Petar says standards help bridge the gap between broad requirements and technical work. Terms such as representative and relevant data sound clear until a team has to decide what they mean for a particular application. During an assessment, the evaluator may need to ask the customer which data is relevant and then turn that answer into concrete checks. Petar mentions ISO 5259 for data quality, ISO 24029 for model robustness, and ISO 42001 for AI management systems. He describes ISO 42001 as covering processes such as documenting models and managing model upgrades, while other standards address project-level data and robustness checks.
Regulation should limit unacceptable uses while industry proves ordinary systems work
Petar expects regulation and standards to develop along parallel tracks. The EU AI Act addresses legal classification and stricter requirements for higher-risk use cases. Standards describe risks, mitigations, and practices that can be applied to technical projects. He argues that companies should first establish that their systems work and then use standards and regulation to set a baseline. He also supports drawing clear limits around uses such as mass surveillance. For mission-critical systems, a trial-and-error approach is unacceptable. Commercial risk already pushes some vendors to validate models because customers will not use systems that could lose money or damage trust.
Some neural-network risks are design problems that guardrails cannot fix
Petar says progress is possible for specialized models in controlled environments, but several limits remain. Long-tail cases are difficult, models may fail to generalize across the full deployment distribution, and small input changes can produce unexpected outputs. He distinguishes benign environments from adversarial ones, where an attacker controls the inputs and actively tries to break the system. Drawing on a point from security expert Apostol B. at NIST, he compares this with secure design in traditional security. If a weakness is built into the design, adding another model or guardrail may not solve it. Teams need to understand which risks have no complete solution and account for them explicitly.
"There is a very big gap of building something that works on your data in the lab to something that you actually then deploy to solve a mission critical task and it actually works reliably in these new environments."Petar Tsankov08:08
Who should watch
You are responsible for a supervised machine learning model that must run reliably in a bank, medical system, or another high-impact application.
Your team uses a model from another vendor and needs an independent way to assess it without receiving or exposing private training data.
You need to connect data quality, model testing, and emerging AI standards in a process that engineers and risk teams can both understand.
