Financial institutions need an inventory of their machine learning work before they can assess risk, controls, or investment value.
2
Charles Radclyffe separates governance into regulatory compliance, risk and safety, and ethics, because each area needs a different response.
3
Reproducibility, data lineage, version control, and auditability matter more for governance than forcing every model to produce a human-readable explanation.
Summary
Charles Radclyffe describes his experience leading AI at Fidelity, where machine learning projects had grown through side-of-desk experiments, senior requests, and work without clear business cases. The first governance task was to discover what existed, decide which projects mattered, and stop low-value work. A regulatory survey then helped the team collect a fuller inventory and understand its controls. Charles argues that financial institutions need a shared view of models in development, pre-production, and production, with risk assessments matched to the possible harm. He separates regulatory compliance from risk and safety and from ethics. Engineers have the largest role in risk and safety, including data quality, bias, transparency, version control, and reproducibility. Ethics concerns the purposes and applications of technology, which require discussion rather than an engineering fix. He is skeptical of treating explainability as the main governance goal and gives greater weight to repeatable results, data audits, and lineage.
AI governance starts by finding the work that already exists
At Fidelity, Charles Radclyffe was brought in to create a strategy for machine learning and an operating model for delivering it. The organization had automation work, plus machine learning projects that developers pursued as side activities or passion projects. Some senior people also commissioned projects on a whim. The first inventory found 15 or 20 projects, but people were reluctant to disclose experiments that lacked a formal mandate. The total eventually reached 26 or 27 initiatives, which forced leadership to compare the work in progress with projects that mattered to the firm. Charles's conclusion was direct: time spent on low-value work should stop so people can focus on more valuable projects.
A regulator can give an organization permission to see its real model risk
After several months of struggling to understand the full scope of activity, Charles's team used a survey from the UK Financial Conduct Authority and the Bank of England to ask for a complete inventory. The regulatory request changed the response from cautious to forthcoming, since employees now had an external reason to disclose their experiments. Within three weeks, the list grew to 60 projects. The team could then see which risks and controls existed and which did not. Charles expected regulators to repeat this work because they were concerned about systemic risk across financial institutions, so he argued for preparing an inventory and risk-management process before every annual request became a fire drill.
A single inventory must cover models across different platforms
Charles describes an inventory as the basic governance layer. It should show which models are being experimented with, which are in pre-production, and which are in production. A large organization may use many platforms and development environments, so a governance tool tied to one platform cannot provide a firm-wide view. Once the inventory exists, the organization can apply different risk-assessment frameworks to models with different levels of potential harm. Developers should follow those frameworks, and the organization should be able to audit that they did so. Charles also connects governance with collaboration, provenance, productivity, and visibility into problems such as poor data handling.
Governance should be proportional to the possible harm
Charles does not support regulation for its own sake, and he does not think every system needs the same controls. He uses autonomous vehicle development as an example of a high-risk activity where teams should act before future rules arrive, because a failure could kill people. Optimizing a video-conferencing network has a different risk profile, since a failure would not have the same physical consequences. The organization should decide its risk appetite and put stronger controls around higher-risk work. He expects best practice to develop over time, as it has in software development, where enterprise work generally comes with expectations around governance and documentation.
Financial services became more cautious after the financial crisis
Charles says large financial institutions are very cautious about machine learning, especially where systems make investment or customer decisions without a human in the loop. He links that caution to the reputational damage caused by the financial crisis. Compliance moved from being viewed as a separate department that stopped commercial work to becoming part of how the industry operates. He gives chatbots as an example of a problem: an employee speaking for a bank normally needs approval and scripted messages, while a system generating language on the fly could produce unapproved statements. He thinks financial services may now be overly cautious, but he sees strong controls around trading and market activity.
The most serious failures can come from how technology is used
Charles worries that organizations can focus too much on engineering quality and too little on the application of a system. He uses Cambridge Analytica as an example where the issue was not a failure by incompetent engineers. The concern was the morally questionable use of capable technology. In financial services, he sees the main risk as becoming too slow and cautious, while in other settings the risk may be a harmful application. Engineers can improve the technology they build, but they may not control how another party uses it. That is why governance needs a way to discuss intended use, responsibility, and possible harm beyond technical performance.
Reproducibility is a better governance target than explainability alone
In response to a question about explainable models, Charles argues that the industry may be using explainability as a label for a much wider concern. He does not know of a current regulation requiring it. The European Commission white paper he cites discusses documenting work and being able to reproduce a model's results later, even after changes to the software or data. Charles considers that more important than producing a simple account of why one prediction occurred. He also thinks teams should weigh factors such as energy consumption and whether a less expensive approximation is sufficient. His preferred foundation is an audit of the data, model version control, and the ability to repeat results under the same conditions.
Governance has three separate domains with different owners
Charles separates regulatory compliance, risk and safety, and ethics. Compliance means meeting the rules that apply, with public-policy work and lobbying as possible activities for organizations that want to influence future rules. Risk and safety includes explainability, bias, discrimination from biased data, and transparency. Charles treats these as engineering problems that can be addressed through standards, processes, tools, and better practice. Ethics covers questions such as when facial recognition should be used, the purposes of systems, and issues such as killer robots or the gender of chatbots. Those questions need conversation with people. MLOps mainly addresses the risk-and-safety side through auditability, traceability, and reproducibility.
"The problem is that we need to go through an exercise to discover what are the things that were really important to the firm, but also what are the things that would be important but we're not doing yet."Charles Radclyffe12:27
Who should watch
You work in a bank or another regulated firm and machine learning projects are appearing outside the formal delivery process.
Your team needs to build an inventory, risk assessment, or audit trail across several model-development platforms.
You are trying to decide how much explainability, documentation, and human review a model actually needs.
