Quillmere: an AI governance layer on an unchanged system

Quillmere is an illustrative case, not a real incident. A chemical site puts a complete AI governance layer in place: policy, risk procedure, register. A vendor’s monthly model update then changes how control room alarms are ranked, and a reactor temperature alarm reaches the operator too late. The independent trip stops the runaway.

All names of companies, sites, equipment and people in this case are invented. Any resemblance to a real organisation, site or event is coincidental.

What happened?

Quillmere runs exothermic batch reactors. After adopting several AI tools, the board asked for AI governance. A consultant wrote an AI policy, an AI risk procedure and an AI register, aligned with ISO/IEC 42001. An internal audit found the layer complete. The operating management system itself did not change.

One of the registered tools is a vendor’s AI system that ranks process alarms in the control room, to help operators through alarm floods. On each reactor, the risk assessment credits a high-temperature alarm with operator response as one protection layer, and a high-high temperature trip as another, independent one.

The vendor updates the model every month. During a restart after maintenance, the latest version ranked the reactor high-temperature alarm low among the start-up alarms. The operator responded late. The instrumented trip shut the reactor down before the runaway developed. No one was hurt.

What did the governance layer miss?

The investigation found the AI policy fully complied with. The tool was registered, risk assessed and approved. What had not changed was the system the plant runs on.

• Management of change defined a change as hardware, procedures or setpoints. The monthly model updates never went through it.
• The barrier register did not record that an AI now sat in the decision path of the alarm and operator response layer.
• Operators were told about the tool by email. They were not consulted before it went in, and nobody asked how they used the ranking during start-ups.
• The vendor contract did not require notice of model changes.

What would M-OPS OMI have asked?

M-OPS OMI goes through the requirements Quillmere already had. Management of change needed modifying, so that a model update counts as a change. The barrier register needed modifying, to record the AI in the decision path of the alarm layer and to review whether that layer still counts as independent. The workforce consultation requirement applied as it stood; it was not applied to the tool. A new element was needed: contractual notice of vendor model changes, each one triggering management of change.

If your AI governance sits beside your management system, contact me.