Explainability in CTCAE AI: Seeing the Reasoning, Not Just the Result

If you cannot see why the model said "Grade 3," should you ever trust it?

The phrase "black box AI" gets used so often it risks becoming background noise. In CTCAE decision support, it should be a red flag.

When an AI system recommends a CTCAE term and grade—"Pneumonitis, Grade 3"—the crucial question is not just whether it is right this time. The question is whether the clinician can see why the system reached that conclusion and quickly assess whether it makes sense.

What explainability should look like for CTCAE workflows

Explainability in this context is not about exposing the internal mathematics of the model. It is about making the clinical reasoning path visible:

• Highlighting the exact sentences in the note that mention the symptom and its impact ("short of breath walking from bed to bathroom," "requires 2L oxygen at rest").

• Showing the relevant lab values and imaging findings, with timing.

• Displaying the CTCAE definition side by side, with the parts that match the evidence clearly aligned.

When that is done well, the clinician does not have to "trust the model." They simply see an organized view of the same evidence they would have had to gather manually, plus a suggested label that they can accept, edit, or reject.

Explainability as a defense against deskilling and bias

Explainability is not just an interface feature; it is a defense against deeper risks.

• It reduces automation bias by forcing clinicians to engage with the evidence instead of blindly clicking "accept."

• It supports skill preservation, because clinicians continue to mentally rehearse the mapping from symptom story to CTCAE grade.

• It surfaces model blind spots, since clinicians can see when the AI has overemphasized a minor phrase or ignored critical context.

In other words, explainability keeps AI in its proper role: as a tool that organizes information, not as a hidden judge.

The danger of "click-to-accept" black boxes

Some AI systems offer little more than a prediction and a confidence score. "Grade 2, 0.93 probability." In CTCAE workflows, that is dangerously thin.

Without clear evidence traces, clinicians face a bad choice:

• Either spend extra time reconstructing the evidence themselves (defeating the purpose of automation), or

• Accept the suggestion based on trust in a number they do not fully understand.

The first path leads to frustration and abandonment. The second leads straight to deskilling and potential harm.

If a vendor cannot show you how their CTCAE AI explains itself to clinicians, they are asking you to import a black box into a domain where transparency is mandatory.

System 2 support: slowing clinicians down when it matters

One of the most valuable roles for explainable AI is to act as a "System 2" trigger—inviting slow, careful reasoning exactly when it is most needed.

For example, the system might:

• Mark certain suggestions as "borderline" and require a confirmatory step.

• Highlight conflicts between PRO-CTCAE and clinician documentation ("patient reports severe interference with walking, but clinician note suggests mild impact").

• Flag patterns of multiple moderate toxicities that together raise concern.

In each case, the AI is not trying to replace judgment; it is trying to ensure that the clinician brings their full attention to a case that deserves it.

Explainability as part of governance, not just UX

Finally, explainability should be integrated into governance:

• Audits should include review of the explanations shown at the time decisions were made.

• Discrepancies between AI suggestions and final CTCAE grades can be analyzed in light of what evidence was presented.

• Policy decisions (for example, when to allow "one-click accept") can be revisited based on real-world behavior.

When explainability is treated as part of the safety case, not just a user experience enhancement, CTCAE AI becomes far easier to defend—to clinicians, to patients, and to regulators.

Marc Saint-jour, MD

marc@burna.ai