AI solves enigmatic immune disease?

Unfortunately, in my write up of our new paper, I committed the cardinal sin of science communication:

I forgot to say we used machine learning!

Clearly I should have led with "AI solves enigmatic immune disease!"

This paper is actually a pretty good example of what AI actually does for understanding immune disorders. Maybe it gives an incremental advance over modern statistical methods. Random Forests (the best performing AI approach) identified the same immunological signatures as multivariable logistic regression (a conventional statistical analysis), so we didn't learn any new biology from AI.

It did improve diagnostic AUC by ~2.6% though. That's not nothing - incremental improvements are at the heart of clinical advances. I'll take a ~2.6% improvement where ever I can get it.

But will even that ~2.6% increase in diagnostic discrimination be reproducible and useful? Here I am not so sure. The main immunological signatures look very robust, but the "AI boost" doesn't come from identifying additional signature factors, it comes from interaction between these signals, and that interaction is the part that is most prone to over-fitting. On a repeat study, the overall diagnostic capacity is likely to drop (regression to the mean), and I suspect that the "AI boost" of ~2.6% may drop to 0% or even be negative. If we increase the complexity of the AI model, we do find that it performs worse than the classical statistical model, which points in that direction.

I also doubt that any clinical diagnostic test would use the "AI boost", even if it was reproducible and robust. Actual clinical diagnostics are usually highly simplified, focused on fewer parameters than the research-grade complexity we used here. So if this advance gets into routine clinical diagnostics (which I hope!), I suspect it would focus on just the very strongest individual immune signals, which were identified by both AI and classical statistics.

So overall, AI was a marginally-useful tool in this study, which helped at the edges. But it wasn't transformative. That isn't just a feature of our study - and it isn't simply a matter of better machine-learning algorithms being built. It is a fundamental limitation of AI - medical data is rarely of the right data structure to give the types of advantages you see in other areas (like weather prediction or molecular structure prediction). AI is another useful tool, and every new tool helps. But it will hardly "transform medicine", and only replaces experiments in niche use cases.

Let's use AI where it helps, without contributing to the excessive hype.