# Show me where the tumour is

21 May 2026

Brain tumour classifiers are already accurate, but none can yet prove it is accurate enough to trust as a black box, so I compared models on how well they explain their answers.

[Paper](https://jacob-davies.com/papers/brain_tumor_explainable_diagnoses/)
PDF: https://jacob-davies.com/blog/brain_tumor_explainable_diagnoses/Brain_Tumor_Explainability.pdf

Postgraduate coursework for the AI and Deep Learning module, University of Warwick.

There are over 300,000 new brain cancer cases every year, and brain tumours are among the most expensive cancers to treat per patient. Catching them early makes treatment less intensive and less risky. Diagnosis comes down to MRI scans, and reading those scans takes a specialist neuroradiologist. Where there aren't enough of them, results come back slowly, and in many low and middle-income countries there aren't nearly enough radiologists to go round.

Classifying tumours on MRI is, more or less, solved. Vision models already read these scans accurately, and squeezing out another point of accuracy won't get them into a hospital. What stops them is trust. To let a black box diagnose on its own, you would need an accuracy far beyond what current techniques reach, and beyond what anyone can prove with the way these systems get deployed today. Regulators know this: the MHRA and NICE rules for software as a medical device require transparency, explainability and human oversight, and the EU AI Act and FDA guidance say much the same.

So in practice a clinician stays in charge, and a model earns its place by helping them decide. That means showing why it thinks a scan contains a tumour. I took that as the question: if you had to pick a model for a real clinic, how would you choose between them on how well they explain themselves, rather than on how often they're right? I trained two classic machine learning models and two deep learning models on the same brain MRI dataset, checked their accuracy, then ran explainability tools over all four to see what each one was looking at when it made a call.

## The dataset

I used Masoud Nickparvar's brain tumour MRI dataset from Kaggle: 7,200 scans across four classes, glioma, meningioma, pituitary tumour and no tumour. Each class has 1,400 training images and 400 test images. I split the training images into training and validation sets, and kept the test set untouched until the final evaluation.

## Four models

### HOG + SVM

A histogram of oriented gradients splits the image into 8 × 8 pixel cells, measures the edge directions in each cell and bins them into histograms. Cells are grouped into blocks and normalised (L2-Hys) so lighting changes matter less, and a support vector machine classifies the result. The engineer has to decide up front which gradients carry the signal. I don't have the medical background to tune that, so I used the standard settings from Dalal and Triggs' original paper.

### PCA + random forest

Each scan is shrunk to 64 × 64, giving 4,096 brightness values, which are standardised and reduced to 100 principal components for a random forest. There's no domain knowledge in this at all. PCA keeps whatever varies most, which isn't necessarily the anatomy that matters, so this model is the floor: it tests whether there's any structure in raw pixel space.

### ResNet-18 (CNN)

A convolutional network learns its own filters, from edges up to larger shapes. I fine-tuned PyTorch's pretrained ResNet-18. Fine-tuning reuses features the model already learned elsewhere, which cuts training cost and helps against overfitting. I picked ResNet-18 over deeper variants because it's light, its skip connections avoid vanishing gradients without much tuning, and its transfer learning on medical images is well documented.

| Setting | Value | Why |
|---|---|---|
| Learning rate | 1e-4 | Low, to avoid catastrophic forgetting during full fine-tuning |
| Weight decay | 1e-4 | Light L2 regularisation for a fairly shallow model |
| Epochs | 17 | Validation loss plateaued around epoch 15 |
| Batch size | 64 | Fits in memory; bigger batches give steadier gradients |
| Optimiser | Adam | Per-parameter learning rates suit fine-tuning |
| Scheduler | ReduceLROnPlateau | Halves the learning rate when validation loss stalls |
| Image size | 128 × 128 | A trade-off between resolution and training speed |

### ViT-tiny (vision transformer)

A vision transformer cuts the image into fixed patches and runs them through layers of self-attention, so every patch can look at every other patch. Cost grows with the square of the patch count, so I fine-tuned timm's `vit_tiny_patch16_224`, the smallest standard ViT that still uses 16 × 16 patches on 224 × 224 images. It has 12 transformer blocks with 3 attention heads each.

| Setting | Value | Why |
|---|---|---|
| Learning rate | 1e-4 | Low, for full fine-tuning |
| Weight decay | 1e-2 | Higher than the CNN, as is standard for transformer fine-tuning with AdamW |
| Epochs | 6 | Validation loss plateaued around epoch 4 or 5 |
| Batch size | 32 | 224 × 224 inputs use more memory per pass |
| Optimiser | AdamW | Decoupled weight decay |
| Scheduler | CosineAnnealingLR | Smooth decay to near zero by the last epoch |

## How they scored

Accuracy isn't the focus here, but a model has to clear it before its explanations are worth reading. I scored every model on the 1,600 test images with four metrics: accuracy, macro recall (the mean of the per-class recalls), macro F1 and tumour recall. Tumour recall treats the three tumour classes as one, so it asks only whether a tumour got flagged as a tumour. That stops a mix-up between two similar-looking tumour types from counting as a miss.

![Grouped bar chart of accuracy, macro recall, macro F1 and tumour recall for the four models. PCA + RF is lowest on every metric, HOG + SVM next, and ResNet-18 and ViT-tiny highest, with ViT slightly ahead except on tumour recall](metrics.webp)

*All four metrics for all four models on the held-out test set.*

| Model | Accuracy | Macro recall | Macro F1 | Tumour recall |
|---|---|---|---|---|
| PCA + RF | 0.870 | 0.870 | 0.866 | 0.940 |
| HOG + SVM | 0.905 | 0.905 | 0.902 | 0.960 |
| ResNet-18 | 0.944 | 0.944 | 0.943 | **0.980** |
| ViT-tiny | **0.949** | **0.949** | **0.948** | 0.979 |

The order is the same on almost every metric: PCA + RF, then HOG + SVM, then the two deep learning models. The deep learning models learn features specific to the task, and that pays off here, where nearly every scan is easy and the gap between models comes from a handful of hard cases.

The ViT beats the CNN by about 0.005 on the first three metrics, which fits the literature: self-attention lets it relate distant parts of the image. On tumour recall the order flips and the CNN wins by 0.001. The margin is tiny, but it's a 0.006 swing against the trend, so I went to the confusion matrices to see where it came from.

## Where they go wrong

![Four confusion matrices, one per model, with true class on the rows and predicted class on the columns. Every model gets all 400 no-tumour scans right. Most errors are gliomas predicted as meningioma or no tumour](confusion.webp)

*Four-class confusion matrices on the test set.*

Nearly all the tumours the deep learning models missed were gliomas: 24 of the ViT's 25 missed tumours and 21 of the CNN's 24. Gliomas, and low-grade gliomas especially, grow into the surrounding tissue, so on an MRI they have low contrast and weak edges. A CNN can learn filters tuned to faint gradients, which may be why it catches a few more. The numbers are too small to call it significant. Still, if different models turn out to be good at different things, that matters for a diagnostic aid, where a model is most useful on the cases people find hard.

Telling glioma from meningioma is a known weak spot for deep learning models, and it's the most common mix-up in these matrices too. I measured it with a glioma-meningioma confusion rate:

$$
\text{confusion rate} = \frac{\text{meningiomas called glioma} + \text{gliomas called meningioma}}{\text{true gliomas} + \text{true meningiomas}}
$$

![Bar chart of glioma-meningioma confusion rate: HOG + SVM 0.091, PCA + RF 0.114, ResNet-18 0.071, ViT-tiny 0.060. Lower is better](glioma-meningioma.webp)

*How often each model mixes up glioma and meningioma. Lower is better.*

The ranking matches the headline metrics. The errors also run mostly one way. Gliomas get called meningioma far more often than the reverse: 53 against 4 for the CNN, and 45 against 3 for the ViT. That fits the low-contrast glioma explanation.

## Asking the models why

A clinician can only use a model if they can see what it based each prediction on. The explanation also tells us whether the model learned real features that should hold up on new data.

I used two kinds of explanation. The first is occlusion sensitivity, which works the same way for every model, so it's a fair comparison. It slides a grey patch across the scan and records how much the model's confidence in the correct class drops when each area is covered.

![Grid of occlusion heatmaps. Rows are a glioma, a meningioma, a no-tumour scan and a pituitary tumour; columns are the input MRI then HOG + SVM, PCA + RF, ResNet-18 and ViT-tiny. On the meningioma row, HOG + SVM and PCA + RF light up the skull and wide areas, while ResNet-18 and ViT-tiny light up only the tumour](occlusion.webp)

*Occlusion sensitivity for four correctly classified scans. Red marks the areas whose removal hurt the prediction most.*

This explains a lot of the performance gap. The classic models lean on the skull and the edges of the brain. The deep learning models focus on small areas in and around the tumour. The meningioma row shows it best: PCA + RF spreads its attention over the whole image, HOG + SVM uses a large patch with the skull prominent, the CNN picks out the tumour, and the ViT focuses on the middle of it.

The second kind is each model's native explanation, its own internal evidence:

- HOG + SVM: the gradient magnitude in each 8 × 8 cell. This is what HOG feeds in, not the SVM's attribution.
- PCA + RF: the random forest's feature importance projected back through the PCA components. This is global, so it's the same for every scan.
- ResNet-18: Grad-CAM on layer 4, the deepest convolutional stage before global average pooling throws away the spatial layout.
- ViT-tiny: attention rollout across all 12 blocks.

![Grid of native explanations for the same four scans. HOG cell magnitude lights up the skull outline; RF importance is a fixed pattern across every scan; Grad-CAM shows a broad red area over the tumour; attention rollout shows a tight patch on the tumour](native.webp)

*Native explanations. These come from different methods, so the colours can't be compared across columns.*

They tell the same story as the occlusion maps. On the meningioma, HOG + SVM looks almost entirely at the skull and both deep learning models look at the tumour. The CNN's Grad-CAM covers a broad area around it, while the ViT's attention sits tightly on it.

HOG has one extra trick: you can also plot which gradient directions made each cell important.

![Four MRI scans beside their HOG gradient-orientation plots, which show small oriented strokes that are brightest around the outline of the skull](hog.webp)

*What HOG + SVM sees: the direction and strength of the gradients in each cell.*

In principle that's more transparent than anything the deep learning models offer. In practice I found it very hard to turn into anything a clinician could use.

## Could this help in a clinic?

On accuracy, the CNN and the ViT are within half a point of each other, so the choice comes down to their explanations. The ViT's attention sits tightly on the tumour, while the CNN's Grad-CAM covers a broad area around it, and a clinician can check a tight map against the scan faster. HOG + SVM would be ruled out on explanations alone even if its accuracy caught up, because its maps point at the skull. All of this assumes the maps are right, which I haven't yet tested (see the limitations below).

Explainability tools like these are what software as a medical device needs to be transparent and auditable. I'd propose a system like this as a computer-aided diagnosis tool rather than an autonomous one. A clinician still makes every call, and the model acts as a second pair of eyes on every scan, which cuts the cost of a misdiagnosis. The heatmaps help here: they show a clinician straight away which part of the scan drove the prediction, which should speed up reading and let each clinician get through more scans.

That's a long way off. The dataset is pre-processed, every image is a slice that shows a diagnosable cross-section, and none of it comes from a clinical pipeline. Real-world performance would need validating on real clinical data, and CE marking would also require ongoing monitoring and audit trails.

## Limitations

The biggest gap is that I haven't checked the heatmaps against expert annotations or ground-truth segmentation masks. A heatmap can look plausible without matching the tumour tissue, and if it doesn't, it could give a clinician false confidence in a wrong prediction. The next step is to rerun this on a dataset with segmentation masks and have a medical professional review the results.

## What's next

A 2025 review by Aburass et al. describes medical imaging moving from CNNs to vision transformers, and my ViT's small lead fits that. It's still open whether general-purpose models beat architectures designed for a specific domain. 3D transformers that read the whole brain volume, instead of one slice, are another promising direction.

The code and notebooks are on GitHub: [JacobRhys/brain_tumor_explainable_diagnoses](https://github.com/JacobRhys/brain_tumor_explainable_diagnoses).