What does deep learning see? Insights from a classifier trained to predict contrast enhancement phase from CT images

Kenneth A. Philbrick, Kotaro Yoshida, Dai Inoue, Zeynettin Akkus, Timothy Kline, Alexander D. Weston, Panagiotis Korfiatis, Naoki M Takahashi, Bradley J Erickson

Research output: Contribution to journalArticle

9 Scopus citations


OBJECTIVE. Deep learning has shown great promise for improving medical image classification tasks. However, knowing what aspects of an image the deep learning system uses or, in a manner of speaking, sees to make its prediction is difficult. MATERIALS AND METHODS. Within a radiologic imaging context, we investigated the utility of methods designed to identify features within images on which deep learning activates. In this study, we developed a classifier to identify contrast enhancement phase from whole-slice CT data. We then used this classifier as an easily interpretable system to explore the utility of class activation map (CAMs), gradient-weighted class activation maps (Grad-CAMs), saliency maps, guided backpropagation maps, and the saliency activation map, a novel map reported here, to identify image features the model used when performing prediction. RESULTS. All techniques identified voxels within imaging that the classifier used. SAMs had greater specificity than did guided backpropagation maps, CAMs, and Grad-CAMs at identifying voxels within imaging that the model used to perform prediction. At shallow network layers, SAMs had greater specificity than Grad-CAMs at identifying input voxels that the layers within the model used to perform prediction. CONCLUSION. As a whole, voxel-level visualizations and visualizations of the imaging features that activate shallow network layers are powerful techniques to identify features that deep learning models use when performing prediction.

Original languageEnglish (US)
Pages (from-to)1184-1193
Number of pages10
JournalAmerican Journal of Roentgenology
Issue number6
StatePublished - Dec 1 2018



  • Class activation map (CAM)
  • Computer-aided diagnosis
  • Contrast enhancement phase
  • Convolutional neural network (CNN)
  • CT
  • Deep learning
  • Gradient-weighted class activation map (Grad-CAM)
  • Guided backpropagation
  • Machine learning
  • Saliency activation map
  • Saliency map

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Cite this