This paper tackles automated categorization of Age-related Macular
Degeneration (AMD), a common macular disease among people over 50. Previous
research efforts mainly focus on AMD categorization with a single-modal input,
let it be a color fundus image or an OCT image. By contrast, we consider AMD
categorization given a multi-modal input, a direction that is clinically
meaningful yet mostly unexplored. Contrary to the prior art that takes a
traditional approach of feature extraction plus classifier training that cannot
be jointly optimized, we opt for end-to-end multi-modal Convolutional Neural
Networks (MM-CNN). Our MM-CNN is instantiated by a two-stream CNN, with
spatially-invariant fusion to combine information from the fundus and OCT
streams. In order to visually interpret the contribution of the individual
modalities to the final prediction, we extend the class activation mapping
(CAM) technique to the multi-modal scenario. For effective training of MM-CNN,
we develop two data augmentation methods. One is GAN-based fundus / OCT image
synthesis, with our novel use of CAMs as conditional input of a high-resolution
image-to-image translation GAN. The other method is Loose Pairing, which pairs
a fundus image and an OCT image on the basis of their classes instead of eye
identities. Experiments on a clinical dataset consisting of 1,099 color fundus
images and 1,290 OCT images acquired from 1,099 distinct eyes verify the
effectiveness of the proposed solution for multi-modal AMD categorization