Revisiting Convolutional Neural Networks from the Viewpoint of Kernel-Based Methods

Convolutional neural networks, as most artificial neural networks, are frequently viewed as methods different in essence from kernel-based methods. In this work we translate several classical convolutional neural networks into kernel-based counterparts. Each kernel-based counterpart is a statistical model called a convolutional kernel network with parameters that can be learned from data. We provide an alternating minimization algorithm with mini-batch sampling and implicit partial differentiation to learn from data the parameters of each convolutional kernel network. We also show how to obtain inexact derivatives with respect to the parameters using an algorithm based on two inter-twined Newton iterations. The models and the algorithms are illustrated on benchmark datasets in image classification. We find that the convolutional neural networks and their kernel counterparts often perform similarly. Supplemental appendices and code for the article are available online.</p

GAP43 immunofluorescence increased in 6-OHDA monkeys and had minimal colabeling with TH.

<p>Cardiac sections of control (<b>A–D</b>) and 6-OHDA-treated (<b>F–I</b>) animals were labeled for GAP43 (red) and TH (green) immunofluorescence and counterstained with DAPI (blue nuclei). Arrows indicate GAP43 expression in nerve bundles. Arrowheads indicate colocalization of GAP43-ir and TH-ir. Inset (<b>d</b>) shows a positive control for GAP43 labeling in caudorostral diencephalon of rhesus embryonic day 38, run in parallel to these stainings. (<b>E</b>) Averaged scores demonstrate an increase of GAP43-ir in 6-OHDA monkeys. (t(8) = 9.073, <i>P</i><0.001). (<b>J</b>) Global GAP43 scores negatively correlate with AAT of TH-ir in cardiac nerve bundles. ***<i>P</i><0.001. AAT, area above threshold; GAP43, growth associated protein 43; TH, tyrosine hydroxylase.</p

6-OHDA-treated monkeys have decreased PGP9.5-ir in cardiac nerve bundles and fibers.

<p>(<b>A–D</b>) Microphotographs of PGP9.5-positive nerve bundles (<b>A,B</b>) and fibers (<b>C,D</b>) in control and 6-OHDA-treated animals. (<b>E</b>) Global PGP9.5-ir cardiac nerve bundle was reduced for both AAT (t(8) = 3.7, <i>P</i> = 0.006) and OD quantification (t(8) = 4.21, <i>P</i> = 0.003) in 6-OHDA compared to control animals. (<b>F</b>) PGP9.5-ir nerve bundle AAT was also decreased between groups in the lateral wall of the left ventricle. Within the 6-OHDA group, a significant effect of wall (ANOVA: F(3,16) = 4.632, <i>P</i> = 0.016) with the greatest reduction in the lateral compared to the septal wall (^a<i>P</i> = 0.022) was found. (<b>G</b>) Quantification of PGP9.5-ir fiber area density was significantly reduced in 6-OHDA-treated monkeys compared to controls (t(8) = 3.026, <i>P</i> = 0.016). (<b>H</b>) TH to PGP9.5 positive fibers ratio was significantly decreased in the anterior (t(8) = 4.335, <i>P</i> = 0.002) and inferior walls (t(8) = 4.43, <i>P</i> = 0.002) in 6-OHDA animals compared to controls. Within the 6-OHDA group there was a significant reduction in ratio between walls (ANOVA; F(3,16) = 3.299, <i>P</i> = 0.047) with significant loss in the inferior compared to septal wall (^b<i>P</i> = 0.044). Scale bar  = 25 µm. <i>*P</i><0.05, **<i>P</i><0.01, ^a<i>P</i> = 0.02, ^b<i>P</i> = 0.044. AAT, area above threshold; OD, optical density; PGP9.5, protein gene product 9.5; TH, tyrosine hydroxylase.</p

Measures of cardiac TH and PGP9.5 expression correlated with <i>in vivo</i> MHED uptake.

<p>Nerve bundle AAT quantification of TH-ir correlated with TH-ir fibers (<b>A</b>) and AAT of PGP9.5-ir nerve bundles (<b>B</b>). Area density quantification of TH-ir fibers highly correlated with PGP9.5-ir fibers (<b>C</b>). <i>In vivo</i> MHED uptake correlated with TH-ir (<b>D</b>) and PGP9.5-ir fiber area density (<b>E</b>). Despite the significant interaction between MHED and TH-ir fibers, MHED uptake did not correlate with AAT quantification of TH-ir nerve bundles (r² = 0.446, <i>P</i> = 0.070) (not shown). AAT, area above threshold; MHED, meta-hydroxyephedrine; PGP9.5, protein gene product 9.5; TH, tyrosine hydroxylase.</p

Cardiac α-synuclein was moderately affected by 6-OHDA.

<p>(<b>A–D</b>) Microphotographs of nerve bundles immunostained for soluble α-synuclein (<b>A,B</b>), phosphorylated α-synuclein (<b>C,D</b>) and proteinase K resistant α-synuclein (<b>E,F</b>) in control (<b>A,C,E</b>) and 6-OHDA-treated (<b>B,D,F</b>) monkeys. Insets (<b>a–c</b>) display immunostained nigral tissue from a Thy1-α-synuclein mouse processed in parallel for each specific antibody and used as a positive control to validate staining. (<b>G</b>) Regional reduction of soluble α-synuclein expression was found in the septal wall (t(8) = 2.787, <i>P</i> = 0.024) of 6-OHDA compared to control monkeys. Within each group, soluble α-synuclein scores did not show significant differences by wall in control animals (ANOVA; F(3,16) = 3.115, <i>P</i> = 0.056) and in 6-OHDA-treated animals (ANOVA: F(3,16) = 0.772, <i>P</i> = 0.526). (<b>H</b>) Average scores of soluble α-synuclein-ir in nerve bundles correlated with AAT TH-positive nerve bundles (r² = 0.570, <i>P</i> = 0.012). Scale bar  = 25 µm, inset scale bar  = 100 µm. <i>*P</i><0.05. AAT, area above threshold; TH, tyrosine hydroxylase.</p

6-OHDA-treated monkeys have reduced TH-ir in cardiac nerve bundles and fibers.

<p>(<b>A,B</b>) Microphotographs of H&E stained nerve bundles identified as a collection of nerve fibers (white asterisk) and intact epineurium indicated by white arrows. (<b>C</b>) Global TH-ir in cardiac nerve bundle was reduced for both AAT (t(8) = 3.508, <i>P</i> = 0.008) and OD quantification (t(8) = 3.403, <i>P</i> = 0.009) in 6-OHDA compared to control animals. (<b>D,E</b>) Microphotographs of TH-ir in matching nerve bundles. (<b>F</b>) Regional wall quantification showed reduced AAT of TH-ir in anterior (t(8) = 5.913, <i>P</i><0.0001), inferior (t(8) = 3.096, <i>P</i> = 0.015), and septal (t(8) = 3.009, <i>P</i> = 0.017) walls in 6-OHDA-treated monkeys compared to controls. In the control group there were also significant differences in OD quantification of TH (F(3,16) = 5.213, <i>P</i> = 0.011) with more TH-ir in the septal wall compared to the lateral (<i>P</i> = 0.043) and inferior walls (<i>P</i> = 0.12) (<i>post hoc</i> Bonferroni multiple comparisons analysis; not shown). (<b>G,H,J,K</b>) Microphotographs of cardiac TH positive fibers in either transverse (<b>G,H</b>) or longitudinal orientation (<b>J,K</b>) in control and 6-OHDA monkeys; black arrows indicate TH-ir fibers. (<b>I,L</b>) TH-ir fiber area density was significantly reduced in 6-OHDA compared to control animals for global (<b>I</b>) analysis (t(8) = 4.934, <i>P</i> = 0.001) and between all walls of the left ventricle (<b>L</b>) including the anterior (t(8) = 4.324, <i>P</i> = 0.003), inferior (t(8) = 5.37, <i>P</i> = 0.001), lateral (t(8) = 5.081,<i>P</i> = 0.001) and septal walls (t(8) = 3.866, <i>P</i> = 0.005). Scale bar  = 25 µm. <i>*P</i><0.05, **<i>P</i><0.01, ***<i>P</i><0.001. AAT, area above threshold; OD, optical density; TH, tyrosine hydroxylase.</p

α-synuclein expression in the adrenal medulla was not affected by 6-OHDA treatment.

<p>Microphotographs of adrenal medulla immunostained for soluble α-synuclein (<b>A,B,C</b>), phosphorylated α-synuclein (<b>D,E,F</b>) and proteinase K resistant α-synuclein (<b>G,H,K</b>) in 6-OHDA-treated (<b>C,F,I</b>) and low or high α-synuclein expressing control monkeys. Insets display nigral tissue from a Thy1-α-synuclein mouse immunolabeled for each antibody in parallel and used as a positive control to validate processing. Scale bar  = 25 µm, inset scale bar  = 50 µm.</p

6-OHDA-treated monkeys have reduced TH-ir and AADC-ir in the adrenal medulla.

<p>(<b>A-F</b>) Microphotographs of adrenal medulla sections stained for H&E (<b>A,B</b>), TH (<b>C,D</b>) and AADC (<b>E,F</b>) from control and 6-OHDA monkeys. Squares in A-F corresponds to respective insets a-f showing a higher magnification image of the selected area. (<b>G</b>) OD (TH: t(8) = 3.455, <i>P</i> = 0.009; AADC: t(8) = 2.192, <i>P</i> = 0.06) and (<b>H</b>) AAT (TH: t(5.791) = 4.437, <i>P</i> = 0.005; AADC: t(8) = 3.180, <i>P</i> = 0.013) quantifications of immunostainings showed significant reductions in 6-OHDA monkeys. Scale bar  = 50 µm; inset scale bar  = 50 µm. <i>*P</i><0.05, **<i>P</i><0.01. AADC, aromatic l-amino acid; AAT, area above threshold; OD, optical density; TH, tyrosine hydroxylase.</p

PGP9.5 and TH immunofluorescense demonstrated colabeling and decreased expression in cardiac nerve bundles of 6-OHDA monkeys.

<p>Microphotographs of immunofluorescence staining of PGP9.5 (red; <b>A,E</b>), TH (green; <b>B,F</b>) and counterstained with DAPI (blue nuclei; <b>C,G</b>) in controls (A–D) and 6-OHDA-treated animals (E–H). Colocalization of PGP9.5 and TH expression is identified as yellow. PGP9.5, protein gene product 9.5; TH, tyrosine hydroxylase.</p

Similar nitrotyrosine-ir and HLA-DR-ir were observed across groups.

<p>(<b>A–D</b>) Microphotographs of a region of the left ventricle containing a nerve bundle, blood vessel and cardiomyocytes immunostained for nitrotyrosine (<b>A,B</b>) and human leukocyte antigen (HLA-DR) (<b>C,D</b>) in control (<b>A,C</b>) and 6-OHDA-treated monkeys (<b>B,D</b>). (<b>E</b>) The sum of nitrotyrosine scores in 6-OHDA-treated animals was significantly lower in the subepicardial compared to the subendocardial layer (t(8) = 2.921, <i>P</i> = 0.019), while no difference were found between layers in the control group. (<b>F</b>) Summed nitrotyrosine scores in nerve bundles by wall were not different between groups, but within 6-OHDA-treated animals a significant effect was found between walls (ANOVA: F(3,16) = 3.803, <i>P</i> = 0.031), with greater nitrotyrosine-ir scores in the septal than the anterior wall (^a<i>P</i> = 0.031). Scale bar  = 25 µm. <i>*P</i> = 0.019, ^a<i>P</i> = 0.031.</p