Implicit bias of (stochastic) gradient descent for rank-1 linear neural network

Studying the implicit bias of gradient descent (GD) and stochastic gradient descent (SGD) is critical to unveil the underlying mechanism of deep learning. Unfortunately, even for standard linear networks in regression setting, a comprehensive characterization of the implicit bias is still an open problem. This paper proposes to investigate a new proxy model of standard linear network, rank-1 linear network, where each weight matrix is parameterized as a rank-1 form. For over-parameterized regression problem, we precisely analyze the implicit bias of GD and SGD---by identifying a “potential” function such that GD converges to its minimizer constrained by zero training error (i.e., interpolation solution), and further characterizing the role of the noise introduced by SGD in perturbing the form of this potential. Our results explicitly connect the depth of the network and the initialization with the implicit bias of GD and SGD. Furthermore, we emphasize a new implicit bias of SGD jointly induced by stochasticity and over-parameterization, which can reduce the dependence of the SGD's solution on the initialization. Our findings regarding the implicit bias are different from that of a recently popular model, the diagonal linear network. We highlight that the induced bias of our rank-1 model is more consistent with standard linear network while the diagonal one is not. This suggests that the proposed rank-1 linear network might be a plausible proxy for standard linear net

Implicit bias of adversarial training for deep neural networks

We provide theoretical understandings of the implicit bias imposed by adversarial training for homogeneous deep neural networks without any explicit regularization. In particular, for deep linear networks adversarially trained by gradient descent on a linearly separable dataset, we prove that the direction of the product of weight matrices converges to the direction of the max-margin solution of the original dataset. Furthermore, we generalize this result to the case of adversarial training for non-linear homogeneous deep neural networks without the linear separability of the dataset. We show that, when the neural network is adversarially trained with ℓ2 or ℓ∞ FGSM, FGM and PGD perturbations, the direction of the limit point of normalized parameters of the network along the trajectory of the gradient flow converges to a KKT point of a constrained optimization problem that aims to maximize the margin for adversarial examples. Our results theoretically justify the longstanding conjecture that adversarial training modifies the decision boundary by utilizing adversarial examples to improve robustness, and potentially provides insights for designing new robust training strategies

An enzyme-responsive and transformable PD-L1 blocking peptide-photosensitizer conjugate enables efficient photothermal immunotherapy for breast cancer

Mild photothermal therapy combined with immune checkpoint blockade has received increasing attention for the treatment of advanced or metastatic cancers due to its good therapeutic efficacy. However, it remains a challenge to facilely integrate the two therapies and make it potential for clinical translation. This work designed a peptide-photosensitizer conjugate (PPC), which consisted of a PD-L1 antagonist peptide (CVRARTR), an MMP-2 specific cleavable sequence, a self-assembling motif, and the photosensitizer Purpurin 18. The single-component PPC can self-assemble into nanospheres which is suitable for intravenous injection. The PPC nanosphere is cleaved by MMP-2 when it accumulates in tumor sites, thereby initiating the cancer-specific release of the antagonist peptide. Simultaneously, the nanospheres gradually transform into co-assembled nanofibers, which promotes the retention of the remaining parts within the tumor. In vivo studies demonstrated that PPC nanospheres under laser irradiation promote the infiltration of cytotoxic T lymphocytes and maturation of DCs, which sensitize 4T1 tumor cells to immune checkpoint blockade therapy. Therefore, PPC nanospheres inhibit tumor growth efficiently both in situ and distally and blocked the formation of lung metastases. The present study provides a simple and efficient integrated strategy for breast cancer photoimmunotherapy