Understanding the Covariance Structure of Convolutional Filters

Neural network weights are typically initialized at random from univariate distributions, controlling just the variance of individual weights even in highly-structured operations like convolutions. Recent ViT-inspired convolutional networks such as ConvMixer and ConvNeXt use large-kernel depthwise convolutions whose learned filters have notable structure; this presents an opportunity to study their empirical covariances. In this work, we first observe that such learned filters have highly-structured covariance matrices, and moreover, we find that covariances calculated from small networks may be used to effectively initialize a variety of larger networks of different depths, widths, patch sizes, and kernel sizes, indicating a degree of model-independence to the covariance structure. Motivated by these findings, we then propose a learning-free multivariate initialization scheme for convolutional filters using a simple, closed-form construction of their covariance. Models using our initialization outperform those using traditional univariate initializations, and typically meet or exceed the performance of those initialized from the covariances of learned filters; in some cases, this improvement can be achieved without training the depthwise convolutional filters at all

The effect of family structure on parent-adolescent conflict : interpersonal conflict management styles of parents in single and dual-parent households

M.A. University of Hawaii at Manoa 2012.Includes bibliographical references.This study examined differences in parent-adolescent conflict in single-parent and dual-parent households from the perspective of parents with adolescent-age children. Previous research has shown differences in family communication and interpersonal conflict as a result of the family structure (i.e., whether there are one or two parents in the home). Single and partnered parents were recruited via students at local middle and high schools, parent-centered organizations, and via snowball sampling. Participants completed demographic information as well as the Thomas-Kilmann Conflict MODE Instrument. Single parents completed the measure as individuals; parents from dual-parent households completed the survey together. Results showed no differences in reported frequency of conflict between the two family structures. Single parents reported the use of compromising more than the other conflict styles, and more than dual parents. Single parents reported using collaborating significantly less than dual parents. Implications, limitations and future directions are discussed

The birth of the RCMI Clinical Research Center is a joint venture of the University of Hawaii and Kapiolani Health.

Hawaii established a Clinical Research Center with collaboration from the University of Hawaii Pacific Biomedical Research Center, the John A. Burns School of Medicine and Kapiolani Health via a five year award from the Research Centers in Minority Institutions of the National Institutes of Health. Support offered includes consultative services for protocol design; epidemiological and biostatistical analysis; design of study forms; and data and specimen collection and analysis.P20RR/AI11091/RR/NCRR NIH HHS/United State

Mimetic Initialization of Self-Attention Layers

It is notoriously difficult to train Transformers on small datasets; typically, large pre-trained models are instead used as the starting point. We explore the weights of such pre-trained Transformers (particularly for vision) to attempt to find reasons for this discrepancy. Surprisingly, we find that simply initializing the weights of self-attention layers so that they "look" more like their pre-trained counterparts allows us to train vanilla Transformers faster and to higher final accuracies, particularly on vision tasks such as CIFAR-10 and ImageNet classification, where we see gains in accuracy of over 5% and 4%, respectively. Our initialization scheme is closed form, learning-free, and very simple: we set the product of the query and key weights to be approximately the identity, and the product of the value and projection weights to approximately the negative identity. As this mimics the patterns we saw in pre-trained Transformers, we call the technique "mimetic initialization"