IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization

Fine-tuning pre-trained language models (PTLMs), such as BERT and its better variant RoBERTa, has been a common practice for advancing performance in natural language understanding (NLU) tasks. Recent advance in representation learning shows that isotropic (i.e., unit-variance and uncorrelated) embeddings can significantly improve performance on downstream tasks with faster convergence and better generalization. The isotropy of the pre-trained embeddings in PTLMs, however, is relatively under-explored. In this paper, we analyze the isotropy of the pre-trained [CLS] embeddings of PTLMs with straightforward visualization, and point out two major issues: high variance in their standard deviation, and high correlation between different dimensions. We also propose a new network regularization method, isotropic batch normalization (IsoBN) to address the issues, towards learning more isotropic representations in fine-tuning by dynamically penalizing dominating principal components. This simple yet effective fine-tuning method yields about 1.0 absolute increment on the average of seven NLU tasks.Comment: AAAI 202

Incorporating Triple Bottom Line Strategies into Corporations

In this new era of globalization, the competition among multi-national corporations in various industries is tighter than ever. Due to this intense struggle for corporations to grow their bottom line, the corporations have recognized that a new approach to business is necessary. Corporations have realized that they can no longer just think about their economical profitability to survive in this environment, but they also have to be good stewards to the people they employ and be conscious of the impact the corporation\u27s daily operations have on the environment. Because of this new found recognition, corporations have started using phrases such as being sustainable, being responsible and being good corporation citizens, in their corporation reporting and publications. The work of this thesis, will analyze the sustainability/responsibility/citizenship report and websites of five-U.S. multi-national corporations in various industries. This research will look at what sustainability/responsibility/citizenship activities corporations are claiming to be doing and verify the validity of those activities from additional independent sources. Furthermore, this thesis will look at whether or not these activities meet the definition of what the corporations are claiming to be sustainable/responsible/citizenship. After performing the research, this thesis has concluded that the activities that each of the companies is claiming to be implementing are indeed being implemented and that these activities do help the individual corporation to achieve its own definition of what is sustainable/responsible/citizenship

Sprinklers: A Randomized Variable-Size Striping Approach to Reordering-Free Load-Balanced Switching

Internet traffic continues to grow exponentially, calling for switches that can scale well in both size and speed. While load-balanced switches can achieve such scalability, they suffer from a fundamental packet reordering problem. Existing proposals either suffer from poor worst-case packet delays or require sophisticated matching mechanisms. In this paper, we propose a new family of stable load-balanced switches called "Sprinklers" that has comparable implementation cost and performance as the baseline load-balanced switch, but yet can guarantee packet ordering. The main idea is to force all packets within the same virtual output queue (VOQ) to traverse the same "fat path" through the switch, so that packet reordering cannot occur. At the core of Sprinklers are two key innovations: a randomized way to determine the "fat path" for each VOQ, and a way to determine its "fatness" roughly in proportion to the rate of the VOQ. These innovations enable Sprinklers to achieve near-perfect load-balancing under arbitrary admissible traffic. Proving this property rigorously using novel worst-case large deviation techniques is another key contribution of this work

A Complete Characterization of Unitary Quantum Space

Motivated by understanding the power of quantum computation with restricted number of qubits, we give two complete characterizations of unitary quantum space bounded computation. First we show that approximating an element of the inverse of a well-conditioned efficiently encoded 2^k(n) x 2^k(n) matrix is complete for the class of problems solvable by quantum circuits acting on O(k(n)) qubits with all measurements at the end of the computation. Similarly, estimating the minimum eigenvalue of an efficiently encoded Hermitian 2^k(n) x 2^k(n) matrix is also complete for this class. In the logspace case, our results improve on previous results of Ta-Shma by giving new space-efficient quantum algorithms that avoid intermediate measurements, as well as showing matching hardness results. Additionally, as a consequence we show that preciseQMA, the version of QMA with exponentially small completeness-soundess gap, is equal to PSPACE. Thus, the problem of estimating the minimum eigenvalue of a local Hamiltonian to inverse exponential precision is PSPACE-complete, which we show holds even in the frustration-free case. Finally, we can use this characterization to give a provable setting in which the ability to prepare the ground state of a local Hamiltonian is more powerful than the ability to prepare PEPS states. Interestingly, by suitably changing the parameterization of either of these problems we can completely characterize the power of quantum computation with simultaneously bounded time and space