Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity

In some settings neural networks exhibit a phenomenon known as grokking, where they achieve perfect or near-perfect accuracy on the validation set long after the same performance has been achieved on the training set. In this paper, we discover that grokking is not limited to neural networks but occurs in other settings such as Gaussian process (GP) classification, GP regression and linear regression. We also uncover a mechanism by which to induce grokking on algorithmic datasets via the addition of dimensions containing spurious information. The presence of the phenomenon in non-neural architectures provides evidence that grokking is not specific to SGD or weight norm regularisation. Instead, grokking may be possible in any setting where solution search is guided by complexity and error. Based on this insight and further trends we see in the training trajectories of a Bayesian neural network (BNN) and GP regression model, we make progress towards a more general theory of grokking. Specifically, we hypothesise that the phenomenon is governed by the accessibility of certain regions in the error and complexity landscapes

Steering Language Generation: Harnessing Contrastive Expert Guidance and Negative Prompting for Coherent and Diverse Synthetic Data Generation

Large Language Models (LLMs) hold immense potential to generate synthetic data of high quality and utility, which has numerous applications from downstream model training to practical data utilisation. However, contemporary models, despite their impressive capacities, consistently struggle to produce both coherent and diverse data. To address the coherency issue, we introduce contrastive expert guidance, where the difference between the logit distributions of fine-tuned and base language models is emphasised to ensure domain adherence. In order to ensure diversity, we utilise existing real and synthetic examples as negative prompts to the model. We deem this dual-pronged approach to logit reshaping as STEER: Semantic Text Enhancement via Embedding Repositioning. STEER operates at inference-time and systematically guides the LLMs to strike a balance between adherence to the data distribution (ensuring semantic fidelity) and deviation from prior synthetic examples or existing real datasets (ensuring diversity and authenticity). This delicate balancing act is achieved by dynamically moving towards or away from chosen representations in the latent space. STEER demonstrates improved performance over previous synthetic data generation techniques, exhibiting better balance between data diversity and coherency across three distinct tasks: hypothesis generation, toxic and non-toxic comment generation, and commonsense reasoning task generation. We demonstrate how STEER allows for fine-tuned control over the diversity-coherency trade-off via its hyperparameters, highlighting its versatility

uPAR Induces Expression of Transforming Growth Factor β and Interleukin-4 in Cancer Cells to Promote Tumor-Permissive Conditioning of Macrophages

Cancer cells condition macrophages and other inflammatory cells in the tumor microenvironment so that these cells are more permissive for cancer growth and metastasis. Conditioning of inflammatory cells reflects, at least in part, soluble mediators (such as transforming growth factor β and IL-4) that are released by cancer cells and alter the phenotype of cells of the innate immune system. Signaling pathways in cancer cells that potentiate this activity are incompletely understood. The urokinase receptor (uPAR) is a cell-signaling receptor known to promote cancer cell survival, proliferation, metastasis, and cancer stem cell–like properties. The present findings show that uPAR expression in diverse cancer cells, including breast cancer, pancreatic cancer, and glioblastoma cells, promotes the ability of these cells to condition co-cultured bone marrow–derived macrophages so that the macrophages express significantly increased levels of arginase 1, a biomarker of the alternatively activated M2 macrophage phenotype. Expression of transforming growth factor β was substantially increased in uPAR-expressing cancer cells via a mechanism that requires uPA-initiated cell signaling. uPAR also controlled expression of IL-4 in cancer cells via a mechanism that involves activation of ERK1/2. The ability of uPAR to induce expression of factors that condition macrophages in the tumor microenvironment may constitute an important mechanism by which uPAR promotes cancer progression

Downregulation of 26S proteasome catalytic activity promotes epithelial-mesenchymal transition.

The epithelial-mesenchymal transition (EMT) endows carcinoma cells with phenotypic plasticity that can facilitate the formation of cancer stem cells (CSCs) and contribute to the metastatic cascade. While there is substantial support for the role of EMT in driving cancer cell dissemination, less is known about the intracellular molecular mechanisms that govern formation of CSCs via EMT. Here we show that β2 and β5 proteasome subunit activity is downregulated during EMT in immortalized human mammary epithelial cells. Moreover, selective proteasome inhibition enabled mammary epithelial cells to acquire certain morphologic and functional characteristics reminiscent of cancer stem cells, including CD44 expression, self-renewal, and tumor formation. Transcriptomic analyses suggested that proteasome-inhibited cells share gene expression signatures with cells that have undergone EMT, in part, through modulation of the TGF-β signaling pathway. These findings suggest that selective downregulation of proteasome activity in mammary epithelial cells can initiate the EMT program and acquisition of a cancer stem cell-like phenotype. As proteasome inhibitors become increasingly used in cancer treatment, our findings highlight a potential risk of these therapeutic strategies and suggest a possible mechanism by which carcinoma cells may escape from proteasome inhibitor-based therapy

Cancer Informatics for Cancer Centers (CI4CC): Building a Community Focused on Sharing Ideas and Best Practices to Improve Cancer Care and Patient Outcomes.

Cancer Informatics for Cancer Centers (CI4CC) is a grassroots, nonprofit 501c3 organization intended to provide a focused national forum for engagement of senior cancer informatics leaders, primarily aimed at academic cancer centers anywhere in the world but with a special emphasis on the 70 National Cancer Institute-funded cancer centers. Although each of the participating cancer centers is structured differently, and leaders' titles vary, we know firsthand there are similarities in both the issues we face and the solutions we achieve. As a consortium, we have initiated a dedicated listserv, an open-initiatives program, and targeted biannual face-to-face meetings. These meetings are a place to review our priorities and initiatives, providing a forum for discussion of the strategic and pragmatic issues we, as informatics leaders, individually face at our respective institutions and cancer centers. Here we provide a brief history of the CI4CC organization and meeting highlights from the latest CI4CC meeting that took place in Napa, California from October 14-16, 2019. The focus of this meeting was "intersections between informatics, data science, and population science." We conclude with a discussion on "hot topics" on the horizon for cancer informatics