7 research outputs found
Deep neural networks architectures from the perspective of manifold learning
Despite significant advances in the field of deep learning in ap-plications
to various areas, an explanation of the learning pro-cess of neural network
models remains an important open ques-tion. The purpose of this paper is a
comprehensive comparison and description of neural network architectures in
terms of ge-ometry and topology. We focus on the internal representation of
neural networks and on the dynamics of changes in the topology and geometry of
a data manifold on different layers. In this paper, we use the concepts of
topological data analysis (TDA) and persistent homological fractal dimension.
We present a wide range of experiments with various datasets and configurations
of convolutional neural network (CNNs) architectures and Transformers in CV and
NLP tasks. Our work is a contribution to the development of the important field
of explainable and interpretable AI within the framework of geometrical deep
learning.Comment: 11 pages, 12 figures, PRAI2023. arXiv admin note: substantial text
overlap with arXiv:2204.0862
Applying language models to algebraic topology: generating simplicial cycles using multi-labeling in Wu's formula
Computing homotopy groups of spheres has long been a fundamental objective in
algebraic topology. Various theoretical and algorithmic approaches have been
developed to tackle this problem. In this paper we take a step towards the goal
of comprehending the group-theoretic structure of the generators of these
homotopy groups by leveraging the power of machine learning. Specifically, in
the simplicial group setting of Wu's formula, we reformulate the problem of
generating simplicial cycles as a problem of sampling from the intersection of
algorithmic datasets related to Dyck languages. We present and evaluate
language modelling approaches that employ multi-label information for input
sequences, along with the necessary group-theoretic toolkit and non-neural
baselines.Comment: 20 page
Artificial Text Boundary Detection with Topological Data Analysis and Sliding Window Techniques
Due to the rapid development of text generation models, people increasingly
often encounter texts that may start out as written by a human but then
continue as machine-generated results of large language models. Detecting the
boundary between human-written and machine-generated parts of such texts is a
very challenging problem that has not received much attention in literature. In
this work, we consider and compare a number of different approaches for this
artificial text boundary detection problem, comparing several predictors over
features of different nature. We show that supervised fine-tuning of the
RoBERTa model works well for this task in general but fails to generalize in
important cross-domain and cross-generator settings, demonstrating a tendency
to overfit to spurious properties of the data. Then, we propose novel
approaches based on features extracted from a frozen language model's
embeddings that are able to outperform both the human accuracy level and
previously considered baselines on the Real or Fake Text benchmark. Moreover,
we adapt perplexity-based approaches for the boundary detection task and
analyze their behaviour. We analyze the robustness of all proposed classifiers
in cross-domain and cross-model settings, discovering important properties of
the data that can negatively influence the performance of artificial text
boundary detection algorithms
ICML 2023 Topological Deep Learning Challenge:Design and Results
This paper presents the computational challenge on topological deep learning that was hosted within the ICML 2023 Workshop on Topology and Geometry in Machine Learning. The competition asked participants to provide open-source implementations of topological neural networks from the literature by contributing to the python packages TopoNetX (data processing) and TopoModelX (deep learning). The challenge attracted twenty-eight qualifying submissions in its two month duration. This paper describes the design of the challenge and summarizes its main findings.</p
ICML 2023 topological deep learning challenge. Design and results
This paper presents the computational challenge on topological deep learning that was hosted within the ICML 2023 Workshop on Topology and Geometry in Machine Learning. The competition asked participants to provide open-source implementations of topological neural networks from the literature by contributing to the python packages TopoNetX (data processing) and TopoModelX (deep learning). The challenge attracted twenty-eight qualifying submissions in its two-month duration. This paper describes the design of the challenge and summarizes its main finding