4,586 research outputs found
Knowledge Distillation and Continual Learning for Optimized Deep Neural Networks
Over the past few years, deep learning (DL) has been achieving state-of-theart performance on various human tasks such as speech generation, language translation, image segmentation, and object detection. While traditional machine learning models require hand-crafted features, deep learning algorithms can automatically extract discriminative features and learn complex knowledge from large datasets. This powerful learning ability makes deep learning models attractive to both academia and big corporations.
Despite their popularity, deep learning methods still have two main limitations: large memory consumption and catastrophic knowledge forgetting. First, DL algorithms use very deep neural networks (DNNs) with many billion parameters, which have a big model size and a slow inference speed. This restricts the application of DNNs in resource-constraint devices such as mobile phones and autonomous vehicles. Second, DNNs are known to suffer from catastrophic forgetting. When incrementally learning new tasks, the model performance on old tasks significantly drops. The ability to accommodate new knowledge while retaining previously learned knowledge is called continual learning. Since the realworld environments in which the model operates are always evolving, a robust neural network needs to have this continual learning ability for adapting to new changes
Endogenous measures for contextualising large-scale social phenomena: a corpus-based method for mediated public discourse
This work presents an interdisciplinary methodology for developing endogenous measures of group membership through analysis of pervasive linguistic patterns in public discourse. Focusing on political discourse, this work critiques the conventional approach to the study of political participation, which is premised on decontextualised, exogenous measures to characterise groups. Considering the theoretical and empirical weaknesses of decontextualised approaches to large-scale social phenomena, this work suggests that contextualisation using endogenous measures might provide a complementary perspective to mitigate such weaknesses.
This work develops a sociomaterial perspective on political participation in mediated discourse as affiliatory action performed through language. While the affiliatory function of language is often performed consciously (such as statements of identity), this work is concerned with unconscious features (such as patterns in lexis and grammar). This work argues that pervasive patterns in such features that emerge through socialisation are resistant to change and manipulation, and thus might serve as endogenous measures of sociopolitical contexts, and thus of groups.
In terms of method, the work takes a corpus-based approach to the analysis of data from the Twitter messaging service whereby patterns in users’ speech are examined statistically in order to trace potential community membership. The method is applied in the US state of Michigan during the second half of 2018—6 November having been the date of midterm (i.e. non-Presidential) elections in the United States. The corpus is assembled from the original posts of 5,889 users, who are nominally geolocalised to 417 municipalities. These users are clustered according to pervasive language features. Comparing the linguistic clusters according to the municipalities they represent finds that there are regular sociodemographic differentials across clusters. This is understood as an indication of social structure, suggesting that endogenous measures derived from pervasive patterns in language may indeed offer a complementary, contextualised perspective on large-scale social phenomena
Leveraging a machine learning based predictive framework to study brain-phenotype relationships
An immense collective effort has been put towards the development of methods forquantifying brain activity and structure. In parallel, a similar effort has focused on collecting experimental data, resulting in ever-growing data banks of complex human in vivo neuroimaging data. Machine learning, a broad set of powerful and effective tools for identifying multivariate relationships in high-dimensional problem spaces, has proven to be a promising approach toward better understanding the relationships between the brain and different phenotypes of interest. However, applied machine learning within a predictive framework for the study of neuroimaging data introduces several domain-specific problems and considerations, leaving the overarching question of how to best structure and run experiments ambiguous. In this work, I cover two explicit pieces of this larger question, the relationship between data representation and predictive performance and a case study on issues related to data collected from disparate sites and cohorts. I then present the Brain Predictability toolbox, a soft- ware package to explicitly codify and make more broadly accessible to researchers the recommended steps in performing a predictive experiment, everything from framing a question to reporting results. This unique perspective ultimately offers recommen- dations, explicit analytical strategies, and example applications for using machine learning to study the brain
Equivariant Polynomials for Graph Neural Networks
Graph Neural Networks (GNN) are inherently limited in their expressive power.
Recent seminal works (Xu et al., 2019; Morris et al., 2019b) introduced the
Weisfeiler-Lehman (WL) hierarchy as a measure of expressive power. Although
this hierarchy has propelled significant advances in GNN analysis and
architecture developments, it suffers from several significant limitations.
These include a complex definition that lacks direct guidance for model
improvement and a WL hierarchy that is too coarse to study current GNNs. This
paper introduces an alternative expressive power hierarchy based on the ability
of GNNs to calculate equivariant polynomials of a certain degree. As a first
step, we provide a full characterization of all equivariant graph polynomials
by introducing a concrete basis, significantly generalizing previous results.
Each basis element corresponds to a specific multi-graph, and its computation
over some graph data input corresponds to a tensor contraction problem. Second,
we propose algorithmic tools for evaluating the expressiveness of GNNs using
tensor contraction sequences, and calculate the expressive power of popular
GNNs. Finally, we enhance the expressivity of common GNN architectures by
adding polynomial features or additional operations / aggregations inspired by
our theory. These enhanced GNNs demonstrate state-of-the-art results in
experiments across multiple graph learning benchmarks
Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies
Recently, a new paradigm called Differentiable Search Index (DSI) has been
proposed for document retrieval, wherein a sequence-to-sequence model is
learned to directly map queries to relevant document identifiers. The key idea
behind DSI is to fully parameterize traditional ``index-retrieve'' pipelines
within a single neural model, by encoding all documents in the corpus into the
model parameters. In essence, DSI needs to resolve two major questions: (1) how
to assign an identifier to each document, and (2) how to learn the associations
between a document and its identifier. In this work, we propose a
Semantic-Enhanced DSI model (SE-DSI) motivated by Learning Strategies in the
area of Cognitive Psychology. Our approach advances original DSI in two ways:
(1) For the document identifier, we take inspiration from Elaboration
Strategies in human learning. Specifically, we assign each document an
Elaborative Description based on the query generation technique, which is more
meaningful than a string of integers in the original DSI; and (2) For the
associations between a document and its identifier, we take inspiration from
Rehearsal Strategies in human learning. Specifically, we select fine-grained
semantic features from a document as Rehearsal Contents to improve document
memorization. Both the offline and online experiments show improved retrieval
performance over prevailing baselines.Comment: Accepted by KDD 202
Cohesive subgraph identification in large graphs
Graph data is ubiquitous in real world applications, as the relationship among entities in the applications can be naturally captured by the graph model. Finding cohesive subgraphs is a fundamental problem in graph mining with diverse applications. Given the important roles of cohesive subgraphs, this thesis focuses on cohesive subgraph identification in large graphs.
Firstly, we study the size-bounded community search problem that aims to find a subgraph with the largest min-degree among all connected subgraphs that contain the query vertex q and have at least l and at most h vertices, where q, l, h are specified by the query. As the problem is NP-hard, we propose a branch-reduce-and-bound algorithm SC-BRB by developing nontrivial reducing techniques, upper bounding techniques, and branching techniques.
Secondly, we formulate the notion of similar-biclique in bipartite graphs which is a special kind of biclique where all vertices from a designated side are similar to each other, and aim to enumerate all maximal similar-bicliques. We propose a backtracking algorithm MSBE to directly enumerate maximal similar-bicliques, and power it by vertex reduction and optimization techniques. In addition, we design a novel index structure to speed up a time-critical operation of MSBE, as well as to speed up vertex reduction. Efficient index construction algorithms are developed.
Thirdly, we consider balanced cliques in signed graphs --- a clique is balanced if its vertex set can be partitioned into CL and CR such that all negative edges are between CL and CR --- and study the problem of maximum balanced clique computation. We propose techniques to transform the maximum balanced clique problem over G to a series of maximum dichromatic clique problems over small subgraphs of G. The transformation not only removes edge signs but also sparsifies the edge set
DCNFIS: Deep Convolutional Neuro-Fuzzy Inference System
A key challenge in eXplainable Artificial Intelligence is the well-known
tradeoff between the transparency of an algorithm (i.e., how easily a human can
directly understand the algorithm, as opposed to receiving a post-hoc
explanation), and its accuracy. We report on the design of a new deep network
that achieves improved transparency without sacrificing accuracy. We design a
deep convolutional neuro-fuzzy inference system (DCNFIS) by hybridizing fuzzy
logic and deep learning models and show that DCNFIS performs as accurately as
three existing convolutional neural networks on four well-known datasets. We
furthermore that DCNFIS outperforms state-of-the-art deep fuzzy systems. We
then exploit the transparency of fuzzy logic by deriving explanations, in the
form of saliency maps, from the fuzzy rules encoded in DCNFIS. We investigate
the properties of these explanations in greater depth using the Fashion-MNIST
dataset
Participation to Partnerships: An Exploration of Reconciliation, Co-management and their Interconnection in the South Australian Context
Contemporary ecological and political crises – ranging from climate change and biodiversity loss to conflicts over territory and sovereignty – pose considerable challenges to human and more-than-human coexistence. These crises continue despite growing efforts to manage them, which is partially due to the fact that the former are deeply interconnected and the latter are not. This study explores the importance of integrating these efforts in the South Australian context. Specifically, it investigates (i) the contribution of the state’s reconciliation initiatives to human coexistence, (ii) the contribution of the state’s co-management program to human and more-than-human coexistence as well as (iii) the implications of their existing and potential interconnection. A theoretical framework that integrates elements of Indigenous methodologies and post-structuralism and an embedded case study design were employed for this investigation, with reconciliation and co-management representing the two subunits of analysis. On this basis, collaborations were established with groups and individuals involved in these fields, and data was collected using in-depth interviews, the observation of co-management meetings and the selection of relevant documents. Parallel thematic analyses were conducted to produce initial findings, which were further refined through feedback meetings with the groups and individuals involved in this study. These processes were guided by the principles of ethical and culturally safe research in Indigenous contexts and Lincoln and Guba’s criteria for qualitative research integrity. Regarding reconciliation, findings reveal a comparatively high dissatisfaction with official reconciliation initiatives among Aboriginal people, which is connected to a lack of genuine opportunities to shape their design and implementation. However, they also reveal that the unique circumstances and aspirations of different Aboriginal groups and individuals mean that some choose to make the most of imperfect initiatives, while others choose to reject them. In recognition of this complexity, this study recommends the redistribution of direct and discursive control over reconciliation initiatives to the right people in each context and outlines initial steps towards it. Regarding co-management, findings reveal that co-management strengthens the management that occurs on the ground, but cannot fully make up for resource restrictions and external pressures. They further reveal that collaboration is restricted to park management matters and does not extend to decisions about the terms and conditions of this collaboration, which affects the realisation and realisability of Aboriginal people’s aspirations disproportionately. Moving forward, this study recommends a transition from co-management of protected areas to co-design and co-administration of the broader framework under which it occurs. Regarding the interconnection of reconciliation and co-management, findings reveal instances of mutual reinforcement, where progress in the space of reconciliation has benefitted co-management and vice versa, but also missed opportunities. They further illustrate that integrating the two fields can not only help avoid mutual harm and maximise mutual reinforcement, but also drive their transformations by rendering existing limitations more visible. This study concludes that interconnected crises require equally interconnected solutions and clarifies that these must take the form of holistic situated solutions rather than one universal solution. To let them emerge, it calls for transitions from participation to partnerships on all scales.Thesis (Ph.D.) -- University of Adelaide, School of Social Sciences, 202
OCM 2023 - Optical Characterization of Materials : Conference Proceedings
The state of the art in the optical characterization of materials is advancing rapidly. New insights have been gained into the theoretical foundations of this research and exciting developments have been made in practice, driven by new applications and innovative sensor technologies that are constantly evolving.
The great success of past conferences proves the necessity of a platform for presentation, discussion and evaluation of the latest research results in this interdisciplinary field
- …