59,017 research outputs found
Function-Space Regularization in Neural Networks: A Probabilistic Perspective
Parameter-space regularization in neural network optimization is a
fundamental tool for improving generalization. However, standard
parameter-space regularization methods make it challenging to encode explicit
preferences about desired predictive functions into neural network training. In
this work, we approach regularization in neural networks from a probabilistic
perspective and show that by viewing parameter-space regularization as
specifying an empirical prior distribution over the model parameters, we can
derive a probabilistically well-motivated regularization technique that allows
explicitly encoding information about desired predictive functions into neural
network training. This method -- which we refer to as function-space empirical
Bayes (FSEB) -- includes both parameter- and function-space regularization, is
mathematically simple, easy to implement, and incurs only minimal computational
overhead compared to standard regularization techniques. We evaluate the
utility of this regularization technique empirically and demonstrate that the
proposed method leads to near-perfect semantic shift detection,
highly-calibrated predictive uncertainty estimates, successful task adaption
from pre-trained models, and improved generalization under covariate shift.Comment: Published in Proceedings of the 40th International Conference on
Machine Learning (ICML 2023
Pattern Reification as the Basis for Description-Driven Systems
One of the main factors driving object-oriented software development for
information systems is the requirement for systems to be tolerant to change. To
address this issue in designing systems, this paper proposes a pattern-based,
object-oriented, description-driven system (DDS) architecture as an extension
to the standard UML four-layer meta-model. A DDS architecture is proposed in
which aspects of both static and dynamic systems behavior can be captured via
descriptive models and meta-models. The proposed architecture embodies four
main elements - firstly, the adoption of a multi-layered meta-modeling
architecture and reflective meta-level architecture, secondly the
identification of four data modeling relationships that can be made explicit
such that they can be modified dynamically, thirdly the identification of five
design patterns which have emerged from practice and have proved essential in
providing reusable building blocks for data management, and fourthly the
encoding of the structural properties of the five design patterns by means of
one fundamental pattern, the Graph pattern. A practical example of this
philosophy, the CRISTAL project, is used to demonstrate the use of
description-driven data objects to handle system evolution.Comment: 20 pages, 10 figure
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns
visual concepts, words, and semantic parsing of sentences without explicit
supervision on any of them; instead, our model learns by simply looking at
images and reading paired questions and answers. Our model builds an
object-based scene representation and translates sentences into executable,
symbolic programs. To bridge the learning of two modules, we use a
neuro-symbolic reasoning module that executes these programs on the latent
scene representation. Analogical to human concept learning, the perception
module learns visual concepts based on the language description of the object
being referred to. Meanwhile, the learned visual concepts facilitate learning
new words and parsing new sentences. We use curriculum learning to guide the
searching over the large compositional space of images and language. Extensive
experiments demonstrate the accuracy and efficiency of our model on learning
visual concepts, word representations, and semantic parsing of sentences.
Further, our method allows easy generalization to new object attributes,
compositions, language concepts, scenes and questions, and even new program
domains. It also empowers applications including visual question answering and
bidirectional image-text retrieval.Comment: ICLR 2019 (Oral). Project page: http://nscl.csail.mit.edu
Transformation From Semantic Data Model to Rdf
There have been several efforts to use relational model and database to store and manipulate Resource Description Framework (RDF). They have one general disadvantage, i.e. one is forced to map the model of semantics of RDF into relational model, which will end up in constraints and additional properties, such as, validating each assertion against the RDF schema which also stored as a triplets table. In this paper, we introduce Semantic Data Model as a proposed data model language to store and manipulate Resource Description Framework. This study also tries to prescribe the procedure on transforming a semantic data model into a RDF data model. Keyworsd: Semantic Data Model, Resource Description Framework
- …