175,836 research outputs found

    Grammar Filtering For Syntax-Guided Synthesis

    Full text link
    Programming-by-example (PBE) is a synthesis paradigm that allows users to generate functions by simply providing input-output examples. While a promising interaction paradigm, synthesis is still too slow for realtime interaction and more widespread adoption. Existing approaches to PBE synthesis have used automated reasoning tools, such as SMT solvers, as well as works applying machine learning techniques. At its core, the automated reasoning approach relies on highly domain specific knowledge of programming languages. On the other hand, the machine learning approaches utilize the fact that when working with program code, it is possible to generate arbitrarily large training datasets. In this work, we propose a system for using machine learning in tandem with automated reasoning techniques to solve Syntax Guided Synthesis (SyGuS) style PBE problems. By preprocessing SyGuS PBE problems with a neural network, we can use a data driven approach to reduce the size of the search space, then allow automated reasoning-based solvers to more quickly find a solution analytically. Our system is able to run atop existing SyGuS PBE synthesis tools, decreasing the runtime of the winner of the 2019 SyGuS Competition for the PBE Strings track by 47.65% to outperform all of the competing tools

    BizBench: A Quantitative Reasoning Benchmark for Business and Finance

    Full text link
    As large language models (LLMs) impact a growing number of complex domains, it is becoming increasingly important to have fair, accurate, and rigorous evaluation benchmarks. Evaluating the reasoning skills required for business and financial NLP stands out as a particularly difficult challenge. We introduce BizBench, a new benchmark for evaluating models' ability to reason about realistic financial problems. BizBench comprises 8 quantitative reasoning tasks. Notably, BizBench targets the complex task of question-answering (QA) for structured and unstructured financial data via program synthesis (i.e., code generation). We introduce three diverse financially-themed code-generation tasks from newly collected and augmented QA data. Additionally, we isolate distinct financial reasoning capabilities required to solve these QA tasks: reading comprehension of financial text and tables, which is required to extract correct intermediate values; and understanding domain knowledge (e.g., financial formulas) needed to calculate complex solutions. Collectively, these tasks evaluate a model's financial background knowledge, ability to extract numeric entities from financial documents, and capacity to solve problems with code. We conduct an in-depth evaluation of open-source and commercial LLMs, illustrating that BizBench is a challenging benchmark for quantitative reasoning in the finance and business domain.Comment: Work in progres

    Learning Logic Programs by Discovering Higher-Order Abstractions

    Full text link
    Discovering novel abstractions is important for human-level AI. We introduce an approach to discover higher-order abstractions, such as map, filter, and fold. We focus on inductive logic programming, which induces logic programs from examples and background knowledge. We introduce the higher-order refactoring problem, where the goal is to compress a logic program by introducing higher-order abstractions. We implement our approach in STEVIE, which formulates the higher-order refactoring problem as a constraint optimisation problem. Our experimental results on multiple domains, including program synthesis and visual reasoning, show that, compared to no refactoring, STEVIE can improve predictive accuracies by 27% and reduce learning times by 47%. We also show that STEVIE can discover abstractions that transfer to different domain

    LILO: Learning Interpretable Libraries by Compressing and Documenting Code

    Full text link
    While large language models (LLMs) now excel at code generation, a key aspect of software development is the art of refactoring: consolidating code into libraries of reusable and readable programs. In this paper, we introduce LILO, a neurosymbolic framework that iteratively synthesizes, compresses, and documents code to build libraries tailored to particular problem domains. LILO combines LLM-guided program synthesis with recent algorithmic advances in automated refactoring from Stitch: a symbolic compression system that efficiently identifies optimal lambda abstractions across large code corpora. To make these abstractions interpretable, we introduce an auto-documentation (AutoDoc) procedure that infers natural language names and docstrings based on contextual examples of usage. In addition to improving human readability, we find that AutoDoc boosts performance by helping LILO's synthesizer to interpret and deploy learned abstractions. We evaluate LILO on three inductive program synthesis benchmarks for string editing, scene reasoning, and graphics composition. Compared to existing neural and symbolic methods - including the state-of-the-art library learning algorithm DreamCoder - LILO solves more complex tasks and learns richer libraries that are grounded in linguistic knowledge

    Az iskolában elsajátított tudás minőségének értékelése = Evaluation of the quality of students' knowledge

    Get PDF
    Az iskolában elsajátított tudás szerveződésének feltárása érdekében a természettudományos fogalmakat a középpontba állítva vizsgáltuk a fogalmak fejlődését. Megjelent a vizsgálatok eredményeit összegző könyv. Több elemzés mutatta be a tárgyi tudás szerveződésének merevségeit, a többszörös reprezentáció hiányát, és az ebből fakadó alkalmazási problémákat. Sor került a központi jelentőségűnek bizonyult képességek (induktív gondolkodás, analógiás gondolkodás) tanulásban játszott konkrét szerepének kimutatására, a fejlesztési lehetőségek feltérképezésére (pl. problémamegoldás), és fejlesztő kísérletekre (pl. az induktív gondolkodás fejlesztése, az analógiás gondolkodás), továbbá azoknak az iskolában alkalmazható tanítási módszereknek a kipróbálására, amelyek a képességek hatékonyabb fejlesztését segítik (probléma alapú tanítás, projekt módszer, a feladatlapok és az IKT eszközök alkalmazása). Megjelent az analógiás gondolkodás fejlesztésével kapcsolatos kísérleteket bemutató könyv. A komplex problémamegoldás terén a nemzetközi elemzéseknél részletesebb adatfelvételre kerül sor, több korosztály felmérésével lehetőség nyílt a fejlődés elemzésére. Megjelent a vizsgálatokat bemutató könyv. A program több téren továbbfejlesztette a tudás szerveződésével és a képességek fejlődésével kapcsolatos kutatások elméleti kereteit és módszertanát, például a kompetencia-skálák készítésének elméleti megalapozása és a széles életkori intervallumot átfogó fejlődéséi skálák készítése terén. | The main focus of the project has been the study of the organization of knowledge mastered at school. Both propositional knowledge (concept organization, conceptual change) and propositional knowledge (analogical reasoning, inductive reasoning, complex problem solving), as well as the interaction of the two forms of knowledge have been studied. The empirical studies revealed the problems of conceptual change that may be attributed to the lack of deeper understanding and multiple representations. Beyond assessing the development of skills, intervention experiments were also carried out. The intervention methods resulted in significant developmental changes in analogical and inductive reasoning. The research project resulted in a new synthesis of the forms of organization of school knowledge and contributed to the development of methods e.g. to the theoretical bases of creating competency scales and establishing developmental scales by the means of IRT

    IRE: A Framework For Inductive Reverse Engineering

    Get PDF
    abstract: Reverse engineering is critical to reasoning about how a system behaves. While complete access to a system inherently allows for perfect analysis, partial access is inherently uncertain. This is the case foran individual agent in a distributed system. Inductive Reverse Engineering (IRE) enables analysis under such circumstances. IRE does this by producing program spaces consistent with individual input-output examples for a given domain-specific language. Then, IRE intersects those program spaces to produce a generalized program consistent with all examples. IRE, an easy to use framework, allows this domain-specific language to be specified in the form of Theorist s, which produce Theory s, a succinct way of representing the program space. Programs are often much more complex than simple string transformations. One of the ways in which they are more complex is in the way that they follow a conversation-like behavior, potentially following some underlying protocol. As a result, IRE represents program interactions as Conversations in order to more correctly model a distributed system. This, for instance, enables IRE to model dynamically captured inputs received from other agents in the distributed system. While domain-specific knowledge provided by a user is extremely valuable, such information is not always possible. IRE mitigates this by automatically inferring program grammars, allowing it to still perform efficient searches of the program space. It does this by intersecting conversations prior to synthesis in order to understand what portions of conversations are constant. IRE exists to be a tool that can aid in automatic reverse engineering across numerous domains. Further, IRE aspires to be a centralized location and interface for implementing program synthesis and automatic black box analysis techniques.Dissertation/ThesisMasters Thesis Computer Science 201

    Middle-Out Reasoning for Logic Program Synthesis

    Get PDF
    We propose a novel approach to automating the synthesis of logic programs: Logic programs are synthesized as a by-product of the planning of a verification proof. The approach is a two-level one: At the object level, we prove program verification conjectures in a sorted, first-order theory. The conjectures are of the form 8args \Gamma\Gamma\Gamma\Gamma! : prog(args \Gamma\Gamma\Gamma\Gamma! ) $ spec(args \Gamma\Gamma\Gamma\Gamma! ). At the meta-level, we plan the object-level verification with an unspecified program definition. The definition is represented with a (second-order) meta-level variable, which becomes instantiated in the course of the planning

    Event-Object Reasoning with Curated Knowledge Bases: Deriving Missing Information

    Full text link
    The broader goal of our research is to formulate answers to why and how questions with respect to knowledge bases, such as AURA. One issue we face when reasoning with many available knowledge bases is that at times needed information is missing. Examples of this include partially missing information about next sub-event, first sub-event, last sub-event, result of an event, input to an event, destination of an event, and raw material involved in an event. In many cases one can recover part of the missing knowledge through reasoning. In this paper we give a formal definition about how such missing information can be recovered and then give an ASP implementation of it. We then discuss the implication of this with respect to answering why and how questions.Comment: 13 page
    corecore