Combating False Negatives in Adversarial Imitation Learning

In adversarial imitation learning, a discriminator is trained to differentiate agent episodes from expert demonstrations representing the desired behavior. However, as the trained policy learns to be more successful, the negative examples (the ones produced by the agent) become increasingly similar to expert ones. Despite the fact that the task is successfully accomplished in some of the agent's trajectories, the discriminator is trained to output low values for them. We hypothesize that this inconsistent training signal for the discriminator can impede its learning, and consequently leads to worse overall performance of the agent. We show experimental evidence for this hypothesis and that the 'False Negatives' (i.e. successful agent episodes) significantly hinder adversarial imitation learning, which is the first contribution of this paper. Then, we propose a method to alleviate the impact of false negatives and test it on the BabyAI environment. This method consistently improves sample efficiency over the baselines by at least an order of magnitude.Comment: This is an extended version of the student abstract published at 34th AAAI Conference on Artificial Intelligenc

Trial without Error: Towards Safe Reinforcement Learning via Human Intervention

AI systems are increasingly applied to complex tasks that involve interaction with humans. During training, such systems are potentially dangerous, as they haven't yet learned to avoid actions that could cause serious harm. How can an AI system explore and learn without making a single mistake that harms humans or otherwise causes serious damage? For model-free reinforcement learning, having a human "in the loop" and ready to intervene is currently the only way to prevent all catastrophes. We formalize human intervention for RL and show how to reduce the human labor required by training a supervised learner to imitate the human's intervention decisions. We evaluate this scheme on Atari games, with a Deep RL agent being overseen by a human for four hours. When the class of catastrophes is simple, we are able to prevent all catastrophes without affecting the agent's learning (whereas an RL baseline fails due to catastrophic forgetting). However, this scheme is less successful when catastrophes are more complex: it reduces but does not eliminate catastrophes and the supervised learner fails on adversarial examples found by the agent. Extrapolating to more challenging environments, we show that our implementation would not scale (due to the infeasible amount of human labor required). We outline extensions of the scheme that are necessary if we are to train model-free agents without a single catastrophe

BC-IRL: Learning Generalizable Reward Functions from Demonstrations

How well do reward functions learned with inverse reinforcement learning (IRL) generalize? We illustrate that state-of-the-art IRL algorithms, which maximize a maximum-entropy objective, learn rewards that overfit to the demonstrations. Such rewards struggle to provide meaningful rewards for states not covered by the demonstrations, a major detriment when using the reward to learn policies in new situations. We introduce BC-IRL a new inverse reinforcement learning method that learns reward functions that generalize better when compared to maximum-entropy IRL approaches. In contrast to the MaxEnt framework, which learns to maximize rewards around demonstrations, BC-IRL updates reward parameters such that the policy trained with the new reward matches the expert demonstrations better. We show that BC-IRL learns rewards that generalize better on an illustrative simple task and two continuous robotic control tasks, achieving over twice the success rate of baselines in challenging generalization settings

Addressing the new generation of spam (Spam 2.0) through Web usage models

New Internet collaborative media introduce new ways of communicating that are not immune to abuse. A fake eye-catching profile in social networking websites, a promotional review, a response to a thread in online forums with unsolicited content or a manipulated Wiki page, are examples of new the generation of spam on the web, referred to as Web 2.0 Spam or Spam 2.0. Spam 2.0 is defined as the propagation of unsolicited, anonymous, mass content to infiltrate legitimate Web 2.0 applications.The current literature does not address Spam 2.0 in depth and the outcome of efforts to date are inadequate. The aim of this research is to formalise a definition for Spam 2.0 and provide Spam 2.0 filtering solutions. Early-detection, extendibility, robustness and adaptability are key factors in the design of the proposed method.This dissertation provides a comprehensive survey of the state-of-the-art web spam and Spam 2.0 filtering methods to highlight the unresolved issues and open problems, while at the same time effectively capturing the knowledge in the domain of spam filtering.This dissertation proposes three solutions in the area of Spam 2.0 filtering including: (1) characterising and profiling Spam 2.0, (2) Early-Detection based Spam 2.0 Filtering (EDSF) approach, and (3) On-the-Fly Spam 2.0 Filtering (OFSF) approach. All the proposed solutions are tested against real-world datasets and their performance is compared with that of existing Spam 2.0 filtering methods.This work has coined the term ‘Spam 2.0’, provided insight into the nature of Spam 2.0, and proposed filtering mechanisms to address this new and rapidly evolving problem

Literature Review of Credit Card Fraud Detection with Machine Learning

This thesis presents a comprehensive examination of the field of credit card fraud detection, aiming to offer a thorough understanding of its evolution and nuances. Through a synthesis of various studies, methodologies, and technologies, this research strives to provide a holistic perspective on the subject, shedding light on both its strengths and limitations. In the realm of credit card fraud detection, a range of methods and combinations have been explored to enhance effectiveness. This research reviews several noteworthy approaches, including Genetic Algorithms (GA) coupled with Random Forest (GA-RF), Decision Trees (GA-DT), and Artificial Neural Networks (GA-ANN). Additionally, the study delves into outlier score definitions, considering different levels of granularity, and their integration into a supervised framework. Moreover, it discusses the utilization of Artificial Neural Networks (ANNs) in federated learning and the incorporation of Generative Adversarial Networks (GANs) with Modified Focal Loss and Random Forest as the base machine learning algorithm. These methods, either independently or in combination, represent some of the most recent developments in credit card fraud detection, showcasing their potential to address the evolving landscape of digital financial threats. The scope of this literature review encompasses a wide range of sources, including research articles, academic papers, and industry reports, spanning multiple disciplines such as computer science, data science, artificial intelligence, and cybersecurity. The review is organized to guide readers through the progression of credit card fraud detection, commencing with foundational concepts and advancing toward the most recent developments. In today's digital financial landscape, the need for robust defense mechanisms against credit card fraud is undeniable. By critically assessing the existing literature, recognizing emerging trends, and evaluating the effectiveness of various detection methods, this thesis aims to contribute to the knowledge pool within the credit card fraud detection domain. The insights gleaned from this comprehensive review will not only benefit researchers and practitioners but also serve as a roadmap for the enhancement of more adaptive and resilient fraud detection systems. As the ongoing battle between fraudsters and defenders in the financial realm continues to evolve, a deep understanding of the current landscape becomes an asset. This literature review aspires to equip readers with the insights needed to address the dynamic challenges associated with credit card fraud detection, fostering innovation and resilience in the pursuit of secure and trustworthy financial transactions

Development and Validation of a Proof-of-Concept Prototype for Analytics-based Malicious Cybersecurity Insider Threat in a Real-Time Identification System

Insider threat has continued to be one of the most difficult cybersecurity threat vectors detectable by contemporary technologies. Most organizations apply standard technology-based practices to detect unusual network activity. While there have been significant advances in intrusion detection systems (IDS) as well as security incident and event management solutions (SIEM), these technologies fail to take into consideration the human aspects of personality and emotion in computer use and network activity, since insider threats are human-initiated. External influencers impact how an end-user interacts with both colleagues and organizational resources. Taking into consideration external influencers, such as personality, changes in organizational polices and structure, along with unusual technical activity analysis, would be an improvement over contemporary detection tools used for identifying at-risk employees. This would allow upper management or other organizational units to intervene before a malicious cybersecurity insider threat event occurs, or mitigate it quickly, once initiated. The main goal of this research study was to design, develop, and validate a proof-of-concept prototype for a malicious cybersecurity insider threat alerting system that will assist in the rapid detection and prediction of human-centric precursors to malicious cybersecurity insider threat activity. Disgruntled employees or end-users wishing to cause harm to the organization may do so by abusing the trust given to them in their access to available network and organizational resources. Reports on malicious insider threat actions indicated that insider threat attacks make up roughly 23% of all cybercrime incidents, resulting in $2.9 trillion in employee fraud losses globally. The damage and negative impact that insider threats cause was reported to be higher than that of outsider or other types of cybercrime incidents. Consequently, this study utilized weighted indicators to measure and correlate simulated user activity to possible precursors to malicious cybersecurity insider threat attacks. This study consisted of a mixed method approach utilizing an expert panel, developmental research, and quantitative data analysis using the developed tool on simulated data set. To assure validity and reliability of the indicators, a panel of subject matter experts (SMEs) reviewed the indicators and indicator categorizations that were collected from prior literature following the Delphi technique. The SMEs’ responses were incorporated into the development of a proof-of-concept prototype. Once the proof-of-concept prototype was completed and fully tested, an empirical simulation research study was conducted utilizing simulated user activity within a 16-month time frame. The results of the empirical simulation study were analyzed and presented. Recommendations resulting from the study also be provided

Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review

This systematic literature review comprehensively examines the application of Large Language Models (LLMs) in forecasting and anomaly detection, highlighting the current state of research, inherent challenges, and prospective future directions. LLMs have demonstrated significant potential in parsing and analyzing extensive datasets to identify patterns, predict future events, and detect anomalous behavior across various domains. However, this review identifies several critical challenges that impede their broader adoption and effectiveness, including the reliance on vast historical datasets, issues with generalizability across different contexts, the phenomenon of model hallucinations, limitations within the models' knowledge boundaries, and the substantial computational resources required. Through detailed analysis, this review discusses potential solutions and strategies to overcome these obstacles, such as integrating multimodal data, advancements in learning methodologies, and emphasizing model explainability and computational efficiency. Moreover, this review outlines critical trends that are likely to shape the evolution of LLMs in these fields, including the push toward real-time processing, the importance of sustainable modeling practices, and the value of interdisciplinary collaboration. Conclusively, this review underscores the transformative impact LLMs could have on forecasting and anomaly detection while emphasizing the need for continuous innovation, ethical considerations, and practical solutions to realize their full potential

Plausible Cause : Explanatory Standards in the Age of Powerful Machines

Much scholarship in law and political science has long understood the U.S. Supreme Court to be the apex court in the federal judicial system, and so to relate hierarchically to lower federal courts. On that top-down view, exemplified by the work of Alexander Bickel and many subsequent scholars, the Court is the principal, and lower federal courts are its faithful agents. Other scholarship takes a bottom-up approach, viewing lower federal courts as faithless agents or analyzing the percolation of issues in those courts before the Court decides. This Article identifies circumstances in which the relationship between the Court and other federal courts is best viewed as neither top-down nor bottom-up, but side-by-side. When the Court intervenes in fierce political conflicts, it may proceed in stages, interacting with other federal courts in a way that is aimed at enhancing its public legitimacy. First, the Court renders a decision that is interpreted as encouraging, but not requiring, other federal courts to expand the scope of its initial ruling. Then, most federal courts do expand the scope of the ruling, relying upon the Court\u27s initial decision as authority for doing so. Finally, the Court responds by invoking those district and circuit court decisions as authority for its own more definitive resolution. That dialectical process, which this Article calls reciprocal legitimation, was present along the path from Brown v. Board of Education to the unreasoned per curiams, from Baker v. Carr to Reynolds v. Sims, and from United States v. Windsor to Obergefell v. Hodges-as partially captured by Appendix A to the Court\u27s opinion in Obergefell and the opinion\u27s several references to it. This Article identifies the phenomenon of reciprocal legitimation, explains that it may initially be intentional or unintentional, and examines its implications for theories of constitutional change and scholarship in federal courts and judicial politics. Although the Article\u27s primary contribution is descriptive and analytical, it also normatively assesses reciprocal legitimation given the sacrifice of judicial candor that may accompany it. A Coda examines the likelihood and desirability of reciprocal legitimation in response to President Donald Trump\u27s derision of the federal courts as political and so illegitimate

Plausible Cause : Explanatory Standards in the Age of Powerful Machines

The Fourth Amendment\u27s probable cause requirement is not about numbers or statistics. It is about requiring the police to account for their decisions. For a theory of wrongdoing to satisfy probable cause-and warrant a search or seizure-it must be plausible. The police must be able to explain why the observed facts invite an inference of wrongdoing, and judges must have an opportunity to scrutinize that explanation. Until recently, the explanatory aspect of Fourth Amendment suspicion- plausible cause -has been uncontroversial, and central to the Supreme Court\u27s jurisprudence, for a simple reason: explanations have served, in practice, as a guarantor of statistical likelihood. In other words, forcing police to articulate theories of wrongdoing is the means by which courts have traditionally ensured that (roughly) the right persons, houses, papers, and effects are targeted for intrusion. Going forward, however, technological change promises to disrupt the harmony between explanatory standards and statistical accuracy. Powerful machines enable a previously impossible combination: accurate predictions unaccompanied by explanations. As that change takes hold, we will need to think carefully about why explanation-giving matters. When judges assess the sufficiency of explanations offered by police (and other officials), what are they doing? If the answer comes back to error­ reduction-if the point of judicial oversight is simply to maximize the overall number of accurate decisions-machines could theoretically do the job as well as, if not better than, humans. But if the answer involves normative goals beyond error-reduction, automated tools-no matter their power-will remain, at best, partial substitutes for judicial scrutiny. This Article defends the latter view. I argue that statistical accuracy, though important, is not the crux of explanation-giving. Rather, explanatory standards-like probable cause-hold officials accountable to a plurality of sometimes-conflicting constitutional and rule-of-law values that, in our legal system, bound the scope of legitimate authority. Error-reduction is one such value. But there are many others, and sometimes the values work at cross purposes. When judges assess explanations, they navigate a space of value­pluralism: they identify which values are at stake in a given decisional environment and ask, where necessary, if those values have been properly balanced. Unexplained decisions render this process impossible and, in so doing, hobble the judicial role. Ultimately, that role has less to do with analytic power than practiced wisdom. A common argument against replacing judges, and other human experts, with intelligent machines is that machines are not (yet) intelligent enough to take up the mantle. In the age of powerful algorithms, however, this turns out to be a weak-and temporally limited-claim. The better argument, I suggest in closing, is that judging is not solely, or even primarily, about intelligence. It is about prudence