Models of human preference for learning reward functions

The utility of reinforcement learning is limited by the alignment of reward functions with the interests of human stakeholders. One promising method for alignment is to learn the reward function from human-generated preferences between pairs of trajectory segments, a type of reinforcement learning from human feedback (RLHF). These human preferences are typically assumed to be informed solely by partial return, the sum of rewards along each segment. We find this assumption to be flawed and propose modeling human preferences instead as informed by each segment's regret, a measure of a segment's deviation from optimal decision-making. Given infinitely many preferences generated according to regret, we prove that we can identify a reward function equivalent to the reward function that generated those preferences, and we prove that the previous partial return model lacks this identifiability property in multiple contexts. We empirically show that our proposed regret preference model outperforms the partial return preference model with finite training data in otherwise the same setting. Additionally, we find that our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human-aligned. Overall, this work establishes that the choice of preference model is impactful, and our proposed regret preference model provides an improvement upon a core assumption of recent research. We have open sourced our experimental code, the human preferences dataset we gathered, and our training and preference elicitation interfaces for gathering a such a dataset.Comment: 16 pages (40 pages with references and appendix), 23 figure

Values, practices, and the utilization of empirical critiques in the clinical triad

Empirical science is composed of a set of mutually reinforcing values and practices. A potential difficulty arises when empirical knowledge products are disseminated to other groups with an interest in research findings. The danger is that researchers\u27 values and practices will be deemed superior to those of other parties, and codified across different contexts without consideration of their effectiveness in achieving broader goals of science (e.g., sharing important knowledge about treating problems in living). Alternatively, understanding and respecting how the values and practices of different groups are situated in local decision-making contexts can open up creative ways for enhancing collaboration between different members of the research and clinical practice community. Taking Scheel\u27s (this issue) thorough and well-crafted critique of research on dialectical behavior therapy as an example, we explore the way values, practices, and local decision-making contexts affect researchers\u27, practitioners\u27, and clinical administrators\u27 reactions to empirical knowledge products

Baastrup's Disease, Interspinal Bursitis, and Dorsal Epidural Cysts: Radiologic Evaluation and Impact on Treatment Options

Baastrup's disease or "kissing spines syndrome" was first described as a cause of lumbar pain before computerized tomography (CT) and magnetic resonance imaging (MRI) scanning existed. The diagnosis was based on x-ray studies, which showed that the spinous processes, especially in the lower lumbar spine, became approximated to each other and this was a generator of positional back pain. Biomechanically, the interspinous and supraspinous ligaments that are degenerated in Baastrup's disease normally contribute significantly to sagittal alignment. Ligamentous stenosis and anterolisthesis would be the expected pathology with deterioration of these ligaments and were initially described on CT and MRI in patients with symptoms similar to Baastrup's disease as isolated individual case reports. This review will highlight the relationship between the various clinical presentations, biomechanics, and overlap of Baastrup's disease with interspinous bursitis, segmental stenosis, and instability, presenting them as a disease continuum rather than as separate disease processes

Evaluation and Interventional Management of Pain After Vertebral Augmentation Procedures

INTRODUCTION: A small subset of patients who underwent successful vertebral compression fracture (VCF) augmentation procedures may develop subsequent pain requiring spinal injections. In a retrospective analysis, we determined whether the pain was related to the original fracture site or to another area within the lumbar or thoracic spine. The pain occurred either at the same/adjacent level and/or non-adjacent level as the VCF. Interventional treatments primarily targeted the facet joints, specifically in the form of facet joint blocks and/or radiofrequency ablation to the medial branches. The pattern of facet injections relative to the original fracture level was studied. Additionally, the elapsed time between the vertebral augmentation and the subsequent interventional blocks was also evaluated. METHODS: A total of 56 patients sustained VCFs. 12 of these patients underwent interventional procedures after vertebral augmentation procedures. The level(s) of same/adjacent level and non-adjacent level pain were determined via physical examination and/or imaging studies. These levels were subsequently treated with interventional procedures primarily focused on the facet joints. The time period of the injections varied from two weeks status post-vertebral augmentation to as late as 304 weeks (5.8 years) status post-vertebral augmentation. RESULTS: We performed 25 vertebral augmentation procedures on these 12 patients. 15 lumbar, eight lower thoracic, and two mid-thoracic VCFs were augmented. 9/14 cases of blocks included those performed at non-adjacent levels, whereas 5/14 cases of blocks were performed only at the same and/or adjacent levels as the VCF. For the events in which thoracic VCFs were augmented, 6/7 (or 86%) had developed non-adjacent level pain in areas of the lumbar spine.  The time from vertebral augmentation procedure to subsequent pain procedure ranged from two weeks to five plus years. The average time elapsed was 83 weeks. Only one case required blocks performed within the first six weeks after vertebral augmentation. In this case, the blocks included those at non-adjacent levels. A total of 4/12 cases (33%) had a block within 12 weeks of the original vertebral augmentation procedure. Lumbar spine imaging showed that at least 9/12 patients had pre-existing significant lumbar pathology at the time of fracture treatment. This may have contributed to the later development of pain. CONCLUSION: Pain after a successful vertebral augmentation is typically non-acute (i.e., beyond six weeks). Mechanisms other than the primary VCF are usually responsible for non-adjacent level pain, which are present a majority of the time on reviewing the patients' diagnostic studies. These mechanisms usually take many weeks to develop and subsequently elicit pain that requires additional interventional pain procedures. In our study, the pain is usually related to the pre-existing degenerative spondylosis and stenosis rather than the fracture site. This study shows that the facet joints in closely related lumbar degenerative changes are the cause of pain in this patient group. These procedures should be explored with pain after vertebral augmentation, especially in those patients with known or suspected spinal degeneration and/or poor biomechanics

Delayed Recognition of Thoracic and Lumbar Vertebral Compression Fractures in Minor Accident Cases

Osteoporotic vertebral compression fractures (VCFs) in the elderly are commonly diagnosed after a minor fall or trauma; however, the majority of these patients have either been previously evaluated for osteoporosis or are already under some form of medical treatment for osteoporosis at the time of the fall. Although accidents are a known cause of VCFs, these fractures are too often undiagnosed. In reviewing a group of patients seen after minor falls or automobile accidents who were complaining of general spine pain, we found a smaller subgroup with previously undiagnosed VCFs. These fractures were also the initial signs of a previously unrecognized osteoporotic process. Initial diagnosis, treatment, and therapy were usually focused on other spinal segments (i.e. mainly the lumbar spine) until both the VCF and the osteoporosis were identified. The purpose of this report is to raise awareness and discuss the steps for improved diagnosis of osteoporotic VCFs. A retrospective analysis was conducted on a large group of patients from one pain/accident clinic in a 24 month period. These patients were diagnosed with VCFs subsequent to the initial evaluation due to either persistent pain after conservative therapy or complaints of pain beyond the original injured area (i.e. typically the lumbar spine). At this point, a more detailed history was taken, including any past treatment for osteoporosis, or previous falls or injury to exclude the possibility of pre-existing fractures. A more focused examination of the painful area was completed, consisting of percussion at the fracture site identified on magnetic resonance imaging (MRI) or computed tomography (CT) scan. If possible, a bone scan was ordered to separate acute and subacute traumatic fractures from old/chronic fractures. Additionally, we surveyed two other similar pain/accident clinics who saw a comparable number and population of patients diagnosed with VCFs within a 24 month period to make a comparison of the number of VCFs they identified. Ten out of approximately 2700 patients seen over a 24 month period sustained acute thoracic or lumbar VCFs during a minor accident and were not previously diagnosed with osteoporosis. Since approximately 30% of the 2,700 patients had new accidents, 10 out of 800 new patients (1.25%) were found to have VCFs without a known history of osteoporosis. Two other surveyed pain/accident, clinics saw a similar number and population of patients in the same time period; however, each only diagnosed two or three VCFs while examining a similar number of patients in the clinic. In these two other clinics, a much lower percentage (0.3%) of patients were diagnosed with new VCFs. Awareness of the possibility of osteoporotic VCFs is the first step in detecting them. This study reveals the presence of a small but real risk of overlooking osteoporotic VCFs in minor trauma cases. When necessary, repeat or obtain better quality imaging in spinal segments affected by persistent pain. The thoracolumbar junction (i.e. T12 & L1 vertebrae) is especially at risk for sustaining VCFs. The delayed recognition of these VCFs and the patient's underlying osteoporosis after minor accident cases could present a major problem, as the critical time for patients to receive the proper medical or surgical treatments responsible for correcting and preventing further spinal deformity and pain has been reduced

Attributions of responsibility for addiction: the effects of gender and type of substance.

In 1997, 248 urban university students in central Massachusetts rated responsibility for addiction using the Attributions of Responsibility for Addiction Scale (ARAS), developed for this study with university-based financial support. The vignette-based factorial design varied sex of drug user and type of addictive substance. Factor analysis yielded two subscales: internal and external responsibility attributions; the dependent variable was the internal-to-external attribution ratio. Analysis of variance indicated main effects for substance type and drug abuse experience and showed interaction effects for participant\u27s sex by user\u27s sex and user\u27s sex by substance type. Authors discuss implications, study limitations, and future research

Barriers to dissemination of evidence-based practices: addressing practitioners\u27 concerns about manual-Based Psychotherapies

The last several years have seen much debate over the appropriateness and viability of empirically supported manual-based psychotherapies for clinical practice. While the majority of discussions have focused on the strengths or weaknesses of evidence-based treatments, and the differences between research and clinical practice, scant attention has been paid to addressing the actual concerns of practitioners in clinical settings. Based on the available research, and our experiences with training and supervision in manual-based treatments, we discuss practitioners\u27 most common concerns, including (a) effects on the therapeutic relationship, (b) unmet client needs, (c) competence and job satisfaction, (d) treatment credibility, (e) restriction of clinical innovation, and (f) feasibility of manual-based treatments. Rather than arguing that these concerns are unwarranted, we suggest future directions the field must take if evidence-based treatments are to be viable and effective in clinical practice. Starting with the assumption that these treatments have much (but not everything) to offer practitioners in clinical settings leads to qualitative and quantitative research questions involving all parties with an interest in evidence-based practice

Targeted Intraspinal Radiofrequency Ablation for Lumbar Spinal Stenosis

INTRODUCTION: By using a combination of magnetic resonance imaging (MRI) and computed tomography (CT) of the lumbar spine, it is possible to distinguish between spinal stenosis caused by bone compression and specific soft tissue epidural intraspinal lesions that cause localized spinal canal stenosis and neural compression. Examples include facet cysts and yellow ligament hypertrophy. Many of these patients are elderly with medical comorbidities that make open surgery problematic. MATERIALS & METHODS: This is a study of patients with predominantly soft tissue stenosis being treated with targeted intraspinal radiofrequency (RF) heat ablation. This novel procedure is performed under local anesthesia in an outpatient setting using intra-operative imaging. Fine tip 20 gauge RF electrodes (Stryker® PA, USA) are precisely placed under radiologic guidance in the identified soft tissue causing the posterior compression of the lumbar spinal canal. After sensory and motor testing to make sure there is a safe distance of the needle tip from the nearby nerve roots to avoid any neural effect, multiple targeted lesions correlated by the MRI or CT scan are made in the fibrous and cystic soft tissue. Lesions are created using a focused 2 or 5 mm tip at 60 degrees centigrade (°C) for either 30 or 60 seconds. This heat causes sufficient shrinking of the targeted soft tissue resulting in relative reduction of the soft tissue component of the stenosis. This relative reduction in the stenosis of the spinal canal, similar to that measured with interspinous devices, provides long-term relief of symptoms, signs, and improvement of spinal motion in patients with lumbar stenosis. This report will review the spinal anatomy, and development and history of using RF in and around the nerve roots and epidural space, as it relates to lumbar stenosis. Examples of before and after MRI scans demonstrate the radiologic reduction in the size of the lesions. This soft tissue reduction correlates with patients' improvement in pain and clinical symptoms. Follow-up of the patients up to 30 months shows that the effect of RF heat on the soft tissue is long lasting. RESULTS: In our long-term follow-up of greater than six months, 58% of RF treated patients had lasting relief of clinical symptoms, back pain, and claudication with increased spinal movement. This reduction in pain and improvement in motion allows patients to continue more aggressive physical therapy and muscle strengthening that secondarily can improve their symptoms. Post-procedure follow-up MRI scans in multiple patients have shown a clear reduction in soft tissue lesion size. Long-term follow-up demonstrated that 58% of patients treated with RF targeted ablation have not required further intervention and 22% went on to other surgical treatments for lumbar spinal stenosis. CONCLUSION: By reducing the soft tissue component of the stenosis with RF ablation and creating relatively more epidural space, targeted intraspinal RF may be a possible minimally invasive, percutaneous non-surgical alternative to treatment in a number of patients where soft tissue lumbar stenosis is the main cause of patients' symptoms. This technique offers a simple and safe additional method to relieve symptoms of lumbar stenosis and possibly compression within the neural foramina, especially in the elderly