RLBOA: A modular reinforcement learning framework for autonomous negotiating agents

Negotiation is a complex problem, in which the variety of settings and opponents that may be encountered prohibits the use of a single predefined negotiation strategy. Hence the agent should be able to learn such a strategy autonomously. To this end we propose RLBOA, a modular framework that facilitates the creation of autonomous negotiation agents using reinforcement learning. The framework allows for the creation of agents that are capable of negotiating effectively in many different scenarios. To be able to cope with the large size of the state and action spaces and diversity of settings, we leverage the modular BOA-framework. This decouples the negotiation strategy into a Bidding strategy, an Opponent model and an Acceptance condition. Furthermore, we map the multidimensional contract space onto the utility axis which enables a compact and generic state and action description. We demonstrate the value of the RLBOA framework by implementing an agent that uses tabular Q-learning on the compressed state and action space to learn a bidding strategy.We show that the resulting agent is able to learn well-performing bidding strategies in a range of negotiation settings and is able to generalize across opponents and domains

Postural change in volunteers: sympathetic tone determines microvascular response to cardiac preload and output increases

Purpose: Microvascular perfusion may be a non-invasive indicator of fluid responsiveness. We aimed to investigate which of the microvascular perfusion parameters truly reflects fluid responsiveness independent of sympathetic reflexes. Methods: Fifteen healthy volunteers underwent a postural change from head up tilt (HUT) to the supine position, diminishing sympathetic tone, followed by a 30° passive leg raising (PLR) with unaltered tone. Prior to and after the postural changes, stroke volume (SV) and cardiac output (CO) were measured, as well as sublingual microcirculatory perfusion (sidestream dark field imaging), skin perfusion, and oxygenation (laser Doppler flowmetry and reflectance spectroscopy). Results: In responders (subjects with >10 % increase in CO), the HUT to supine change increased CO, SV, and pulse pressure, while heart rate, systemic vascular resistance, and mean arterial pressure decreased. Additionally, microvascular flow index, laser Doppler flow, and microvascular hemoglobin oxygen saturation and concentration also increased. Conclusion: When preload and forward flow increase in association with a decrease in sympathetic activity, microvascular blood flow increases in the skin and in the sublingual area. When preload and forward flow increase with little to no change in sympathetic activity, only sublingual functional capillary density increases. Therefore, our results indicate that sublingual functional capillary density is the best pa

Tissue perfusion and oxygenation to monitor fluid responsiveness in critically ill, septic patients after initial resuscitation: a prospective observational study

Fluid therapy after initial resuscitation in critically ill, septic patients may lead to harmful overloading and should therefore be guided by indicators of an increase in stroke volume (SV), i.e. fluid responsiveness. Our objective was to investigate whether tissue perfusion and oxygenation are able to monitor fluid responsiveness, even after initial resuscitation. Thirty-five critically ill, septic patients underwent infusion of 250 mL of colloids, after initial fluid resuscitation. Prior to and after fluid infusion, SV, cardiac output sublingual microcirculatory perfusion (SDF: sidestream dark field imaging) and skin perfusion and oxygenation (laser Doppler flowmetry and reflectance spectroscopy) were measured. Fluid responsiveness was defined by a ≥5 or 10 % increase in SV upon fluids. In responders to fluids, SDF-derived microcirculatory and skin perfusion and oxygenation increased, but only the increase in cardiac output, mean arterial and pulse pressure, microvascular flow index and relative Hb concentration and oxygen saturation were able to monitor a SV increase. Our proof of principle study demonstrates that non-invasively assessed tissue perfusion and oxygenation is not inferior to invasive hemodynamic measurements in monitoring fluid responsiveness. However skin reflectance spectroscopy may be more helpful than sublingual SDF

Nitroglycerin reverts clinical manifestations of poor peripheral perfusion in patients with circulatory shock

Introduction: Recent clinical studies have shown a relationship between abnormalities in peripheral perfusion and unfavorable outcome in patients with circulatory shock. Nitroglycerin is effective in restoring alterations in microcirculatory blood flow. The aim of this study was to investigate whether nitroglycerin could correct the parameters of abnormal peripheral circulation in resuscitated circulatory shock patients.Methods: This interventional study recruited patients who had circulatory shock and who persisted with abnormal peripheral perfusion despite normalization of global hemodynamic parameters. Nitroglycerin started at 2 mg/hour and doubled stepwise (4, 8, and 16 mg/hour) each 15 minutes until an improvement in peripheral perfusion was observed. Peripheral circulation parameters included capillary refill time (CRT), skin-temperature gradient (Tskin-diff), perfusion index (PI), and tissue oxygen saturation (StO2) during a reactive hyperemia test (RincStO2). Measurements were performed before, at the maximum dose, and after cessation of nitroglycerin infusion. Data were analyzed by using linear model for repeated measurements and are presented as mean (standard error).Results: Of the 15 patients included, four patients (27%) responded with an initial nitroglycerin dose of 2 mg/hour. In all patients, nitroglycerin infusion resulted in significant changes in CRT, Tskin-diff, and PI toward normal at the maximum dose of nitroglycerin: from 9.4 (0.6) seconds to 4.8 (0.3) seconds (P <0.05), from 3.3°C (0.7°C) to 0.7°C (0.6°C) (P <0.05), and from [log] -0.5% (0.2%) to 0.7% (0.1%) (P <0.05), respectively. Similar changes in StO2 and RincStO2 were observed: from 75% (3.4%) to 84% (2.7%) (P <0.05) and 1.9%/second (0.08%/second) to 2.8%/second (0.05%/second) (P <0.05), respectively. The magnitude of changes in StO2 was more prono

Clinical assessment of peripheral perfusion to predict postoperative complications after major abdominal surgery early: A prospective observational study in adults

Introduction: Altered peripheral perfusion is strongly associated with poor outcome in critically ill patients. We wanted to determine whether repeated assessments of peripheral perfusion during the days following surgery could help to early identify patients that are more likely to develop postoperative complications.Methods: Haemodynamic measurements and peripheral perfusion parameters were collected one day prior to surgery, directly after surgery (D0) and on the first (D1), second (D2) and third (D3) postoperative days. Peripheral perfusion assessment consisted of capillary refill time (CRT), peripheral perfusion index (PPI) and forearm-to-fingertip skin temperature gradient (Tskin-diff). Generalized linear mixed models were used to predict severe complications within ten days after surgery based on Clavien-Dindo classification.Results: We prospectively followed 137 consecutive patients, from among whom 111 were included in the analysis. Severe complications were observed in 19 patients (17.0%). Postoperatively, peripheral perfusion parameters were significantly altered in patients who subsequently developed severe complications compared to those who did not, and these parameters persisted over time. CRT was altered at D0, and PPI and Tskin-diff were altered on D1 and D2, respectively. Among the different peripheral perfusion parameters, the diagnostic accuracy in predicting severe postoperative complications was highest for CRT on D2 (area under the receiver operating characteristic curve = 0.91 (95% confidence interval (CI) = 0.83 to 0.92)) with a sensitivity of 0.79 (95% CI = 0.54 to 0.94) and a specificity of 0.93 (95% CI = 0.86 to 0.97). Generalized mixed-model analysis demonstrated that abnormal peripheral perfusion on D2 and D3 was an independent predictor of severe postoperative complications (D2 odds ratio (OR) = 8.4, 95% CI = 2.7 to 25.9; D2 OR = 6.4, 95% CI = 2.1 to 19.6).Conclusions: In a group of patients assessed following major abdominal surgery, peripheral perfusion alterations were associated with the development of severe complications independently of systemic haemodynamics. Further research is needed to confirm these findings and to explore in more detail the effects of peripheral perfusion-targeted resuscitation following major abdominal surgery

Heterogeneously Catalyzed Continuous-Flow Hydrogenation Using Segmented Flow in Capillary Columns

Segmented flow in standard GC capillary columns, with a heterogeneous Pd catalyst on the walls, gave rapid information about catalytic processes in them. The residence time and conversion was monitored visually, greatly simplifying bench-scale optimization. Examples show the benefits of the elimination of pore diffusion and axial dispersion. Further, we demonstrated how to quickly identify deactivating species in multistep synthesis without intermediate workup

Trajectories of renal biomarkers and new-onset heart failure in the general population:Findings from the PREVEND study

AIMS: Renal dysfunction is one of the most critical risk factors for developing heart failure (HF). However, the association between repeated measures of renal function and incident HF remains unclear. Therefore, this study investigated the longitudinal trajectories of urinary albumin excretion (UAE) and serum creatinine and their association with new-onset HF and all-cause mortality.METHODS AND RESULTS: Using group-based trajectory analysis, we estimated trajectories of UAE and serum creatinine in 6881 participants from the Prevention of Renal and Vascular End-stage Disease (PREVEND) study and their association with new-onset HF and all-cause death during the 11-years of follow-up. Most participants had stable low UAE or serum creatinine. Participants with persistently higher UAE or serum creatinine were older, more often men, and more often had comorbidities, such as diabetes, a previous myocardial infarction or dyslipidaemia. Participants with persistently high UAE had a higher risk of new-onset HF or all-cause mortality, whereas stable serum creatinine trajectories showed a linear association for new-onset HF and no association with all-cause mortality.CONCLUSION: Our population-based study identified different but often stable longitudinal patterns of UAE and serum creatinine. Patients with persistently worse renal function, such as higher UAE or serum creatinine, were at a higher risk of HF or mortality.</p

HFrEF subphenotypes based on 4210 repeatedly measured circulating proteins are driven by different biological mechanisms

Background: HFrEF is a heterogenous condition with high mortality. We used serial assessments of 4210 circulating proteins to identify distinct novel protein-based HFrEF subphenotypes and to investigate underlying dynamic biological mechanisms. Herewith we aimed to gain pathophysiological insights and fuel opportunities for personalised treatment. Methods: In 382 patients, we performed trimonthly blood sampling during a median follow-up of 2.1 [IQR:1.1–2.6] years. We selected all baseline samples and two samples closest to the primary endpoint (PEP; composite of cardiovascular mortality, HF hospitalization, LVAD implantation, and heart transplantation) or censoring, and applied an aptamer-based multiplex proteomic approach. Using unsupervised machine learning methods, we derived clusters from 4210 repeatedly measured proteomic biomarkers. Sets of proteins that drove cluster allocation were analysed via an enrichment analysis. Differences in clinical characteristics and PEP occurrence were evaluated. Findings: We identified four subphenotypes with different protein profiles, prognosis and clinical characteristics, including age (median [IQR] for subphenotypes 1–4, respectively:70 [64, 76], 68 [60, 79], 57 [47, 65], 59 [56, 66]years), EF (30 [26, 36], 26 [20, 38], 26 [22, 32], 33 [28, 37]%), and chronic renal failure (45%, 65%, 36%, 37%). Subphenotype allocation was driven by subsets of proteins associated with various biological functions, such as oxidative stress, inflammation and extracellular matrix organisation. Clinical characteristics of the subphenotypes were aligned with these associations. Subphenotypes 2 and 3 had the worst prognosis compared to subphenotype 1 (adjHR (95%CI):3.43 (1.76–6.69), and 2.88 (1.37–6.03), respectively). Interpretation: Four circulating-protein based subphenotypes are present in HFrEF, which are driven by varying combinations of protein subsets, and have different clinical characteristics and prognosis. Clinical Trial Registration: ClinicalTrials.gov Identifier: NCT01851538 https://clinicaltrials.gov/ct2/show/NCT01851538. Funding: EU/ EFPIA IMI2JU BigData@Heart grant n° 116074, Jaap Schouten Foundation and Noordwest Academie.</p

Machine learning–based biomarker profile derived from 4210 serially measured proteins predicts clinical outcome of patients with heart failure

Aims Risk assessment tools are needed for timely identification of patients with heart failure (HF) with reduced ejection fraction (HFrEF) who are at high risk of adverse events. In this study, we aim to derive a small set out of 4210 repeatedly measured proteins, which, along with clinical characteristics and established biomarkers, carry optimal prognostic capacity for adverse events, in patients with HFrEF. Methods and results In 382 patients, we performed repeated blood sampling (median follow-up: 2.1 years) and applied an aptamer-based multiplex proteomic approach. We used machine learning to select the optimal set of predictors for the primary endpoint (PEP: composite of cardiovascular death, heart transplantation, left ventricular assist device implantation, and HF hospitalization). The association between repeated measures of selected proteins and PEP was investigated by multivariable joint models. Internal validation (cross-validated c-index) and external validation (Henry Ford HF PharmacoGenomic Registry cohort) were performed. Nine proteins were selected in addition to the MAGGIC risk score, N-terminal pro-hormone B-type natriuretic peptide, and troponin T: suppression of tumourigenicity 2, tryptophanyl-tRNA synthetase cytoplasmic, histone H2A Type 3, angiotensinogen, deltex-1, thrombospondin-4, ADAMTS-like protein 2, anthrax toxin receptor 1, and cathepsin D. N-terminal pro-hormone B-type natriuretic peptide and angiotensinogen showed the strongest associations [hazard ratio (95% confidence interval): 1.96 (1.17–3.40) and 0.66 (0.49–0.88), respectively]. The multivariable model yielded a c-index of 0.85 upon internal validation and c-indices up to 0.80 upon external validation. The c-index was higher than that of a model containing established risk factors (P = 0.021). Conclusion Nine serially measured proteins captured the most essential prognostic information for the occurrence of adverse events in patients with HFrEF, and provided incremental value for HF prognostication beyond established risk factors. These proteins could be used for dynamic, individual risk assessment in a prospective setting. These findings also illustrate the potential value of relatively ‘novel’ biomarkers for prognostication.</p

Machine learning–based biomarker profile derived from 4210 serially measured proteins predicts clinical outcome of patients with heart failure

Aims: Risk assessment tools are needed for timely identification of patients with heart failure (HF) with reduced ejection fraction (HFrEF) who are at high risk of adverse events. In this study, we aim to derive a small set out of 4210 repeatedly measured proteins, which, along with clinical characteristics and established biomarkers, carry optimal prognostic capacity for adverse events, in patients with HFrEF. Methods and results: In 382 patients, we performed repeated blood sampling (median follow-up: 2.1 years) and applied an aptamer-based multiplex proteomic approach. We used machine learning to select the optimal set of predictors for the primary endpoint (PEP: composite of cardiovascular death, heart transplantation, left ventricular assist device implantation, and HF hospitalization). The association between repeated measures of selected proteins and PEP was investigated by multivariable joint models. Internal validation (cross-validated c-index) and external validation (Henry Ford HF PharmacoGenomic Registry cohort) were performed. Nine proteins were selected in addition to the MAGGIC risk score, N-terminal pro-hormone B-type natriuretic peptide, and troponin T: suppression of tumourigenicity 2, tryptophanyl-tRNA synthetase cytoplasmic, histone H2A Type 3, angiotensinogen, deltex-1, thrombospondin-4, ADAMTS-like protein 2, anthrax toxin receptor 1, and cathepsin D. N-terminal pro-hormone B-type natriuretic peptide and angiotensinogen showed the strongest associations [hazard ratio (95% confidence interval): 1.96 (1.17–3.40) and 0.66 (0.49–0.88), respectively]. The multivariable model yielded a c-index of 0.85 upon internal validation and c-indices up to 0.80 upon external validation. The c-index was higher than that of a model containing established risk factors (P = 0.021). Conclusion: Nine serially measured proteins captured the most essential prognostic information for the occurrence of adverse events in patients with HFrEF, and provided incremental value for HF prognostication beyond established risk factors. These proteins could be used for dynamic, individual risk assessment in a prospective setting. These findings also illustrate the potential value of relatively ‘novel’ biomarkers for prognostication