Foley Music: Learning to Generate Music from Videos

In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We then formulate music generation from videos as a motion-to-MIDI translation problem. We present a Graph$-$Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements. The MIDI event can then be converted to realistic music using an off-the-shelf music synthesizer tool. We demonstrate the effectiveness of our models on videos containing a variety of music performances. Experimental results show that our model outperforms several existing systems in generating music that is pleasant to listen to. More importantly, the MIDI representations are fully interpretable and transparent, thus enabling us to perform music editing flexibly. We encourage the readers to watch the demo video with audio turned on to experience the results.Comment: ECCV 2020. Project page: http://foley-music.csail.mit.ed

Prediction of Suicide-Related Events by Analyzing Electronic Medical Records from PTSD Patients with Bipolar Disorder

Around 800,000 people worldwide die from suicide every year and it’s the 10th leading cause of death in the US. It is of great value to build a mathematic model that can accurately predict suicide especially in high-risk populations. Several different ML-based models were trained and evaluated using features obtained from electronic medical records (EMRs). The contribution of each feature was calculated to determine how it impacted the model predictions. The best-performing model was selected for analysis and decomposition. Random forest showed the best performance with true positive rates (TPR) and positive predictive values (PPV) of greater than 80%. The use of Aripiprazole, Levomilnacipran, Sertraline, Tramadol, Fentanyl, or Fluoxetine, a diagnosis of autistic disorder, schizophrenic disorder, or substance use disorder at the time of a diagnosis of both PTSD and bipolar disorder, were strong indicators for no SREs within one year. The use of Trazodone and Citalopram at baseline predicted the onset of SREs within one year. Additional features with potential protective or hazardous effects for SREs were identified by the model. We constructed an ML-based model that was successful in identifying patients in a subpopulation at high-risk for SREs within a year of diagnosis of both PTSD and bipolar disorder. The model also provides feature decompositions to guide mechanism studies. The validation of this model with additional EMR datasets will be of great value in resource allocation and clinical decision making

DeepBiomarker: Identifying Important Lab Tests from Electronic Medical Records for the Prediction of Suicide-Related Events among PTSD Patients

Identifying patients with high risk of suicide is critical for suicide prevention. We examined lab tests together with medication use and diagnosis from electronic medical records (EMR) data for prediction of suicide-related events (SREs; suicidal ideations, attempts and deaths) in post-traumatic stress disorder (PTSD) patients, a population with a high risk of suicide. We developed DeepBiomarker, a deep-learning model through augmenting the data, including lab tests, and integrating contribution analysis for key factor identification. We applied DeepBiomarker to analyze EMR data of 38,807 PTSD patients from the University of Pittsburgh Medical Center. Our model predicted whether a patient would have an SRE within the following 3 months with an area under curve score of 0.930. Through contribution analysis, we identified important lab tests for suicide prediction. These identified factors imply that the regulation of the immune system, respiratory system, cardiovascular system, and gut microbiome were involved in shaping the pathophysiological pathways promoting depression and suicidal risks in PTSD patients. Our results showed that abnormal lab tests combined with medication use and diagnosis could facilitate predicting SRE risk. Moreover, this may imply beneficial effects for suicide prevention by treating comorbidities associated with these biomarkers

Foley Music: Learning to Generate Music from Videos

In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We then formulate music generation from videos as a motion-to-MIDI translation problem. We present a Graph−Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements. The MIDI event can then be converted to realistic music using an off-the-shelf music synthesizer tool. We demonstrate the effectiveness of our models on videos containing a variety of music performances. Experimental results show that our model outperforms several existing systems in generating music that is pleasant to listen to. More importantly, the MIDI representations are fully interpretable and transparent, thus enabling us to perform music editing flexibly. We encourage the readers to watch the supplementary video with audio turned on to experience the results.ONR MURI (N00014-16-1-2007

An Emulation of Randomized Trials of Administrating Antipsychotics in PTSD Patients for Outcomes of Suicide-Related Events

Post-traumatic stress disorder (PTSD) is a prevalent mental disorder marked by psychological and behavioral changes. Currently, there is no consensus of preferred antipsychotics to be used for the treatment of PTSD. We aim to discover whether certain antipsychotics have decreased suicide risk in the PTSD population, as these patients may be at higher risk. A total of 38,807 patients were identified with a diagnosis of PTSD through the ICD9 or ICD10 codes from January 2004 to October 2019. An emulation of randomized clinical trials was conducted to compare the outcomes of suicide-related events (SREs) among PTSD patients who ever used one of eight individual antipsychotics after the diagnosis of PTSD. Exclusion criteria included patients with a history of SREs and a previous history of antipsychotic use within one year before enrollment. Eligible individuals were assigned to a treatment group according to the antipsychotic initiated and followed until stopping current treatment, switching to another same class of drugs, death, or loss to follow up. The primary outcome was to identify the frequency of SREs associated with each antipsychotic. SREs were defined as ideation, attempts, and death by suicide. Pooled logistic regression methods with the Firth option were conducted to compare two drugs for their outcomes using SAS version 9.4 (SAS Institute, Cary, NC, USA). The results were adjusted for baseline characteristics and post-baseline, time-varying confounders. A total of 5294 patients were eligible for enrollment with an average follow up of 7.86 months. A total of 157 SREs were recorded throughout this study. Lurasidone showed a statistically significant decrease in SREs when compared head to head to almost all the other antipsychotics: aripiprazole, haloperidol, olanzapine, quetiapine, risperidone, and ziprasidone (p p value < 0.0004). In addition, olanzapine was associated with higher SREs than quetiapine and risperidone, and ziprasidone was associated with higher SREs than risperidone. The results of this study suggest that certain antipsychotics may put individuals within the PTSD population at an increased risk of SREs, and that careful consideration may need to be taken when prescribed

An Emulation of Randomized Trials of Administrating Antipsychotics in PTSD Patients for Outcomes of Suicide-Related Events

Post-traumatic stress disorder (PTSD) is a prevalent mental disorder marked by psychological and behavioral changes. Currently, there is no consensus of preferred antipsychotics to be used for the treatment of PTSD. We aim to discover whether certain antipsychotics have decreased suicide risk in the PTSD population, as these patients may be at higher risk. A total of 38,807 patients were identified with a diagnosis of PTSD through the ICD9 or ICD10 codes from January 2004 to October 2019. An emulation of randomized clinical trials was conducted to compare the outcomes of suicide-related events (SREs) among PTSD patients who ever used one of eight individual antipsychotics after the diagnosis of PTSD. Exclusion criteria included patients with a history of SREs and a previous history of antipsychotic use within one year before enrollment. Eligible individuals were assigned to a treatment group according to the antipsychotic initiated and followed until stopping current treatment, switching to another same class of drugs, death, or loss to follow up. The primary outcome was to identify the frequency of SREs associated with each antipsychotic. SREs were defined as ideation, attempts, and death by suicide. Pooled logistic regression methods with the Firth option were conducted to compare two drugs for their outcomes using SAS version 9.4 (SAS Institute, Cary, NC, USA). The results were adjusted for baseline characteristics and post-baseline, time-varying confounders. A total of 5294 patients were eligible for enrollment with an average follow up of 7.86 months. A total of 157 SREs were recorded throughout this study. Lurasidone showed a statistically significant decrease in SREs when compared head to head to almost all the other antipsychotics: aripiprazole, haloperidol, olanzapine, quetiapine, risperidone, and ziprasidone (p &lt; 0.0001 and false discovery rate-adjusted p value &lt; 0.0004). In addition, olanzapine was associated with higher SREs than quetiapine and risperidone, and ziprasidone was associated with higher SREs than risperidone. The results of this study suggest that certain antipsychotics may put individuals within the PTSD population at an increased risk of SREs, and that careful consideration may need to be taken when prescribed