Foley Music: Learning to Generate Music from Videos

In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We then formulate music generation from videos as a motion-to-MIDI translation problem. We present a Graph$-$Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements. The MIDI event can then be converted to realistic music using an off-the-shelf music synthesizer tool. We demonstrate the effectiveness of our models on videos containing a variety of music performances. Experimental results show that our model outperforms several existing systems in generating music that is pleasant to listen to. More importantly, the MIDI representations are fully interpretable and transparent, thus enabling us to perform music editing flexibly. We encourage the readers to watch the demo video with audio turned on to experience the results.Comment: ECCV 2020. Project page: http://foley-music.csail.mit.ed

Eyes wide open: perceived exploitation and its consequences

Drawing on the array of literature on exploitation from several social science disciplines, we propose a new way of seeing employer-employee relationships by introducing the concept of perceived exploitative employee-organization relationships, distinguish it from related concepts, and conduct five studies to develop a scale and test our theoretical model of the effects of such employee perceptions. Contributing to the Employee-Organization Relationships and workplace emotions literatures, perceived exploitation is defined as employees’ perceptions that they have been purposefully taken advantage of in their relationship with the organization, to the benefit of the organization itself. We propose and find that such perceptions are associated with both outward-focused emotions of anger and hostility toward the organization and inward-focused ones of shame and guilt at remaining in an exploitative job. In two studies including construction workers and a time-lagged study of medical residents, we find that the emotions of anger and hostility partially mediate the effects of perceived exploitation on employee engagement, revenge against the organization, organizational commitment, and turnover intentions, whereas the emotions of shame and guilt partially mediate the effects of perceived exploitation on employee burnout, silence, and psychological withdrawal

BRCA mutational status shapes the stromal microenvironment of pancreatic cancer linking clusterin expression in cancer associated fibroblasts with HSF1 signaling

Tumors initiate by mutations in cancer cells, and progress through interactions of the cancer cells with non-malignant cells of the tumor microenvironment. Major players in the tumor microenvironment are cancer-associated fibroblasts (CAFs), which support tumor malignancy, and comprise up to 90% of the tumor mass in pancreatic cancer. CAFs are transcriptionally rewired by cancer cells. Whether this rewiring is differentially affected by different mutations in cancer cells is largely unknown. Here we address this question by dissecting the stromal landscape of BRCA-mutated and BRCA Wild-type pancreatic ductal adenocarcinoma. We comprehensively analyze pancreatic cancer samples from 42 patients, revealing different CAF subtype compositions in germline BRCA-mutated vs. BRCA Wild-type tumors. In particular, we detect an increase in a subset of immune-regulatory clusterin-positive CAFs in BRCA-mutated tumors. Using cancer organoids and mouse models we show that this process is mediated through activation of heat-shock factor 1, the transcriptional regulator of clusterin. Our findings unravel a dimension of stromal heterogeneity influenced by germline mutations in cancer cells, with direct implications for clinical research

The spinal muscular atrophy with pontocerebellar hypoplasia gene VRK1 regulates neuronal migration through an amyloid-β precursor protein-dependent mechanism

Spinal muscular atrophy with pontocerebellar hypoplasia (SMA-PCH) is an infantile SMA variant with additional manifestations, particularly severe microcephaly. We previously identified a nonsense mutation in Vaccinia-related kinase 1 (VRK1), R358X, as a cause of SMA-PCH. VRK1-R358X is a rare founder mutation in Ashkenazi Jews, and additional mutations in patients of different origins have recently been identified.VRK1is a nuclear serine/threonine protein kinaseknownto play multiple roles in cellular proliferation, cell cycle regulation, and carcinogenesis. However, VRK1 was not known to have neuronal functions before its identification as a gene mutated in SMA-PCH. Here we show that VRK1-R358X homozygosity results in lack of VRK1 protein, and demonstrate a role for VRK1 in neuronal migration and neuronal stem cell proliferation. Using shRNA in utero electroporation in mice, we show that Vrk1 knockdown significantly impairs cortical neuronal migration, and affects the cell cycle of neuronal progenitors. Expression of wild-type human VRK1 rescues both proliferation and migration phenotypes. However, kinase-dead human VRK1 rescues only the migration impairment, suggesting the role of VRK1 in neuronal migration is partly noncatalytic. Furthermore, we found that VRK1 deficiency in human and mouse leads to downregulation of amyloid-β precursor protein (APP), a known neuronal migration gene. APP overexpression rescues the phenotype caused by Vrk1 knockdown, suggesting that VRK1 affects neuronal migration through an APP-dependent mechanism.This study was supported by grants from the Israel Science Foundation (Grant 702/13 to P.R. and E.L.-L., and Grant 47/10 and 322/13 to O.R. and T.S.); by the Minerva Foundation with funding from the Federal German Ministry for Education and Research, the Jérôme Lejeune Foundation, the Benoziyo Center for Neurological diseases, the Kekst Family Center for Medical Genetics, and the David and Fela Shapell Family Center for Genetic Disorders Research (to O.R.); by the JAE (Programa Junta para la Amplición de Estudios) Consejo Superior de Investigaciones Científicas Fondo Social Europeo fellowship (to L.C.); by the Ministerio de Educación, Ciencia e Innovación (SAF2010-14935 and SAF2013-44810R, to P.A.L.); and by Kutxa-Fundación Inbiomed (to P.A.L.).Peer Reviewe

VRK1 regulates Cajal body dynamics and protects coilin from proteasomal degradation in cell cycle

Cajal bodies (CBs) are nuclear organelles associated with ribonucleoprotein functions and RNA maturation. CBs are assembled on coilin, its main scaffold protein, in a cell cycle dependent manner. The Ser-Thr VRK1 (vaccinia-related kinase 1) kinase, whose activity is also cell cycle regulated, interacts with and phosphorylates coilin regulating assembly of CBs. Coilin phosphorylation is not necessary for its interaction with VRK1, but it occurs in mitosis and regulates coilin stability. Knockdown of VRK1 or VRK1 inactivation by serum deprivation causes a loss of coilin phosphorylation in Ser184 and of CBs formation, which are rescued with an active VRK1, but not by kinase-dead VRK1. The phosphorylation of coilin in Ser184 occurs during mitosis before assembly of CBs. Loss of coilin phosphorylation results in disintegration of CBs, and of coilin degradation that is prevented by proteasome inhibitors. After depletion of VRK1, coilin is ubiquitinated in nuclei, which is partly mediated by mdm2, but its proteasomal degradation occurs in cytosol and is prevented by blocking its nuclear export. We conclude that VRK1 is a novel regulator of CBs dynamics and stability in cell cycle by protecting coilin from ubiquitination and degradation in the proteasome, and propose a model of CB dynamics.L. C. and M. S-G. were funded by JAE-CSIC-Fondo Social Europeo fellowships. This work was supported by grants from Ministerio de Ciencia e Innovación (SAF2010-14935), Ministerio de Economía y Competitividad (SAF2013-44810R, SAF2014-57791-REDC), and Junta de Castilla y León-Consejería de Educación (CSI002U14) to P.A.L;and grant (702/13) from the Israel Science Foundation to P.R. and E.L.L.Peer reviewe

Self-supervised learning of audio-visual objects from video

Our objective is to transform a video into a set of discrete audio-visual objects using self-supervised learning. To this end, we introduce a model that uses attention to localize and group sound sources, and optical flow to aggregate information over time. We demonstrate the effectiveness of the audio-visual object embeddings that our model learns by using them for four downstream speech-oriented tasks: (a) multi-speaker sound source separation, (b) localizing and tracking speakers, (c) correcting misaligned audio-visual data, and (d) active speaker detection. Using our representation, these tasks can be solved entirely by training on unlabeled video, without the aid of object detectors. We also demonstrate the generality of our method by applying it to non-human speakers, including cartoons and puppets. Our model significantly outperforms other self-supervised approaches, and obtains performance competitive with methods that use supervised face detection

Foley Music: Learning to Generate Music from Videos

In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We then formulate music generation from videos as a motion-to-MIDI translation problem. We present a Graph−Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements. The MIDI event can then be converted to realistic music using an off-the-shelf music synthesizer tool. We demonstrate the effectiveness of our models on videos containing a variety of music performances. Experimental results show that our model outperforms several existing systems in generating music that is pleasant to listen to. More importantly, the MIDI representations are fully interpretable and transparent, thus enabling us to perform music editing flexibly. We encourage the readers to watch the supplementary video with audio turned on to experience the results.ONR MURI (N00014-16-1-2007