Learning Robot Activities from First-Person Human Videos Using Convolutional Future Regression

We design a new approach that allows robot learning of new activities from unlabeled human example videos. Given videos of humans executing the same activity from a human's viewpoint (i.e., first-person videos), our objective is to make the robot learn the temporal structure of the activity as its future regression network, and learn to transfer such model for its own motor execution. We present a new deep learning model: We extend the state-of-the-art convolutional object detection network for the representation/estimation of human hands in training videos, and newly introduce the concept of using a fully convolutional network to regress (i.e., predict) the intermediate scene representation corresponding to the future frame (e.g., 1-2 seconds later). Combining these allows direct prediction of future locations of human hands and objects, which enables the robot to infer the motor control plan using our manipulation network. We experimentally confirm that our approach makes learning of robot activities from unlabeled human interaction videos possible, and demonstrate that our robot is able to execute the learned collaborative activities in real-time directly based on its camera input

Forecasting Hands and Objects in Future Frames

This paper presents an approach to forecast future presence and location of human hands and objects. Given an image frame, the goal is to predict what objects will appear in the future frame (e.g., 5 seconds later) and where they will be located at, even when they are not visible in the current frame. The key idea is that (1) an intermediate representation of a convolutional object recognition model abstracts scene information in its frame and that (2) we can predict (i.e., regress) such representations corresponding to the future frames based on that of the current frame. We design a new two-stream convolutional neural network (CNN) architecture for videos by extending the state-of-the-art convolutional object detection network, and present a new fully convolutional regression network for predicting future scene representations. Our experiments confirm that combining the regressed future representation with our detection network allows reliable estimation of future hands and objects in videos. We obtain much higher accuracy compared to the state-of-the-art future object presence forecast method on a public dataset

Identifying First-person Camera Wearers in Third-person Videos

We consider scenarios in which we wish to perform joint scene understanding, object tracking, activity recognition, and other tasks in environments in which multiple people are wearing body-worn cameras while a third-person static camera also captures the scene. To do this, we need to establish person-level correspondences across first- and third-person videos, which is challenging because the camera wearer is not visible from his/her own egocentric video, preventing the use of direct feature matching. In this paper, we propose a new semi-Siamese Convolutional Neural Network architecture to address this novel challenge. We formulate the problem as learning a joint embedding space for first- and third-person videos that considers both spatial- and motion-domain cues. A new triplet loss function is designed to minimize the distance between correct first- and third-person matches while maximizing the distance between incorrect ones. This end-to-end approach performs significantly better than several baselines, in part by learning the first- and third-person features optimized for matching jointly with the distance measure itself

Characterization of macular lesions in punctate inner choroidopathy with spectral domain optical coherence tomography.

PURPOSE: Punctate inner choroidopathy (PIC) is an ocular inflammatory disease. Spectral domain optical coherence tomography (SD-OCT) allows detailed visualization of retinal and choroidal structures. We aimed to describe the retinal changes on SD-OCT associated with PIC lesions localized in the macula. METHODS: Retrospective case series: PIC lesions not associated with choroidal neovascularization (CNV) and captured by macular SD-OCT scans were identified and characterized. RESULTS: Twenty-seven PIC lesions from seven patients (eight eyes) were identified and classified into four categories according to disease activity and temporal changes. Among clinically inactive patients, two main patterns were noted on OCT: (1) retinal pigment epithelium (RPE) elevation with sub-RPE hyper-reflective signals and (2) localized disruption of outer retinal layers with choroid and Bruch\u27s membrane (BM) generally spared. Clinically active patients demonstrated lesions with intact BM with RPE elevation that fluctuated with disease activity and sub-RPE hyper-reflective signals. Photoreceptor-associated bands on SD-OCT (PRs) were not visible during active disease, but returned to normal visibility when lesions were clinically stable. Seven lesions in patients without clinically detected activity demonstrated alteration of RPE elevation. CONCLUSION: SD-OCT can provide detailed structural characteristics of PIC lesions. RPE elevation is noted in many lesions while BM and choroid are spared. Photoreceptor-associated bands on SD-OCT appear compressed during clinically active stages and are visible during stabilization. OCT may provide information on activity not detected clinically

Importance of proper diagnosis for management: multifocal choroiditis mimicking ocular histoplasmosis syndrome

PURPOSE: The study aims to evaluate a series of patients with initial diagnosis of ocular histoplasmosis syndrome (OHS) with progression and response to treatments consistent with multifocal choroiditis (MFC). METHODS: Retrospective review of nine patients referred for management of recurrent OHS lesions. Serology panel was conducted to rule out autoimmune and infectious causes. RESULTS: Clinical examination revealed multiple small, punched-out peripheral chorioretinal scars, and peripapillary atrophy. Histoplasma antigen/antibody was negative in all patients. Fluorescein angiography and optical coherence tomography confirmed active inflammation in five patients. Immunomodulatory therapy (IMT) was initiated to control active inflammation. While on IMT, visual acuity stabilized or improved in three patients with no recurrence of CNV or lesion activities over the follow-up period. CONCLUSIONS: MFC may initially masquerade as OHS. Clinical characteristics of recurrent MFC and absence of histoplasma titer may lead to consideration of IMT and other proper treatments for MFC

Strongly adhesive dry transfer technique for van der Waals heterostructure

That one can stack van der Waals materials with atomically sharp interfaces has provided a new material platform of constructing heterostructures. The technical challenge of mechanical stacking is picking up the exfoliated atomically thin materials after mechanical exfoliation without chemical and mechanical degradation. Chemically inert hexagonal boron nitride (hBN) has been widely used for encapsulating and picking up vdW materials. However, due to the relatively weak adhesion of hBN, assembling vdW heterostructures based on hBN has been limited. We report a new dry transfer technique. We used two vdW semiconductors (ZnPS3 and CrPS4) to pick up and encapsulate layers for vdW heterostructures, which otherwise are known to be hard to fabricate. By combining with optimized polycaprolactone (PCL) providing strong adhesion, we demonstrated various vertical heterostructure devices, including quasi-2D superconducting NbSe2 Josephson junctions with atomically clean interface. The versatility of the PCL-based vdW stacking method provides a new route for assembling complex 2D vdW materials without interfacial degradation.Comment: Accepted for publication in 2D Material