Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text

Real world multimedia data is often composed of multiple modalities such as an image or a video with associated text (e.g. captions, user comments, etc.) and metadata. Such multimodal data packages are prone to manipulations, where a subset of these modalities can be altered to misrepresent or repurpose data packages, with possible malicious intent. It is, therefore, important to develop methods to assess or verify the integrity of these multimedia packages. Using computer vision and natural language processing methods to directly compare the image (or video) and the associated caption to verify the integrity of a media package is only possible for a limited set of objects and scenes. In this paper, we present a novel deep learning-based approach for assessing the semantic integrity of multimedia packages containing images and captions, using a reference set of multimedia packages. We construct a joint embedding of images and captions with deep multimodal representation learning on the reference dataset in a framework that also provides image-caption consistency scores (ICCSs). The integrity of query media packages is assessed as the inlierness of the query ICCSs with respect to the reference dataset. We present the MultimodAl Information Manipulation dataset (MAIM), a new dataset of media packages from Flickr, which we make available to the research community. We use both the newly created dataset as well as Flickr30K and MS COCO datasets to quantitatively evaluate our proposed approach. The reference dataset does not contain unmanipulated versions of tampered query packages. Our method is able to achieve F1 scores of 0.75, 0.89 and 0.94 on MAIM, Flickr30K and MS COCO, respectively, for detecting semantically incoherent media packages.Comment: *Ayush Jaiswal and Ekraam Sabir contributed equally to the work in this pape

Zero-Shot Learning by Convex Combination of Semantic Embeddings

Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the embedding space is trained jointly with the image transformation. In other cases the semantic embedding space is established by an independent natural language processing task, and then the image transformation into that space is learned in a second stage. Proponents of these image embedding systems have stressed their advantages over the traditional \nway{} classification framing of image understanding, particularly in terms of the promise for zero-shot learning -- the ability to correctly annotate images of previously unseen object categories. In this paper, we propose a simple method for constructing an image embedding system from any existing \nway{} image classifier and a semantic word embedding model, which contains the \n class labels in its vocabulary. Our method maps images into the semantic embedding space via convex combination of the class label embedding vectors, and requires no additional training. We show that this simple and direct method confers many of the advantages associated with more complex image embedding schemes, and indeed outperforms state of the art methods on the ImageNet zero-shot learning task

Learning SO(3) Equivariant Representations with Spherical CNNs

We address the problem of 3D rotation equivariance in convolutional neural networks. 3D rotations have been a challenging nuisance in 3D classification tasks requiring higher capacity and extended data augmentation in order to tackle it. We model 3D data with multi-valued spherical functions and we propose a novel spherical convolutional network that implements exact convolutions on the sphere by realizing them in the spherical harmonic domain. Resulting filters have local symmetry and are localized by enforcing smooth spectra. We apply a novel pooling on the spectral domain and our operations are independent of the underlying spherical resolution throughout the network. We show that networks with much lower capacity and without requiring data augmentation can exhibit performance comparable to the state of the art in standard retrieval and classification benchmarks.Comment: Camera-ready. Accepted to ECCV'18 as oral presentatio

Directed Trans-Differentiation of Thymus Cells into Parathyroid-Like Cells Without Genetic Manipulation

Replacement of a diseased organ with an autologously derived tissue is an ideal therapy for some medical problems. However, it is difficult to recreate many adult human tissues in vitro due to the functionally necessary architecture of most organs and the lack of understanding of methods to direct the development of the organ of interest. The parathyroid gland is ideal for in vitro organ development because this gland is relatively simple, is transplantable, and is commonly affected by a surgical complication rather than an autoimmune disease. We have investigated thymus as a source of autologous endoderm and parathyroid-like precursor cells. Human thymus cells were treated with a differentiation protocol we developed with human embryonic stem cells (The Bingham Protocol) that utilizes timed exposures to Activin A and soluble Sonic hedgehog (Shh). We incrementally changed the protocol to optimize the differentiation of the thymus cells into parathyroid-like cells. The final protocol used 50-ng/mL Activin A and 100-ng/mL Shh over 13 weeks. The differentiated cells expressed the parathyroid markers parathyroid hormone (PTH), calcium sensing receptor, chemokine receptor type-4 (CXCR4), and chorian-specific transcription factor (GCM2) as measured by reverse transcription-polymerase chain reaction and PTH enzyme-linked immunosorbent assay. Cultured thymus cells without Activin A or Shh exposure did not secrete PTH nor express similar markers. The differentiated cells released PTH, which was suppressed in response to increased calcium concentration. The chemically differentiated cells did not form tumors in immune-compromised mice. Our protocol recreated cells with markers of parathyroid tissue that responded as parathyroid cells to physiologic stimuli. This approach is a further step toward a strategy to restore parathyroid function using autologous cells that were directed to differentiate by nongenetic in vitro manipulation.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/90464/1/ten-2Etec-2E2011-2E0170.pd

Detecting Sarcasm in Multimodal Social Platforms

Sarcasm is a peculiar form of sentiment expression, where the surface sentiment differs from the implied sentiment. The detection of sarcasm in social media platforms has been applied in the past mainly to textual utterances where lexical indicators (such as interjections and intensifiers), linguistic markers, and contextual information (such as user profiles, or past conversations) were used to detect the sarcastic tone. However, modern social media platforms allow to create multimodal messages where audiovisual content is integrated with the text, making the analysis of a mode in isolation partial. In our work, we first study the relationship between the textual and visual aspects in multimodal posts from three major social media platforms, i.e., Instagram, Tumblr and Twitter, and we run a crowdsourcing task to quantify the extent to which images are perceived as necessary by human annotators. Moreover, we propose two different computational frameworks to detect sarcasm that integrate the textual and visual modalities. The first approach exploits visual semantics trained on an external dataset, and concatenates the semantics features with state-of-the-art textual features. The second method adapts a visual neural network initialized with parameters trained on ImageNet to multimodal sarcastic posts. Results show the positive effect of combining modalities for the detection of sarcasm across platforms and methods.Comment: 10 pages, 3 figures, final version published in the Proceedings of ACM Multimedia 201

Zero-Shot Hashing via Transferring Supervised Knowledge

Hashing has shown its efficiency and effectiveness in facilitating large-scale multimedia applications. Supervised knowledge e.g. semantic labels or pair-wise relationship) associated to data is capable of significantly improving the quality of hash codes and hash functions. However, confronted with the rapid growth of newly-emerging concepts and multimedia data on the Web, existing supervised hashing approaches may easily suffer from the scarcity and validity of supervised information due to the expensive cost of manual labelling. In this paper, we propose a novel hashing scheme, termed \emph{zero-shot hashing} (ZSH), which compresses images of "unseen" categories to binary codes with hash functions learned from limited training data of "seen" categories. Specifically, we project independent data labels i.e. 0/1-form label vectors) into semantic embedding space, where semantic relationships among all the labels can be precisely characterized and thus seen supervised knowledge can be transferred to unseen classes. Moreover, in order to cope with the semantic shift problem, we rotate the embedded space to more suitably align the embedded semantics with the low-level visual feature space, thereby alleviating the influence of semantic gap. In the meantime, to exert positive effects on learning high-quality hash functions, we further propose to preserve local structural property and discrete nature in binary codes. Besides, we develop an efficient alternating algorithm to solve the ZSH model. Extensive experiments conducted on various real-life datasets show the superior zero-shot image retrieval performance of ZSH as compared to several state-of-the-art hashing methods.Comment: 11 page

Shifts in perception of size after adaption to gratings.

Thesis (M.S.

In wildness is the preservation of the world

Passage of the Wilderness Act of 1964 and establishment of the National Wilderness Preservation System represent an historic achievement, an assertion of idealism in a materialist-powered society. The four federal land-management agencies, however, have failed to meet their legislative mandate. Thus Earth Day 1990 provides a singular opportunity to address the unfinished agenda with consideration of a new agency to be called the United States Wilderness Service

D.C. Background on Predator Control Legislation

The tragic fiasco of federal predator control as we have known it is finished. The American people will no longer tolerate it. In this age of environmental concern, the people will not allow their tax dollars to be diverted for such a destructive and wasteful war against living wild creatures for the exclusive benefit of the sheep industry. There is now no turning back to old ways. Indiscriminate trapping, shooting and poisoning have reduced some of the rarest, most beautiful and superbly adapted species of our wildlife heritage to the brink of extinction, although they consitutue a resource that could be enjoyed by all and harvested by sportsmen under sound management principles. The war on predators has been waged with little scientific knowledge of their beneficial role in the biotic community, and without moral or ethical consideration for man\u27s responsibility in preserving natural life as an integral part of the environment. As I wrote in the January, 1971, issue of Field and Stream, the Division of Wildlife Services, an agency of the Interior Department, has had one prime goal at the root of its existence: to kill wildlife. It has for years gotten away with murder -- the murder of wolves, mountain lions, coyotes, bobcats, foxes, badgers -- as well as anything else that might be handy

Career success: the role of teenage career aspirations, ambition value and gender in predicting adult social status and earnings

Links between family social background, teenage career aspirations, educational performance and adult social status attainment are well documented. Using a contextual developmental framework, this article extends previous research by examining the role of gender and teenage ambition value in shaping social status attainment and earnings in adulthood. Drawing on data from an 18-year British follow up study we tested a path model linking family background factors (such as family social status and parental aspirations) and individual agency factors in adolescence (in particular, career aspirations and ambition value) to social status attainment and earnings in adulthood. The findings suggest that ambition value is linked to adult earnings. That is, young people for whom it is important to get on in their job earn more money in adulthood than their less ambitious peers. The findings also confirm that teenage career aspirations are linked to adult social status attainment, and suggest that family background factors, teenage career aspirations and ambition value interact to influence social status attainment and earnings in adulthood. Gender differences are discusse