Fine-graind Image Classification via Combining Vision and Language

Fine-grained image classification is a challenging task due to the large intra-class variance and small inter-class variance, aiming at recognizing hundreds of sub-categories belonging to the same basic-level category. Most existing fine-grained image classification methods generally learn part detection models to obtain the semantic parts for better classification accuracy. Despite achieving promising results, these methods mainly have two limitations: (1) not all the parts which obtained through the part detection models are beneficial and indispensable for classification, and (2) fine-grained image classification requires more detailed visual descriptions which could not be provided by the part locations or attribute annotations. For addressing the above two limitations, this paper proposes the two-stream model combining vision and language (CVL) for learning latent semantic representations. The vision stream learns deep representations from the original visual information via deep convolutional neural network. The language stream utilizes the natural language descriptions which could point out the discriminative parts or characteristics for each image, and provides a flexible and compact way of encoding the salient visual aspects for distinguishing sub-categories. Since the two streams are complementary, combining the two streams can further achieves better classification accuracy. Comparing with 12 state-of-the-art methods on the widely used CUB-200-2011 dataset for fine-grained image classification, the experimental results demonstrate our CVL approach achieves the best performance.Comment: 9 pages, to appear in CVPR 201

SILICA NANOPOROUS CONFINEMENT EFFECTS ON IONIC LIQUID PROPERTIES FOR BETTER DESIGN OF SMALL MOLECULE SEPARATION, ELECTROCHEMICAL DEVICES AND DRUG DELIVERY

Silica nanoconfinement provides a high level of control of ionic liquids (ILs) in localizing catalysts, creating distinct environment for tuning reactivity and controlling the partition of solvents, reactants and products. Silica thin films with two different pore sizes (2.5 nm and 8 nm) were synthesized to study the effect of nanopore confinement on ionic liquids 1-butyl-3-methylimidazolium hexafluorophosphate ([BMIM][PF6]), and 1-butyl-3-methylimidazolium chloride ([BMIM][Cl]). Silica thin films with accessible 8 nm pore diameters were synthesized using evaporation-induced self-assembly (EISA) with Pluronic P123 as templating surfactant on a chemically neutral modified substrate. The silica films with similar orthogonal aligned mesostructured but smaller pore size (2.5 nm) were produced through cetyltrimethylammonium bromide (CTAB) templated EISA. The perpendicularly oriented channels were achieved by doping the silica matrix with small amount of titania, which destabilized the nanoporous structure during calcination so that the isolated micelles connect with each other when the films go through thermal contraction during calcination. In situ grazing-incidence small angle X-ray scattering (GISAXS) was performed and it revealed this structure transformation. To broaden the application of this CTAB templated film, a sugar surfactant was added to bind with titania precursor and disperse titania on the pore surface instead of through out the entire matrix. The absorption of water by ILs is among the most concerning properties when they are utilized in catalysis systems for example, the dehydration of glucose to 5-(hydroxymethyl)furfural (HMF) using [BMIM][Cl] as solvent. Thus, [BMIM][Cl] confined in the 8-nm-pore-diameter silica thin films was investigated and compared to that of bulk ionic liquid. Transmission Fourier transform infrared spectra (FTIR) were collected in situ at room temperature while the relative humidity (RH) of the environment was changed. Pore confinement effects were interpreted from the C-H stretching bands shift and OH stretching band growth. Deconvolution of OH stretching bands shows that weakly coordinated water is promoted in confined [BMIM][Cl], which may affect mechanisms of solubilization and catalysis in confined ILs. To understand the effect of pore confinement on transport, the two silica thin films were modified with physically absorbed [BMIM][PF6], chemically tethered 3-methyl-1-[3-(trimethoxysilyl) propyl]-1-imidazolium group ([TMS-MIM]+) and with the presence of both. Electrochemical impedance spectroscopy (EIS) was performed to investigate the permeability of hydrophilic and hydrophobic redox groups through the thin films with different treatments and different pore sizes. Both films with chemically tethered IL possess much higher resistance to hydrophilic molecules than hydrophobic molecules with 14-fold lower permeability through 8-nm tethered films and 30-fold lower for 2.5-nm tethered films. This work successfully produced highly selective silica thin films for selective separation of hydrophilic and hydrophobic species. Crystallization of ILs is another attractive property of ILs to study but lack of direct characterization results. This dissertation work introduced in situ probing using grazing-incidence wide-angle X-ray scattering (GIWAXS) to characterize the crystallization behavior of [BMIM][PF6] and [BMIM][Cl] under confinement of the two silica thin films with and without ILs tethering as described above. While certain crystallization behavior can be predicted by the classic Gibbs-Thomson theory, confined ILs in most cases studied do not follow the predictions of Gibbs-Thomson theory. One extreme case is when [BMIM][PF6] were confined in [TMS-MIM] tethered 2.5-nm porous silica thin films, the composite did not melt at room temperature while [BMIM][PF6] in bulk melts around -11 °C. This work successfully demonstrates how different confining condition changes the crystallization behavior of the two ILs through the novel in situ GIWAXS characterization, which benefits further studies for drug delivery and battery research

Revisiting Event Argument Extraction: Can EAE Models Learn Better When Being Aware of Event Co-occurrences?

Event co-occurrences have been proved effective for event extraction (EE) in previous studies, but have not been considered for event argument extraction (EAE) recently. In this paper, we try to fill this gap between EE research and EAE research, by highlighting the question that ``Can EAE models learn better when being aware of event co-occurrences?''. To answer this question, we reformulate EAE as a problem of table generation and extend a SOTA prompt-based EAE model into a non-autoregressive generation framework, called TabEAE, which is able to extract the arguments of multiple events in parallel. Under this framework, we experiment with 3 different training-inference schemes on 4 datasets (ACE05, RAMS, WikiEvents and MLEE) and discover that via training the model to extract all events in parallel, it can better distinguish the semantic boundary of each event and its ability to extract single event gets substantially improved. Experimental results show that our method achieves new state-of-the-art performance on the 4 datasets. Our code is avilable at https://github.com/Stardust-hyx/TabEAE.Comment: Accepted to ACL 2023 main conferenc