User-Aware Prefix-Tuning is a Good Learner for Personalized Image Captioning

Image captioning bridges the gap between vision and language by automatically generating natural language descriptions for images. Traditional image captioning methods often overlook the preferences and characteristics of users. Personalized image captioning solves this problem by incorporating user prior knowledge into the model, such as writing styles and preferred vocabularies. Most existing methods emphasize the user context fusion process by memory networks or transformers. However, these methods ignore the distinct domains of each dataset. Therefore, they need to update the entire caption model parameters when meeting new samples, which is time-consuming and calculation-intensive. To address this challenge, we propose a novel personalized image captioning framework that leverages user context to consider personality factors. Additionally, our framework utilizes the prefix-tuning paradigm to extract knowledge from a frozen large language model, reducing the gap between different language domains. Specifically, we employ CLIP to extract the visual features of an image and align the semantic space using a query-guided mapping network. By incorporating the transformer layer, we merge the visual features with the user's contextual prior knowledge to generate informative prefixes. Moreover, we employ GPT-2 as the frozen large language model. With a small number of parameters to be trained, our model performs efficiently and effectively. Our model outperforms existing baseline models on Instagram and YFCC100M datasets across five evaluation metrics, demonstrating its superiority, including twofold improvements in metrics such as BLEU-4 and CIDEr

Observing Exoplanets with High-Dispersion Coronagraphy. II. Demonstration of an Active Single-Mode Fiber Injection Unit

High-dispersion coronagraphy (HDC) optimally combines high contrast imaging techniques such as adaptive optics/wavefront control plus coronagraphy to high spectral resolution spectroscopy. HDC is a critical pathway towards fully characterizing exoplanet atmospheres across a broad range of masses from giant gaseous planets down to Earth-like planets. In addition to determining the molecular composition of exoplanet atmospheres, HDC also enables Doppler mapping of atmosphere inhomogeneities (temperature, clouds, wind), as well as precise measurements of exoplanet rotational velocities. Here, we demonstrate an innovative concept for injecting the directly-imaged planet light into a single-mode fiber, linking a high-contrast adaptively-corrected coronagraph to a high-resolution spectrograph (diffraction-limited or not). Our laboratory demonstration includes three key milestones: close-to-theoretical injection efficiency, accurate pointing and tracking, on-fiber coherent modulation and speckle nulling of spurious starlight signal coupling into the fiber. Using the extreme modal selectivity of single-mode fibers, we also demonstrated speckle suppression gains that outperform conventional image-based speckle nulling by at least two orders of magnitude.Comment: 10 pages, 7 figures, accepted by Ap

Lessons for WFIRST CGI from ground-based high-contrast systems

The Coronagraph Instrument (CGI) for NASA's Wide Field Infrared Survey Telescope (WFIRST) will constitute a dramatic step forward for high-contrast imaging, integral field spectroscopy, and polarimetry of exoplanets and circumstellar disks, aiming to improve upon the sensitivity of current ground-based direct imaging facilities by 2-3 orders of magnitude. Furthermore, CGI will serve as a pathfinder for future exo-Earth imaging and characterization missions by demonstrating wavefront control, coronagraphy, and spectral retrieval in a new contrast regime, and by validating instrument and telescope models at unprecedented levels of precision. To achieve this jump in performance, it is critical to draw on the experience of ground-based high-contrast facilities. We discuss several areas of relevant commonalities, including: wavefront control, post-processing of integral field unit data, and calibration and observing strategies