7 research outputs found
Overview of the TREC 2023 Product Product Search Track
This is the first year of the TREC Product search track. The focus this year
was the creation of a reusable collection and evaluation of the impact of the
use of metadata and multi-modal data on retrieval accuracy. This year we
leverage the new product search corpus, which includes contextual metadata. Our
analysis shows that in the product search domain, traditional retrieval systems
are highly effective and commonly outperform general-purpose pretrained
embedding models. Our analysis also evaluates the impact of using simplified
and metadata-enhanced collections, finding no clear trend in the impact of the
expanded collection. We also see some surprising outcomes; despite their
widespread adoption and competitive performance on other tasks, we find
single-stage dense retrieval runs can commonly be noncompetitive or generate
low-quality results both in the zero-shot and fine-tuned domain.Comment: 14 pages, 4 figures, 11 tables - TREC 202
Generic Intent Representation in Web Search
This paper presents GEneric iNtent Encoder (GEN Encoder) which learns a
distributed representation space for user intent in search. Leveraging large
scale user clicks from Bing search logs as weak supervision of user intent, GEN
Encoder learns to map queries with shared clicks into similar embeddings
end-to-end and then finetunes on multiple paraphrase tasks. Experimental
results on an intrinsic evaluation task - query intent similarity modeling -
demonstrate GEN Encoder's robust and significant advantages over previous
representation methods. Ablation studies reveal the crucial role of learning
from implicit user feedback in representing user intent and the contributions
of multi-task learning in representation generality. We also demonstrate that
GEN Encoder alleviates the sparsity of tail search traffic and cuts down half
of the unseen queries by using an efficient approximate nearest neighbor search
to effectively identify previous queries with the same search intent. Finally,
we demonstrate distances between GEN encodings reflect certain information
seeking behaviors in search sessions
Orca 2: Teaching Small Language Models How to Reason
Orca 1 learns from rich signals, such as explanation traces, allowing it to
outperform conventional instruction-tuned models on benchmarks like BigBench
Hard and AGIEval. In Orca 2, we continue exploring how improved training
signals can enhance smaller LMs' reasoning abilities. Research on training
small LMs has often relied on imitation learning to replicate the output of
more capable models. We contend that excessive emphasis on imitation may
restrict the potential of smaller models. We seek to teach small LMs to employ
different solution strategies for different tasks, potentially different from
the one used by the larger model. For example, while larger models might
provide a direct answer to a complex task, smaller models may not have the same
capacity. In Orca 2, we teach the model various reasoning techniques
(step-by-step, recall then generate, recall-reason-generate, direct answer,
etc.). More crucially, we aim to help the model learn to determine the most
effective solution strategy for each task. We evaluate Orca 2 using a
comprehensive set of 15 diverse benchmarks (corresponding to approximately 100
tasks and over 36,000 unique prompts). Orca 2 significantly surpasses models of
similar size and attains performance levels similar or better to those of
models 5-10x larger, as assessed on complex tasks that test advanced reasoning
abilities in zero-shot settings. make Orca 2 weights publicly available at
aka.ms/orca-lm to support research on the development, evaluation, and
alignment of smaller LMsComment: Added url to model weights fixed typo in Author nam
Contrastive Post-training Large Language Models on Data Curriculum
Alignment serves as an important step to steer large language models (LLMs)
towards human preferences. In this paper, we explore contrastive post-training
techniques for alignment by automatically constructing preference pairs from
multiple models of varying strengths (e.g., InstructGPT, ChatGPT and GPT-4). We
carefully compare the contrastive techniques of SLiC and DPO to SFT baselines
and find that DPO provides a step-function improvement even after continueing
SFT saturates. We also explore a data curriculum learning scheme for
contrastive post-training, which starts by learning from "easier" pairs and
transitioning to "harder" ones, which further improves alignment. Finally, we
scale up our experiments to train with more data and larger models like Orca.
Remarkably, contrastive post-training further improves the performance of Orca,
already a state-of-the-art instruction learning model tuned with GPT-4 outputs,
to exceed that of ChatGPT