RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models

Researchers have successfully applied large language models (LLMs) such as ChatGPT to reranking in an information retrieval context, but to date, such work has mostly been built on proprietary models hidden behind opaque API endpoints. This approach yields experimental results that are not reproducible and non-deterministic, threatening the veracity of outcomes that build on such shaky foundations. To address this significant shortcoming, we present RankVicuna, the first fully open-source LLM capable of performing high-quality listwise reranking in a zero-shot setting. Experimental results on the TREC 2019 and 2020 Deep Learning Tracks show that we can achieve effectiveness comparable to zero-shot reranking with GPT-3.5 with a much smaller 7B parameter model, although our effectiveness remains slightly behind reranking with GPT-4. We hope our work provides the foundation for future research on reranking with modern LLMs. All the code necessary to reproduce our results is available at https://github.com/castorini/rank_llm

Comparison of Foveated Downsampling Techniques in Image Recognition

Foveation is an important part of human vision, and a number of deep networks have also used foveation. However, there have been few systematic comparisons between foveating and non-foveating deep networks, and between different variable-resolution downsampling methods. Here we define several such methods, and compare their performance on ImageNet recognition with a custom Densenet network. The best variable-resolution method slightly outperforms uniform downsampling. Thus in our experiments, foveation does not substantially help or hinder object recognition in deep networks

Vector Search with OpenAI Embeddings: Lucene Is All You Need

We provide a reproducible, end-to-end demonstration of vector search with OpenAI embeddings using Lucene on the popular MS MARCO passage ranking test collection. The main goal of our work is to challenge the prevailing narrative that a dedicated vector store is necessary to take advantage of recent advances in deep neural networks as applied to search. Quite the contrary, we show that hierarchical navigable small-world network (HNSW) indexes in Lucene are adequate to provide vector search capabilities in a standard bi-encoder architecture. This suggests that, from a simple cost-benefit analysis, there does not appear to be a compelling reason to introduce a dedicated vector store into a modern "AI stack" for search, since such applications have already received substantial investments in existing, widely deployed infrastructure

LEARN: A multi-centre, cross-sectional evaluation of Urology teaching in UK medical schools

OBJECTIVE: To evaluate the status of UK undergraduate urology teaching against the British Association of Urological Surgeons (BAUS) Undergraduate Syllabus for Urology. Secondary objectives included evaluating the type and quantity of teaching provided, the reported performance rate of General Medical Council (GMC)-mandated urological procedures, and the proportion of undergraduates considering urology as a career. MATERIALS AND METHODS: LEARN was a national multicentre cross-sectional study. Year 2 to Year 5 medical students and FY1 doctors were invited to complete a survey between 3rd October and 20th December 2020, retrospectively assessing the urology teaching received to date. Results are reported according to the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). RESULTS: 7,063/8,346 (84.6%) responses from all 39 UK medical schools were included; 1,127/7,063 (16.0%) were from Foundation Year (FY) 1 doctors, who reported that the most frequently taught topics in undergraduate training were on urinary tract infection (96.5%), acute kidney injury (95.9%) and haematuria (94.4%). The most infrequently taught topics were male urinary incontinence (59.4%), male infertility (52.4%) and erectile dysfunction (43.8%). Male and female catheterisation on patients as undergraduates was performed by 92.1% and 73.0% of FY1 doctors respectively, and 16.9% had considered a career in urology. Theory based teaching was mainly prevalent in the early years of medical school, with clinical skills teaching, and clinical placements in the later years of medical school. 20.1% of FY1 doctors reported no undergraduate clinical attachment in urology. CONCLUSION: LEARN is the largest ever evaluation of undergraduate urology teaching. In the UK, teaching seemed satisfactory as evaluated by the BAUS undergraduate syllabus. However, many students report having no clinical attachments in Urology and some newly qualified doctors report never having inserted a catheter, which is a GMC mandated requirement. We recommend a greater emphasis on undergraduate clinical exposure to urology and stricter adherence to GMC mandated procedures