Large Language Model (LLM) and Generative Pre-trained Transformer (GPT) technology has the ability to generate responses to questions provided in text format. While such technology produces text content with plausible answers to a given question, it is difficult to judge the trustworthiness of the generated answers. It has been shown that answers generated by such technology may not always be factually accurate. This disclosure describes automated techniques to improve the credibility and confidence of machine-generated answers by using results from a search engine. Per the techniques, search results corresponding to a user input query are used to generate an augmented query that is provided as input to an LLM. Such augmentation adds context and can improve the quality and reliability of answers generated by the LLM. Further, the techniques can also be utilized to add citations to the generated answer and to indicate a confidence level in the generated answer

Park, Hyun Jin

Ryu, Changwan

English

Technical Disclosure Common

Technical Disclosure Commons Defensive Publications Series May 2023 Query Augmentation Using Search Engine Results to Improve Answers Generated by Large Language Models Hyun Jin Park Changwan Ryu Follow this and additional works at: https://www.tdcommons.org/dpubs_series Recommended Citation Park, Hyun Jin and Ryu, Changwan, "Query Augmentation Using Search Engine Results to Improve Answers Generated by Large Language Models", Technical Disclosure Commons, (May 11, 2023) https://www.tdcommons.org/dpubs_series/5889 This work is licensed under a Creative Commons Attribution 4.0 License. This Article is brought to you for free and open access by Technical Disclosure Commons. It has been accepted for inclusion in Defensive Publications Series by an authorized administrator of Technical Disclosure Commons. Query Augmentation Using Search Engine Results to Improve Answers Generated by Large Language Models  ABSTRACT Large Language Model (LLM) and Generative Pre-trained Transformer (GPT) technology has the ability to generate responses to questions provided in text format. While such technology produces text content with plausible answers to a given question, it is difficult to judge the trustworthiness of the generated answers. It has been shown that answers generated by such technology may not always be factually accurate. This disclosure describes automated techniques to improve the credibility and confidence of machine-generated answers by using results from a search engine. Per the techniques, search results corresponding to a user input query are used to generate an augmented query that is provided as input to an LLM. Such augmentation adds context and can improve the quality and reliability of answers generated by the LLM. Further, the techniques can also be utilized to add citations to the generated answer and to indicate a confidence level in the generated answer. KEYWORDS● Generative AI ● Large Language Model (LLM) ● Generative Pre-trained Transformer (GPT) ● Query augmentation ● Query context ● Generated text ● Citation confidence ● Online search 2Park and Ryu: Query Augmentation Using Search Engine Results to Improve AnswersPublished by Technical Disclosure Commons, 2023BACKGROUND Large Language Model (LLM) and Generative Pre-trained Transformer (GPT) technology has the ability to generate responses to questions provided in text format. LLMs are trained on a large corpus of text. Training of such a model can include masking, where certain words/phrases (tokens) in a block of text are masked (omitted) and the task for the model is to predict the masked token. While such technology produces text content with plausible answers to a given question, it is difficult to judge the trustworthiness of the generated answers. It has been shown that answers generated by such technology may not always be factually accurate. DESCRIPTION This disclosure describes automated techniques to improve the credibility and confidence of machine-generated answers by using results from a search engine. Per the techniques, search results corresponding to a user input query are used to generate an augmented query that is provided as input to an LLM. Such augmentation adds context and can improve the quality and reliability of answers generated by the LLM. Further, the techniques can also be utilized to add citations to the generated answer and to indicate a confidence level in the generated answer. The availability of a citation informs the user regarding the information sources that support the generated text and enables the user to access such sources to verify the answer. 3Defensive Publications Series, Art. 5889 [2023]https://www.tdcommons.org/dpubs_series/5889 Fig 1: Query augmentation using search engine results to improve answers generated by a Large Language Model (LLM) Fig. 1 illustrates an example method to use results from a search engine to improve answers generated by an LLM. The query text is obtained (102), e.g., as input by a user. One or more queries based on the query text are executed on a search engine (104). Summary text from the search engine results is extracted by an LLM (106). Augmented query text is constructed by augmenting the summary text results along with the original query (108). The augmented query text is provided as input to the LLM (110). Based on the augmented query text, which provides context and related information (in the form of the added summary text) to the LLM, the LLM generates answers (112) that are an improvement upon the answers generated for the user input 4Park and Ryu: Query Augmentation Using Search Engine Results to Improve AnswersPublished by Technical Disclosure Commons, 2023query text without augmentation. References are added to the answer and the final answer is presented to the user (114). The improvement can include better answers due to a more in-depth query that incorporates more context, as well as additional citations that help the user verify the generated answer. Example of use Consider a user that wishes to find out who invented the telephone. Shown below are example responses from a chatbot that makes use of an LLM, with no query augmentation, as well as the current response from a search engine. In each, the user submits a query and receives a response. Current chatbot: A user asks a query to a chatbot, in response to which text is generated (e.g., using a large language model that powers the chatbot) and provided to the user, as shown in the example below:  Query to Chatbot Who invented the telephone ? Answer from Chatbot Alexander Graham Bell is credited with inventing the telephone. He was a Scottish-born scientist, inventor, and engineer who lived in the 19th century. On March 10, 1876, Bell was granted a patent for an invention he called the "improvement in telegraphy," which was more commonly known as the telephone. This invention revolutionized the way people communicated with each other and paved the way for modern telecommunication technology.  Search engine: A user submits a query to a search engine. In response, the search engine provides a set of links and snippets of text from each of the links, as shown in the example below: 5Defensive Publications Series, Art. 5889 [2023]https://www.tdcommons.org/dpubs_series/5889Search Query: Who invented the telephone?   Search results: Link 1 and associated snippet Link 2 and associated snippet Link 3 and associated snippet  Per techniques of this disclosure, top links identified by the search engine in response to the query text input by a user are provided to a large language model that powers the chatbot) to obtain a summary of corresponding pages. For example, for the links 1, 2, and 3 above, the corresponding summary text generated by the LLM is:  Summary Text for Link 1: Alexander Graham Bell is widely credited with inventing the telephone. He was awarded the first US patent for an "improvement in telegraphy" in 1876, which described the process of transmitting vocal or musical sounds telegraphically. However, several inventors such as Elisha Gray and Antonio Meucci had also developed similar devices and made substantial contributions to the invention of the telephone. …   Summary Text for Link 2: Alexander Graham Bell (1847-1922) was a Scottish-born scientist, inventor, engineer and innovator who is credited with inventing the first practical telephone. He also made significant contributions to the fields of communication and sound recording. …  Summary Text for Link 3: Alexander Graham Bell is credited with inventing the first practical telephone in 1876, but the development of the telephone was a collaborative effort involving multiple inventors and companies.     Per techniques of this disclosure, the summary text is utilized to augment the 6Park and Ryu: Query Augmentation Using Search Engine Results to Improve AnswersPublished by Technical Disclosure Commons, 2023question posed by the user to the chatbot. The more detailed prompt can lead to improvements in the answer provided by the chatbot, as shown below. The yellow highlights indicate that the LLM has incorporated context from the search results included in the augmented query.   Query to Chatbot [Summary text for link 1] [Summary text for link 2] [Summary text for link 3] Who invented the telephone ? Answer from Chatbot The telephone as we know it today is the result of the collective efforts and innovations of multiple inventors, including Alexander Graham Bell, Elisha Gray, and Antonio Meucci. Alexander Graham Bell is widely credited with inventing the first practical telephone and was awarded the first US patent for it in 1876. However, it was a result of the combined effort and contributions of several inventors and companies, which played a major role in the development and widespread adoption of the telephone.  The summaries can also be used to improve the answer supplied by the chatbot in other ways. Citation creation is improved by generating new citation queries to the LLM, asking how well certain search result summaries explain portions of the LLM generated answer. As a result, the chatbot can identify relevant sources for different portions of the generated answer. In addition, the text summaries can also be used to indicate the confidence level and thoroughness of the citation. The proposed search augmentation is performed as follows: For a user’s {query_text} (e.g., “Who invented the telephone?”), the generated {answer_text} as output by an LLM may be “Alexander Graham Bell”.  7Defensive Publications Series, Art. 5889 [2023]https://www.tdcommons.org/dpubs_series/5889Per techniques of this disclosure, the {query_text} is first run on a search engine to obtain additional information, e.g., a list of [ {searched_link_and_contents_k}... ] = SearchEngine( {query_texts} ) , k=0...N-1.  For each {searched_link_and_contents_k}, summary text is obtained by using the LLM: {summary_text_k} = LLM ( “summarize {searched_contents}_k “) An augmented query is constructed as: augmented_query_text} = “{summary_text_1}... {summary_text_k} ... {summary_text_(N-1)} {query_texts}” A new generated answer is obtained by feeding the {augmented_query_text} into LLM as input - {final_answer_texts} = LLM( {augmented_query_text} ). The {final_answer_texts} can be decomposed into a list of [ { answer_sentence_k} ]. A list of references can be attached to the final answer.  To include a list of references, for each { answer_sentence_k }, an LLM query is built as {cite_query_k} = “Text_1 is ‘{summary_text_1}’ ,,, Text_(N-1) is ‘{summary_text_(N-1)}’, which text best explains {answer_sentence_k}”. Extract reference text id r is obtained by = LLM ( {cite_query_k} ).  Further, another query to the LLM in the form {confidence_query_k} = “Text_r is ‘{summary_text_r}’, How thoroughly Text_r explains {answer_sentence_k} ? Answer with confidence level from 0 to 10”, can be utilized to extract a confidence level L from output = LLM ( {confidence_query_k} ). If the confidence level is below a threshold, the search can be rerun with the answer_sentence_k to obtain additional summary_text to obtain additional citations.  The techniques described herein can be implemented in any chatbot or other interface that utilizes a LLM to generate answers to user queries. 8Park and Ryu: Query Augmentation Using Search Engine Results to Improve AnswersPublished by Technical Disclosure Commons, 2023CONCLUSION This disclosure describes automated techniques to improve the credibility and confidence of machine-generated answers by using results from a search engine. Per the techniques, search results corresponding to a user input query are used to generate an augmented query that is provided as input to an LLM. Such augmentation adds context and can improve the quality and reliability of answers generated by the LLM. Further, the techniques can also be utilized to add citations to the generated answer and to indicate a confidence level in the generated answer. REFERENCES 1. “Failure archive for ChatGPT and similar models” available online at https://github.com/giuven95/chatgpt-failures accessed April 10, 2023. 2. “Google’s main Bard example contains a mistake” available online at https://github.com/giuven95/chatgpt-failures accessed April 10, 2023. 9Defensive Publications Series, Art. 5889 [2023]https://www.tdcommons.org/dpubs_series/5889

Query Augmentation Using Search Engine Results to Improve Answers Generated by Large Language Models

https://www.tdcommons.org/cgi/viewcontent.cgi?article=6994&context=dpubs_series

Query Augmentation Using Search Engine Results to Improve Answers Generated by Large Language Models

Abstract

Similar works

Full text

Available Versions

Technical Disclosure Common