Facilitating Discovery of Virtual Assistant Capabilities via Augmented Reality

Users may not be aware of all possible tasks that a virtual assistant can perform. This disclosure describes techniques for user-permitted use of augmented reality (AR) technology to help users discover contextually relevant tasks that a virtual assistant could perform. With user permission, the relevant tasks are determined by recognizing entities and/or locations within the camera view of the user’s device. Additional data from device sensors and/or other known user information such as devices owned or apps installed can also be utilized with user permission. The contextually relevant tasks are recommended to the user as an augmented message, via audio, or other interfaces, thus helping the user discover virtual assistant capabilities

Improved Dictation Using Virtual Assistant

Writing on small form factor devices such as smartphones is inefficient due to the small dimensions of such devices. This disclosure describes techniques that support natural-language, context-aware voice commands that refer to individual words, sentences, paragraphs, sections, chapters, etc. of text content. Various features are supported, such as: automatically splitting text into paragraphs; creating lists; using different fonts for subheadings, chapter titles, etc.; inserting footnotes; adding citations and references; etc. To assist the user in navigating the document, the transcribed text is displayed overlaid with annotations that indicate available commands; numbered textual components; etc. Machine learning and natural language processing techniques are used to automatically differentiate between text and commands