The disclosed technology addresses technical challenges associated with augmented reality (AR) techniques. The disclosed technology can provide for determining an augmented reality (AR) effect applied to a content item. An audio description associated with the AR effect can be generated based on signals associated with the AR effect and the content item. The audio description associated with the AR effect can be provided. By providing audio descriptions associated with AR effects, the disclosed technology allows a broad range of users, including users with vision impairments, to enjoy content items created with AR effects

Afshar, Jaleh

English

Technical Disclosure Common

Technical Disclosure Commons Defensive Publications Series September 2021 SYSTEM FOR GENERATING ACCESSIBILITY FOR AUGMENTED REALITY INTERFACES Jaleh Afshar Follow this and additional works at: https://www.tdcommons.org/dpubs_series Recommended Citation Afshar, Jaleh, "SYSTEM FOR GENERATING ACCESSIBILITY FOR AUGMENTED REALITY INTERFACES", Technical Disclosure Commons, (September 21, 2021) https://www.tdcommons.org/dpubs_series/4609 This work is licensed under a Creative Commons Attribution 4.0 License. This Article is brought to you for free and open access by Technical Disclosure Commons. It has been accepted for inclusion in Defensive Publications Series by an authorized administrator of Technical Disclosure Commons. brought to you by COREView metadata, citation and similar papers at core.ac.ukprovided by Technical Disclosure Common  1  SYSTEM FOR GENERATING ACCESSIBILITY FOR AUGMENTED REALITY INTERFACES  DETAILED DESCRIPTION  [0001]  Keywords: augmented reality (AR), accessibility, vision impairment, audio description, computer vision. [0002]  A system, such as a social networking system, communication platform, or content sharing system, can enable users to create content items and share the created content items with other users. Creating content items and sharing the created content items allow users to engage with each other. For example, users can engage with each other by commenting on created content items or creating their own content items in response to the created content items. In some cases, users can create content items based on various augmented reality (AR) techniques. For example, an effect creator (e.g., a user who designs and creates an AR effect) can create a new AR effect and share it with other users via a content sharing system. Effect users (e.g., users who use an AR effect to create content items) can apply the new AR effect to, for example, an image or a video to create a content item. The created content item can be shared with effect consumers (e.g., users who consume created content items based on AR effects) who access the created content item. For example, an effect user of a content sharing system can create a content item by applying an AR effect that is visual to a video. The effect user can share the created content item, which includes the visual AR effect, with other users via the content sharing system. The other users can engage with each other based on the created content item, for example, by commenting on the created content item. In this example, because the created content item including the AR effect is visual, the created content item may be partially or entirely inaccessible to users with vision impairments. Accordingly, these users  may not be able to fully enjoy the created content item and may not be afforded opportunities to engage with other users in relation to the created content item. Thus, as illustrated by this example, conventional AR techniques pose disadvantages in  relation to their accessibility to users with vision impairments. 2Afshar: SYSTEM FOR GENERATING ACCESSIBILITY FOR AUGMENTED REALITY INTERFAPublished by Technical Disclosure Commons, 2021  2  [0003]  The technology disclosed herein provides for generating an audio description associated with a content item created with an AR effect. The audio description can be generated based on various signals associated with the AR effect or the created content item. An effect consumer can access the created content item and be provided with the audio description. The audio description can allow the effect consumer to hear information describing the created content item and the AR effect. For example, an effect creator can design a new AR effect. Various signals or types of information describing the design of the new AR effect, such as a description of the new AR effect provided by the effect creator, can be used for generating an audio description associated with the AR effect. An effect user can use the new AR effect to create a content item by applying the new AR effect, for example, to a video. Information regarding the use of the new AR effect, such as identification of new objects detected within the video, can be additionally included in the audio description. The created content item can be shared with an effect consumer. The effect consumer can access the created content item and be provided with an audio description based on the various signals associated with the design of the new AR effect and the use of the new AR effect. For example, as discussed, the audio description can include the description of the new AR effect provided by the effect creator and the identification of new objects detected within the video after the new AR effect is applied by the effect user. The audio description accordingly allows the effect consumer to more fully appreciate aspects of the created content item attributable to the AR effect. Thus, the technology disclosed herein allows content items created with AR effects to be enjoyed by a broad range of users, including users with vision impairments.  Generating Audio Descriptions [0004]  The technology disclosed herein can generate audio descriptions associated with an AR effect based on various signals. In some cases, the various signals can be associated with the design of the AR effect. The signals associated with the design of the AR effect can be provided by an effect creator associated with the AR effect. The effect creator can provide information, such as a name or a description for the AR effect. For example, an effect creator can provide a written description for an AR 3Defensive Publications Series, Art. 4609 [2021]https://www.tdcommons.org/dpubs_series/4609  3  effect. An audio description associated with the AR effect can include, for example, an audio reading of the written description. In some cases, signals associated with design of an AR effect can be determined from objects and transformations associated with the AR effect. The objects and transformations associated with an AR effect can be determined based on a scan of a source file associated with the AR effect. For example, an AR effect that is applied to a content item can add an object to the content item. The addition of the object can be determined based on a scan of a source file associated with the AR effect. An audio description associated with the AR effect can include audio based on the added object. The audio can include, for example, an audio identification of the added object. As another example, an AR effect that is applied to a content item can apply a transformation to the content item. An audio description associated with the AR effect can include audio based on the transformation. For example, the audio can include audio describing the transformation. In some cases, an AR effect can include multiple stages and each stage can involve different objects or transformations. For example, the AR effect can be a game and display different objects and transformations as the game advances through its stages. The stages and the objects and transformations displayed during the stages can be determined, for example, based on a scan of an AR effect source file associated with the game.  4Afshar: SYSTEM FOR GENERATING ACCESSIBILITY FOR AUGMENTED REALITY INTERFAPublished by Technical Disclosure Commons, 2021  4   Figure 1  [0005]  Figure 1 illustrates an example scenario associated with design of an AR effect where an effect creator uploads the AR effect using an example interface. As illustrated in Figure 1, the example interface includes a field for a name of the AR effect where an effect creator can provide the name of the AR effect. An audio description for the AR effect can be generated based on the name of the AR effect (e.g., “Roll for Initiative”). For example, the audio description can include an audio reading of the name of the AR effect. As illustrated in Figure 1, the example interface includes a field for an audio description of the AR effect where an effect creator can provide a written description on which the audio description of the AR effect can be based (e.g., “2D-sided red die appears on forehead and rolls to a random number”). For example, the audio description can include an audio reading of the written description of the AR effect.  Roll for Initiative20-sided red die appears on forehead and rolls to a random numberSubmitReplaceUpload Your AR EffectNameRoll_for_Initiative.source_fileAudio Description – Describe this effect for people using screen readersIcon205Defensive Publications Series, Art. 4609 [2021]https://www.tdcommons.org/dpubs_series/4609  5   Figure 2  [0006]  Figure 2 illustrates an example scenario associated with design of an AR effect where an effect creator uses an example interface to design transformations to be applied by the AR effect. As illustrated in Figure 2, the example interface includes a panel (e.g., “AR Transformations”) for designing transformations to be applied by the AR effect. The panel includes various transformations that can be applied by the AR effect (e.g., “Eyeliner Color,” “Eyeliner Opacity,” “Eyeshadow Color,” “Eyeshadow Opacity,” “Eyelash Length,” “Lip Color,” “Reflectivity,” “Saturation”). An audio description for the AR effect can be generated based on the transformations to be applied by the AR effect. For example, the audio description can include audio identifying the transformations to be applied by the AR effect. [0007]  In some cases, the various signals on which generation of an audio description associated with an AR effect are based can be associated with the application of the AR effect to a content item. The signals associated with the application of the AR effect can be based on how the AR effect was applied by an effect user and what is depicted in a content item to which the AR effect was applied. In general, applying an AR effect to a content item adds an object to the content item or applies a transformation to the content item. Computer vision techniques can be applied to the content item before the AR effect is applied to the content item and after the AR 6Afshar: SYSTEM FOR GENERATING ACCESSIBILITY FOR AUGMENTED REALITY INTERFAPublished by Technical Disclosure Commons, 2021  6  effect is applied to the content item to determine a delta (e.g., differences) that arises through application of the AR effect to the content item. An audio description associated with the AR effect can be generated based on the delta. For example, an AR effect can be designed to add an object to a video. When the AR effect is applied to the video, the AR effect adds the object to the video to create a new video. Computer vision techniques can be applied to the video and the new video to determine a delta between the video and the new video. An audio description associated with the AR effect can be generated based on the delta. For example, the audio description can describe the object added to the video to create the new video and, for example, describe where the object is in relation to other objects identified in the new video.             Figure 3  [0008]  Figure 3 illustrates an example scenario associated with application of an AR effect to a content item. As illustrated in Figure 3, an AR effect was applied to a video (the depicted left video) to create a new video (the depicted right video). The video depicts a person. The new video depicts a 20-sided die over the forehead of the 187Defensive Publications Series, Art. 4609 [2021]https://www.tdcommons.org/dpubs_series/4609  7  person. The 20-sided die displays a number (e.g., “18”). Computer vision techniques can be applied to the video and the new video to determine a delta between the video and the new video. As illustrated in Figure 3, the delta can include the 20-sided die displaying the number 18 located over the forehead of the person. An audio description for the AR effect can be generated based on the delta. For example, the audio description can include audio identifying the 20-sided die added to the video, audio describing that the 20-sided die displays the number 18, and audio describing where the 20-sided die is located.  Audio Descriptions During Consumption [0009]  The technology disclosed herein can provide an audio description associated with an AR effect. For example, the audio description can be provided to an effect consumer accessing a content item created based on an AR effect. In some cases, the audio description can be provided in response to an interaction by the effect consumer. For example, the effect consumer can select an option or interact with a button to hear an audio description. The audio description can be provided to the effect consumer in response to the selection of the option or the interaction with the button. In some cases, the audio description can be provided as part of a screen reader application. An effect consumer can enable a screen reader application. When the effect consumer accesses content items created based on an AR effect, the screen reader application can recite an audio description of the AR effect.  8Afshar: SYSTEM FOR GENERATING ACCESSIBILITY FOR AUGMENTED REALITY INTERFAPublished by Technical Disclosure Commons, 2021  8   Figure 4  [0010]  Figure 4 illustrates an example scenario associated with providing an audio description. As illustrated in Figure 4, an AR effect for adding a 20-sided die was applied to a video depicting a person. An audio description for the AR effect can be generated based on the various signals described herein. For example, the audio description can be based on a description provided by an effect creator and computer vision techniques. As illustrated in Figure 4, the audio description includes an identification of the person depicted in the video (e.g., “Jane Doe”). The audio description includes a name of the AR effect (e.g., “Roll for Initiative”). The audio description can indicate that the person depicted in the video is using the AR effect. The audio description can include a description of the AR effect, for example, based on a description provided by an effect creator (e.g., “A 20-sided red die appears on their forehead and rolls to a random number.”). The audio description can include a description of the 20-sided die added to the video, such as a resulting action of the 20-sided die. (e.g., “The number is 18.”).  18 Jane Doe is using the Roll for Initiative AR effect.A 20-sided red die appears on their forehead and rolls to a random number.The number is 18.9Defensive Publications Series, Art. 4609 [2021]https://www.tdcommons.org/dpubs_series/4609  9  Scope of Disclosed Technology [0011]  Certain details in this disclosure are set forth in order to provide an understanding of various embodiments of the technology that is disclosed. A reference herein to an “embodiment”, “instance”, “example”, or the like are not necessarily all referring to the same embodiment. Nor are separate or alternative embodiments mutually exclusive of other embodiments. Various features are herein provided or described through text and diagrams. The features may be variously combined and included in some embodiments, but also variously omitted in other embodiments. Similarly, various features are described that may be preferences or requirements for some embodiments, but not for other embodiments. Accordingly, the disclosure of the embodiments of the technology is intended to be illustrative, but not limiting, of the scope of the technology. The features of the disclosed technology may be variously combined, separated, removed, reordered, or replaced in a manner other than as expressly described and depicted herein.  [0012]  The disclosed technology can be categorized in one or more technical fields. The disclosed technology can be appropriately combined with other technologies categorized in the same technical fields in which the disclosed technology can be categorized. Further, the disclosed technology can be appropriately combined with other technologies in technical fields that are different from the technical fields in which the disclosed technology can be categorized. The disclosed technology has broad application to other technologies and other technical fields, and can be appropriately implemented, adjusted, modified, or combined therewith.  10Afshar: SYSTEM FOR GENERATING ACCESSIBILITY FOR AUGMENTED REALITY INTERFAPublished by Technical Disclosure Commons, 2021  10  ABSTRACT The disclosed technology addresses technical challenges associated with augmented reality (AR) techniques. The disclosed technology can provide for determining an augmented reality (AR) effect applied to a content item. An audio description associated with the AR effect can be generated based on signals associated with the AR effect and the content item. The audio description associated with the AR effect can be provided. By providing audio descriptions associated with AR effects, the disclosed technology allows a broad range of users, including users with vision impairments, to enjoy content items created with AR effects. 11Defensive Publications Series, Art. 4609 [2021]https://www.tdcommons.org/dpubs_series/4609

SYSTEM FOR GENERATING ACCESSIBILITY FOR AUGMENTED REALITY INTERFACES

https://www.tdcommons.org/cgi/viewcontent.cgi?article=5695&context=dpubs_series

SYSTEM FOR GENERATING ACCESSIBILITY FOR AUGMENTED REALITY INTERFACES

Abstract

Similar works

Full text

Available Versions

Technical Disclosure Common