Decoder Side Multiplane Images using Geometry Assistance SEI for MPEG Immersive Video

International audienceThe MPEG Immersive Video (MIV) standard enables a novel technology denoted as decoder side depth estimation (DSDE) by introducing a dedicated Geometry Absent profile. In DSDE only texture information is coded and the corresponding geometry is reconstructed on the decoder side. MIV further enables the coding of side-information useful to the geometry reconstruction, denoted as Geometry Assistance SEI message. An emerging format for immersive video are Multiplane Images, which is investigated for feasibility in coding systems due to their promising rendering quality with complex sequences. In this work, we show that MIV can be used to construct block-based Multiplane Images on the decoder-side and to enhance the view synthesis performance utilizing the Geometry Assistance SEI. In a complexity-aware setting using only 32 planes, up to 6 dB of quality improvement is achieved compared to the reference

Motion Compensation-based Low-Complexity Decoder Side Depth Estimation for MPEG Immersive Video

International audienceDecoder-Side Depth Estimation (DSDE) is a system firstly enabled in the novel MPEG Immersive Video (MIV) coding standard. In DSDE, only texture components are coded, while the depth is estimated at the decoder-side. This is motivated by previous work, which has shown high coding gain and pixel rate savings in DSDE. However, the computational complexity remains a concern, as high quality depth search has a high runtime and memory requirement. In this work we extend the concept of depth estimation to depth recovery. Using this mode, the decoder-side depth information is recovered through motion compensation utilizing the displacement vectors contained in the texture bitstream. This strategy enables us to replace most of the complex depth estimation processes with a simple motion compensation step, a decision that is drawn on the encoder-side and signaled per coding unit. With only minor losses in terms of synthesis PSNR and similar perceptual quality in terms of MS-SSIM, the complexity is significantly reduced. Depending on the acceptable loss, up to 80% of the moving objects depth may be motion compensated instead of estimated by a depth estimator translating into a speed-up of a factor of 104 for inter-frames compared to the reference depth estimator

The evolving SARS-CoV-2 epidemic in Africa: Insights from rapidly expanding genomic surveillance

INTRODUCTION Investment in Africa over the past year with regard to severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) sequencing has led to a massive increase in the number of sequences, which, to date, exceeds 100,000 sequences generated to track the pandemic on the continent. These sequences have profoundly affected how public health officials in Africa have navigated the COVID-19 pandemic. RATIONALE We demonstrate how the first 100,000 SARS-CoV-2 sequences from Africa have helped monitor the epidemic on the continent, how genomic surveillance expanded over the course of the pandemic, and how we adapted our sequencing methods to deal with an evolving virus. Finally, we also examine how viral lineages have spread across the continent in a phylogeographic framework to gain insights into the underlying temporal and spatial transmission dynamics for several variants of concern (VOCs). RESULTS Our results indicate that the number of countries in Africa that can sequence the virus within their own borders is growing and that this is coupled with a shorter turnaround time from the time of sampling to sequence submission. Ongoing evolution necessitated the continual updating of primer sets, and, as a result, eight primer sets were designed in tandem with viral evolution and used to ensure effective sequencing of the virus. The pandemic unfolded through multiple waves of infection that were each driven by distinct genetic lineages, with B.1-like ancestral strains associated with the first pandemic wave of infections in 2020. Successive waves on the continent were fueled by different VOCs, with Alpha and Beta cocirculating in distinct spatial patterns during the second wave and Delta and Omicron affecting the whole continent during the third and fourth waves, respectively. Phylogeographic reconstruction points toward distinct differences in viral importation and exportation patterns associated with the Alpha, Beta, Delta, and Omicron variants and subvariants, when considering both Africa versus the rest of the world and viral dissemination within the continent. Our epidemiological and phylogenetic inferences therefore underscore the heterogeneous nature of the pandemic on the continent and highlight key insights and challenges, for instance, recognizing the limitations of low testing proportions. We also highlight the early warning capacity that genomic surveillance in Africa has had for the rest of the world with the detection of new lineages and variants, the most recent being the characterization of various Omicron subvariants. CONCLUSION Sustained investment for diagnostics and genomic surveillance in Africa is needed as the virus continues to evolve. This is important not only to help combat SARS-CoV-2 on the continent but also because it can be used as a platform to help address the many emerging and reemerging infectious disease threats in Africa. In particular, capacity building for local sequencing within countries or within the continent should be prioritized because this is generally associated with shorter turnaround times, providing the most benefit to local public health authorities tasked with pandemic response and mitigation and allowing for the fastest reaction to localized outbreaks. These investments are crucial for pandemic preparedness and response and will serve the health of the continent well into the 21st century

Compression et synthèse pour représentation de contenus immersifs adaptés au 6DoF

A novel immersive video coding standard has been finalized 2022 by the Moving Picture Experts Group (MPEG) denoted as MPEG Immersive Video (MIV), MPEG-I Part 12. The MIV standard can be used to enable free navigation within a scene. However, appropriate compression of the geometry cannot be expected as widely used 2D video codecs do not support dedicated depth coding tools. Furthermore, several disadvantages like higher bit- and pixel rate requirements are connected with the compression of geometry. In this thesis Decoder Side Depth Estimation (DSDE) is proposed and further developed as an alternative coding system to MIV, which provides significant coding gain, pixel rate savings and improved perceptual quality. We further propose various novel improvements of DSDE, involving the partial transmission of geometry, the transmission of side information and the exploitation of the texture bitstream in order to further improve the coding gain and to reduce the complexity. Finally, we show that our proposals can be used to enhance the performance of more recent, neural network-based rendering methods like multiplane images.Une nouvelle norme de codage vidéo immersive a été finalisée par le Moving Picture Experts Group (MPEG). Il s'agit de la norme MPEG Immersive Video (MIV), MPEG-I Part 12. La norme MIV peut être utilisée pour permettre une navigation libre dans une scène. Cependant, il ne faut pas s'attendre à une compression appropriée de la géométrie, car les codecs vidéo 2D largement utilisés ne prennent pas en charge les outils dédiés au codage de la profondeur. En outre, la compression de la géométrie présente plusieurs inconvénients, tels que des exigences plus élevées en termes de débit binaire et de taux de pixel. Dans cette thèse, Decoder Side Depth Estimation (DSDE) est proposé et développé comme un système de codage alternatif au MIV, qui offre un gain de codage significatif, des économies de taux de pixel et une meilleure qualité perceptuelle. Nous proposons en outre plusieurs nouvelles améliorations de DSDE, impliquant la transmission partielle de la géométrie, la transmission d'informations latérales et l'exploitation du flux binaire de la texture afin d'améliorer encore le gain de codage et de réduire la complexité. Enfin, nous montrons que nos propositions peuvent être utilisées pour améliorer les performances de méthodes de rendu plus récentes, basées sur les réseaux neuronaux, comme les images multiplans

Compression et synthèse pour représentation de contenus immersifs adaptés au 6DoF

A novel immersive video coding standard has been finalized 2022 by the Moving Picture Experts Group (MPEG) denoted as MPEG Immersive Video (MIV), MPEG-I Part 12. The MIV standard can be used to enable free navigation within a scene. However, appropriate compression of the geometry cannot be expected as widely used 2D video codecs do not support dedicated depth coding tools. Furthermore, several disadvantages like higher bit- and pixel rate requirements are connected with the compression of geometry. In this thesis Decoder Side Depth Estimation (DSDE) is proposed and further developed as an alternative coding system to MIV, which provides significant coding gain, pixel rate savings and improved perceptual quality. We further propose various novel improvements of DSDE, involving the partial transmission of geometry, the transmission of side information and the exploitation of the texture bitstream in order to further improve the coding gain and to reduce the complexity. Finally, we show that our proposals can be used to enhance the performance of more recent, neural network-based rendering methods like multiplane images.Une nouvelle norme de codage vidéo immersive a été finalisée par le Moving Picture Experts Group (MPEG). Il s'agit de la norme MPEG Immersive Video (MIV), MPEG-I Part 12. La norme MIV peut être utilisée pour permettre une navigation libre dans une scène. Cependant, il ne faut pas s'attendre à une compression appropriée de la géométrie, car les codecs vidéo 2D largement utilisés ne prennent pas en charge les outils dédiés au codage de la profondeur. En outre, la compression de la géométrie présente plusieurs inconvénients, tels que des exigences plus élevées en termes de débit binaire et de taux de pixel. Dans cette thèse, Decoder Side Depth Estimation (DSDE) est proposé et développé comme un système de codage alternatif au MIV, qui offre un gain de codage significatif, des économies de taux de pixel et une meilleure qualité perceptuelle. Nous proposons en outre plusieurs nouvelles améliorations de DSDE, impliquant la transmission partielle de la géométrie, la transmission d'informations latérales et l'exploitation du flux binaire de la texture afin d'améliorer encore le gain de codage et de réduire la complexité. Enfin, nous montrons que nos propositions peuvent être utilisées pour améliorer les performances de méthodes de rendu plus récentes, basées sur les réseaux neuronaux, comme les images multiplans

Compression et synthèse pour représentation de contenus immersifs adaptés au 6DoF

A novel immersive video coding standard has been finalized 2022 by the Moving Picture Experts Group (MPEG) denoted as MPEG Immersive Video (MIV), MPEG-I Part 12. The MIV standard can be used to enable free navigation within a scene. However, appropriate compression of the geometry cannot be expected as widely used 2D video codecs do not support dedicated depth coding tools. Furthermore, several disadvantages like higher bit- and pixel rate requirements are connected with the compression of geometry. In this thesis Decoder Side Depth Estimation (DSDE) is proposed and further developed as an alternative coding system to MIV, which provides significant coding gain, pixel rate savings and improved perceptual quality. We further propose various novel improvements of DSDE, involving the partial transmission of geometry, the transmission of side information and the exploitation of the texture bitstream in order to further improve the coding gain and to reduce the complexity. Finally, we show that our proposals can be used to enhance the performance of more recent, neural network-based rendering methods like multiplane images.Une nouvelle norme de codage vidéo immersive a été finalisée par le Moving Picture Experts Group (MPEG). Il s'agit de la norme MPEG Immersive Video (MIV), MPEG-I Part 12. La norme MIV peut être utilisée pour permettre une navigation libre dans une scène. Cependant, il ne faut pas s'attendre à une compression appropriée de la géométrie, car les codecs vidéo 2D largement utilisés ne prennent pas en charge les outils dédiés au codage de la profondeur. En outre, la compression de la géométrie présente plusieurs inconvénients, tels que des exigences plus élevées en termes de débit binaire et de taux de pixel. Dans cette thèse, Decoder Side Depth Estimation (DSDE) est proposé et développé comme un système de codage alternatif au MIV, qui offre un gain de codage significatif, des économies de taux de pixel et une meilleure qualité perceptuelle. Nous proposons en outre plusieurs nouvelles améliorations de DSDE, impliquant la transmission partielle de la géométrie, la transmission d'informations latérales et l'exploitation du flux binaire de la texture afin d'améliorer encore le gain de codage et de réduire la complexité. Enfin, nous montrons que nos propositions peuvent être utilisées pour améliorer les performances de méthodes de rendu plus récentes, basées sur les réseaux neuronaux, comme les images multiplans

Compression and synthesis for representation of immersive content adapted to 6DoF

Une nouvelle norme de codage vidéo immersive a été finalisée par le Moving Picture Experts Group (MPEG). Il s'agit de la norme MPEG Immersive Video (MIV), MPEG-I Part 12. La norme MIV peut être utilisée pour permettre une navigation libre dans une scène. Cependant, il ne faut pas s'attendre à une compression appropriée de la géométrie, car les codecs vidéo 2D largement utilisés ne prennent pas en charge les outils dédiés au codage de la profondeur. En outre, la compression de la géométrie présente plusieurs inconvénients, tels que des exigences plus élevées en termes de débit binaire et de taux de pixel. Dans cette thèse, Decoder Side Depth Estimation (DSDE) est proposé et développé comme un système de codage alternatif au MIV, qui offre un gain de codage significatif, des économies de taux de pixel et une meilleure qualité perceptuelle. Nous proposons en outre plusieurs nouvelles améliorations de DSDE, impliquant la transmission partielle de la géométrie, la transmission d'informations latérales et l'exploitation du flux binaire de la texture afin d'améliorer encore le gain de codage et de réduire la complexité. Enfin, nous montrons que nos propositions peuvent être utilisées pour améliorer les performances de méthodes de rendu plus récentes, basées sur les réseaux neuronaux, comme les images multiplans.A novel immersive video coding standard has been finalized 2022 by the Moving Picture Experts Group (MPEG) denoted as MPEG Immersive Video (MIV), MPEG-I Part 12. The MIV standard can be used to enable free navigation within a scene. However, appropriate compression of the geometry cannot be expected as widely used 2D video codecs do not support dedicated depth coding tools. Furthermore, several disadvantages like higher bit- and pixel rate requirements are connected with the compression of geometry. In this thesis Decoder Side Depth Estimation (DSDE) is proposed and further developed as an alternative coding system to MIV, which provides significant coding gain, pixel rate savings and improved perceptual quality. We further propose various novel improvements of DSDE, involving the partial transmission of geometry, the transmission of side information and the exploitation of the texture bitstream in order to further improve the coding gain and to reduce the complexity. Finally, we show that our proposals can be used to enhance the performance of more recent, neural network-based rendering methods like multiplane images

Compression et synthèse pour représentation de contenus immersifs adaptés au 6DoF

A novel immersive video coding standard has been finalized 2022 by the Moving Picture Experts Group (MPEG) denoted as MPEG Immersive Video (MIV), MPEG-I Part 12. The MIV standard can be used to enable free navigation within a scene. However, appropriate compression of the geometry cannot be expected as widely used 2D video codecs do not support dedicated depth coding tools. Furthermore, several disadvantages like higher bit- and pixel rate requirements are connected with the compression of geometry. In this thesis Decoder Side Depth Estimation (DSDE) is proposed and further developed as an alternative coding system to MIV, which provides significant coding gain, pixel rate savings and improved perceptual quality. We further propose various novel improvements of DSDE, involving the partial transmission of geometry, the transmission of side information and the exploitation of the texture bitstream in order to further improve the coding gain and to reduce the complexity. Finally, we show that our proposals can be used to enhance the performance of more recent, neural network-based rendering methods like multiplane images.Une nouvelle norme de codage vidéo immersive a été finalisée par le Moving Picture Experts Group (MPEG). Il s'agit de la norme MPEG Immersive Video (MIV), MPEG-I Part 12. La norme MIV peut être utilisée pour permettre une navigation libre dans une scène. Cependant, il ne faut pas s'attendre à une compression appropriée de la géométrie, car les codecs vidéo 2D largement utilisés ne prennent pas en charge les outils dédiés au codage de la profondeur. En outre, la compression de la géométrie présente plusieurs inconvénients, tels que des exigences plus élevées en termes de débit binaire et de taux de pixel. Dans cette thèse, Decoder Side Depth Estimation (DSDE) est proposé et développé comme un système de codage alternatif au MIV, qui offre un gain de codage significatif, des économies de taux de pixel et une meilleure qualité perceptuelle. Nous proposons en outre plusieurs nouvelles améliorations de DSDE, impliquant la transmission partielle de la géométrie, la transmission d'informations latérales et l'exploitation du flux binaire de la texture afin d'améliorer encore le gain de codage et de réduire la complexité. Enfin, nous montrons que nos propositions peuvent être utilisées pour améliorer les performances de méthodes de rendu plus récentes, basées sur les réseaux neuronaux, comme les images multiplans

Decoder Side Multiplane Images using Geometry Assistance SEI for MPEG Immersive Video

International audienceThe MPEG Immersive Video (MIV) standard enables a novel technology denoted as decoder side depth estimation (DSDE) by introducing a dedicated Geometry Absent profile. In DSDE only texture information is coded and the corresponding geometry is reconstructed on the decoder side. MIV further enables the coding of side-information useful to the geometry reconstruction, denoted as Geometry Assistance SEI message. An emerging format for immersive video are Multiplane Images, which is investigated for feasibility in coding systems due to their promising rendering quality with complex sequences. In this work, we show that MIV can be used to construct block-based Multiplane Images on the decoder-side and to enhance the view synthesis performance utilizing the Geometry Assistance SEI. In a complexity-aware setting using only 32 planes, up to 6 dB of quality improvement is achieved compared to the reference