The heterogeneity in the contemporary multimedia environments requires a format-agnostic adaptation framework for the consumption of digital video content. Scalable bitstreams can be used in order to satisfy as many circumstances as possible. In this paper, the scalable extension on the H.264/AVC specification is used to obtain the parent bitstreams. The adaptation along the combined scalability axis of the bitstreams is done in a format-independent manner. Therefore, an abstraction layer of the bitstream is needed. In this paper, XML descriptions are used representing the high-level structure of the bitstreams by relying on the MPEG-21 Bitstream Syntax Description Language standard. The exploitation of the combined scalability is executed in the XML domain by implementing the adaptation process in a Streaming Transformation for XML (STX) stylesheet. The algorithm used in the transformation of the XML description is discussed in detail in this paper. From the performance measurements, one can conclude that the STX transformation in the XML domain and the generation of the corresponding adapted bitstream can be realized in real time

De Neve, Wesley

De Schrijver, Davy

De Wolf, Koen

Lambert, Peter

Van de Walle, Rik

Van Deursen, Davy

English

Ghent University Academic Bibliography

XML-driven Exploitation of Combined Scalabilityin Scalable H.264/AVC BitstreamsDavy De Schrijver, Wesley De Neve, Koen De Wolf, Peter Lambert, Davy Van Deursen, and Rik Van de WalleDepartment of Electronics and Information Systems – Multimedia LabGhent University – IBBTGaston Crommenlaan 8 bus 201, B-9050 Ledeberg-Ghent, BelgiumEmail: davy.deschrijver@ugent.beAbstract—The heterogeneity in the contemporary multimediaenvironments requires a format-agnostic adaptation frameworkfor the consumption of digital video content. Scalable bitstreamscan be used in order to satisfy as many circumstances as possible.In this paper, the scalable extension on the H.264/AVC speciﬁ-cation is used to obtain the parent bitstreams. The adaptationalong the combined scalability axis of the bitstreams is done ina format-independent manner. Therefore, an abstraction layerof the bitstream is needed. In this paper, XML descriptions areused representing the high-level structure of the bitstreams byrelying on the MPEG-21 Bitstream Syntax Description Languagestandard. The exploitation of the combined scalability is executedin the XML domain by implementing the adaptation processin a Streaming Transformation for XML (STX) stylesheet. Thealgorithm used in the transformation of the XML descriptionis discussed in detail in this paper. From the performancemeasurements, one can conclude that the STX transformationin the XML domain and the generation of the correspondingadapted bitstream can be realized in real time.I. INTRODUCTIONNowadays, digital video content can be accessed bydifferent users in heterogeneous environments. Two compo-nents, in particular scalable bitstreams and a format-agnosticadaptation framework, are needed in order to control the hugediversity in content and resource constraints such as terminalcapabilities, band width, and user preferences. In this paper,both technologies are brought together to adapt the scalablebitstreams by making use of a format-agnostic engine.The aim of Scalable Video Coding (SVC) is to encode avideo sequence once, after which the generated bitstreamcan be adapted by using simple truncation operations. Theseoperations make it possible to extract bitstreams containing alower frame rate, spatial resolution, and/or visual quality fromthe original coded bitstream. Therefore, an SVC bitstreamcontains three scalability axes (temporal, spatial, and SNR)along which adaptations can be executed. Every scalabilityaxis is independently accessible but it is also possible to adaptthe bitstream by truncating along multiple axes at the sametime. This results in combined scalability and this type ofscalability will be exploited in this paper. Hereby, we make useof bitstreams compliant with the Joint Scalable Video Model(JSVM) version 6 specification [1].The scalable bitstreams will be adapted by a format-independent engine. Therefore, we will describe the high-levelstructure of the bitstreams in XML. These XML descriptionswill be generated by relying on the MPEG-21 BitstreamSyntax Description Language (MPEG-21 BSDL, [2]) frame-work. This gives us the possibility to shift the focus ofthe content customization process to the XML domain. Theadaptation process in the XML domain can be realized by atransformation engine without knowledge of the underlyingcoding format. Such an engine typically takes a stylesheet,representing the transformation actions, as input. In this paper,we will make use of Streaming Transformations for XML(STX, [3]) and we will pay special attention to the implemen-tation of an STX stylesheet exploiting the combined scalabilitycharacteristic of JSVM6-coded bitstreams.The performance of doing the adaptation process in the XMLdomain will be investigated in order to verify the usefulness ofa fully XML-driven adaptation framework for scalable videocontent adaptation.The outline of this paper is as follows. In Section II, afully XML-driven adaptation framework is explained. Thehigh-level structure of the scalable bitstreams is discussedin Section III. Section IV describes the adaptation processin the XML domain. More precisely, the STX stylesheetimplementing the combined scalability is discussed. The per-formance results and accompanying discussing are providedin Section V. Finally, a conclusion is given in Section VI.II. XML-DRIVEN ADAPTATION FRAMEWORKThe MPEG-21 Digital Item Adaptation (DIA) standardenables the customization of multimedia content in hetero-geneous environments. One of the building blocks of DIAis MPEG-21 BSDL. This language allows to build an inter-operable description-driven framework in which multimediacontent can be adapted in a format-agnostic manner [4]. InFig. 1, an overview of our fully XML-driven framework forvideo content adaptation is given. The core of the frameworkis the automatic generation of XML descriptions containinginformation about the high-level structure of the scalablebitstreams and the adaptation thereof. Explanatory notes ofthe indicated numbers in Fig. 1 are provided below.(1) The Bitstream Syntax Schema (BS Schema) representsthe high-level structure of the coding format of the scal-able bitstream as speciﬁed in an international standard.The language used to construct such a BS Schema isBSDL, which is standardized in the DIA speciﬁcation.Adaptation Decision-Taking EngineUsage EnvironmentDescriptionSteeringDescriptionParametersSTX Engine STX StylesheetXML Description (BSD)Transformed BSDBSDtoBinBS SchemaBSDLinkDescriptionOriginalScalable BitstreamBintoBSDAdapted Bitstream: Half Frame Rate(2)<bitstream><header>0 25</header><frame>26 125</frame><frame>126 352</frame> <frame>353 541</frame> <frame>541 845</frame>  </bitstream><bitstream><header>0 25</header><frame>26 125</frame><frame>353 541</frame> </bitstream>(1)(3)(4)(5)Fig. 1. Overview of a fully XML-driven framework for video content adaptation based on MPEG-21 DIA(2) The Bitstream Syntax Description (BSD) of a (scalable)bitstream is an XML document and generated by aformat-agnostic parser. This parser, i.e. the BintoBSDParser, takes the original bitstream and accompanying BSSchema as input to generate the BSD.(3) A scalable bitstream has to be adapted according to ausage environment. Therefore, an engine is needed toprovide an adequate decision of the properties of thescalability axes along which the adaptation has to be exe-cuted. This Adaptation Decision-Taking Engine (ADTE)makes use of the Usage Environment Description (UED)describing the terminal capabilities, network characteris-tics, and user preferences. Because of the format-agnosticnature of this engine, the ADTE has to know whichadapted versions can be generated from the original bit-stream. This information about the scalability possibilitiesof the bitstream is described in the steering description.The outcome of the ADTE is a set of parameters, whichhave to be transmitted to the XML transformation engine.The BSDLink description deﬁnes these parameters andgives the ADTE the necessary information to map thecorrect values to the parameters.(4) Once the ADTE has stipulated the adaptation parameters,the generated BSD can be transformed by using a ubiq-uitous transformation language like STX. The adaptationis represented by a stylesheet and the format-neutral STXengine uses the parameters to generate the transformedBSD. This means that the adaptation is executed in theXML domain instead of on the bitstream itself.(5) The format-agnostic BSDtoBin Parser creates an adaptedbitstream, using the transformed BSD, the correspondingBS Schema, and the original (scalable) bitstream.Note that the 4 engines in the framework are format-agnosticmeaning that the code base of these parsers does not have tobe rewritten to support the adaptation of bitstreams compliantwith other coding formats. Furthermore, all communication(i.e., the descriptions) within the framework is XML driven.It is the ﬁrst time that a fully XML-driven framework isevaluated in which JSVM-coded bitstreams are used.III. SCALABLE EXTENSION OF H.264/MPEG-4 AVCThe video coding specification used, in particular JSVM6, isan extension of the non-scalable single-layered H.264/MPEG-4 Advanced Video Coding scheme (H.264/AVC). Conse-quently, a JSVM decoder can decode H.264/AVC bitstreamsand the base layer of a scalable bitstream should be compli-ant with H.264/AVC. Note, the fundamental building blocksof JSVM bitstreams are Network Abstraction Layer Units(NALUs), similar to H.264/AVC bitstreams.Fig. 2 shows the structure of a JSVM6-coded bitstreamproviding three temporal levels, two spatial layers, and aquality enhancement layer. This means that a JSVM6-encodergenerates scalable bitstreams containing following scalabilityproperties along which adaptations can be executed at the sametime (i.e., combined scalability).• Temporal scalability is obtained by using hierarchical Bpicture.• By down-sampling the original video sequence, spatialscalability is obtained.• To obtain quality or SNR scalability, one has to addition-ally code quality enhancement layers on top of a qualitybase layer. The JSVM6 speciﬁcation allows Coarse andFine Grain Scalability (CGS and FGS). In case of CGS,a complete enhancement layer is removed. While in caseof FGS, the enhancement layers can be truncated at anyarbitrary byte position.In in-depth explanation of the structure of JSVM6-codedbitstreams is discussed in [1].I0 P4B2B1 B3I0 P4B2B1 B3+FGS+FGS+FGS+FGS+FGS1st spatial layer(e.g., 176x144)2nd spatial layer(e.g., 352x288)= Temporal dependency= Spatial dependencyH.264/AVC CompliantFig. 2. Structure of a JSVM-coded bitstreamNALU-typeSEIput NALU  in buffer AbufferAparameters:- width- height- frame rate- bit ratetype SEI messagestartparsing layer infoscalability info SEIcorrect frame ratesave temporal levelcorrect width and heightyesnosave spatial layeryessave bitrate single layerbitrates single layersstore direct dependency information in buffer BbufferBnomore layers?yescalculatecumulated bit rate for each layernocalculate quality layer and truncation ratiotemporallevelspatial layerquality layertruncationratiodo nothingstore temporal level of framecurrent temp levelsub_seq_infoSEIother(pic_width <= width)and (pic_height <= height)SPSstore SPS idoutput buffer A to transformed BSDyesarray “SPS ids”do nothingnoSPS id of PPS element of array “SPS ids”?PPSoutput buffer A to transformed BSDyesdo nothingno(current temp level) <= (temporal level)AVC compliant base layeroutput buffer A to transformed BSDdo nothingyes noenhancement layersavetemporal level; spatial layer; quality levelcurrent templevelcurrent spatial layercurrent qualitylevel(current temp layer) <= (temporal level)noyescurrent spatial layer – spatial layer> 0output buffer A to transformed BSD< 0current quality layer – quality layer= 0do nothingoutput buffer A to transformed BSD> 0 < 0truncate quality layer, i.e., modify slice payload=0output modified buffer A to transformed BSDFig. 3. Data ﬂow of our STX stylesheet for exploiting combined scalabilityIV. EXPLOITATION OF COMBINED SCALABILITY IN XMLThe combined scalability of a JSVM bitstream can beexploited when a desired frame rate, spatial resolution, andbit rate are given to the STX transformation engine, whichare the parameters in Fig. 1. This engine is steered byan STX stylesheet representing the adaptation in the XMLdomain and transforms the generated BSD. To obtain such aBSD, a BS Schema for the JSVM6 speciﬁcation is developedwhich we discuss in detail in [5]. This BS Schema describesthe necessary information in XML in order to execute thecombined scalability. More precisely, the following syntacti-cal data structures are described: Supplemental EnhancementInformation (SEI) messages, Sequence Parameter Sets (SPSs),Picture Parameter Sets (PPSs), and the NALU headers.The data ﬂow of our STX stylesheet for the exploitation of thecombined scalability is shown in Fig. 3. As one can observe,the adaptation process can be divided into three main parts.In the ﬁrst step, the scalability information SEI message isinterpreted. Based on the information encapsulated in this SEImessage, the desired temporal, spatial, and quality level isdetermined. Furthermore, the truncation ratio is calculated incase the last quality layer is an FGS layer. The obtained valuesare stored in the stylesheet and are used during the decisionstep of the NALUs representing slices of the bitstream.In the second step, the parameter sets are parsed. Only theSPSs needed to decode the desired spatial layer will bekept in the BSD and the other SPSs are removed from thedescription. Because each PPS refers to an SPS (using thesequence parameter set id syntax element), we canalso remove all PPSs referring to removed SPSs.In the third step, the NALUs representing slices of the bit-stream are parsed. Hereby, we have to make a distinctionbetween slices belonging to an AVC-compliant base layer andbelonging to an enhancement layer. The main difference isthat the NALUs of the AVC-compliant base layer containno scalability information. Therefore, a sub seq info SEImessage precedes these NALUs containing information of thetemporal level. The NALUs of the enhancement layers containthe information about the temporal, spatial, and quality levelin the NALU header. Based on these values and the internalstored values from in the ﬁrst step, a decision can be made toremove, to keep, or to truncate the NALU in question.TABLE IPERFORMANCE RESULTS OF THE ADAPTATION ENGINEOriginal Bitstream BintoBSD Parser STX Transformation BSDtoBin ParserName #Frames Size (KB) ET (s) sizep (KB) sizec (KB) Desired Bitstream ET (s) sizep (KB) sizec (KB) ET (s) OB (Kbit/s)176x144@15:120 0.77 36.5 3 0.76 130.38Seq 1 75 3389 4.35 601 12.3 352x288@30:700 0.83 133 4 0.87 740.56704x576@60:5000 0.92 392 8 0.99 5334.63176x144@15:120 1.04 67 3 0.80 127.79Seq 2 150 5980 6.94 1072 20.7 352x288@30:700 1.14 325 7 0.95 727.45704x576@60:5000 1.31 764 13 1.10 5269.10176x144@15:120 1.43 126 4 0.81 121.24Seq 3 300 11451 12.23 2013 37.2 352x288@30:700 1.75 637 11 1.04 707.79704x576@60:5000 2.10 1509 24 1.33 5142.94176x144@15:120 2.12 198 5 0.99 119.81Seq 4 480 19966 19.44 3148 57.3 352x288@30:700 2.36 781 13 1.10 700.42704x576@60:5000 2.98 2404 37 1.58 5125.12V. PERFORMANCE RESULTSMethodology. To evaluate the performance of our XML-driven adaptation framework, we have generated four JSVM6-compliant scalable bitstreams. Each bitstream contains a partof the well-known ice sequence with a resolution of 704×576at a frame rate of 60Hz. The encoder generates bitstreams with5 temporal, 3 spatial, and 4 quality levels.For each bitstream, the corresponding BSD is generated byusing an optimized BintoBSD Parser as explained in [6].These BSDs are subject to the transformation reﬂecting theadaptation in the XML domain. From each bitstream, 3 partialstreams are generated containing a resolution of 176× 144 at15Hz (120 Kbit/s), a resolution of 352 × 288 at 30Hz (700Kbit/s), and a resolution of 704× 576 at 60Hz (5000 Kbit/s).The STX engine used is Joost (version 2005-05-21). Finally, amodiﬁed BSDtoBin Parser of the MPEG-21 reference softwareversion 1.2.1 is used to generate the adapted bitstreams.Discussion of the results. The performance results of thedifferent steps are given in Table I. The BintoBSD Parser is theﬁrst engine of which the performance was investigated. TheExecution Times (ETs) of the parser, the sizes of the resultingXML descriptions in plain text (sizep), and the sizes of thecompressed BSDs (sizec by using EasyZip v3.5) are given inthe table. Hereby, we can conclude that the ET is linear as afunction of the length of the sequence. The sizes of the plain-text generated BSDs is substantially compared to the originalbitstream, approximately 15% of the size of the bitstream. Bycompressing the BSDs, the overhead originates from the XMLdescription is negligible, roughly 0.3%.The STX engine is the next step in the adaptation process thatwas measured. The ETs are linear as a function of the lengthof bitstream if the sequence is long enough (meaning that thestart-up time can be ignored). The sizes (in plain text andcompressed) of the transformed BSDs represent the inﬂuenceof the adaptation parameters on the available NALUs.Finally, the ETs of the BSDtoBin Parser are given resultingin a fast engine. Note, the transformation together with thegeneration of the adapted bitstream is realized in almost realtime. The Obtained Bit rates (OBs) of the adapted bitstreamsapproach the desired rates very well. This means that ouradaptation engine can generate bitstreams containing a desiredbit rate without knowledge of the underlying coding format.VI. CONCLUSIONSIn this paper, a format-agnostic framework for scalablevideo content adaptation was proposed in which all commu-nication is based on XML descriptions. Not only the usageenvironment but also the high-level structure of the scalablebitstreams is described in XML by using MPEG-21 BSDL.This gives us the opportunity to shift the adaptation processto the XML domain. The JSVM6-coded bitstreams can beadapted along the three scalability axes at the same time (i.e.,combined scalability). The transformation exploiting the com-bined scalability of the XML descriptions was implemented inSTX. From the performance results, we can conclude that theexecution time of the transformation and the generation of theadapted bitstreams is linear as a function of the length of thesequences. Furthermore, we have proven that our XML-drivenframework can execute the adaptations in real time.ACKNOWLEDGEMENTSThe research activities that have been described in thispaper were funded by Ghent University, the InterdisciplinaryInstitute for Broadband Technology (IBBT), the Institute forthe Promotion of Innovation by Science and Technology inFlanders (IWT), the Fund for Scientific Research-Flanders(FWO-Flanders), the Belgian Federal Science Policy Office(BFSPO), and the European Union.REFERENCES[1] J. Reichel, H. Schwarz, and M. Wien, “Joint Scalable Video ModelJSVM-6,” Doc. JVT-S202, April 2005.[2] G. Panis, A. Hutter, J. Heuer, H. Hellwagner, H. Kosch, C. Timmerer,S. Devillers, and M. Amielh, “Bitstream syntax description: a tool formultimedia resource adaptation within MPEG-21,” Signal Processing:Image Communication, vol. 18, no. 8, pp. 721–747, 9 2003.[3] O. Becker, “Transforming XML on the ﬂy,” in Proceedings of XMLEurope, May 2003.[4] S. Devillers, C. Timmerer, J. Heuer, and H. Hellwagner, “Bitstream syntaxdescription-based adaptation in streaming and constrained environments,”IEEE Transactions on Multimedia, vol. 7, no. 3, pp. 463–470, 6 2005.[5] D. De Schrijver, W. De Neve, K. De Wolf, S. Notebaert, andR. Van de Walle, “XML-based customization along the scalability axesof H.264/AVC scalable video coding,” in Proceedings of IEEE ISCAS2006, Island of Kos, Greece, 5 2006, pp. 465–468.[6] D. De Schrijver, W. De Neve, K. De Wolf, and R. Van de Walle, “Gen-erating MPEG-21 BSDL descriptions using context-related attributes,” inProceedings of the 7th IEEE ISM, Irvine, (CA, USA), 12 2005, pp. 79–86.

XML-driven exploitation of combined scalability in scalable H.264/AVC bitstreams

https://biblio.ugent.be/publication/416748/file/448051

XML-driven exploitation of combined scalability in scalable H.264/AVC bitstreams

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography