In today’s fast paced world, the time available to watch

long sports programmes is decreasing, while the number of sports channels is rapidly increasing. Many viewers desire the facility to watch just the highlights of sports events.

This paper presents a simple, but effective, method for generating sports video highlights summaries. Our method detects semantically important events in sports programmes by using the Scale Factors in the MPEG audio bitstream to generate an audio amplitude profile of the program. The Scale Factors for the subbands corresponding to the voice bandwidth give a strong indication of the level of commentator and/or spectator excitement. When periods of sustained high audio amplitude have been detected and ranked, the corresponding video shots may be concatenated to produce a summary of the program highlights. Our method uses only the Scale Factor information that is directly accessible from the MPEG bitstream, without any decoding, leading to highly efficient computation. It is also rather more generic than many existing techniques, being particularly suitable for the more popular sports televised in Ireland such as soccer, Gaelic football, hurling, rugby, horse racing and motor racing

Marlow, Seán

Murphy, Noel

O'Connor, Noel E.

Sadlier, David A.

English

The imminent rapid expansion in the number of TV channels is driving the need for efficient digital video indexing, browsing and playback systems. For the past four years, the Centre for Digital Video Processing in DCU has been working towards the provision of such a system. The current stage of development is demonstrated on our Web-based digital video system called Físchlár [1,2], which is now in hourly use by over 1000 registered users. At present a user can pre-set the recording of TV broadcast programmes and can choose from a set of different browser interfaces which allow navigation through the recorded programmes. As our research has developed we have been adding increased functionality such as personalisation and\ud
programme recommendation, automatic recording, SMS/WAP/PDA alerting, searching, summarising and so on. There have been many approaches to automatic highlight detection and summarization of sports programmes\ud
[3-6]. However, most methods are oriented to particular sports and/or specific edit effects or need much\ud
computational effort. Therefore we decided to take a more basic and generic approach based on the observation\ud
that, in most common sports coverage with a commentator and spectators, exciting events are usually characterized by an increase in the audio amplitude, particularly in the speech band. Measurement of the audio amplitude is described in Section II and a case study is presented in Section III. Section IV summarises results while conclusions and areas of continuing and future work are outlined in Section V

Name not available

Seán Marlow, David A. Sadlier, Noel O’Connor, Noel Murphy.Centre for Digital Video Processing,Dublin City University,Ireland.email: marlows@eeng.dcu.ie   phone: +353-1- 7005120Voice Processing for Automatic TV Sports Program Highlights DetectionI INTRODUCTIONThe imminent rapid expansion in the number of TV channels is driving the need for efficient digital videoindexing, browsing and playback systems. For the past four years, the Centre for Digital Video Processing inDCU has been working towards the provision of such a system. The current stage of development isdemonstrated on our Web-based digital video system called Físchlár [1,2], which is now in hourly use by over1000 registered users. At present a user can pre-set the recording of TV broadcast programmes and canchoose from a set of different browser interfaces which allow navigation through the recorded programmes. Asour research has developed we have been adding increased functionality such as personalisation andprogramme recommendation, automatic recording, SMS/WAP/PDA alerting, searching, summarising and so on.There have been many approaches to automatic highlight detection and summarization of sports programmes[3-6]. However, most methods are oriented to particular sports and/or specific edit effects or need muchcomputational effort. Therefore we decided to take a more basic and generic approach based on the observationthat, in most common sports coverage with a commentator and spectators, exciting events are usuallycharacterized by an increase in the audio amplitude, particularly in the speech band. Measurement of the audioamplitude is described in Section II and a case study is presented in Section III. Section IV summarises resultswhile conclusions and areas of continuing and future work are outlined in Section V.II AUDIO PROFILE GENERATIONa) MPEG Bitstream ProcessingThe Físchlár system captures television broadcasts and encodes the programmes according to the MPEG-1digital video standard with the audio signal coded in line with the Layer-II profile [8]. Unlike many other audiocompression algorithms, which make assumptions about the nature of the audio source, MPEG-1 Audio exploitsthe perceptual restrictions of the human auditory system, via psychoacoustic weighting of the bit allocation foreach frequency subband, to attain its compression [7].The MPEG-1 Layer-II compression algorithm encodes audio signals by dividing the frequency spectrum of theaudio signal, bandlimited to 20kHz, into 32 subbands. The subbands are assigned individual bit-allocationsaccording to the audibility of quantisation noise within each subband. A pyschoacoustic model of the earanalyses the audio signal and provides this information to the quantiser. Layer-II frames consist of 1152samples; 3 groups of 12 samples from each of 32 subbands. A group of 12 samples gets a bit-allocation and, ifthis is non-zero, a scalefactor. Scalefactors are weights that scale groups of 12 samples such that they fully usethe range of the quantiser . The scalefactor for such a group is determined by the next largest value (given in alook-up table) to the maximum of the absolute values of the 12 samples. Thus it provides an indication of themaximum power exhibited by any one of the 12 samples within the group.b) Amplitude of the Speech BandMost of the energy in a speech signal lies between 0.1kHz – 4kHz.  According to the MPEG-1 Layer-II audiostandard, the maximum allowable frequency component in the audio signal is at 20kHz. At the encoder, thefrequency spectrum (0 – 20kHz) is divided uniformly into 32 subbands, each having bandwidth of 0.625kHz [7].Thus, subbands 2 through 7 represent the frequency range from 0.625kHz – 4.375kHz. See Figure 1.                   Speech    1      2      3      4     5     6     7      8                      31    32  0.625kHz          4.375HzFigure 1:   MPEG-1 Layer-II Frequency SubbandsBy limiting the audio examination to these subbands, which approximate the range of the speech band, wefurther concentrate the audio investigation on commentator vocals. Therefore, the influence of the commentatoron the generation of the audio amplitude profile is increased. It was expected that the examination of subbands2 though 7 would provide for a reasonable trade-off between rejection of low-frequency background noise(typically present in sports programmes which would naturally upset results) and the capture of excited speech.c) Boundary DetectionOne of the problems with the audio amplitudes technique is caused by the inclusion of supplementary contentwhich typically accompanies the main event in a sports programme. Features such as player profiles, highlightsof recent events etc. tend to contain attributes such as commentator dialogue and spectator noise, similar to thatof the main event. The problem is that these features generally have audio amplitudes comparable to that of theevent of interest. To combat this problem, the system must be able to detect the temporal boundaries of themain feature within the overall sports programme. This is done by searching through the audio track forextended periods of sustained volume.Segments such as interviews, studio discussions, archive video clips, etc. which make up the peripheral content,are flagged by the intermittent occurrence of brief moments of silence. For example, short silences exist inbetween sentences spoken by an anchorperson, when switching from anchorperson to video clips, or betweenadvertisements. In contrast, the main event in a sports broadcast features relatively long periods of sustainedvolume due to the continuous presence of background noise.  On this basis it may be automaticallydistinguished from the supplementary content. i.e. the temporal boundaries of the main event within the overallprogramme may be detected. For the summary generation, the probing domain is restricted to lie within theseboundaries.III CASE  STUDYa) TaskThe following is an illustration of the automatic generation of a 10-minute summary of a terrestrial TV broadcastof a sports event via the discussed technique. The experimental subject is the UEFA Cup Final 2001 featuringLiverpool FC Vs Alaves FC. This was a near 3-hour soccer match broadcast, resulting in a 5-4 victory forLiverpool FC. The programme featured the main event plus studio discussions and analysis, player profiles,highlights of related events and advertisement breaks.b) Amplitude ProfilesA second-by-second audio amplitude profile was established by a superposition of all the scalefactors fromsubbands 2-7 over a window length of one second. See Figure 2.AudioAmplitude                 0    1   2   3   4   5    6   7   8   9         Time (s)Figure 2:  Per-Second Audio Amplitude ProfileA frame-by-frame audio amplitude profile was established by a superposition of all the scalefactors fromsubbands 2-7 over a window of length corresponding to one video frame (˜1/25s). See Figure 3.AudioAmplitude       Frame       Number                 0    1   2   3   4   5    6   7   8   9Figure 3:  Per-Frame Audio Amplitude Profilec) Boundary DetectionThe overall structure of the near 3-hour subject, as captured by Físchlár, is described below. In terms ofsummary generation, an asterisk identifies segments of interest.Ads ~3 minsInterviews ~14mins* 1st half ~51minsStudio analysis ~14mins* 2nd half ~49minsStudio analysis ~4 mins* Extra time ~26minsStudio analysis ~6 minsBoundary Detection using the per-frame audio amplitude profile with a silence threshold ofSth = 0.033 * overall mean audio amplitudeeliminated all but the 1st half, 2nd half and extra time from subsequent  processing.d) Summary GenerationThe per-second audio amplitude profiles of segments 1-3 (above) were examined. A loudness threshold, Lth,was defined and initalised to the value corresponding to the largest peak found. See Figure 4.    Segment-1     Segment-2    Segment-3Figure 4:  Examination of Segments 1-3An audio amplitude peak is defined as loud if it exceeds Lth. Ignoring isolated peaks, Lth was gradually reduceduntil it began to pick out loud periods of at least 3-seconds in duration (audio surges). See Figure 5.Figure 5:  Decreasing Lth and detecting audio surgesLthTimeFigure 5 shows three sections which extend beyond the current value of Lth. The second and third have timespans of 4 seconds and 3 seconds respectively. Thus both are recognised as audio surges. The first section isignored since with a length of 2 seconds, it does not meet the minimum surge threshold of 3-seconds. Lth wasfurther reduced until the amount of detected surges was sufficient such that a 10-minute video summary couldbe produced. The summary was then generated by first matching up the video clips within the combinedaudio/video track which temporally align with the audio surges. Then, a pre-clip buffer of 1 shot and a post clipbuffer of 2 shots was appended (to make viewing the amalgamation less visually disturbing). Finally these clipswere extracted from the audio/video stream and (chronologically) concatenated to generate the highlightssummary. See Figure 6.Figure 6:  Summary generationIV RESULTSThe analysis returned 18 individual clips corresponding to the following descriptions, comprising a summarylength of just over 10-minutes:1. Teams come out # 10. General Play *2. Goal ? 11. General Play *3. Goal ? 12. Substitution #4. Penalty Offence ? 13. Controversial Foul #5. Goal ? 14. Goal ?6. Irrelevant content ? 15. General Play *7. Goal ? 16. Goal ?8. Goal ? 17. Red Card Offence ?9. Yellow Card Offence ?   18. General Play *For the purposes of evaluation, the twenty clips returned were examined and classified into four categoriesaccording to significance. Ten clips seemed to depict very significant moments of the feature and hence weredescribed as definite highlights (?). The inclusion of definite highlights in the summary is always preferred. Threeof the clips returned seemed to represent moments of arguably lesser significance. These were described by theterm semi-highlights (#), and their inclusion in the summary is desired once all definite highlights already havebeen. The system returned four further clips containing content of considerably less significance, labeledlowlights (*). Inclusion of lowlights would typically not be tolerated except when the combined length of alldefinite and semi-highlight clips fails to satisfy the desired length of the summary.A slight error in the boundary detection results allowed for a minor amount of peripheral content to be includedwithin the summary probing domain. Consequently, the summary contained a clip containing irrelevant content.This result was labeled error (?), and inclusion of clips of this type in the summary is undesired under anycircumstances.The objective was to automatically generate a 10-minute summary of the sports broadcast UEFA Cup Final2001.Pure audio analysis yielded a 10.3 minute long amalgamation of 18 individual clips from the programme.From these, seventeen clips related to the main feature; it was noted that ten corresponded to largely importantmoments of the feature, a further three exhibited content of a more debatable significance, and the remainingfour presented content of a more inconsequential nature.It is important to note that a full qualitative analysis of the video content has yet been performed, therefore itcurrently remains unknown whether or not the programme contains any further, undetected, content which wouldhave deserved the definite- or quasi-highlight label. If so, then a number of the included lowlight clips wouldrepresent false positives. This number is currently being evaluated and thus the inclusion of all the lowlight clipsis not yet indisputably justified. However, concentrating on the amount of true-positives returned, 72% of thesummary is comprised of significant material and viewed as a whole, the amalgamation gives a very coherentsynopsis of the dramatics of the feature.V SUMMARY OF RESULTSThe work reported here is a preliminary investigation into the usefulness of pure audio analysis forsummarisation of (limited types of) sports programmes. A further eight 10-minute summaries were generatedfrom various other broadcast sports programmes.. The content of returned clips, which make up the finalsummary, is listed in Table 1. Again, a full qualitative analysis of the subject content has not yet been performed,so attention should be paid primarily to the number of true-positive returns (definite & semi-highlights).To the best of our knowledge, examination of scalefactor weights represents the most efficient method ofconstructing the audio amplitude profiles from encoded MPEG audio. The example in Section III was performedon a P3-700MHz machine running the Mandrake Linux operating system, and took 2-minutes computing time toretrieve the 18 clip locations directly from the audio bitstream information.Table 1: Classification of clips included inautomatic summariesVI CONCLUSIONS AND FUTURE WORKFrom the preliminary results presented above, we can see that the audio analysis we perform makes asignificant contribution to summarisation of sports TV broadcasts.  We are working to improve the usefulness ofthe sports summary by analyzing the prosody (pitch, amplitude and duration of phonemes) of the commentator’svoice, detecting slow-motion replays, tracking individual players using word-spotting, removing ad breaks anddetecting the goal area in football matches.In a real scenario, automatic summarisation of such broadcasts would depend on some combination of ananalysis of the closed captions (teletext), analysis at the visual level, as well as the analysis we have reportedhere.  In separate work we are working on analysing TV broadcasts in those other domains.REFERENCES[1] Lee H., Smeaton A., O'Toole C., Murphy N., Marlow S. &   O'Connor N., The Físchlár Digital VideoRecording, Analysis, and Browsing System, Proc. Content based  Multimedia Information Access (RIAO 2000),Vol. 2, pp. 1390-1399, Paris, France, 12-14  April 2000.[2] Center of Digital Video Processing/ Físchlár web site: http://www.cdvp.dcu.ie.[3] Intille S. Tracking using a local closed-world assumption: Tracking in the football domain, Proc. SPIE Storageand Retrieval for Image and Video Databases, pp. 216-227, 1997.[4] Kawashima T. et al, Indexing of baseball telecast for content-based video retrieval, Proc. IEEE Int. Conf.Image Processing, pp. 871-875, 1998.[5] Saur D. et al, Automated analysis and annotation of basketball video, Proc. SPIE Storage and Retrieval forImage and Video Databases, pp. 176-187, 1997.[6] Yow D. et al Analysis and presentation of soccer highlights from digital video, Proc. Asian Conf. ComputerVision, 1995[7] Pan, D., A Tutorial on MPEG/Audio Compression, IEEE Multimedia Journal, pp. 60-74, 1995.This material is based upon work supported by the IST programme of the EU in the project IST-2000-32795 SCHEMA. Thesupport of the Informatics Research Initiative of Enterprise Ireland is gratefully acknowledged.Clip ClassificationSportsBroadcast     Total ClipsReturnedDefinite(?)Semi(#)Low(*)Error(?)1.Soccer 18 10 3 4 12.GaelicFootball21 9 11 1 03.Ice Hockey 20 9 6 5 04.GaelicFootball24 12 11 1 05.Rugby 18 7 8 3 06.Soccer 17 9 7 0 17.Rugby 20 8 6 6 08.Soccer 14 3 9 2 09.Field Hockey 22 8 9 5 0

Voice processing for automatic TV sports program highlights detection

In today’s fast paced world, the time available to watch
long sports programmes is decreasing, while the number of sports channels is rapidly increasing. Many viewers desire the facility to watch just the highlights of sports events.
This paper presents a simple, but effective, method for generating sports video highlights summaries. Our method detects semantically important events in sports programmes by using the Scale Factors in the MPEG audio bitstream to generate an audio amplitude profile of the program. The Scale Factors for the subbands corresponding to the voice bandwidth give a strong indication of the level of commentator and/or spectator excitement. When periods of sustained high audio amplitude have been detected and ranked, the corresponding video shots may be concatenated to produce a summary of the program highlights. Our method uses only the Scale Factor information that is directly accessible from the MPEG bitstream, without any decoding, leading to highly efficient computation. It is also rather more generic than many existing techniques, being particularly suitable for the more popular sports televised in Ireland such as soccer, Gaelic football, hurling, rugby, horse racing and motor racing

O\u27Connor, Noel E.

Irish Universities

Audio processing for automatic TV sports program highlights detection

The imminent rapid expansion in the number of TV channels is driving the need for efficient digital video indexing, browsing and playback systems. For the past four years, the Centre for Digital Video Processing in DCU has been working towards the provision of such a system. The current stage of development is demonstrated on our Web-based digital video system called Físchlár [1,2], which is now in hourly use by over 1000 registered users. At present a user can pre-set the recording of TV broadcast programmes and can choose from a set of different browser interfaces which allow navigation through the recorded programmes. As our research has developed we have been adding increased functionality such as personalisation and
programme recommendation, automatic recording, SMS/WAP/PDA alerting, searching, summarising and so on. There have been many approaches to automatic highlight detection and summarization of sports programmes
[3-6]. However, most methods are oriented to particular sports and/or specific edit effects or need much
computational effort. Therefore we decided to take a more basic and generic approach based on the observation
that, in most common sports coverage with a commentator and spectators, exciting events are usually characterized by an increase in the audio amplitude, particularly in the speech band. Measurement of the audio amplitude is described in Section II and a case study is presented in Section III. Section IV summarises results while conclusions and areas of continuing and future work are outlined in Section V

In today’s fast paced world, the time available to watch\ud
long sports programmes is decreasing, while the number of sports channels is rapidly increasing. Many viewers desire the facility to watch just the highlights of sports events.\ud
This paper presents a simple, but effective, method for generating sports video highlights summaries. Our method detects semantically important events in sports programmes by using the Scale Factors in the MPEG audio bitstream to generate an audio amplitude profile of the program. The Scale Factors for the subbands corresponding to the voice bandwidth give a strong indication of the level of commentator and/or spectator excitement. When periods of sustained high audio amplitude have been detected and ranked, the corresponding video shots may be concatenated to produce a summary of the program highlights. Our method uses only the Scale Factor information that is directly accessible from the MPEG bitstream, without any decoding, leading to highly efficient computation. It is also rather more generic than many existing techniques, being particularly suitable for the more popular sports televised in Ireland such as soccer, Gaelic football, hurling, rugby, horse racing and motor racing

_________________________________________________________ISSC 2002, Cork.  June 25-26Audio Processing for Automatic TV Sports ProgramHighlights DetectionSeán Marlow, David A. Sadlier, Noel O’Connor, Noel Murphy.Visual Media Processing Group,Dublin City University,Ireland.email: marlows@eeng.dcu.ie   phone: +353-1- 7005120__________________________________________________________________________________________Abstract -- In today’s fast paced world, the time available to watchlong sports programmes is decreasing, while the number of sportschannels is rapidly increasing. Many viewers desire the facility towatch just the highlights of sports events.This paper presents a simple, but effective, method for generatingsports video highlights summaries. Our method detectssemantically important events in sports programmes by using theScale Factors in the MPEG audio bitstream to generate an audioamplitude profile of the program. The Scale Factors for the sub-bands corresponding to the voice bandwidth give a strongindication of the level of commentator and/or spectator excitement.When periods of sustained high audio amplitude have beendetected and ranked, the corresponding video shots may beconcatenated to produce a summary of the program highlights.Our method uses only the Scale Factor information that is directlyaccessible from the MPEG bitstream, without any decoding, leadingto highly efficient computation. It is also rather more generic thanmany existing techniques, being particularly suitable for the morepopular sports televised in Ireland such as soccer, Gaelic football,hurling, rugby, horse racing and motor racing._________________________________________________________________________________________I INTRODUCTIONThe imminent rapid expansion in the number of TVchannels is driving the need for efficient digitalvideo indexing, browsing and playback systems. Forthe past four years, the Centre for Digital VideoProcessing in DCU has been working towards theprovision of such a system. The current stage ofdevelopment is demonstrated on our Web-baseddigital video system called Físchlár [1,2], which isnow in hourly use by over 1000 registered users. Atpresent a user can pre-set the recording of TVbroadcast programmes and can choose from a set ofdifferent browser interfaces which allow navigationthrough the recorded programmes. As our researchhas developed we have been adding increasedfunctionality such as personalisation and programmerecommendation, automatic recording,SMS/WAP/PDA alerting, searching, summarisingand so on.To initiate the recording of a programme, a userbrowses the TV schedule and selects thoseprogrammes to be recorded - our system will thenautomatically record (digitally) that programme atbroadcast time, much the same as a home VCR.After we record a programme, we then automaticallysegment it using our shot boundary detectiontechnique based on colour histogram comparison, sothat the content becomes easily browsable throughour various user interfaces. The analysed programmeis then added to our archive of recorded programmeswhich a user can scroll through and then select onefor browse/playback. As a user browses through aprogramme he/she can then stream the video to theirdesktop.There have been many approaches to automatichighlight detection and summarization of sportsprogrammes [3-7]. However, most methods areoriented to particular sports and/or specific editeffects or need much computational effort. Thereforewe decided to take a more basic and genericapproach based on the observation that, in mostcommon sports coverage with a commentator andspectators, exciting events are usually characterizedby an increase in the audio amplitude, particularly inthe speech band. Measurement of the audioamplitude is described in Section II and a  case studyis presented in Section III. Section IV summarisesresults while conclusions and areas of continuing andfuture work are outlined in Section V.II AUDIO PROFILE GENERATIONa) MPEG Bitstream ProcessingThe Físchlár system captures television broadcastsand encodes the programmes according to theMPEG-1 digital video standard with the audio signalcoded in line with the Layer-II profile [8]. Unlikemany other audio compression algorithms, whichmake assumptions about the nature of the audiosource, MPEG-1 Audio exploits the perceptualrestrictions of the human auditory system, viapsychoacoustic weighting of the bit allocation foreach frequency subband, to attain its compression[7].The MPEG-1 Layer-II compression algorithmencodes audio signals by dividing the frequencyspectrum of the audio signal, bandlimited to 20kHz,into 32 subbands which approximate the ear’s criticalbands. The subbands are assigned individual bit-allocations according to the audibility of quantisationnoise within each subband. A pyschoacoustic modelof the ear analyses the audio signal and provides thisinformation to the quantiser. Layer-II frames consistof 1152 samples; 3 groups of 12 samples from eachof 32 subbands. A group of 12 samples gets a bit-allocation and, if this is non-zero, a scalefactor.Scalefactors are weights that scale groups of 12samples such that they fully use the range of thequantiser (the encoder uses a different scalefactor foreach of the three groups of 12 samples within eachsubband only if necessary). The scalefactor for sucha group is determined by the next largest value(given in a look-up table) to the maximum of theabsolute values of the 12 samples. Thus it providesan indication of the maximum power exhibited byany one of the 12 samples within the group [9,10].b) Amplitude of the Speech BandMost of the energy in a speech signal lies between0.1kHz – 4kHz.  According to the MPEG-1 Layer-IIaudio standard, the maximum allowable frequencycomponent in the audio signal is at 20kHz. At theencoder, the frequency spectrum (0 – 20kHz) isdivided uniformly into 32 subbands, each havingbandwidth of 0.625kHz [4]. Thus, subbands 2through 7 represent the frequency range from0.625kHz – 4.375kHz. See Figure 1.                   Speech    1      2      3      4      5      6      7      8                      31    320.625kHz 4.375HzFigure 1:  MPEG-1 Layer-II Frequency SubbandsFor sports programme audio tracks, by strictlylimiting the audio examination to these subbands,which approximate the range of the speech band, wefurther concentrate the audio investigation oncommentator vocals. Therefore, the influence of thecommentator on the generation of the audioamplitude profile is increased. This is clearlydesirable since it bolsters the assumption that theprofile will be an accurate indicator of thesignificance of the content.It was expected that the examination of subbands 2though 7 would provide for a reasonable trade-offbetween rejection of low-frequency backgroundnoise (typically present in sports programmes whichwould naturally upset results) and the capture of thefundamental frequency for excited speech.c) Boundary DetectionOne of the problems with the audio amplitudestechnique is caused by the inclusion ofsupplementary content which typically accompaniesthe main event in a sports programme. Features suchas player profiles, highlights of recent events etc.tend to contain attributes such as commentatordialogue and spectator noise, similar to that of themain event. The problem is that these featuresgenerally have audio amplitudes comparable to thatof the event of interest.To combat this problem, the system must be able todetect the temporal boundaries of the main featurewithin the overall sports programme. This is done bysearching through the audio track for extendedperiods of sustained volume.Segments such as interviews, studio discussions,archive video clips, etc. which make up theperipheral content, are flagged by the intermittentoccurrence of brief moments of silence. For example,short silences exist in between sentences spoken byan anchorperson, when switching from anchorpersonto video clips, or between advertisements. Incontrast, the main event in a sports broadcast featuresrelatively long periods of sustained volume due tothe continuous presence of background noise.  Onthis basis it may be automatically distinguished fromthe supplementary content. i.e. the temporalboundaries of the main event within the overallprogramme may be detected. For the summarygeneration, the probing domain is restricted to liewithin these boundaries.III CASE  STUDYa) TaskThe following is an illustration of the automaticgeneration of a 10-minute summary of a terrestrialTV broadcast of a sports event via the discussedtechnique. The experimental subject is the UEFACup Final 2001 featuring Liverpool FC Vs AlavesFC. This was a near 3-hour soccer match broadcast,resulting in a 5-4 victory for Liverpool FC. Theprogramme featured the main event plus studiodiscussions and analysis, player profiles, highlightsof related events and advertisement breaks.b) Amplitude ProfilesA second-by-second audio amplitude profile wasestablished by a superposition of all the scalefactorsfrom subbands 2-7 over a window length of onesecond. See Figure 2.AudioAmplitude                 0    1   2   3   4   5    6   7   8   9         Time (s)Figure 2:  Per-Second Audio Amplitude ProfileA frame-by-frame audio amplitude profile wasestablished by a superposition of all the scalefactorsfrom subbands 2-7 over a window of lengthcorresponding to one video frame (≈1/25s). SeeFigure 3.AudioAmplitudeFrameNumber                 0    1   2   3   4   5    6   7   8   9Figure 3:  Per-Frame Audio Amplitude Profilec) Boundary DetectionThe overall structure of the near 3-hour subject, ascaptured by Físchlár, is described below. In terms ofsummary generation, segments of interest areidentified by an asterisk.Ads ~3 minsInterviews ~14mins* 1st half ~51minsStudio analysis ~14mins* 2nd half ~49minsStudio analysis ~4 mins* Extra time ~26minsStudio analysis ~6 minsA silence threshold was empirically determined asSth = 0.033 * overall mean audio amplitudeUsing the per-frame audio amplitude profile and Sth,the entire audio track of the subject was examinedfor periods of continuous volume lasting, at least 1-minute. It was found that sustained volumesexceeding Sth occur during the following videoframes:309 – 3504 ~2mins3867 – 6608 ~1min6984 – 13577 ~4mins15037 – 19467 ~3mins19553 – 23405 ~2mins* 26248 – 102751 ~51mins106225 – 109935 ~2mins112696 – 115354 ~1min* 123629 – 198950 ~50mins199586 – 201168 ~1min* 201252 – 244086 ~28mins245527 – 247690 ~1minFurther thresholding at a length of 10-minutes rejectsall segments except for three (identified by anasterisk), which correspond almost precisely to thesegments of interest mentioned previously (i.e. thetemporal boundaries of the match play segmentswere accurately detected). Changing units to secondsthese are:• Segment-1: 1050s – 4110s• Segment-2 : 4945s – 7958s• Segment-3: 8050s – 9763sOnly the content which resides within theseboundaries is eligible for inclusion in the summary.Hence, further audio processing is restrictedaccordingly. The boundary detection preamble is nota crucial component of the summarisation procedure,i.e. in the event of failure, the main audio analysisprocedure would still be expected to produce amoderately successful summary. However, it isbeneficial tool which prevents the consideration ofirrelevant material and thus lightens the workload ofsubsequent procedures.d) Summary GenerationThe per-second audio amplitude profiles of segments1-3 (above) were examined. A loudness threshold,Lth, was defined and initalised to the valuecorresponding to the largest peak found. See Figure4.    Segment-1     Segment-2    Segment-3Figure 4:  Examination of Segments 1-3An audio amplitude peak is defined as loud if itexceeds Lth. Ignoring isolated peaks, Lth wasgradually reduced until it began to pick out loudperiods of at least 3-seconds in duration (audiosurges). See Figure 5.Figure 5:  Decreasing Lth and detecting audio surgesFigure 5 shows three sections which extend beyondthe current value of Lth. The second and third havetime spans of 4 seconds and 3 seconds respectively.Thus both are recognised as audio surges. The firstsection is ignored since with a length of 2 seconds, itdoes not meet the minimum surge threshold of 3-seconds. Lth was further reduced until the amount ofdetected surges was sufficient such that a 10-minutevideo summary could be produced. The summarywas then generated by first matching up the videoclips within the combined audio/video track whichtemporally align with the audio surges. Then, a pre-clip buffer of 1 shot and a post clip buffer of 2 shotswas appended (to make viewing the amalgamationless visually disturbing). Finally these clips wereextracted from the audio/video stream and(chronologically) concatenated to generate thehighlights summary. See Figure 6.Figure 6:  Summary generationIV RESULTSThe analysis returned 18 individual clipscorresponding to the following descriptions,comprising a summary length of just over 10-minutes:1. Teams come out # 10. General Play *2. Goal ♥ 11. General Play *LthTime3. Goal ♥ 12. Substitution #4. Penalty Offence ♥ 13. Controversial Foul #5. Goal ♥ 14. Goal ♥6. Irrelevant content ∆ 15. General Play *7. Goal ♥ 16. Goal ♥8. Goal ♥ 17. Red Card Offence ♥9. Yellow Card Offence ♥   18. General Play *For the purposes of evaluation, the twenty clipsreturned were examined and classified into fourcategories according to significance. Ten clipsseemed to depict very significant moments of thefeature and hence were described as definitehighlights (♥). The inclusion of definite highlights inthe summary is always preferred.Three of the clips returned seemed to representmoments of arguably lesser significance. These weredescribed by the term quasi-highlights (#), and theirinclusion in the summary is desired once all definitehighlights already have been. The system returnedfour further clips containing content of considerablyless significance, labeled lowlights (*). Inclusion oflowlights would typically not be tolerated exceptwhen the combined length of all definite and quasi-highlight clips fails to satisfy the desired length ofthe summary.A slight error in the boundary detection resultsallowed for a minor amount of peripheral content tobe included within the summary probing domain.Consequently, the summary contained a clipcontaining irrelevant content. This result was labelederror (∆), and inclusion of clips of this type in thesummary is undesired under any circumstances.The objective was to automatically generate a 10-minute summary of the sports broadcast UEFA CupFinal 2001.Pure audio analysis yielded a 10.3 minutelong amalgamation of 18 individual clips from theprogramme. From these, seventeen clips related tothe main feature; it was noted that ten correspondedto largely important moments of the feature, a furtherthree exhibited content of a more debatablesignificance, and the remaining four presentedcontent of a more inconsequential nature.It is important to note that a full qualitative analysisof the video content has yet been performed,therefore it currently remains unknown whether ornot the programme contains any further, undetected,content which would have deserved the definite- orquasi-highlight label. If so, then a number of theincluded lowlight clips would represent falsepositives. This number is currently being evaluatedand thus the inclusion of all the lowlight clips is notyet indisputably justified. However, concentrating onthe amount of true-positives returned, 72% of thesummary is comprised of significant material andviewed as a whole, the amalgamation gives a verycoherent synopsis of the dramatics of the feature.V SUMMARY OF RESULTSThe work reported here is a preliminary investigationinto the usefulness of pure audio analysis forsummarisation of (limited types of) sportsprogrammes. A further eight 10-minute summarieswere generated from various other broadcast sportsprogrammes.. The content of returned clips, whichmake up the final summary, is listed in Table 1.Again, a full qualitative analysis of the subjectcontent has not yet been performed, so attentionshould be paid primarily to the number of true-positive returns (definite & quasi-highlights).To the best of our knowledge, examination ofscalefactor weights represents the most efficientmethod of constructing the audio amplitude profilesfrom encoded MPEG audio. The example in SectionIII was performed on a P3-700MHz machine runningthe Mandrake Linux operating system, and took 2-minutes computing time to retrieve the 18 cliplocations directly from the audio bitstreaminformation.Table 1: Classification of clips included in automaticsummariesVI CONCLUSIONS AND FUTURE WORKFrom the preliminary results presented above, wecan see that the audio analysis we perform makes asignificant contribution to summarisation of sportsTV broadcasts.  We are working to improve theusefulness of the sports summary by analyzing theprosody (pitch, duration of phonemes and amplitude)of the commentator’s voice, detecting slow-motionreplays, tracking individual players using word-Clip ClassificationSports Broadcast Total ClipsReturnedDefinite(♥)Quasi-(#)Low(*)Error(∆)1.Soccer 18 10 3 4 12.GaelicFootball 21 9 11 1 03.Ice Hockey 20 9 6 5 04.GaelicFootball 24 12 11 1 05.Rugby 18 7 8 3 06.Soccer 17 9 7 0 17.Rugby 20 8 6 6 08.Soccer 14 3 9 2 09.Field Hockey 22 8 9 5 0spotting, removing ad breaks [9,10] and detecting thegoal area in football matches.In a real scenario, automatic summarisation of suchbroadcasts would depend on some combination of ananalysis of the closed captions (teletext), analysis atthe visual level, as well as the analysis we havereported here.  In separate work we are working onanalysing TV broadcasts in those other domains.REFERENCES[1] Lee H., Smeaton A., O'Toole C., Murphy N., MarlowS. &   O'Connor N., The Físchlár Digital Video Recording,Analysis, and Browsing System, Proc. Content basedMultimedia Information Access (RIAO 2000), Vol. 2, pp.1390-1399, Paris, France, 12-14  April 2000.[2] Center of Digital Video Processing/ Físchlár website: http://www.cdvp.dcu.ie.[3] Intille S. Tracking using a local closed-worldassumption: Tracking in the football domain, Proc. SPIEStorage and Retrieval for Image and Video Databases, pp.216-227, 1997.[4] Kawashima T. et al, Indexing of baseball telecast forcontent-based video retrieval, Proc. IEEE Int. Conf. ImageProcessing, pp. 871-875, 1998.[5] Saur D. et al, Automated analysis and annotation ofbasketball video, Proc. SPIE Storage and Retrieval forImage and Video Databases, pp. 176-187, 1997.[6] Yow D. et al Analysis and presentation of soccerhighlights from digital video, Proc. Asian Conf. ComputerVision, 1995[7] Pan, D., A Tutorial on MPEG/Audio Compression,IEEE  Multimedia Journal, pp. 60-74, 1995.[8] Rao, K.R. & Hwang, J.J., Techniques & Standards forImage, Video and Audio Coding. Prentice Hall, 1996.

The imminent rapid expansion in the number of TV channels is driving the need for efficient digital video indexing, browsing and playback systems. For the past four years, the Centre for Digital Video Processing in DCU has been working towards the provision of such a system. The current stage of development is demonstrated on our Web-based digital video system called Físchlár [1,2], which is now in hourly use by over 1000 registered users. At present a user can pre-set the recording of TV broadcast programmes and can choose from a set of different browser interfaces which allow navigation through the recorded programmes. As our research has developed we have been adding increased functionality such as personalisation and

programme recommendation, automatic recording, SMS/WAP/PDA alerting, searching, summarising and so on. There have been many approaches to automatic highlight detection and summarization of sports programmes

[3-6]. However, most methods are oriented to particular sports and/or specific edit effects or need much

computational effort. Therefore we decided to take a more basic and generic approach based on the observation

that, in most common sports coverage with a commentator and spectators, exciting events are usually characterized by an increase in the audio amplitude, particularly in the speech band. Measurement of the audio amplitude is described in Section II and a case study is presented in Section III. Section IV summarises results while conclusions and areas of continuing and future work are outlined in Section V

Audio processing for automatic TV sports program highlights detection

Abstract

Similar works

Full text

Available Versions

Name not available

Irish Universities

Irish Universities

Name not available

DCU Online Research Access Service

DCU Online Research Access Service