Goodwin, Michael

Prandoni, Paolo

Vetterli, Martin

Infoscience - École polytechnique fédérale de Lausanne

Sound Representation and Modification With MultiresolutionSinusoidal ModelsMichael Goodwin*, Paolo Prandonit, and Martin Vetterlil*Silicon Graphics, inc., hlountain View, CA and t EPFL, Lausanne, SwitzerlandAbstract: In the standard sinusoidal model, the analysis and synthesis are typicdy carried out at fixed frame rates.This can result in representation error for signals with dynamic behavior. To improve the modehng performance fornonstationary signals, the sinusoidd analysis-synthesis can be carried out in a multiresolution framework based onadaptive time segmentation; the adaptation is guided by a metric such as ratedistortion. This approach improvesthe model accuracy while preserving the ability to carry out the rich class of modifications Worded by the standardsinusoidd model.SINUSOIDAL MODELINGIn sinusoidd modeling, a signal is represented M a sum of evolving sinusoidd partids. Analysis involves findingframe-by-frame estimates of the amplitude, frequency, and ph=e of the constituent partids [1, 2, 3]. In the synthesisstage, these parameters are connected from frame to frame by a fine tracking process and then interpolated to derivesample-rate control functions for a bank of osciHators; the interpolation is carried out based on underlying synthesisframes, which are estabhshed by the analysis stride. Typicdy, the frames are of fixed size as depicted in Fig. l(a).~Ie such models have proven useful for speech and audio processing, using a fixed frame size has various drawbackswhich motivate the use of adaptive frame sizes such as those shown in Fig. l(b); these drawbacks are discussed below.The resolution of the sinusoidd model is fimited by the choice of the analysis frame size and stride. For longframes, the time resolution is inadequate for capturing signal dynamics such as attack transients. For short frames,the frequency resolution is degraded such that estimation of sinusoidd components becomes difficult. Furthermore,the original signal may not behave in the manner assumed by the line tracking and parameter interpolation used inthe synthesis. FinMy, the sum-of-pmtids model simply has difficulty representing broadband noisebke processes suchas breath noise. As a result of these various limitations, the sinusoid~ analysis-synthesis process yields a nonzeroresidual. Noise-based models of the residud have been developed, but such approaches are primarily useful formodefing the aforementioned noiselike processes [2, 4]; while these methods do improve the synthesis reafism, theyare hmited in that noise-based models are not effective for representing coherent featwes. For noise-b=ed residu~models to be effective, then, it is necessary to modify the sinusoidd model so as to reduce coherent artifacts, forinstance pre-echoin the reconstruction of signal onsets. Pre-echo, a well-known diffictity in audio coding, is of interesthere since high-qu~ty music synthesis requires preservation of note attacks [2, 3].Pre-echo is introduced in the sinusoidd model in the follow~ing way. Before a signal onset, there is an analysisframe in which the signal is not present and no partids are found. Thereafter, partids are identified in the framein which the onset occurs; the hne tracking interprets these as new partids and connects them via interpolation tozero-amptitude partids in the previous frame [1]. This results in a smooth amphtude envelope for eaeh partial insteadof a sharp onset. The pre-echo caused by hnear amplitude interpolation is shown in Fig. 1 for two simple signals,and in Fig. 2 for a saxophone note.The pre-echo problem in the sinusoidd model can be deviated by casting the modehng into a multiresolutionframework [3, 5, 6]. This paper is concerned with muftiresolution based on time-varying segmentation, in whichshort frames are used near transients, which improves time loc~zation, and long frames are used for regions withstat ionary behavior, which improves frequency resolution and dOWS for Codng gtin. such adaptivity is of greatimportance in signal modebng [3].AnalysiswindowsSynthesisframesAnalysiswindowsSynthesisframesI 1 t I I I 1 1 1 1(a) Fixed segmentation} 1 # 1 I 1 1 i 1 r I(b) Adaptive segmentation1 (c) ,“- \--- --,,0 “.-t t .0 w lW lW m1‘ (91o-i 20 m lW lm m‘ h) io-1 10 m lm 1= mTime (samples) Time (samples)Figure 1: Analysis windows and synthesis frames in sinusoidd models with (a) fixed and (b) adaptive resolution.Using a fixed frame size introduces pre-echo; modehng the onsets of (c) a sinusoid and (d) a simple harmonic signalusing the analysis windows depict ed leads to (e,f ) deloc~zed reconstruction and (g ,h) residuds with artifacts.81oCeH boundary markers/l\1 2 3 4 5. D451,0Time (samples)Figure 2: Depiction of a dynamic segmentation algorithm and a modefing example: (a) the onset of a saxophonenote, (b) a fixed resolution reconstruction and (c) ‘the corresponding residud, where the dashed lines indicate thesynthesis frames; and, (d) a dynamic multiresolution model and (e) its residud, which exhibits fewer artifacts.ADAPTIVE SEGMENTATIONIn the sinusoidd model, the analysis windows do not have to satisfy an overlapadd property as in the perfectreconstruction STFT [3]. Thus, time-varying windows such as those depicted in Fig. l(b) can be readily used. Suchwindows correspond to a time-varying synthesis segmentation; the god, then, is to derive such a segmentation whichoptimizes some metric such as the mean-squared reconstruction error. This optimization can be carried out by anexhaustive global search in which each possible segmentation is considered in turn. If the optimization metric isadditive and independent on disjoint segments, however, an exhaustive evacuation involves redundant computation.This redundancy can be removed by formdating the computation as a dynamic program. This approach is based ontreating the time span of the signal as a concatenation of cells. The boundaries between these ce~s, which win becalled ma rke rg, serve as nodes in the dynamic program; dowable segment lengths correspond to integer multiples ofthe ceH size [7].The operation of a dynamic segmentation dgonthm is depicted in Fig. 2; the expression Dab represents themetric associated with the signal model on the segment between markers a and b. At each marker, the dgonthmcomputes and records the minimum modebng metric to reach that marker; it dso records the length of the lastsegment in the corresponding segmentation, which is the optimal segmentation up to that point in the signal, andthe sinusoidd parameters computed for that particdar segment using an analysis window of a corresponding scale(see Fig. l(b)). When the end of the signaf is reached, the optimal segmentation can be recovered by backtrackingthrough the recorded lengths. The computation at a given marker thus amounts to evaluating the modebng metricon each segment that leads to that marker; this is done by analyzing the signal with a window based on the segmentlength, synthesizing the model by tracking and interpolating the analysis data back to the data recorded for themarker at the start of the segment, and computing the reconstruction error; the tracking is simplified by constrainingthe maximum number of partids to be the same in short segments as in long segments. Fig. 2 depicts a fixed modelof a saxophone onset and an improved model derived by a dynamic segmentation dgori thm. Additiond details onsignal-adaptive segmentation for sinusoidal modeling can be found in [3, 6]. Such models Wow the same extensivemodification capabihties as the standard sinusoidd model while providng improved accuracy; the analysis-synthesisresidu~ for such multiresolution models can be effectively described using noise-based methods.[1][2][3][4][5][6][7]REFERENCESR. J. McAtiay and T. F. Quatieri. Speech analysis/synthesis based on a sinusoidd representation. IEEE Trans-acting on Acougtics, Speech, and Signal Proce99ing, 34(4):744 – 754, August 1986,X. Serra and J. Smith. Spectral modeling synthesis: A sound analysis/synthesis system based on a deterministicplus stochastic decomposition. Computer Mu9ic Journal, 14(4):12-24, Winter 1990.M. Goodwin. Adaptive Signal Mode19: Theory, Algorithm, and .4udio Application. PhD thesis, Berkeley, 1997.M. Goodwin. Residud modehng in music analysis-synthesis. ICASSP- 1996,2:1005-1008.S. Levine, T. Verma, and J. Smith. Ahas-free, multiresolution sinusoidd modehng for polyphonic wideband audio.IEEE Work9hop on Application of Signal Proce99ing to Audio and Acougticg, October 1997.M. Goodwin. Multiresolution sinusoidd modehng using adaptive segmentation. ICASSP-1998.P. Prandoni, M. Goodwin, and M. Vetterh. Optimal segmentation for signal modehng and compression. ICASSP-1997, 3:202%2032.82

Sound representation and modification with multiresolution, sinusoidal models

Abstract

Similar works

Full text

Available Versions

Infoscience - École polytechnique fédérale de Lausanne