We present methods used in our submission to the Sequence Prediction ChallengE (SPiCe’16) 1 .

The two methods used to solve the competition tasks were spectral learning and a count

based method. Spectral learning led to better results on most of the problems

Grzes, Marek

Liza, Farhana Ferdousi

Kent Academic Repository

A Spectral Method that Worked Well in the SPiCe’16CompetitionFarhana Ferdousi Liza fl207@kent.ac.ukMarek Grzes´ m.grzes@kent.ac.ukSchool of ComputingUniversity of KentCanterbury, Kent, CT2 7NY, UKEditor:AbstractWe present methods used in our submission to the Sequence Prediction ChallengE (SPiCe’16)1.The two methods used to solve the competition tasks were spectral learning and a countbased method. Spectral learning led to better results on most of the problems.Keywords: Spectral Learning, Rank, Sequence Prediction, Hyperparameters1. Description of the SPiCe 2016 CompetitionThe Sequence PredictIction ChallengE (SPiCe) was an on-line competition about guessingthe next element in a sequence. Training datasets consist of whole sequences and the aim isto learn a model that allows the ranking of potential next symbols for a given test sequence(prefix), that is, the most likely options for a single next symbol. The evaluation processwas interactive: on submission of an answer for a prefix, another prefix was presented andso on. Once rankings for all prefixes were submitted then the score (NDCG5 explainedbelow) of the submission was computed. The score is a ranking metric based on normaliseddiscounted cumulative gain computed from the ranking of 5 potential next symbols startingfrom the most probable next symbol. Suppose the test set is made of prefixes y1, . . . , yMand the distinct next symbols ranking submitted for yi is (aˆi1, . . . , aˆi5) sorted from morelikely to less likely. The target probability distribution of possible next symbols given theprefix yi, (p(.|yi)), was known to the organisers. Thus, the exact measure for prefix yi couldbe computed using the following equation:NDCG5(aˆi1, . . . , aˆi5) =∑5k=1p(aˆik|yi)log2(k+1)∑5k=1pklog2(k+1)where p1 ≥ p2 ≥ . . . ≥ p5 are the top 5 values in the distribution p(.|yi).The competition used real-world data from different fields (Natural Language Process-ing, Biology, Signal Processing, Software Verification, etc.), and some synthetic data. Thedata description was not available during the competition.The distributions p(.|yi) were computed differently depending on whether the data issynthetic (and the model that generated it is available) or real. For synthetic data, the true1. http://spice.lif.univ-mrs.fr/data.php1conditional distribution over the next symbol was used. For real data, where the string yiis obtained as a prefix of a longer string yiax, the conditional distribution was computedas follows p(a′|y) = δa=a′ . Note that in this case, p1 = 1 and p2 = ... = p5 = 0. Thus,when applying this metric to real data, NDCG5(aˆi1, . . . , aˆi5) =1log2(j+1), where j is suchthat aˆj = a (and j = ∞ if a is not in the list of predicted next symbols). The score of asubmission was the sum of the scores on each prefix in the test sample, normalised by thenumber of prefixes in the test set.2. Our ApproachSpectral learning algorithms (Balle et al., 2014; Hsu et al., 2012) are promising thanks totheir theoretical guarantees of consistency. Another advantage is that they require relativelylittle parameter tuning, where the main parameter is the number of hidden states, n. Ourmain approach was the method proposed by Balle et al. (2014), and we built our solutionsusing its implementation described in Denis et al. (2016). We also used an n-gram modelwith smoothing to compare results on different datasets, and in several cases, the n-grammethod was better than spectral learning.3. Details of the Spectral MethodThe approach that gave us the best score on most problems is the algorithm proposed by(Balle et al., 2014). The algorithm follows from a duality result between minimal WeightedFinite Automata (WFA) and factorisation of Hankel matrices (Hf ). Weighted Finite Au-tomata are classical finite automata and Hankel Matrix of a function is a bi-infinite matrix.The method provides an efficient algorithm that implements the ideas of the followingLemma to find a rank factorisation of a complete sub-block H of Hf and obtains from it aminimal WFA for a function f which maps strings to real numbers.Lemma 1 If H = PS is a rank factorization, then the WFA A = 〈α1, α∞, {Aσ}〉 withα1⊤ = h⊤λ,SS+, α∞ = P+hP,λ, and Aσ = P+HσS+, is minimal for f .3.1 Description of the AlgorithmThe purpose of the algorithm is to compute a minimal WFA for a function (f) defined onstrings f : Σ∗ → R with finite rank n. Here, Σ be a finite alphabet, where σ is an arbitrarysymbol in Σ, and the set of finite strings over Σ is denoted by Σ∗. The algorithm assumesthat the B = (P, S) is a complete basis for f , and given that basis along with values of f (inour case the values are frequency of substrings) on a set of strings W , where W = {λ,Σ},as input, the algorithm does a rank factorisation of a sub-block of the Hankel matrix tobe able to apply the formulas given in Lemma 1. Particularly, the algorithm assumes thatP Σ′S ∪ P ∪ S ⊆ W . Consequently, the values of f on a set of strings can compute thevectors hλ,S and hP,λ, where λ is an empty string. Thus, the algorithm only requires tocompute the rank factorisation of Hλ to be able to apply the formulas given in Lemma 1.The compact SVD for a matrix gives rank factorisation. The SVD of a p× s matrix Hλof ranks n can be written as Hλ = ULV⊤, where U ∈ Rp×n and V ∈ Rs×n are orthogonalmatrices, and L ∈ Rn×n is a diagonal matrix containing the singular values of Hλ. In2the algorithm, this formulation has been written differently as follows Hλ = (HλV )V⊤.This can be written because V ⊤V = I and V + = V ⊤, when V is orthgonal. With thisfactorization, the equations in Lemma 1 can be written as α1⊤ = h⊤λ,SV , α∞ = (HλV )+hp,λ,and Aσ = (HλV )+HσV and these are used in the learning phase of the algorithm. One ofthe applications of this algorithm is to determine the probabilities of sequences as follows:P (x1:T ) = (α1)⊤Ax1 . . . Axtα∞.In practice, the H and Hσ are not known exactly because the number of finite stringscan be infinite and as a result there will be some estimation error. However, it is possibleto apply the algorithm on approximate Hˆ and Hˆσ. Here, approximation means the Hˆ andHˆσ can be calculated from a number of strings in W rather than from all strings of W .3.2 Relevant ParametersWhen this algorithm is used for practical purposes with approximate Hˆλ and Hˆσ, for σ ∈Σ, hˆλ,S , and hˆP,λ, the algorithm receives as input the number of states n of the target WFA.This is due to the fact that noise can make the rank of Hˆσ different from the rank of the Hσand the algorithm has to ignore some small singular values of Hˆσ, which correspond to zerosin the original matrix. This is done by computing a truncated SVD of Hˆλ up to dimensionn. As the zeros in the original matrix do not have much impact on the final result, Bailly(2011) has shown that when empirical Hankel matrices are sufficiently accurate, singularvalues of Hˆ can yield accurate estimates of the number of states n in the target. Usuallythe number of states is chosen based on some sort of cross-validation procedure.The other important parameter to choose when using the algorithm is the basis. Ingeneral there are infinitely many complete basis and for a function of rank n there alwaysexist complete basis with |P | = |S| = n. In practice, choosing a basis can be done inthe form P = S = Σ≤k for some k > 0 (Hsu et al., 2012; Siddiqi et al., 2009). Anotherapproach is to choose a basis that contains the most frequent elements observed in thesample, which can be either strings, prefixes, suffixes, or substrings (Balle et al., 2012).We used the second approach in our method with substrings. Note that the basis vectorcan be chosen from a subblock of the Hankel matrix where the rows and columns of theHankel matrix correspond to the substrings and the cells of the Hankel matrix contain thefrequencies of the corresponding substrings. In the algorithm, we also choose the length ofthese substrings. Instead of using substrings, it is also possible to use the prefixes as a rowand suffixes as a column. In such a case, the cell of the Hankel matrix can be calculatedas the frequencies of the corresponding strings. If the data is informative enough and thefrequencies are high enough, the Hankel matrix gives a complete basis without the costlyneed to look at all possible rows and columns.3.3 Our Approach to Parameter TuningHaving the code written by Denis et al. (2016), our main task was to tune the parametersof the algorithm. Specifically, we had to select values for three parameters: the number ofstates, n, the maximum length of the substring considered in the row of the Hankel matrix,nR, and the maximum length of the substring considered in the column of the Hankelmatrix, nC. The objective was to maximise the score (NDCG5) on the problems includedin the competition.3We did not apply any of the more sophisticated methods for hyperparameter tuning—such as random search (Bergstra and Bengio, 2012), Bayesian optimisation (Bergstra et al.,2011) or grid-based search (Larochelle et al., 2007)—because the method quickly becomesinfeasible when the rank, i.e., the number of states is too high, and we had to stop manyexperiments manually. Similar to Larochelle et al. (2007), our method included a combi-nation of multi-resolution search, coordinate ascent, and manual search, with a significantutilisation of the last method.On all problems, our method first initialises nR and nC to 4 and n to 5. Note that thenumber of rows (columns) in the Hankel matrix is much larger than nR (nC). In the secondstep, the algorithm starts the process of tuning the number of states n because this wasthe most important parameter in our preliminary experiments. Random walk is used toselect new values of n with the step size being depended on the size of the domain, i.e., thenumber of observations and the number of sequences. Thus, when nR and nC were keptconstant, the value of n was increased or decreased randomly based on the score (NDCG5),i.e., a form of coordinate ascent was performed on n. After the highest score was achievedby tuning n, n was frozen, and the algorithm used the same randomised procedure to tunenR (see Figure 3). Finally, the same procedure was executed to tune the parameter nC(see Figure 6). After tuning nR, and nC, we did not tune n again because, for a given n, avery small improvement was usually observed after tuning nR and nC. On some problems,increasing the values of n, nR and nC to a large number was not possible as the algorithmwas becoming intractable.4. An Alternative Approach: N-gram with SmoothingWe also used 3-gram with Kneser-Ney smoothing (Chen and Goodman, 1996) for the com-petition datasets.5. Experimental resultsThe spectral algorithm led to good scores on problems 1, 2, 3, 9, and 12 (see Figure 1). Acommon characteristic of all these problems is that they have small numbers of hidden states(low rank). The scores of the predictions made using spectral learning on problems 4, 5, 7,8, 10, 11, and 13 were not improving with the change of the value of n (see Figure 2), andincreasing the size of the basis vector was making the algorithm intractable. On problems5, 8, and 10, n-gram with smoothing gave slightly better results than the correspondingbest spectral solution (see Figure 5). On problem 6, increasing the rank improved the scorewhereas for problem 11, increasing the rank made the algorithm intractable. On problem14 and 15, higher rank did not work at all (see Figure 4). The final parameters for theproblems can be found in the accompanying website2.6. ConclusionThe methods described in this paper placed us in the 4th position with score 9.4082437158whereas the winning score was 10.4498481750.2. https://www.cs.kent.ac.uk/people/staff/mg483/SPiCe2016/4AcknowledgmentsWe thank the organising committee for their effort to set up the competition. We also thankRe´mi Eyraud for answering our questions related to their software package.ReferencesRaphael Bailly. Quadratic weighted automata: Spectral algorithm and likelihood maxi-mization. Journal of Machine Learning Research, 20:147–162, 2011.Borja Balle, Ariadna Quattoni, and Xavier Carreras. Local loss optimization in operatormodels: A new insight into spectral learning. In ICML. icml.cc / Omnipress, 2012. URLhttp://dblp.uni-trier.de/db/conf/icml/icml2012.html#BalleQC12.Borja Balle, Xavier Carreras, Franco M Luque, and Ariadna Quattoni. Spectral learning ofweighted automata. Machine Learning, 96(1-2):33–63, 2014.James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimiza-tion. J. Mach. Learn. Res., 13:281–305, February 2012. ISSN 1532-4435. URLhttp://dl.acm.org/citation.cfm?id=2188385.2188395.James S. Bergstra, Re´mi Bardenet, Yoshua Bengio, and Bala´zs Ke´gl. Algorithms for hyper-parameter optimization. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, andK. Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages2546–2554. Curran Associates, Inc., 2011.Stanley F. Chen and Joshua Goodman. An empirical study of smoothing techniquesfor language modeling. In Proceedings of the 34th Annual Meeting on Associa-tion for Computational Linguistics, ACL ’96, pages 310–318, Stroudsburg, PA, USA,1996. Association for Computational Linguistics. doi: 10.3115/981863.981904. URLhttp://dx.doi.org/10.3115/981863.981904.Arrivault Denis, Benielli Dominique, Denis Fracois, and Eyraud Remi. Sp2learn: A toolboxfor the spectral learning of weighted automata. The 13th International Conference onGI, 2016.Daniel Hsu, Sham M Kakade, and Tong Zhang. A spectral algorithm for learning hiddenmarkov models. Journal of Computer and System Sciences, 78(5):1460–1480, 2012.Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio.An empirical evaluation of deep architectures on problems with many factors of variation.In Proc. of ICML, pages 473–480. ACM, 2007.Sajid M. Siddiqi, Byron Boots, and Geoffrey J. Gordon. Reduced-rank hidden markovmodels. CoRR, abs/0910.0902, 2009. URL http://arxiv.org/abs/0910.0902.5Appendix ANo of States0 50 100 150 200Score0.550.60.650.70.750.80.850.90.95No of states vs Score (Small number of states)Dataset 1Dataset 2Dataset 9Dataset 3Dataset 12Figure 1: Good Score (SM)No of States0 200 400 600 800 1000Score0.250.30.350.40.450.50.550.6No of states vs ScoreDataset 4Dataset 5Dataset 7Dataset 8Dataset 10Dataset 13Figure 2: Low Score (SM)nC=nR3 4 5 6 7 8 9Score0.760.780.80.820.840.860.880.90.920.94Substring length vs ScoreDataset 9 (Rank = 44)Dataset 9 (Rank = 44)Dataset 6 (Rank = 250)Figure 3: Impact of basis on scoreNo of States0 100 200 300 400 500Score0.20.30.40.50.60.70.80.9No of states vs ScoreDataset 6 (higher rank works)Dataset 11(higher rank intractable)Dataset 14 (higher rank does not work)Dataset 15 (higher rank does not work)Figure 4: No of states vs ScoreProblem no4 5 6 7 8 9 10Score0.350.40.450.50.550.60.65spectral vs n-gramSpectral3-gram with KN smoothingFigure 5: SM vs n-gramnC2 3 4 5 6 7 8 9 10Score0.8210.8220.8230.8240.8250.8260.8270.8280.829Constant nR and Variable nC vs ScoreDataset 3 (Rank=5, nR=nC)Dataset 3 (Rank=5,nR=10,nC=Variable)Figure 6: Same and different nR, nC6

A Spectral Method that Worked Well in the SPiCe'16 Competition

http://kar.kent.ac.uk/57327/1/liza16spice.pdf

A Spectral Method that Worked Well in the SPiCe'16 Competition

Abstract

Similar works

Full text

Available Versions

Kent Academic Repository