Enumeration of coalescent histories for caterpillar species trees and pp-pseudocaterpillar gene trees

For a fixed set $X$ containing $n$ taxon labels, an ordered pair consisting of a gene tree topology $G$ and a species tree $S$ bijectively labeled with the labels of $X$ possesses a set of coalescent histories -- mappings from the set of internal nodes of $G$ to the set of edges of $S$ describing possible lists of edges in $S$ on which the coalescences in $G$ take place. Enumerations of coalescent histories for gene trees and species trees have produced suggestive results regarding the pairs $(G,S)$ that, for a fixed $n$, have the largest number of coalescent histories. We define a class of 2-cherry binary tree topologies that we term $p$-pseudocaterpillars, examining coalescent histories for non-matching pairs $(G,S)$, in the case in which $S$ has a caterpillar shape and $G$ has a $p$-pseudocaterpillar shape. Using a construction that associates coalescent histories for $(G,S)$ with a class of "roadblocked" monotonic paths, we identify the $p$-pseudocaterpillar labeled gene tree topology that, for a fixed caterpillar labeled species tree topology, gives rise to the largest number of coalescent histories. The shape that maximizes the number of coalescent histories places the "second" cherry of the $p$-pseudocaterpillar equidistantly from the root of the "first" cherry and from the tree root. A symmetry in the numbers of coalescent histories for $p$-pseudocaterpillar gene trees and caterpillar species trees is seen to exist around the maximizing value of the parameter $p$. The results provide insight into the factors that influence the number of coalescent histories possible for a given gene tree and species tree

Asymptotic properties of the number of matching coalescent histories for caterpillar-like families of species trees

Coalescent histories provide lists of species tree branches on which gene tree coalescences can take place, and their enumerative properties assist in understanding the computational complexity of calculations central in the study of gene trees and species trees. Here, we solve an enumerative problem left open by Rosenberg (IEEE/ACM Transactions on Computational Biology and Bioinformatics 10: 1253-1262, 2013) concerning the number of coalescent histories for gene trees and species trees with a matching labeled topology that belongs to a generic caterpillar-like family. By bringing a generating function approach to the study of coalescent histories, we prove that for any caterpillar-like family with seed tree t , the sequence h_n describing the number of matching coalescent histories of the n th tree of the family grows asymptotically as a constant multiple of the Catalan numbers. Thus, h_n ∼ β(t) c_n, where the asymptotic constant β(t) > 0 depends on the shape of the seed tree t. The result extends a claim demonstrated only for seed trees with at most eight taxa to arbitrary seed trees, expanding the set of cases for which detailed enumerative properties of coalescent histories can be determined. We introduce a procedure that computes from t the constant β(t) as well as the algebraic expression for the generating function of the sequence h_n