F_ST vs. genetic similarity in various population pairs.

<p>Pairwise distances are colored red or blue within populations and black between populations. (A) Even at a relatively low F_ST of 0.02 all within-population pairs among the Uygur and Adygei samples are genetically more similar than all the between-population pairs. (B) Separation is more ambiguous among Native Americans. Despite a relatively high F_ST of 0.09, there is substantial overlap between Maya-Maya (red) and Maya-Surui (black) samples. E_ST values are more consistent with the within- vs.-between population overlap and the dissimilarity fraction (ω).</p

Scatter plots indicating a positive correlation for <i>SD</i>_T and <i>F</i>_ST.

<p>Each dot represents the two statistics computed for data sampled from our population model with 1000 SNPs and allele frequencies from Beta distributions. The Pearson product-moment correlation coefficient of F_ST and SD_T is 0.67 for plot (A) with panmictic populations, 0.38 for plot (B) with slightly varying structure in subpopulations, 0.14 for plot (C) with highly varying structure in subpopulations, and for 0.94 for plot (D) with three panmictic populations.</p

On the Apportionment of Population Structure

<div><p>Measures of population differentiation, such as F_ST, are traditionally derived from the partition of diversity within and between populations. However, the emergence of population clusters from multilocus analysis is a function of genetic <i>structure</i> (departures from panmixia) rather than of diversity. If the populations are close to panmixia, slight differences between the mean pairwise distance within and between populations (low F_ST) can manifest as strong separation between the populations, thus population clusters are often evident even when the vast majority of diversity is partitioned within populations rather than between them. For any given F_ST value, clusters can be tighter (more panmictic) or looser (more stratified), and in this respect higher F_ST does not always imply stronger differentiation. In this study we propose a measure for the partition of structure, denoted E_ST, which is more consistent with results from clustering schemes. Crucially, our measure is based on a statistic of the data that is a good measure of internal structure, mimicking the information extracted by unsupervised clustering or dimensionality reduction schemes. To assess the utility of our metric, we ranked various human (HGDP) population pairs based on F_ST and E_ST and found substantial differences in ranking order. E_ST ranking seems more consistent with population clustering and classification and possibly with geographic distance between populations. Thus, E_ST may at times outperform F_ST in identifying evolutionary significant differentiation.</p></div

A simulation of <i>SD</i>_T and <i>SD</i>_S under a two panmictic population model demonstrating the divergent behavior of these two statistics with an increasing number of SNP loci.

<p>SNP frequencies are modeled on <i>Beta</i> distributions (as in [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0160413#pone.0160413.ref005" target="_blank">5</a>]). (A) with <i>F</i>_ST = 0.10. (B) with <i>F</i>_ST = 0.03.</p

PCA plots from simulated SNP data of four populations (80 samples each) demonstrating the much pronounced decrease in <i>SD</i>_S for panmictic populations (red and green) relative to structured ones (black and blue), for two different patterns of internal structure, as the number of SNP loci processed by the PCA scheme is increased.

<p>(A-B) from 1K SNPs to 8K SNPs, where structure results from some random linkage disequilibrium (LD) pattern. (C-D) from 1K SNPs to 8K SNPs, where structure results from an LD pattern resembling admixture.</p

E_ST as a function of SNP sample size.

<p>E_ST was estimated in various population pairs with gradually increasing SNP sample size from 10 to 660,755. As expected, E_ST initially rises rather steeply, but tends to plateau before reaching the 660,755 SNP point. This suggests that we are approaching the maximal resolving power of genetic markers in this dataset, and adding markers beyond this point should not have a significant effect on -cluster separation and E_ST.</p

F_ST and E_ST vs. Clustering with increasing SNP count.

<p>Multidimensional scaling (MDS) plots with Russian (n = 25) and Chinese (n = 34) samples with increasing SNP count from top to bottom (10, 100, 1000, 10,000, and 660,755 SNPs). Two clusters gradually emerge as SNP count increases, along with an increase in E_ST, while F_ST remains relatively constant.</p

Positive correlation between F_ST and E_ST.

<p>R = 0.61. Note that E_ST has a much broader range, spanning nearly the entire 0–1 interval while F_ST only goes as high as 0.2 in these HGDP populations.</p

Pairwise F_ST (above diagonal) and E_ST (below diagonal) in 5 New World and 5 Old World HGDP populations.

<p>Pairwise F_ST (above diagonal) and E_ST (below diagonal) in 5 New World and 5 Old World HGDP populations.</p

Zooming into a Russian (n = 25) and Chinese (n = 34) Neighbor Joining tree of individual similarities.

<p>(A) The length of the red branch compared to the overall tree length is a rough proxy to F_ST. (B) 10x magnification highlights the structure within and between populations. The blue branches inversely associate with E_ST values. (C) 100x magnification reveals fine substructure within Russian samples. Individual branches (black) were removed in B and C for clarity.</p