Classical rate theories often fail in cases where the observable(s) or order
parameter(s) used are poor reaction coordinates or the observed signal is
deteriorated by noise, such that no clear separation between reactants and
products is possible. Here, we present a general spectral two-state rate theory
for ergodic dynamical systems in thermal equilibrium that explicitly takes into
account how the system is observed. The theory allows the systematic estimation
errors made by standard rate theories to be understood and quantified. We also
elucidate the connection of spectral rate theory with the popular Markov state
modeling (MSM) approach for molecular simulation studies. An optimal rate
estimator is formulated that gives robust and unbiased results even for poor
reaction coordinates and can be applied to both computer simulations and
single-molecule experiments. No definition of a dividing surface is required.
Another result of the theory is a model-free definition of the reaction
coordinate quality (RCQ). The RCQ can be bounded from below by the directly
computable observation quality (OQ), thus providing a measure allowing the RCQ
to be optimized by tuning the experimental setup. Additionally, the respective
partial probability distributions can be obtained for the reactant and product
states along the observed order parameter, even when these strongly overlap.
The effects of both filtering (averaging) and uncorrelated noise are also
examined. The approach is demonstrated on numerical examples and experimental
single-molecule force probe data of the p5ab RNA hairpin and the apo-myoglobin
protein at low pH, here focusing on the case of two-state kinetics
