Measuring the performance of applications running over live wide area internets (WAINs) is important and difficult. Directly observed performance and availability can help application developers and network managers. To compensate for the difficulty of controlling important variables such as message path and network load, we developed the Layered Refinement (LR) measurement methodology. LR tools use generic user-level software to collect data simultaneously at important software layers. LR data reduction techniques reveal interesting phenomena on network routing, latency, and availability, in addition to application responsetime and RPC performance over the Internet. We illustrate the application of LR with a series of measurements and some of the phenomena observed
