People often tell me they don't understand performance tool output because they can't tell what's "good" or "bad." It can be hard as performance is subjective. What's good for one user may be bad for another. There are also cases where I can't tell either: The tools only provide clues for further analysis.
Brendan Gregg from Sun's Fishworks team makes an interesting discovery about inducing disk latency. For more details, see Brendan's blog entry: (not available anymore)
A tool to monitor I/O latency in real time. It shows disk latency in the same way as ping shows network latency.
Homepage: https://github.com/koct9i/ioping/ (migrated from http://code.google.com/p/ioping/)
Please send your patches, issues and questions to https://github.com/koct9i/ioping/issues/
Time is Money. Understanding application responsiveness and latency is critical but good characterization of bad data is useless. Gil Tene discusses some common pitfalls encountered in measuring latency and response time behavior. He introduces how simple, open sourced tools can be used to improve and gain higher confidence in both latency measurement and reporting.