Performance testing and tuning is a science moreso than an art. There are certain tips and tricks that you can generally apply once you have some experience, caching being one of the easiest ones, but many times these will only mask the underlying problem. When it comes to tuning an application to eliminate performance bottlenecks, it’s important to measure a baseline, then form a hypothesis, make a configuration change, and then measure its effect. This is the scientific method in practice, and it generally works much more effectively than randomly applying hacks to your code in places you think (but don’t know, not having measured) will help.
http://stevesmithblog.com/blog/real-world-performance-and-the-stir-trek-web-site/