What version of Apache are you running? Is the site's content heavy on static content or heavy on dynamic content? Is the dynamic content server generated, or are you relying heavily on client side technologies like java and java script? And what are your server side scripts written in? Any modules loaded into apache?

Hi Miker,
Your area to focus on if you've got the server ship shape, is the software serving up the pages on the the server.
If you are using a php based content management system, you can compress page output using gzip or zlib compression. Doing so will speed up page delivery a great deal. Most content management systems have a switch to turn that option on in the administration backend.
Again, if you are using php, then make sure that you have an op-code cache extension running.
If your content management system has a cache or throttle system, turn them on to make your site accessible to as many visitors as possible under heavy load.
If you have things like javascript includes for popular libraries such as jQuery, you can load those libraries from places like google's ajax apis. Doing so can provide a speed boost if you have lots of javascript going on, especially if a visitor has just come from another site which is doing the same. Web browsers cache javascripts for faster operation, so in the same session a browser will re-use the already downloaded script with any site you visit which uses the method.
Make sure that the css and page output are valid by checking it using the W3C's validators.
Use closure to compile custom javascipt includes and Code Beautifier to reduce the size of cascading style sheets.
If you are using an open source content management system and you understand the code it's written in, take a look at it in areas of importance (Like it's template system or url route mechanism) To see if there is anything you can do to make the code run faster.
If you are running any parsers on database output which could be putting undue strain on your server, cache the output and/or limit their usage. It's much better to parse once and store the output in a database and retrieve it when a user wants it than to parse it every time a user requests it.

Hope that helps! ;)

Thanks,
Tony

I think not a lot of sites, then there will not be such a problem

Personal view

You can Cache your website to make it load faster and Gzip the site before sending data out of the server.

Check the article on my blog techstream

CACHING YOUR SITE
Compress Your Web Site
Active Gzip Compression In HTACCESS

If you have further quarries post it here or in Tech Stream Discuess