Your area to focus on if you've got the server ship shape, is the software serving up the pages on the the server.
If you are using a php based content management system, you can compress page output using gzip or zlib compression. Doing so will speed up page delivery a great deal. Most content management systems have a switch to turn that option on in the administration backend.
Again, if you are using php, then make sure that you have an op-code cache extension running.
If your content management system has a cache or throttle system, turn them on to make your site accessible to as many visitors as possible under heavy load.
Make sure that the css and page output are valid by checking it using the W3C's validators.
Use closure to compile custom javascipt includes and Code Beautifier to reduce the size of cascading style sheets.
If you are using an open source content management system and you understand the code it's written in, take a look at it in areas of importance (Like it's template system or url route mechanism) To see if there is anything you can do to make the code run faster.
If you are running any parsers on database output which could be putting undue strain on your server, cache the output and/or limit their usage. It's much better to parse once and store the output in a database and retrieve it when a user wants it than to parse it every time a user requests it.
Hope that helps! ;)