The question is a bit too broad, but let me try to give you some hints.

You can examine the Top500 list, use the report generation tool. For example, you can see that Linux is used in most (87%) of the listed sites: http://www.top500.org/stats/list/32/osfam

The number of nodes varies, I have seen as little as 16 nodes and as many as 2048. Roadrunner, the fastest system in the November 2008 ranking, has over 128 thousand cores which I believe translates to 16 thousand nodes. It uses PowerXCell processors and Infiniband for interconnect. For message passing, Voltaire offers several options on top of Infiniband, MPI is of course included, which is, I believe the most common API used on Linux clusters.