No matter how fast the network, it is not likely to compete well with fast storage systems and discs that are Sata/SAS/FC connected. Several factors come to the front.
1. Raw disc throughput.
2. Disc controller I/O capabilities
These two above affect NFS as well as local file systems since the data has to (usually) reside on a disc somewhere, unless you are using RAM discs...
3. Network packet overhead - every packet sent requires an ack return. Dropped packets require resends.
4. Contributing to that is packet size - does your NFS implementation(s) support jumbo packets, or are they limited to the 1500 byte (more or less) standard TCP/IP and Ethernet packet sizes?
5. Are you running your NFS mounts in sync mode (each write waits until commit-to-disc is assured)? For good performance over a high-speed network, you probably want to mount your shares in async mode.
6. Is this a read-mostly environment, or write-intensive?
In any case, this is going to be an expensive proposition, so if you can you might want to test out a similar configuration at an HP capacity planning center. They can set up an environment much as yours is since you are using mostly HP gear, and you can go there and test with your software, data, and system loads. They (used to) have centers in several locations in the US. I've used the ones in Paramus, New Jersey and in Cupertino, California, though I have to admit that the last time I did was about 9-10 years ago. An eon in Internet Time! :-)
