July 23, 2010

Slow PCI performance on Intel Xeon server

I'm in the process of porting some of my company's software to Linux. I have an application that loads an FPGA image at runtime over the PCI bus on to our custom hardware.

The application works fine on my development PC, which is an Intel Core 2 Duo PC. The FPGA load takes about 25 seconds, about the same as the Windows port on the same hardware. However, on an Intel Xeon server, the same load operation takes 5 minutes.

Both systems are running RHEL 5.3 x86_64 (kernel 2.6.18-el5). All of the software is the same, the only difference is the hardware. I've attached a PCI analyzer to see if I could find anything unusual. Everything looks fine to me, except for the time between successive PCI writes. On the Xeon server, successive writes are spaced much further apart than on the Core 2 PC - the difference is an order of magnitute (I've forgotten the exact numbers).

The drivers I'm using are provided by a third party: Jungo. I'm using v10.11 of their x86_64 Linux driver.

I suspect that this is a software issue, possibly related to the driver that interacts with the PCI bus interface. The Xeon processor is a Nehalem part, W5590. Red Hat claims that it is supported, but I'm not sure how well supported it is. Their knowledge base article is: http://kbase.redhat.com/faq/docs/DOC-15262.

I'm looking for suggestions that will help me identify the problem. I'm new to Linux, but I'm not afraid to dig into the kernel code if I have to.