Linux.com

I'm doing some performance evaluation of an in-house app. We're working on figuring how it will scale. I'm hitting the app hard, and trying to figure out what the bottleneck is. here's a sample of output from vmstat while I'm hitting it:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 15173388 3432 59740 0 0 0 8 62 91 0 0 100 0 0 0 0 15173388 3432 59740 0 0 0 0 54 85 0 0 100 0 0 1 0 15170248 3436 59736 0 0 2 30 130 212 2 0 98 0 0 0 0 15166960 3440 59800 0 0 4 2 327 694 2 0 98 0 0 0 0 15145812 3440 59800 0 0 0 0 2067 3110 12 0 87 0 6 0 0 15068504 3448 59792 0 0 2 10 2995 3437 22 4 65 0 5 0 0 14930312 3448 59792 0 0 2 0 4401 4672 31 5 58 0 1 0 0 14781876 3448 59792 0 0 0 2 3849 4626 32 3 58 0 2 0 0 14640516 3456 59784 0 0 0 10 3753 4541 29 5 56 0 6 0 0 14492708 3456 59784 0 0 0 18 4333 5065 24 4 68 0 12 0 0 14354448 3464 59776 0 0 0 26 3778 4786 27 4 64 0 4 0 0 14188900 3464 59776 0 0 0 0 4274 5150 25 3 59 0 7 0 0 14033280 3464 59776 0 0 0 22 4117 5160 28 4 62 0 4 0 0 13860648 3472 59768 0 0 0 8 4557 5486 18 5 69 0 9 0 0 13677500 3472 59768 0 0 0 0 5092 6068 17 4 72 0 2 0 0 13477604 3472 59768 0 0 0 0 4158 5728 37 5 48 0 5 0 0 13237564 3480 59760 0 0 0 32 5674 7095 21 5 64 0 11 0 0 12976792 3480 59760 0 0 0 0 5438 6687 17 6 65 0 0 0 0 12826860 3480 59760 0 0 0 0 3848 4787 10 3 81 0 5 0 0 12645944 3488 59752 0 0 0 8 3051 4118 33 3 54 0 3 0 0 12378988 3488 59752 0 0 0 0 5445 6996 24 6 58 0 12 0 0 12116100 3488 59752 0 0 0 0 6209 7653 17 5 68 0

I have a few things that I'd like to figure out. This is on a system with 4 cores. First, the r values suggest to me that there are often lots of processes waiting to run, so it seems like CPU might be the bottleneck. But the idle value for CPU is generally > 50%. This is a fairly IO (network) intensive application, but I don't know how to interpret these results to know for sure whether IO or CPU is an issue. Why might r be high, with lots of CPU cycles to spare?

So I've run sar and looked at the results. Pretty much what I got out of vmstat, plus a couple of things:
- We're doing 20-25 Mbps in + out on eth0. We have 250Mbps available to us.
- Negligible disk activity.

So I'm still stuck with the question, what's my bottleneck? What else should I be looking at?

Looking at your vmstat results there is no disk activity, no swap, the CPU is not maxed so there are probably concurrent processes waiting for the same data.
The classic example is a mysql database being hammered and using table level locking. Is the app using any sort of external database? How is data shared between threads?