A message for Linux.com registered users: We are in the process of making changes to the Linux forums. Starting Monday, 8/13/18 at 6:00 PM PT, you will be unable to access the forums. They will re-launch as soon as possible on Wednesday, 8/15/18 with new features and improved functionality. Thank you for your patience and stay tuned for the new improved forums.
GlusterFS : Kernel Out of memory
I'm running GlusterFS 3.7.6 on 3 Nodes (3 CentOS Servers) to serve as Apache WebServers. Nowadays i'm keep getting errors something like:
kernel: Out of memory: Kill process 30883 (glusterfs) score 99 or sacrifice child
.. in all of them. It happens like once every week now.
Whenever it happens, the CPU usages jumped to 100% (according to monitors) and then very soon one of the 3 Nodes gone offline, and the cluster seems to stall. Then of course, the whole Web farm is down.
But based on the Apache logs, LB logs, the Web Traffic is still quite normal. There is no sudden PEAK traffic, in order to trigger this panic. Also my Servers are quite far big enough to handle such amount of total traffic so far. (What i'm trying to say is, it is very unlikely to be the case of Apache panic due to high load.)
It rather seems to be something wrong in GlusterFS.
Please help to suggest what seems to be the cause and where should i look into.