August 18, 2010

Java process - RSS/VSS are growing unstopped


We have a weblogic 9.2 server with Java1.5.0.16 on RHEL5.3 that we deploy on it a web service and an Alfresco content management system.

We were running it fine for ~3 years on HP-UX i11.23 and a month ago we moved to Linux RH5.3 and from time to time (it happened 3 times) we noticed that the process is starting to use more and more memory until all the memory and swap on the machine ends.

The process still works fine and all the log files looks normal (as if nothing happened) including GC log.

Glance for process ID 25450:

B0000A Glance C.04.70.000 06:54:05 supra2 x86_64 Current Avg High
CPU Util SU | 2% 2% 2%
Disk Util D D | 97% 97% 97%
Mem Util U U | 98% 98% 98%
Swap Util U U | 60% 60% 60%
Resources PID: 25450, java PPID: 25394 euid: 664 User:afspr04
CPU Usage (util): 5.40 Total RSS : 40.6gb
User CPU : 3.60 Text VSS : 56kb
System CPU : 1.80 Data VSS : 66.1gb
Priority : 15 Stack VSS : 2.0mb
Nice Value : 0 Total VSS : 66.5gb
Blocked On : SLEEP
Major Faults : 235
Minor Faults : 164
Processor : 1
Argv1: weblogic.Server
Cmd : /opt/java1.5.0_16/bin/java -Dweblogic.Name=dmcmsserver -server -javaagent:/opt/MercuryDiagn
ostics/JavaAgent/DiagnosticsAgent/lib/probeagent.jar -D -Dcom.wily.introsco
pe.agent.agentName=DMCMS -Xms7g -Xmx7g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=1792m -XX:MaxNewSize=1792m -X
X:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:+DisableExplicitGC -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Xnoclassg
c -Xloggc:logs/gc.log -Dweblogic.Stderr=/app/afspr04/dmcms_ear_p4/dmcmsdomain/logs/online.l
og -Dweblogic.Stdout=/app/afspr04/dmcms_ear_p4/dmcmsdomain/logs/online.log -Damdocs.system.home=/app/afspr04/dmcms_ear_
p4/properties/jesi -Damdocs.messageHandling.home=/app/afspr04/dmcms_ear_p4/properties/jesi -Djesi.config.loader=amdocs.
ecommerce.esi.utils.config.InterfaceConfigXPathLoader -Damdocs.uams.config.resource=config/mvc/ldap ...

pmap shows the big allocation as anonymous
pmap (sorted by the big once):

25450: /opt/java1.5.0_16/bin/java -Dweblogic.Name=dmcmsserver -server -javaagent:/opt/MercuryDiagnostics/JavaAgent/DiagnosticsAgent/lib/probeagent.jar -Dcom.wily.introscope.agent.agentName=DMCMS -Xms7g -Xmx7g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=1792m -XX:MaxNewSize=1792m -XX:SurvivorRatio=4 -XX:TargetSurvivo
00002ab0f8000000 10518548 rwx-- [anon]
00002ab798009000 8388612 rwx-- [anon]
000000005fcce000 8038976 rwx-- [anon]
00002aac7aab0000 7602176 rwx-- [anon]
00002aaf74000000 5259284 rwx-- [anon]
00002ab688000000 4194308 rwx-- [anon]
00002aae4b930000 1684124 rwx-- [anon]
00002aab80000000 1314836 rwx-- [anon]
00002aab20000000 655376 rwx-- [anon]
00002aac28000000 532488 rwx-- [anon]
00002aac50000000 524292 rwx-- [anon]
00002aaaec000000 327696 rwx-- [anon]
00002aaad8000000 131088 rwx-- [anon]
00002ab658000000 131060 rwx-- [anon]
00002ab0dc000000 131044 rwx-- [anon]
00002aaacc2f5000 114708 rwx-- [anon]
total 69733292K

Have anyone encountered something similar?


Click Here!