Friday, December 1, 2017

JVM Stack/Heap Sizes

Posted on 
  • Looked into JVM stack and heap sizes on dfls83-85
As you know there have been service interruptions in CQWeb. I keep looking at the logs for clues. About the only consistent thing is an error similar to this:

2006-09-28 00:10:20 Ajp13Processor[8009][15] process: invoke
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(Unknown Source)
        at java.net.SocketOutputStream.write(Unknown Source)
        at org.apache.ajp.Ajp13.send(Ajp13.java:525)
        at org.apache.ajp.RequestHandler.finish(RequestHandler.java:495)
        at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
        at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
        at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
        at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
        at java.lang.Thread.run(Unknown Source)

Now "Connection reset by peer" could be an error that the process gets because the service has stopped so this could be more of a symptom than a cure. However searching for "Ajp13Processor socket write error" points me to this post which suggests increasing the stack and heap sizes for the JVM. Problems that are intermittent can be consistent with running out of stack or heap size.<./p>

According to the Clearquest Web Administration Guide:

Controlling Java VM Memory Consumption

You can configure the memory consumption of Java processes used by New ClearQuest Web by adjusting the parameters in property files under the various components.

Windows

This section describes the configuration changes for New ClearQuest Web Java VM memory consumption for processes running on Microsoft Windows. To specify the VM memory consumption:
  1. Open the appropriate configuration file for the New ClearQuest Web component whose memory consumption you want to reconfigure. For the ClearQuest Web application:
    ComponentConfiguration file
    Apache Tomcat ServerC:\Program Files\Rational\Common\rwp\bin\jk_service2.in.properties
    Rational Web PlatformC:\Program Files\Rational\Common\rwp\bin\jk_service2.properties

    For the ClearQuest server:
    ComponentConfiguration file
    ClearQuest Request ManagerC:\Program Files\Rational\ClearQuest\cqweb\cqserver\requestmgr_service.properties
    ClearQuest Registry ServerC:\Program Files\Rational\ClearQuest\cqweb\cqregsvr\cqregsvr_service.properties
  2. Modify the section shown below:
    # # JVM Options #
    # Useful Options:
    # -Xms2m = Initial heap size, modify for desired size
    # -Xmx256m = Maximum heap size, modify for desired size
    # -Xrs = Available in Jdk1.3.1 to avoid JVM termination during logoff
    #
    wrapper.jvm.options=-Xrs -Xms2m -Xmx256m
        

I looked at these config files on the three machines (dfls83-85) and they were pretty much set to the default:

Ltx0062320:for server in 83 84 85; do grep
wrapper.jvm.options= //dfls$server/Rational/Common/rwp/bin/jk_service2*properties
//dfls$server/Rational/ClearQuest/cqweb/cqserver/requestmgr_service.properties
//dfls
$server/Rational/ClearQuest/cqweb/cqregsvr/cqregsvr_service.properties;
done
//dfls83/Rational/Common/rwp/bin/jk_service2.default.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls83/Rational/Common/rwp/bin/jk_service2.in.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls83/Rational/Common/rwp/bin/jk_service2.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls83/Rational/ClearQuest/cqweb/cqserver/requestmgr_service.properties:wrapper.jvm.options=-Xrs

//dfls83/Rational/ClearQuest/cqweb/cqregsvr/cqregsvr_service.properties:wrapper.jvm.options=-Xrs

//dfls84/Rational/Common/rwp/bin/jk_service2.default.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls84/Rational/Common/rwp/bin/jk_service2.in.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls84/Rational/Common/rwp/bin/jk_service2.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls84/Rational/ClearQuest/cqweb/cqserver/requestmgr_service.properties:wrapper.jvm.options=-Xrs

//dfls84/Rational/ClearQuest/cqweb/cqregsvr/cqregsvr_service.properties:wrapper.jvm.options=-Xrs

//dfls85/Rational/Common/rwp/bin/jk_service2.default.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls85/Rational/Common/rwp/bin/jk_service2.in.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls85/Rational/Common/rwp/bin/jk_service2.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls85/Rational/ClearQuest/cqweb/cqserver/requestmgr_service.properties:wrapper.jvm.options=-Xrs

//dfls85/Rational/ClearQuest/cqweb/cqregsvr/cqregsvr_service.properties:wrapper.jvm.options=-Xrs

 
All these machines have 2 gig of main memory and largely just serve CQWeb. Indeed the CQWeb service processes are consuming most of the memory:

 
dfls83

 
dfls84 

 
dfls85

I think we should try setting at least the following:

wrapper.jvm.options=-Xrs -Xms128m -Xmx512m

A restart of all CQ Web services would probably be needed for the changes to become effective. The above settings start off the jvm @ 128m for all 4 processes thus a total memory footprint of 512 Meg and limit each process to 512 Meg max for a total footprint of 2 Gig (when full). We might want to bounce this idea off IBM/Rational support to see if all 4 process should have the same settings or if we show vary them.

No comments:

Post a Comment

Posted on  November 23, 2005 Continued to help several users with connecting to the new Controller Clearquest Database for Vinh...