Shutdown tomcat server node in case of memory depletion – Avoiding Tomcat Out of memory

Friday, 6th June 2014

fix-avoid-tomcat-out-of-memory-logo

Out Of Memory Errors, or OOMEs, are one of the most common problems faced by Apache Tomcat users. Tomcat cluster behind Apache unreachable (causing customer downtimes). OOME errors occur on production servers that are experiencing an unusually high spike of traffic.

Out of memory errors are usually a problem of application and not of Tomcat server. OMEs have become such a persistent topic of discussion in the Apache Tomcat community cause its so difficult to trace to their root cause. Usually 'incorrect' web app code causing Tomcat to run out of memory is usually technically correct.

Most common reasons for Out of Memory errors in application code are:
 

  •     the heap size being too small
  •     running out of file descriptors
  •     more open threads than the host OS allows
  •     code with high amounts of recursion
  •     code that loads a very large file into memory
  •     code that retaining references to objects or classloaders
  •     a large number of web apps and a small PermGen


The following java option -XX:OnOutOfMemoryError= could be added to any of tomcat java application servers in setenv.sh in  JAVA_OPTS= variable in case of regular Out of Memory errors occur making an application unstable.

-XX:OnOutOfMemoryError=<path_to_tomcat_shutdown_script.sh>

Where < path_to tomcat_shutdown_script.sh > is shutdown script(which performs kill <tomcat_pid> if normal shutdown fails) for the tomcat instance.

With this setup if any tomcat instance run out of memory it will be shutdown (shutdown script invoked) – as result the Apache proxy infront of Tomcats should not pass any further requests to this instance and application will visualize / work properly for end customers.

Usually a tomcat_shutdown_script.sh to invoke in case of OOM would initiate a Tomcat server restart something like:

for i in `ps -ef |grep tomcat |grep /my_path_to_my_instance | awk '{print $2}'`
do
kill -9 "$i"
#path and script to start tomcat
done

To prevent blank pages returned to customer because of shutdown_script.sh starting stopping Tomcat you can set in Reverse Apache Proxy something like:
 

<Proxy balancer://mycluster>
   BalancerMember ajp://10.16.166.48:11010/ route=delivery1 timeout=30 retry=1
   BalancerMember ajp://10.16.166.70:11010/ route=delivery2 timeout=30 retry=1
</Proxy>

Where in above example I assume, there are only two tomcat nodes, for more just add respective ones.

Note that if the deployed application along all servers is having some code making it crash all tomcat nodes can get shutdown all time and you can get in a client havoc 🙂

Share this on:

Download PDFDownload PDF

Tags: , , , , , , , , , , ,

2 Responses to “Shutdown tomcat server node in case of memory depletion – Avoiding Tomcat Out of memory”

  1. admin says:
    Firefox 29.0 Firefox 29.0 Windows 7 x64 Edition Windows 7 x64 Edition
    Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0

    Tcat (Enterprise version of Apache tomcat) – is useful in debugging and quickly fixing out of memory (memory leaks).

    Tcat allows adminstrator to create custom e-mail or SMS alerts for any attribute of a server or web application, such as unusual memory activity.

    View CommentView Comment
  2. admin says:
    Firefox 30.0 Firefox 30.0 Windows 7 x64 Edition Windows 7 x64 Edition
    Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0

    Just a small important note regarding connectionTimeout in Tomcat is calculated not in seconds like in Apache but in miliseconds.
    Default value for connectionTimeout="20000" converted to seconds is only 20 seconds.

    Below is example Tomcat config extractions from server.xml:


    <!–You will find the connectors now in BIS6 frontend (System Settings/HTTP services/listener)–>

        <!–Connector port="11004" maxHttpHeaderSize="8192"

                   maxThreads="150" minSpareThreads="25" maxSpareThreads="75"

                   enableLookups="false" redirectPort="11001" acceptCount="100"

                   connectionTimeout="20000" disableUploadTimeout="true" /–>

        <!– Note : To disable connection timeouts, set connectionTimeout value to 0 –>

     

        <!– Define a SSL HTTP/1.1 Connector on port 11001 –>

        <!–You will find the connectors now in BIS6 frontend (System Settings/HTTP services/listener)–>

        <!–Connector port="11001" maxHttpHeaderSize="8192"

                   sSLImplementation="com.seeburger.tksm.SeeSSLImplementation"

                   maxThreads="150" minSpareThreads="25" maxSpareThreads="75"

                   enableLookups="false" acceptCount="100"

                   disableUploadTimeout="true"

                   scheme="https" secure="true" clientAuth="false"

                   keystoreFile="USERS\SSL\TC_sl02296_BIS6_SV_NODE_DT"

                   trustStoreFile="TRUSTED\SSL\*"

                   sslProtocol="${tomcat.ssl.protocol}" algorithm="${tomcat.ssl.algorithm}" /–>

    <!–You will find the connectors now in BIS6 frontend (System Settings/HTTP services/listener)–>

        <!–Connector port="11004" maxHttpHeaderSize="8192"

                   maxThreads="150" minSpareThreads="25" maxSpareThreads="75"

                   enableLookups="false" redirectPort="11001" acceptCount="100"

                   connectionTimeout="20000" disableUploadTimeout="true" /–>

        <!– Note : To disable connection timeouts, set connectionTimeout value to 0 –>

     

        <!– Define a SSL HTTP/1.1 Connector on port 11001 –>

        <!–You will find the connectors now in BIS6 frontend (System Settings/HTTP services/listener)–>

        <!–Connector port="11001" maxHttpHeaderSize="8192"

                   sSLImplementation="com.seeburger.tksm.SeeSSLImplementation"

                   maxThreads="150" minSpareThreads="25" maxSpareThreads="75"

                   enableLookups="false" acceptCount="100"

                   disableUploadTimeout="true"

                   scheme="https" secure="true" clientAuth="false"

                   keystoreFile="USERS\SSL\TC_sl02296_BIS6_SV_NODE_DT"

                   trustStoreFile="TRUSTED\SSL\*"

                   sslProtocol="${tomcat.ssl.protocol}" algorithm="${tomcat.ssl.algorithm}" /–>

     

    View CommentView Comment

Leave a Reply

CommentLuv badge