Posts Tagged ‘apache reverse proxy’

Increase tomcat MaxThreads values to resolve Tomcat timeout issues and sort

Friday, December 11th, 2015

Increase_Tomcat_MaxThreads_values_to_resolve_Tomcat_timeout-issues-and-sort

Thanks God, we have just completed (6 months) Migration few Tomcat and TomEE application servers for PG / PP and Scorpion instances from old environment to a new one for a customer.

Though the separate instances of the old environment are being migrated, the overall design of the Current Mode of Operations (CMO) as they use to call it in corporate World and the Future Mode of Operations (FMO) has differences.

The each of applications on old environment is configured to run in Tomcat failover cluster (2 tomcats on 2 separate machines with unique IP addresses are running) and Apache Reverse Proxy is being used with BalanceMember apache directive in order to drop requests to Tomcat cluster to Tomcat node1 and node2. On the new environment however by design the Tomcat cluster is removed and the application request has to be served by single Tomcat instance.

The migration completed fine and in the beginning in the first day (day 1) and day 2 since the environment went in Production and went through the so-called "GoLive", as called in Corporate World- which is a meathor for launching the application to be used as a production environment for customer, the customer reported TimeOut issues.

Some of the requests according to their report would took up to 4 minutes to serve, after a bit of investigation we found out, that though the environment was moved to one Tomcat the (number) amount of connections to application of end clients did not change, thus the timeouts were caused by default MaxThreads being reached and, we needed to to obviously raise that number. Here is the old Apache RP config where we had the 2 Tomcats between which the RP was load balancing:
 

BalancerMember ajp://10.10.10.5:11010 route=node1 connectiontimeout=10 ttl=60 retry=60
BalancerMember ajp://10.10.10.5:11010 route=node2 connectiontimeout=10 ttl=60 retry=60

ProxyPass / balancer://pool/ stickysession=JSESSIONID
ProxyPassReverse / balancer://pool/


As we needed a work around, we come to conclusion that we just need to increase Timeout on RP first so on Apache Reverse Proxy we placed following httpd.conf Virtualhost ProxyPass (directive) configs :

 

ProxyPass / ajp://10.10.10.5:11010/ keepalive=On timeout=30 connectiontimeout=30 retry=20
ProxyPassReverse / ajp://10.10.10.5:11010/

ProxyPass / ajp://10.10.10.5:11010/ keepalive=On timeout=30 connectiontimeout=30 retry=20
ProxyPassReverse / ajp://10.10.10.5:11010/


and following Apache Timeout directives options:

 

Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15


Even though the developer tried to insist that the problem was in Reverse Proxy timeout config, they were wrong as I checked the RP logs and there was no "maximum connections reached" errors..

As you could guess what left to check was only Tomcat, after quick evaluation of server.xml, it turned out that the MaxThreads directive on old clustered Tomcats was omitted at all, meaning the default MaxThreads Tomcat value of 200 maximum connections were used, however this was not enough as the client was quering the application with about 350 connections / sec.

The solution was of course to raise the Maxthreads to 400 we were pretty lucky that we already had a good dedicated Linux machine where the application was hosted (16GB Ram, 2 CPUs x 2.67 Ghz), thus raising MaxThreads to 400 was not such a big deal.

Here is the final config we used to fix tomcat timeouts:
 

<Connector port="11010" address="10.10.10.80" protocol="AJP/1.3" redirectPort="8443" MaxThreads="400" connectionTimeout="300000" keepAliveTimeout="300000" debug="9" />


One note to make here is the debug="9" options to Connector directive was used to increase debug loglevel of Tomcat, and address="" is the local network IP on which Tomcat instance runs. As you see, we choose to use very high connectionTimeouts (because it is crucial, not to cut requests to applications due to timeouts) in case of application slowness.

We also suspected that there are some Oracle (ORA) database queries slowly served on the SQL backend, that might in future cause more app slowness, but this has to be checked seperately further in time as presently we were checking we did not have our Db person present.

 

How to remove parameters from URL on Apache (Reverse Proxy) with .htaccess and mod_rewrite – Remove query string from Link

Thursday, December 11th, 2014

Desktop/How_to_remove_parameters_from_URL_on_Apache_with_htaccess_and_SAP
Recently I had a task to delete number of set variables (listed parameters) from URL address on a Apache webserver serving as Reverse Proxy. 
To make it more clear exact task was when customers call the URL https://companywebsite-url.com (all subdomains included) the following URL parameters should always be deleted by the reverse proxy

– ebppMode 
– ebppObjectKey 
– ebppObjectType 
– ebppSystem 
– logSys 

The paramets are part of SAP Biller Direct in a Portal (based on the famous SAP database) which is often deployed as a component of Internet Sales (ISA) / Supplier Relationship Management (SRM) / CRM
, if a user is logged in with his Credentials (KID (Key ID) / Admin KID)  into the system. The EBPP part of most variables stands for (Electronic Bill Presentment and Payment).

 By passing above parameters to Website, modes of use, user accounts switched with which user is logged into the system system logs read and other stuff which can turn to be a severe security hole.
As most of Big Companies, does pass there web traffic via a "transparent" Reverse Proxy,it is a good security practice for SAP Biller Direct (including CRM systems( to wipe out this variables

Here is the mod_rewrite working rules that I used to achieve the delete variable from URL address task:
 

   RewriteEngine On
   RewriteCond %{QUERY_STRING} ^(.*)bebppMode=(w*)b(.*)
   RewriteRule (.*) $1?%1%3

   RewriteCond %{QUERY_STRING} ^(.*)bebppObjectKey=(w*)b(.*)
   RewriteRule (.*) $1?%1%3

   RewriteCond %{QUERY_STRING} ^(.*)bebppObjectType=(w*)b(.*)
   RewriteRule (.*) $1?%1%3

   RewriteCond %{QUERY_STRING} ^(.*)bebppSystem=(w*)b(.*)
   RewriteRule (.*) $1?%1%3

   RewriteCond %{QUERY_STRING} ^(.*)logSys=(w*)b(.*)
   RewriteRule (.*) $1?%1%3

   RewriteCond %{QUERY_STRING} ^(.*)&&(.*)
   RewriteRule (.*) $1?%1%3


P.S. I've implemented above Rewrite rules into all Virtualhosts of Applications (in that case all living in the same httpd.conf on SuSE (SLES) 11 SP1 Enterprise Linux server).
To make changes affective, restarted HTTPD Webserver:
 

/etc/init.d/httpd restart


The sesult is:

https://companywebsite-url.com/start.html?page=start&ebppMode=A&ebppSystem=Test

leads to a internal URL redirection

https://companywebsite-url.com/start.html?page=start

without parameters ebppSystem, ebppMode, ebppObjectKey, ebppSystem,   logSys .

Other mod_rewrite rule that works but is too ugly and when I tried it on Debian Linux host was behaving strange (including in the rewrited URL address the directory address of the PHP twice):

RewriteCond %{QUERY_STRING} (.*)(^|&|%26|%20)ebppMode(=|%3D)([^&]+)(.*)$ [OR]
RewriteCond %{QUERY_STRING} (.*)(^|&|%26|%20)ebppObjectKey(=|%3D)([^&]+)(.*)$ [OR]
RewriteCond %{QUERY_STRING} (.*)(^|&|%26|%20)ebppObjectType(=|%3D)([^&]+)(.*)$ [OR]
RewriteCond %{QUERY_STRING} (.*)(^|&|%26|%20)ebppSystem(=|%3D)([^&]+)(.*)$ [OR]
RewriteCond %{QUERY_STRING} (.*)(^|&|%26|%20)logSys(=|%3D)([^&]+)(.*)$

RewriteRule (.*) /$1?%1%5 [R=307]

 Well anyways, with the first bunch of mod_rewrite rule it works fine.

Thanks God Problem Solved 🙂

Apache Reverse Proxy to Tomcat – What is reverse proxying and how to configure it on Linux

Monday, November 4th, 2013

Reverse proxy diagram Apache and Tomcat webserver diagram how to make reverse proxy of tomcat behind Apache

Reverse Proxy is a Proxy server which routes all incoming traffic to secondary Webserver situated behind the Reverse Proxy site. Then all incoming replies from secondary webserver (which is not visible) from the internet gets routed back to Reverse Proxy service. The result is it seems like all incoming and outgoing HTTP requests are served from Reverse Proxy host where in reality, reverse proxy host just does traffic redirection. Problem with reverse proxies is it is one more point of failure the good side of it can protect and route only certain traffic to your webserver, preventing the behind reverse proxy located server from crackers malicious HTTP requests.

Very common use of Reverse Proxy is whether traffic has to be served by Tomcat webservers. Probably one of most common configuration is Apache (configured to serve traffic) as reverse proxy and one or more Tomcat Webservers to which web traffic is re-routed.
By default Tomcat listens for client connections on port 8082 (-. http://localhost:8082), Apache is then used to route all requests serving as Reverse Proxy incoming on port

Reverse Proxy:80 to -> Tomcat server (secondary host):8082

  • So what are benefits (PROS) of Reverse Proxy?

        a) Webservers in Internal Network will be less vulnerable to attacks from the Internet
        b) Reduces risk to sensitive data – Only existent resources on Proxied server will available
        c) Enables makes easy Load Balancing and Fail Over

  • What are disadvantages (CONS) of Reverse Proxy?

         a) If a single instalce of Reverse Proxy is available, this creates a single point of failure

To have reverse proxy it is necessary that Apache server is configured with enabled mod_proxy
Creating a reverse proxy is being done as follows (Assuming you have to do it for a Virtualhost)

Open Apache config with VirtualHost for which Reverse Proxy has to be set, whether only one domain is set on server there is no need for <VirtualHost *:80> opening and close directives but same rules has to be set for primary domain.

<VirtualHost *:80>
  ServerName your-domain.com
  ServerAlias *.your-domain.com
  ProxyRequests off 
  ProxyPassMatch /server-status !
  ProxyPassMatch /server-info !
  ProxyPassMatch /ldap-status !
  ProxyPass / http://10.10.10.1:8082/ keepalive=on timeout=30 connectiontimeout=60 retry=20
  ProxyPassReverse / http://10.10.10.1:8082/ 
</VirtualHost>

In above example is 10.10.10.1 is local IP address of DMZed host running Tomcat server. That's all necessary assuming that on port 8082 Catalina Tomcat is running. Of course same scenario can be used not only with Tomcat but with DMZ-ed Webservers with Apache, Nginx or Lighttpd.