Posts Tagged ‘slowness’

Increase tomcat MaxThreads values to resolve Tomcat timeout issues and sort

Friday, December 11th, 2015

Increase_Tomcat_MaxThreads_values_to_resolve_Tomcat_timeout-issues-and-sort

Thanks God, we have just completed (6 months) Migration few Tomcat and TomEE application servers for PG / PP and Scorpion instances from old environment to a new one for a customer.

Though the separate instances of the old environment are being migrated, the overall design of the Current Mode of Operations (CMO) as they use to call it in corporate World and the Future Mode of Operations (FMO) has differences.

The each of applications on old environment is configured to run in Tomcat failover cluster (2 tomcats on 2 separate machines with unique IP addresses are running) and Apache Reverse Proxy is being used with BalanceMember apache directive in order to drop requests to Tomcat cluster to Tomcat node1 and node2. On the new environment however by design the Tomcat cluster is removed and the application request has to be served by single Tomcat instance.

The migration completed fine and in the beginning in the first day (day 1) and day 2 since the environment went in Production and went through the so-called "GoLive", as called in Corporate World- which is a meathor for launching the application to be used as a production environment for customer, the customer reported TimeOut issues.

Some of the requests according to their report would took up to 4 minutes to serve, after a bit of investigation we found out, that though the environment was moved to one Tomcat the (number) amount of connections to application of end clients did not change, thus the timeouts were caused by default MaxThreads being reached and, we needed to to obviously raise that number. Here is the old Apache RP config where we had the 2 Tomcats between which the RP was load balancing:
 

BalancerMember ajp://10.10.10.5:11010 route=node1 connectiontimeout=10 ttl=60 retry=60
BalancerMember ajp://10.10.10.5:11010 route=node2 connectiontimeout=10 ttl=60 retry=60

ProxyPass / balancer://pool/ stickysession=JSESSIONID
ProxyPassReverse / balancer://pool/


As we needed a work around, we come to conclusion that we just need to increase Timeout on RP first so on Apache Reverse Proxy we placed following httpd.conf Virtualhost ProxyPass (directive) configs :

 

ProxyPass / ajp://10.10.10.5:11010/ keepalive=On timeout=30 connectiontimeout=30 retry=20
ProxyPassReverse / ajp://10.10.10.5:11010/

ProxyPass / ajp://10.10.10.5:11010/ keepalive=On timeout=30 connectiontimeout=30 retry=20
ProxyPassReverse / ajp://10.10.10.5:11010/


and following Apache Timeout directives options:

 

Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15


Even though the developer tried to insist that the problem was in Reverse Proxy timeout config, they were wrong as I checked the RP logs and there was no "maximum connections reached" errors..

As you could guess what left to check was only Tomcat, after quick evaluation of server.xml, it turned out that the MaxThreads directive on old clustered Tomcats was omitted at all, meaning the default MaxThreads Tomcat value of 200 maximum connections were used, however this was not enough as the client was quering the application with about 350 connections / sec.

The solution was of course to raise the Maxthreads to 400 we were pretty lucky that we already had a good dedicated Linux machine where the application was hosted (16GB Ram, 2 CPUs x 2.67 Ghz), thus raising MaxThreads to 400 was not such a big deal.

Here is the final config we used to fix tomcat timeouts:
 

<Connector port="11010" address="10.10.10.80" protocol="AJP/1.3" redirectPort="8443" MaxThreads="400" connectionTimeout="300000" keepAliveTimeout="300000" debug="9" />


One note to make here is the debug="9" options to Connector directive was used to increase debug loglevel of Tomcat, and address="" is the local network IP on which Tomcat instance runs. As you see, we choose to use very high connectionTimeouts (because it is crucial, not to cut requests to applications due to timeouts) in case of application slowness.

We also suspected that there are some Oracle (ORA) database queries slowly served on the SQL backend, that might in future cause more app slowness, but this has to be checked seperately further in time as presently we were checking we did not have our Db person present.

 

Linux: Fixing Qmail server qmail-smtpd port 25 slow (lagged) connect problem

Thursday, May 16th, 2013

qmail logo fixing qmail mail SMTP port 25 connect delays

After updating my Debian Squeeze to latest stable packages from repository with standard:
# apt-get update && apt-get upgrade

I routinely checked, if afterwards all is fine with Qmail?, just to find out connect to port 25 was hell delayed about 40-50 seconds before qmail responds with standard assigned Mail Greeting.
I Googled long time to see if I can find a post or forum thread discussing, exact issue, but though I found similar discussions I didn't found anything that exactly match problem. Thus I decided to follow the good old experimental try / fail method to figure out what causes it.

elow is pastes from telnet, illustrating delays in Qmail SMTP greeting respond:

# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

I spend about 2 hours, checking Qmail for the standard so common errors, usually causing it to not work properly following my previous article testing qmail installation problems

After going, through all of possible causes the only clue for problems, were some slowness with spamassassin. This brought me the idea that something is done wrong with spamassassin .I tried disabling, Spamassassin Razon and Pyzor restarting spamd through (in my case done not via the standard start/stop debian script) but through daemontools with svc and qmailctl i.e.:

# svc -d /service/spamd
# svc -u /service/spamd
# svc -a /service/spamd

qmailctl restart
* Stopping qmail-smtpdssl.
* Stopping qmail-smtpd.
* Sending qmail-send SIGTERM and restarting.
* Restarting qmail-smtpd.
* Restarting qmail-smtpdssl.
* Restarting qmail-pop3d.
This doesn't help, so I continued trying to figure out, what is wrong .One assumption for slow  qmail-smtpd responce was of course slow DNS resolve issues. I checked /etc/resolv.conf to find out server is configured to use local  configured DJBDNS server as first line DNS resolver. I used djbdns for it is simple and easy to configure, however it is a bit obsolete so it was possible bottleneck. After commenting line to use localhost 127.0.0.1
and settings as primary DNS Google Public DNS 8.8.8.8, problem persisted so problems with hosts resolving was obviously not the problem.

I pondered for about 30 minutes, checking again all logs and checking machine processes. Just to remember before I experienced similar issues caused by unresolving RBL (blacklist IP) hosts. I checked configured SPF records in
(process list) and noticed following 4 hosts;

# ps auxwwf

7190 ?        S      0:00 tcpserver -vR -l /var/qmail/control/me -c 30 -u 89 -g 89 -x /etc/tcp.smtp.cdb 0 25 rblsmtpd -t0 -r zen.spamhaus.org -r dnsbl.njabl.org -r dnsbl.sorbs.net -r bl.spamcop.net qmail-smtpd /var/qmail/control/me /home/vpopmail/bin/vchkpw /bin/true
 

I checked one by one hosts and find out 1st two hosts in line are no longer resolving (blacklist is no longer accessible) as before:

 

zen.spamhaus.org, dnsbl.njabl.org

DNSBL (DNS blocklist) is configured on this host via /service/qmail-smtpd/run, hence to remove two unresolvable hosts forcing the weird qmail-smtpd connect delay I had to modify in it:

RBL_BAD="zen.spamhaus.org dnsbl.njabl.org dnsbl.sorbs.net bl.spamcop.net"

to

RBL_BAD="dnsbl.sorbs.net bl.spamcop.net"

After a close examinations in mail server config /var/qmail/control/spfrules, found one other Unresolvable SPF Blacklist host configured ;
# cat /var/qmail/control/spfrules
include:spf.trusted-forwarder.org

To move that one I null-ed file:

# cat /dev/null > /var/qmail/control/spfrules

Finally to take affect all changes, launched Qmail start:

# qmailctl restart
Restarting qmail:
* Stopping qmail-smtpdssl.
* Stopping qmail-smtpd.
* Sending qmail-send SIGTERM and restarting.
* Restarting qmail-smtpd.
* Restarting qmail-smtpdssl.
* Restarting qmail-pop3d.

To check all was fine afterwards, again used telnet:

# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 This is Mail Pc-Freak.NET ESMTP

Mail greeting now appears in about 2-3 seconds time.

 

 

Little Registry Cleaner (Free Software / Open Source Windows XP Registry Cleaner)

Saturday, December 17th, 2011

Little Registry Cleaner - Free and Open Source Software Windows XP Registry cleaner / Alternative to Registry Booster
Have you ever wondered, if there is a free (open source) software that could fix Windows XP registry irregularities e.g. (obsolete or unwanted items that build up in the registry over time.)?

I did not either until now, however when I had to fix, few Windows XP computers which was not maintained for a long time fixing the Windows registry was necessery to make the sluggerish computers improve their overall stability and performance.

The reasons of the slowness in computers who run for a long time by users who does not have a "computer culture" are obvious.
Windows programs which has incorrectly placed registry records withint the Windows registry database, Programs which on Uninstall / Removal left behind a lot of registry records just to hang around because of impotent (coders), or records created on purpose on program uninstall to intentionally further track the user behavior etc.
Other reasons why Windows registry gots bloated with time, are due to Malware or polymorphic Viruses which load them selves everytime on Windows load using some obscure registry records.

Though I'm not a big proprietary software lover still my job as a system administrator , enforces me to fix some broken Windowses.
I haven't fixed Windows machines for a long time, so my memories on programs that clean up registry are from my young years.

The software, I've used before to fix Windows 2000 / XP Registry was:

1. Registry Booster

From my current perspective of a free software hobbyist / evangelist it was important for me to clean up the Windows PCs with a program that is Free or Open Source Software.
When I'm asked to fix some Windows computer I always do my best to make most of the programs that roll on the PC to be FOSS.

Using FOSS instead of downloading from torrents, some cracked software has multiple benefits.

1. Usually Free Software is more stable and more robust2. FOSS software for Windows usually does not come with Malware / Spyware as many of the cracked proprietary software

3. Free and Open Source Programs are simplistic in interface and way of use

A bit of research if there is a Free (Open Source) Software immediately lead me to a program called Little Registry Cleaner
You can see a screenshot of the program in the beginning of the article, the program is very easy to install and use and uses some .NET framework classes so right before installing it installs .NET library (code).

The use results of Little Registry Cleaner were amazing. Even though it is a free software the program found and fixed more registry problems than its competitor Windows Registry Booster! 😉
 

Speed up your DNS resolve if your Internet Service Provider DNS servers fail or resolve slowly / Privacy concerns of public DNS services use

Wednesday, March 30th, 2011

In my experience with many network Internet Service Providers by so far I’ve encountered a lot of DNS oddities and therefore surfing (web) and mail slowness.

It’s sometimes very irritating especially in cases, when I use my internet over Wireless public or university wireless networks.
In principle many of the Wireless routers which distribute the internet especially in organizations are badly configured and the slowness with DNS resolvings is an absolute classic.
If you haven’t encountered that slowness in opening web pages when connected from your University’s canteen, whether it’s fill with people for the lunch break, then I should say you’re really lucky!

My personal experience with this bad configured devices DNS services has been quite negative and every now and then I use to set and use public DNS servers like OpenDNS and Google DNS

Very often when I connect to a wireless network with my notebook running Debian Linux and the internet is too slow in opening pages I automatically set the Google or OpenDNS servers as a default DNS IP resolving servers.

1. DNS IP addresses of Google Public DNS are:

8.8.8.7
and
8.8.8.8

2. OpenDNS Public DNS servers has the IP addresses of:

208.67.222.222
208.67.222.220

I do set up and use the upper public DNS services addresses via the commands:

3. Set and use Google Public DNS services on my Linuxdebian:~# cp -rpf /etc/resolv.conf /etc/resolv.conf.orig
debian:~# echo "nameserver 8.8.8.7n nameserver 8.8.8.8 n" > /etc/resolv.conf;

I first create backu pof my resolv.conf under the name resolv.conf.orig just to make sure I can revert back to my old DNSes if I need them at some point.

If you prefer to use the OpenDNS services for some let’s say privacy reasons, you do it in the same manner as in the above commands, you only change the IP addresses. 4. Configure and use the OpenDNS public DNS services

debian:~# cp -rpf /etc/resolv.conf /etc/resolv.conf.orig
debian:~# echo "nameserver 208.67.222.222n nameserver 208.67.222.220 n" > /etc/resolv.conf;

Of course using Public DNS services has it’s disadvantages over the domain resolving speed up advantage.
One major issue is that Public DNS services are running on a top of a cloud and if you have red my previous article Cloud Computing a possible threat to users privacy and system administrator employment you might be agaist the idea of using a services which are powered by cloud.

The other primary concern is related to your SECURITY and a PRIVACY by using Public DNS networks, you risk that your Public DNS provider might use some DNS spoof techniques to mislead you and resolve you common domain names which usually resolve to let’s say 1.1.1.1 to let’s say 1.5.5.10

Even though this kind of practices on a side of a public DNS provider is not a likely scenario the possible implications of Public DNS providers using DNS forgery to fool you about domain names locations is a very serious issue.

As public DNS providers does contain again the good old philosophy of cloud computing embedded in themselves and they strive to become some kind of a standard which people might vote to adopt and use, the future implications of a wide adoption of Public DNS servers might be a terrible thing on internet users privacy!!!

Just think about a future scenario where we users of the Internet are forced to use a number of public DNS servers in order to use the Internet!
Usually a very huge companies are possessing the Public DNS services and do pay for the tech equipment required for building up the cluster clouds which provide the DNS services and therefore, if in the short future public DNS becomes a fashion and (God forbid!) a standard which shifts up the regular ISP DNS servers to resolve domains to IPs then it will be terrible.

The corporations which does own the Public DNS service/s might have a direct control over filtering and censoling information posted on any website on the internet.
Even worser if the world decides to adopt public DNS services somewhere in the future this means that large corporations owning the open dns cluster or clusters will be able to check each and every resolving made by any user on the net.
If you think closely such an information possessed by a company is not the best thing we want.

So let me close up this article, I’m not a fan and an evangelist who preaches the use of Public DNS services. Right on Contrary I do honestly hate the idea behind public DNS.
Nevertheless apart from my personal opinion I’m a practical person and using the public DNS servers every now and then when this will accelerate my access to the internet is still an option I do enjoy.

Maybe it’s time for a free software project (a tor like), which will provide users with an OpenDNS alternative which will run on hobbyist computers around the globe (just like with tor).

What’s rather funny is that the loud name OpenDNS is a big lie in reality OpenDNS is not opened it’s a company owned closed source service 😉