Thursday 28 January 2016

jk_lb_worker.c All tomcat instances failed, no more workers left for recovery

We are getting a issue on our site getting down with following message in mod_jk.log
[Tue Sep 03 16:22:34 2013][22149:1585183040] [info] service::jk_lb_worker.c (1448): All tomcat instances failed, no more workers left for recovery (attempt=0, retry=0)
[Tue Sep 03 16:22:34 2013][22149:1585183040] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=1, retry=0)
[Tue Sep 03 16:22:34 2013][22149:1501264192] [info] service::jk_lb_worker.c (1448): All tomcat instances failed, no more workers left for recovery (attempt=0, retry=0)
[Tue Sep 03 16:22:34 2013][22149:1501264192] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=1, retry=0)
[Tue Sep 03 16:22:34 2013][22149:1616652608] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=0, retry=1)
[Tue Sep 03 16:22:34 2013][22149:1616652608] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=1, retry=1)
[Tue Sep 03 16:22:34 2013][22149:1616652608] [info] service::jk_lb_worker.c (1468): All tomcat instances are busy or in error state
[Tue Sep 03 16:22:34 2013][22149:1616652608] [error] service::jk_lb_worker.c (1473): All tomcat instances failed, no more workers left
[Tue Sep 03 16:22:34 2013]loadbalancer - naf.timesjobs.com 0 0.100549 POST /naf/log.html 200
[Tue Sep 03 16:22:34 2013][22149:1616652608] [info] jk_handler::mod_jk.c (2627): Service error=0 for worker=loadbalancer
[Tue Sep 03 16:22:34 2013][22149:1606162752] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=0, retry=1)
[Tue Sep 03 16:22:34 2013][22149:1606162752] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=1, retry=1)
[Tue Sep 03 16:22:34 2013][22149:1606162752] [info] service::jk_lb_worker.c (1468): All tomcat instances are busy or in error state
[Tue Sep 03 16:22:34 2013][22149:1606162752] [error] service::jk_lb_worker.c (1473): All tomcat instances failed, no more workers left
[Tue Sep 03 16:22:34 2013]loadbalancer - naf.timesjobs.com 0 0.100469 POST /naf/log.html 200
[Tue Sep 03 16:22:34 2013][22149:1606162752] [info] jk_handler::mod_jk.c (2627): Service error=0 for worker=loadbalancer
[Tue Sep 03 16:22:34 2013][22149:1690081600] [info] service::jk_lb_worker.c (1448): All tomcat instances failed, no more workers left for recovery (attempt=0, retry=0)
[Tue Sep 03 16:22:34 2013][22149:1690081600] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=1, retry=0) 
mod-jk.conf

<IfModule mpm_worker_module>
    ThreadLimit         60
    ServerLimit         10
    StartServers        3
    MinSpareThreads     5
    MaxSpareThreads     20
    MaxClients          600
    ThreadsPerChild     60
    MaxRequestsPerChild 0
</IfModule>

While in tomcat server.xml   <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="400" acceptCount="100"/>

Due to which the above error is coming at peak load when apache 600 request coming while tomcat is able to serve only 400.


We changed the setting  <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="600" acceptCount="100"/>

and the problem got resolved.

No comments:

Post a Comment