Showing posts with label tomcat. Show all posts
Showing posts with label tomcat. Show all posts

Wednesday, 1 June 2016

tomcat 7 difference nio (non blocking I/O) vs bio (blocking I/O)

Tomcat has a couple of connectors to choose from. I’ll leave aside the APR connector, and focus on the BIO and NIO.
The BIO connector (blocking I/O) is blocking – it uses a thread pool where each thread receives a request, handles it, responds, and is returned to the pool. During blocking operations (e.g. reading from database or calling an external API) the thread is blocked.
The NIO connector (non-blocking I/O) is a bit more complicated. It uses the java NIO library and multiplexes between requests. It has two thread pools – one holds the the poller threads, which handle all incoming requests and push these requests to be handled by worker threads, held in another pool. Both pool sizes are configurable.
When to prefer NIO vs BIO depends on the use case. If you mostly have regular request-response usage, then it doesn’t matter, and even BIO might be a better choice (as seen in my previous benchmarks). If you have long-living connections, then NIO is the better choice, because it can server more concurrent users without the need to dedicate a blocked thread to each. The poller threads handle the sending of data back to the client, while the worker threads handle new requests. In other words, neither poller, nor worker threads are blocked and reserved by a single user.
With the introduction of async processing servlet it became easier to have the latter scenario from the previous paragraph. And maybe that was one of the reasons to switch the default connector from BIO to NIO in Tomcat 8. It’s an important thing to have in mind, especially because they didn’t exactly change the “default value”.
The default value is always “HTTP/1.1″, but in Tomcat 7 that “uses an auto-switching mechanism to select either a blocking Java based connector or an APR/native based connector”, while in Tomcat 8 “uses an auto-switching mechanism to select either a non blocking Java NIO based connector or an APR/native based connector”. And to make things even harder, they introduced a NIO2 connector. And to be honest, I don’t know which one of the two NIO connectors is used by default.
So even if you are experienced with tomcat configuration, have in mind this change of defaults. (And generally I’d recommend reading the documentation for all the properties and play with them on your servers)
for more detail check http://techblog.bozho.net/tomcats-default-connectors/

Thursday, 28 January 2016

jk_lb_worker.c All tomcat instances failed, no more workers left for recovery

We are getting a issue on our site getting down with following message in mod_jk.log
[Tue Sep 03 16:22:34 2013][22149:1585183040] [info] service::jk_lb_worker.c (1448): All tomcat instances failed, no more workers left for recovery (attempt=0, retry=0)
[Tue Sep 03 16:22:34 2013][22149:1585183040] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=1, retry=0)
[Tue Sep 03 16:22:34 2013][22149:1501264192] [info] service::jk_lb_worker.c (1448): All tomcat instances failed, no more workers left for recovery (attempt=0, retry=0)
[Tue Sep 03 16:22:34 2013][22149:1501264192] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=1, retry=0)
[Tue Sep 03 16:22:34 2013][22149:1616652608] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=0, retry=1)
[Tue Sep 03 16:22:34 2013][22149:1616652608] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=1, retry=1)
[Tue Sep 03 16:22:34 2013][22149:1616652608] [info] service::jk_lb_worker.c (1468): All tomcat instances are busy or in error state
[Tue Sep 03 16:22:34 2013][22149:1616652608] [error] service::jk_lb_worker.c (1473): All tomcat instances failed, no more workers left
[Tue Sep 03 16:22:34 2013]loadbalancer - naf.timesjobs.com 0 0.100549 POST /naf/log.html 200
[Tue Sep 03 16:22:34 2013][22149:1616652608] [info] jk_handler::mod_jk.c (2627): Service error=0 for worker=loadbalancer
[Tue Sep 03 16:22:34 2013][22149:1606162752] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=0, retry=1)
[Tue Sep 03 16:22:34 2013][22149:1606162752] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=1, retry=1)
[Tue Sep 03 16:22:34 2013][22149:1606162752] [info] service::jk_lb_worker.c (1468): All tomcat instances are busy or in error state
[Tue Sep 03 16:22:34 2013][22149:1606162752] [error] service::jk_lb_worker.c (1473): All tomcat instances failed, no more workers left
[Tue Sep 03 16:22:34 2013]loadbalancer - naf.timesjobs.com 0 0.100469 POST /naf/log.html 200
[Tue Sep 03 16:22:34 2013][22149:1606162752] [info] jk_handler::mod_jk.c (2627): Service error=0 for worker=loadbalancer
[Tue Sep 03 16:22:34 2013][22149:1690081600] [info] service::jk_lb_worker.c (1448): All tomcat instances failed, no more workers left for recovery (attempt=0, retry=0)
[Tue Sep 03 16:22:34 2013][22149:1690081600] [info] service::jk_lb_worker.c (1457): All tomcat instances failed, no more workers left (attempt=1, retry=0) 
mod-jk.conf

<IfModule mpm_worker_module>
    ThreadLimit         60
    ServerLimit         10
    StartServers        3
    MinSpareThreads     5
    MaxSpareThreads     20
    MaxClients          600
    ThreadsPerChild     60
    MaxRequestsPerChild 0
</IfModule>

While in tomcat server.xml   <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="400" acceptCount="100"/>

Due to which the above error is coming at peak load when apache 600 request coming while tomcat is able to serve only 400.


We changed the setting  <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="600" acceptCount="100"/>

and the problem got resolved.