I have just managed to repeat the error. 2 of us, from 2 different
workstations, hammered our website for a minute, by rapidly clicking on
links within the same site before it ended up always responding with a
standard Error 503 (Service unavailable) coming from the Apache frontend. I
checked all the log files, and again the only errors recorded were found in
the mod_jk.log:
....
[Fri Jan 19 11:34:42 2007] [0404:2392] [info] mod_jk.c (2142): Service
error=0 for worker=ajp13
[Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (948):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat is down (127.0.0.1:8009), err=-53
[Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1566):
(ajp13) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c (1877):
(ajp13) receiving from tomcat failed, recoverable operation attempt=1
[Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c (1916):
(ajp13) sending request to tomcat failed, recoverable operation attempt=2
[Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1928):
(ajp13) Connecting to tomcat failed. Tomcat is probably not started or is
listening on the wrong port
....
The 2 connections between Apache and Tomcat according the the netstat -a
command just after the web server started the 503 (Service unavilable)
responses (hightlighted in bold):
Proto Local Address Foreign Address State
TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:2093 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
TCP DAOHPRW2:2110 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED
TCP DAOHPRW2:2077 daohprp2.essex.ac.uk:microsoft-ds TIME_WAIT
TCP DAOHPRW2:2176 sernt29.essex.ac.uk:epmap ESTABLISHED
TCP DAOHPRW2:2177 sernt29.essex.ac.uk:1025 ESTABLISHED
TCP DAOHPRW2:2181 sernt29.essex.ac.uk:microsoft-ds TIME_WAIT
TCP DAOHPRW2:2194 sernt50.essex.ac.uk:epmap ESTABLISHED
TCP DAOHPRW2:2204 sernt5.essex.ac.uk:epmap ESTABLISHED
TCP DAOHPRW2:2215 sernt25.essex.ac.uk:epmap ESTABLISHED
TCP DAOHPRW2:2230 dalinux6a.essex.ac.uk:netbios-ssn
ESTABLISHED
TCP DAOHPRW2:2250 sernt2.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:2253 bodger.essex.ac.uk:microsoft-ds TIME_WAIT
TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:http daohpr1.essex.ac.uk:1528 TIME_WAIT
TCP DAOHPRW2:http daohpr2.essex.ac.uk:1936 TIME_WAIT
TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:2232 sernt23.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:2237 sernt9.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:2245 serlx08.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:2249 sernt2.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:2261 sernt23.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
Half another later, long after the connection_pool_timeout, connect_timeout
and prepost_timeout, the netstat -a command still shows this:
Proto Local Address Foreign Address State
TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED
TCP DAOHPRW2:2754 sernt29.essex.ac.uk:microsoft-ds TIME_WAIT
TCP DAOHPRW2:2782 sernt29.essex.ac.uk:epmap ESTABLISHED
TCP DAOHPRW2:2783 sernt29.essex.ac.uk:1025 ESTABLISHED
TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:microsoft-ds daohpr1.essex.ac.uk:1541 ESTABLISHED
TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
Yet if I then try again to access the web service running on Tomcat, mod_jk
fails to re-establish a connection, according to the mod_jk.log:
[Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (948):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat is down (127.0.0.1:8009), err=-53
[Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (1566):
(ajp13) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c (1877):
(ajp13) receiving from tomcat failed, recoverable operation attempt=0
[Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c (1916):
(ajp13) sending request to tomcat failed, recoverable operation attempt=1
[Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (948):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat is down (127.0.0.1:8009), err=-53
[Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1566):
(ajp13) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c (1877):
(ajp13) receiving from tomcat failed, recoverable operation attempt=1
[Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c (1916):
(ajp13) sending request to tomcat failed, recoverable operation attempt=2
[Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1928):
(ajp13) Connecting to tomcat failed. Tomcat is probably not started or is
listening on the wrong port
[Fri Jan 19 12:50:52 2007] ajp13 daohprw2-a 0.281333 GET /ohpr/servlet/
HTTP/1.1 200
[Fri Jan 19 12:50:52 2007] [0404:2404] [info] mod_jk.c (2142): Service
error=0 for worker=ajp13
Only Apache continues to respond with a standard 503 error according to its
access.log:
daohpr1.essex.ac.uk - - [19/Jan/2007:12:50:51 +0000] "GET /ohpr/servlet/
HTTP/1.1" 503 1101 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
In short: There were only 2 established connections, Tomcat was running all
the time, and, as expected, only saw 2 active HTTPSessions, yet the rapid
hammering on weblinks within the website somehow blocked the connections
between Apache and Tomcat. And now Tomcat keeps just listening on port 8009,
but mod_jk is unable to establish any TCP connections between Apache and
Tomcat. Only a complete server machine re-boot gets things back to normal.
Any ideas what might have gone wrong? I am still using this
workers.properties:
# Define 1 real worker using ajp13
worker.list=ajp13
# Set properties for worker1 (ajp13)
worker.ajp13.type=ajp13
worker.ajp13.host=localhost
worker.ajp13.port=8009
worker.ajp13.connection_pool_timeout=600
worker.ajp13.connection_pool_minsize=10
worker.ajp13.connect_timeout=15000
worker.ajp13.prepost_timeout=10000
Regards
Juergen Neuhoff
--
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@(protected)
To unsubscribe, e-mail: users-unsubscribe@(protected)
For additional commands, e-mail: users-help@(protected)
