Hi,

our new shiny 4x opteron dual core server just stoped responding. I've
sent it a kill -QUIT and found out that 678 Threads were hanging in
the same "StackTrace". I won't paste it 678 times, but once :-) Is it
normal for code to hang around there, or is it an indication for a
deadlock of some kind?

"http-8580-Processor15824" daemon prio=1 tid=0xbc176920 nid=0x1d81 waiting for m
onitor entry [0xd507e000..0xd507f5c0]
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:270)
    - waiting to lock <0x61c8de58> (a org.apache.jasper.servlet.JspServletWr
apper)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile (JspServlet.java:2
92)
    at org.apache.jasper.servlet.JspServlet.service (JspServlet.java:236)
    at javax.servlet.http.HttpServlet.service (HttpServlet.java:810)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:237)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:157)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp
atcher.java:703)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(Applica
tionDispatcher.java:463)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationD
ispatcher.java:398)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDis
patcher.java:312)
    at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.
java:1069)
    at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestP
rocessor.java:274)
    at org.apache.struts.action.RequestProcessor.processForwardConfig(Reques
tProcessor.java:455)
    at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(Ti
lesRequestProcessor.java:320)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.ja
va:279)
    at org.apache.struts.action.ActionServlet.process (ActionServlet.java:148
2)
    at org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:507)
    at de.friendscout.datingr4.shared.presentation.controller.ControllerServ
let.doGet(ControllerServlet.java:153)
    at javax.servlet.http.HttpServlet.doHead (HttpServlet.java:270)
    at javax.servlet.http.HttpServlet.service (HttpServlet.java:714)
    at de.friendscout.datingr4.shared.presentation.controller.ControllerServ
let.service(ControllerServlet.java:189)
    at javax.servlet.http.HttpServlet.service (HttpServlet.java:810)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:237)

The line in question is:
       if (options.getDevelopment() || firstTime ) {
          synchronized (this) {
            if (firstTime) {
               firstTime = false;
            }
            ctxt.compile();
          }
       } else {


There are also other, unusual monitor-waitings in the log file.
Shortly before the thread dump the catalina.out contains multiple
entries of type:

org.apache.catalina.core.StandardHostValve@(protected)
Processing ErrorPage[errorCode=404, location=/down/404.html]
ClientAbortException: java.net.SocketException: Connection reset
 at org.apache.coyote.tomcat5.OutputBuffer.doFlush (OutputBuffer.java:331)
 at org.apache.coyote.tomcat5.OutputBuffer.flush (OutputBuffer.java:297)
 at org.apache.coyote.tomcat5.CoyoteResponse.flushBuffer (CoyoteResponse.java:537)
 at org.apache.coyote.tomcat5.CoyoteResponseFacade.flushBuffer (CoyoteResponseFacade.java:238)
 at org.apache.catalina.core.StandardHostValve.status (StandardHostValve.java:303)
 at org.apache.catalina.core.StandardHostValve.invoke (StandardHostValve.java:147)
 at org.apache.catalina.core.StandardValveContext.invokeNext (StandardValveContext.java:104)
 at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:117)
 at org.apache.catalina.core.StandardValveContext.invokeNext (StandardValveContext.java:102)
 at org.apache.catalina.core.StandardPipeline.invoke (StandardPipeline.java:520)
 at org.apache.catalina.core.StandardEngineValve.invoke (StandardEngineValve.java:109)
 at org.apache.catalina.core.StandardValveContext.invokeNext (StandardValveContext.java:104)
 at org.apache.catalina.core.StandardPipeline.invoke (StandardPipeline.java:520)
 at org.apache.catalina.core.ContainerBase.invoke (ContainerBase.java:929)
 at org.apache.coyote.tomcat5.CoyoteAdapter.service (CoyoteAdapter.java:160)
 at org.apache.coyote.http11.Http11Processor.process (Http11Processor.java:793)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:702)
 at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:571)
 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:644)
 at java.lang.Thread.run (Thread.java:595)

StandardWrapperValve[controller]: Servlet.service() for servlet
controller threw exception
java.lang.IllegalStateException: getOutputStream() has already been
called for this response
java.lang.IllegalStateException: getOutputStream() has already been
called for this response
 at org.apache.coyote.tomcat5.CoyoteResponse.getWriter (CoyoteResponse.java:599)
 at org.apache.coyote.tomcat5.CoyoteResponseFacade.getWriter (CoyoteResponseFacade.java:163)
 at javax.servlet.ServletResponseWrapper.getWriter (ServletResponseWrapper.java:111)
 at org.apache.jasper.runtime.JspWriterImpl.initOut (JspWriterImpl.java:122)
 at org.apache.jasper.runtime.JspWriterImpl.flushBuffer (JspWriterImpl.java:115)
 at org.apache.jasper.runtime.PageContextImpl.release (PageContextImpl.java:182)
 at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext (JspFactoryImpl.java:115)
 at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext (JspFactoryImpl.java:75)
....
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:157)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:214)
    at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValv
eContext.java:104)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.jav
a:520)
    at org.apache.catalina.core.StandardContextValve.invokeInternal(Standard
ContextValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:152)
    at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValv
eContext.java:104)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.jav
a:520)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:137)
    at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValv
eContext.java:104)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:117)
    at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValv
eContext.java:102)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.jav
a:520)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:109)
    at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValv
eContext.java:104)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.jav
a:520)
    at org.apache.catalina.core.ContainerBase.invoke (ContainerBase.java:929)
    at org.apache.coyote.tomcat5.CoyoteAdapter.service (CoyoteAdapter.java:16
0)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:793)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ssConnection(Http11Protocol.java:702)
    at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java
:571)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadP
ool.java:644)
    at java.lang.Thread.run (Thread.java:595)

and
org.apache.catalina.core.StandardHostValve@(protected)
Processing ErrorPage[errorCode=500, location=/down/500.html]
ClientAbortException: java.net.SocketException: Broken pipe
 at org.apache.coyote.tomcat5.OutputBuffer.doFlush (OutputBuffer.java:331)
 at org.apache.coyote.tomcat5.OutputBuffer.flush (OutputBuffer.java:297)
 at org.apache.coyote.tomcat5.CoyoteResponse.flushBuffer (CoyoteResponse.java:537)
 at org.apache.coyote.tomcat5.CoyoteResponseFacade.flushBuffer (CoyoteResponseFacade.java:238)
 at org.apache.catalina.core.StandardHostValve.status (StandardHostValve.java:303)
 at org.apache.catalina.core.StandardHostValve.throwable (StandardHostValve.java:244)
 at org.apache.catalina.core.StandardHostValve.invoke (StandardHostValve.java:145)
 at org.apache.catalina.core.StandardValveContext.invokeNext (StandardValveContext.java:104)
 at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:117)
 at org.apache.catalina.core.StandardValveContext.invokeNext (StandardValveContext.java:102)
 at org.apache.catalina.core.StandardPipeline.invoke (StandardPipeline.java:520)
 at org.apache.catalina.core.StandardEngineValve.invoke (StandardEngineValve.java:109)
 at org.apache.catalina.core.StandardValveContext.invokeNext (StandardValveContext.java:104)
 at org.apache.catalina.core.StandardPipeline.invoke (StandardPipeline.java:520)
 at org.apache.catalina.core.ContainerBase.invoke (ContainerBase.java:929)
 at org.apache.coyote.tomcat5.CoyoteAdapter.service (CoyoteAdapter.java:160)
 at org.apache.coyote.http11.Http11Processor.process (Http11Processor.java:793)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:702)
 at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:571)
 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:644)
 at java.lang.Thread.run (Thread.java:595)


any input is highly appreciated.

Regards
Leon

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@(protected)
For additional commands, e-mail: users-help@(protected)