I am following up on my own thread because I found the culprit!
I turns out that the JVM was leaking native memory (that's why profiling
the heap usage did not show anything, nor tracking file descriptors).
The process virtual size (VSZ in top) was growing slowly but surely,
until the OS starts denying malloc calls, triggering the error described
below. This is why the OOM only happened in native code and never in
straight java code.
It turns out that the culprit is struts 1.2.9. It ships with
commons-fileupload 1.0, which uses File.deleteOnExit() to clean up after
itself. File.deleteOnExit() leaks native memory (as documented here:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4513817): this means
that on a struts 1.2.9 webapp with fileupload functionality we leak
native memory on each upload...
Replacing the jar with the latest version (1.1.1) fixes the problem.
- Renaud
Renaud Bruyeron wrote:
>
> We are running into OOM errors that we think are related to this:
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4705373
>
> Our setup:
> JDK 1.5.0_08b3 on Debian Linux (2.4.21SMP)
> tomcat 5.5.17
>
> We have 3 webapps in the tomcat instance with fairly similar
> WEB-INF/lib content
> (struts + spring + hibernate, etc) - roughly 25-30MB worth of .jar files.
>
> We have JAVA_OPTS="-server -XX:MaxPermSize=128M
> -XX:+HeapDumpOnOutOfMemoryError -Xms$1024m -Xmx1024m" and we are
> running on dual-xeon, 4GB main memory hardware.
> RAM is never exhausted, we have on average 3GB of free swap, and we
> never actually consume all of the non-swap RAM.
>
> This is an example of the top of the stacktraces we experience *after
> a while* (i.e. after a couple of days of operation, sometimes more):
>
java.lang.OutOfMemoryError> at
java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.<init>(ZipFile.java:203)
> at java.util.jar.JarFile.<init>(JarFile.java:132)
> at java.util.jar.JarFile.<init>(JarFile.java:70)
> at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:56)
> at
>
sun.net.www.protocol.jar.URLJarFile.getJarFile (
URLJarFile.java:41)
> at
>
sun.net.www.protocol.jar.JarFileFactory.get (
JarFileFactory.java:68)
> at
>
sun.net.www.protocol.jar.JarURLConnection.connect (
JarURLConnection.java:102)
>
> at
> org.apache.jasper.compiler.TagLibraryInfoImpl.<init>(TagLibraryInfoImpl.java:175)
>
> at
>
org.apache.jasper.compiler.Parser.parseTaglibDirective (
Parser.java:423)
> at
>
org.apache.jasper.compiler.Parser.parseDirective (
Parser.java:492)
> at
>
org.apache.jasper.compiler.Parser.parseElements (
Parser.java:1552)
> at
org.apache.jasper.compiler.Parser.parse (
Parser.java:126)
> at
>
org.apache.jasper.compiler.ParserController.doParse (
ParserController.java:211)
>
> at
>
org.apache.jasper.compiler.ParserController.parse (
ParserController.java:116)
>
> at
>
org.apache.jasper.compiler.Parser.processIncludeDirective (
Parser.java:335)
>
> at
>
org.apache.jasper.compiler.Parser.parseIncludeDirective (
Parser.java:372)
> at
>
org.apache.jasper.compiler.Parser.parseDirective (
Parser.java:484)
> at
>
org.apache.jasper.compiler.Parser.parseElements (
Parser.java:1552)
> at
org.apache.jasper.compiler.Parser.parse (
Parser.java:126)
> at
>
org.apache.jasper.compiler.ParserController.doParse (
ParserController.java:211)
>
> at
>
org.apache.jasper.compiler.ParserController.parse (
ParserController.java:116)
>
> at
>
org.apache.jasper.compiler.Parser.processIncludeDirective (
Parser.java:335)
>
> at
>
org.apache.jasper.compiler.Parser.parseIncludeDirective (
Parser.java:372)
> at
>
org.apache.jasper.compiler.Parser.parseDirective (
Parser.java:484)
> at
>
org.apache.jasper.compiler.Parser.parseElements (
Parser.java:1552)
> at
org.apache.jasper.compiler.Parser.parse (
Parser.java:126)
> at
>
org.apache.jasper.compiler.ParserController.doParse (
ParserController.java:211)
>
> at
>
org.apache.jasper.compiler.ParserController.parse (
ParserController.java:100)
>
> at
>
org.apache.jasper.compiler.Compiler.generateJava (
Compiler.java:155)
> at
org.apache.jasper.compiler.Compiler.compile (
Compiler.java:295)
> at
org.apache.jasper.compiler.Compiler.compile (
Compiler.java:276)
> at
org.apache.jasper.compiler.Compiler.compile (
Compiler.java:264)
> at
>
org.apache.jasper.JspCompilationContext.compile (
JspCompilationContext.java:563)
>
> at
>
org.apache.jasper.servlet.JspServletWrapper.service (
JspServletWrapper.java:303)
>
> at
>
org.apache.jasper.servlet.JspServlet.serviceJspFile (
JspServlet.java:314)
> at
>
org.apache.jasper.servlet.JspServlet.service (
JspServlet.java:264)
> at
javax.servlet.http.HttpServlet.service (
HttpServlet.java:802)
> ...
>
> Has anyone else seen this?
> The answer is yes, as per
> http://mail-archives.apache.org/mod_mbox/tomcat-users/200508.mbox/%3C20050809123043.6923E10FB2B8@(protected)
>
> Note that the user above had a different stacktrace involving the
> webapp classloader accessing resources inside jars: we *DO* experience
> this kind of stacktrace as well (for example when using javamail APIs,
> which tries to open a META-INF/mailcap file), and I think it is the
> same underlying problem.
>
> What is the best approach to this problem?
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6280693 suggests
> that the underlying JDK bug has been fixed, however this is a JDK6 fix
> only at this point.
> The original bug
> (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4705373) suggests
> that using
> ZipInputStream instead of ZipFile would also help.
> Please note that our applications do not use the java.util.zip
> package: all the exceptions are thrown while inside tomcat (either the
> WebappClassLoader or inside jasper).
>
> If anyone has some info on this problem, please let me know
>
> - Renaud
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@(protected)
To unsubscribe, e-mail: users-unsubscribe@(protected)
For additional commands, e-mail: users-help@(protected)
