<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Arial","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Arial","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='color:#1F497D'>We’ve had some oddities with
commons pooling ourselves. One thing we found counter intuitive is we actually
had to specify the dummy query to get the reaping to actually work. We have code
like:<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>setValidationQuery("select
1");<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>in our database setup code. Not
sure if you do that.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Scott<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<p class=MsoNormal><b><span style='font-size:7.5pt;color:#1F497D'><o:p> </o:p></span></b></p>
<p class=MsoNormal><b><span style='font-size:7.5pt;color:#1F497D'><o:p> </o:p></span></b></p>
<p class=MsoNormal><b><span style='font-size:7.5pt;color:#1F497D'>Scott Oster</span></b><span
style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p>
<div>
<p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'> </span><span
style='color:#1F497D'><o:p></o:p></span></p>
</div>
<p class=MsoNormal><span style='font-size:7.5pt;color:#1F497D'>Senior Research
Specialist<br>
Biomedical Informatics Department<br>
Ohio State University<br>
</span><span style='font-family:"Calibri","sans-serif";color:#1F497D'><a
href="mailto:oster@bmi.osu.edu"><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>oster@bmi.osu.edu</span></a><o:p></o:p></span></p>
<p class=MsoNormal><b><span style='font-size:7.5pt;color:#1F497D'>lab:</span></b><span
style='font-size:7.5pt;color:#1F497D'> 614-292-9845</span><span
style='font-family:"Calibri","sans-serif";color:#1F497D'><br>
</span><b><span style='font-size:7.5pt;color:#1F497D'>voice mail:</span></b><span
style='font-size:7.5pt;color:#1F497D'> 614-292-8680</span><span
style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p>
</div>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Blezek, Daniel J
(GE, Research) [mailto:blezek@crd.ge.com] <br>
<b>Sent:</b> Thursday, June 07, 2007 11:18 AM<br>
<b>To:</b> Scott Oster; dart@public.kitware.com<br>
<b>Subject:</b> RE: [Dart] database/memory errors?<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;color:blue'>Hi Scott,</span><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;color:blue'> This is a
hard to find bug that I've been battling for ~1 year. Seems that the
connection pooling mechanism doesn't return connections to the pool from time
to time, and the reaping process doesn't work correctly either (it's supposed
to issue a dummy query and kill the connection if it fails). Since we're
dropping support for 1.4, I think we can try an upgrade of the connection
pooling library, and/or use the Java's mechanism (I remember reading about it,
but can't recall the specifics).</span><span style='font-size:12.0pt;
font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;color:blue'>If this starts to
happen more regularly, you could attach Eclipse (or any other debugger) to your
Dart, if you start it with the debugger enabled and a port open. This
would be helpful for me to try to track this down.</span><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;color:blue'>-dan</span><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
<blockquote style='margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p>
<div class=MsoNormal align=center style='text-align:center'><span
style='font-size:12.0pt;font-family:"Times New Roman","serif"'>
<hr size=2 width="100%" align=center>
</span></div>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'>
dart-bounces+blezek=crd.ge.com@public.kitware.com
[mailto:dart-bounces+blezek=crd.ge.com@public.kitware.com] <b>On Behalf Of </b>Scott
Oster<br>
<b>Sent:</b> Thursday, June 07, 2007 9:26 AM<br>
<b>To:</b> dart@public.kitware.com<br>
<b>Subject:</b> [Dart] database/memory errors?</span><span style='font-size:
12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p>
<p class=MsoNormal>This morning I found some submissions missing from my
dartboard, so I looked at the error log and found it filled with errors like
this:<o:p></o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><07
Jun 2007 07:51:19,349> [SocketListener0-4] ERROR dart.server.Database<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>
closeConnection: Dart's count: 6 Pool count: 7<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Database:
7 active 3 idle Dart's Count: 6<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Got
connection: org.apache.commons.dbcp.PoolableConnection@9e4b4d<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>
dart.server.track.TemporalTrack.getTrackId(TemporalTrack.java:105)<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>
dart.server.TrackManager.getTrackIds(TrackManager.java:93)<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>
dart.server.servlet.Dashboard.doGet(Dashboard.java:232)<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Got
connection: org.apache.commons.dbcp.PoolableConnection@19458f6<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>
dart.server.servlet.Dashboard.doGet(Dashboard.java:146)<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>
javax.servlet.http.HttpServlet.service(HttpServlet.java:596)<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal>Restarting the server seems to have cleared up the problems
. The server (1.0.7) has been running fine for several months. It
appears Dart was having a problem getting a connection to the database.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Looking back at the logs, the problem seems to have started
several days ago (though this morning was the first visible symptom), when the
SocketListener printed out warnings for all my projects like <i>“jetty.servlet.ServletHandler
– Error for <project name> java.lang.OutOfMemoryError: Java heap space”</i><o:p></o:p></p>
<p class=MsoNormal>Following that this exeception was printed:<o:p></o:p></p>
<pre>[SimpleThreadPoolWorker-7] ERROR dart.server.Database - Something was wrong with this connection, marking it invalid.<o:p></o:p></pre><pre>java.sql.SQLException: Already closed.<o:p></o:p></pre><pre> at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:77)<o:p></o:p></pre><pre> at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:180)<o:p></o:p></pre><pre> at dart.server.Database.closeConnection(Database.java:217)<o:p></o:p></pre><pre> at dart.server.Container.closeConnection(Container.java:158)<o:p></o:p></pre><pre> at dart.server.TrackManager.reindexTracks(TrackManager.java:176)<o:p></o:p></pre><pre> at dart.server.task.ReindexTrackTask.execute(ReindexTrackTask.java:28)<o:p></o:p></pre><pre> at dart.server.task.ScheduledTask.execute(ScheduledTask.java:36)<o:p></o:p></pre><pre> at org.quartz.core.JobRunShell.run(JobRunShell.java:203)<o:p></o:p></pre><pre> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)<o:p></o:p></pre>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>From that point on, the log was filled with the messages
shown at the top (“Got connection…”).<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Obviously the OutOfMemory has something to do with it, but
Dart continued to work fine until this morning. If I had to guess, I’d
say for some reason the OutOfMemory is causing database connections to be
somehow corrupted and never recovered. Searching my old logs I have quite
a few occurances of this error, though this is the first time I’ve had to
restart the server.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Scott<span style='font-size:10.0pt;font-family:"Courier New"'><o:p></o:p></span></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><b><span style='font-size:7.5pt'><o:p> </o:p></span></b></p>
<p class=MsoNormal><b><span style='font-size:7.5pt'><o:p> </o:p></span></b></p>
<p class=MsoNormal><b><span style='font-size:7.5pt'>Scott Oster</span></b><span
style='font-family:"Calibri","sans-serif"'><o:p></o:p></span></p>
<div>
<p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'> </span><o:p></o:p></p>
</div>
<p class=MsoNormal><span style='font-size:7.5pt'>Senior Research Specialist<br>
Biomedical Informatics Department<br>
Ohio State University<br>
</span><span style='font-family:"Calibri","sans-serif"'><a
href="mailto:oster@bmi.osu.edu"><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>oster@bmi.osu.edu</span></a><o:p></o:p></span></p>
<p class=MsoNormal><b><span style='font-size:7.5pt'>lab:</span></b><span
style='font-size:7.5pt'> 614-292-9845</span><span style='font-family:"Calibri","sans-serif"'><br>
</span><b><span style='font-size:7.5pt'>voice mail:</span></b><span
style='font-size:7.5pt'> 614-292-8680</span><span style='font-family:"Calibri","sans-serif"'><o:p></o:p></span></p>
<p class=MsoNormal><o:p> </o:p></p>
</blockquote>
</div>
</div>
</body>
</html>