<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v =
"urn:schemas-microsoft-com:vml" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:m =
"http://schemas.microsoft.com/office/2004/12/omml"><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.3059" name=GENERATOR>
<STYLE>@font-face {
font-family: Calibri;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Arial","sans-serif"
}
LI.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Arial","sans-serif"
}
DIV.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Arial","sans-serif"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
PRE {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"; mso-style-priority: 99; mso-style-link: "HTML Preformatted Char"
}
SPAN.EmailStyle17 {
COLOR: windowtext; FONT-FAMILY: "Arial","sans-serif"; mso-style-type: personal-compose
}
SPAN.HTMLPreformattedChar {
FONT-FAMILY: "Courier New"; mso-style-priority: 99; mso-style-link: "HTML Preformatted"; mso-style-name: "HTML Preformatted Char"
}
.MsoChpDefault {
mso-style-type: export-only
}
DIV.Section1 {
page: Section1
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></HEAD>
<BODY lang=EN-US vLink=purple link=blue>
<DIV dir=ltr align=left><SPAN class=203251415-07062007><FONT face=Arial
color=#0000ff size=2>Hi Scott,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=203251415-07062007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=203251415-07062007><FONT face=Arial
color=#0000ff size=2> This is a hard to find bug that I've been battling
for ~1 year. Seems that the connection pooling mechanism doesn't return
connections to the pool from time to time, and the reaping process doesn't work
correctly either (it's supposed to issue a dummy query and kill the connection
if it fails). Since we're dropping support for 1.4, I think we can try an
upgrade of the connection pooling library, and/or use the Java's mechanism (I
remember reading about it, but can't recall the specifics).</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=203251415-07062007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=203251415-07062007><FONT face=Arial
color=#0000ff size=2>If this starts to happen more regularly, you could attach
Eclipse (or any other debugger) to your Dart, if you start it with the debugger
enabled and a port open. This would be helpful for me to try to track this
down.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=203251415-07062007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=203251415-07062007><FONT face=Arial
color=#0000ff size=2>-dan</FONT></SPAN></DIV><BR>
<BLOCKQUOTE dir=ltr style="MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B>
dart-bounces+blezek=crd.ge.com@public.kitware.com
[mailto:dart-bounces+blezek=crd.ge.com@public.kitware.com] <B>On Behalf Of
</B>Scott Oster<BR><B>Sent:</B> Thursday, June 07, 2007 9:26 AM<BR><B>To:</B>
dart@public.kitware.com<BR><B>Subject:</B> [Dart] database/memory
errors?<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=Section1>
<P class=MsoNormal>This morning I found some submissions missing from my
dartboard, so I looked at the error log and found it filled with errors like
this:<o:p></o:p></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'"><07 Jun 2007
07:51:19,349> [SocketListener0-4] ERROR
dart.server.Database<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">
closeConnection: Dart's count: 6 Pool count: 7<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">Database: 7 active 3 idle
Dart's Count: 6<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">Got connection:
org.apache.commons.dbcp.PoolableConnection@9e4b4d<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">
dart.server.track.TemporalTrack.getTrackId(TemporalTrack.java:105)<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">
dart.server.TrackManager.getTrackIds(TrackManager.java:93)<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">
dart.server.servlet.Dashboard.doGet(Dashboard.java:232)<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">Got connection:
org.apache.commons.dbcp.PoolableConnection@19458f6<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">
dart.server.servlet.Dashboard.doGet(Dashboard.java:146)<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">
javax.servlet.http.HttpServlet.service(HttpServlet.java:596)<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'"><o:p> </o:p></SPAN></P>
<P class=MsoNormal>Restarting the server seems to have cleared up the problems
. The server (1.0.7) has been running fine for several months. It
appears Dart was having a problem getting a connection to the
database.<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Looking back at the logs, the problem seems to have started
several days ago (though this morning was the first visible symptom), when the
SocketListener printed out warnings for all my projects like
<I>“jetty.servlet.ServletHandler – Error for <project name>
java.lang.OutOfMemoryError: Java heap space”</I><o:p></o:p></P>
<P class=MsoNormal>Following that this exeception was printed:<o:p></o:p></P><PRE>[SimpleThreadPoolWorker-7] ERROR dart.server.Database - Something was wrong with this connection, marking it invalid.<o:p></o:p></PRE><PRE>java.sql.SQLException: Already closed.<o:p></o:p></PRE><PRE> at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:77)<o:p></o:p></PRE><PRE> at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:180)<o:p></o:p></PRE><PRE> at dart.server.Database.closeConnection(Database.java:217)<o:p></o:p></PRE><PRE> at dart.server.Container.closeConnection(Container.java:158)<o:p></o:p></PRE><PRE> at dart.server.TrackManager.reindexTracks(TrackManager.java:176)<o:p></o:p></PRE><PRE> at dart.server.task.ReindexTrackTask.execute(ReindexTrackTask.java:28)<o:p></o:p></PRE><PRE> at dart.server.task.ScheduledTask.execute(ScheduledTask.java:36)<o:p></o:p></PRE><PRE> at org.quartz.core.JobRunShell.run(JobRunShell.java:203)<o:p></o:p></PRE><PRE> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)<o:p></o:p></PRE>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>From that point on, the log was filled with the messages
shown at the top (“Got connection…”).<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Obviously the OutOfMemory has something to do with it, but
Dart continued to work fine until this morning. If I had to guess, I’d
say for some reason the OutOfMemory is causing database connections to be
somehow corrupted and never recovered. Searching my old logs I have
quite a few occurances of this error, though this is the first time I’ve had
to restart the server.<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Scott<SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'"><o:p></o:p></SPAN></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal><B><SPAN
style="FONT-SIZE: 7.5pt"><o:p> </o:p></SPAN></B></P>
<P class=MsoNormal><B><SPAN
style="FONT-SIZE: 7.5pt"><o:p> </o:p></SPAN></B></P>
<P class=MsoNormal><B><SPAN style="FONT-SIZE: 7.5pt">Scott
Oster</SPAN></B><SPAN
style="FONT-FAMILY: 'Calibri','sans-serif'"><o:p></o:p></SPAN></P>
<DIV>
<P class=MsoNormal><SPAN
style="FONT-FAMILY: 'Calibri','sans-serif'"> </SPAN><o:p></o:p></P></DIV>
<P class=MsoNormal><SPAN style="FONT-SIZE: 7.5pt">Senior Research
Specialist<BR>Biomedical Informatics Department<BR>Ohio State
University<BR></SPAN><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'"><A
href="mailto:oster@bmi.osu.edu"><SPAN
style="FONT-SIZE: 7.5pt; COLOR: blue; FONT-FAMILY: 'Arial','sans-serif'">oster@bmi.osu.edu</SPAN></A><o:p></o:p></SPAN></P>
<P class=MsoNormal><B><SPAN style="FONT-SIZE: 7.5pt">lab:</SPAN></B><SPAN
style="FONT-SIZE: 7.5pt"> 614-292-9845</SPAN><SPAN
style="FONT-FAMILY: 'Calibri','sans-serif'"><BR></SPAN><B><SPAN
style="FONT-SIZE: 7.5pt">voice mail:</SPAN></B><SPAN style="FONT-SIZE: 7.5pt">
614-292-8680</SPAN><SPAN
style="FONT-FAMILY: 'Calibri','sans-serif'"><o:p></o:p></SPAN></P>
<P class=MsoNormal><o:p> </o:p></P></DIV></BLOCKQUOTE></BODY></HTML>