[Dart] database/memory errors?

[Scott Oster]

This morning I found some submissions missing from my dartboard, so I
looked at the error log and found it filled with errors like this:

<07 Jun 2007 07:51:19,349> [SocketListener0-4] ERROR
dart.server.Database

        closeConnection: Dart's count: 6 Pool count: 7

Database: 7 active 3 idle Dart's Count: 6

Got connection: org.apache.commons.dbcp.PoolableConnection at 9e4b4d

 
dart.server.track.TemporalTrack.getTrackId(TemporalTrack.java:105)

        dart.server.TrackManager.getTrackIds(TrackManager.java:93)

        dart.server.servlet.Dashboard.doGet(Dashboard.java:232)

 

Got connection: org.apache.commons.dbcp.PoolableConnection at 19458f6

        dart.server.servlet.Dashboard.doGet(Dashboard.java:146)

        javax.servlet.http.HttpServlet.service(HttpServlet.java:596)

        javax.servlet.http.HttpServlet.service(HttpServlet.java:689)

 

 

Restarting the server seems to have cleared up the problems .  The
server (1.0.7) has been running fine for several months.  It appears
Dart was having a problem getting a connection to the database.

 

Looking back at the logs, the problem seems to have started several days
ago (though this morning was the first visible symptom), when the
SocketListener printed out warnings for all my projects like
"jetty.servlet.ServletHandler - Error for <project name>
java.lang.OutOfMemoryError: Java heap space"

Following that this exeception was printed:

[SimpleThreadPoolWorker-7] ERROR dart.server.Database  - Something was
wrong with this connection, marking it invalid.
java.sql.SQLException: Already closed.
        at
org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java
:77)
        at
org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.clo
se(PoolingDataSource.java:180)
        at dart.server.Database.closeConnection(Database.java:217)
        at dart.server.Container.closeConnection(Container.java:158)
        at dart.server.TrackManager.reindexTracks(TrackManager.java:176)
        at
dart.server.task.ReindexTrackTask.execute(ReindexTrackTask.java:28)
        at dart.server.task.ScheduledTask.execute(ScheduledTask.java:36)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
        at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java
:520)

 

>From that point on, the log was filled with the messages shown at the
top ("Got connection...").

 

Obviously the OutOfMemory has something to do with it, but Dart
continued to work fine until this morning.  If I had to guess, I'd say
for some reason the OutOfMemory is causing database connections to be
somehow corrupted and never recovered.  Searching my old logs I have
quite a few occurances of this error, though this is the first time I've
had to restart the server.

 

Scott

 

 

 

 

Scott Oster

 

Senior Research Specialist
Biomedical Informatics Department
Ohio State University
oster at bmi.osu.edu <mailto:oster at bmi.osu.edu> 

lab: 614-292-9845
voice mail: 614-292-8680

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://public.kitware.com/pipermail/dart/attachments/20070607/eb55b69d/attachment.html