[Dart] database/memory errors?

[Scott Oster]

We've had some oddities with commons pooling ourselves.  One thing we
found counter intuitive is we actually had to specify the dummy query to
get the reaping to actually work.  We have code like:

setValidationQuery("select 1");

in our database setup code.  Not sure if you do that.

 

Scott

 

 

 

Scott Oster

 

Senior Research Specialist
Biomedical Informatics Department
Ohio State University
oster at bmi.osu.edu <mailto:oster at bmi.osu.edu> 

lab: 614-292-9845
voice mail: 614-292-8680

 

From: Blezek, Daniel J (GE, Research) [mailto:blezek at crd.ge.com] 
Sent: Thursday, June 07, 2007 11:18 AM
To: Scott Oster; dart at public.kitware.com
Subject: RE: [Dart] database/memory errors?

 

Hi Scott,

 

  This is a hard to find bug that I've been battling for ~1 year.  Seems
that the connection pooling mechanism doesn't return connections to the
pool from time to time, and the reaping process doesn't work correctly
either (it's supposed to issue a dummy query and kill the connection if
it fails).  Since we're dropping support for 1.4, I think we can try an
upgrade of the connection pooling library, and/or use the Java's
mechanism (I remember reading about it, but can't recall the specifics).

 

If this starts to happen more regularly, you could attach Eclipse (or
any other debugger) to your Dart, if you start it with the debugger
enabled and a port open.  This would be helpful for me to try to track
this down.

 

-dan

	 

	
________________________________


	From: dart-bounces+blezek=crd.ge.com at public.kitware.com
[mailto:dart-bounces+blezek=crd.ge.com at public.kitware.com] On Behalf Of
Scott Oster
	Sent: Thursday, June 07, 2007 9:26 AM
	To: dart at public.kitware.com
	Subject: [Dart] database/memory errors?

	This morning I found some submissions missing from my dartboard,
so I looked at the error log and found it filled with errors like this:

	<07 Jun 2007 07:51:19,349> [SocketListener0-4] ERROR
dart.server.Database

	        closeConnection: Dart's count: 6 Pool count: 7

	Database: 7 active 3 idle Dart's Count: 6

	Got connection:
org.apache.commons.dbcp.PoolableConnection at 9e4b4d

	
dart.server.track.TemporalTrack.getTrackId(TemporalTrack.java:105)

	
dart.server.TrackManager.getTrackIds(TrackManager.java:93)

	        dart.server.servlet.Dashboard.doGet(Dashboard.java:232)

	 

	Got connection:
org.apache.commons.dbcp.PoolableConnection at 19458f6

	        dart.server.servlet.Dashboard.doGet(Dashboard.java:146)

	
javax.servlet.http.HttpServlet.service(HttpServlet.java:596)

	
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)

	 

	 

	Restarting the server seems to have cleared up the problems .
The server (1.0.7) has been running fine for several months.  It appears
Dart was having a problem getting a connection to the database.

	 

	Looking back at the logs, the problem seems to have started
several days ago (though this morning was the first visible symptom),
when the SocketListener printed out warnings for all my projects like
"jetty.servlet.ServletHandler - Error for <project name>
java.lang.OutOfMemoryError: Java heap space"

	Following that this exeception was printed:

	[SimpleThreadPoolWorker-7] ERROR dart.server.Database  -
Something was wrong with this connection, marking it invalid.
	java.sql.SQLException: Already closed.
	        at
org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java
:77)
	        at
org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.clo
se(PoolingDataSource.java:180)
	        at
dart.server.Database.closeConnection(Database.java:217)
	        at
dart.server.Container.closeConnection(Container.java:158)
	        at
dart.server.TrackManager.reindexTracks(TrackManager.java:176)
	        at
dart.server.task.ReindexTrackTask.execute(ReindexTrackTask.java:28)
	        at
dart.server.task.ScheduledTask.execute(ScheduledTask.java:36)
	        at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
	        at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java
:520)

	 

	From that point on, the log was filled with the messages shown
at the top ("Got connection...").

	 

	Obviously the OutOfMemory has something to do with it, but Dart
continued to work fine until this morning.  If I had to guess, I'd say
for some reason the OutOfMemory is causing database connections to be
somehow corrupted and never recovered.  Searching my old logs I have
quite a few occurances of this error, though this is the first time I've
had to restart the server.

	 

	Scott

	 

	 

	 

	 

	Scott Oster

	 

	Senior Research Specialist
	Biomedical Informatics Department
	Ohio State University
	oster at bmi.osu.edu <mailto:oster at bmi.osu.edu> 

	lab: 614-292-9845
	voice mail: 614-292-8680

	 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://public.kitware.com/pipermail/dart/attachments/20070607/de0e6809/attachment.htm