[Dart] database/memory errors?

Thu Jun 7 16:04:30 EDT 2007

Hey Scott,

  I just updated to DBPC 1.2.2, which has a number of bug fixes.  I've
also added support for the reaper thread, so (perhaps) you can configure
this if you continue to run out of connections.  The documentation has
also been updated.

  This is revision 423 in the Dart subversion repository.

-dan

________________________________

	From: dart-bounces+blezek=crd.ge.com at public.kitware.com
[mailto:dart-bounces+blezek=crd.ge.com at public.kitware.com] On Behalf Of
Scott Oster
	Sent: Thursday, June 07, 2007 3:23 PM
	To: dart at public.kitware.com
	Subject: RE: [Dart] database/memory errors?

	We've had some oddities with commons pooling ourselves.  One
thing we found counter intuitive is we actually had to specify the dummy
query to get the reaping to actually work.  We have code like:

	setValidationQuery("select 1");

	in our database setup code.  Not sure if you do that.

	Scott

	Scott Oster

	Senior Research Specialist
	Biomedical Informatics Department
	Ohio State University
	oster at bmi.osu.edu <mailto:oster at bmi.osu.edu> 

	lab: 614-292-9845
	voice mail: 614-292-8680

	From: Blezek, Daniel J (GE, Research) [mailto:blezek at crd.ge.com]

	Sent: Thursday, June 07, 2007 11:18 AM
	To: Scott Oster; dart at public.kitware.com
	Subject: RE: [Dart] database/memory errors?

	Hi Scott,

	  This is a hard to find bug that I've been battling for ~1
year.  Seems that the connection pooling mechanism doesn't return
connections to the pool from time to time, and the reaping process
doesn't work correctly either (it's supposed to issue a dummy query and
kill the connection if it fails).  Since we're dropping support for 1.4,
I think we can try an upgrade of the connection pooling library, and/or
use the Java's mechanism (I remember reading about it, but can't recall
the specifics).

	If this starts to happen more regularly, you could attach
Eclipse (or any other debugger) to your Dart, if you start it with the
debugger enabled and a port open.  This would be helpful for me to try
to track this down.

	-dan

________________________________

		From: dart-bounces+blezek=crd.ge.com at public.kitware.com
[mailto:dart-bounces+blezek=crd.ge.com at public.kitware.com] On Behalf Of
Scott Oster
		Sent: Thursday, June 07, 2007 9:26 AM
		To: dart at public.kitware.com
		Subject: [Dart] database/memory errors?

		This morning I found some submissions missing from my
dartboard, so I looked at the error log and found it filled with errors
like this:

		<07 Jun 2007 07:51:19,349> [SocketListener0-4] ERROR
dart.server.Database

		        closeConnection: Dart's count: 6 Pool count: 7

		Database: 7 active 3 idle Dart's Count: 6

		Got connection:
org.apache.commons.dbcp.PoolableConnection at 9e4b4d

dart.server.track.TemporalTrack.getTrackId(TemporalTrack.java:105)

dart.server.TrackManager.getTrackIds(TrackManager.java:93)

dart.server.servlet.Dashboard.doGet(Dashboard.java:232)

		Got connection:
org.apache.commons.dbcp.PoolableConnection at 19458f6

dart.server.servlet.Dashboard.doGet(Dashboard.java:146)

javax.servlet.http.HttpServlet.service(HttpServlet.java:596)

javax.servlet.http.HttpServlet.service(HttpServlet.java:689)

		Restarting the server seems to have cleared up the
problems .  The server (1.0.7) has been running fine for several months.
It appears Dart was having a problem getting a connection to the
database.

		Looking back at the logs, the problem seems to have
started several days ago (though this morning was the first visible
symptom), when the SocketListener printed out warnings for all my
projects like "jetty.servlet.ServletHandler - Error for <project name>
java.lang.OutOfMemoryError: Java heap space"

		Following that this exeception was printed:

		[SimpleThreadPoolWorker-7] ERROR dart.server.Database  -
Something was wrong with this connection, marking it invalid.
		java.sql.SQLException: Already closed.
		        at
org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java
:77)
		        at
org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.clo
se(PoolingDataSource.java:180)
		        at
dart.server.Database.closeConnection(Database.java:217)
		        at
dart.server.Container.closeConnection(Container.java:158)
		        at
dart.server.TrackManager.reindexTracks(TrackManager.java:176)
		        at
dart.server.task.ReindexTrackTask.execute(ReindexTrackTask.java:28)
		        at
dart.server.task.ScheduledTask.execute(ScheduledTask.java:36)
		        at
org.quartz.core.JobRunShell.run(JobRunShell.java:203)
		        at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java
:520)

		From that point on, the log was filled with the messages
shown at the top ("Got connection...").

		Obviously the OutOfMemory has something to do with it,
but Dart continued to work fine until this morning.  If I had to guess,
I'd say for some reason the OutOfMemory is causing database connections
to be somehow corrupted and never recovered.  Searching my old logs I
have quite a few occurances of this error, though this is the first time
I've had to restart the server.

		Scott

		Scott Oster

		Senior Research Specialist
		Biomedical Informatics Department
		Ohio State University
		oster at bmi.osu.edu <mailto:oster at bmi.osu.edu> 

		lab: 614-292-9845
		voice mail: 614-292-8680

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://public.kitware.com/pipermail/dart/attachments/20070607/0b9ce6fc/attachment-0001.htm