I have recently run into a fun problem that I just can't get my head around. One of my rails applications is causing some problems for one of my clients. The application itself is setup with apache acting as a load balancer in front of 5 mongrel instances. Those all run on the same box as the database which is SQL Server 2005 running on a windows 2003 server.
At random times in the last week one of the mongrel instances seems to become unresponsive when talking to the database. The apache load balancer doesn't sense anything is wrong so it keeps sending requests to the hung mongrel. Of course these requests are back logged due to the single threaded nature of the mongrel server. So in essence al connections to this mongrel stop until the mongrel service is rebooted.
I have not been able to determine anything through the the logs files that points to one specific issue. Furthermore this problem is not occurring at a regular interval at all. I feel like I am trying to find a moving needle in a haystack.
I will keep this post updated as I find out more information. If anyone has any advice or ways to help isolate the hung mongrel please shoot me a comment.
