Expecting Rain

I've been rather silent the last few days. With good reason, I've been dealing with a crazy bug in a piece of software our group wrote. The application is 400,000 lines of code long.

Many times as I've searched for bugs in code I get to the point where I think it is 3rd party software only to be proven wrong -- that it is my software. It's always embarrassing when I make the statement that it isn't my problem, but someone else's only to be wrong. Once again, I believe the problem is elsewhere.

Here is the scenario. We have an application which makes many requests to a REST styled web service. We have many clients who run this application. We have noticed that randomly one client will get another client's response. It doesn't happen often, it isn't hardly noticeable, but it is wrong. We are using Tomcat as our web server, our web service is powered by a servlet which I believe isn't handling concurrent requests. To test this, I created a simplified version of our servlet. I pass two parameters to this service, a client id and a random string. The servlet then does a MD5 hash of that string and passes it back along with the client id and the original random string. I'm basically just doing some processing to take up a cpu cycle or two. I have a client with creates 8 different threads, each with their own client id. I then begin make a huge number of requests to the servlet and check to make sure the response's client id matches the requester's client id. It fails after about 1000 requests.

I'll keep looking through this problem and see if I can find a solution.

As a side note, I have the same "servlet" running as a ruby cgi script, it does the same thing, being ran from my same client. I'm at my 16000 request without an error...