Another testing success story

I’ve been re-architecting our typical .NET web application on a new project we are starting. There are some huge changes: we are adopting MVC over WebForms, Entity Framework over NetTiers, and we are finally adopting a new workflow engine.

To mitigate the risks of starting a project with all these new variables we’ve started stressing the application as soon as we could. And soon meant running load tests over a very crude application.

Web stressing an application on such an early stage poses a challenge on the team itself, as most of the time at least a couple of scenarios fail to execute properly, but this is the phase where the feedback we are getting from the load tests can deeply influence the refactoring of the architecture itself.

We’ve started the load tests with some simple CRUD and workflow scenarios. All it took was 10 virtual users and less than a minute and we’ve exhausted some critical resources on the server (database connection and workflow engine handles). Some of the Entity Framework data layer calls started failing, most of the workflow engine calls failed, and we could no longer login on the workflow engine. We were also eating up memory usage. None of these errors were identified by the development team on the development process.

After a quick code review we’ve identified a bunch of places we’ve missed to call Dispose. We could now run the same load tests with no errors, so we’ve programmed a load test to find out how many virtual users can we project so that the application server’s CPU stays below 80%. The test failed with the following error:

“Limit of 250 virtual users exceeded”

Yes, this is my kind of error! We’ve re-programmed the load test to stay within the licensing limits and left it running for the night: less of 30% to CPU usage on the application server, no error and no leaks. Cool!