Does Sonar scale well ?
As Sonar is an enterprise quality tool, it must scale well when number of projects and snapshots by project grow over time. We consider this capability to be a plain Sonar functionality. So we manage it as the others by launching regression tests on each new release and trying to tune performance from one release to the other.
Ok fine, but what means "scale" in Sonar's context and what is the protocol used to measure this capability ? As a good coach, we decided to answer this question by asking new ones :
What could be called a reference project ?
- We decided to select Struts as this reference project with its 500 classes & 30'000 lines of code.
What could be considered as a significant number of history snapshots ?
- We decided to fix this value to 150. It emulates 1 cycle per day for 6 months or 1 cycle per week for 3,5 years
Which measures are relevant for Sonar users ?
- Collecting time : the time used by the sonar maven plugin to statically and dynamically analyze source code and to inject data into Sonar database
- Consolidation time : the time used by sonar web module to aggregate information provided by the sonar maven plugin
- Size of the Sonar database
- Memory used by Sonar web application
- Loading time of the Sonar home page
- Size of the Sonar home page
When do we want to do those measures and show how performance evolve when number of projects grows ?
- For 1 then 50 then 100 and finally when 200 projects are placed under quality control with Sonar
How can we automate "at best" this protocol to be able to play regression tests for each release at low cost ?
- We've developed a mechanism which is able to make a two dimensional replication of the reference project in the Sonar database.
You can take a look to the detailed description of this protocol and the evolution of each measures from one version to the other, but the final conclusion is :
Sonar is ready to welcome 200 projects having each one 150 snapshots. It can certainly do more but we haven't do any tests over this limit for now.
In addition to this benchmark, a public instance of Sonar called Nemo (http://nemo.sonar.codehaus.org) is up and running and is a way to show you that Sonar really scales with hundred of Open Source projects placed under quality control.