KDE Quality Assurance Meeting Report

On the weekend of December 10th & 11th, a small group of nine KDE contributors met in Hamburg to work on quality assurance checks for KDE's code base. This not only covered C++ sourcecode but also other aspects such as checking the state of the API documentation, looking for common errors in KDE's manuals and evaluating the usability of KDE applications. Read on for the full report.

Hosted by the kind folks of froglogic, saturday started with people arriving at the froglogic office and saying hello to each other. As people arrived, it became clear that there was some interest in looking to what degree froglogic Squish would be useful for testing the user interface of KDE applications, so a small Squish demonstration took place. Also, the original project (the infamous English Breakfast Network) which started the whole idea of having a QA meeting wasn't known to some people so a laptop was attached to the projector and a short "Browsing around the EBN" session was held. It was a nice opportunity to show off how the results of (different) automatic checks could be presented in a reasonably uniform manner.

After having some food, Ellen Reitmayr of OpenUsability.org fame was kind enough to educate the rest of us about usability. The fact that a whole bunch of factors which influence the usability of the user interface can easily be checked automatically came as a relieving surprise (so the idea wasn't all that crazy after all!). During her explanations, it became apparent that even though many things are terribly difficult to check automatically it would be a big achievement if we could have nightly checks for the low hanging fruits. Getting them out of the way (and there are a lot of them) would give the usability people more time to focus on the things which actually require human intervention.

The rest of the afternoon was spent on letting ideas for checks bounce off each other, brainstorming about the feasibility of automatic checking for such things (just imagine people occasionally moaning something like "You can't possibly do this and that automatically!" or "Such a tool would give so many false hits, it'd clutter the output and make the valid reports about issues harder to spot!"). A little bit of hacking was done as well (talking about and looking into the sources of different C++ parsers) - in particular, KDevelop developer Harald Fernengel hacked up an QTestLib-based test based on Dirk's binary compatibility checker (unfortunately nobody seems to know about 'gmake bcheck') which checks whether a given interface composed of a number of C++ classes is binary compatible to another interface, which would be useful for KDE's libraries (so that your KDE applications don't break after upgrading KDE's libraries to a new minor release because of binary incompatibility). In the evening, people had food together and then visited Hamburg's central Christmas fair, having some of the infamous German Glühwein together. :-)

While Saturday was much more about talking and exchanging ideas, a lot more work got done on Sunday. Reginald hacked up a little Squish script to show off how Squish can be used to automatically check the menus of different KDE applications for some of the usability points Ellen mentioned the day before (the picture of Squish opening and closing all menus available in an application - and the fact that one script was used to test different applications - caused some "oh!" and "ah!" and nodding among the viewers). Adriaan (who runs the English Breakfast Network) was given a free Squish license so that you should see the results of various usability tests showing up on the web pages. Work was also done on porting the QTestLib-based tests in KDE's libraries to the new QTestLib as stored in the qt-copy module of KDE's repository. In general, Sunday's soundscape was much influenced by the sound of people typing on their laptop keyboards. The day finished with a dinner together, after which the first persons left to venture back to the mundane world.

A few noteworthy things we got out of this weekend:

• Two days were not enough. KDE has no real centralised testing framework in place, so a lot of time was spent on discussing very basic problems.
• Usability is a lot less diffuse a term than most people think. A lot of things which make up the usability of a program (as defined by the KDE HIG) can indeed be checked automatically. Having a Squish license for running Squish on the EBN so that we can start generating the reports is a very nice achievement.
• There doesn't seem to be a useful and free static C++ analyser, which is a pity. Maybe using Roberto Raggi's C++ parser to do something like that would be worthwhile, but it is still a lot of work.
• How to present whatever results we get out of any checks and integrating that information into the workflow of volunteers working on KDE is a tricky issue - having five-digit numbers of issues on the front page easily scares potentially helping hands away. This is indeed a social problem, and we could have spent the entire weekend on just evaluating different approaches at how to present rules appropriately.
• Visiting the Reeperbahn in Hamburg on a Sunday night is quite boring.