My general experience of static analysers is that, almost without exception, it's more trouble than it's worth to apply one to an existing codebase.
A static analyser will tend to impose some set of fairly arbitrary rules on your code; generally those rules will be only implicitly specified, and they'll boil down to 'Don't do anything that this particular analyser's algorithms can't understand well enough to be confident of'.
When I say 'arbitrary', I don't mean to imply that the rules are completely without value; probably you will manage to write safer code if you can stick within the rules, for any ruleset implemented by a not-totally-hopeless analyser. It's just that the rules won't be the only possible way to keep yourself safe; there will be other sets of rules which would also keep you reasonably safe, none of which is a strict superset of any other.
Your existing codebase – assuming you have a nontrivially large one to which you're considering retrospectively applying an analyser – will almost certainly give a zillion warnings when you test-run the analyser. And as you go through them, you'll find yourself getting a strong feeling of 'But, dammit, that's perfectly safe, why can't you see that?', because whoever wrote the code was working to one of those other sets of implicit keeping-yourself-safe rules, and the real complaint is not that the code is unsafe, but just that it doesn't happen to match this analyser's particular set of guidelines for keeping yourself safe.
If you don't have an existing codebase at all, and you're planning to write a fresh load of code right from scratch, that makes life much easier. In that situation, you can commit up front to keeping the code clean under some particular static analyser(s); you can fix the complaints as they come up while you're writing the code in the first place, and there won't be too many of them at a time. Probably quite a lot of analysers will give you some actual benefit, if run in that mode.
But if you do have an existing codebase which has never previously gone through a particular static analyser, then I'd almost always bet on you getting five zillion reports of 'problems', nearly all of which will strike you as deficiencies in the analyser, and you'll almost certainly get the strong feeling that fixing them all in one big blitz would introduce more bugs than it solves.
The one exception I've ever seen is Coverity, which is specifically tuned to have a low false-positive rate. I've run that over previously un-staticked code of my own (namely PuTTY), and I found it to have a false positive rate of about 1 in 3, which is higher than Coverity's marketing would have you believe (I think they say more like 1 in 10), but by the standards of any other analyser I've ever seen the output of, 1 in 3 is still very, very good, and made it well worth trawling through the whole set of problem reports and fixing all the sensible ones.
But the flip side is that one of the ways Coverity lowers its false positive rate is by giving you a lot of benefit of the doubt. It will shout about things it's really sure of, of course, but a lot of the rest of the time, it will try to infer from your own code on the assumption that you knew what you were doing. E.g. if you test a pointer for NULL before use in 18 cases but miss the check in the 19th, it'll point out the discrepancy; but if you never tested the pointer for NULL, then it will assume you had a good reason for believing you didn't need to – even if you didn't really. So if you're after a probabilistic reduction in bug rates, Coverity is probably worth a go; but if what you wanted from a static analyser was some kind of guarantee or near-guarantee that any code which gets through it is almost certainly free of a given class of bug, then that's one thing Coverity won't give you.
(Also, I found Coverity's various checkers to be quite variable in quality, with the least critically important ones doing the best job. Its absolute sweet spot, for me, was spotting resource leaks on error handling paths, where you take an early return from the middle of a complicated function and miss out a free or fclose or some such; but the checkers designed to report exploitable security holes did a much more patchy job, flagging up a lot more false positives and not finding much of value.)
TL;DR: Coverity will probably do you some good if you're prepared to pay for it and don't demand guarantees. Everything else I've ever seen (apart from the odd completely useless one) might be useful if you wrote a codebase from scratch taking care to keep it clean, but likely to be more trouble than it's worth to retrospectively apply to any nontrivial amount of existing code.
no subject
Date: 2015-11-02 01:05 pm (UTC)A static analyser will tend to impose some set of fairly arbitrary rules on your code; generally those rules will be only implicitly specified, and they'll boil down to 'Don't do anything that this particular analyser's algorithms can't understand well enough to be confident of'.
When I say 'arbitrary', I don't mean to imply that the rules are completely without value; probably you will manage to write safer code if you can stick within the rules, for any ruleset implemented by a not-totally-hopeless analyser. It's just that the rules won't be the only possible way to keep yourself safe; there will be other sets of rules which would also keep you reasonably safe, none of which is a strict superset of any other.
Your existing codebase – assuming you have a nontrivially large one to which you're considering retrospectively applying an analyser – will almost certainly give a zillion warnings when you test-run the analyser. And as you go through them, you'll find yourself getting a strong feeling of 'But, dammit, that's perfectly safe, why can't you see that?', because whoever wrote the code was working to one of those other sets of implicit keeping-yourself-safe rules, and the real complaint is not that the code is unsafe, but just that it doesn't happen to match this analyser's particular set of guidelines for keeping yourself safe.
If you don't have an existing codebase at all, and you're planning to write a fresh load of code right from scratch, that makes life much easier. In that situation, you can commit up front to keeping the code clean under some particular static analyser(s); you can fix the complaints as they come up while you're writing the code in the first place, and there won't be too many of them at a time. Probably quite a lot of analysers will give you some actual benefit, if run in that mode.
But if you do have an existing codebase which has never previously gone through a particular static analyser, then I'd almost always bet on you getting five zillion reports of 'problems', nearly all of which will strike you as deficiencies in the analyser, and you'll almost certainly get the strong feeling that fixing them all in one big blitz would introduce more bugs than it solves.
The one exception I've ever seen is Coverity, which is specifically tuned to have a low false-positive rate. I've run that over previously un-staticked code of my own (namely PuTTY), and I found it to have a false positive rate of about 1 in 3, which is higher than Coverity's marketing would have you believe (I think they say more like 1 in 10), but by the standards of any other analyser I've ever seen the output of, 1 in 3 is still very, very good, and made it well worth trawling through the whole set of problem reports and fixing all the sensible ones.
But the flip side is that one of the ways Coverity lowers its false positive rate is by giving you a lot of benefit of the doubt. It will shout about things it's really sure of, of course, but a lot of the rest of the time, it will try to infer from your own code on the assumption that you knew what you were doing. E.g. if you test a pointer for NULL before use in 18 cases but miss the check in the 19th, it'll point out the discrepancy; but if you never tested the pointer for NULL, then it will assume you had a good reason for believing you didn't need to – even if you didn't really. So if you're after a probabilistic reduction in bug rates, Coverity is probably worth a go; but if what you wanted from a static analyser was some kind of guarantee or near-guarantee that any code which gets through it is almost certainly free of a given class of bug, then that's one thing Coverity won't give you.
(Also, I found Coverity's various checkers to be quite variable in quality, with the least critically important ones doing the best job. Its absolute sweet spot, for me, was spotting resource leaks on error handling paths, where you take an early return from the middle of a complicated function and miss out a free or fclose or some such; but the checkers designed to report exploitable security holes did a much more patchy job, flagging up a lot more false positives and not finding much of value.)
TL;DR: Coverity will probably do you some good if you're prepared to pay for it and don't demand guarantees. Everything else I've ever seen (apart from the odd completely useless one) might be useful if you wrote a codebase from scratch taking care to keep it clean, but likely to be more trouble than it's worth to retrospectively apply to any nontrivial amount of existing code.