 Mib SoftwareHigh Reuse Software Development and Consulting reusable and open software. |
Some Implications of Bazaar Size
Third Draft. Aug 11, 1998Copyright 1997-1998, Forrest J. Cavalier, III Mib Software
All rights reserved. Comments welcome!
| Some Implications of Bazaar SizeThird Draft. Aug 11, 1998Copyright 1997-1998, Forrest J. Cavalier, III Mib Software All rights reserved. Comments welcome! |
| The analysis by Forrest Cavalier is silent on how open source testing is better than closed source testing. He mentions a number of "reduction factors" which apply to bazaar (open source) testing, but these same factors (and more) also apply to cathedral (closed source) testing. When I first read his comments on debugging by testing in bazaar projects, I got the impression that he was criticizing bazaar methods as compared to closed source methods. His analysis seemed to ignore the "open source" part of the Bazaar (its defining attribute, I thought) and seemed to predict more frequent failure than cathedral methods. I got the impression that the point of the argument was that the Cathedral was better than the Bazaar. After corresponding with him, I don't think he intended to do that, but rather just offer an analysis on "effective size" of a large volunteer effort. I believe that he could have more clearly indicated that open source debugging, despite the pitfalls mentioned, is better than closed source debugging. Availability of source (in open source testing by end users) enhances debugging as compared to closed source end-user testing. Here is a start at listing some of those advantages. Regarding 2.1 Factors affecting defect "installation" In "Strategies to maintain effective size", he suggests "Reduce the risk and effort to upgrading." By having the source code, you can never be stopped from re-installing an older, better-performing version, so there is a lot less risk when considering upgrading. And the biggest risk reduction in having the open source is the confidence that any potential bugs are fixable by someone. Regarding 2.2 Factors affecting defect "reachability" and "triggering" Existence of "dead code" (compile-time optional) sections is mitigated somewhat by the fact that the open source tester can try more "compile-time conditional sections" than one with closed source (who can't recompile with control over the setting of the conditionals.) Even the "dead" sections in open source can be seen by inspection. Open source testers can also do single-stepping (a powerful defect locating technique in itself.) While single-stepping through one conditional branch, it is possible to notice an error even if it is not reachable, or triggered (even if in a "dead" or inactivated branch.) A tester with source might be single-stepping through a function (on the way to an already-noticed bug perhaps) and detect a bug that otherwise would be missed by a closed source tester or a non-testing inspector. An open source tester can use the source to know learn to purposely manipulate the run conditions and program sequence to reach some bit of the program source of interest, even if it is normally run infrequently. A tester without source is blindfolded. Regarding strategies to counter low effective size" He mentions the existence of a fundamental difference between blindfolded testers and open source testers, with the statement, "Rely only on user-developers to test." In a Cathedral, this is a small group. In a Bazaar it is potentially everyone. Regarding 2.3 Factors affecting defect "detection" There's not much difference in this step between open and closed source methods. But the countering strategy "Motivate users to look for defects" happens automatically in the Bazaar (at least for testers who can potentially fix defects), since their name will get enshrined in the changes header of the fixed source file. The (properly muted) pride in such an event is probably a much greater motivation than the slim chance of being listed in the credits of the final closed source program as "one of the many indispensible beta testers." Regarding 2.4 Factors affecting defect "reporting" "Incorrect attribution to operator error" (mentioned by F. Cavalier) can be fixed by looking at ("inspection" of) the source. Open source encourages the "learned response" to become to look instead of assume the error was their own. After defect detection There are important differences which come into play AFTER a defect is reported (where the defect reporting reduction analysis of F. Cavalier stops.) If a bug report makes it back to a closed source developer group, and they don't have time, don't see the need (they may have a different notion of the program's purpose), lose the report, can't find the solution, or for any reason fail to fix the bug, the user is at a loss. But with open source, anyone in the community can (and therefore at least sometimes will) step up to the plate and fix the bug themselves. F. Cavalier mentions "once a defect is advertised, the effective size for the tasks is no longer determined by the pass-through portions of the preceding sequence." This fix-the-bug step *is* the final pass-through portion for both open and closed source projects. For an open source project, the likelihood of a defect fix after reporting is nearly 100 percent, or else it wasn't really bug that mattered to the community in the first place. Closed source projects are subject to the staffing limitations of a smaller developer pool. Additionally, the quality of error reports are also increased if the testers can inspect the source and give the project team a better pointer to the problem (perhaps pinpoint the procedure, even if the user doesn't feel competent to rewrite it himself). |