[wordup] Is software accurate ...

[Adam Shand]

Via: Simon Horsburgh <simon.horsburgh at stonebow.otago.ac.nz>
More: http://www.agresearch.cri.nz/Science/Statistics/exceluse1.htm

From: Adam Warner <usenet at consulting.net.nz>
Newsgroups: nz.comp

Hi all,

I thought some of you might be interested in the journal article "Is it
safe to assume that software is accurate?" by B. D. McCullough (2000),
International Journal of Forecasting, 16, 349-357.

The author's conclusion is that it is not safe to assume that statistical
and econometric software packages are accurate. The short literature
review uncovered packages that are so inaccurate that they give incorrect
answers to simple textbook problems. Poor random number generators are
also mentioned, and how one ruined a Monte Carlo study (See here for info
about Monte Carlo studies: http://csep1.phy.ornl.gov/mc/node1.html)

The article goes on to discuss why user friendliness is valued more than
accuracy. It's stated that it's simply this consequence: "Developers do
not supply accuracy because the vast majority of users do not demand it."

   To perform sophisticated statistical analyses, users need possess
   knowledge of little more than how to point and click a mouse.... These
   same users are shocked, simply shocked, to discover that computer
   numbers are not exact, and the phrase `cumulated rounding error' is
   wholly foreign to them.

And some users who point out errors have discovered little concern for
correcting them by some vendors:

   Many of those errors persisted through subsequent versions of the
   software, and some of those errors remain uncorrected even today, nine
   years after developers were first informed.

   ...

   Of course, some developers are extremely responsive to even unpublished
   errors, fixing them almost immediately. The problem is that the user
   who cares about accuracy has no idea whether his developers falls into
   the former category or the latter.

There is a vivid illustration of problems with packages that tout double
precision computations but only have single precision graphics. Data
points can just disappear off a graph.

In the section "6. What can be done?" McCullough argues that "Journals and
researchers should seriously consider actively encouraging replicable
research" and data/code archives.

   For researchers who do not write clean, clearly commented code and
   otherwise do not adhere to the standard of replicable research, the
   cost will be substantial--but it should be. The purpose of replication
   has always been to maintain high quality research.


When you consider how many important economic, engineering, and social
planning decisions are made on the basis of forecasting and statistical
techniques it's troubling to discover how inaccurate critical pieces of an
academic/technican's toolkit may be.

Regards,
Adam