[Host's Note: Debbie sent me a note after I had distributed LO15712 saying
she had hit the send key too early and had more to say. Here's the
additional. ...Rick]
Here's the rest - sorry for the extra burden.
...I would have to "trifurcate" the original query to address it:
1. how well is the information technology (IT) project
completed relative to IT PRODUCTION CRITERIA - e.g.,
code efficiency and elegance, etc.?
2. how well does the product satisfy the initial goals
and specifications? DOES IT DO WHAT IT WAS SUPPOSED TO?
3. what does the project contribute to the organization's
BOTTOM LINE?
These are very different, and each has its own evaluation needs and, if
you're lucky, promising evaluation methodologies.
1. an example of an IT production methodology is the Pittsburgh Software
Engineering Institute's function point system which improves on the purely
quantitative output measure of thousands of lines of code (KLOC) written
- might be worth asking them about that and more suggestions for project
evaluation techniques. James Martin and others whose writing touches on
software engineering, e.g., Ed Yourdon, would cover relevant issues here.
2. I see the goal attainment question as a typical business process one
where evaluation procedures are not specific to IT.
3. This is the biggie - what did the organization get? Many organizations
don't even know how many computers they have, so how could they figure out
what they get from IT? Thomas K. Landauer published a book in 1995 called
THE TROUBLE WITH COMPUTERS: USEFULNESS, USABILITY, AND PRODUCTIVITY. (And
of course MIT's Nobel prize winner Robert Solow is widely quoted as saying
'we see computers everywhere except in the productivity statistics' - IT
spending keeps on escalating and explanations of the "productivity
paradox" with it.)
Paul Strassmann has done a lot of work on this subject and might be worth
a look as input to fashioning your approach. His most recent book is THE
SQUANDERED COMPUTER - EVALUATING THE BUSINESS ALIGNMENT OF INFORMATION
TECHNOLOGIES (1997). One of his points that makes a lot of sense to me is
looking at Return on Management rather than investment; what you get from
IT depends most on how you use it, not on the box or its cost. He wrote in
COMPUTERWORLD a couple years ago that based on a survey of our largest
firms, 82% (if this memory has it right) spent more on IT than they got
from it. He has a web site if you want to explore.
Chris Kemerer and Lorin Hitt at MIT have published interesting material on
value derived (or not) from IT. Their articles might be of indirect
interest, and a call to their office might get you to more applied
resources at MIT.
Hope this discourse is of some use and/or interest, even though it doesn't
neatly answer the query. There must be tons of practitioners out there who
know eons more than I. Failing a credible metric, benchmarking against
your competition can be a useful tool.
Last but far from least, I would want my evaluation process to extend not
only to indirect benefits but especially to unexpected ones. I've seen it
said that American Airlines makes more from its Sabre reservation system
(IT) than it does from transporting humans and their possessions.
Regards - Debbie Roth
--Debbie Roth <dr@sprintmail.com>
Learning-org -- An Internet Dialog on Learning Organizations For info: <rkarash@karash.com> -or- <http://world.std.com/~lo/>