IT analysts’ credibility gap

If nothing else, the recent public squabble involving the Gartner Group, IDC
and the Open Source community should wake up investors and business leaders
to one very big, very unhappy fact: A lot of professional market
research firms do incredibly sloppy work. Many of them use “research methods”
that wouldn’t pass muster in a first year marketing or statistics class.
And, far from being objective analysts of marketing trends, in many cases,
their opinions are for sale.

Take note, you company managers who are just about to go over the
waterfall because a “projected” market never showed up, and you investors who
are watching your IRAs crumble into dust. You may have been
had.

If you haven’t been following NewsForge over the last few days you
may not have heard that — despite evidence at your company (and mine, and
every firm I deal with) — there aren’t any Linux servers out there, or hardly
any. That’s according to research done by Jeff Hewitt, an
analyst with Dataquest, a business unit of the Gartner Group. In a
report widely
leaked to the press courtesy of Microsoft, Hewitt claimed
that his research showed that Linux constituted less than 9% of the server
market.

This number doesn’t square with anecdotal reports of “viral” growth
of the Linux server market, and it certainly contradicts reports
generated by Dataquest chief competitor, IDC. IDC and its high-profile
analyst Dan Kusnetzky have published estimates that show Linux with a market
share roughly three times the size of the Dataquest numbers. More than six
months ago, Kusnetzky was claiming that Linux held better
than 24% of the server market.

Something is clearly wrong with at least one research company’s
methodology, and it is a good illustration of why product developers, sales people
or stock promoters who cite third-party commercial research as the sole
basis for doing anything ought to be dismissed almost immediately.

Hewitt has already had to defend his research. His methodology
consisted of telephone interviews with 724 “responsible” spokespersons. He
conducted reasonably valid statistical methodology; his sample was
over-represented with interviews of large mainframe and supercomputer users, and
under-represented by small systems users. He balanced these numbers using traditional
estimates of the installed base of these various platforms. What he didn’t
do was certify whether his respondents were in an effective position to
know about systems installed in-house or validate his assumption that
the number of pre-configured units and bare-bones “white” systems predicted
the installed base of Linux servers at corporate sites. These are both
the type of data collection error that would get you into trouble in
any college market research course.

Linux breaks a lot of rules, one of which is that very few Linux
servers began life that way. In most of the organizations I’ve worked with,
Linux boxes have been converted from something else — buffed up and
placed into service to fill a hole as an intranet box or workgroup machine.
They were almost never “purchased” servers. None of the Linux machines at
one site I work at were ever “purchased” — they were recycled, configured
with Linux and used to fill in until management decided what to do in the
interim, if they ever figured it out at all.

Kusnetzky tells a similar
story. While doing a survey, he was told by the CFO of a
major New York bank that the firm wasn’t using and didn’t plan to use any
Linux servers. According to the company’s IT staff, there were more than 100
servers in the company running Linux for various server applications. Hewitt
should know this. It is this type of preliminary research that is missing from
Hewitt’s research and one of the reasons his conclusions seem so wrong.
If Dataquest’s survey of “responsible people” didn’t take this
into account — if his respondents included no engineers but just CFOs or
other “responsible” corporate officials, then its a certain bet that he
managed to miss 90% of the machines out there.

This is not to state that Hewitt is wrong or that his research is
necessarily any worse than that of his competition. It is true that of all the
analysts in IDCs stable, Kusnetzky has one of the best reputations. Unlike a
lot of analysts, he is not a Pollyanna telling customers what they want
to hear. In the past year, he practically told Novell it no longer had
a viable business. He’s also got a reasonably solid track record. Last
fall he predicted
the decline in server sales with a greater degree of
precision than anyone else. And he’s not a Linux shill. He’s been less than
shy about questioning Linux as a business model, suggesting that Linux
software was likely to generate very low financial numbers into
the foreseeable future.

But that doesn’t mean that his high Linux numbers are solidly based
in fact either. As early as 1999, Kusnetzky admitted Linux was “a
market that has been really difficult for us to get our arms
around.” He told CNN, “We have no real way to track the number of copies
of Linux that are installed.” Coming from an analyst who has generated
some fairly precise numbers and percentages, that should be a concern.
Kusnetzky has said himself that all “we can keep track of is the money generated
by commercial shipments.” These are precisely the numbers that
Hewitt claims to be following.

So what to make of all this? I’d suggest caution and lots of it.
While IDC has made a real effort not to tell businesses what they want to
hear, nearly all commercial market research firms spend far more time
marketing their research than collecting it. Many of the survey designers are
young and not very experienced. Research information has a short shelf life
and only rarely do market research firms get confronted with their errors.
This makes for a very sloppy research environment.

Further, many companies are too deeply involved with their clients
to provide stiff analysis. Most companies shopping for market research are
looking for good news, not bad. It’s a rare marketing firm that goes
broke telling clients what they want to hear, and many times analysts depend
heavily on the firms they claim to analyze. For example,
Dataquest been very heavily involved in promotional events such as one held last
year that plugged data storage systems. The conference was underwritten by storage
vendors who heavily promoted Dataquest “research.” The company’s “analysts”
dominated the list of speakers at that event. If you believe that the “research”
presented at that show was objective, I have a bridge in Brooklyn I’d
like to sell you.

So it is important to understand that you need to do some of your
own market research and not rely exclusively on third parties. If market
reports don’t seem to make sense, the reason may be that they are flat out
wrong.

Mr. Kusnetzky’s name was misspelled “Kuznetsky” in this column when it first appeared. The mistake has been corrected. – Robin Miller, EiC