The Syndrome of Useless Information

I occasionally elevate comments and replies to the level of full posts if I feel they better illustrate and clarify significant points I raise.

In my Jan. 9, 2010 post "Does the CEO of Google Use Google? - And: Platform, Platform, Who's Got The Platform?" I lamented the intrusion of ill-informed, reductionistic, "database platform"-centric views of non-clinical IT personnel into healthcare.

I received the following feedback from an anonymous commenter:

Anonymous wrote:

Or then again, there's this:

"Non-physicians may reach correct diagnoses by using Google: a pilot study."

http://www.ncbi.nlm.nih.gov/pubmed/19130327

"Researchers found that almost six-in-10 difficult cases can be solved by using the world wide web as a diagnostic aid."

http://www.dailymail.co.uk/news/article-415562/Doctors-using-Google-diagnose-illnesses.html

I think this post [i.e., my HC Renewal post about Google's CEO and "platforms" - ed.] misses the larger point that Gawande and Schmidt were addressing (and to some extent agree on) - the "art" of the practice of medicine must be transformed to the science of the practice of medicine. And today, science and IT are necessarily intertwined in practical terms.

The future will no doubt involve ever more sophisticated and useful clinical DSS. Historically speaking, invoking the complexity argument is fraught with issue as technological advances provide better and better ways to represent and manage it.

January 16, 2010 10:03:00 AM EST

Anonymous apparently meant these article references to exemplify the coming Age of Cybernetic Miracles in medicine.

My response to this anonymous feedback was placed in the comment thread of the "Does the CEO use Google" post and covered a number of bases.

I've reproduced my response, and added additional explanatory notes not in my original response in bold red italics below:

MedInformaticsMD wrote:

Re: Anonymous Jan 16 @ 10:03 AM-

The article [the first, at nlm- ed.] you cite reports the following:

Non-physicians may reach correct diagnoses by using Google: a pilot study.

OBJECTIVE: We endeavoured to determine whether individuals who are not physicians are likely to arrive at correct diagnoses [note the stated objective carefully -- "are likely to arrive at correct diagnoses" - ed.] by using Internet resources.

METHODS: In this prospective study four non-physicians used Google to search for diagnoses. They reviewed the 26 diagnostic cases presented in the case records of the New England Journal of Medicine during 2005; they were blind to the correct diagnoses. The main measurement was the percentage of correct diagnoses arrived at by non-physicians by using Google. The diagnostic success of the four non-physicians was compared to that of four young physicians.

RESULTS: The average diagnostic success of non-physicians was 22.1% (95% confidence interval [CI] 4.5-39.7%). There was no statistically significant difference between the non-physicians regarding this outcome (p = 0.11). They took 8.9 +/- 6.7 (mean +/- standard deviation) minutes for case record reading and 17.4 +/- 7.9 minutes for Google searching per case. Non-physicians performed worse than physicians (50.9% [95% CI 37.4-64.5%]) in regard to diagnostic success (p <0.001).

CONCLUSION: Non-physicians, at least those who have similar characteristics to the participants in the present study, may occasionally reach correct diagnoses by performing a brief web-based search. Doctors should realise that patients may assume a more active role in their health decision-making process and take this development into consideration in physician-patient interaction.

--------------------------

This article suffers from what can be termed "the syndrome of useless information."

[It is, in fact, a failure or negative finding with regard to its stated objective - ed.]

Let's see:

The average diagnostic success of non-physicians was 22.1%

They omit the opposite semantic: that the google-armed nonphysicians got more than three quarters of the diagnoses WRONG. Will you trust your grandmother to them?

Non-physicians performed worse than [young, a.k.a. inexperienced] physicians (50.9% [95% CI 37.4-64.5%]) in regard to diagnostic success (p <0.001).

Young, inexperienced physicians (trainees or residents?) arriving at 51% of the diagnoses correctly using ONLY google (not, for example, Harrison's, and other standard medical texts that would not be highly usable to most non-physicians lacking background to fully understand it) is not unexpected.

What is surprising is that the article omitted an essential control group: *experienced* physicians. It's not that they're hard to find.

The article concludes:

Non-physicians, at least those who have similar characteristics to the participants in the present study, may occasionally reach correct diagnoses by performing a brief web-based search.

Occasionally reach correct diagnoses? How about "most often reach incorrect diagnoses?" Cf. a broken clock is still occasionally correct.

[I repeat, the article is, in fact, a failure or negative finding with regard to its stated objective of "determining whether individuals who are not physicians are likely to arrive at correct diagnoses." They are in fact unlikely to get it right, with only a 22% hit rate, although the authors appear to have de-emphasized that fact. This will only make physicians' work harder as they "take this development into consideration", i.e., deal with patients armed with search engine-gleaned misdiagnoses - ed.]

I do not view this article as revealing anything of practical value other than perhaps the dangers of allowing non-physicians armed with search engines to think they can perform medical diagnosis to any meaningful extent.

Anonymous wrote: the "art" of the practice of medicine must be transformed to the science of the practice of medicine. And today, science and IT are necessarily intertwined in practical terms.

Reductionist views of non-medical IT personnel spouting off about 'database platforms' will not advance the science or the art of medicine.

Also, yes, IT is a tool of science; however, it and its designers and implementers facilitate science; the enablers of science are: scientists, using their insight, creativity, ingenuity, experience and expertise.

Concerning the second article [at dailymail.co.uk - ed.], I note:

"But they [the authors] stress the efficiency of the search and the usefulness of the retrieved information depend on the searchers' knowledge base." [The 'searchers' in this case were apparently - experienced physicians - ed.]

In other words, search engines can facilitate experts. That is not exactly new knowledge.

The 'art' of medicine, i.e., judgment, likewise is both indispensible, and irreproducible via "database platforms." Perhaps one day with advances in cognitive computing we will get to that point, but at present we can't even do as well as a cat. Note the statement from the IBM P.I. that "there are no computers that can even remotely approach the remarkable feats the mind perform."

Until we get there, I think it's not unreasonable to hold off on non-clinicians touting reductionist information retrieval-centric views.

Anonymous wrote: Historically speaking, invoking the complexity argument is fraught with issue as technological advances provide better and better ways to represent and manage it.

You must not have read my original post. It is advances in information science (i.e., in informatics) that will provide those advances, not advances in [information] technology. IT is a tool; information science is an activity of the mind.

Your statements clearly demonstrate a conflation of information technology and information science.

Computers facilitate information science research, but they are certainly not its sine qua non.

I should also add that the second article referenced by "Anonymous" refers to "obscure conditions such as Cushing's syndrome." I'm not exactly sure to whom Cushing's disease is an obscure disease. A non-medical IT person, perhaps?

Now, back to reality. Computers serve as aids to clinicians, when the IT is "done well." However, there is no substitute -- except in sci fi -- for expertise.

Medical expertise can only come from ~4 years of premed and then 4+ years of hard medical study in a wide variety of preclinical and clinical sciences leading to the M.D. degree, 3-4 years of postgraduate residency, and often 2+ additional years of postdoctoral fellowship training beyond residency for specialists (it should be noted, unfortunately, that there are no additional medical degrees beyond M.D., although residency and fellowship training often makes pursuing a Ph.D. in most fields seem like a cakewalk). Add to all that years of additional clinical experience in actual practice.

I am quite disturbed that this study and experience has become so casually regarded outside of medicine, for example in the IT sector. I think it reflects poorly on the IT culture specifically, and on our current culture as a whole.

Medical judgment is borne of that study and experience, not of healthcare computer dabbling typical of the business IT/MIS world in hospitals, and the all-too-often technical bachelor's degrees held by leaders in that domain. Google CEO Schmidt at least is a true computer scientist with a doctorate in CS; that does not, however, qualify him to comment on medical and medical informatics-related issues as in my prior post.

Stated frankly: he and many others with similar views are venturing far outside their competencies and as a result are talking nonsensically.

Incidentally, I believe it's time to move on from continually responding to the non-medical, IT-personnel proffered platformorrhea on how IT databases, decision support, artificial intelligence etc. will transform medicine from an art to a science, revolutionize medicine, and similar positions trafficked for at least several decades now.

They are largely manifestations of the Syndrome of Inappropriate Overconfidence in Computing (SICC syndrome), a term I coined in the 1990's and described is some detail here in a post entitled "A 21st Century Plague? The Syndrome of Inappropriate Over-Confidence in Computing." At that post I stated that:

I, for one, would welcome a cessation of claims that IT will "revolutionize" any field that depends primarily on cognition, such as biomedicine, and a return to more temperate attitudes instead of the almost bellicose grandiosity about HIT we see today. That is to say, that HIT - with proper contributions from the aforementioned specialties [e.g., social science, social informatics, biomedical informatics, HCI, etc.] will facilitate better health care, not "revolutionize" it.

In the future, similar comments as those from "Anonymous" will simply get referred back to the post you are now reading and the bolded one on SICC above.

Finally, I will not hold my breath for Emergency Medical Hologram Mark I, or R2D2 and C3PO to appear anytime soon.

-- SS