PatrickMD.net

Thoughts on Medicine and Software

PatrickMD.net header image 4

Why You Shouldn’t Trust Wolfram|Alpha For Medicine

May 23rd, 2009 by Patrick
Respond

Wolfram|Alpha comparing Mayo Clinic to Olmsted

Wolfram|Alpha is a “computational knowledge engine” which is supposed to make a large variety of databases available through simple text query. The idea is that instead of digging through many databases, have one big database that can combine disparate results.

As with anything so ambitious, WA has collected plenty of criticism, especially because of the opaque natural language interface. Why does “life expectancy of 30 yo US woman in 2002″ work, but not “life expectancy of 30 yo US woman from 2002-2003″? Farhad Manjoo at Slate has a great article showing how straying from the example queries leads to the infamous “Wolfram|Alpha isn’t sure what to do with your input” response. SomethingAwful has put together a collection of ridiculous queries including
how much money BloodRayne made per millisecond.

The WA health examples demonstrate a number of problems which makes me not trust all of the results. Their example of “Mayo Clinic, Olmsted Medical Center” is supposed to compare two large medical centers in Rochester, Minn. However, it actually compares the Mayo Clinic satellite in Jacksonville, FL, with Rochester. Even that apples-to-oranges comparison is hampered because there is no data in WA for Mayo in Jacksonville. Try finding data on Mount Sinai Hospital in Miami — the only Mount Sinai that WA admits to knowing is in New York City (and has no affiliation with the one in Miami.)

Wolfram|Alpha calculates target exercise heartrate

WA says that for a 50 year old with resting HR 60, the maximum HR is 180 and the target exercise HR is 132-156. However, the AHA says that target rate is 50%-80% of the maximum, and that a 50 year old’s maximum is 170. Maybe WA’s numbers are better, but it’s tough to tell without references or justification.

Wolfram|Alpha calculates 10 year coronary heart disease risk

WA says it calculates heart disease risk based on the Framingham study, but I get different results. (Assuming LDL 111, HDL 54, BP 120/80, nonsmoker, not diabetic.) Using the male score sheet from Wilson, Prediction of Coronary Heart Disease Using Risk Factor Categories. Circulation 1998 97 (18): 1837-1847., I get 6%, versus WA’s 4.6%.

As the score sheets just return whole numbers, WA is likely using the Framingham model which is discussed in the paper. However, even using that I get 5.4%, a solid 0.8% more than WA’s result. (for sticklers, my work is after the “more”.)

I would encourage anyone who uses Wolfram|Alpha for medical decisions to run the numbers on their own. I would like WA to be more explicit about where the data is coming from and how results are derived.
[Read more →]

Tags:   · · · No Comments.

Cancelled Kindle subscription to NEJM

May 22nd, 2009 by Patrick
Respond

Kindle screen shot of New England Journal

I cancelled my Kindle subscription to the New England Journal today. I am still considering an online subscription.

What really annoyed me this week was realizing that the Kindle doesn’t get Early Release articles until they show up in the print magazine. As soon as this content is on the website, it should be available for the Kindle too.

Journals on the Kindle are apparently just electronic versions of the paper editions, without any fancy additions. No interactive component, no updates to corrected content, none of the things that a constant Internet connection offers. (After all, the 3G connection on my Kindle is faster than my iPhone’s connection.)

The price certainly wasn’t great either; a Kindle subscription from Amazon runs $9*12 = $108 per year. The print/online subscription from NEJM is $159, and the online-only subscription is $99. It’s cheaper if you are a resident or medical student.

Finally, note that the $108/year to Amazon does not include subscriber access to the NEJM website.

Tags:   · No Comments.

Reading the New England Journal on Kindle 2

May 10th, 2009 by Patrick
Respond

Kindle showing NEJM article

Recently I received a Kindle 2 (not the same thing as a Kindle DX.) The E-Ink display is really cool. The screen looks great when read in bright light. It doesn’t need any current to maintain the screen image (like an Etch-A-Sketch) so battery life is very long. Kindle runs the Linux operating system with Java for the UI.

Kindle is great for reading novels and long-form articles. I read A Study in Scarlet in about two hours and forgot I wasn’t reading from a book halfway through.

Kindle screen shot of New England Journal

I got a trial subscription for the New England Journal of Medicine and found that the experience was better than I expected, but not great. Reading the Perspective and review articles was better than reading them on a web browser, but research articles were difficult, especially because any tables or figures are very difficult to read. You can zoom into a figure slightly with several arduous clicks using the 5-way tool, but the result is not worth the trouble. The image at the top of this article shows what NEJM looks like on a Kindle, and a screenshot dump appears to the right.

After experimentation with some medical and non-medical works, I have to say that Kindle is terrible for reference works because:

  • Substantial latency to user input, particularly cursor movements. Next and previous page are the fastest commands.
  • It’s difficult to select links from a long list (such as a table of contents or search result) using the 5-way cursor device.
  • There’s no good interface for magnifying pictures and tables. I kept touching the screen before I remembered that there is no touchscreen on a Kindle.

Jakob Nielsen talks about some of these points in his Kindle 2 usability review.

Kindle viewing Google Reader (RSS)

I was surprised that the web browser works well with mobile feeds of newspapers and Google Reader, which I use for RSS. The image to the right shows the Google Operating System blog in Reader.

I loaded my Kindle with a variety of classics from various free e-book sources.

While Kindle 2 does not read PDFs natively, you can translate any PDF to Amazon’s Kindle format for free.

Tags:   · · No Comments.

Minor New York State License Search Update

December 16th, 2008 by Patrick
Respond

New York MD License Plate, 1955

I added the ability to search for a physician’s NPI. Enjoy.

Tags: No Comments.

How do you keep up with the medical literature?

December 15th, 2008 by Patrick
Respond

Ves Dimov at the Clinical Cases and Images blog is working on the best way to keep abreast of news in the medical literature. Dr. Dimov’s “5 tips” for staying up to date is a great first stop. The first point is to follow the RSS feeds of the major journals using a feed reader such as my favorite, Google Reader.

I don’t like the journal RSS feeds, even though I agree they are a good resource. Unlike a medical blog which kicks out one or three posts a day, the journal feeds are silent until an issue comes out and 20-30 updates appear in the feed. Each article, whether a book review or a major study, gets one entry. There’s no linkage between opinion pieces and the studies that prompted them. Pictures (such as important graphs or clinical images) are not part of the feed. I found that I prefer the way that a medical blog will discuss an article instead of the simple summary and link that I get now.

Using medical blogs can be better than the raw RSS. Dr. Dimov and others put together weekly reviews. I recently subscribed to Physician’s First Watch which provides one-line notes about major articles. First Watch is available by email or RSS. I’m still experimenting to see which is the best route for me. I have been so busy lately that I am barely checking my email, and never open Google Reader.

More recently, some physicians have begun to use Twitter, which is a combination of IM and social networking. RSS requires some technical knowhow to understand and set up. Twitter is accessible to anyone who has ever sent an IM. I can’t agree that Twitter wins. The social aspect leads to a lot of less than informational chatter. 140 characters is really small! It’s barely enough to mention a title and a hyperlink, much less why the hyperlink is worth following.

The solution is less about the technology and more about the content. I believe that human-generated summaries, properly hyperlinked, are the only way to digest the steady stream of medical literature out there. In the end, you still have to read the full article.

Tags:   · · · No Comments.

AMIA 2008: Using MedLEE to Classify Smoking Status

November 16th, 2008 by Patrick
Respond

The i2b2 NLP Shared Task in 2006 had two parts. The first was to deidentify discharge summaries. A separate task was to identify a patient’s smoking status based on a discharge summary. There were a number of successful methods used for this task which are described in the January 2008 issue of JAMIA.

My project was to further evaluate the utility of semantic features in this task, and determine how well semantic features would perform with a simpler classifier. To generate semantic features I used Columbia’s MedLEE medical language processor.

The rule-based classifier using MedLEE semantic features performed better than I expected with an F-measure of 0.83. The Boostexter classifier trained with semantic MedLEE features was competitive with the top-performing smoking classifier in the Shared Task, with microaveraged precision of 0.90, recall of 0.89, and F-measure of 0.89.

Above is the slide presentation I gave this past Sunday. The full paper is available below.

McCormick PJ, Elhadad N, Stetson PD. Use of Semantic Features to Classify Patient Smoking Status. AMIA 2008 Symposium Proceedings. 2008. PMID 18998969. [PDF]

Tags:   · · · · No Comments.

Google’s Voice Search

November 15th, 2008 by Patrick
Respond

Google has migrated their old Voice Search application to the iPhone as part of their Google Mobile App software. Alex Chitu has a nice screenshot of the original interface from 2002. Google continues to run GOOG-411.

The core voice recognition algorithms used in the industry are mostly the same ideas optimized over the last 20 years, benefiting from the increases in processor power and storage. The big difference is in the volume and nature of data collected for the acoustic and language models. Google is legendary for their insatiable appetite for all kinds of data. The recent debut of a many-to-many translation service shows that they have plenty of data for advanced language models.

It’s not clear that speech recognition is the best tool for undirected tasks (i.e. interpreting responses to “What do you want to search for?”) I recall a few startups that used cheap human transcribers instead of speech recognition, such as Jott.

I plan to poll people with iPhones to see if they find the voice search feature worth using more than once. I don’t think I will get much out of it personally, because A) I type much faster than I speak (even on the iPhone), and B) I often search for proper names and abbreviations which are likely not high up in the language model.

Tags:   · · · No Comments.

Consultant Recommended Orders, Coming to a Hospital Near You?

November 15th, 2008 by Patrick
Respond

Martin Were, the top winner of the AMIA 2008 Student Paper Competition, tackled the problem of improving the implementation of consultant recommendations. As an intern this topic is near and dear to me.

When a consult is called, particularly in a large academic hospital, the consultant will generally leave a note with recommendations. While consultants can input orders directly for the patient, it is considered best for the primary team to enter all orders so that they have a full understanding of what treatments the patient is getting. Exceptions are made for complex orders, such as dialysis instructions and chemotherapy dosing.

According to Dr. Were’s research, only half of all geriatric consultant recommendations are followed. Maybe the primary team doesn’t agree. Or, the team didn’t see all the recommendations. In some cases it’s not clear how to dose a recommended medication.

At the university hospital, Were extended the existing CPOE system to allow a consulting service to enter actual orders. This forced the consultants to be specific with their recommendations. The primary team was prompted to accept or reject the suggestions. The picture above is Figure 4 from his paper showing the primary team interface.

Were piloted the tool with geriatrics consultants and the hospitalist service. Intervention patients had 249 recommendations versus 192 for the controls (p<0.05). 78% of intervention recommendations were implemented versus 59% for controls. Providers indicated in a survey that the system improved quality and saved time.

I would like to see Consultant Recommended Orders (CROs) implemented in more hospitals. I'm curious to see what objections other physicians have to this idea.

Update: The full paper will appear in the next issue of JAMIA. A preprint is available at the JAMIA website for subscribers.

Were MC, Abernathy G, Hui SL, Kempf C, Weiner M. Using Computerized Provider Order Entry and Clinical Decision Support to Improve Referring Physicians’ Implementation of Consultants’ Medical Recommendations. AMIA 2008 Symposium Proceedings. 2008. p. 803. PMID 18952934. [PDF]

Tags:   · · · · · No Comments.

MedLEE commercialized

November 14th, 2008 by Patrick
Respond

I saw on HISTalk this week that Columbia University’s MedLEE system is being commercialized by a new startup with the somewhat dry name of NLP International Corporation. It appears that Columbia’s Science & Technology Ventures office has helped create this startup and granted it an exclusive MedLEE license.

MedLEE has been around for awhile, so I’m surprised that this commercialization is happening now. It’s a great system that I used for my AMIA project. I hope this startup can build some great applications and deliver benefits to the wider industry, with a reward for those who worked so hard to build it over the last decade-plus.

You can try a MedLEE demo here.

Tags:   · · No Comments.

NLP Making Indiana MRSA Reporting Very Accurate

November 14th, 2008 by Patrick
Respond

A paper I really liked in the Student Finalist competition at AMIA 2008 was Jeff Friedlin’s project to use NLP processing to automate the identification of MRSA lab reports for state-level reporting. The Regenstrief Institute runs an electronic lab reporting system at the Indiana Network For Patient Care, which is a regional center that collects HL7 lab messages from hospitals throughout Indiana. The state of Indiana now requires that any positive MRSA result (not just invasive cases) be reported. The existing system had been using LOINC codes to identify positive cases. This was missing many positive reports because of lab systems that communicate in free text, usually with OBX segments in the HL7 message.

Dr. Friedlin sorted through all the types of lab messages received by the regional center and created an NLP system built on Regenstrief’s REX processor to identify those with MRSA positive results. He then tested his system with one year’s worth of data. To calculate accuracy he reviewed 64,554 messages himself to generate a gold standard. The results were fantastic, with a sensitivity of 99.96%, a specificity of 99.71%, and a PPV of 99.81%.

One side effect of this great work is that it led to a huge increase in positive MRSA reports for the state, because so many were being missed by the old system. He showed a slide with this increase during the presentation but I don’t have the numbers available. Reportedly his presentation later in the conference overflowed.

Friedlin J, Grannis S, Overhage JM. Using Natural Language Processing to Improve Accuracy of Automated Notifiable Disease Reporting. AMIA 2008 Symposium Proceedings. 2008. p.207-11. PMID 18999177.

Image Credit: estherase on Flickr

Tags:   · · · No Comments.