http://www.wired.com/design/2013/08/an-app-thats-looking-to-document-the-worlds-pill-supply/
Big Problem, Smart Solution: An App That IDs the Pills You’re Taking
By LIZ STINSON
Two years ago Patrick Hymel’s grandfather was being treated for metastatic prostate cancer. The 90-year-old had a host of pills he was supposed to take–around 12 total, including one that was critical to his treatment. It turns out, for six weeks he was taking his medications incorrectly without anyone noticing. “Not one of the eight doctors he was seeing, none of the nurses he talked to, none of us [his family] caught it,” says Hymel. “And I realized that no one can easily check to see, do we know what this patient is actually taking, and are these the correct pills?”
In our over-medicated world, there’s room for multiple levels of error.
Hymel, a former ER doctor turned healthcare entrepreneur in Birmingham, Ala., figured there had to be an easier, more reliable way to monitor a patient’s pill regimen than just relying on information from the patients themselves, who are often dealing with multiple chronic conditions. In our over-medicated world, there’s room for multiple levels of error, beginning at the moment a doctor prescribes you a medication and extending all the way to the pharmacist filling the order, and a patient ingesting a drug.
This led Hymel and his business partner, Stephen Brossette, to create MedSnap, a new iPhone app that uses computer vision technology to identify prescription pills and the possible interactions a patient might experience if certain drugs are taken together. “We want to be able to validate what patients take, and we want a technology that could eventually allow patients to verify each dose themselves,” Hymel says. “Because it’s no small task to ask someone who is sick to self-administer 12 medications a day.”
On a basic level, MedSnap is as easy to use as depositing a check with your phone. First you position your pills and capsules flat on the the surface of the plastic MedSnap tray, then you open the the app and take a photo. From there, the images are matched against MedSnap’s visual pill library and within seconds, a screen pops up identifying each medication, its common uses and the possible interactions it might have if taken with the other drugs you snapped. “We set out to invent technology that could make that type of validation very quick, elegant and doable,” he says, adding that the app works without a wireless or cellular connection in case of emergency scenarios.
Sounds pretty basic, but for busy doctors, nurses and pharmacists, a mobile app that quickly and accurately accounts for a patient’s medicinal regimen while identifying potentially harmful drug interactions could be a huge time saver, not to mention a way to cut down on hospital readmissions due to improper medicine usage. Currently, the app is rolling out to various health care organizations and will be integrated into their electronic health records, but Hymel says that eventually he’d like it to be used at home by individual patients.
Using computer vision technology it captures the imprint, color and size of a pill. Image: MedSnap
Yes, MedSnap is an app for your iPhone, but at its core its more than that. The team is really on a quest to document the world’s pill supply. But compiling the data it takes to build a comprehensive database of medications is neither quick nor easy, which is why MedSnap designed a process that runs, in the words of Hymel, like a pill identification factory. It begins with a unique form of crowdsourcing.
Since the app’s beta release in December, pharmacists from around the country have been snapping photos of the pills they have in stock. Those images are independently audited and verified by two member of the SnapLab team (mostly made up of local pharmacists and pharmacy students) who cross reference the pills with images from the FDA and other professional third-party sources.
From there, each pill is uploaded to a visual pill library, which is a major database of images that resides on the MedSnap servers. “It’s an interesting form of crowdsourcing because this data has to be right,” Hymel explains. “We can’t depend on the wisdom of the crowd entirely because we can’t tolerate any errors.” The MedSnap team has gone to medical research facilities, hospitals and drug distributors to do reference snaps, which allows them to collect the entirety of a facility’s pill stock. Plus, every time a pill is snapped by a user, he or she is prompted to verify the identity against MedSnap’s image models.
It’s a lot of input, and so far, the database contains more than 3,300 over-the counter and prescription medications, including their color, imprint and size variations—”That’s by far the most common pills in circulation,” Hymel says. Sorry, this does not include the random pills you have left over from New Year’s Eve.
At minimum, MedSnap requires 30 images of each side of a pill, meaning, they have to get at least 60 images for every medication in the database. Bonus if the images come from different lighting scenarios and angles. This guarantees that the computer vision algorithms will be able to recognize a pill’s shape, color and grain regardless of the environment in which it’s captured.
To ensure this quality, MedSnap built its own computer vision technology to enhance color recognition (it currently can differentiate between 244,000 shades of pill color) as well as boosting the app’s segmenting abilities, the process that allows the computer to separate one pill from the next on a tray full of multi-colored medications. Theoretically, patients can unload as many pills as they have in their fabled “big brown bag” as long as it fits into the rectangular surface on the tray.
I played around with the app, and for the most part, it worked as described. My first batch of pills was included a Ibuprofen, a generic brand Aleve and a Zertec, all of which MedSnap recognized with ease, while assuring me that it wouldn’t be a problem to take all three if I were to have a raging headache during allergy season. Trouble only arose when I began snapping my generic supplements and vitamins, which Hymel warned me would turn up as “ambiguous” since they aren’t recognized due to their lack of imprint (the letters and symbols that are printed or etched onto a pill).
One of my super-vitamin horse pills showed up as Keppra, a medication that helps to cure certain types of seizures caused by epilepsy, which with its oblong shape and nude coloring, I does look deceptively similar to my multi-vitamin. When I told Hymel about this, he said that errors like this are actually a good way to refine the image bank, and that I should take a few reference pics of the pill in question. “The more images we have,” he says. “The better we can get.”
