How to transcribe medical prescriptions with AI

Lately I was chatting with a business man who has done amazing things in the medical imaging field.

While conversing, he told me that he was working with his technical team on a recurrent problem that doctors have.

This recurrent problem is the transcription of medical notes and medical prescription.

He told me that they tried document AI APIs from Azure and other big cloud providers but the results are not up to their requirements.

Specifically, he said that non textual parts of the prescriptions are hard to transcribe.

For example, if you have YES or NO questions in the doctor's notes, then the document API failed to "understand" them.

Another example is, if you have checkboxes in the prescriptions then the document API failed to categorize them into checked or unchecked.

Fast forward a few weeks later, he asked me if I heard about Qwen2-VL models.

At that time, I had heard about them and saw some people test them on different vision or language tasks, but I hadn't done any testing from my side yet.

He asked if these models could address the shortcomings that they noticed when using document AI APIs from the big cloud providers.

Immediately, I asked him to provide me with some samples where the cloud APIs failed so that I can test them with Qwen2-VL models.

Upon testing, I was blown away by the results!

These models are truly powerful in understanding documents.

They can understand machine written text, human handwritten text and all sorts of symbols and structures like checkboxes, YES/NO questions, ...etc.

Below you can see a dummy prescription image. I asked the model to transcribe it and I asked it to add an "X" next to the choices that were checked.
​
The transcription:

What the model gave me:

Truly impressive!

I can hear you say, well, this can be done by OpenAI models, so what's the big fuss about?

Well, Qwen2-VL has open sourced some of its models!

This is an extremely important point, especially in healthcare applications.

So basically, this model can be downloaded and integrated into a specific workflow that's later deployed directly in a doctor's clinic.

Btw, this is a business opportunity if you can execute on it!

​

Secret Way to Learn from the Best People in Medical Imaging

Here's a secret way to learn from the best people doing medical imaging stuff.
​
First of all, what are the best people in medical imaging actually doing?
​
The answer is: they're building useful tools.
​
If you want to learn from these people, then you should check out their code.
​
So where do you find useful tools in medical imaging that are open source and which you can check their code?
​
Top place is 3D Slicer extensions!
​
3D Slicer is open source and its extensions are open source as well.
​
To find the code for these tools, you can either check online github repos that correspond to those tools. Or you can simply download the extension through 3D Slicer and you'll have the code directly on your computer!
​
You can find some amazing algorithms implemented within those tools.
​
If you study those algorithms and learn how to repurpose them for other use cases in medical imaging then you'll definitely gain a ton of skills in the medical imaging field which will set you apart from the competition.
​
Now go download some extensions!