How 3D Medical Images Impact my ML Pipelines at several steps

I used to mostly deal with 2D images. But for the past couple of years, I’ve been dealing with 3D medical images. Here are the differences and how they impact the whole ML pipeline.

2D and 3D images differ in 2 main ways:

• the amount of information contained in each one of them,
• and the size of the data itself.

Before we continue, I want to highlight 2 things.

• When I am talking about 2D, I am mostly talking about natural images or high resolution images that don’t exceed 3k x 3k resolution.
• And when I am talking about 3D images, I am mostly talking about 3D medical scans like CT or MRI scans.

Now, back to the main question, how do differences between these 2 types of data affect your whole ML pipeline choices?

If you try to build a machine learning pipeline for this type of data you’ll encounter the following challenges:

• Preprocessing is generally a lot less expensive in 2D than 3D.
• Learning features from 3D requires more complex ML models than learning features from 2D.
• Storage requirement is much less for 2D than 3D.
• Real time processing is generally not possible for 3D while it’s usually the norm for 2D.

This means that for each step of your ML pipeline you’ll need to have specific considerations when working with 3D.

For example, how do you retrieve data efficiently from your storage and send it to your ML model?

How do you run your inference when it can take several minutes to finish?

What’s the efficient way to send the output of your model back to whichever server or service that’s consuming that output?

How do you visualize such output on different platforms?

All of these are questions that we deal with every day and we still do not have a perfect answer.

Moreover, the majority of the tools and techniques developed by the computer vision community are made for 2D.

So when working with 3D medical data, you have a lot less options. This poses a real challenge for building production grade AI systems for 3D medical imaging.

But it’s because of these challenges that we at PYCAD have been focused on this niche, dealing with 3D medical data and building ML solutions using this data.

Now, I haven’t even gotten started talking about the extra challenges that you get as a bonus when dealing with specific medical data types like DICOM, NIFTI, …etc.

But all in all, it’s been truly a pleasant experience working with 3D medical data. When we build AI systems that can handle such data for our clients, it feels like we unlocked new potentials.

It’s always a pleasure to see our clients reactions when they realize the amount of value they can unlock from their 3D medical data!

​

X Post of the Day

Med-Gemini achieves state of the art on various medical imaging tasks!

Tanishq Mathew Abraham, Ph.D. @iScienceLuvr Google DeepMind has written a SECOND paper about Med-Gemini, introducing 3 new models to the family! 🔥 Med-Gemini achieves state-of-the-art on various medical imaging tasks! 🩻🔬🧬 IMO multimodal AI is the future of medicine so excited to see this paper, let's dive in! ↓↓↓ https://twitter.com/i/web/status/1789216212704018469 10:49 AM • May 11, 2024 116 Retweets 692 Likes Read 12 replies

​