How we helped accelerate inference time for a client's AI product

Below is a screenshot of a benchmark we did for a client of ours. The goal was to accelerate inference time. This client has 4 different models taking in CBCT scans as input. We were able to decrease the time shown in red on the left (synch) to what you're seeing on the right (asynch). Here's how.

As most of you already know by now, we run an AI development agency focused solely on medical imaging.

We've been building models for our clients to solve technically challenging problems in medical imaging field.

One thing that is challenging when building such models is inference time. It's never real-time. And I don't think it will be real time any time soon!

This is due to the complexity of the problem.

Let me explain.

Most of the models we build for our clients take in 3 dimensional arrays as inputs. These arrays are the pixels representation found in DICOM, NIFTI or NRRD files.

So think about it for a second.

How complex would a model be if it needs to handle this type of data?

The answer is, very.

This means that not only the data is complex, but the model itself is complex.

One ML model can take several minutes to finish processing one input!

For this specific client, we deployed all of their models through containerized APIs.

Now how did we accelerate the inference time of all models?

We ran them asynchronously.

Since the models are being run as separate REST APIs, we used asyncio to run these APIs in parallel.

Asyncio is a python library that allows you to run asynchronous processes. This can squeeze so much performance, especially when you have constraints like one server running all the models. Which was the case for this specific client.

Running these models asynchronously was possible in this specific use case because the models didn't depend on each other. If they did depend on each other, then it would've been a different story.

There are several other techniques that we can apply to accelerate the process even further. But we're not going to get into that in this post.

But here's what you should take from all of this.

Building ML models is just the start. There is a ton of engineering needed to create a product around these ML models. Keep this in mind next time you work on an AI project.

X Post of the Day

Cloud providers are pushing many features for medical imaging! It's a sign that the field is growing at a very fast pace!

Oge Marques @ProfessorOge Uncover the power of decoding DICOM medical images with AWS HealthImaging using nvJPEG2000. Explore GPU-accelerated solutions for efficient image processing. Learn more in our blog. #ai #medicalimaging https://bit.ly/3U8H6Il 12:47 PM • Apr 20, 2024 0 Retweets 2 Likes

​