Animating Images Without Finetuning

Generating images from text is cool. But do you know what's cooler? Animating those images like the gif below!

So how can you animate images that were generated using text to image models?

Introducing AnimateDiff.

This is a new approach that can be built on top of text to image models such as stable diffusion.

Here's how it works from a high level perspective:

Given a base text2image model (e.g., Stable Diffusion), AnimateDiff method first trains a motion modeling module on video datasets to encourage it to distill motion priors.

In simpler terms, it tries to model what motion is in videos.

During this stage, only the parameters of the motion module are updated, thereby preserving the feature space of the base text2image model.

At inference, the once-trained motion module can turn any personalized model tuned upon the base text2image model into an animation generator.

Then it can produce diverse and personalized animated images via iteratively denoising process.

Below is a diagram of this process.

This model can make your still images full of life and without any finetuning of the text-to-image model.

You can read more about AnimateDiff in their original paper. You can also check the code here.

​

I want to help you understand and do more with AI and Machine Learning

So lately I've been exploring different ways to make this newsletter more useful for you as a reader. In order to do that, I thought I should directly ask you!

So Reader, how can AIFEE newsletter be more useful for you?

In order to help you answer this question, you can ask yourself the following questions:

• What kind of topics would I like to see more of?
• What is one area of AI and machine learning that I find most interesting and would love to understand it more?
• What is one area of AI that can help me with my business but I'm not too sure how it works and how exactly can it help me so I want to know more about it?
• What is one area of AI that I want to specialize in and maybe pursue a career and would love to understand it better?

Answering any of these questions will be a tremendously appreciated feedback!

Believe me, I don't take your feedback lightly. In fact, this newsletter subscribers are the first people I share my ideas with when it comes to doing any sort of content creation.

Although I have a larger following on LinkedIn, I just love and prefer bouncing ideas with you because I believe we truly have a beautiful community here and I want this community to thrive. That can only be done if I can truly help you achieve your goals with AI and machine learning.