How to have more control over text-to-image diffusion models

In text-to-image generation models, you can specify many attributes to customize your generated image.

You can for example specify the color of the hair of a person, or if an animal is large or small.

But what if you want to push the level of control even further?

For example, imagine adding as input to your diffusion model, an image of edges which could be used to highlight more details of the generated object.

Now, such a solution exists! It’s a neural network that’s used as a control mechanism for diffusion models.

The network is called ControlNet.

The figure below illustrates an example of how you can control details by sending as input:

• The prompt: “a high-quality, detailed, and professional image”.
• An image of Canny edges that was extracted from another image.

Important things to understand:

1 - We don’t use the original image of the deer as an input. We just extract edges from it, and then use the image of the edges as extra input.

2 - We don’t mention anything about the object at hand (in this case the deer).

​

​

How do you go from training locally to training on the cloud as quickly as possible?

Imagine this scenario.

You are training a deep learning model locally.

After some time, you realize that you don't have enough resources locally and that your training will take a long time.

You decide to switch to training on the cloud.

What you would usually need to do is to rent an instance on a cloud provider and then setup everything on that instance, just like you have on your local machine.

Another approach would be to dockerize your code and then push your docker image to a container registry on one of the cloud providers.

Both of these approaches can be tedious.

Here's another solution that you may not know about.

If you're training a tensorflow model, you can use a tool called Tensorflow Cloud.

With a few lines of code, you can go from training locally to training on google cloud.

The code snippet below illustrates this. And a full colab notebook example can be found here.