GoolyBib Offices – Weekly Creative Meeting

You’re in your weekly creative meeting and someone mentions DALL-E, the AI art model. The CEO dismisses it, saying it won’t be able to make creative assets that are consistent with our brand. You interject to explain Textual Inversion with Stable Diffusion.

Ok so let me get this right: Stable Diffusion is like an open source version of DALL-E?
And it has this functionality DALL-E doesn’t, Textual Inversion?
Alright and what that means is we can ‘train’ it on our brand, our products, etc and then we can use them later when generating creative assets for our ads?
Brilliant, this sounds promising, give it a whirl and report back.

AI models like DALL-E, Midjourney, and Stable Diffusion can ‘imagine’ almost any concept you can think of that’s in the public domain. They’ve trained on more images than any human artist could encounter in a lifetime, and they know how to draw all of them. However there are plenty of concepts that they don’t yet understand, and often these more niche concepts are the most important for business use cases. For example unless you’re Coca Cola, the popular AI models are unlikely to have seen many images of your brand or product in the training set. If you’re inventing something new it’ll definitely not be present in the models parameters. However there is a technique for ‘training’ the model to understand a specific concept you dictate. It’s called Textual Inversion, and so far it only works only on Stable Diffusion, the open source alternative to DALL-E. The process is that you upload 3 - 5 images of a concept – an object or style – and run the training script. That finds a space in the existing model that corresponds to the combination of features your sample images have, and then adds a label there, a token, so you’re able to reuse that pointer next time. 

Say for example you wanted to add the concept of a specific product you were about to launch called the GoolyBib. You could upload 3 - 5 images and train the model for 1-4 hours on a GPU, a specialised graphics card suited to running ML models, and then that’s saved to a concepts library. Once you import the library you can reference that concept again using the token you designated, for example <goolybib>. Now in prompts you can say “<goolybib> photoshoot in high detail”, or “photo in street style of <goolybib>”, or “elon musk buying a <goolybib>”, whatever images you need to create. You can also train styles, so for example rather than training on your product, you could train on a consistent brand style, and use that in your images to make sure they’re recognizable and easier to get approval to use. This solves a key issue with AI models that have precluded them from being used by businesses in great numbers: you need consistency in order to make the images useful.

Textual Inversion

Hey, we're gonna learn about textual inversion, which is kind of like an add on for stable diffusion a open source competitor to Dali the AI that lets you create images with just words.
So what is textual inversion? So normally when you use Dali or stable diffusion you run a prompt and you would type in something like, here we go, want a picture of Jaja beings in the Roman Senate in the style of the tale of two empires collection.
So what do I mean there Jar jar binks is from star wars, right? So that's a unique branded asset that Disney owns if Disney wanted to automatically generate images of Jar jar binks.
They could maybe use this, but you can see it's not very consistent, right? It doesn't quite understand the concept of JJA banks and therefore it's kind of giving you some weird aliens instead but also, but you know, we're not gonna worry too much about that.
That's Disney's intellectual property. I don't wanna get sued. So what we're gonna focus on more is like the tale of two empires collection.
And this is just something really niche that I found that I loved in the star wars Reddit this guy just likes to Photoshop old paintings with pictures of star wars.
So he has like you know, here's the ship above Rome, or like a 80 you know above the sub ruins.
And these are really reminiscent of I think it's Thomas Cole who does the empire series. So that's why I think it's like a tale of two empires star wars empire, and Thomas Cole over all the Roman empire.
Cool. So I wanted to replicate this style because it obviously doesn't understand, right? Like you know, it kind of gets it like in this one, that's kind of the style, but in the background, but everything else doesn't match.
So what textual inversion lets you do? So lets you train stable diffusion on a concept. So this is a good example here.
They've trained it on this monkey, they call it Bonzo bony. And then they can use that in a prompt. So you can say a piano being played by bony and run and then you've got it here.
So so this, this is like a library on hugging face. Hugging face are done a lot to democratize using this tool and definitely recommend you check it out, but there's, you know, tons in here.
You know, people put loads and loads of these libraries up. This is a funny one. I just found someone was trying to train the library on a concept of ans right.
So they obviously want to use her for some prompts. I'm not sure what is private. So we can, we can actually add our own concepts to this library and the way you do it is you can use this training, Google CoLab,  Which you can get a copy of here in in this page.
And then there's also the inference CoLab here. So the training CoLab is in order to train it on a new concept and add it to the library.
And then the inference CoLab is to use that concept. And I found I needed to use both because you'll see later the training collab uses a ton of space and memory and compute power and Google collab kinda complains about that unless you're using the paid version.
Cool. So this is the inference one. So we were just, you know, kind of using this prompt and there's no special concepts in here, right?
We're not using textual inversion but we can train it on, on this, right? So it kind of gives an example here.
They've trained it on a teapot, they just give three pictures of the teapot and then they can use that same teapot in different you know, different situations like there's a movie poster of the teapot or a teapot as a death star whatever you want.
So this is like really simple to use. You don't actually need to know how to code. You can just go through and click play on each one of these or you can open them up if you so you can click like show code.
If you wanna see what it's actually doing. One thing you will need is a hugging face account. So when when you run this, it will give you a link to where you need to go and sign up and then it it'll give you a token which you need to be a right access token, not just read access and then that will let you contribute to this, how you face library.
Cool. So this is the path here that you know, that, that is gonna lead you to this library.  But you know, really, I think anyone can make, make one of these libraries.
And then and then you just need to input the URLs. So I've input six images here. I, I just uploaded these to Google Firebase storage but you could really upload it anywhere or just link to it out on the web.
And you can run this to see if the pictures come through. So what I did is I just took this guy's paintings.
I chopped them up into six different styles and sorry, six different images. And then I uploaded them. You don't need six, like I've actually seen it work pretty well with just two or three.
But I think and, and it does take a lot longer to train if it's if it's more images, but in this case, cuz it was like a relatively complicated concept.
It's like combined star wars with like the older Roman and empire then I wanted to give it plenty of examples.
Cool. So, and there's a link there to his artwork, if you are interested. And what you have to do is you choose your, is this an object or a style?
So in this case it's a style, but if you're, you know, training under teapot or, and D YaaS it would be an object.
You shouldn't Objectify people. The placeholder of token is what you want it to be called in your prompts. So if you say I want, you know Jaja bins in the room and send it in the style of two empires then that's what you put in your prompt.
And, and, and anyone else can use this as well. And then you wanna use initializer the token. So it's a word that summarizes what the new concept is in this case.
It's like a painting to type of painting. So that's what I've used here. Cool. You can then run this code to to train the model.
It says it takes one to four hours. For me it took more like five or six. It took a really long time, I think because I had extra pictures for this one.
But the really cool thing here is that, you know, Google actually lets you when you connect here, it will complain at you and say, this is a big file, but <laugh> but it will let you run it for free.
And it gives you a GPU. Now GPU is like a specialized computer that just you know, it is, it's very, very good at working with machine learning.
And that's really what you're doing here is you're teaching the algorithm how to understand that concept. So it's basically searching the existing model for a space in that model.
Like kind of, if you think about this, like a multi-dimensional model and it's searching all those dimensions and finding one particular point in space where it's most similar to this style or most similar to AMA, so most similar to teapot.
And and then it's just giving that a label. And, and that does take some time. So it is doing a lot of processing and it won't really work unless you have a GPU.
So Google collab, I think is your best option, cuz it's very generous that they provide one for free. If you wanted to use your own GPU on Google cloud that's usually in the range of like $200 a month.
If you want to use that and that's just for one GPU it runs faster if you, I think if you use more GPS or better GPS but if you're just a hobbyist, then Google CoLab is is great.
Then once this code is run then you can run the code with a newly trained model. And and first thing you want to do though is save it to the library of concepts.
So I think you can also run it without saving it to the concepts, but I didn't try that. You know I, I, I, there wasn't anything.
I, I cared about being in the public domain, so and it's not my artwork, so I was happy to just kind of leave this concept out there.
And then and you know, kind of link link back to this guy. So people know where it came from. So The, you wanna name the concept and then just run this, it doesn't take very long.
And then and then you're ready to kind of use it. So I set up the pipeline and then I tried to run stable diffusion.
And this is the example of the prompt. So instead of saying like what we had before in the style, in the style of the tale of two empires collection, now it already knows what that is.
So you can say in the style of two empires and I run it, but then I run into this issue, good is out of memory.
And that's because it only has 14 or 15 gig of memory and 12 was already allocated. It didn't have enough space.
Pie torch is what is used to do the machine learning. And that's already kind of used up everything in this initial training.
But that's fine because you don't necessarily need that. You can bring any like I, I just like kind of left this where it was, and then moved across to to this to the inference notebook because the library is there in the public now you can actually just pull it in.
So what, the way that you do that is where it says load your concept here. You just put the link through to the SD concepts library, a tale of two empires, which is, you know, what I put in here.
And this is you know, the, basically what saved to the library was generated. And then I've linked through to his artwork here.
So he gets all the credit. Cool. So so once you've you know, once you've loaded that like it loaded here and then then you can just run all this stuff again, and then you can actually use it in the prompt.
So just to compare in terms of styles, this is what we got from stable diffusion when it didn't know what two empires was, right.
It also doesn't know what drag I thinks is I guess, but well maybe it's kind of purposely not infringing a Disney cup, right.
I'm not sure. But this is what we got afterwards. And this looks a lot better. It's much closer to that style.
In fact, this one is brilliant. It looks very much like that style. Like if you kind of compare that to, you know, some of these you know, with the storm troopers here then you know, that looks actually pretty good.
And, and this one as well is like pretty good. This is more cartoony. And then you have this one doesn't know, this one kind of is like, close this one's better.
But you can work on this and you can improve the prompt. Now that it knows what that style is it's gonna give you much better results, which match much more what you're thinking of.
If you wanted to, you could train it on the concept of J our as well and use both prompts. So you could say like You know, know if you have like Gira in the, in the room and Senate you could also train it on the idea of a room and Senate, if you think it's not getting that.
But it's I'll say only train it on the things that it doesn't get naturally, because it does take a very long time to train and you know, how can you use this?
Well there's tons of different ways. You know, I think the main way that I can imagine this being used for a brand is you know, you probably want to use stable diffusion to generate some creative assets, but they need to be in the right style, right?
Like they need to be in your brand star guide. And if I, you know, had a brand star guide saved as a concept that just said, I dunno, my brand name in here then you know, we could actually kind of use that in our prompts and then get a very consistent look and feel every time.
So I I did this I, you know, this was before textual in, in version. But if I was doing this again, this this illustration that I made for my book I would probably use textual inversion now, cuz you can see you know, in this in this prompt engineering case study kind of went through so if you go sax AFR prompt engineering, I went through a lot of hassles to try to find a style that I liked.
This is actually Thomas Cole full of Rome course of empire. This is I think one of the ones in inspired this other guy to put star wars into it.
And I wanted to get that style and eventually I kind of found that style and and then generated a bunch of these.
But if I could train stable diffusion on that style, then I don't need to include a full prompt every time.
Right. Like we can you know, if this is the prompt faded or painting on canvas and hum Hudson river school style of an ancient samurai soldier arriving a futuristic utopian Shanghai city from behind wide angle.
So if you think about it, you kind of store a lot of that into a into a variable, right? Like into a concept in this library and then and then just get consistent results every time.
So this is really good if you're making a brown star guide really good if you are trying to kind of make a, a video game and you want a consistent character that always shows up very good if you're making you know, a movie I think you could use stable diffusion to create a movie eventually.
This is one way to keep it more consistent. So it doesn't kind of change the style on you every time or change the character on you every time.
And you could also, I think use this, like if you're creating blog illustrations you know, this is for my book, but you can do the same thing for your blog and have, you know, something that looks like this every time, but with just different characters.
Cool. Okay. Hopefully that's helpful and enjoy playing around with this.

Why is Textual Inversion a big deal for brands?

Which model does Textual Inversion work on?

How fast does training the model on a new concept with Textual Inversion take with a GPU?

Why do we need to use a computer with a GPU to train Stable Diffusion?

In the future all brand style guides will include links to the brand’s Textual Inversion concept library, so when you use AI art models to create new assets they’ll always be on brand.

Vexpower Learning Certificate of Completion

Congratulations Carina Obster 💪