Using AI to generate minifigures, part 3
Posted by Huw,
During last year's lockdown we published many articles by guest authors. Here's a follow-up on one of them, by Pawel, an Artificial Intelligence engineer:
I am returning with another article about AI generated minifigures. If you are not familiar with the previous articles (part 1, part 2) I encourage you to read them to get the general feeling of how the magic works.
I have now started my PhD in image generation so have had an opportunity to dive into the topic and the results are amazing...
The advancements in AI algorithms are astonishing, and almost every day better and better AI enhanced solutions are designed. This time, for the task of minifigures images generation, I decided to use what is called the state of the art (the best currently available) model in this topic. It is called StyleGAN2 and was designed by NVIDIA engineers.
If you encountered generated HD images of people faces, this is where the StyleGAN2 is applied. However, the basics of the algorithm are almost the same as I described in the first part of this article series, just more sophisticated. I used Brickset (on some of them you can see generated Brickset logo) and minifigs.blog images for model training.
Once again, the input to the model is just some set of random numbers and the output is 256 x 256 image of the minifigure.
The image below shows a sample of them and as you can see the results that I now get blow away those from my previous articles, both in quality and diversity.
I encourage you to download this image to take a closer look at more of them.
You can easily recognise some casual city minifigures with classic smiley face, ninjas, or some fancy superheroes. Once again you can generate those images endlessly and try to find your favourites. Mine is definitely this guy:
StyleGAN2 have one more amazing feature - the minifigures are represented as some points in an abstract space. Imagine that you have two minifigures represented as two points. Now, you can generate all the intermediate minifigures that correspond to all the intermediate points between two previously selected minifigures. In a result you get some cool morphing animations like this:
I am very satisfied with those results and can't wait to do some more research and share it with you.
80 likes
26 comments on this article
That morphing effect is tremendously cool.
Another example of nowadays gratuitous artificial intelligence. I think you could get similar results by building a correlation matrix among minifig parts from the inventory and using that distribution for the random generator. Thinking twice, I will propose that as an exercise for my students next semester :-)
Lepin would love these.
The one that stood out for me was the red ninja on the RH column, 2nd from the top (next to Stormtrooper Batman).
Some of them look like designs that are good enough to base actual character designs off of, such that anyone using them would have to negotiate some sort of licensing deal with the originator.
Pawel, this is such an incredible improvement! These look absolutely amazing, I love how far the machine has come since part 2. Can’t wait for the next update!
I love how the Batman (I see it's being nicknamed Stormtrooper Batman now) figure has both the jagged cape and pointy ears helmet, so the AI has clearly recognised those two elements being used together a lot! Such a fascinating topic.
Stormtrooper Batman (Battrooper? Stormbat?) Not the figure we deserve, but the one we need.
AI can certainly do interesting things, but those Minifigures look to me like they've lost a battle with a blow-torch and some acetone...
Thx. Didn't wanna sleep today anyway
I guess they’re ok, sort of interesting but nothing special.
What happens when you apply StyleGan2 to Bionicles or Hero Factory figures? I don’t know if there are enough from which to learn, but if so, what would the outputs be? I suspect they would look quite plausible.
This is getting scary.
They look so real in the downloaded image!
Reading the comments and I see we all agree that Stormtrooper Batman frikkin’ rules
@pecadorl said:
"Another example of nowadays gratuitous artificial intelligence. I think you could get similar results by building a correlation matrix among minifig parts from the inventory and using that distribution for the random generator. Thinking twice, I will propose that as an exercise for my students next semester :-) "
Maybe I don’t get what your saying, but wouldn’t that just create a bunch random themed minfigures out of existing parts, and not actually mix prints and parts together? So you wouldn’t get the storm trooper Batman helmet, for instance?
Thanks for all of your comments!
@Zander said:
"What happens when you apply StyleGan2 to Bionicles or Hero Factory figures? I don’t know if there are enough from which to learn, but if so, what would the outputs be? I suspect they would look quite plausible."
Unfortunately, that kind of algorithm needs a lot of data to train (usually thousands), so there are simply not enough sets.
@pecadorl said:
"Another example of nowadays gratuitous artificial intelligence. I think you could get similar results by building a correlation matrix among minifig parts from the inventory and using that distribution for the random generator. Thinking twice, I will propose that as an exercise for my students next semester :-)"
I'm not quite sure if I understand correctly what you mean, but assembling a whole minifigure out of generated parts is quite a tricky task.
I've been fan of your work since part 1, but the improvement in new examples is really something else! Keep up the good work!
2-3/30 figs have batman ears. Seems about on par with the source data >.<
@WallaceFredrickson said:
" @pecadorl said:
"Another example of nowadays gratuitous artificial intelligence. I think you could get similar results by building a correlation matrix among minifig parts from the inventory and using that distribution for the random generator. Thinking twice, I will propose that as an exercise for my students next semester :-) "
Maybe I don’t get what your saying, but wouldn’t that just create a bunch random themed minfigures out of existing parts, and not actually mix prints and parts together? So you wouldn’t get the storm trooper Batman helmet, for instance?
"
You're correct.
Even so, if you want figures with new parts I think that it should be much easier to train a different network for each part type. And then mix the parts with the correlation matrix.
@pablo94 said:
"Thanks for all of your comments!
@Zander said:
"What happens when you apply StyleGan2 to Bionicles or Hero Factory figures? I don’t know if there are enough from which to learn, but if so, what would the outputs be? I suspect they would look quite plausible."
Unfortunately, that kind of algorithm needs a lot of data to train (usually thousands), so there are simply not enough sets.
@pecadorl said:
"Another example of nowadays gratuitous artificial intelligence. I think you could get similar results by building a correlation matrix among minifig parts from the inventory and using that distribution for the random generator. Thinking twice, I will propose that as an exercise for my students next semester :-)"
I'm not quite sure if I understand correctly what you mean, but assembling a whole minifigure out of generated parts is quite a tricky task.
"
I think that should be easier to generate minifigure parts than whole minifigures.
"Stormtrooper Batman" is my favorite.
I posted this before even reading the comments, and now after reading I see a lot of people think similarly :)
Wow, these results are phenomenal! I did a very similar project on minifig faces last year: https://www.echevarria.io/blog/lego-face-vae/index.html
I must admit, Stormtrooper Batman is awesome, but Baby Yoda Batman would be the ultimate minifigure.
At the same time, it would be interesting to see a whole Star Wars-Batman minifigure line. You could have Darth Maul Joker, a Twilek Riddler, and Mandalorian Catwoman. Of course it would have to be brick built, but Jabba and Clayface would go great together.
They look like disgusting freaks
Overall, my hands rock back/forth...but:
-I'm the 'camp' about Stormtrooper Batman...although, I think of him more as 'Hazmat Batman' (hmmm...'HazBat'???), a 'Radiation Suit Batman' ('BatRad'???) or E-Suit Batman (you know, when he Batman...IN SPAAAAACE!!!:))
-The three figure N,S, and E of 'Bats' looks like aaawesome additions to Ninjago...which is something I never thought I'd be typing:)
-The red one two below the black 'ninja' might fit with Ninjago...'cept that fronts a little busy.:)
LOL! They all look like they should be found in some cheap shop under the brand name BlockMen, or festooning a table in a market/carboot sale amongst all the other FAKE figs those types like to con children with.