Enhanced set inventories and part information

 

  • Enhanced set inventories and part information

    Posted by Huw, 27 Nov 2012 08:51. Filed under Brickset.

    If you've been reading the forum or have looked around the site menus recently, you will have read about or noticed that we've been able to obtain inventories for around 4,000 sets and information for about 20,000 parts.

    Over the last few weeks I've been putting it to good use and gradually adding features to the site that I hope you find useful. There's still a lot more I want to do but I think what's there now is worth announcing. Here's what's new:

    • On set details pages, if an inventory is available, a 'Parts' tab is shown at the top which will show all parts and quantities in a compact format. Here's an example for Frodo's Cooking Corner.
    • A link to a tabular listing of the inventory is provided on that page, which for the same set, looks like this. You can sort the columns by clicking on the header.
    • Parts can be browsed by category or colour. Note that these are official LEGO names, not made-up ones that are used at other sites.
    • You can search for parts by part number, design number or name, from the top search box. Here's an example results page.
    • In the results, it shows the number of sets each part appears in. Click on one to see the sets it appears in, for example, Sets containing 4212471, Dark Green Angle Plate 1X2 / 2X2.
    • Each part has a detail page like this one, that shows information about the part, and also other colours it comes in.
    • If you're logged in, you can see sets you own containing a particular part using the 'Your collection' menu on the part detail page.
    • As new inventories are added, they'll be listed on the Latest inventories page.

    I think that's all... The source of the data is nowhere near complete, and often inventories are missing parts, particularly licensed parts, so the information presented is only as good as the source data. An important next step to take, probably in the New Year, will be to enable you, the site's users, to validate inventories using the one printed in the instructions.

    Do you have any ideas about what more could be done with the data? One obvious thing, which you can expect to see in the next few weeks, is some sort of 'My Parts' listing which will show all the parts you own in your sets. 

    (Thanks to the team over at The Daily Brick for helping with the technology to acquire the inventories)


    48 comments, latest one posted 5 months ago. (Commenting has ended on this article)
    2555 views


 
 

Comments

Posted by lomis in United States, 27 Nov 2012 09:34

This works great as far as I can see.
Thank you!

Posted by icey in Denmark, 27 Nov 2012 09:35

Amazing... I'm impressed!

Posted by DaveBey in United States, 27 Nov 2012 09:41

Neat!

I realize there are other sites/tools available for this but will users be able to track individual parts here at Brickset?

Posted by rosconiner1 in United States, 27 Nov 2012 09:42

awesome new feature! good job!

Posted by copperwonder96 in United States, 27 Nov 2012 09:44

The layout of part information is nice.

If I were buying a set to find certain pieces, this is useful. Or if I recently purchased a number of sets and wanted an inventory to report on (an excel sheet version) that would be great too. The most powerful feature i feel however, is the trade/sell factor. Obviously Bricklink does this already but their interface is not good in my opinion. What would also be great is to connect the Pick-a-Brick store on Lego.com with this database to indicate (Available, not Available to buy). I often use the long part number (the absolute part number) to find pieces in the Pick-a-Brick database from the inventory on the back of instruction books so the attributes seem to be there...just need to figure out how to dump and refresh Pick-a-Brick inventory into this massive catalog.

Cheers!

Posted by lego613master in United States, 27 Nov 2012 09:48

A thing you NEED to implement:
I sometimes get LEGO at garage sales, and I do not know which sets it contains. A awesome feature would be to have a more advanced parts finder, a 'master list' if you will, of all LEGO parts ever created. The color could be selected at the top of the page.

Posted by ChicoCheco in Norway, 27 Nov 2012 09:48

Good job! This seems to me like quite a time demanding and complex task. I wonder HUW, how many hours per DAY in average you for us to improve Brickset? Is there any time left for build something?

Posted by lego613master in United States, 27 Nov 2012 09:49

Oh, and maybe a feature where you can omit parts from sets, if they're lost, broken, or traded.

Posted by wolfgang514 in United States, 27 Nov 2012 09:56

This is great! I really like being able to get a set's parts list without having to download the instructions. Thanks Huw!

Posted by Goony in Denmark, 27 Nov 2012 10:06

Hi HUW

An addition that would be cool would be an alternative version of my parts.
As in part not bought in sets but from PaB, Bricklink etc. Maybe with an option to see which sets you could build out of your collection of "loose bricks"
Thanks for an awesome site

Posted by georgebjones in United States, 27 Nov 2012 10:08

Huw-Since you are taking suggestions, would it be possible to indicate whether the part is in stock to order via replacement parts? For example, if you go to the replacement parts page at https://service.lego.com/en-us/replacementparts/ (Sorry, US version) select your age bracket and location, then select you wish to purchase a piece, then using a different set than you have above, 4192, you get to this page: https://service.lego.com/en-us/replacementparts/#WhatBrickBuy/4192. Parts you can order are listed with a green "add to bag" button, and parts unavailable are listed with a red "out of stock" button.

Another thing I can think of is if it is possible to see which parts are not in any sets, only in 1 set, in 2 sets, etc. I saw your link in the forum to an "undocumented report" but it returned nothing for me when I tried it, which was about 5 minutes ago.

EDIT: It looks like the inventory was updated to not list any of the parts that appear in 0 sets. :( The rest is still valid, though.
PS - you have done an amazing job with this. You are largely responsible for ushering in a new golden age of LEGO. Great job, your work is greatly appreciated.
-George

Posted by shaase in Canada, 27 Nov 2012 10:12

Huw, yes I've noticed and been paying attention; you almost have complete inventories done in a short period of time.

Not sure if this will affect how you extract the information from Lego.com, but they recently updated the "Missing Parts" section of their website and you can only enter set numbers with a maximum of 5 digits. In the past you could enter sets/gear/collectable minifigures like 3300014, 852293 and 4568039 (respectively); this is no longer the case. So for these sets and and future ones that will have more than 5 digits (don't follow the normal numbering system) will you be able to still obtain the inventories?

In the future will you allow Brickset memebers to complete the inventories? I've notice that sometimes a piece is not inventoried but if you know it's ElementID and replace the number in the following URL http://cache.lego.com/media/bricks/5/2/#######.jpg it does exist it just hasn't been assigned to the set. The Part Browser page may help in Brickset members track down missing parts, as I believe you extracted all the images from the Lego site regardless if it belongs to a set. Is this how people developed the Series 10 & 11 minifigures?

Posted by oldfan in United States, 27 Nov 2012 10:21

Huw,

Looking toward the future: if the parts information is removed from the source, will the information still be present on Brickset? Are you storing the data, or just linking to the source?

Posted by Leemonlime in United States, 27 Nov 2012 10:48

Oh wow. . .that looks incredible. Are there any thoughts about linking to DAT files for a given part as well?

Posted by jengafett in United Kingdom, 27 Nov 2012 11:08

My mum sells lego as a buisness on ebay and uses the bricklink part inventories to see the part number.
But this is one step better as it shows the part number and the part colour number!
She does get a lot of messages saying can i get a 36782903 and it takes her ages finding the piece out, rather than a part number like 2004 in yellow.
These part inventories have to now be better than bricklink!

Posted by Huw in United Kingdom, 27 Nov 2012 11:09

So many questions! Let me address them all in one go:

> I realize there are other sites/tools available for this but will users be able to track individual parts here at Brickset?

I'm not sure whether to or not. Would it really be useful? How many people actually track every piece they own?

> a 'master list' if you will, of all LEGO parts ever created

I could produce a list of every unique design ID fairly easily, but it wouldn't be 'all LEGO parts ever created', just those that are in sets inventoried at LEGO.com, which does go back to 1990-ish but certainly doesn't include them all.

> I wonder HUW, how many hours per DAY in average you for us to improve Brickset? Is there any time left for build something?

Too many, but it's a labour of love. I don't seem to find much time for MOC-ing these days, unfortunately, though.

> would it be possible to indicate whether the part is in stock to order via replacement parts?

That's a great idea. If I can find out how I will do so.

>I saw your link in the forum to an "undocumented report" but it returned nothing for me when I tried it

LEGO contacted me today and were a bit 'upset' that unreleased part images were used to create the CMF 10/11 images over the weekend and have removed them from their server. I've decided that, to prevent embarassing them further, parts which don't appear in an inventory will not be shown any more.

> but they recently updated the "Missing Parts" section of their website and you can only enter set numbers with a maximum of 5 digits

The method I am using does not suffer from that problem; see the list of recently added ones as proof...

> In the future will you allow Brickset members to complete the inventories?

Yes I do plan to do that as a matter of priority next year.

> if the parts information is removed from the source, will the information still be present on Brickset?

The inventories have been saved locally but parts images are referenced at LEGO.com. If they remove them, then they no longer appear here (see above). Having said that, should them being removed become a problem, I have a copy of every one of them on my hard disk and could serve them from Brickset if needed.

> Are there any thoughts about linking to DAT files for a given part as well?

No, I don't plan to link to BrickLink or any other unofficial parts database, because they don't use the same numbering system and maintaining a cross reference would be impossible.

Posted by fxdl96 in Canada, 27 Nov 2012 11:15

your work is much appreciated, thank you!

Posted by bluemodern in United States, 27 Nov 2012 11:23

This is great! Thanks for all your hard work Huw!

Posted by binaryeye in United States, 27 Nov 2012 11:25

This is fantastic. Thanks!

My only suggestion is that when searching for the sets a specific part appears in, it would be helpful if the results listed the quantity of that part in each set, and displayed the sets in order of descending quantity. Of course, I realize this may be a lot more work, because it would require a completely new set view rather than the default set view, which seems to be what is currently used for these results.

Posted by shaase in Canada, 27 Nov 2012 11:42

Huw, thanks for answering our questions; have two more for you.

Currently the Team GB/Series 8 and some recent sets have incomplete inventories due to Lego; can you rerun the script/program you developed should Lego update these in the future?

I now can pretty much can get the information I wanted, which is finding a piece of a certain type/colour when I need to track down more when building a MOCs. Will you eventually build a list where a person can essentially view a list of all the pieces they own? I realize it would be incomplete due to Lego have incomplete inventories but its a start.

Thanks again

Posted by leemcg in United Kingdom, 27 Nov 2012 11:43

It seems that there is some duplication of colours in the LEGO database, If you look at your example of the angle plates, for instance, there are two in bright red (and two in dark orange, bright orange and light purple).

Is this (maybe) because of a slight change in their red colouring?

In any case it's not obvious that the difference is completely helpful in this case. Those parts are surely the same. They even have identical jpgs (different files, but they are pixel for pixel the same), so if it's a change in colour, it's not one that TLG think is visible.

Great stuff!

Posted by binaryeye in United States, 27 Nov 2012 12:08

Sorry, one more suggestion: Could the "in X sets" be added to the expanded inventory page, and be a sortable field? This would make it easier to determine whether a set has any "rare" parts. I realize the expanded inventory is already packed with information, so adding another field may not be possible.

Also, a question: Does the number displayed on the parts tab reflect the number of parts shown on the parts page, or the number of pieces in the set? Put another way, can one assume the inventory is complete if the number on the tab matches the piece count of the set?

Posted by Huw in United Kingdom, 27 Nov 2012 14:09

More responses...

> it would be helpful if the results listed the quantity of that part in each set, and displayed the sets in order of descending quantity

Good idea. Not easily implemented as you suggest but I'll add it to the to-do list to investigate further

> can you rerun the script/program you developed should Lego update these in the future?

Yes. The date an inventory is fetched is recorded in the database. Although it's not doing so yet, I plan to re-fetch them after a certain amount of time in case there are changes

>It seems that there is some duplication of colours in the LEGO database

Indeed there are, and it will be due to LEGO issuing new part numbers, perhaps due to a new mould being used or something. It won't be due to colour variations. Unfortunately there is not much that can be done about it that doesn't involve an ummanageable amount of manual data entry.

> Could the "in X sets" be added to the expanded inventory page, and be a sortable field?

Yes, no problem, I'll do that right away.

> Does the number displayed on the parts tab reflect the number of parts shown on the parts page

At the moment there is no correlation. The number on the tab is that in the set database which has come from the box or LEGO.com But I do plan to start comparing numbers, and provide some sort of probability that an inventory is complete.

Great suggestions and questions, keep 'em coming...

Posted by Duq in Ireland, 27 Nov 2012 14:23

Another improvement for Brickset even if it doesn't all work for me...
At the moment I don't get images in the popup with available colours (Clicking the design ID below a part on the inventory tab).
In a future release will you make the part number and design ID on the full screen / tabular inventory clickable like on the parts tab?
Techie remark: you've got an encoding problem somewhere. The degree symbol isn't rendering properly. See for example http://www.brickset.com/parts/?part=4211135. The name of the slope 45 becomes "Roof Tile 1X2/45°".
Strange thing is, that's what you would get if utf-8 is rendered as ascii but the page is actually rendering as utf-8...
Last request: on that same part detail page, can you put the part name after the number (and in the same font and size) just like you do with sets?

Posted by Huw in United Kingdom, 27 Nov 2012 14:55

> At the moment I don't get images in the popup with available colours

Thanks, now fixed. It was due to the lazy image loading (that you've had problems with before!)

> In a future release will you make the part number and design ID on the full screen / tabular inventory clickable like on the parts tab?

Sure, that will be easy enough

> Techie remark: you've got an encoding problem somewhere.

Yes I did have, when I imported some inventories, hence this problem on some of the parts. I think I've prevented it now, but I need to go through and sort out any parts that were affected.

> can you put the part name after the number

OK, consider it done. I'll start work on some of the requests tomorrow.

Posted by spae in Belgium, 27 Nov 2012 15:02

Splendid! Works fine.
Thank you very much.

Posted by Cam'n'Stu in United Kingdom, 27 Nov 2012 15:30

Great work Huw. Brickset is going from strength to strength recently!

Posted by kangojack in Australia, 27 Nov 2012 16:23

Huw,

What about adding an option for adding parts to you wanted list?
The list could then be exported to excell, and become your shopping list. E.g. shopping on Bricklink.

I often do this via a paper process for purchasing the accessories for the minifigs.

Keep up the good work!

Posted by The Big Legoski in Netherlands, 27 Nov 2012 17:35

Wow!
I am speechless, I am without speech...............
Every time I check this site you (Huw) have added new features, and improved other ones. Fantastic! Really wonderful, great job, this enhanced set inventories and parts info will certainly be useful.

Posted by leemcg in United Kingdom, 27 Nov 2012 17:35

>>It seems that there is some duplication of colours in the LEGO database

>Indeed there are, and it will be due to LEGO issuing new part numbers, perhaps due to a new mould being used or something. It won't be due to colour variations. Unfortunately there is not much that can be done about it that doesn't involve an ummanageable amount of manual data entry.

Ah, that was my original guess (new mould), but it seemed odd that it was in a section that said "all colours of this part". Surely you know then that these are actually the same part, for them to appear on the same summary page.

But I wonder if I wasn't clear - it seems that there are duplicates both due to a different part number (which I suppose would correspond to a different mould), but also where the part number is the same. Your searches above show this well.

So in the angle plate search results, in the 1x2, 2x2 section, there are two which are yellowish green, but these seem to be different parts with different part numbers 92411 and 44728, but in red both of the duplicates have the same part number 44728.

What is it that makes these two "Bright Reds" with the same part number, appear as different parts? Are there many colour IDs with the same name, or maybe something silly like an invisible typo (extra space?) in one of the names.

On the related matter, I can see that it would be potentially manual to match different part numbers and know when some are the same part, or where the difference matters. Although there may be ways around manual data entry. You've already used text matching (I presume - to group all the 1x2,2x2 plates), and you could also consider image matching. If LEGO use the same image (to the pixel) and the same name, surely we could consider them the same part?

Brilliant stuff!

Posted by Lego_Moose in Canada, 27 Nov 2012 18:07

Thanks for the features Huw.
Only thing is I would love a direct link to add a part to Bricklink wanted list.
But from what I understand that this is not possible due to different part numbers.
Love being able to see what other sets a part is in and what colours it came in.
To be really useful though you need inventories for all sets.
Hopefully the Brickset community can help with that in the future.
Also noticed that you have inventory for #10040 but not for the original #6285 .
Will this be easy to fix for other rereleased sets.

Posted by LEGOIndianaMogens in Denmark, 27 Nov 2012 19:06

Those people at LEGO are not very good at naming pieces. Some are poorly translated from Danish, like the coffin lid from Monster Fighters (chest lid), others aren't even translated, like the Fabuland shovel (spade).
But it's nice to see ID numbers for pieces that aren't in LEGO Digital Designer. I'm quite interested in those kinds of things.

About the dublicated parts. My farther works at LEGO, and he says when a piece is not used in a while, the partnumber becomes invalid or something, and next time they use that piece, it recieves a new partnumber.
He also told me, old pieces from the 70's - mid 90's partnumbers were just the part's ID number + color ID number (2x4 brick (3001) + red (21) = 300121).
When i one day looked at the part list for 6242 soldier's fort's instructions, i noticed the pirate pieces' numbers were changed, because they haven't been used in many years.

Posted by multiplex in Canada, 27 Nov 2012 20:49

Thanks Huw, this is a very useful feature

Posted by ninjagoyo in United States, 28 Nov 2012 01:22

Huw, you are a beast! Rock on!

Posted by Huw in United Kingdom, 28 Nov 2012 03:41

> Ah, that was my original guess (new mould), but it seemed odd that it was in a section that said "all colours of this part". Surely you know then that these are actually the same part, for them to appear on the same summary page.

Yes indeed, design number and colour name will match. But there is still the problem that in LEGO's inventories, some sets will have a 2x2 black plate with number xxxx and others will have what to us appears to be the same part but with number yyyy. That is why, for now at least, they will need to remain as separate entries in the list.

> Only thing is I would love a direct link to add a part to Bricklink wanted list.

I do not plan integrating with BrickLink, and not only because of the part number issue.

Posted by leemcg in United Kingdom, 28 Nov 2012 06:02

^ I quite understand. And having looked into it a little now, I can see a reason not to link those different pieces together (independent of the data structuring issues), and that's that LEGO doesn't do so, so won't understand if you need to talk to them.

The new section on the replacement parts area "buy parts", is clearly what LEGO used to use when you phoned them up. I recently needed some sand green 1x3 slopes, which I couldn't find on BrickLink in the UK. When I asked LEGO, I gave them an arbitrary set number that they were in. No, sorry they said. Having looked at the way you've displayed the data, I can now see that if only I'd chosen the most recent set, 4738, they would have found the piece just fine, and that I was asking about a different older 1x3 sand green slope.

I would therefore definitely like to second the request above for the "availability at LEGO" (and maybe price) to be available. Maybe with a checkbox at the top, which could shade out unavailable elements, in the way LEGO does.

Posted by Huw in United Kingdom, 28 Nov 2012 07:08

^ The availability data is exposed through the means I obtain inventories, but obviously as it's time-sensisitive I can't cache it so I'd need to check every time an inventory is requested. Certainly not impossible by any means but not a 5 minute job. So, leave it with me... :)

Posted by leemcg in United Kingdom, 28 Nov 2012 08:13

It might not be as time sensitive as it seems. Unavailable seems to be generally "not being made any more, old pieces", rather than "Out of stock for now". I'd certainly be happy enough for it to be cached from even a month ago, or even a once in a while manual refresh if people notice there has been a major change.

Posted by Huw in United Kingdom, 28 Nov 2012 10:20

I've implemented live stock checking on parts lists, e.g.

http://www.brickset.com/detail/inventory/?Set=10000-1

It goes off to LEGO.com and checks if parts are in stock, then shows a red or green icon in the right column.

You may need to press ctrl-F5 to refresh the stylesheet to see it properly.

Next step will be to save the stock levels locally, and show them in other parts views.

Posted by Duq in Ireland, 28 Nov 2012 15:13

Thanks for the quick turnaround Huw, that's brilliant!

Posted by georgebjones in United States, 28 Nov 2012 15:39

Awesome job, Huw!! This is so fantastic. I love data and more data. There are so many possibilities. One other thing. Would it be possible to search for parts based on their "rarity?" For example, see which parts show up in only 1 set, or 5 or less sets, maybe which parts are in sets from a certain date range? A complicated example would be:
All parts that were in only 2 or fewer sets in the color Dark Green from 2006-2009.

I don't know if that is making any sense. I don't know how complicated you want it to be, either. Sometimes giving users too much control can backfire and I certainly would not want that.

Posted by Huw in United Kingdom, 28 Nov 2012 17:17

All that can be done fairly easily, in fact the database stored procedure supports that query already, but building a web interface might take a little while.

Posted by legoguru in Hong Kong, 28 Nov 2012 23:28

That is excellent, thanks brick set team

Posted by AlexWeston in United Kingdom, 29 Nov 2012 01:58

Fantastic new features - well done for all your hard work.

Posted by masterX244 in Germany, 29 Nov 2012 06:22

When that goes on and gets better on the old sets inventories you could almost make peeron useless and your site to a all in one one except to part shopping

Posted by Duq in Ireland, 29 Nov 2012 12:10

Huw, can the parts tables be added to the Advanced Query tool? That would give georgebjones and others (like me...) a nice toy -sorry, tool to play with.

Posted by Huw in United Kingdom, 29 Nov 2012 12:38

20

Posted by Acadia in United States, 02 Dec 2012 14:14

I'm now going to stop using bricklink for inventories. It isn't as accurate as your site Huw.

Commenting has ended on this article.