Displaying set inventories

 

  • Displaying set inventories

    Posted by Huw, 29 Aug 2012 04:18. Filed under Brickset.

    A feature that is often requested is to display set inventories. To do so requires a massive amount of data that Brickset does not currently have. There are three sites that do:

    • BrickLink: This has the most complete set of inventories that have been collated over the last 12 years. Unfortunately, despite me requesting permission to license their data so I can display it here, I have not been able to reach agreement with their community, some of whom seem to be a little short-sighted in terms of seeing the likely benefit for BrickLink of doing so (it would drive traffic their way and generally raise awareness of the site)
    • Peeron: Peeron has been in existence for a similar amount of time and has amassed a large number of inventories independently of BrickLink. Unfortunately it's as good as dead now, and no new inventories have been added since March.
    • LEGO.com customer services: LEGO publishes partial inventories on their replacement parts page and this is a source that hasn't been 'exploited' by the fan community, until now....

    The LEGO Customer Services replacement parts page is not designed to show complete set inventories and to be honest it's a bit of a PITA to use given it shows so few parts at a time, so I asked super-coder Brent to write something to extract the inventory data from the page, which I could then save in the Brickset database. He did just that and I've run it and the results are shown on the set details pages, top row of tabs.

    There are inventories for over 4500 sets. Most are from 1998 onwards but there are few from as far back as 1980. The different parts in a set can be ascertained, but not the quantities. For some reason, it appears that none of the inventories are complete. I have no idea why this is, but if you look at, say, Palpatine's Arrest, it's as if the most desirable/rare parts are missing, perhaps deliberately.

    Anyway, the script has run, the data is available and displayed on the set details pages. But is it actually any use? I'm not convinced that it is, but I welcome your views. Could the data form the basis of more comprehensive inventories compiled by Brickset users, using the data that's there and comparing it with that in the back of the instructions? That would not be without its problems and would still be a massive undertaking but it's certainly possible...


    48 comments, latest one posted 8 months ago. (Commenting has ended on this article)
    5001 views


 
 

Comments

Posted by Alemas in Germany, 29 Aug 2012 05:01

Ok, found a bug already: The Bricklink link in the Parts tab opens two tabs instead of one. :P

Posted by Huw in United Kingdom, 29 Aug 2012 05:18

Only, it seems, if you use the link to the tab contents directly, but not in normal use.

Posted by CapnRex101 in United Kingdom, 29 Aug 2012 05:22

^^ I am not getting that problem, it opens as a new tab as I would have expected.

I will check as many instruction manuals as I have to hand, mostly Star Wars. Having quickly checked the inventory for Saesee Tiin's Jedi Starfighter, I have found that one of the white 3x1 slopes is missing, one is displayed but there are in fact two in the set.

I have to say though, having got used to Bricklink fulfilling this function, I might simply forget that Brickset does it as well now, and go to Bricklink automatically. Only time will tell I suppose and any new functions are welcome. I do find Bricklink occasionally difficult to navigate, and Peeron is just too slow on the uptake to be of any use I find.

Posted by Mickitat in Germany, 29 Aug 2012 05:28

Too bad Bricklink is not playing along. Even though their database is great, the user interface is desastrous. It would have been great to see it implemented here with a decent interface.

Posted by Holodoc in Netherlands, 29 Aug 2012 05:29

If there is a way to extract the information of the parts lists of the building instructions, we can use the official data from TLC for it.
I don't know if any OCR-program can solve that, but on the parts lists, there are always the thumbnail pics of the part and underneath you get the amount and Lego ID of the brick.
On the other hand this would deliver pure data - hard to read, as the color is hidden in the ID as well.
Like
20x 2347654
1x 4357633
...

That data would have to be "pimped" to make it readable: Find the right icon to put the info next to and have the list sorted, either by color (the BL-way) or by brick (Peeron, which I prefer).
Doable? OCR-experts, please take over! ;-)

Posted by Matthew in United Kingdom, 29 Aug 2012 05:47

^ That would take a lot of typing...

Posted by Bab in Italy, 29 Aug 2012 05:55

Good job! But, there's something necessary that's missing...

1) The real useful inventory of a set, must contains parts counts for each part.
2) I'm a programmer, so I always try to avoid redudancy. :-) This is the case. With BrickLink that has the biggest and most complete (after LEGO's one, obviously) inventory catalog, it's just a loss of time and wasted effort for you to implement and mantain another database and its own "scraper".

I'm just curious... Can I ask you what's the matter with BrickLink?
Why they don't agree with a direct access to their data?
If I'm not wrong, there's a Export Catalog (or something...) function on BrickLink that can export the entire database. And anyway, what about if they make available a web service for the Inventory function?

Posted by brix in United Kingdom, 29 Aug 2012 05:56

You could use this information to make a 'my pieces' section at the top. This would be very useful if you want to find out how many of a certain brick your collection contains in order to use them for a moc

Posted by CapnRex101 in United Kingdom, 29 Aug 2012 05:59

^ I absolutely second that idea, that would be excellent!

Posted by bezetkace in Poland, 29 Aug 2012 06:11

Why is Ki-Adi-Mundi listed in Palpatine's Arrest?
There are 6 not 7 minifigs in this set (no Ki-Adi-Mundi).

Posted by Purno in Netherlands, 29 Aug 2012 06:38

Is there anything preventing you from designing a script which just reads BrickLink pages and copies its info to your own database? A BrickLink inventory page shows all the info a script would need, in a logical way. I'm no programmer however. Just brainstorming here.

I guess it won't be the most efficient way, but if they don't like to co-operate, so be it. I'm not sure if they actually own the copyright on their inventory data?

Posted by caperberry in United Kingdom, 29 Aug 2012 07:01

In lieu of BL inventories I think this is a worthwhile feature, I really like the layout actually - 'at a glance' arranged by colour.

I'd make the "incomplete" warning at the top super-big and highlighted though!

Posted by Huw in United Kingdom, 29 Aug 2012 07:06

I can, and have, extracted the data from BrickLink using their export function but as I say I cannot reach agreement with them there to display it here.

Believe me I have tried but some of the more vocal members of their community are unable to see the benefits to BrickLink of doing so.

@bezetkace, because I *do* have permission to use BrickLink's minifig information (given before Dan passed) and that's where it comes from. The data is incorrect at BrickLink and if you see the note underneath the images you can submit a change request to get it changed and I encourage you to do so.

Posted by BrianOg in France, 29 Aug 2012 07:11

I'm happy enough just to click through on the links you provide to the bricklink page Huw

Posted by moss in United Kingdom, 29 Aug 2012 07:27

excellent work!

It would be great if we could export out the parts list in a XML file, we can then load it into BrickLink through their upload tool. Although we would need the parts quantity as @Bab mentioned to make it work completely.

Posted by Cerlin in Austria, 29 Aug 2012 07:33

It's a shame about Peeron though. I loved the "try to build" function, that shows you what parts in what colour your collection is missing to build a set you don't own. Is there a similar function on bricklink?

Posted by Pravi_diablo in Slovenia, 29 Aug 2012 07:39

what about rebrickable.com? I like fresh design and a lot of inventories.

Posted by bluelion3 in United States, 29 Aug 2012 07:42

If you start with the inventory data obtained from the LEGO replacement parts page, you could allow the Brickset community to update and validate the inventories. On day one, you would not have the complete inventory for all sets, but over time, people would update them and you would eventually have better data. I think this is what Bricklink does anyway. When a new set appears, it takes a little while until a Bricklink member enters the inventory to the site and another validates it. This would be a sort of open source approach to the inventories, and would be prone to some errors, but the validation step would help that.

Posted by Huw in United Kingdom, 29 Aug 2012 07:46

^^ Rebrickable gets theirs from BrickLink.

Posted by RikTheVeggie in Canada, 29 Aug 2012 08:30

I agree with bluelion3, a community effort to validate and complete this data would be the best idea IMO. With official instructions available to download from LEGO, it would even be possible for people to verify sets they do not own. Maybe have a forum group dedicated to verifying the sets so we can organise ourselves to do so? I'd recommend at least 2 independent verifications by different people before a listing is marked as 'verified correct & complete'.

With regards to data, I have a similar problem with minifig images for the myBrickset app - as it stands I'm currently unable to get my hands on any images I can get permission to store offline (low quality) and display on demand (high quality) which means I'm unable to include minifigs in the app (several people have asked for this). Luckily Huw's such a nice chap he gave me permission to use all of his photos for sets & collectable minifigs :)

Posted by emilewski in United States, 29 Aug 2012 08:55

To be blunt, the new parts function is insufficient. We need an accurate inventory, we need quantities, we need part description and we need to be able to sort by all of those fields. What we need is what Huw has already programmed using the Bricklink export function. All we are lacking is the permission from Bricklink.

I have tried to re-energize that discussion on their side:

http://www.bricklink.com/message.asp?ID=619350

But ultimately, don't we just need Eric's permission, not the Bricklink community? It is a private business, not a co-op after all.

Failing that, the ability for the Brickset community to upload and maintain parts lists would work, but is a huge duplication of effort since that is already being done on Bricklink.

Posted by BeardedCastleGuy in United States, 29 Aug 2012 09:04

I like it, even if it is incomplete, I can see this being useful already. I would echo the idea of seeing if some mechanism being setup to verify contents (and as an improvement perhaps adding part quantities). I know I'd be willing to verify/add inventories for some sets if it meant it went into an accessible DB here (aka a version of the spiffy looking parts page Huw has above).

It's too bad that Bricklink is being short sighted on the inventory issue but not much we can do about that (other than perhaps setting up a inventory FB system here where you don't have to be worried about their issues.

With the 4500 sets you currently have at least some information for it would take some time to very parts (and or add quantities) even with a community effort. One other thought, if you do go forward with creating a more complete parts DB perhaps you could see if you can get permission to get the older part inventory info from Peeron? I know I contributed one or two inventories there long ago.

I can also see the downside to doing this, more work for Huw (and likely the other admins) and heavier server load so even if this doesn't turn into a more mature product (or even stay around) I *DO* appreciate that Huw is continuing to explore way to make Brickset even better.

Posted by DarthZared in United Kingdom, 29 Aug 2012 09:04

I like the idea Huw, but with all due respect, why have a new feature when it isn't complete or there are problems with it? And if this website is called Brickset, not Brickpart, then why do we need the add-on? A link to Bricklink's individual inventories for each set in the side-bar or if they just won't give permission then maybe your point about Brickset users editing the part lists to fix and update them would be a good idea, as I'm sure many of us would love to help out doing so. :)

Posted by jengafett in United Kingdom, 29 Aug 2012 09:41

As my mum is selling lego she always wants to find the part number. It would be good if brickset could have a set inventory as it would really help my mums job as peeron hasnt got nearly any of the new sets, which is a bother as my mum is just picking up newer sets which people have grown out of.

I also think it would help other people with there sets if they lose a piece from a set or have seen a piece they want to build a MOC with and want to know what its called.

I know it would take a lot of work but maybe the brickset community could contribute?

Posted by shaase in Canada, 29 Aug 2012 09:49

I was sorry to see Peeron inventories were slowly becoming less and less frequent over the last couple of years. I prefer the inventories from Peeron/Rebrickable as the images for a piece is usually the same (and nicer) regardless of colour. Bricklink same piece images seem to be a mixture of images from number of sources.

I definitely would like to see inventories here as then, really we would only have need for one site; unless you are looking to buy/sell (but links are provided here). Brickset would then have set details, minifig details, inventories, instructions, reviews and other useful tidbits about a set.

It would be nice if Bricklink would help out since most of us probably use bricklink as well. I would hate for this community to start from scratch when something has already been done, but perhaps starting with Lego's Customer Services replacement parts page is a start and then the Brickset community could work at revising the inventories when errors are found.

The only problem with Lego is I've become use to the part numbers used by peeron/bricklink/rebrickable

Posted by richmannn1 in United Kingdom, 29 Aug 2012 09:55

That feature is soooo cool Huw. Thanks a lot.

Posted by lomis in United States, 29 Aug 2012 10:02

I like it!

Three things:
1) Allow addition of parts where the LEGO Customer Services replacement parts page did not have the part or for older sets.
2) Change the view from a top-to-bottom inline view to a table view (with columns).
3) As others have mentioned: add the number of each piece in the set.

Otherwise, this is a great start.

Posted by DrDaveWatford in United Kingdom, 29 Aug 2012 10:08

Without the quantity of each part the inventory is of little value IMHO.

Regarding the use of Bricklink data, who or what is the 'Bricklink Community" ? Certainly those store owners too short-sighted to see the benefits of sharing the data (and thus standing against everything that Dan stood for, as I understand it) are individuals without any decision-making authority anyway. So genuine question - who is "in charge" there these days ?

Posted by lovaquero in United States, 29 Aug 2012 10:22

^ Good question. If you ask the BL forums it boils down a large handful of very vocal members. If you ask me, no one is in charge. It is kinda like medievil Europe over there - everyone with their own little fiefdom... They are still trying to pick up all the pieces from the site hack many weeks back...

Posted by onejaguar in United States, 29 Aug 2012 10:41

Where does rebrickable.com get their data? They apparently have an API you can use to query it: http://rebrickable.com/api/

Posted by emilewski in United States, 29 Aug 2012 10:52

^^Bricklink is a private business...I would think if Eric owns it, it is his decision alone if he wants to allow Huw to link/use the parts export function. It is a win-win for everyone, including Bricklink (more traffic to their site so more awareness and sales, and if Brickset caches the data locally it will not be a big bandwidth issue).

Posted by Huw in United Kingdom, 29 Aug 2012 11:04

The issue, as I understand it, is not so much with the inventories themselves at BrickLink, which you could argue are in the public domain, and which as you say are reused elsewhere at places like Rebrickable, but the photos. When people submit a photo to BL, apparently they retain copyright on the image.

Had provision been made early in BL's life for (c) to be assigned to BL then the owners (the Jezek family) could make a decision to licence them, but as it is they don't and thus nobody can make a decision.

Posted by shaase in Canada, 29 Aug 2012 11:21

^ Huw then what about obtaining the inventories from Bricklink and using images from Rebrickable, Peeron and Lego (as done in your example)?

Personally I'm not a fan of the piece images on Bricklink

Posted by jimred in United Kingdom, 29 Aug 2012 11:22

^ That's interesting Huw. So if you were able to obtain pictures of all the parts from another source, could you then use the BL inventories?

Posted by Huw in United Kingdom, 29 Aug 2012 11:33

Possibly, yes...

Posted by emilewski in United States, 29 Aug 2012 12:21

Now we are getting somewhere. Supplying pictures for the parts is much easier than trying to create inventory counts from scratch for all the sets. And as shaase said, the Bricklink images are not that great anyway.

Posted by vwong19 in United States, 29 Aug 2012 13:25

Bricklink images are inconsistent and not great. If a photos are redone through the Brickset community, Bricklink would be the ones asking Brickset for permission to use their images...

Posted by DrDaveWatford in United Kingdom, 29 Aug 2012 15:13

Maybe the Bogers (Peeron owners) would be willing to sanction the use of their part images ? That would mean far fewer to photograph de novo than if starting from scratch...

Posted by Duq in Ireland, 29 Aug 2012 19:13

What you have at the moment is useless; an incomplete inventory is as good as having no inventory.
If Brickset would offer only a database of inventories that would be redundant although the user interface at BrickSet would probably be much better than Bricklink (which isn't that hard...).
Where Brickset could offer added value is in combining setlists with inventories, showing how many of a certain part I should have (although after years of Bricklink and Pick a Brick it would be seriously incomplete). That is a function I used to have at Peeron, which imported my setlist from Lugnet. Both those site are unfortunately practically dead and the functionality hasn't been replicated anywhere else as far as I know.
If Brickset is going to offer a parts inventory then I hope the search function will be at least as good as Peeron, which I always preferred to Bricklink.
As for pictures, wouldn't LDraw go a long way for pictures of lots of regular parts? That would give a more consistent look than photos from hundreds of different photographers. For printed parts you would still need photos. Contacting the Bogers is a good idea for those. And of course for minifigs Brickset already has a pretty impressive image library.

Posted by DrDaveWatford in United Kingdom, 29 Aug 2012 20:49

The incomplete LEGO inventories on the Brickset set pages do at least come complete with images of the parts in appropriate colours; would it not therefore be possible to combine the Bricklink INVENTORIES (which are it seems not subject to copyright) and the LEGO inventory PART IMAGES, i.e. combine the two ? I'm assuming that some older part images would not be available in this way, but they could presumably be added by members of the BrickSET community....?

Posted by TBOC in United States, 30 Aug 2012 00:18

Is anyone else upset about Peeron dying out? man, they were the best...

Posted by Brickmatic in United States, 30 Aug 2012 01:42

Wait, so it would be ok to use the Bricklink database without pictures?

I'm not sure it would even matter to get permission when it comes to the data sans pictures. "Databases are protected by U.S. copyright law as compilations, but only if they are selected, coordinated, or arranged in such a way that they constitute an original work of authorship. This copyright protection extends only to the compilation itself, which means that some of the data can be extracted from a protected database without violating the copyright in the database. If the database is unprotected by copyright law, the entire database may be copied unless such acts are prohibited by contract." However, in other jurisdictions there may be additional rights afforded to databases. Yet still, Bricklink is based in the US.

On the other hand, the pictures of parts are most certainly protected. But why use such a random assortment of photos? It would be much better to use a standard set of computer generated images. The only problem is that computer models of parts, such as in LDraw, are also copyrighted.

I think a crowd sourced solution to filling gaps in the database would be great. Validation of submitted information would go a long way to keep things accurate. Also, even if you can't use the Bricklink database directly, you should be able to use it to validate Brickset data.

Posted by nthom in Australia, 30 Aug 2012 04:18

Hi, I am the creator of Rebrickable. I have put a lot of effort into mapping Bricklink and LDraw part numbers and automating the generation of my rendered parts images. However, I would be very happy to have Brickset use the images or some other variation I can generate for you.

Also while most of my set inventories do come from Bricklink, over the last few weeks I have implemented functionality that allows users to submit new inventories which has been getting some good use.

FYI if you're not familiar with my parts images, here is a nice example from one of the MOCs: http://rebrickable.com/aj/aj_export_parts?fmt=grid&src=MOC-0112

Posted by jwsmart in United States, 30 Aug 2012 07:58

Just having the LEGO Element ID & Picture is very useful - especially if you ever have to communicate with LEGO customer service to get a part. Thanks Huw!

Posted by guardiangames in United States, 30 Aug 2012 12:29

Bricklink needs some competition. The lack of competition has allowed them to get away with being sub-par in many ways.

Posted by BeardedCastleGuy in United States, 30 Aug 2012 13:04

One of the things that I like about the current, incomplete implementation of the Set inventories is the grouping/layout/view of having all the parts of one color next to each other while also *NOT* having this long list you have to scroll around to find stuff. To me it's a nice clean layout that presents the information in a compact easy to use format. Basically I like the fact it is closer to what LEGO puts into the instructions now-days rather what other websites use. Or to put it more plainly, @Huw the current implementation just 'feels' like Brickset.

Regarding the images, having a consistent standard (or even two standard's) would help give (and maintain) the clean uniform look the current inventory has. If this feature goes ahead and the appropriate agreements reached etc, you could use images such as the Rebrickable ones (for example) with a supplemental standard for photo's in those cases that the nice looking Rebrickable ones are not available. Or if you desire to go with all photo images of the pieces starting out with a consistent standard would ensure uniformity. Specifying a uniform background color, angle of the shot, lighting etc would allow images from a variety of sources to be more seamless. Personally the images generated by @nthom look like they fit the bill for what the inventory feature would require quite nicely...

I will go to Bricklink to find something in an inventory when I have to but honestly the overall UI of the site is dated something of a pita to navigate especially if you don't use it that often.
I'd love, that is LOVE to have set inventory functionality available here because navigation and the site UI/layout are SO much better and to me, intuitive to use. If that means pitching in somehow then I'll pitch in, mind you my photography skills need some work but I could verify inventories or whatnot.

Edit:
@guardiangames, thing is Brickset does NOT compete with Bricklink, Brickset is a Fan community (currently) centered around the Set Guide and Forums. Bricklink is a marketplace for new and used LEGO sets and parts that happens to REQUIRE the inventory functionality to for it to work. I see set inventories @ Brickset as an extension to the Set Guide, an enhancement, versus something that's lack would cause major disruptions to the sites functionality.

Posted by The Lego Train Station in United Kingdom, 31 Aug 2012 06:16

Here are my thoughts: for what I use peeron for, it still seems perfectly useable. I have only used the bricklink inventory once or twice and it wasn't great as an useable interface (with the search functions being annoying and such). If we could use the Bricklink inventories with a better interface and the images from peeron that would be reat. But any way, this website is BrickSET and not BrickPART as somebody else said up there and I am perfectly happy with clicking the links to peeron and such, just my opinions...

Posted by xarquar in United States, 02 Sep 2012 01:23

I haven't seen if anyone has commented on this, but the inventory feature is useful for quick look up of ordering parts for LUGBulk. The reason being that the Lego part/color number is used, which can be a task to look up for a complete LUGBulk order.

Commenting has ended on this article.