New server, new site...
Posted by Huw,
What a tumultuous week that was, but Brickset is back and better than ever!
It all started on Tuesday last week when the site was experiencing network errors connecting to the database. The solution, I was told, was to move both servers to a new network. That started on Thursday but for one reason or another took a couple of days to be completed. Then, when it was moved, the server was running perfectly but nobody could connect to it reliably. I suspected a denial of service attack on the site, having found some very dodgy-looking usage in the server logs but the cause is now believed to be a DNS amplification attack on another server on the same network which was causing it to be flooded with bad traffic making the sites hosted there unreachable.
The team at OnRamp Indiana did a great job at trying to control it and keeping me informed of progress but with the problem continuing and no end in sight yesterday morning I felt I had to do something.
I started looking round for alternative hosting providers following similar, but shorter lived problems, last summer so I already had one lined up who understood my requirements and had provided a quote. So it was just a case of pressing the button, as it were.
The order was placed at 8am yesterday, by midday the server had been built, by 2pm I'd configured it and by 4pm the database had been copied. After a quick change to the DNS settings at CloudFlare, Brickset was back on air.
I wondered whether to re-host the old code, or just bite the bullet and launch the new site at the same time. Given that the new site had been used and tested by most of you over the last month and most of the problems had been ironed out, I thought I might as well launch it.
The site is now on a dedicated server and seems to be running incredibly quickly. Complex database queries that used to take 1-2 seconds on the old server now run in about a quarter of that. The CloudFlare content delivery network will also be helping to speed up the delivery of static content to you, from servers located around the world.
After breathing a sigh of relief at tea-time yesterday I thought I'd take a break for the rest of the day and tackle the outstanding problems today. I hope you'll excuse me for doing so.
The known major issues are:
- Country detection: Because all traffic comes through CloudFlare the method I'm currently using to detect your country isn't working. I believe CloudFlare provide another means to determine it so I will be investigating that today. In the meantime you can click on the flag at the top-right to change it. If the flag looks funny press crtl-F5.
- Time zone: The server is now in the UK so the time-zone code needs to be changed, everything is 5 hours ahead of reality at the moment.
- It seems emails sent when resetting passwords are not being received. The server is sending me email OK so I'll delve into the code and figure out the problem later.
- The scheduled tasks to pull in data from Amazon etc. and crunch the database overnight are not running yet.
Finally, for those of you that think the site is too bright: you'll find a setting in your profile where you can elect to have 'Brickset Blue' back again.
Welcome back,everyone. Normal service has been resumed...
0 likes
81 comments on this article
Congrats and long live Brickset! ))
Well done Huw! I think after the week you've had, you more than deserved a bit of a break last night!
It took about an hour for the password reset email to come through, Outlook had also sent them to junk inbox. All good now.
Thanks for the quick response Huw & for keeping us lurkers updated.
The flag is OK for me (Belgium).
Thank you so much for all your work!
Wow! what a difference. The site is so fast now!
I really missed it last week.
Thank you!
Well done Huw, bet that's a week you don't want to relive anytime soon. Thanks for your all your efforts.
Country detection should now be operating correctly: please let me know.
Flag is all good now. Thanks for the quick response.
Time offset should also be OK now. The default is London time rather than EST now, but if you're logged in you can change it here: http://brickset.com/profile/preferences.
country detection test... aussie aussie aussie
my preferred country and flag shows Australia
but here on the comments it says United States
Not sure where the flag is supposed to be showing, but my profile STILL shows Yugoslavia, even though the country ceased to exist almost 8 years ago...
Congratulations and thank you for your hard work Huw!
Nice one Huw. We appreciate your hard work!
On the new site, how do you update your minifigs owned list and how do you send your sets to rebrickable? Is it all done automatically now? If so, is there a delay for when these changes show up? Because I just added a new set as owned and the changes are not yet there. New site is beautiful BTW.
Great work Huw. New site is superb. Much appreciated!
Three cheers for all involved in getting Brickset up and running again.
Jaffa cakes all round!
Hang on!
Europe?? Europe!!
I'm not in Europe! I'm in the UK!
Thanks for the great work Huw. I love the new "Brickset Blue" option, it's a nice touch that reminds you of the old site, with the amazingness of the new one.
However, like misb, I'm also in Australia and have my location/flag set as Australia, but my country says United States. I guess this is to do with the CloudFlare issue?
Applause from Austria! Good job! Thank you very much!
Ooo. My comments look pretty! Yay!
Love the new look Huw ;)
Okay, it should show Germany as location.... *posting to see*
-> Negative... but time is correct and I have the German flag on top of the site.
everything seems to be working for me, in the usa. Thanks i know it was alot of hard work!
Great new site, thanks for all the updates. Country and flag are correct
Great new site!!!! And it's working faster than ever.
Has this change affected the brickset app? I have tried logging in on both my kindle and phone but it says unable to connect.
@Sammael: Yes, that makes no sense (except in Germany, where Serbia is called Jugoslawien, pronounced Yugoslavien but with a bigger accent no doubt).
^^ Just tested and myBrickset cannot login, I'll investigate after work.
Great job getting the site back up Huw, we missed you! Country flag & timezone showing ok here with no manual changes. Comment country is wrong though, it shows US instead of Canuck-Land.
@ilike...
It is "Serbien" in German too! Since 1992 if I remember the year right.
Well, lets see if my country is still United States.
An aussie here too and it is showing that I'm in the U.S also... but I have the Aussie flag
I've tweaked the country detection code again.
If it's still not showing correctly, clear cookies for brickset.com then view this page: http://brickset.com/_test/country. The country cookie lasts a day so if you can't clear them it should sort itself out tomorrow.
Email me with any anomalies.
Nice to have you back, and in new form!
One bug I just noticed was that when I clicked on the "See more discounts at Amazon.co.uk" link and it directs to a new.brickset.com URL that is isn't being styled (CSS).. I'm guessing it needs to be have the new. part removed!
Oh Canada! country test.. looks great Huw!
Minor issue - posting shows the comment, editing then saving shows a coloured bar but no text until after a page refresh (Windows, Chrome, latest stable version).
^ I noticed that too, I'm guessing it's a CloudFlare cache thing but I'll add it to the list of things to investigate :-)
^^ Ah yes, I need to change those links, thanks for reminding me.
Another issue I noticed was that when I am on the Amazon Shopper/Price Comparator page and I select to only show sets I want, that page (/buy/amazon/mycollection-wanted) shows the green and orange boxes on the right but there's no tick box in them.
Also, this is probably not a bug, but it'd be nice if the list on that page showed it with theme and year separators, like the all sets page does (/buy/amazon). It's kind of a mess to browse through when there aren't any separators at all.
^ That page does need some work and I'll be attending to it when time permits.
This is to test that comments say I'm in the UK and not the US...
Looks great! Thanks for all of your hard work (and the option to add a little blue back :-)
Great job Huw and thanks.
Yay, brickset, welcome back. I have missed you so much.
Thanks to @huw and the team for working so hard to bring Brickset back to us.
Just noticed my comment is saying I am in Europe not UK. I suppose technically I am in both (until Scotland gets its independence). I will check back in 24 hours to see if it has sorted itself.
Great job on persevering through all the recent difficulties. The new site is looking good!
@Huw, just tried the blue setting, and I like it very mcuh, although I love the new design. On my 24" screen the blue just seems to calm everything down.
@huw, love the optional blue background! Thanks!...(c:
@guilhorn14 and all myBrickset users : there's a confirmed issue communicating with brickset.com. Huw's investigated on his end (thanks Huw!) and supplied a fix, I shall test this tonight and if it works will get an update out ASAP to resolve the issue.
ACM looks like it is not loading the style sheet. It was looking fine earlier today, I don't know what happened but now is not formatted.
^^ Just click on the buy tab and you should be taken to a page where you can select which stores to view discounts from.
"Better than ever" is certainly the truth. I love the new design! When I first accidentally stumbled upon dev.brickset.com and saw this design last year, I was excited. It certainly hasn't disappointed me!
At last. Welcome BACK! Really missed you last week.
Sweet! Glad to see Brickset back up, and the new one running. Out of curiosity, how do I change my avatar?
The avatars are setup at Gravatar.com, something you've done already by the looks od things.
Well everything is back to normal here (except for the site-design obviousley), great to have the Brickset-blue back again.
One little bug, which may have been present on the old site already, but I just discovered: when I click "My Sets" there's the announcement how many of my sets need a better image. There are two that already have official (and in one case quite large) Lego-images.
Keep up the good work.
Site looks great, I'm glad Brickset is back :)
Nice job and most definitely faster and better experience both from a page loading and overall new look to site.
One more comment, I'd like to request that the (settings-enabled) blue background colour match the blue background in the new logo, instead of using the old "Brickset Blue". The new colour scheme is much more satured and "pop", and the old blue is a bit bleak and mismatching.
By klatu003 in United States, 05 Feb 2014 09:21
Thanks Huw for restoring Brickset. Take a break and pat yourself on the back.
"You don't miss your water, until the well runs dry" and last week was mighty thirsty in the AFOL world without your essential database and news.
I, for one, love the new bright, fresh look.
Yeah! Whahoo! Brickset is back!
@Huw: I used the comment field of "my sets", now this field disappeared. Are the data lost, or do you reintegrate this function including the data?
Brilliant! Good to see Brickset is back up and running.
With "Brickset Blue", things look a touch more familiar. :P
Thanks. :)
Is that going to affect the app that I have for my kindle and phone?
@tsi, it's shown on the set details page now. Very few people were actually using it, so it, and a few other lesser used features, have been removed. This article explains the reasoning behind doing so:
http://boagworld.com/usability/stop-focusing-on-edge-cases/
I love this new site.
Great job on sorting out the timezone issues Huw!
*problem fixed itself* :-)
Big thumbs up for all you hard work Huw. The new site is great.
I was terrified because the site wouldn't load and I couldn't get my lego fix. And today, it finally loaded. And EVERYTHING WAS GOOD WITH THE WORLD. The update is wonderful and the mobile site is flipping great. I can't wait for my wife to roll her eyes as I scroll endlessly on this new and improved site! WOOHOO.
@Huw: Your calm and patient demeanor, your dedication to your users and your rational approach are amazing. It is rare to see a large "niche online community" which is run by someone so sincere and capable.
So many communities are led by arrogant dictators, folks who mean well but get wrapped up in the cult of themselves. Folks unwilling to take critical feedback, and unwilling to take risks. Glad to see the converse is possible too.
Last year, I wrote an article about Niche online communities which might be interested to you and the readers of this site @ http://tomalphin.com/2013/02/niche-online-communities.html
Likewise, I did another post looking critically at a popular niche online community website, and showing how it could be adapted to a modern app for tablets and phones. It may also interest your readers, especially folks who have built apps to access your site. @ http://tomalphin.com/2013/05/adapting-an-existing-website-to-a-modern-application.html
Sincerely,
---tom
(P.S. folks may also be interested in my lego Architecture 30-day challenge @ http://tomalphin.com/2013/10/lego-architecture-studio-30-day-challenge.html )
^ That first article is spot-on and if you changed 'beer' for 'LEGO' you'd be talking about Brickset exactly :-)
Sorry to keep commenting, but now Brickset keeps logging me out every time I close a browsing session, even when "Remember me" is checked. Before the new site was launched, I used to be able to stay logged in (on both old and new, I must point out) for months on end, now I have to log in every few hours as I close my browser frequently. anyone else having this problem or is it because I cleared all my cookies for the site yesterday? And if it's not just me, can it be rectified please? :-) I hate having to log in every time I open Brickset ;-)
But as many others have said, great job on the new site otherwise Huw! :-)
I haven't notice that personally. The user-identifying cookie expiration is still set for a very long time. Mine expires on Wed, 02 Nov 2016 09:11:02 GMT.
If you have FireFox and the web developer toolbar (or other means of viewing cookies) check what it says for the one called ASPXAUTH.
Nice job on the background option Huw! Looks perfect. And like almost everyone says, thanks for all your hard work and dedication to everyone at Brickset.
It's great to have Brickset back! Over the last week, I was really struggling to get info. Goes to show how much Brickset is part of your life!
Awesome! I love this. Thank you so much, Huw!
^^^^I've been having the same problem as TheOneVeyronian. I use Google Chrome, if it makes any difference. I only noticed it after all of the server problems.
Again, thanks for all your hard work on the site! The Brickset Blue option is really nice!
^ It should be sorted now, log off and on again and you should get a persistent cookie.
Awesome. :P Seriously though, it's good to see Brickset back up and running smoothly. Thanks Brickset team for all the work you guys do in keeping us LEGO fans "informed."
Sort of missing the "Brickset Blue," but I imagine that will pass. This new site is great.
Go to your profile page and you can get it back.
The new site looks great and loads so quickly! Thanks for all of your hard work, Huw and everyone else involved!
Is the link to iOS 'Brick by Brick' broken? I can't view sets, catalog, etc.
Congratulations with the new site and thank you for all your hard work!
AMAZING... The result is brilliant! Much better than anticipated! Congratulations to you and the team behind this great homepage. Thanks for letting us use it!
The country is now correctly showing as Serbia, woohoo! Thanks, Huw!