What a tumultuous week that was, but Brickset is back and better than ever!
It all started on Tuesday last week when the site was experiencing network errors connecting to the database. The solution, I was told, was to move both servers to a new network. That started on Thursday but for one reason or another took a couple of days to be completed. Then, when it was moved, the server was running perfectly but nobody could connect to it reliably. I suspected a denial of service attack on the site, having found some very dodgy-looking usage in the server logs but the cause is now believed to be a DNS amplification attack on another server on the same network which was causing it to be flooded with bad traffic making the sites hosted there unreachable.
The team at OnRamp Indiana did a great job at trying to control it and keeping me informed of progress but with the problem continuing and no end in sight yesterday morning I felt I had to do something.
I started looking round for alternative hosting providers following similar, but shorter lived problems, last summer so I already had one lined up who understood my requirements and had provided a quote. So it was just a case of pressing the button, as it were.
The order was placed at 8am yesterday, by midday the server had been built, by 2pm I'd configured it and by 4pm the database had been copied. After a quick change to the DNS settings at CloudFlare, Brickset was back on air.
I wondered whether to re-host the old code, or just bite the bullet and launch the new site at the same time. Given that the new site had been used and tested by most of you over the last month and most of the problems had been ironed out, I thought I might as well launch it.
The site is now on a dedicated server and seems to be running incredibly quickly. Complex database queries that used to take 1-2 seconds on the old server now run in about a quarter of that. The CloudFlare content delivery network will also be helping to speed up the delivery of static content to you, from servers located around the world.
After breathing a sigh of relief at tea-time yesterday I thought I'd take a break for the rest of the day and tackle the outstanding problems today. I hope you'll excuse me for doing so.
The known major issues are:
- Country detection: Because all traffic comes through CloudFlare the method I'm currently using to detect your country isn't working. I believe CloudFlare provide another means to determine it so I will be investigating that today. In the meantime you can click on the flag at the top-right to change it. If the flag looks funny press crtl-F5.
- Time zone: The server is now in the UK so the time-zone code needs to be changed, everything is 5 hours ahead of reality at the moment.
- It seems emails sent when resetting passwords are not being received. The server is sending me email OK so I'll delve into the code and figure out the problem later.
- The scheduled tasks to pull in data from Amazon etc. and crunch the database overnight are not running yet.
Finally, for those of you that think the site is too bright: you'll find a setting in your profile where you can elect to have 'Brickset Blue' back again.
Welcome back,everyone. Normal service has been resumed...