Redeemer.Blog

As we gear up for an architecture change and server update for Neoseeker, I thought I'd write some behind the scenes information about our recent v1.0.3 launch of the forums. This post might get a bit geek-techie so please mind that its intended to be somewhat technical.

Normally when we update the site, the process can take anywhere from a few minutes to several hours depending on the scope of the upgrade. Updating files is not a problem, even though we run the site on multiple servers we have a system from which we can deploy necessary files with a point and click interface, and which then rsyncs the entire change to all of the servers.

The v1.0.3 update took a total of several days to launch - so long that for a while we were running at half capacity because some servers were running the old code, and others were running the new code. We had to take the updated servers off the site rotation in order to keep the user experience consistent. Here's the story behind the whole 1.0.3 "fiasco".

Under the Hood

On the surface v1.0.3 had very few changes, though some are nice, like Notifications from private forums and "quick preview" Ajax features that didn't reload pages.

Under the hood alot was done to try to make our work simpler in the future, including:

• Updating to PHP 5.3
• Complying with PHP 5.3 Strict Standards as much as possible (this was a huge whack of work).
• Employing a robust Autoload strategy for class files
• Starting to use a Dependency Injection Framework (DIF) for our common objects
• Rewriting the way moderators are loaded to make it more portable and useable
• Rewriting the posting of threads and replies in anticipation of creating NeoAPI calls for forum posts.
• moving our templates compilation dir into a central common location
• completely change our beta and dev architectures to match the live URL structure
• redesign the NeoAPI framework, and migrate some of our own Ajax calls to use actual NeoAPI calls instead of using our older methods. The preview post feature for instance uses the nTags translate API call.

Neoseeker's codebase is very old and a lot of my time was spent just trying to quell the Strict Standards and rewriting various things.

However, for almost all of the above, the actual time to update the code to the servers would still have been negligible. Our challenge was that we were updating to PHP5.3 for the first time and our new code made use of 5.3 only features like Late Static Binding.

This meant that v1.0.3 would NOT run on a non 5.3 server and since we had yet to move to VM's, every server needs to be compiled with Apache/PHP5.3 individually - a time consuming process. We inevitably hit PHP snags that didn't show up in our beta testing VMs, and as we were changing our code architecture in preparation for our pending 2011 server architecture update we encountered problems relating to those changes too.

Between all that, we were trying out a different deploy method by relying on Source Control checkouts rather than rsync'ing from a local source checkout. This last change seemed innocent enough, but it created more problems that we had anticipated when different servers refused to update selective files at selective times. I can tell you those were fun (ugh).

This lead to MANY hours of troubleshooting and constantly trying to update files to the live servers using a new deploy scheme, with new rewrite rules in Apache and a huge number of other changes. In total we were putting together the first stages of 4 months of planning and changes, the end result of which users would only see a few features they could tangibly use :(.

Tomorrow, we'll hopefully be putting together the second half of that planning: 3 brand new servers and a shift to improved server management by deploying VMWare Virtualised Servers.

This second stage is less software related, and more hardware and server related. We've been testing and qualifying VMWare backed VM images of our FreeBSD servers since December and those have been finalised for a while. We also know the code architecture is ready for the move because we've been running this "VM" ready code since the v1.0.3 launch, and have been using the exact VM image as our beta and development platform for the past 3 months. If you've been part of the Neoseeker Beta team you've actually been using the new VM architecture, running on one of the actual production hardware servers until yesterday.

This is our second foray into using VMs to run the site, the first one didn't work so well and resulted in poor performance and unmanageable issues, so we're crossing our fingers this time around :D.

When we successfully get the new architecture going I'll try to write a post-mortem on how that goes and what was behind the scenes there :).