Redeemer.Blog

This post is mostly for historical purposes, and for those who are interested in this sort of tech babble.


As of this morning, Neoseeker and all its associated projects have been migrated to UTF-8. This was a long and arduous process that tomchu completed for us successfully. All told over some 72hrs of work was required by various people to finish this update which will largely go unnoticed and has no immediate feature impact on the site.

Background

This entire process started with initial research and testing I did many years ago when we determined that foreign characters, especially Japanese titles of games, were not stored properly on Neoseeker. At the time it was deemed not feasible to make a conversion due to lack of manpower and understanding of character sets in general. The world itself was not quite ready either and MySQL especially had only just started really supporting character collations in MySQL 4.1.

In Feb 2008 key zer reported a bug relating to Russian text being counted improperly. I thought it was something we could fix easily, but after 20hrs of research, testing, and various other work decided it was just too tough, and the risk of corruption of existing data too high.

In Feb 2009 I decided we would take this project up once again, spurred by the need to properly support collation of accented European characters in search features we will be introducing on the site. This time around, tomchu, our Server Dude was tasked with the majority of the R&D of the update.

It was decided that the entire site had to be disabled for this update to be done properly, and the data dumped completely into SQL dump files and reimported after some manipulation. This procedure is remarkably similar to the recommended procedure to upgrade major variants of MySQL (our database software) itself, so we decided to kill 2 birds with one stone.

And now, March 2009 we accomplish 2 of 3 main "core" updates to our site platform: support for UTF-8 and migrating to MySQL 5.

What now?

This was not an easy journey and we're not done yet - alot of code has to be tweaked still on TOP of what's already been done, and many UTF-8 quirks have surfaced, but now that we're up and running again we can look forward to the minor but definitely tangible benefits of supporting non-ASCII characters straight in the site.