We’re currently converting a number of INNODB tables over to the new barracuda format.

ETA 3 hours.

Warez-BB is temporarily unreachable. We are working on fixing this issue. We regret the inconvenience caused by this downtime.

EDIT:

Accessibility issues have been fixed.

Boy have we been busy rolling out updates to many aspects of the community. Here is just some things that has been going on behind the scenes.

1. Searching has been improved. The bug that caused Graveyard topics to appear in normal results even when explicitly specifying not to has been fixed. This bug was caused to a flaw in the design of the DB schema. There has also been other enhancements to the search backend. One you will not notice is a decrease in IO when indexing. This tweak will decrease HDD IO by two folds. There has also been a bug fixed in the search where the flood limit feature was not working properly.

2. Index page has had moderators removed. This is currently only a cosmetic change with the code that drives this feature still intact. We plan to remove it when we rewrite the the index page’s PHP code. The biggest problem with the index page is that it requires numerous loops to generate the page. This is unnecessary and we will be using better programming techniques to improve it’s efficiency there. There is also a planned rewrite of the expand/collapse feature on the index page. This is mainly to fix the invalid use of arrays identifiers in HTML, in laymen terms it means we’ll fix the expand/collapse issue with the General category.

3. Fixes to Topic Views. This has probably been said before, however we have altered the script that handles the committing of topic views to the database. The script now scales better and can handle hundreds of thousands of topics that need to be committed compared to only a few ten thousands from before. We have also set up a different memcache process dedicated for buffering topic view updates. This prevents topic view entries from being pushed out of the cache because of other incoming data. Memcache has also been upgraded to their latest versions.

4. MySQL setting tweaks. One of the extremely useful features of the Percona MySQL binaries (64bit) is that it includes a patch that allows for more than one background read/write threads. This is extremely useful on multicore servers such as ours.

MySQL works by using threads within one process because many servers these days are multicore. Back when single core procs were still mainstream, programmers will program only to work on one core. However, with multiple cores in today’s procs, threading is needed to scale up performance because each thread can only work within one core. Having multiple read/write threads increases the number of writes/reads that the system can request to the harddrive and this ultimately improves the performance provided that the HDD hasn’t maxed out already (which in our case, it hasn’t).

There has also been improved contention issues between the log buffer and buffer pool that are detailed here. This also helps MySQL’s performance.

PS. If you’re using the INNODB Plugin version 1.0.3. Make sure you turn on tcmalloc memory allocator with:

innodb_use_sys_malloc = 1

PS2. Useful INNODB tweaking resource here.

What we hope/wish/want in the future.

One of the biggest issues with a large database and a large userbase is that whenever we restart MySQL, we get hit with a large number of requests. Most of which we cannot serve as fast as we normally would. A rough guesstimate is that pages will be generated in 0.5 seconds (compare that to ~0.1 second page generation we normally get). The culprit is because of the slow (10K rpm) harddrive. The data must be read from the harddrive into the memory in order to provide a fast query response.

What we want/wish/hope for is a feature in MySQL that allows us to choose which tables between which ranges (between which topic_id for eg) we want to populate the RAM with first before MySQL begins accepting connections. This is useful because it allows us to warm up the cache first and it also allows us to warm up the cache quicker because a single read between ranges are sequential. And with HDD’s sequential reads/writes are much faster than random reads/writes. FYI, random reads/writes are what we perform when the site goes online.

Currently the only option is to create a script that simulates the activity of a normal user to warm up the cache before we enable the site online. But this method is hacky and would require a lot of work to get the timing correct.

Recently MySQLPerformanceBlog posted a topic about how to shut down an INNODB database faster. Because shutting down MySQL takes over 5 minutes for us, this nifty trick has made us even more trigger happy when restarting MySQL.

We will be shutting MySQL in a few minutes.

Along with this restart, we will also be updating a few settings to reflect the increased load we’re facing.

We’re currently shutting down MySQL for some configuration updates. These updates are to fine tune the database to the workload that WBB faces. We will also be reindexing the search to reflect some fixes that we’ve made to prevent Graveyard results from showing up when we explicitly choose not to show results from that forum.

ETA 1 hour. We will only be down for 15 minutes with the MySQL updates.

Amidst doing a lot of updates to the backend script and hence a lot of uploaded files, I constantly received messages from members about errors. These errors are due to our users loading a page with a half uploaded PHP script. And since PHP is stupid, it’ll parse the file and try and execute it anyway. So to improve the user experience whilst we’re performing maintenance and to alleviate time wasted looking at errors that auto fixes itself once the file is uploaded, I wanted a way to atomically upload a file.

So I searched through the documentation of Proftpd and found a nifty option that is turned off by default. This option basically allows me to upload files to a tmp directory and then move the file to where I intend to upload to, providing an atomic like upload experience.
No longer should you see errors from half uploaded PHP files.

I noticed (along with a handful from my MSN) that our site went down momentarily for a few minutes. There was evident packet loss with the whole C-class network. I’m still not sure what caused it but whatever it is, it has been fixed but I would take it with a grain of salt as it may happen again.

We have just restarted our MySQL server with some new settings to reflect the latest configuration updates.

The tinkering has just started and has shown improvements already.

We have successfully upgraded to XtraDB and Percona MySQL builds. We’re now in the process of converting parts of the database over to the new Barracuda format.

We have performed this upgrade before with the INNODB Plugin, the difference today is that XtraDB is essentially the new INNODB Plugin enhanced by the Percona guys. So kudos to them for the hard work on fixing the scalability issues.

We have just commenced procedings to upgrade our MySQL to the latest version of XtraDB and Percona. This will most definitely result in downtime. A back up is being taken to prevent any data loss.

ETA 3 hours.