Sometime Sunday when most of the WBBians are asleep, we’ll be performing a MySQL upgrade. This is should only result in at most 30 minutes of downtime.

We’ve been informed by the DC the network issue has been resolved.

As of the time of this post, WBB is experiencing slowness. This is due to a network problem and is not related to our servers. Over the past few minutes it has gotten worse.

Our datacenter has been contacted of this matter and we hope to have a fully operational site soon.

We’re sorry for the inconvenience.

In the past two days we have been performing fixes to the search index database and the usual pruning of our Graveyard. We were faced with the task of pruning 200,000 topics and many more replies. This process resulted in either script crashes with the prune script itself (now fixed) or lag in loading pages. Because we would prune 100,000 posts (not topics) at a time, this caused INNODB to constantly flush the delete updates to disk constantly.

This flushing causes dips in performance where for a second, MySQL will stop responding to requests. This performance phenomena is illustrated below:

Previously we were the orange colored line (XtraDB5). This seemingly small single second penalty (illustrated by the dips on the graph) results in notable lag because the number of requests that needs to be served within that second adds up.

With the fix detailed in this post, we enforced constant predictable flushing updates behavior to disk. This results in less maximum performance but higher average performance as illustrated by XtraDB5-adaptive (yellow).

Ultimately, this means the performance of the site will not lag in the future when faced with intensive updates maintenance such as pruning.

One of the tools that I use to admin servers used to be DNSreport from DNSstuff.com. It was an invaluable tool to diagnose problems related to DNS and emailing. Most of the basic issues are sometimes overlooked whilst configuring servers and it can drive you up the wall if you don’t cover the basics again (properly).

Since DNSstuff requires you to pay (atleast the last time I checked), once my subscription expired I looked else where.

DNSsy is one of those tools that I now use. It does a similar job to DNSreport but it is free and easy to use. We recently fixed a bug with emailing with the help of DNSsy.