A small configuration error prevented the avatars from being synced. This has now been fixed.

Since last night I am glad to say that indexing the search doesn’t appear to be causing any 500 errors anymore. The culprit was due to the fact that we would ask for 250 posts each query and this caused some locking issues with MySQL.

A “locking issue” is caused when we’re trying to read a part of the database which is being written to, as a result that read query is blocked or stopped and caused to wait before the writes are submitted.

By reducing the number of posts to read at each query to 10, we’ve decreased locking contention and improved smoothness in indexing the search.

The bad news?

Well, the 500 errors are still occurring and we need to see what queries is now causing it. The worse case that we could see is that our resources on both servers are maxed out already and we need more capacity for PHP and MySQL to grow.

Sorry about that, we were testing out an automation script to flush topic views to MySQL before we restart Memcached. We were also testing out an automation script to keep Memcached running even if it suddenly stops running.

There has been complaints about Warez-BB redirecting to an Islamic site. The reason why is because you have been going to our domain name using .com when our actual domain name is Warez-BB.org.

Please make sure that you go to the correct domain name. The .com domain name is not owned by us. That owner redirects some traffic to us and some to preach about Mohamad and Islam. Here at WBB, we believe that the person doing this is actually giving Islam a bad name. And whoever is doing this should stop this domain hijacking behavior or else punitive action will take place.

After the upgrade to one of our load balancing servers, we have had a series of issues to work our way through. These issues were not easy to solve as they ranged from kernel bugs (hard to find and reproduce) to script bugs (easy peasy).

Here is a list of issues we’re currently working on and the efforts we’re putting in to solve it:

1. The most important of all is the slowness at peak hours along with a few 500 errors. Though we did promise to rid the 500 errors for good prior to our server upgrade, the current 500 errors are caused by a phenomena completely different. The current issue is that we have decreased the number of PHP processes on our main database server and shifted most of the PHP processing to our new Quad core. This should give more resource leeway for MySQL to do it’s thing on our db server. However, from observation so far, the connection between the Quad and db server drops for PHP processes when the search begins to reindex every 2 hours.

The effort? Those with some knowledge about how servers work will realize there is no correlation between the reindexing of the searching (reindexing is done via a C++ program) with the connection between PHP processes. This is why it is such a hard issue to narrow down and reproduce. It was only a few days ago did we realize this, and we have now started playing with the parameters between the search and the PHP processes to narrow this down further. It is only through the behavioral changes through our changed settings can we narrow this down.

Most of my effort so far has been placed towards this.

2. Avatars and emails.

Our avatar problem was just fixed 12 hours ago. The reason why is because we did not notice that we needed to switch servers back from avt2.warez-bb.org to avt.warez-bb.org. Avt2 currently does not have the syncing of avatars set up, so all uploaded avatars was not viewable until we made the switch in our script.

Our emailing problem was due an unknown error between our emailing server (img5) and our servers in HK. We had added a secondary IP to img5 recently and was set as eth0:1. For some stupid and still unknown reason, the main IP that the server uses to connect to the outside world has switched from eth0 to eth0:1. In all my life working on servers, eth0 (or ethX) should always be the main IP. So if someone is reading this and has knowledge of where to configure the main IP for CentOS/Redhat, please contact me via PMing one of our Administrators.

Currently, we’ve authorized our second IP to access our HK servers which is why emailing once again works. To avoid sending too many emails at once, I have removed all pending emails that were in the queue about new replies and PM notifications. However, as of now, all notifications and all emails will be sent out as usual.

3. A large number of our members are receiving email notifications regarding incorrect log ins. Though we have moved to firewall ban all IPs that try and log into multiple accounts, more aggressive techniques are going to be used to thwart incorrect log ins.

I think we may have just fixed the root cause of these session errors. Only time will tell but at the moment we’re pretty happy with how well it’s performing again.

We’ll be currently restarting our server. BRB.

We’re just rebooting the server as we encountered a small problem. Things should be bouncing back shortly.

The bugs related to sessions and search have been fixed. Registrations are now open. However, memcache server just went down – this issue will be fixed as soon as possible.

I’m pleased to announce that the current Quad core server that had recently replaced the Pentium D server is running as expected. It has taken considerable load off our other servers and everything is churning along under load – even at peak hours.

We’re however, still stuck with the same session and search duplication bug. We’re unsure the exact cause of this as it worked perfectly fine with our old server so this may have something to do with the new binary compilation. And it has a little bit to do with a design fault made by phpBB devs. There are a few ways to fix this, but since redesigning the part which generates the random variable hasn’t worked. I suspect it could be due to either the new server or simply too many people logging on now that we have increased capacity.

Whatever it is, we will fix these before we activating registrations again.