We managed to catch this error in the act and have patched up our script to prevent the two most common conditions that has been causing this error.

There has been reports on members from India not being able to access our site. We’re aware of this issue and based on our research this appears to be a problem with the network rather than something that we can fix ourselves. We have already reported this issue to our DC who will forward it to their bandwidth provider and go through the red tape from there.

We’re pleased to announce that from here on now, the 500 errors that plagued our community has been irradiated.

The 500 errors at the moment are due to our efforts to optimize the server.

I’ll save the technical stuff for later, all you need to know is that this upgrade increases performance and it’s going to have a downtime of roughly 2 hours.

A small configuration error prevented the avatars from being synced. This has now been fixed.

Since last night I am glad to say that indexing the search doesn’t appear to be causing any 500 errors anymore. The culprit was due to the fact that we would ask for 250 posts each query and this caused some locking issues with MySQL.

A “locking issue” is caused when we’re trying to read a part of the database which is being written to, as a result that read query is blocked or stopped and caused to wait before the writes are submitted.

By reducing the number of posts to read at each query to 10, we’ve decreased locking contention and improved smoothness in indexing the search.

The bad news?

Well, the 500 errors are still occurring and we need to see what queries is now causing it. The worse case that we could see is that our resources on both servers are maxed out already and we need more capacity for PHP and MySQL to grow.

Sorry about that, we were testing out an automation script to flush topic views to MySQL before we restart Memcached. We were also testing out an automation script to keep Memcached running even if it suddenly stops running.

There has been complaints about Warez-BB redirecting to an Islamic site. The reason why is because you have been going to our domain name using .com when our actual domain name is Warez-BB.org.

Please make sure that you go to the correct domain name. The .com domain name is not owned by us. That owner redirects some traffic to us and some to preach about Mohamad and Islam. Here at WBB, we believe that the person doing this is actually giving Islam a bad name. And whoever is doing this should stop this domain hijacking behavior or else punitive action will take place.

After the upgrade to one of our load balancing servers, we have had a series of issues to work our way through. These issues were not easy to solve as they ranged from kernel bugs (hard to find and reproduce) to script bugs (easy peasy).

Here is a list of issues we’re currently working on and the efforts we’re putting in to solve it:

1. The most important of all is the slowness at peak hours along with a few 500 errors. Though we did promise to rid the 500 errors for good prior to our server upgrade, the current 500 errors are caused by a phenomena completely different. The current issue is that we have decreased the number of PHP processes on our main database server and shifted most of the PHP processing to our new Quad core. This should give more resource leeway for MySQL to do it’s thing on our db server. However, from observation so far, the connection between the Quad and db server drops for PHP processes when the search begins to reindex every 2 hours.

The effort? Those with some knowledge about how servers work will realize there is no correlation between the reindexing of the searching (reindexing is done via a C++ program) with the connection between PHP processes. This is why it is such a hard issue to narrow down and reproduce. It was only a few days ago did we realize this, and we have now started playing with the parameters between the search and the PHP processes to narrow this down further. It is only through the behavioral changes through our changed settings can we narrow this down.

Most of my effort so far has been placed towards this.

2. Avatars and emails.

Our avatar problem was just fixed 12 hours ago. The reason why is because we did not notice that we needed to switch servers back from avt2.warez-bb.org to avt.warez-bb.org. Avt2 currently does not have the syncing of avatars set up, so all uploaded avatars was not viewable until we made the switch in our script.

Our emailing problem was due an unknown error between our emailing server (img5) and our servers in HK. We had added a secondary IP to img5 recently and was set as eth0:1. For some stupid and still unknown reason, the main IP that the server uses to connect to the outside world has switched from eth0 to eth0:1. In all my life working on servers, eth0 (or ethX) should always be the main IP. So if someone is reading this and has knowledge of where to configure the main IP for CentOS/Redhat, please contact me via PMing one of our Administrators.

Currently, we’ve authorized our second IP to access our HK servers which is why emailing once again works. To avoid sending too many emails at once, I have removed all pending emails that were in the queue about new replies and PM notifications. However, as of now, all notifications and all emails will be sent out as usual.

3. A large number of our members are receiving email notifications regarding incorrect log ins. Though we have moved to firewall ban all IPs that try and log into multiple accounts, more aggressive techniques are going to be used to thwart incorrect log ins.