We are experiencing unexpected difficulties and members might not be able to access the board. Our experts are working on fixing the problem.

We apologize for the inconvenience.

Although we are working hard on getting this fixed, members can do their bit by contacting their ISP telling them how important our site is to you and demand them to unblock it.

Firstly, our apologies for not updating this blog as much as we would have liked to. We make no excuses here, we should have provided more updates.

Our site has been up and down for the past 3 days because of DDoS attacks on our servers. This has caused considerable downtime and we have fought hard on defeating these attacks. Currently, our site is up and running and we hope our latest fix will keep our servers up and running for the next few days.

PS: Our entry title is how you may be feeling after trying and trying and trying to access our beloved site, and failing. We feel your pain. Just don’t go around formatting your computer because some idiot technician told you it would fix your access to WBB. =)

For some odd reason, we’re facing network issues. We’re not sure what it is but we are looking into it with the datacenter.

But for now, we have switched to a new IP which works. And those updates should propagate through the internet within the next 4 hours.

If you cannot wait, you may edit your hosts file.

If you cannot be bothered to read the link provided above, just edit your hosts file and add the following line.

Location of hosts file on windows – C:Windows/system32/drivers/etc/hosts
On *NIX the hosts file is located at – /etc/hosts
You may have to open the hosts file from an elevated notepad, if you use vista or windows 7.

Add the subsequent line to your hosts file

119.42.146.34 www.warez-bb.org

We will be taking the site down for 30 minutes for some quick upgrades now.

We’re experiencing a DDOS attack now. We’re working on mitigating the effects of this attack to enable site access to our members. We apologize for the inconvenience caused.

We’re providing notice that our site is being updated with new colors to our legend. Keep a keen eye on it. =)

There will be no downtime.

Our Easter challenge has officially been completed. It has provided our community with an uplift of spirit and fun for everyone. Exactly 2085 members found an egg out of the 241 eggs hidden away, this is the most amount of eggs we have ever hidden at Easters.

We would like to take this opportunity to again thank all those who have participated in hiding the eggs (our Moderation Team, Development Team, Wadio Team, IRC Team, Press Team, and Design Team). But most importantly, we want to thank and congratulate everyone who found an egg. For those who tried their best but couldn’t find one, there will be plenty more chances next year. We are hoping next to give out even more eggs and make this event bigger than it is today, so make sure you join in the fun when it comes around again.

Bye for now. =)

I thought I’d give some insight into how we fixed this emailing issue and what had been causing it.

At WBB, we do not directly send mail when the forum requires an email to be sent. Instead this email gets serialized and saved into a “buffer” database. A script then runs in the background, going through the “buffer” database to send out the emails.

Because you now know the basis of how our email system works, let’s explain the reason why our members were receiving duplicate emails. Our email script is simple, it reads the latest 20 emails from the buffer database, and sends them out through the mail() function. However, this design is prone to one flaw, that is if two instances of the email script are run at the same time, then one script may read the same latest emails the other script has just read in. This will cause one email to be mailed twice.

Now so far, our Admins have not designed this script to use a locking mechanism. A locking mechanism will basically force one instance to run at once. Only when the running instance has finished executing will it release the lock and allow another instance to be run.

This may sound like a good idea, but it becomes a nightmare to handle when the script begins to fail before the lock is released. Because of this, our Admins have just relied on a simple balance between average time execution, the number of emails batched up per instance and the time the next instance is run to prevent this from happening. This has one major flaw, this balance is fragile and if anything unforeseen unbalances it, then we get the duplication issue.

So, what caused our latest problem? Through probing around, there were two issues.

One, our /var/spool/clientmqueue folder contained around 900MB of files. Considering these files are small in size (i.e. 2KB each), we’re looking at about 460,800 files. This may not be the files limit per folder ext3 has, but it surely is the limit for sendmail. FYI, sendmail is the program that we use to send email.

[root@srv6 clientmqueue]# du –max-depth=1 -h
900M    .

This problem leads to the question, why did the folder become so big? The reason why is because /var/spool/clientmqueue is a folder designed for sendmail to queue files before they’re sent. An email sent using sendmail will first be copied to /var/spool/clientmqueue and whatever requested the email to be sent will assume that it was sent successfully. And in many cases, it does get sent. The second step in the process is a sendmail daemon is run to listen for new files in that folder and copy those files to /var/spool/mqueue to be actually sent out by the server. In essence, these folders act as temporary buffers to improve performance similar to our “buffer” database.

To prune the /var/spool/clientmqueue folder, we cannot use the conventional “rm” command. Instead, we use “find . -type f -delete” which we find to be the fastest out of all the solutions for deleting a large number of files.

[root@srv6 clientmqueue]# find . -type f -delete

Now, there comes our second problem and our second question. We in fact do not use sendmail to send emails. sendmail is supposed to be replaced with something superior called Exim. And we trick the OS by deleting the sendmail binary and replacing it with a symbolic link (a shortcut) to Exim. Naturally, the Exim installation should have automatically removed sendmail and replaced it with a symbolic link. Turns out, this was not the case. Sadly, this meant that all this time we were never actually using Exim but using sendmail.

So after all of this reading, the solution of this email problem was to create a shortcut from sendmail to Exim… Anyone want to buy me a beer?

Our site is down due to an IP conflict. We have already changed our DNS to a new IP until the old IP gets restored.

The worse case ETA is 24 hours.

At the moment, we have changed our IP address to 119.42.146.34