We are going to be down for another 2 more hours instead of the protected 6. This means we will be down for a total for 8 hours instead.

I have just spent the last hour editing the configure script because PHP fails to find libraries as specified. On Warez-BB’s servers we install all of our things under a prefix /usr/server. This allows us to separate the junk RPMs that CentOS comes with and the things we custom compile. This is our workaround for preventing our compiled and installed apps to overwrite what comes originally with the CentOS system.

The first problem arises when we must reconfigure ld.config every time we reinstall to include these libraries. This is fair enough considering we set up our own custom prefix. However, CentOS is configured by default to compile and install libraries into the folder /usr/lib – even in a 64bit version of CentOS. For those who do not know, /usr/lib is for 32bit libraries, and /usr/bin64 is for 64bit libraries. Since we’re on a 64bit server, we should be accustomed to running on 64bit libraries instead. But by default, CentOS queries /usr/lib for libraries (regardless of 64bit or 32bit) when we compile. And if it doesn’t find them, it throws out errors, even if it’s found in /usr/lib64.

Just a FYI for those who are confused because we’re talking about 32bit binaries in a 64bit system. Linux can run both 32bit and 64bit binaries at the same time – unlike Windows, which in my opinion, is an epic fail when it comes to the whole 32bit/64bit architecture stuff.

We must force the “configure” script (a configuration script that we must run when we compile something to tell it what to include and exclude) to get libraries from lib64 instead. This is fine until more errors are thrown out because some of required dependencies are installed in /usr/lib instead. This is because the configure script can only look for libraries at lib or lib64 at once. Though, this saga is partially due to my stupidity how 64bit OSes work, we cannot forgive CentOS for installing 64bit libraries by default to a folder destined for 32bit libraries to begin with.

I spent the past hour running around in circles chasing each dependency that failed and editing the configure script to force detect it through – this is, of course, a bitch to do.

The solution? To get the CentOS devs to include both /usr/lib and /usr/lib64 into the configure query. But prioritize the preferred library to /usr/lib64 if is duplication of 32bit and 64bit libraries. Though I am unsure if 32bit libraries can actually link to 64bit binaries, this would have helped me. I guess ultimately, what CentOS should do is dynamically set the library install path (whether /usr/lib or /usr/lib64) when we compile and install programs, they will always be in the correct folders.

An unexpected scripting error on our part caused portions of one of our Administrators topics/posts to be removed. No other topics and posts were removed due to this error and it is an isolated event.

However, because we need to restore back the content. We will be taking 6 hours of downtime starting from now.

Though in the midst of such bad news, we have taken this downtime opportunity to run maintenance that would’ve required 6 hours of downtime anyway. These include:

• Setting up our new img6 server and begin transferring everything that was originally in img5 over to img6.
• Configuring a new and updated kernel that does not crash when it boots up. This will allow us to put our SSD into work since the current kernel does not support EXT4.
• Fix the search bug that caused Firefox to expel “Content encoding error” errors.
• SSL certificate set up for Lilium. This will allow us to push out an update of the latest Lilium version.

Our servers are fired up and running.

We’re shutting down one of our servers for some hardware maintenance.

ETA 40 minutes.

We had experienced a server issue with our image server. The server was rebooted within an hour of the problem occurring and things are up and running once again.

We are also continuing with pruning our Graveyard.

We’re currently pruning the Graveyard.

As per usual, report counts will go down, however, post counts will not be decreased. We do not use the report count in anyway to determine MOTM, so there is nothing to worry about with the loss of stats.

The site will be taken off line within the next few minutes to perform this upgrade.

Just realized that our search was down and doesn’t automatically start up when we reboot. Thanks to all who called this bug in. It has now been fixed.

Kernel upgrade was an epic fail. We’ll be looking into the error message we got before we commit anymore downtime.