Tristan Findley

A report on the recent server outages, and the recent server rebuilds.

Google Analytics results for the outage periods

November 26th, 2009

The site became inaccessible when the IP address was unknowingly duplicated.

Access from most external machines was still possible, however machines on the 134.219 IP Range experienced a loss of connectivity. This was due to the IP Address being reissued to another machine. Connectivity was affected for less than 24hours, and has now been resolved.

November 28th, 2009

During the weekend of November 28th, 2009, all connectivity was lost with the webserver.
The outage was discovered late on Saturday night when I attempted to access the server to carry out some website development. When the server did not respond, I assumed that another IP problem might have occurred. On Monday morning, I accessed the rack, and rebooted the firewall (which at this point had no evidence of damage, except that it had locked up), which restored services. About an hour later, connectivity was once again lost. Upon further inspection of the firewall, it was discovered that water had leaked into the rack from a fresh hole in the ceiling. The firewall, being on top of other servers, had unfortunately received the worst of the water, was destroyed. The hard disk was recoverable, however the motherboard was not.

A replacement firewall has since been installed, and steps taken toward protecting the rack from future leaks.  No damage to the webserver was identified, and all data has been recovered.

Server rebuild

In light of this recent firewall outage, I took the time to rebuild the server. The OS was reinstalled, and reconfigured, and all web-content restored. Services have now resumed, and full site migration should be complete by midnight tonight.

The rebuild has given me the opportunity to improve on certain aspects of the configuration, and to tighten up security on the server. As part of this, certain obsolete services has been removed.

• Teamspeak – Removed
• IceCast – Reinstalled but currently Offline
• Ventrillo – Removed

In addition to the webserver being rebuilt, the firewall has also been replaced with a newer solution. This should provide faster throughput, and an improved stability of the IDS / IPS service.

VOME Site Migration

As part of my contracting work with the VOME Group, I will be migrating the VOME webservices onto my server. Currently, all VOME web services are hosted inside of the ISG at Royal Holloway University. Unfortunately in recent months, the Virtual Infrastructure has been in increasing demand. The migration of the web services to my server will free up these resources for use in research.