Google Analytics results for the outage periods

November 26th, 2009

The site became inaccessible when the IP address was unknowingly duplicated.

Access from most external machines was still possible, however machines on the 134.219 IP Range experienced a loss of connectivity. This was due to the IP Address being reissued to another machine. Connectivity was affected for less than 24hours, and has now been resolved.

November 28th, 2009

During the weekend of November 28th, 2009, all connectivity was lost with the webserver.
The outage was discovered late on Saturday night when I attempted to access the server to carry out some website development. When the server did not respond, I assumed that another IP problem might have occurred. On Monday morning, I accessed the rack, and rebooted the firewall (which at this point had no evidence of damage, except that it had locked up), which restored services. About an hour later, connectivity was once again lost. Upon further inspection of the firewall, it was discovered that water had leaked into the rack from a fresh hole in the ceiling. The firewall, being on top of other servers, had unfortunately received the worst of the water, was destroyed. The hard disk was recoverable, however the motherboard was not.

A replacement firewall has since been installed, and steps taken toward protecting the rack from future leaks.  No damage to the webserver was identified, and all data has been recovered.

Server rebuild

In light of this recent firewall outage, I took the time to rebuild the server. The OS was reinstalled, and reconfigured, and all web-content restored. Services have now resumed, and full site migration should be complete by midnight tonight.

The rebuild has given me the opportunity to improve on certain aspects of the configuration, and to tighten up security on the server. As part of this, certain obsolete services has been removed.

• Teamspeak – Removed
• IceCast – Reinstalled but currently Offline
• Ventrillo – Removed

In addition to the webserver being rebuilt, the firewall has also been replaced with a newer solution. This should provide faster throughput, and an improved stability of the IDS / IPS service.

VOME Site Migration

As part of my contracting work with the VOME Group, I will be migrating the VOME webservices onto my server. Currently, all VOME web services are hosted inside of the ISG at Royal Holloway University. Unfortunately in recent months, the Virtual Infrastructure has been in increasing demand. The migration of the web services to my server will free up these resources for use in research.

The server power-down worked flawlessly, and the server backups were not needed. I will start scheduling regular downtime for server and OS maintenance, which will help keep the OS Kernel up-to-date, as well as allow for hardware upgrades.

This downtime will mark the loss of 390 days of uptime. Equinox (the hosting server) has been running non-stop ever since it was reinstalled in mid 2008.

Unlike previous blogs, I don’t plan on this one being a report on my day-to-day activities. I’ll publish website updates, major events in my life, and professional content.

——

Over the last week, we’ve been running the KPMG course for the students. This has involved setting up an entirely new streamlined lab environment for the students use. After a few small technical issues with the VM deployment system on the first day, I devised a new way of pre-loading the VM’s to the machines. We plan to use this method to deploy the Foundstone lab next week (Thursday / Friday). I will need to secure the OS further for use with new lab sessions. I also plan on offering this system to the staff, so they can create VM appliances for student use, then they are able to run fully customized lab sessions.

I finished my first VOME contact, which included setup and deployment of three desktops and three laptops to the new VOME staff, and also the setup of their Moodle E-Learning space. The next piece of work I do for them will hopefully be website design. I plan to use WordPress for this due to its simplicity and multi-user capability. It will be perfectly suited to their needs.