Log for week 35, 2007

Operator on Duty: Leif Nixon

Thursday

08:59, Nixon: Was informed by Olli that 3D still is not fully running - it is excluded from replication and will need to be manually brought back in sync, but key CERN personnel is on vacation until September 10. EGEE broadcast sent.

08:33, Nixon: RLS is flapping, according to Nagios. Short outages at 05:00, 06:00 and 07:00. Looks like the watchdog script restarted the service sometime after 07:00, and it seems to have behaved since.

Tuesday

15:51, Nixon: Eventful day, this. I was away until lunch, and during that time both UiO and Örestaden lost power briefly. Everything seems to be back up and running now. There are also (unrelated) network problems at FZK - certain IP addresses can't contact the FZK gridftp doors. Might possibly be a broken load-balancing scheme at their end. FZK has been contacted.

Monday

14:42, Nixon: Mattias Ellert reports RLS seems to have recovered spontaneously.
14:00, Nixon: Alex reports that RLS is hanging. Nagios doesn't notice anything wrong.
09:44, Nixon: 3D machine responding to ssh again, after a lengthy downtime since an unplanned power outage last week.
09:30, Nixon: OoD handover, operation log rotated.