Currently being updated.
SNIC operators, please keep in mind that this info is not yet final.
NEW: The Operator on Duty (OoD) alternates between the core NDGF operations team
and the SNIC operations team. The Operator on Duty participates in the WLCG weekly
operation meetings following their shift. The shifts start on Monday mornings, and are
on an On Call basis over the weekend.
NDGF OoD duty times are, in general, but not limited to, 08:00 - 16:00 local time, or some
reasonable DAY shift hours when general operations are underway. This means that doing a
16:00 to 24:00 shift is not an option. If you can not be available for a shift, please swap with
a colleague in your roster, or another OoD operator IN ADVANCE, (if possible.)
Note that NUNOC will be handling after hours operations at a monitoring level.See here for more info.
They will do monitoring between 16:00 and 08:00 CET. When this is fully operational, there
will also be a handover procedure.
Quick start up guide: Duties
The OoD needs to keep an eye on the health of the NDGF systems. This includes:
- monitoring and reacting to NAGIOS alerts.
- checking the state of dCache Pools, FTS, and GANGLIA.
- communicating with site admins when problems arise.
- keeping abreast of incoming requests from admins etc. on support@ndgf.org.
- issuing downtime where needed.
- creating appropriate tickets in the NUNOC and GGUS ticketing systems.
- attending weekly NDGF meetings and the WLCG meeting after their shift.
- filling in the weekly RC Production report (for NDGF).
- passing on the week's status during handover.
Duties in detail.
Daily duties
- Keep an eye on the most important monitors
- Check the support@ndgf.org mailing list in a regular fashion
- Its advisable to fill in the RC production report daily. Less work at the end of the week!
- Be active (or at least logged on!) to the NDGF jabber session.
- Be alert to scheduled maintenance and issue downtimes accordingly.
- (more to come here)
Weekly duties
Monday:
- Participate in the Monday morning handover. There is no set time, but before
noon is preferable. Ensure that you are available if you are taking the current
week's shift.
- Change Nagios SMS notification to go to current OoD. The procedure for this is found here.
- Participate in the WLCG weekly operation meeting.
Friday:
- Submit Production Report Reminder
- Add NUNOC tickets to the NDGF Weekly status wiki page before the meeting.
NDGF weekly meeting starts at 10:00 CET, and is via NDGF's Jabber chat room.
Operations is the first item on the agenda, and you need to be available for
that in case of questions.
OoD Prerequisites
- OoD should have an account on
- {dcache,ftp1,srm,pnfs}.ndgf.org
- For filling in the RC Production Report on Friday, check access for CIC Operations Portal
- (fill in more systems/urls here)
Operator on Duty handover
- Handover
- Old OoD - send an e-mail to support@ndgf.org and announce that you are handing over to the new OoD (and state who that is). Briefly mention any outstanding issues.
- New OoD - acknowledge handover to support@ndgf.org
- New OoD - check the ticketing system for any outstanding issues
- Weekly Operation meeting
- Old OoD and ideally also new OoD participates in the WLCG meeting (16:00, CET). Its advisable to use their call back feature, as this is the way that the participant is identified. Please make sure you identify yourself as Name/NDGF as you will be representing NDGF in this meeting.
Incidents
A detailed list is here .
Contact point
All emails from the OoD concerning operations should be posted through
support@ndgf.org
Site admins will contact NDGF through this address for support also.
Calendar
The OoD calendar is in a google calendar database. You should be able to see and make changes to
that calendar, for instance, if you decide to swap a shift with someone. SNIC people who are rotating on
their duty schedule might also like to add in who (specifically) will be taking that week's shift.
Operation-Operator on Duty
