TO:   All GCN Members
RE:   Recent outages of GCN service
DT:   11 Nov 99

Some of you may be aware of a series of outages in service from the GCN system
within the last week.  This note lists the specific intervals of service loss
and the reasons.  The first three were all related to a problem
with the network within the GCN building, and the fourth was an unrelated
GCN program crash (which because of all the prior service interuptions
went on for a long time because the watchdog program had been terminated
by the prior network failures).


The approximate times of the outages of the last few days are:

1)  Friday, 05 Nov ~17:00 to 06 Nov 13:00 EST local (20 hrs):
The Goddard network service/support people took GCN off the network
because of a perceived problem with one of the GCN computers.  (This perception
later turned out to be in error.)
After failing to get responses from the network service people, on Saturday
I "patched in" a work-around network connection to get back on the air.

2)  Monday, 08 Nov 10:00-16:30 local (6.5 hrs):
The Goddard network service/support people found the "patch" and again
unilaterally removed the connection.  After some meetings with them
where my disfavor on their operational procedures were communicated
(and they agreed that it was improper procedure), sevice was restored.
And during Monday they identified and fixed the fundamental problem
(which was totally unrelated to GCN).

3)  Tuesday, 09 Nov 11:15-16:15 local (5 hrs):
The network router between the network for the building in which GCN resides
and the rest of the world was down.  It had corrupted lookup tables. 
There is a possibility that this problem was fallout from the Friday & Monday
problems.

4)  Wednesday, 10 Nov 04:30-10:00 local (5.5 hrs):
Because the 'watch' program was down, we weren't paged about the outage and
we didn't notice the GCN program crash -- the "watch" program was terminated
when one of the network outages occurred.  We are now more conscientious
about monitoring the monitor programs given this recent spate of problems
and this last lapse in service.


There was definitely a backlog of outgoing e-mail GCN Notices on Tuesday
while the router was corrupted, and there may have been been a similar delay
in distribution on Monday.  This backed-up outgoing mail queues explains
the delays in e-mail delivery and the eventual "pulse" of email notices
you received.

My apologies for these recent losses of service.  Steps are being taken
which will reduce this probablility greatly in the future (after Dec 99).


Sincerely,
Scott


Scott Barthelmy                      NASA-GSFC, Code 661, Greenbelt, MD   20771
PHONE:  301-286-3106                 (work)
FAX:    301-286-1684                 (1st choice,  -1682 2nd choice)
EMAIL:  scott@lheamail.gsfc.nasa.gov
PAGER:  1-800-SKY-PAGE, PIN 2618712  (by phone, voice menu/instructions)
PAGER:  2618712@skymail.com          (by email, 240 characters max per message)
WEB:    http://gcn.gsfc.nasa.gov/gcn