In the CHIPS Oct-Dec 2005 edition, we reported that the Space and Naval Warfare Systems Center (SSC) New Orleans was closed due to damage caused by Hurricanes Katrina and Rita. But we were wrong because only the SSC New Orleans buildings were closed! By implementing the Continuity of Operations Plan (COOP), SSC New Orleans was operating at full speed. Relocated personnel were working at alternate work sites before, during and after Katrina passed to continue the vital work of the center.
Today, approximately 86 of the center's government employees are working in the SSC New Orleans facility, with the remainder working at Naval Air Station Joint Reserve Base (NAS JRB) Forth Worth, Texas (20), Naval Education and Training Command (NETC), NAS Pensacola, Fla., (90) and SSC New Orleans Millington, Tenn., detachment (29). The remainder of the SSC New Orleans staff and contractors are either telecommuting from around the greater New Orleans area or from the SSC New Orleans Washington, D.C., detachment.
CHIPS finally caught up with SSC New Orleans Commanding Officer, Capt. Fred Mingo, in late December (on the run, between flights on his cell phone) to ask him about New Orleans personnel and center operations. Capt. Mingo was recognized with a Department of the Navy Information Management/Information Technology (DON IM/IT) Excellence Award in January for his exemplary leadership and commitment to people and mission in his superb response to Hurricane Katrina.
Capt. Mingo: We have become a virtual organization. The rapid transition to a virtual organization has been one of our success stories. Right after the storm, the entire 504 and 985 area codes failed. Even though I was working from my COOP office at the SSC New Orleans Washington, D.C., detachment, I couldn't receive calls on my 504 area code cell phone. Most of our people were safely out of the storm area, but almost everybody had cell phones originating in the local area code so they couldn't receive calls. The complete failure of the phone system was one reason why it was so hard to find many of our people immediately after the storm.
CHIPS: One of the foremost questions on everybody's mind is the welfare of your personnel?
Capt. Mingo: We are very thankful that everybody made it through the storm safely with no loss of life or physical injury. Several have harrowing stories like swimming from their homes or getting out of their flooded homes at the last minute. Many, however, experienced tremendous personal loss, and they are now working with FEMA (Federal Emergency Management Agency), the Red Cross, their insurance companies and a host of other organizations to rebuild their lives.
Of the 225 civilian and military personnel that report directly to me, 94 have homes that are not habitable, 46 of which were totally destroyed; 76 have a home that sustained minor damage. Despite this loss, everybody has been touched by the generous outpouring of support they have received from their fellow shipmates, those at their temporary work locations, friends and strangers. Some are also saying that they are experiencing closer family ties as a result of this national disaster.
The Navy has really helped with Task Force Navy Family (TFNF). TFNF is providing a host of information and services from counseling to championing issues that require changing Navy, Defense Department or higher policy. All of our people are benefiting from Task Force Navy Family and many are using its services. Simply knowing that TFNF is available is a great relief because we know that the Navy cares.
CHIPS: Did you have an emergency plan prior to the hurricanes?
Capt. Mingo: Yes, we did, and it was executed as planned. Our plan has been iteratively developing for more than seven years, and it is closely coordinated with the Navy Reserve and the Navy Standard Integrated Personnel System (NSIPS) program office.
Today, we (SSC New Orleans, Commander, Navy Reserve Force and NSIPS) have a well-established team and have successfully executed our COOP several times, specifically last year for Hurricane Ivan and then earlier this year for Hurricane Dennis. Both times we successfully COOPed supported systems off-site while those systems without an off-site COOP requirement continued to operate from our SSC New Orleans server farm.
Unfortunately, the magnitude of the disaster resulted in our server farm dropping off the network before the generators ran out of fuel. As a result of this network failure, today we’re in a new phase of our COOP. Customer requirements drove and fund our COOP plan. Before Katrina many of our hosted systems didn’t require off-site COOP, but they became a high priority for restoration after the storm.
To restore these systems, we had to recover servers and then reconnect these systems to the network. We had people in our buildings about a week after the storm, but there was no power or air conditioning. We first flew in by helicopter, and then after the flooding subsided, we drove in, but it was a long and difficult trip, especially the first few weeks after the storm.
Our on-site recovery team started removing key servers and computers from the 100-degree buildings, hand-carrying everything, sometimes down five flights of stairs. There were more than 800 servers on our fifth floor operations deck and personal computers supporting more than 1,000 employees. The workforce included military, civilian and contractors at SSC New Orleans and our two other hosted program staffs: DIMHRS (Defense Integrated Military Human Resources System) and NSIPS. This was a huge, difficult recovery effort, considering the harsh working conditions, but our team was focused,d espite whatever personal losses they may have experienced.
During recovery, our leadership team was locating our people, ensuring all were safe and assisting them in any way that they could. We identified off-site work and billeting locations that effectively and efficiently support our mission. Our various departments communicated using BlackBerries, e-mail, the Internet and conference calls.
CHIPS: How long did it take you to get a full accounting of all personnel?
Capt. Mingo: Just over three weeks for Hurricane Katrina and about a week for Hurricane Rita.
CHIPS: Do you know when your building will be fit for occupancy?
Capt. Mingo: We were hoping we would be back in our facility by early spring, but it will probably be early summer. We are in leased facilities owned and operated by the University of New Orleans foundation. This arrangement means that coordinating restoration efforts are more complex.
CHIPS: Can the city’s infrastructure support a return to duty for SSC New Orleans personnel?
Capt. Mingo: It depends. We are located on the University of New Orleans campus on Lake Pontchartrain. While this area did not flood, a lot of the surrounding area did. Right now people working at the facility have to bring their daily provisions from home. Life has also changed for areas outside the flood zone. Everything is crowded and traffic is terrible, especially for those driving in and out of the city at rush hour. Schools are also a huge concern for those with children.
CHIPS: You and your leadership team have earned much praise for your superb COOP execution and commitment to personnel.
Capt. Mingo: We have a very tight leadership team, and that was one reason that our command could continue to function throughout this disaster. Several personnel on our team did lose everything in Katrina, and their personal loss has helped our understanding for the loss others in the command have experienced. Our team talks and meets regularly not only to discuss operational issues but personnel concerns too.
Our December leadership meeting was significant to the entire command because it was our first meeting back in New Orleans. In addition, we are hosting the SPAWAR Vice Commander, Rear Adm. (sel) Tim Flynn and Mr. Murray Rowe, the new Manpower, Personnel, Training and Education (MPT&E) chief information officer (CIO).
CHIPS: Can you talk about some of the lessons learned from these natural disasters?
Capt. Mingo: COOP is a command-wide effort requiring constant attention and support. You have to practice and stress COOP as a command priority. Paperwork drills won’t do it. SSC New Orleans is an IT service support command, so we have two components to our COOP, one for our command operations and another for our customers’ systems. For these systems we support, actually testing and going through drills that transfer real data, making sure that everything works and can sustain operations from the COOP environment is what it takes to have confidence in continuing operations.
One of our immediate lessons learned is that you need multi-channels for communications, especially for those members unable to access their NMCI e-mail accounts from home computers. Our contact lists quickly filled with multiple e-mail addresses and phone numbers as people moved from location to location.
At the command level, everybody on our leadership team had an NMCI BlackBerry with cell phone capability that was vital to command communications, especially during and immediately after the storm. BlackBerries worked very well for e-mail, text messaging and cell phones. EDS was very supportive throughout the storm. Because our e-mail accounts went over the 50 MB limit, I made a quick phone call to EDS and every team account was expanded to 250 MB within 30 minutes.
In addition to the efforts of our management team, we were greatly assisted by a team of Reservists at SPAWAR Headquarters who tracked many of our people down by persistence and creative associations between known contact information. As a command, we also employed the services and capabilities of the Navy’s Global Distance Support Group or call center. I’m a fan of this capability because it enabled our leadership team to focus on the command mission. The call center staff would not only receive calls and e-mails from our personnel, but they would contact our personnel, if required.
Often the call center staff would call me to say they had spoken with an individual they thought required additional support or to say somebody from our command had asked me to call them. The lesson learned is that our personnel need to know this call center number and its capability before an emergency occurs. It should be standard across the Navy, like a master 911 number for anybody in the Navy. Consolidating this capability is one feature we’re developing for Sea Warrior, but this is a significant effort that will take time to establish.
Naval message traffic was not available and didn’t reach our command for months. Frequently, I would receive a call or e-mail notification referencing tasking that was outlined in a message or on a Web site, but we were not receiving this important information. Consequently, we relied on verbal communications from our headquarters during daily conference calls. Immediately after the storm, many could not access the Internet, so we couldn’t rely on Web sites for information.
Most of our non-COOPed systems requiring post-Katrina restoration supported manpower and personnel. We had to work closely with Navy Personnel Command to generate a customer driven prioritization list that was then signed out at the flag level. More importantly, we included supporting IT infrastructure on this list, to ensure the customer understood and supported the IT infrastructure component of the recovery effort.
At the Navy enterprise level, we need to provide enterprisewide tools and capabilities for commands to easily track and communicate with their entire workforce: military, civilian and contractor. Commands also need to understand there are differences between military and civilian orders, entitlements, benefits and support processes. We often speak of a total force, but there are significant differences, especially regarding safe haven orders. Depending on how these policies are implemented could adversely impact morale.
Today, we’re already incorporating lessons learned from Katrina, but we still have a long way to go because a command COOP must support emergencies well beyond hurricanes. Other natural disasters or terrorist actions don’t provide advance lead time. This is only a start of our lessons learned, which continues to grow.
CHIPS: Are some of your customers changing their systems requirements due to lessons learned from the damage to operations?
Capt. Mingo: Absolutely! Going into this storm, we had a high degree of confidence with our command COOP processes, team and capabilities, but we also knew our customers didn’t have off-site COOP requirements for their systems. COOP is critical when you have the centralized or Web-based systems that support mission critical operations and manpower and personnel systems fall into this category. These systems require failover capability or flexibility to adjust mission requirements. When we’re talking about manpower and personnel, the last thing we want to do is have a system failure that adversely impacts Sailor pay, administrative support, promotion status …
It costs more to deliver this failover capability, but it’s a reality of our mission support today. When we developed the New Order Writing System requirements, (now referred to as the Navy Reserve Order Writing System – NROWS), it included COOP because we needed to ensure Reservists could receive orders reliably, especially in time of crisis. NROWS serves as one of the pillars of the successful COOP we have in place today.
Post Katrina, restoring the Job Advertising Selection System (JASS), for example, was BUPER’s number one priority. JASS advertises job assignments on the Web. It is part of the Sea Warrior capabilities, but the customer did not fund an off-site COOP capability.
As soon as JASS went offline, BUPERS had to double detailer shifts, so it quickly became the top priority for system recovery. From the IT infrastructure perspective, restoring JASS required replicating the SSC New Orleans network and boundary layer 2 (B2) capabilities at NAS JRB Fort Worth to host JASS on the network in a DMZ. (Demilitarized zone refers to a network area that sits between an organization’s internal network and an external network, usually the Internet.)
After Katrina, the MPT&E community realized that existing COOP requirements had to be expanded. Moreover, because we successfully executed our COOP and everything went according to plan, there is leadership confidence in our capabilities. Consequently, our task today is to ensure that we have the right COOP requirements integrated into the supported applications via the Program Objective Memorandum process.
CHIPS: The safety of the staff and continuity of operations are the most important issues in any emergency.
Capt. Mingo: Absolutely! Katrina highlighted that we still have a long way to go with regard to disaster preparedness. There has been and still is a lot of organizational change in the Navy resulting in matrix organizations that blur the lines of traditional command and control.
For example, because SSC New Orleans provides matrix support personnel to both NSIPS and DIMHRS, we had to clearly agree who was going to report and maintain contact data on each individual member. Sounds easy, but there isn’t a system or tools to easily capture and then maintain this data over time, especially for commands with joint personnel.
CHIPS: SSC New Orleans personnel are truly heroes to have met the challenge of meeting the command’s mission under impossible circumstances.
Capt. Mingo: Our people are simply tremendous and leading and working with them is a very rewarding and satisfying experience. Then, watching our recovery take shape, day by day has really been uplifting. There is never a dull day.
In addition to recovery efforts, we are also preparing for a major organizational change for our command. Effective Oct. 1, 2006, SSC New Orleans will transfer from the SPAWAR claimancy to merge with the newly formed Manpower, Personnel, Training and Education (MPT&E) Enterprise. We first started dealing with this command transition in February 2005, and the specifics are continuing to develop as the MPT&E Enterprise is transitioning to an IT shared services construct.
Essentially, SSC New Orleans will disestablish to form the nucleus of a new organization that will become the MPT&E Central Design Agency. While the details of this transition are still being worked, those within the MPT&E information technology domain have been asked to begin supporting this new MPT&E IT shared services construct as a virtual organization under the MPT&E CIO, Mr. Murray Rowe.
As a result of this and in addition to my responsibilities as SSC New Orleans commanding officer under SPAWAR, I have also been acting as lead for the MPT&E IT operations and infrastructure division. In fact, many of our SSC New Orleans employees started working within this MPT&E virtual organization as well. Sounds confusing and it was, especially as we began our post-Katrina coordination and recovery, but there are benefits to this alignment and closer integration with our primary customer is one.
CHIPS: Will your entire staff be in the new organization and all the systems that New Orleans currently supports?
Capt. Mingo: That’s the plan, but we’re still working the details.
CHIPS: What do you want readers to remember about SSC New Orleans operations and personnel?
Capt. Mingo: The devastation around New Orleans and the Gulf Coast is overwhelming. One of our program managers captured the magnitude best when he explained that not only did he lose his home and family possessions, but he lost his childhood memories, a remark that I have heard others make several times. In this case, he, his wife, and their families were born and raised in Chalmette; now everything they had is gone. Fortunately, they found a new home, and he’s working from Millington as his wife and family are settling on the north shore of Lake Pontchartrain, but their lives have forever changed.
Knowing the extra effort our people have gone through to make sure that the mission is successful — despite their personal losses is amazing — it’s a tribute to their strength, resiliency and positive attitude. I have met many that have lost everything, but they’re coming back to pick up the pieces and rebuild. It’s like running a marathon and, at this point, we’re just beginning. But our motto is ’rebuilding … one day at a time.’
I have spent over 14 of my last 16 years stationed in New Orleans so hurricanes and evacuations are nothing new. Even before Katrina, this tour had become my most rewarding due to the quality, professionalism, humor and commitment of our people. Katrina brought out the best our team has to offer. It is an honor and privilege to have this opportunity to have served with and been a part of this inspirational team.
SSC New Orleans provides customer support 24 hours a day, 365 days a year for nine systems:
• Navy Standard Integrated Personnel System (NSIPS)
• Navy Reserve Pay Helpdesk
• Navy Reserve Order Writing System (NROWS)
• Job Advertising and Selection System (JASS)
• Medical Readiness Reporting System (MRRS)
• Inactive Manpower and Personnel Management Information System (IMAPMIS)
• Reserve Headquarters Support (RHS)
• Reserve Standard Training Administration and Readiness Support (RSTARS) (HP)
• Local base operations; electronic data warehousing and Corporate Data Maintenance (CDM), formerly known as the Personnel Pay Assistance Center (PPAC)