Monday, October 31, 2011

Mission Operations - PCVMR Cycle

Reminiscing About the Past

Leading off the previous post, let's delve deeper into the processes that helped provide mission assurance to the crew taking the boat down to operational depth. We spoke of submarines and the mature operational approach that allowed a crew barely out of high school, most with no formal education, to not only function in these demanding environments, but excel and push themselves and their equipment to the extremes. 

Why were we successful? 
It was more than top-notch training. It was more than engineering and equipment superiority. It included a deep knowledge of operational processes that work in orchestration with the equipment and a firm understanding of the mission objectives and risks. The effectiveness of everything was measured and fed back into the processes and equipment.

Provision | Configure | Validate | Monitor | Respond
The PCVMR process cycle provides insight into how we were able to attest to the assurance of our boat to keep us safe and deliver on her mission. Here's how it works.

Provision: Equip yourself with the right systems for your mission. Submarines are equipped with systems appropriate for accomplishing their mission. Ballistic missile and attack submarines have very different missions and very different equipment... and crew and training. The highly specialized Submarine NR-1 was outfitted with equipment and capabilities not found in other subs because that's what her missions required. 

Configure: We sometimes laughed at a few of the Standard Operating Procedures (SOPs), but we respected them. Some would say, "Rules are written in blood." That's because somebody paid a heavy price for that stupid rule to check that breaker or valve lineup twice. Every system had a checklist for every operational lineup. These lineups are thoroughly tested by smart engineers, and every effort is made to follow the book. It's one thing if you're throwing your leftover litter into a McDonalds wastebasket. It's another to dump or pump it overboard underwater. One is casual. The other is very carefully handled. 

Validate: Everything was checked twice before getting underway. Every critical system was reviewed. Every change. Anyone that's spent time underway will recall the repeat-backs required on the phones as you read from a procedure to senior operator. The senior operators then repeated the same requests to watch officers for final permission. Everyone backed each other up to validate actions. Important actions were verified formally by a second person and signed off by all parties involved. Some critical actions required multiple validations and checks based on the affect of the system to the ship's mission. Once everything is known, you have entered an operational steady state, or a known state of operations. 

Monitor: Despite the best intentions to engineer flawless equipment and set everything up correctly, things go wrong. Systems are heavily monitored, automatically and manually, many both, to identify state deviations, or changes in the known state of operations. These may be intentional by the crew and known. These may be intentionally malicious from an external source, or changes could exist because of an inexperienced operator. The monitoring systems (some of which are redundant) help identify the early state of changes to give operators the most time to respond appropriately. Monitoring occurs across many complex related systems, and you need to identify issues as quickly as possible to minimize their impact. 

Respond:  It is the operator's experience and well-rehearsed drills that helps lead the best response. Realistic drills are part of every day life underway in preparation for when something bad happens. You expect something bad to happen. And it does. It's the workflow, methodical analysis, and rapid response that make the difference between "that was close!" and a new SOP. Rules are written in blood. Responses to incidents are debriefed for details that could could have managed the incident better than what was done. After Action Reviews. Post Incident Reviews. The outcomes of these meetings completes the PCVMR Cycle as they affect the Provisioning, Configuring, Validating, Monitoring, and Responding. 

Can you see how this translates to cloud security and audit? We'll dig into that next. It's time to walk out of the bubble and back into the cloud:).