|
Successful development and deployment of mission-critical systems require rigorous testing before deployment,
and thorough auditing and monitoring of ongoing operations after deployment.
This document outlines in broad strokes the steps and requirements that CPrompt employs for mission-critical systems.
For all mission-critical projects, we employ three separate environments or stages. Each environment comprises an entire functional system of both hardware and software which is completely isolated from all other environments, which are:
- Development
- Testing
- Production
Each functional module is separately tested in development, then integrated into a complete pre-production test system and tested again in the test environment. After deployment, we test the production systems a third time.
If programming changes affect core subsystems in any direct way, we will perform full regression tests of the entire system in all three environments (development, test, and production) as it advances during deployment.
For testing, we employee various strategies depending upon the nature of the system under test. These strategies are performed differently in each environment as follows:
- In the development environment, testing is performed by the developer or other technical personnel.
- In the test environment, testing is performed by non-developers; outside beta testers are often used for the more critical jobs. Since the test environment can be opened to Internet access, the client can also use their own testers, if required.
- In the production environment, testing is performed by developers and non-developers. Often, focus groups are employed to evaluate or test system efficacy for recommendations on future tuning of the system.
Hardware and software systems are tested under various load conditions to identify race conditions as well as choke points and certification of capacity. These loads will be simulated for development and test environments, and real-world in the production environment. We often record real-world load conditions and then simulate those conditions in both the development and test environments.
All mission-critical systems that we develop include integrated auditing of all vital operations. Our systems in all environments, and under all load conditions, always run full auditing. This may require additional hardware capacity, but it is the only way to detect and resolve some classes of intermittent run-time errors. This type of auditing is also the most reliable way to predict and mitigate problems before they occur. This is a necessary requirement for the attainment of the high levels of quality assurance needed in mission-critical systems.
All mission-critical systems are monitored by physically separate computer systems and / or personnel. These monitoring systems inspect audit records, log files, run-time operating system state, and environmental conditions such as network traffic, temperature, etc. These monitoring systems are also often part of the fail-over systems which automatically switch in redundant capability as needed.
All deployment of mission-critical systems includes roll-back strategies. No matter how small the change, a full roll-back option will be available should a problem occur in the production environment.
|
