Foreword by Admiral Edmund P. Giambastiani, Jr. 15
Foreword by Dr. William Nylin, Jr. 17
Preface 19
Acknowledgments 23
About the Authors 24
Part One What Is Automated Software Testing and Why Shoul We Automate? 1
1 What Is Effective Automated Software Testing (AST)? 3
1.1 Automated Software Testing Definition 4
1.2 Automated Software Testing Recipes 5
1.3 Advances in AST Technologies 8
1.4 Automating Various Software Testing Types 11
1.5 Providing AST-Based Production Support 16
1.6 Automating Standards Assessments 18
2 Why Automate? 23
2.1 The Challenges of Testing Software Today 24
2.2 Reducing the Time and Cost of Software Testing 26
2.3 Impacting Software Quality 38
2.4 Improvements to Your Software Test Program 42
3 The Business Case
3.1 Definition of the Business Case 51
3.2 Identifying the Business Needs 53
3.3 Justifying Automation in Terms of Cost and Benefits 55
3.4 Risks 65
3.5 Other Considerations 67
4 Why Automated Software Testing Fails and Pitfalls to Avoid 69
4.1 R&D Does Not Generally Focus on Automated or Manual Testing Efforts 71
4.2 AST Myths and Realities 74
4.3 Lack of Software Development Consideration for AST 83
4.4 The Forest for the Trees-Not Knowing Which Tool to Pick 91
4.5 Lack of Automaton Standards across Tool Vendors 94
4.6 Lack of Business Case 97
Part Two How to Automate: Top Six Keys for Automation Payoff 99
5 Key 1: Know Your Requirements 101
5.1 Understand the AST Support Requirements 102
5.2 Additional Information in Support of AST Requirements 114
5.3 When Information Is Not Available 116
5.4 Start Implementing Your Requirements Traceability Matrix(RTM) 124
6 Key 2: Develop the Automated Test Strategy 129
6.1 The AST Strategy Document 131
6.2 Scope and Automated Test Objectives 132
6.3 Identify the Approach 139
6.4 Automated Software Test Framework(ASTF) 146
6.5 AST Environment/Configuration 150
6.6 Automating the RTM 159
6.7 Automated Defect Tracking 164
7 Key 3: Test the Automated Software Test Framework (ASTF) 167
7.1 Verify That the ASTF Meets Specified Requirements and That Features Behave As Expected 169
7.2 Peer-Review All ASTF-Related Artifacts, Including Design,Development, and Test Cases 170
7.3 Verify Requirements and Coverage 182
7.4 Hold a Customer Review 183
8 Key 4: Continuously Track Progress--and Adjust Accordingly 187
8.1 AST Program Tracking and Defect Prevention 188
8.2 AST Metrics 192
8.3 Root Cause Analysis 205
9 Key 5: Implement AST Processes 209
9.1 AST Phases and Milestones 211
9.2 AST Phase 1: Requirements Gathering-Analyze Automated testing Needs 212
9.3 AST Phase 2: Test Case Design and Development 215
9.4 AST Phase 3: Automated Software Testing Framework (ASTF) Test Script Development 216
9.5 AST Phase 4: Automated Test Execution and Results Reporting 217
9.6 AST Phase 5: Program Review and Assessment 218
9.7 Virtual Quality Gates 219
9.8 Process Measurement 220
10. Key 6: Put the Right People on the Project-Know the Skill Sets Required 223
10.1 Program Management 232
10.2 Systems Engineering 236
10.3 Software Development 239
10.4 Configuration Management 242
10.5 Quality Assurance 244
10.6 Subject Matter Experts(SMEs) 246
Appendices
A Process Checklist 251
A.1 AST Phase 1: Requirements Gathering-Analyze Automated testing Needs 252
A.2 AST Phase 2: Test Case Design and Development 253
A.3 AST Phase 3: Automated Software Testing Framework (ASTF) and Test Script Development 254
A.4 AST Phase 4: Automated Test Execution and Results Reporting 255
A.5 AST Phase 5: Program Review and Assessment 256
B. AST Applied to Various Testing Types 257
B.1 Security Testing 257
B.2 Soak Testing 261
B.3 Concurrency Testing 263
B.4 Performance Testing 265
B.5 Code Coverage Testing 266
B.6 Unit Testing 271
C The Right Tool for the Job 275
C.1 Requirements Management (RM) 276
C.2 Unit Testing Frameworks--Example Evaluation 278
C.3 Configuration Management--Example Evaluation 284
C.4 Defect Tracking--Example Evaluation 292
C.5 Security Testing 299
C.6 Automated Software Testing Framework (ASTF)-Example Evaluation 306
C.7 Other STL Support Tools 316
D Case Study: An Automated Software Testing Framework(ASTF) Example 321
D.1 Key Design Features 323
D.2 Test Manager 325
D.3 More on Automated Test Case and Test Code Generation 326
D.4 Results Reporting 328
D.5 Automated Defect Reporting 328
Contributing Authors 331
Index 333