This article is part 3 in the Non-Negotiable Elements of a Secure Software Development Process series and focuses on defining validation criteria. In part 1 of the series, we discussed how security requirements set clear and reasonable expectations that development teams can plan for and meet to satisfy a specific level of security assurance. Part 2 discussed how to use secure architecture, configuration, and coding patterns to satisfy security requirements and reduce the ongoing cost of developing secure code.All three articles are listed below:
Part 1: Security Requirements
Part 2: Secure Architecture, Configuration, and Coding Patterns
Part 3: Validation Criteria
Why Validation Criteria?
Many organizations measure how secure their applications and infrastructure are using assessments. They might use application security assessments, penetration tests, design reviews, threat models, or many other fault finding activities. These can be good risk indicators, and they are often important activities to include within an application security program, but they fall short in actually telling us whether our application is secure. They focus on finding problems, telling us when an asset is insecure, and remain silent about everything else.
With the declaration of security requirements and secure architecture, configuration, and coding patterns (secure patterns), we now have a list of positive characteristics about the application that we would like to affirm. If we can validate these elements, then we can get a more comprehensive understand regarding how our application is secured. We can then focus on using assessments to identify missing security requirements and improve our overall process as a whole instead of a single application.
What to Validate?
If we try to validate the code in a vacuum, we will fall back into the same old process:
- Use a code review tool
- Find problems
- Fix them
Validation Criteria Approaches
Validation steps can focus on testing code, application behavior, or both. The effectiveness of testing code versus application behavior will depend on the security requirement or secure pattern that is being tested. For example, if one were to verify the code/configuration of an application to determine whether it used SSL/TLS, they would fail to test whether the content delivery network (CDN) required SSL/TLS. In this case, testing the application behavior of each infrastructure element would be more effective than testing the code or configuration. On the other hand, validating that an application used only prepared statements would be more effective in code. The final series of slides at the end of the presentation demonstrates this further.
Additionally, validation criteria can affirm controls are in place (positive testing) or find deviations from the defined secure pattern (negative testing). Positive testing should be the primary approach used for all validation criteria. Negative testing should simply be used to provide additional confidence in test results.
Test Case Techniques
Teams can define test cases within their validation criteria in a number of ways. A few examples are listed below. Each is demonstrated in the presentation slides.
- Manual test cases
- Step by step instructions
- Automated test cases
- Grep or Findstr
- Exploratory Testing
- Security testing tools
- Free/Open Source
- Commercial (Not Recommended!)
I don’t recommend starting with commercial automated security testing tools like a static code analyzers or dynamic application testing platform. Many times teams spend a lot of time and money to get it set up and are disappointed with the results. Before investing in one of these tools, it's important to have clear goals, realistic cost and personnel expectations, and a strategy for how the tool will fit in with a larger application security program. If the team is moving towards using a commercial security tool, consider starting with free tools first like FxCop for .NET applications and write your own custom rules.
The key advantage to writing your own manual or automated test cases is that you can customize them for the application and its business context. This isn't something a commercial, general purpose security scanning/code review tool is designed to do.
Once a team has a fairly comprehensive set of security requirements; is effective at implementing their security patterns; and desires to grow further, exploratory testing could become valuable. Exploratory testing should be used to identify vulnerabilities that do not yet have a security requirement already defined. This testing approach is used to improve the overall secure software development process, expand the list of requirements, and improve all the organization’s applications - not to focus on the faults of one single application.
How to Win at Software Security
If all of these components are put in place:
- Security and Compliance Requirements
- Secure Architecture, Configuration, and Coding Patterns that Satisfy the Requirements
- Security Test Cases and Validation Criteria that Affirm Patterns are Implemented