ADVERSARIAL VALIDATION PROCEDURE

Pen Test · Red Team · Purple Team · Cloud Attack Sim · App Test · OT Safety · Finding-to-Closure

Table of Contents

1. Purpose and Scope
2. Engagement Types
3. Annual Cadence and Scoping
4. Rules of Engagement (RoE) Template
5. Red-Team Engagement Charter
6. Purple-Team Detection Validation
7. Cloud Attack Simulation
8. Application Penetration Test Checklist
9. OT Adversarial Assessment, Safety Protocol
10. Finding Rating, Routing, and Retest
11. Systemic Analysis and Program Feedback Loop
12. Roles and Responsibilities
13. Evidence Retention
14. Regulatory and Framework Alignment Summary
15. Document Control

1. Purpose and Scope

The Cybersecurity Policy requires periodic adversarial testing. The exposure management procedure names the adversarial activities CERG performs. The risk taxonomy distinguishes adversarial testing from vulnerability scanning. Until this procedure, the operating model, scope, rules, safety, finding routing, evidence, for adversarial validation was implicit.

This procedure makes it explicit. It applies to every CERG-led or CERG-overseen adversarial engagement: external pen test, internal pen test, cloud red-team / attack simulation, web/mobile/API pen test, OT adversarial assessment, purple-team detection validation, and social-engineering work conducted with the adjacent Awareness function.

In CERG, penetration testing is one engagement type within the broader adversarial validation capability. It does not replace red-team operations, purple-team detection validation, cloud attack simulation, OT-safe assessment, or other authorized testing where those activities are in scope.

Adversarial Validation Is the Antidote to Drift

A control library that exists on paper but has never been tested is a hope. Adversarial validation tests the assumption that the control library implements actual defense. CERG runs it on a cadence; documents the rules; routes findings into exposure management or the risk register; and uses purple-team work to upgrade detection, not just to score the team.

2. Engagement Types

2.1 External Tester Management

When adversarial validation is conducted by an external firm rather than internal staff, the following requirements apply.

Minimum Tester Qualifications

Selection and Vetting Requirements

Vendor Rotation Policy

• External testing vendors are rotated at minimum every 3 years for each scope area (network, application, cloud, OT, social engineering).
• Rotation applies to the firm, not individual testers.
• Exceptions require CISO approval and are documented in the risk register.

Contractual Requirements

• Non-disclosure agreement (NDA) covering all findings, data accessed, and methods used
• Data handling: all customer data accessed during testing is treated as Restricted; no data leaves the organization’s control without explicit authorization
• Evidence destruction: upon engagement closure or within 30 days, the firm certifies destruction of all engagement data, findings drafts, and tool output in their possession
• Right-to-audit: the organization reserves the right to audit the firm’s handling of engagement data
• Notification of subcontractors: the firm must disclose any subcontractors and their qualifications before engagement start

3. Annual Cadence and Scoping

CERG produces an annual Adversarial Validation Plan that schedules the engagements above against the in-scope environments. The plan is reviewed and approved by the CISO; it is updated quarterly to reflect emerging threats and program priorities.

Scoping inputs:

• Highest-residual-score risks in the register.
• Recently introduced systems and architectures (CERG-PRC-AR-001 Phase 5 records).
• Regulatory cadence requirements (CIP, CMMC).
• ATT&CK coverage gaps identified by detection engineering.
• Threat intelligence pointing at adversary activity relevant to the organization.

4. Rules of Engagement (RoE) Template

Every engagement has a signed RoE before any activity begins. The template:

RULES OF ENGAGEMENT - <Engagement Name>      AV-YYYY-NNNN

1. ENGAGEMENT IDENTITY
   Type                  : (External / Internal / Cloud / App / OT / Purple / SE / TT)
   Sponsor (named role)  :
   Test Lead             :
   Period                :
   Time Windows          :

2. OBJECTIVES
   What we are testing   :
   Success criteria      :
   Risk-register / control gaps this engagement is intended to validate

3. SCOPE - IN
   IPs / URLs / Apps / Cloud accounts / OT segments / Identity tenants explicitly in scope

4. SCOPE - OUT
   Explicit out-of-scope: e.g., production-impacting denial-of-service, mass account lockouts, third-party-hosted out-of-scope assets

5. PROHIBITED ACTIONS
   (Defaults below; engagement may add. None may be removed.)
   - No denial-of-service against production
   - No data destruction or modification
   - No persistence beyond engagement window
   - No exfiltration of regulated data (CUI / BCSI / PHI / PII); proof-of-concept counts; data is staged and discarded
   - No exploitation of safety-impacting OT systems (Section 9)
   - No social engineering of named C-suite without explicit prior approval
   - No physical intrusion outside agreed scope

6. SAFETY AND ABORT
   Stop conditions       : (defined; e.g., OT process alarms, anomalous business impact)
   Abort signal          : (named individual + backup; mechanism - phone / Signal channel)
   Operator on-call (OT) : (if OT)
   Incident handoff path : if findings cross the "test" → "real incident" threshold

7. AUTHORIZATION
   Authorization letter signed by   :
   Distribution                     :
   Authority to act in our environment

8. COMMUNICATIONS
   SOC informed?         : Yes - passive (no detection assistance) / Yes - active (purple)
   Liaison on Tester side:
   Liaison on CERG side  :
   Reporting cadence     : (daily standups / weekly status / final report)

9. EVIDENCE COLLECTION
   What is captured      :
   How it's stored       :
   What is sanitized in final report

10. CONFIDENTIALITY
    Findings classification (default: Restricted)
    Handling : encrypted at rest, named distribution, no email of unredacted

11. SIGNATURES
    Tester Lead          :
    CERG Sponsor         :
    System / Business Owner :
    CISO (red team / OT / cloud red team) :

5. Red-Team Engagement Charter

Red-team engagements are scope-broader, time-longer, goal-driven simulations of a realistic adversary. Each red team has a Charter in addition to its RoE.

RED-TEAM CHARTER - <Engagement Name>     AV-YYYY-NNNN

A. ADVERSARY MODEL
   Threat actor profile        :
   Capability                  : (commodity / advanced / nation-state-equivalent)
   Resourcing (time, tools, ops infrastructure)

B. OBJECTIVES - IN PRIORITY ORDER
   Crown Jewel(s) to attempt to reach
   Detection escape required?  :
   Persistence required?       :
   Specific TTPs to emulate    : (ATT&CK technique IDs)

C. ENGAGEMENT WINDOW
   Start / End                 :
   No-Op windows               : (financial close, regulatory exam, major event)

D. PRE-AGREED LIMITS
   Production-impact threshold :
   OT touch                    : (forbidden unless explicit)

E. WHITE-CELL
   Named coordinators on both sides; channels for clarification without breaking the test

6. Purple-Team Detection Validation

Purple work is the collaboration between offensive and detection engineering. It is not a graded exam of the SOC; it is a measurement of detection coverage and the basis for upgrading rules and baselines.

6.1 Cadence and Coverage

• Quarterly purple-team cycle covering a rotating slice of the ATT&CK matrix in scope.
• Annual full-scope purple-team across the IT spine; semi-annual on cloud sub-matrix; annual on OT sub-matrix.
• Each cycle has 8–20 techniques tested; the cycle’s selection is risk-driven (recent threat intel, recent infrastructure changes, prior-cycle failures).

6.2 Validation Workflow

1. Plan, choose techniques (ATT&CK IDs); map to expected detections in the SIEM; document expected alert.
2. Execute, emulate technique in non-production where possible; in production with controlled, documented action where required.
3. Observe, did the expected detection fire? With expected priority? In expected time?
4. Classify, Pass · Pass-with-Latency · Pass-with-Tuning-Needed · Fail (no detection) · Fail-Suppressed (existed but suppressed).
5. Action, Pass: no change. Pass-with-Tuning: tune. Fail / Fail-Suppressed: open detection backlog item + risk register entry until restored.
6. Reflect, feed result into detection inventory; update DT-002 metric in CERG-GOV-MTR-001.

6.3 Purple-Team Outcome to Baseline

The Purple Cycle Updates Detection, Not Just the Scorecard

If a technique fires nine times out of ten because logs were missing for one source, the fix is to fix the source, not to re-grade the test. If a technique fires reliably but the alert routes to the wrong queue, the fix is the queue. Purple cycles end with a list of concrete changes to the detection set; if they don’t, the cycle is incomplete.

7. Cloud Attack Simulation

Cloud engagements emulate adversary activity at the cloud control plane and identity layer, historically the most-attacked surface in modern enterprises.

Cloud engagements explicitly include detection validation; CSPM/SSPM signals and cloud-native detection (GuardDuty, Defender for Cloud, Security Command Center) are exercised.

8. Application Penetration Test Checklist

Application tests use OWASP WSTG (web), MASTG (mobile), and API Security Top 10 as the technical reference. The checklist below is the minimum coverage at scoping; specific apps may need more.

8.1 Social Engineering Engagements

Social engineering tests the human layer of security controls. These engagements carry unique legal, ethical, and operational risks that require explicit guardrails.

Scope Definition

Pretexting Rules and Limitations

• Pretexts must not impersonate law enforcement, government officials, or regulatory authorities.
• Pretexts must not exploit personal tragedy, medical conditions, or family emergencies.
• Pretexts must not target minors or non-employees.
• Pretexts must not threaten employment status, compensation, or benefits.
• All pretexts are defined in the RoE and approved by the Engagement Sponsor.

Prohibited Targets (require explicit CISO approval)

• C-suite executives and their direct administrative staff
• Board members
• Legal counsel (internal and external)
• Human Resources personnel
• Personnel on leave (medical, family, or other protected leave)

Coordination Protocol

Reporting Requirements

Social engineering findings are reported separately from technical findings. The report includes:

• Engagement subtype and scope
• Number of targets and interactions
• Success rate per subtype and per department
• Root cause themes (e.g., lack of verification procedure, urgency exploitation, authority bias)
• Recommended awareness and control improvements
• Comparison to prior social engineering engagements (if applicable)

9. OT Adversarial Assessment: Safety Protocol

OT testing is uniquely constrained. Safety overrides any test objective.

9.1 Prohibited Without Explicit Approval

The following are prohibited absent a documented exception with engineering and operations sign-off:

• Active scanning of live SCADA/HMI/RTU/relay surfaces. Passive monitoring and engineering-supervised authenticated checks only.
• Exploitation that could affect setpoints, alarms, relays, breakers, or control logic.
• Denial-of-service against any OT segment, including saturation testing.
• Disabling of OT monitoring or logging.
• Persistence on OT systems.
• Engagement during operational events: storms, demand peaks, planned outages, regulatory windows.

9.2 Permitted Approaches

• IT-side validation of paths into OT (jump server, vendor remote access, IT/OT crossing).
• Lab / digital-twin replication of OT systems for offensive testing.
• Passive observation and documentation of OT exposure.
• Engineering-supervised authenticated checks in a defined window.
• Tabletop / scenario-based assessment with OT operators.

9.3 Authority and Safety Officer

Every OT engagement has a named OT Safety Officer (an operator with substation / process engineering authority) who can stop the test for any reason without justification. The Safety Officer’s “stop” is binding.

The Cost of Getting OT Testing Wrong

A pen-test misstep on an enterprise web app produces a ticket. A misstep on a relay setting group can shed load or trip a substation. CERG conducts OT adversarial work with the same care a power engineer applies in the field, and structures the engagement so the engineer’s call always wins.

10. Finding Rating, Routing, and Retest

10.1 Rating

Findings are rated using a uniform schema regardless of engagement type, aligning to CERG-PRC-VM-001 severity bands so they can co-route into exposure management:

10.2 Routing

10.3 Retest and Closure

• Critical findings retested within 30 days of remediation claim.
• High findings retested within 60 days.
• Medium/Low retested at next routine engagement or by sampled VM scan.
• A finding moves to Closed only after retest demonstrates remediation; a finding that cannot be remediated moves through the risk register acceptance path with explicit Approver.
• Closed findings inform the threat model for the affected system at next architecture review.

10.4 Escalation for Stalled Remediation

When remediation stalls beyond the retest window, the finding escalates.

Escalation Triggers

Escalation Ladder

Adversarial Testing Lead → Risk Pillar Leader → CISO → Executive Sponsor

At each level, the responsible party must either (a) produce a credible remediation plan with a committed date, (b) initiate a risk acceptance through CERG-PRC-RM-001, or (c) provide a written rationale. Non-response at any level automatically escalates after 5 business days.

Risk Acceptance as the Only Alternative

When remediation is not feasible, risk acceptance requires: a written risk statement per PRC-RM-001 §4.1, documented compensating controls, approval per the authority matrix, and a scheduled re-review date. A finding that is neither remediated nor risk-accepted is a program gap, not an operational delay.

10.5 Adversarial Validation Final Report

Every engagement produces a Final Report - the authoritative record of what was tested, found, and recommended.

Required Sections

Evidence Artifact Standards

• Screenshots: Annotated with arrows or captions highlighting relevant detail; include timestamp and host identifier
• Command output: Full command and output with timestamp; truncation acceptable with notation
• Packet captures: Time-stamped with source/destination; filtered to relevant conversation
• Exploit code: Referenced by tool and version, not reproduced in full

10.6 Inter-Rater Reliability for Finding Ratings

Different testers may rate the same finding differently. The following process ensures severity ratings are consistent and defensible.

Peer Review for High-Severity Findings

All Critical and High findings receive a peer review by a second qualified tester before the finding is finalized. The peer reviewer validates: - The severity rating against the definitions in Section 10.1 - That the evidence supports the severity determination - That the finding is not a duplicate of a known issue

If the peer reviewer disagrees with the severity, the Adversarial Testing Lead facilitates resolution. The final severity is the consensus rating, or the higher of the two if consensus cannot be reached.

Calibration Sessions

Quarterly calibration sessions are conducted using a sample of anonymized findings from recent engagements. All testers independently rate each finding. Results are compared and discussed. The purpose is to: - Align severity interpretation across testers - Identify systematic over-rating or under-rating patterns - Refine severity definitions where ambiguity is found

The Adversarial Testing Lead facilitates calibration and documents outcomes. Calibration results inform tester development and, where appropriate, updates to this procedure’s severity definitions.

Dispute Resolution

11. Systemic Analysis and Program Feedback Loop

An Adaptive program does not stop at fixing individual pen-test findings. It asks: why did four of six DMZ servers share the same misconfiguration? What standard or procedure failed to prevent this? How do we change the program so the next engagement does not repeat the same class of finding?

This section defines the post-engagement systemic analysis that converts adversarial validation findings into program improvements.

11.1 Post-Engagement Systemic Analysis

After every adversarial validation engagement, the Adversarial Testing Lead produces two outputs: a findings report (existing) and a systemic analysis (new). The systemic analysis answers four questions:

1. Which findings share a common root cause? Example: four findings relate to default credentials on newly provisioned systems : the provisioning process does not enforce credential rotation. This is not four separate problems; it is one systemic weakness manifesting in four locations.
2. Which findings indicate a standard or procedure gap? Example: DMZ web servers all had the same TLS misconfiguration : STD-CFG-001 or STD-NET-001 does not specify DMZ TLS requirements clearly enough.
3. Which findings indicate a control effectiveness gap? Example: WAF was deployed per CB-001 but three bypass techniques worked : the WAF control is Implemented but not effective (per CEF-001).
4. Which findings indicate a detection gap? Example: testers pivoted laterally for four hours without an alert : detection rules need updating for the observed technique chain.

11.2 Systemic Finding Routing

Each systemic finding is routed to the appropriate owner for program-level action:

Routed systemic findings are tracked in the improvement register (IMPREG-001) per the improvement lifecycle. The source field records the engagement ID and date.

11.3 Prevention Verification

At the next adversarial validation engagement of the same type, scope, or target environment, the engagement plan includes re-testing the systemic weaknesses identified in the prior engagement.

• If the same class of finding does NOT recur: the systemic improvement is verified Effective in IMPREG-001.
• If the same class of finding recurs: the improvement is marked Ineffective or Partially Effective, re-opened in IMPREG-001, and the root cause analysis is re-examined. The fact that a deliberate program change did not prevent recurrence is itself a critical lesson.

The Recurrence Test

An assessor reviewing two consecutive pen test reports should see: (a) the specific vulnerabilities from the prior test are closed, AND (b) the systemic weakness that produced them no longer produces new findings of the same class. If the second report finds different specific vulnerabilities but the same systemic root cause : different default credentials on different systems : the improvement failed. The program that only fixes individual findings and never the systemic cause is operating at Repeatable, not Adaptive.

11.4 Annual Trend Analysis

Annually, the Adversarial Testing Lead produces an adversarial validation trend analysis covering all engagements in the trailing 12 months:

• Are total findings increasing or decreasing? Segmented by severity.
• Are systemic weaknesses repeating across engagements?
• Which control families (per CB-001) produce the most findings? The most Critical and High findings?
• Which control families show improvement (fewer findings year-over-year)?
• How many systemic findings were routed to IMPREG-001, and what was their verification outcome?

The trend analysis feeds into: - The control effectiveness program (CEF-001): control families with persistently high finding rates trigger a control failure analysis - The risk appetite calibration (RMF-001): a finding-rate trend that undermines a risk appetite assumption triggers recalibration - The annual maturity assessment (MAT-001): finding-rate trends are evidence of control effectiveness or ineffectiveness

12. Roles and Responsibilities

13. Evidence Retention

Final reports are distributed under the confidentiality terms in the RoE. Full reports are not emailed; access is via a controlled location with audit logging.

Findings Carry Sensitive Information About How to Compromise Us

An unsanitized pen-test report is a road map for attackers and should be treated as Restricted. Distribution is named, encrypted, and audited; “wider sharing” requests are reviewed by the Engagement Sponsor.

14. Regulatory and Framework Alignment Summary

15. Document Control

Source: procedures/CERG-PRC-AV-001_Adversarial_Validation_Procedure.md · Download .md · View on GitHub