Hitchhiker's Guide — Safe OT Assessment and Hardening

The operational companion to the Phase 10 WARMUP. The WARMUP explains why OT is safety-constrained security with inverted priorities; this guide is the how — the absolute safety rules, the assessment sequence, the digital-twin range, passive baselining, conduit review, safe simulated fixtures, and the recovery exercise. Everything here is simulation or supplied captures: no active scan, fuzz, write, firmware action, or controller restart on real equipment, ever.

Table of Contents

The Rules (Non-Negotiable)

  • simulation or supplied captures only unless a plant-approved procedure says otherwise;
  • no active scan, fuzz, write command, firmware action, or controller restart on real equipment (an Nmap can crash a PLC — Phase 10 WARMUP, Chapter 6);
  • safety and operations approve containment and maintenance (Chapter 11);
  • record process context for every observation (the same action is safe or dangerous depending on process state).

Assessment Sequence

  1. identify the process and safety consequences (consequence-driven risk, Chapter 2);
  2. validate any existing asset inventory passively;
  3. map zones, conduits, remote access, and cloud flows (Chapters 4, 8);
  4. review identities, change control, backups, and recovery images;
  5. monitor normal protocol roles/commands to baseline (Chapter 12);
  6. prioritize vulnerabilities with compensating controls (Chapter 9 — not blind patch SLAs);
  7. tabletop the incident and recovery with operations.

Build a Digital-Twin Range

Use simulated PLC/RTU, HMI, historian, engineering workstation, industrial DMZ, remote-access gateway, and a passive sensor. Represent process variables and safe ranges so detections can reason about command context (e.g. a setpoint outside the envelope) without touching real equipment. Keep the range physically/logically separate from production and household automation.

Passive Baseline

Observe: communicating assets, protocol roles, initiators, function/method use, polling intervals, firmware/project identifiers (where supplied), and normal maintenance windows. Record uncertainty and do not infer that silence means absence — passive monitoring only sees what talks during the capture window (Chapter 6). Validate with operators and engineering documentation.

The Asset-Identity Caveat

An IP address is not a durable asset identity. Address reuse, NAT, replacement devices, duplicate vendor strings, and stale captures require the inventory to carry confidence and conflicts (Phase 10 WARMUP, Chapter 7) — never overstate completeness, because segmentation and response decisions depend on it.

Conduit Review

For each flow, document: source role, destination role, protocol/function, direction, authentication, business/process need, expected frequency, owner, logging, and failure behavior. Reject: enterprise-to-controller bypass, broad any/any rules, unowned write functions, and cloud paths that fail into uncontrolled operation. Segmentation rules must correspond to approved operational flows, not arbitrary reachability (Chapters 4, 8).

Hardening

  • an industrial DMZ and one-way (diode) patterns where appropriate;
  • allowlisted flows and authenticated jump access;
  • MFA, time-bounded vendor access, session recording under policy;
  • controller/program/config backups with restore validation;
  • passive detections for new assets, role changes, unexpected commands, and engineering access;
  • secure gateway buffering and failure-safe cloud disconnection (data out only, no control in — Chapter 10).

Safe (Simulated) Attack Fixtures

Use simulation only to test: a new asset, an unexpected engineering login, a write command from the wrong role, a setpoint outside the declared envelope, a repeated rejected command, loss of the historian feed, a remote-access session outside its window, and a cloud-gateway disconnect. Validate the alerts and operator actions without sending malformed traffic to real devices.

Recovery Exercise

Practice: isolating remote access; switching to local/manual operation where the simulation allows (the OT lifeline, Chapter 11); comparing controller logic with a signed known-good copy; restoring engineering projects/config; validating process state and interlocks; coordinating vendor/safety approval; and returning in stages. Cyber recovery is complete only when operations confirms safe process behavior — not when the malware is removed.

Deliverables

  • a process and consequence model;
  • a confidence-rated asset inventory;
  • a zones/conduits matrix and diagram;
  • controller/project backup and restore evidence;
  • passive detection fixtures with process context;
  • a safety-approved incident decision tree (who approves what — Chapter 11);
  • vulnerability-treatment records explaining patch, compensating control, and residual risk.

Worked Examples

Reading a Modbus capture (from a supplied pcap, never a live device)

Time     Src(role)        Dst(role)     Function                       Read as...
0.000    10.5.2.10 (HMI)  10.5.1.20 PLC  Read Holding Registers (0x03)  normal poll
1.001    10.5.2.10 (HMI)  10.5.1.20 PLC  Read Holding Registers (0x03)  normal poll (1s interval)
2.310    10.5.4.7  (ENG)  10.5.1.20 PLC  Write Single Register  (0x06)  ← engineering WRITE

Modbus has no authentication — the write at 2.310s looks identical whether it came from the legitimate engineering workstation or an attacker who reached the control segment (Phase 10 WARMUP, Ch 5). Two detections fall out: (a) a write from a non-engineering source/zone (architecture-policy violation), and (b) a setpoint outside the declared process envelope (protocol misuse with process context). Both are passive — you never inject traffic.

A conduit policy row

src role   dst role   proto/func        dir  auth  process need        freq   owner  failure
HMI        PLC        Modbus 0x03 read   →    none  operator monitoring 1/s    ctrl-eng  fail-safe
ENG WS     PLC        Modbus 0x06 write  →    jump+ logic changes only  rare   ctrl-eng  manual fallback
historian  cloud      MQTT (one-way)     →    mTLS  analytics           5/s    ops    buffer; no inbound

Reject anything not on the list — especially enterprise→controller bypass, any/any rules, unowned write functions, and cloud paths that can send control inward.

The safety-aware decision tree (excerpt)

ALERT: unexpected write to PLC controlling pump speed
  process state idle?  ──yes──> isolate ENG segment (ops + ctrl-eng approve)
        │ no
        └──> DO NOT blindly isolate (could trip an unsafe state)
             → switch to MANUAL/local operation first (operators)  → then isolate
             → safety officer holds veto on anything affecting safety

The interviewer is checking whether you reach for the IT reflex ("isolate now") or the OT-correct answer ("manual operation first, joint approval, safety veto" — WARMUP Chapter 11).

Interview Drills

  1. "Secure remote vendor maintenance access." No persistent VPN into the control zone. Broker via the iDMZ + jump host with MFA, least privilege to the specific equipment, time-boxed sessions, full session recording, approval workflow; changes follow change control with control-eng sign-off.
  2. "Respond to suspicious HMI behavior." Preserve evidence passively; engage control-eng + safety to assess process state; decide containment jointly — isolating/rebooting mid-process may be unsafe, so the option may be manual operation first. No attribution; scope via passive logs; weigh blast radius and safety.
  3. "Connect a historian to cloud analytics safely." One-way only (diode/outbound-only, authenticated, least-priv); cloud can never send control inward; buffer locally so an outage doesn't disrupt the process; failure-safe. Threat-model device→gateway→cloud→operator→supply-chain.
  4. "Handle an unpatchable PLC with a known CVE." Patching may be impossible/unsafe, so rank by consequence × exposure and apply compensating controls: tighten segmentation, passive detection, restrict function codes/source-dest, keep a recovery image, coordinate a vendor fix in a planned window. The remediation's own outage/safety cost is part of the decision.
  5. "Why is active scanning forbidden in OT?" A scan/fuzz can crash fragile PLCs → outage or unsafe physical state. Discovery is passive; where active is unavoidable it needs control-eng approval, a maintenance window, and failure analysis. Safety and availability outrank confidentiality.

References

  • NIST SP 800-82 (OT security); ISA/IEC 62443 (zones/conduits, security levels).
  • CISA ICS advisories and recommended practices; MITRE ATT&CK for ICS.
  • Modbus / DNP3 / OPC UA / Profinet specs and vendor secure-deployment guides.
  • Case studies: Stuxnet, Ukraine grid (2015/2016), TRITON/TRISIS, Oldsmar.
  • Robert M. Lee / Dragos and SANS ICS writings; Wireshark industrial dissectors.