Incident Evidence Preservation

Evidence preservation during incidents.

Snapshot

The instinct during an incident is to fix what is broken. The discipline during an incident, especially a security incident, is to preserve evidence first. Once the system has been remediated, the data that would have proved root cause, attribution, and scope is gone. The first action when a security incident is suspected is freezing the state.

What snapshotting actually means in practice:

• EBS snapshots before remediation.: Take EBS snapshots (or equivalent for other cloud volumes) of every potentially-affected disk before any remediation work begins. The snapshots are point-in-time, immutable, and preserved separately from the running infrastructure. They capture the state the attacker left behind.
• Memory dumps if available.: RAM contents at the time of compromise often hold critical evidence: in-flight credentials, decrypted data, attacker-injected processes that did not write to disk. Modern cloud platforms support memory acquisition (AWS EC2 has options; on-prem requires forensic tools). Memory is volatile; capturing it is time-sensitive.
• Container state snapshots.: For containerized workloads, the running container's filesystem and process state can be exported with docker commit, kubectl debug, or equivalent. The image goes to a forensic registry separate from production, where it can be analyzed without rebuilding the workload.
• Network captures if relevant.: When the incident involved network exfiltration or lateral movement, packet captures from the time of the incident are golden evidence. If you have them already (VPC flow logs, sniffers, mirroring), preserve them. If you do not, this is the moment to start capturing.
• Before remediation, always.: The order matters: snapshot first, then remediate. Restarting a compromised host, redeploying a pod, or rotating credentials all destroy evidence that was on those systems. The snapshot has to be in flight before any of those happen.

The snapshot discipline is the reason forensic investigations succeed or fail. A snapshot taken in the first 30 minutes of an incident is worth more than any analysis tool you can buy.

Logs

The second pillar is log preservation. Most logging systems have retention policies that auto-delete data after some window. During an incident, that auto-deletion is the enemy. The goal is to lock the relevant logs out of the deletion process and preserve them indefinitely.

• Preserve relevant logs.: Application logs, system logs, audit logs, network flow logs, authentication logs, anything that touched the affected systems during the relevant time window. The window is usually broader than expected because attacker dwell time before detection averages weeks.
• Lock retention.: Apply legal hold or equivalent retention lock to the relevant log buckets. The logs cannot be deleted, expired, or compacted. The lock is independent of normal lifecycle policies; it overrides them. This is a deliberate operational action, not an automatic one.
• Copy to forensic store.: The logs are also copied to a separate forensic storage tier. The copy is read-only, durable, and access-controlled. Even if the operational logs are accidentally lost, the forensic copy survives.
• Don't let auto-deletion erase.: The most common evidence loss is logs aging out under normal retention while the investigation is still in flight. Identify which logs are at risk based on their retention policy and act on them within hours of the incident, not days.
• Time-of-day matters for some logs.: Some logs (load balancer access logs, CDN logs) age out very fast by default, often days. Others (audit logs, identity provider logs) are typically kept longer. Prioritize the fast-aging ones first.

Log preservation is unglamorous and time-sensitive. The team that has practiced it knows which logs to grab in what order and can do it in the first hour. The team that has not practiced it loses critical evidence to retention policies that nobody remembered to override.

Chain of custody

The third pillar is documentation: chain of custody. Every piece of evidence needs to be associated with who collected it, when, how, and where it has been stored since. This sounds bureaucratic; for any incident that might end up in litigation, regulatory action, or law enforcement involvement, it is non-negotiable.

• Documented and signed.: Every evidence item gets a custody record: what it is, when it was collected, by whom, how it was collected, where it is stored, who has accessed it since, and who currently has access. The record is signed by the collector at the time of collection.
• For litigation.: If the evidence ever needs to be presented in court, in a regulatory proceeding, or to law enforcement, the chain of custody is the credibility document. Without it, the evidence's integrity can be challenged. With it, the evidence is admissible.
• Cryptographic signing where possible.: Snapshots, log archives, and forensic copies are cryptographically hashed and signed at the moment of collection. The hash proves the evidence has not been altered since. The signature attaches the hash to a specific person and timestamp.
• Access logged forever.: Every access to the evidence (read, copy, analysis) is logged. The log is itself preserved and signed. The chain extends from the original collection through every subsequent touch, so any tampering can be detected.
• Trained custody.: The team members likely to be first responders during a security incident have basic forensic training. They know what to capture, in what order, with what documentation. The training is annual; the practice is part of the tabletop exercises.

Evidence preservation is the discipline that turns "we had an incident" into "we have a defensible record of what happened." Nova AI Ops integrates with the snapshot, logging, and custody mechanisms an organization uses, automates the legal-hold process during declared incidents, and produces the structured custody records that compliance and legal teams need without requiring manual effort during the incident.