Leaked credentials are one of the fastest ways for attackers to move from “outside” to “inside.” A single exposed API key, SSH private key, database password, or cloud access token can enable data access, privilege escalation, crypto-mining, or destructive actions within minutes. This incident response playbook for leaked credentials is designed to be used under pressure: it prioritizes immediate containment, reliable rotation, and verification that the blast radius is understood.
Goal: stop active abuse, invalidate the leaked secret(s), restore secure access for legitimate systems, and reduce the chance of recurrence—while producing evidence for audits and post-incident learning.
What counts as “leaked credentials”?
Any secret that an unauthorized party could obtain, including:
- API keys, access tokens, OAuth refresh tokens
- Cloud IAM keys (e.g., AWS access keys, GCP service account keys)
- Database credentials and connection strings
- SSH private keys, TLS private keys, signing keys
- Application secrets (JWT signing secrets, webhook secrets)
- Hardcoded credentials in code, containers, images, CI logs
Preparation: define roles, sources of truth, and “break-glass” access
Credential leaks are easier to handle when you’ve pre-decided who does what. At minimum, define these roles:
- Incident Commander (IC): coordinates, timelines, decisions
- Security Lead: risk assessment, forensics direction
- Platform/Cloud Owner: IAM changes, key rotation, infra updates
- App Owner: service redeploy, config changes, feature flags
- Comms/Legal: notifications, customer and regulator obligations
Also ensure you have:
- Centralized logging (cloud audit logs, CI logs, VCS audit logs)
- A secrets inventory (where secrets live, what they access, owners)
- Documented rotation procedures per secret type
- “Break-glass” access that is auditable and time-bound
Severity triage: decide how fast you must act
Use a simple matrix to standardize urgency and communication.
| Severity | Example | Expected Response |
|---|---|---|
| Critical | Cloud admin key leaked; prod DB password posted publicly | Immediate containment (minutes), rotate/disable, forensic review |
| High | Scoped token leaked but can access sensitive data | Contain within an hour, rotate, verify access logs |
| Medium | Non-prod secrets leaked; limited privileges | Rotate same day, fix exposure path, monitor |
| Low | Expired token leaked; no access possible | Confirm expiration, improve controls, document |
Phase 1: Identification (confirm the leak and scope it)
Start by capturing the facts you know and the artifacts you must preserve:
- Where was it found? Git repo, CI logs, Slack, vendor ticket, public paste site, container image, endpoint response.
- What is the secret type? API key, IAM key, password, private key, token, certificate.
- Which environment? production, staging, dev, third-party SaaS.
- What can it access? services, data sets, permissions, network reach.
- Is there evidence of use? auth logs, cloud audit logs, app logs, unusual IPs, spikes in requests.
Preserve evidence before changes when possible (without delaying containment): export relevant audit logs, CI job logs, VCS commit history, and the leak location URL/screenshot. This supports later root-cause analysis and any compliance reporting.
Quick scoping checklist
- Identify the secret’s owner and system-of-record (who can rotate it?)
- List dependent workloads (apps, cron jobs, integrations) that will break on rotation
- Determine whether the secret was exposed publicly (assume compromise if yes)
- Search for duplicates: same secret reused in other places or environments
Phase 2: Containment (stop the bleeding)
Containment aims to prevent further unauthorized access while you prepare safe rotation.
- Disable or revoke the credential (preferred) rather than only rotating in code later.
- Block malicious access paths temporarily: IP blocks, WAF rules, conditional access policies.
- Reduce privileges immediately if you can’t revoke instantly (e.g., detach admin policies).
- Pause affected pipelines that might keep leaking (e.g., CI logs echoing env vars).
Rule of thumb: if a credential was exposed publicly or to an untrusted party, treat it as compromised—even if you don’t yet see abuse in logs.
Example: revoke an AWS access key (CLI)
If you’ve identified a specific access key ID, deactivating it is immediate containment.
# Deactivate the compromised access key
aws iam update-access-key \
--user-name <IAM_USER> \
--access-key-id <ACCESS_KEY_ID> \
--status Inactive
# Optional: delete after confirming replacement is working
aws iam delete-access-key \
--user-name <IAM_USER> \
--access-key-id <ACCESS_KEY_ID>
Phase 3: Eradication (remove the exposure source)
Eradication addresses how the secret leaked in the first place.
- Remove secrets from code and history: rewrite Git history if necessary, invalidate caches, and ensure forks are handled.
- Fix CI/CD logging: mask variables, stop printing env vars, prevent debug dumps in prod.
- Update developer workflows: pre-commit secret scanning, protected branches, code review gates.
- Harden endpoints: ensure APIs don’t accidentally return secrets in error messages.
Example: scan Git history for likely secrets
# Basic pattern checks (use dedicated secret scanners in real workflows)
# Search for common key prefixes in repository history
git log -p | grep -E "(AKIA[0-9A-Z]{16}|BEGIN PRIVATE KEY|xox[baprs]-)" -n
Note: use specialized secret-scanning tools in CI for better detection and fewer false positives; the snippet above is a last-resort triage aid.
Phase 4: Recovery (rotate safely and restore service)
Rotation is not just generating a new value. It’s a controlled sequence that ensures dependencies are updated and the old secret is invalidated at the right time.
Recommended rotation order
- Generate replacement secrets in the target system (IdP, cloud IAM, database, SaaS).
- Deploy updates to all consumers (apps, jobs, integrations) using your configuration mechanism.
- Verify functionality (health checks, synthetic transactions, integration tests).
- Revoke the old secret once the new one is confirmed working.
- Monitor for auth failures and suspicious retries.
Zero-downtime patterns
- Dual-credential window: allow both old and new for a short period (where supported), then disable old.
- Versioned secrets: consumers read the latest version; roll forward by bumping version.
- Staged rollout: rotate one service at a time; watch error budgets.
Phase 5: Verification (prove the incident is contained)
After containment and rotation, confirm that unauthorized access is no longer possible and that no additional footholds were created.
- Audit logs review: look for suspicious IPs, user agents, unusual regions, impossible travel, or new access patterns.
- Check for persistence: new IAM users, API tokens created, SSH keys added, new OAuth apps authorized.
- Data access validation: confirm whether sensitive tables/buckets were queried or exfiltrated.
- Alert tuning: add detections for the exact abuse pattern you observed.
Communication: internal updates and external notifications
Credential leaks can trigger customer, partner, or regulatory obligations depending on what the credential accessed. Keep communications factual and timestamped.
Internal status update template
- What happened: e.g., “API key exposed in public repo”
- Impact: systems/data potentially accessed
- Actions taken: revocation, rotation, monitoring enabled
- Current risk: contained/not contained, evidence of misuse yes/no
- Next steps: remaining rotations, forensics tasks, timeline for RCA
When to escalate
Escalate to legal/compliance if the credential could access regulated data, if you see evidence of exfiltration, or if incident reporting time windows may apply.
Post-incident: root cause analysis and prevention controls
The playbook isn’t complete until you reduce recurrence. A strong post-incident review should produce specific control changes and owners.
Common root causes
- Hardcoded secrets committed during debugging
- Over-privileged credentials used for convenience
- Long-lived tokens with no rotation policy
- Secrets copied into tickets/chat without redaction
- CI systems printing environment variables or stack traces
High-impact preventative controls
- Short-lived credentials: prefer federated auth and expiring tokens over static keys.
- Least privilege by default: scope secrets to a single service and environment.
- Automated rotation: rotate on schedule and on-demand after any suspected leak.
- Secret scanning: pre-commit + CI + repository hosting scans.
- Centralized secrets storage: minimize copies, enable audit trails and approvals.
- Runbooks: per secret type (cloud, database, SaaS) with tested steps.
Printable checklist: incident response playbook for leaked credentials
- Triage: confirm secret type, environment, potential access.
- Preserve evidence: capture logs, URLs, commit hashes, timestamps.
- Contain: revoke/disable credential; block abuse paths if needed.
- Scope: identify all consumers; search for reuse across systems.
- Rotate: generate replacement; deploy to all consumers.
- Invalidate old: delete or permanently revoke after verification.
- Verify: review audit logs; check for persistence or new tokens/users.
- Communicate: internal updates; external notifications if required.
- Prevent: implement scanning, least privilege, automation, and training.
Closing note
Credential leaks are inevitable in complex systems; unmanaged fallout isn’t. If you standardize this workflow, rehearse it, and automate rotation and auditing, you can shrink both impact and downtime. Teams that rely on a dedicated secrets management platform (for example, Vaulify) often find it easier to centralize rotation, reduce secret sprawl, and produce audit-ready access records.