Synthetic
MaxArc Global Health Impact Platform
Service Reliability, Operational Continuity, Backup, Recovery & Observability Governance
Cross-module service reliability, continuity, recovery, and observability governance.
A synthetic governance demonstration and simulation only — not a production monitoring platform, not a live backup service, not a disaster-recovery system, not a cloud-control plane, not a network-management system, not a production incident-response engine, not an automated failover service, and not a replacement for authorized infrastructure or operational teams. Synthetic service status is not live production status; a dashboard indicator is not production telemetry; a single successful check does not establish reliability. Degraded, maintenance, unknown, and available states remain distinct. This module never restarts, stops, modifies, fails over, or recovers production services, creates no real backups, and performs no real restoration. Source-module records remain authoritative. AI is assistive only and never restarts, fails over, recovers, backs up, restores, certifies, or authorizes anything.
Tier-0/1
Synthetic status
≠ available
Synthetic
≠ outage
≠ available
Simulated
Remain visible
Controlled
Human-reviewed
Traceable
Remain visible
Gap
Draft / review
Synthetic
Synthetic
Remain visible
Not verified
Remain visible
Not current RP
Simulation
≠ production proof
Remain partial
Remain visible
Evidence gap
Observed ≠ target
Observed ≠ target
Remain visible
Missing / stale
Remain visible
Remain visible
Human review
Remain visible
2nd review
Blocked / pending
Human review required
Append-only
Services (synthetic status ≠ live production status)
Every service identifies source module, criticality, tier, owner, operational owner, status, continuity plan, backup policy, RTO/RPO/MTD, minimum service level, dependencies, observability status, and review dates. Degraded, maintenance, unknown, and available remain distinct; this module never restarts, stops, fails over, or recovers production services.
| Service | Name / owners | Status / objectives | Continuity / backup / audit |
|---|---|---|---|
| SVC-FUND fund-accountability · tier-0 · critical |
Fund Accountability Service owner service-owner · ops operational-owner |
available RTO 1h · RPO 15m · MTD 2h |
plan CP-FUND · backup BP-CRIT · obs instrumented · audit AUD-SVC-1 |
| SVC-STOCK stockpile-logistics · tier-1 · high |
Stockpile & Logistics Service owner service-owner · ops operational-owner |
degraded degraded RTO 4h · RPO 1h · MTD 8h |
plan CP-STOCK · backup BP-HIGH · obs partial · audit AUD-SVC-2 |
| SVC-LAB lab-operations · tier-0 · critical |
Laboratory Operations Service owner service-owner · ops operational-owner |
available RTO 1h · RPO 15m · MTD 2h |
plan CP-LAB · backup BP-CRIT · obs instrumented · audit AUD-SVC-3 |
| SVC-PCON patient-continuity · tier-0 · critical |
Patient Continuity Service owner service-owner · ops operational-owner |
available RTO 1h · RPO 15m · MTD 2h |
plan CP-PCON · backup BP-CRIT · obs instrumented · audit AUD-SVC-4 |
| SVC-RLOC restricted-patient-locator · tier-1 · critical |
Restricted Patient Locator Service owner service-owner · ops operational-owner |
unavailable RTO 1h · RPO 15m · MTD 2h |
plan CP-RLOC · backup BP-CRIT · obs instrumented · audit AUD-SVC-5 |
| SVC-ASSET asset-management · tier-2 · high |
Asset Management Service owner service-owner · ops operational-owner |
available RTO 4h · RPO 1h · MTD 8h |
plan CP-ASSET · backup BP-HIGH · obs instrumented · audit AUD-SVC-6 |
| SVC-PERF program-performance · tier-2 · medium |
Program Performance Service owner service-owner · ops operational-owner |
available RTO 12h · RPO 6h · MTD 24h |
plan CP-PERF · backup BP-STD · obs instrumented · audit AUD-SVC-7 |
| SVC-AI ai-intelligence · tier-1 · high |
AI Intelligence Service owner service-owner · ops operational-owner |
unknown RTO 4h · RPO 1h · MTD 8h |
plan CP-AI · backup BP-HIGH · obs telemetry-missing · audit AUD-SVC-8 |
| SVC-ROLL country-rollout · tier-3 · medium |
Country Rollout Service owner service-owner · ops operational-owner |
available RTO 12h · RPO 6h · MTD 24h |
plan CP-ROLL · backup BP-STD · obs instrumented · audit AUD-SVC-9 |
| SVC-ROOM authorized-review-room · tier-2 · medium |
Authorized Review Room Service owner service-owner · ops operational-owner |
available RTO 12h · RPO 6h · MTD 24h |
plan CP-ROOM · backup BP-STD · obs instrumented · audit AUD-SVC-10 |
| SVC-PACKET executive-review-packet · tier-3 · medium |
Review Packet & Download Center owner service-owner · ops operational-owner |
maintenance RTO 12h · RPO 6h · MTD 24h |
plan CP-PACKET · backup BP-STD · obs instrumented · audit AUD-SVC-11 |
| SVC-IDENT identity-access-governance · tier-0 · critical |
Identity & Access Governance Service owner service-owner · ops operational-owner |
available RTO 1h · RPO 15m · MTD 2h |
plan CP-IDENT · backup BP-CRIT · obs instrumented · audit AUD-SVC-12 |
| SVC-DQ data-quality-reconciliation · tier-2 · high |
Data Quality & Reconciliation Service owner service-owner · ops operational-owner |
available RTO 4h · RPO 1h · MTD 8h |
plan CP-DQ · backup BP-HIGH · obs instrumented · audit AUD-SVC-13 |
| SVC-RISK risk-incident-case-governance · tier-1 · high |
Risk, Incident & Case Governance Service owner service-owner · ops operational-owner |
available RTO 4h · RPO 1h · MTD 8h |
plan CP-RISK · backup BP-HIGH · obs instrumented · audit AUD-SVC-14 |
| SVC-POLICY policy-compliance-control-governance · tier-1 · high |
Policy & Control Governance Service owner service-owner · ops operational-owner |
available RTO 4h · RPO 1h · MTD 8h |
plan CP-POLICY · backup BP-HIGH · obs instrumented · audit AUD-SVC-15 |
| SVC-PLATFORM policy-compliance-control-governance · tier-0 · critical |
Impact Platform Core Service owner service-owner · ops operational-owner |
available RTO 1h · RPO 15m · MTD 2h |
plan CP-PLATFORM · backup BP-CRIT · obs instrumented · audit AUD-SVC-16 |
Dependencies (existence ≠ availability; fallback ≠ effectiveness)
Module, infrastructure, network, data, identity, and evidence dependencies. Unknown status remains unknown; single points of failure remain visible; critical dependencies require explicit continuity handling; downstream impact and cross-module dependencies remain traceable.
| Dependency | Path / impact | Status | Degraded behavior / audit |
|---|---|---|---|
| DEP-FUND module |
SVC-FUND → SVC-IDENT (identity) impact fund approvals blocked |
available fallback fallback-available |
read-only ledger view · review reviewed · audit AUD-DEP-1 |
| DEP-STOCK data |
SVC-STOCK → SVC-DQ (reconciliation) impact stock reconciliation delayed |
degraded fallback fallback-degraded |
queued reconciliation · review reviewed · audit AUD-DEP-2 |
| DEP-LAB network |
SVC-LAB → result routing path impact result routing halted |
available fallback fallback-available |
local queue then sync · review reviewed · audit AUD-DEP-3 |
| DEP-PCON module |
SVC-PCON → SVC-LAB (results) impact follow-up gaps |
available fallback fallback-available |
manual follow-up list · review reviewed · audit AUD-DEP-4 |
| DEP-RLOC identity |
SVC-RLOC → SVC-IDENT (authorization) impact restricted locator disabled (fails safe closed) |
unavailable SPOF fallback no-fallback-by-design |
access denied by default · review under-review · audit AUD-DEP-5 |
| DEP-ASSET data |
SVC-ASSET → custody records store impact custody updates delayed |
available fallback fallback-available |
offline custody log · review reviewed · audit AUD-DEP-6 |
| DEP-PERF module |
SVC-PERF → SVC-DQ (verified data) impact reporting stale |
available fallback fallback-available |
last-verified snapshot · review reviewed · audit AUD-DEP-7 |
| DEP-AI evidence |
SVC-AI → cross-module evidence index impact AI signals unavailable (no autonomous action) |
unknown fallback fallback-unknown |
signals suppressed pending human review · review under-review · audit AUD-DEP-8 |
| DEP-ROLL network |
SVC-ROLL → country connectivity path impact rollout assessment delayed |
degraded fallback fallback-degraded |
offline assessment capture · review reviewed · audit AUD-DEP-9 |
| DEP-IDENT-PLATFORM identity |
SVC-PLATFORM → identity authorization core impact all authorized actions blocked (fails safe) |
available SPOF fallback no-fallback-by-design |
deny-by-default · review reviewed · audit AUD-DEP-10 |
| DEP-NET-PLATFORM infrastructure |
SVC-PLATFORM → synthetic hosting node impact platform unreachable |
available fallback fallback-available |
offline site operation · review reviewed · audit AUD-DEP-11 |
| DEP-IDENT identity |
SVC-IDENT → authorization decision store impact authorization decisions unavailable |
available fallback fallback-available |
cached deny-by-default · review reviewed · audit AUD-DEP-12 |
| DEP-DQ data |
SVC-DQ → source-module records (authoritative) impact reconciliation blocked |
available fallback fallback-available |
queued reconciliation · review reviewed · audit AUD-DEP-13 |
| DEP-RISK module |
SVC-RISK → SVC-POLICY (controls) impact case-control linkage delayed |
available fallback fallback-available |
manual linkage · review reviewed · audit AUD-DEP-14 |
| DEP-POLICY evidence |
SVC-POLICY → control evidence store impact control testing evidence unavailable |
available fallback fallback-available |
read-only evidence · review reviewed · audit AUD-DEP-15 |
| DEP-ROOM module |
SVC-ROOM → SVC-POLICY (evidence) impact review packages stale |
available fallback fallback-available |
cached package view · review reviewed · audit AUD-DEP-16 |
| DEP-PACKET module |
SVC-PACKET → SVC-ROOM (packages) impact download center offline |
maintenance fallback fallback-available |
catalog-only view · review reviewed · audit AUD-DEP-17 |
Service-level objectives (governance targets, not contracts)
- SLO-1 — availability 99.0%SVC-FUND · planning target, not a contractual commitment · audit AUD-SLO-1
- SLO-2 — latency <800ms p95SVC-LAB · planning target, not a contractual commitment · audit AUD-SLO-2
- SLO-3 — freshness <15m data ageSVC-PCON · planning target, not a contractual commitment · audit AUD-SLO-3
- SLO-4 — completeness >=98% recordsSVC-RLOC · planning target, not a contractual commitment · audit AUD-SLO-4
- SLO-5 — recovery-time-objective 1hSVC-IDENT · planning target, not a contractual commitment · audit AUD-SLO-5
- SLO-6 — recovery-point-objective 15mSVC-STOCK · planning target, not a contractual commitment · audit AUD-SLO-6
- SLO-7 — maximum-tolerable-downtime 2hSVC-DQ · planning target, not a contractual commitment · audit AUD-SLO-7
- SLO-8 — minimum-service-level core-read-and-authorized-writeSVC-PLATFORM · planning target, not a contractual commitment · audit AUD-SLO-8
Service-level indicators (distinct from objectives)
- SLI-1 — availability observed 98.6% current at-riskindicator ≠ objective · basis synthetic sampled demonstration values (not live telemetry) · audit AUD-SLI-1
- SLI-2 — latency observed 910ms p95 current breachingindicator ≠ objective · basis synthetic sampled demonstration values (not live telemetry) · audit AUD-SLI-2
- SLI-3 — freshness observed unknown missing unknownindicator ≠ objective · basis synthetic sampled demonstration values (not live telemetry) · audit AUD-SLI-3
- SLI-4 — completeness observed 97.1% stale at-riskindicator ≠ objective · basis synthetic sampled demonstration values (not live telemetry) · audit AUD-SLI-4
- SLI-5 — availability observed 99.4% current meetingindicator ≠ objective · basis synthetic sampled demonstration values (not live telemetry) · audit AUD-SLI-5
- SLI-6 — synchronization-lag observed 42m current breachingindicator ≠ objective · basis synthetic sampled demonstration values (not live telemetry) · audit AUD-SLI-6
Continuity plans (lifecycle states remain distinct)
Draft, pending-review, approved, active, expired, superseded, blocked, and withdrawn remain distinct. Missing approval blocks active status; expired or superseded plans are never current; plan existence does not prove readiness; activation requires an attributable human decision; conflicting versions remain visible; history is never silently overwritten.
| Plan | Status | Approval | Posture / audit |
|---|---|---|---|
| CP-FUND v3.0 SVC-FUND / fund-accountability |
active effective 2025-01-15 · expires 2026-01-31 |
recovery-approver 2nd review | missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-1 |
| CP-LAB v2.1 SVC-LAB / lab-operations |
active effective 2025-01-15 · expires 2026-03-31 |
recovery-approver 2nd review | missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-2 |
| CP-PCON v1.2 SVC-PCON / patient-continuity |
approved effective 2025-01-15 · expires 2026-02-28 |
recovery-approver 2nd review | missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-3 |
| CP-RLOC v0.3 SVC-RLOC / restricted-patient-locator |
draft effective — · expires — |
no approver 2nd review | missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-4 |
| CP-IDENT v4.0 SVC-IDENT / identity-access-governance |
active effective 2025-01-15 · expires 2026-05-31 |
recovery-approver 2nd review supersedes CP-IDENT-OLD |
missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-5 |
| CP-IDENT-OLD v3.0 SVC-IDENT / identity-access-governance |
superseded effective 2025-01-15 · expires 2025-05-31 |
recovery-approver 2nd review superseded by CP-IDENT · |
missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-6 |
| CP-STOCK v1.0 SVC-STOCK / stockpile-logistics |
expired effective 2025-01-15 · expires 2025-01-31 |
recovery-approver | missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-7 |
| CP-AI v0.9 SVC-AI / ai-intelligence |
blocked effective — · expires — |
no approver 2nd review | missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-8 |
| CP-ROLL v1.0 SVC-ROLL / country-rollout |
pending-review effective — · expires 2026-06-30 |
no approver 2nd review | missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-9 |
| CP-PACKET v0.5 SVC-PACKET / executive-review-packet |
withdrawn effective — · expires — |
no approver | missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-10 |
| CP-CONFLICT v3.0-alt SVC-FUND / fund-accountability |
pending-review conflicting version effective — · expires 2026-01-31 |
no approver 2nd review | missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-11 |
| CP-PLATFORM v5.0 SVC-PLATFORM / platform-core |
active effective 2025-01-15 · expires 2026-04-30 |
recovery-approver 2nd review | missing approval blocks active · expired/superseded ≠ current · audit AUD-CP-12 |
Offline operations (controlled mode, not unrestricted access)
- OFF-1 — no-connectivity-facility-operation offline pendingSVC-PCON · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-1
- OFF-2 — intermittent-connectivity-facility-operation intermittent delayedSVC-LAB · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-2
- OFF-3 — delayed-synchronization intermittent delayedSVC-STOCK · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-3
- OFF-4 — conflicting-offline-updates offline conflictSVC-DQ · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-4
- OFF-5 — duplicate-synchronization-candidates intermittent duplicate-candidateSVC-DQ · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-5
- OFF-6 — stale-reference-data offline staleSVC-POLICY · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-6
- OFF-7 — incomplete-evidence-synchronization offline incompleteSVC-ROOM · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-7
- OFF-8 — identity-authorization-expiration-while-offline offline blocked authorization expired — deniedSVC-IDENT · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-8
- OFF-9 — commodity-movement-pending-synchronization intermittent pendingSVC-STOCK · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-9
- OFF-10 — laboratory-result-routing-pending-synchronization intermittent pendingSVC-LAB · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-10
- OFF-11 — patient-continuity-event-pending-synchronization offline pendingSVC-PCON · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-11
- OFF-12 — asset-custody-update-pending-synchronization intermittent pendingSVC-ASSET · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-12
- OFF-13 — program-result-pending-synchronization intermittent pendingSVC-PERF · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-13
- OFF-14 — audit-event-synchronization-backlog offline backlogSVC-PLATFORM · authorization preserved · audit preserved · later data does not silently overwrite · audit AUD-OFF-14
Synchronization recovery (later data never silently overwrites)
- SYNC-1 — offline -> reconciliation queue -> human-reviewed merge conflict-pending-reviewlater data overwrites: no · duplicates remain candidates: yes · semantic reconciliation proven: no
- SYNC-2 — offline -> duplicate-candidate review duplicate-pending-reviewlater data overwrites: no · duplicates remain candidates: yes · semantic reconciliation proven: no
- SYNC-3 — offline -> audit backlog -> blocked reconciliation backloglater data overwrites: no · duplicates remain candidates: yes · semantic reconciliation proven: no
- SYNC-4 — intermittent -> delayed sync -> reconciled delayedlater data overwrites: no · duplicates remain candidates: yes · semantic reconciliation proven: no
Backups (no real backups created; presence ≠ verification)
This task creates no real backups. Backup presence is not verification; completion is not integrity proof; integrity is not restoration proof; an encryption-status label is not cryptographic verification. Failed, partial, unverified, expired, and integrity-concern backups remain visible; expired backups are not current recovery points; missing evidence blocks verified status; records never contain real storage locations, credentials, keys, or production paths.
Backup policies
- BP-CRIT — critical services (synthetic) criticalfreq every-15-minutes-synthetic · retention 90-days-synthetic · RPO 15m · verification required · audit AUD-BP-1
- BP-HIGH — high-criticality services (synthetic) highfreq hourly-synthetic · retention 60-days-synthetic · RPO 1h · verification required · audit AUD-BP-2
- BP-STD — standard services (synthetic) standardfreq daily-synthetic · retention 30-days-synthetic · RPO 24h · verification required · audit AUD-BP-3
| Backup | Completion / verification | Encryption / retention | Posture / audit |
|---|---|---|---|
| BK-1 SVC-FUND dataset (synthetic) |
completed verified integrity intact |
encryption-status:labelled-encrypted · retention retained · expires 2025-09-30 | presence ≠ verification · completion ≠ integrity · label ≠ crypto proof · audit AUD-BK-1 |
| BK-2 SVC-IDENT dataset (synthetic) |
completed verified integrity intact |
encryption-status:labelled-encrypted · retention retained · expires 2025-09-30 | presence ≠ verification · completion ≠ integrity · label ≠ crypto proof · audit AUD-BK-2 |
| BK-3 SVC-STOCK dataset (synthetic) |
partial unverified integrity unknown |
encryption-status:labelled-encrypted · retention retained · expires 2025-08-31 | presence ≠ verification · completion ≠ integrity · label ≠ crypto proof · audit AUD-BK-3 |
| BK-4 SVC-LAB dataset (synthetic) |
failed verification-failed integrity unknown |
encryption-status:label-missing · retention retained · expires 2025-08-31 | presence ≠ verification · completion ≠ integrity · label ≠ crypto proof · audit AUD-BK-4 |
| BK-5 SVC-PCON dataset (synthetic) |
completed unverified integrity unknown |
encryption-status:labelled-encrypted · retention retained · expires 2025-09-30 | presence ≠ verification · completion ≠ integrity · label ≠ crypto proof · audit AUD-BK-5 |
| BK-6 SVC-PERF dataset (synthetic) |
completed verified integrity integrity-concern |
encryption-status:labelled-encrypted · retention retained · expires 2025-08-01 | presence ≠ verification · completion ≠ integrity · label ≠ crypto proof · audit AUD-BK-6 |
| BK-7 SVC-ROLL dataset (synthetic) |
completed verified integrity intact |
encryption-status:labelled-encrypted · retention expired · expires 2025-01-10 | presence ≠ verification · completion ≠ integrity · label ≠ crypto proof · audit AUD-BK-7 |
| BK-8 SVC-PLATFORM dataset (synthetic) |
completed verified integrity intact |
encryption-status:labelled-encrypted · retention retained · expires 2025-09-30 | presence ≠ verification · completion ≠ integrity · label ≠ crypto proof · audit AUD-BK-8 |
Recovery exercises & restoration simulations (simulation only)
Results use pass, partial, fail, blocked, not-tested, and not-applicable. A successful exercise does not prove production recoverability; RTO and RPO remain distinct; observed performance remains distinguishable from target; failed and partial exercises remain visible; later success does not erase prior failures; missing evidence blocks a pass; restoration simulation changes no production data; recovery approval does not authorize production restoration; independent review is required for high-criticality services.
| Exercise | Result / SoD | Target vs observed | Gaps / audit |
|---|---|---|---|
| EX-1 SVC-FUND · tabletop |
pass SoD compliant |
RTO 1h / observed 52m RPO 15m / observed 12m |
no gaps · simulation only · audit AUD-EX-1 |
| EX-2 SVC-LAB · functional-restore-simulation |
partial SoD compliant |
RTO 1h / observed 1h48m RPO 1h / observed 1h30m |
result-routing revalidation incomplete · simulation only · audit AUD-EX-2 |
| EX-3 SVC-IDENT · failover-simulation |
fail SoD blocked |
RTO 1h / observed >4h (not met) RPO 15m / observed unknown |
authorization cache did not warm; deny-by-default held · simulation only · audit AUD-EX-3 |
| EX-3-R SVC-IDENT · failover-simulation-retest |
partial retest of EX-3 SoD compliant |
RTO 1h / observed 1h20m RPO 15m / observed 18m |
improved but RTO still unmet · simulation only · audit AUD-EX-4 |
| EX-4 SVC-PCON · tabletop |
blocked SoD compliant |
RTO 1h / observed n/a RPO 15m / observed n/a |
missing evidence blocked a pass · simulation only · audit AUD-EX-5 |
| EX-5 SVC-STOCK · tabletop |
not-tested SoD compliant |
RTO 4h / observed n/a RPO 1h / observed n/a |
not exercised this period · simulation only · audit AUD-EX-6 |
| EX-6 SVC-ROLL · tabletop |
not-applicable SoD compliant |
RTO 12h / observed n/a RPO 6h / observed n/a |
no gaps · simulation only · audit AUD-EX-7 |
| EX-7 SVC-PLATFORM · full-restore-simulation |
pass SoD compliant |
RTO 1h / observed 58m RPO 15m / observed 14m |
no gaps · simulation only · audit AUD-EX-8 |
Outages (declaration & restoration require human decisions)
- OUT-1 — SVC-RLOC restored highcontinuity activation: yes (recovery-approver) · validation reviewed · restoration ≠ complete until validation reviewed · audit AUD-OUT-1
- OUT-2 — SVC-STOCK partially-restored mediumcontinuity activation: yes (recovery-approver) · validation partial · restoration ≠ complete until validation reviewed · audit AUD-OUT-2
- OUT-3 — SVC-AI investigating lowcontinuity activation: no · validation pending · restoration ≠ complete until validation reviewed · audit AUD-OUT-3
Maintenance windows (maintenance ≠ outage)
- MW-1 — SVC-PACKET in-progress maintenance ≠ outagesynthetic catalog refresh · approver operational-owner
- MW-2 — SVC-PERF planned maintenance ≠ outagesynthetic index rebuild · approver operational-owner
Observability & alerts (synthetic telemetry, not production)
Synthetic metrics, logs, and traces are not production telemetry. Alert generation is not incident confirmation; an alert is not proof of outage, attack, loss, or failure. Stale alerts, false positives, and unacknowledged alerts remain visible; AI may prioritize alerts but never restarts or fails over services; alert closure requires attributable human review.
Observability signals
- availability — SVC-FUND value nominal-synthetic current oksynthetic telemetry, not production · confidence medium · audit AUD-OBS-1
- latency — SVC-LAB value elevated-synthetic current oksynthetic telemetry, not production · confidence medium · audit AUD-OBS-2
- freshness — SVC-PCON value unknown missing unknownsynthetic telemetry, not production · confidence low · audit AUD-OBS-3
- completeness — SVC-RLOC value elevated-synthetic current oksynthetic telemetry, not production · confidence medium · audit AUD-OBS-4
- queue-depth — SVC-IDENT value nominal-synthetic current oksynthetic telemetry, not production · confidence medium · audit AUD-OBS-5
- synchronization-lag — SVC-STOCK value elevated-synthetic current breachingsynthetic telemetry, not production · confidence medium · audit AUD-OBS-6
- backup-status — SVC-DQ value nominal-synthetic current oksynthetic telemetry, not production · confidence medium · audit AUD-OBS-7
- error-rate — SVC-PLATFORM value elevated-synthetic current breachingsynthetic telemetry, not production · confidence medium · audit AUD-OBS-8
- dependency-status — SVC-FUND value nominal-synthetic current oksynthetic telemetry, not production · confidence medium · audit AUD-OBS-9
- capacity — SVC-LAB value elevated-synthetic stale oksynthetic telemetry, not production · confidence medium · audit AUD-OBS-10
- storage — SVC-PCON value nominal-synthetic current oksynthetic telemetry, not production · confidence medium · audit AUD-OBS-11
| Alert | Severity / status | Escalation / incident | Posture / audit |
|---|---|---|---|
| AL-1 SVC-LAB · from OBS-8 |
high open unacknowledged |
escalation escalation-pending · incident — | alert ≠ incident · alert ≠ proof · closure requires human review · audit AUD-AL-1 |
| AL-2 SVC-STOCK · from OBS-6 |
medium acknowledged acknowledged |
escalation not-escalated · incident OUT-2 | alert ≠ incident · alert ≠ proof · closure requires human review · audit AUD-AL-2 |
| AL-3 SVC-AI · from OBS-3 |
low open unacknowledged · stale |
escalation not-escalated · incident — | alert ≠ incident · alert ≠ proof · closure requires human review · audit AUD-AL-3 |
| AL-4 SVC-PERF · from OBS-2 |
low closed acknowledged · false-positive |
escalation not-escalated · incident — | alert ≠ incident · alert ≠ proof · closure requires human review · audit AUD-AL-4 |
| AL-5 SVC-RLOC · from OBS-1 |
critical acknowledged acknowledged |
escalation escalated · incident OUT-1 | alert ≠ incident · alert ≠ proof · closure requires human review · audit AUD-AL-5 |
Resilience risks (SPOF, capacity, storage, connectivity, sync)
- RR-1 — single-point-of-failure high SPOFidentity authorization is a synthetic single point of failure · status mitigating · audit AUD-RR-1
- RR-2 — dependency medium SPOFrestricted locator depends entirely on identity availability (fails safe closed) · status accepted · audit AUD-RR-2
- RR-3 — capacity mediumreconciliation queue capacity risk under backlog · status open · audit AUD-RR-3
- RR-4 — storage lowsynthetic backup storage growth risk · status open · audit AUD-RR-4
- RR-5 — connectivity mediumcountry connectivity variability · status mitigating · audit AUD-RR-5
- RR-6 — synchronization mediumsynchronization conflict risk during offline operation · status mitigating · audit AUD-RR-6
- RR-7 — observability-gap mediumtelemetry freshness missing for AI service · status open · audit AUD-RR-7
Post-incident reviews (separate from restoration)
- PIR-1 — outage OUT-1 completedidentity dependency was a single point of failure; fail-safe held · separate from restoration · audit AUD-PIR-1
- PIR-2 — outage OUT-2 in-progressreconciliation dependency degraded; partial restoration remains partial · separate from restoration · audit AUD-PIR-2
Corrective actions (completion ≠ effectiveness)
- CA-1 — corrective-action completed effectivecompletion ≠ effectiveness · residual low · linked OUT-1 · audit AUD-CA-CA-1
- CA-2 — dependency-reduction in-progress overdue pendingcompletion ≠ effectiveness · residual high · linked RR-1 · audit AUD-CA-CA-2
- CA-3 — resilience-improvement completed ineffectivecompletion ≠ effectiveness · residual medium · linked OUT-2 · audit AUD-CA-CA-3
- CA-4 — recovery-plan-improvement in-progress pendingcompletion ≠ effectiveness · residual high · linked EX-3 · audit AUD-CA-CA-4
- CA-5 — capacity-improvement open pendingcompletion ≠ effectiveness · residual medium · linked RR-3 · audit AUD-CA-CA-5
- CA-6 — monitoring-improvement open pendingcompletion ≠ effectiveness · residual low · linked OBS-3 · audit AUD-CA-CA-6
- CA-7 — backup-improvement in-progress overdue pendingcompletion ≠ effectiveness · residual medium · linked BK-4 · audit AUD-CA-CA-7
- CA-8 — synchronization-control-improvement open pendingcompletion ≠ effectiveness · residual medium · linked OFF-4 · audit AUD-CA-CA-8
- CA-9 — offline-procedure-improvement open pendingcompletion ≠ effectiveness · residual low · linked CP-STOCK · audit AUD-CA-CA-9
- CA-10 — documentation-improvement open pendingcompletion ≠ effectiveness · residual low · linked PIR-1 · audit AUD-CA-CA-10
- CA-11 — training-action open pendingcompletion ≠ effectiveness · residual low · linked RR-5 · audit AUD-CA-CA-11
- CA-12 — preventive-action open pendingcompletion ≠ effectiveness · residual medium · linked EX-3 · audit AUD-CA-CA-12
- CA-13 — independent-review open pendingcompletion ≠ effectiveness · residual high · linked RR-1 · audit AUD-CA-CA-13
Effectiveness reviews
- ER-1 — action CA-1 effectiveaction verified effective · 2nd review effectiveness-second-reviewer
- ER-2 — action CA-3 ineffectiveaction completed but did not resolve residual risk · 2nd review effectiveness-second-reviewer
- ER-3 — action CA-2 pendingawaiting completion and second review · 2nd review effectiveness-second-reviewer
Separation-of-duties rules
Violations remain blocked or pending, identify the violated rule, require reassignment, independent review, or second review, remain auditable, and are never silently overridden.
| Rule | Enforcement | Override / audit |
|---|---|---|
| service owner cannot independently approve final continuity readiness SOD-1 · service-owner-cannot-approve-continuity-readiness |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| continuity-plan author cannot independently provide final approval SOD-2 · continuity-author-cannot-final-approve |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| backup operator cannot independently verify backup integrity SOD-3 · backup-operator-cannot-verify-integrity |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| backup record creator cannot independently certify restoration readiness SOD-4 · backup-creator-cannot-certify-restoration-readiness |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| recovery-exercise performer cannot independently provide final exercise approval SOD-5 · exercise-performer-cannot-approve-exercise |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| restoration operator cannot independently approve final restoration validation SOD-6 · restoration-operator-cannot-approve-validation |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| outage reporter cannot independently close the outage SOD-7 · outage-reporter-cannot-close-outage |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| alert creator cannot independently close a high-severity alert SOD-8 · alert-creator-cannot-close-high-severity-alert |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| dependency owner cannot independently certify fallback effectiveness SOD-9 · dependency-owner-cannot-certify-fallback |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| synchronization operator cannot independently resolve a high-risk conflict SOD-10 · sync-operator-cannot-resolve-high-risk-conflict |
pending | never silently overridden auditable · requires reassignment / independent / second review |
| offline-site operator cannot independently approve return to normal operation SOD-11 · offline-operator-cannot-approve-return-to-normal |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| corrective-action owner cannot independently verify high-risk effectiveness SOD-12 · ca-owner-cannot-verify-high-risk-effectiveness |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| infrastructure assessor cannot independently attest production readiness SOD-13 · infra-assessor-cannot-attest-production-readiness |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| AI signal generator cannot independently trigger failover, restart, recovery, or closure SOD-14 · ai-generator-cannot-trigger-failover-restart-recovery-closure |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| fund-accountability owner cannot independently attest continuity of financial controls SOD-15 · fund-owner-cannot-attest-financial-continuity |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| warehouse operator cannot independently certify continuity of commodity custody SOD-16 · warehouse-operator-cannot-certify-commodity-continuity |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| laboratory operator cannot independently certify result-routing recovery SOD-17 · lab-operator-cannot-certify-result-routing-recovery |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| patient-continuity operator cannot independently certify follow-up recovery SOD-18 · pcon-operator-cannot-certify-followup-recovery |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| restricted-locator operator cannot independently certify privacy-control restoration SOD-19 · rloc-operator-cannot-certify-privacy-restoration |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| asset custodian cannot independently certify asset-record recovery SOD-20 · asset-custodian-cannot-certify-asset-recovery |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| program-result submitter cannot independently certify reporting recovery SOD-21 · program-submitter-cannot-certify-reporting-recovery |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| rollout assessor cannot independently authorize country deployment recovery SOD-22 · rollout-assessor-cannot-authorize-deployment-recovery |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| review-room author cannot independently certify evidence-package recoverability SOD-23 · room-author-cannot-certify-evidence-recoverability |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| packet author cannot independently attest download-center continuity SOD-24 · packet-author-cannot-attest-download-continuity |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| identity approver cannot independently certify access-control recovery SOD-25 · identity-approver-cannot-certify-access-recovery |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| data-quality correction owner cannot independently certify reconciliation recovery SOD-26 · dq-owner-cannot-certify-reconciliation-recovery |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| incident investigator cannot independently certify case-system restoration SOD-27 · incident-investigator-cannot-certify-case-restoration |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
| compliance-control owner cannot independently attest control recovery effectiveness SOD-28 · compliance-owner-cannot-attest-control-recovery |
blocked | never silently overridden auditable · requires reassignment / independent / second review |
Evidence (presence ≠ verification; conflicts remain visible)
- EVID-1 — recovery-exercise-report present verified currentpolicy-compliance-control-governance · EX-1
- EVID-2 — backup-verification-record present verified currentpolicy-compliance-control-governance · BK-1
- EVID-3 — backup-verification-record missing missing n/apolicy-compliance-control-governance · BK-4
- EVID-4 — continuity-activation-record present verified currentrisk-incident-case-governance · OUT-1
- EVID-5 — sync-reconciliation-log present conflicting currentdata-quality-reconciliation · SYNC-1
- EVID-6 — continuity-plan-approval present verified supersededpolicy-compliance-control-governance · CP-IDENT-OLD
AI resilience signals (human review required)
- stale-continuity-plan — CP-STOCK continuity plan expired and stale confidence mediumhuman review: yes · autonomous action: no
- dependency-concentration — identity authorization concentrated across modules confidence mediumhuman review: yes · autonomous action: no
- single-point-of-failure — restricted locator fully depends on identity availability confidence mediumhuman review: yes · autonomous action: no
- missing-backup-evidence — BK-4 has no verification evidence confidence mediumhuman review: yes · autonomous action: no
- unmet-recovery-objective — EX-3 failover did not meet RTO confidence mediumhuman review: yes · autonomous action: no
- recovery-gap-priority — identity failover gap is high priority confidence mediumhuman review: yes · autonomous action: no
- stale-alert — AL-3 alert is stale confidence mediumhuman review: yes · autonomous action: no
- duplicate-alert-candidate — possible duplicate alert pattern flagged for review confidence mediumhuman review: yes · autonomous action: no
- overdue-resilience-action — CA-2 is overdue confidence mediumhuman review: yes · autonomous action: no
- synchronization-backlog — OFF-14 audit synchronization backlog detected confidence mediumhuman review: yes · autonomous action: no
- recommend-independent-review — EX-3 recommended for independent review confidence mediumhuman review: yes · autonomous action: no
- potentially-ineffective-action — CA-3 completed but potentially ineffective confidence mediumhuman review: yes · autonomous action: no
Approval chains
- AC-1 — continuity-plan CP-FUNDcontinuity-owner:author(complete) → technical-reviewer:review(complete) → recovery-approver:approve(complete)
- AC-2 — recovery-exercise EX-3recovery-exercise-performer:perform(complete) → independent-reviewer:independent-review(pending) → recovery-exercise-approver:approve(blocked)
- AC-3 — outage-restoration OUT-1restoration-operator:restore(complete) → restoration-validator:validate(complete)
Immutable audit events (append-only)
Continuity-plan versions, activations and deactivations, backup verification decisions, recovery exercises and retests, outage declaration and restoration, alert acknowledgement and closure, and corrective-action effectiveness decisions all require new linked events; deletion is not an allowed governance control; source-module audit records remain authoritative. This is not a production cryptographic audit ledger.
- AUD-LINK-CPV-1 — continuity plan version active continuity-plan-versionrecovery-approver · policy-compliance-control-governance · 2025-04-20T10:00:00Z · immutable: yes
- AUD-LINK-ACT-1 — continuity activation activated continuity-activationrecovery-approver · risk-incident-case-governance · 2025-04-21T10:00:00Z · immutable: yes
- AUD-LINK-DEACT-1 — continuity deactivation deactivated continuity-deactivationrecovery-approver · risk-incident-case-governance · 2025-04-22T10:00:00Z · immutable: yes
- AUD-LINK-BKV-1 — backup verification verified backup-verificationbackup-integrity-reviewer · policy-compliance-control-governance · 2025-04-23T10:00:00Z · immutable: yes
- AUD-LINK-EX-1 — recovery exercise fail recovery-exerciserecovery-exercise-performer · policy-compliance-control-governance · 2025-04-24T10:00:00Z · immutable: yes
- AUD-LINK-EXR-1 — recovery retest partial recovery-retestrecovery-exercise-performer · policy-compliance-control-governance · 2025-04-25T10:00:00Z · immutable: yes
- AUD-LINK-OUTD-1 — outage declaration declared outage-declarationoutage-reporter · risk-incident-case-governance · 2025-04-26T10:00:00Z · immutable: yes
- AUD-LINK-OUTR-1 — outage restoration restored outage-restorationrestoration-validator · risk-incident-case-governance · 2025-04-27T10:00:00Z · immutable: yes
- AUD-LINK-ALA-1 — alert acknowledgement acknowledged alert-acknowledgementalert-operator · risk-incident-case-governance · 2025-04-28T10:00:00Z · immutable: yes
- AUD-LINK-ALC-1 — alert closure closed alert-closurealert-approver · program-performance · 2025-04-20T10:00:00Z · immutable: yes
- AUD-LINK-ERD-1 — corrective action effectiveness ineffective corrective-action-effectivenesseffectiveness-second-reviewer · policy-compliance-control-governance · 2025-04-21T10:00:00Z · immutable: yes
Services by module
- fund-accountability 1
- stockpile-logistics 1
- lab-operations 1
- patient-continuity 1
- restricted-patient-locator 1
- asset-management 1
- program-performance 1
- ai-intelligence 1
- country-rollout 1
- authorized-review-room 1
- executive-review-packet 1
- identity-access-governance 1
- data-quality-reconciliation 1
- risk-incident-case-governance 1
- policy-compliance-control-governance 2
Services by tier / status
- tier-0 5
- tier-1 5
- tier-2 4
- tier-3 2
- available 12
- degraded 1
- unavailable 1
- unknown 1
- maintenance 1
Continuity by state
- active 4
- approved 1
- draft 1
- superseded 1
- expired 1
- blocked 1
- pending-review 2
- withdrawn 1
Recovery by result
- pass 2
- partial 2
- fail 1
- blocked 1
- not-tested 1
- not-applicable 1
AI posture — assistive only, non-autonomous
AI assists with
- identify stale or missing continuity plans
- detect dependency concentration
- identify single points of failure
- detect missing backup evidence
- compare recovery objectives with synthetic exercise results
- identify unmet recovery objectives
- prioritize recovery gaps
- identify stale, duplicate, or conflicting alerts
- summarize synthetic outages and recovery chronology
- identify overdue resilience actions
- detect synchronization backlogs and conflicts
- recommend cases for independent or second review
- reconcile module, dependency, plan, backup, recovery, and evidence references
- suggest resilience improvements
- identify potentially ineffective corrective actions
AI must never
- restart, stop, reload, fail over, or recover services
- create or restore backups
- alter production infrastructure
- change network, DNS, firewall, certificate, PM2, Apache, systemd, database, or environment configuration
- declare or close a production outage
- authorize continuity activation
- authorize restoration
- approve backup integrity
- certify recoverability
- certify continuity readiness
- suppress failed backups, alerts, outages, or recovery gaps
- fabricate telemetry or evidence
- notify external parties
- authorize deployment
- bypass human approval, independent review, second review, separation of duties, evidence, expiration, or audit controls
AI is assistive only. It may prioritize and summarize for authorized human reviewers, but never restarts, fails over, recovers, backs up, restores, certifies, or authorizes anything, and never bypasses human approval, independent review, second review, separation of duties, evidence, expiration, or audit controls.
Runtime boundary & module coverage
- Impact runtime:
impact.maxarchealth.comon127.0.0.1:3201. - Covers reliability, continuity, backup, recovery, observability, and resilience for fund accountability, stockpile & logistics, lab operations, patient continuity, restricted locator, asset management, program performance, AI intelligence, country rollout, the authorized review room, the executive review packet & controlled download center, identity & access governance, data quality & reconciliation, risk, incident & case governance, and policy, compliance & control governance — without coupling to MaxTrax EHR.
- No Apache, PM2, systemd, firewall, DNS, TLS, production backup, production log, or production database is read, polled, or modified.
- Medical Library boundary remains separate at
library.maxarchealth.com(port3101) and is not used here. - MaxTrax EHR remains separate; this is not a full EHR and implements no live integration or real database mutation.