Requirements and implementation evidence¶
This matrix tracks the Goldman Sachs AI Control Layer task and the submitted FastFence feature description against the current implementation. Sources are the supplied CRIETRIA AI Control Layer.pdf, RULES AI Control Layer.pdf, and the project submission. It is a traceability record, not a jury score or security certification. Tests listed below are reproducible verification targets; historical measurements retain their original scope in Testing.
Partner task requirements¶
| Requirement | Current implementation and source | Verification | Boundaries |
|---|---|---|---|
| Central policy engine and live configuration | Immutable policy/feed snapshots, local files or trusted HTTP bundle; policy storage, configuration providers | tests/unit/test_config_providers.py, tests/integration/test_policy_source_conflict.py, tests/unit/test_policy_persistence_bounds.py |
Invalid updates retain the last valid snapshot. One management writer per local source; independent processes are not coordinated. Identities are startup configuration. |
| Deterministic and AI-based controls | Local privacy, signatures and scoped literal predicates plus actual Laya semantic assessment; inspection, Laya provider | tests/unit/test_laya_semantic_runtime.py, tests/unit/test_named_semantic_rules.py, tests/integration/test_laya_raw_provider.py; separate live report in Testing |
Meaning-based classification is approximate. Exact character rules use deterministic predicates. Input blocks skip upstream work; output blocks cannot reverse work already executed. |
| Budget and resource limits | Atomic process-local reservations and settlement in the execution pipeline; execution | tests/unit/test_budgets.py, tests/integration/test_gateway.py |
Counters reset on restart. Independent instances have separate allowances. Token units and configured costs are estimates. |
| Known exploit/signature mitigation and feed changes | Versioned bounded textual signatures, normalization and constrained decoding; matcher | tests/unit/test_signature_matching.py, configuration-provider tests; recorded inert attack variants in Testing |
Text inspection does not inspect model binaries or establish comprehensive CVE protection. Quoted dangerous patterns can be conservatively blocked. |
| Dashboard, metrics and exportable audit | Management console with active controls, resource usage, recent decisions, local search and JSONL export; HTTP routes | evaluation/smoke_console.py, evaluation/smoke_audit_ui.py, tests/unit/test_telemetry_classification.py |
Bounded audit is in memory and omits raw content. No durable SIEM backend or cross-instance event store is implied. |
| Positive and negative automated self-tests | Isolated unit/integration suite, coverage gate, clean-clone smoke and real-browser CI; workflow | uv run pytest, scripts/smoke_clean_install.py, Chromium suites |
CI uses explicit model fixtures and offline configuration where appropriate. Actual model evidence is reported separately; passing fixtures are not inference evidence. |
Submitted product features¶
| Feature | Current implementation and verification | Boundary or outstanding work |
|---|---|---|
| Agent/model/tool interception | REST, bounded OpenAI-compatible text chat and authenticated MCP; tests/integration/test_mcp_models.py, test_openai.py, test_gateway.py |
Only explicitly routed operations are protected. The default runtime has no simulated business handlers. Real business adapters must be registered; examples are separate. |
| Natural-language policy authoring and generated regression cases | Describe a fast rule drafts a supported bounded configuration proposal. Local before/after preview checks reviewed expectations; activation saves the exact proposal. tests/integration/test_policy_regression_diff.py, tests/unit/test_policy_regression_generation.py |
Generated cases require human review. This feature is different from named semantic rule sample testing. Unsupported arbitrary prose is not silently compiled into a fast predicate. |
| Named semantic rules | Add Laya rule accepts instruction, direction and target; actual-model sample preview followed by explicit versioned policy review. tests/unit/test_named_semantic_rules.py, tests/integration/test_semantic_rule_preview.py, console browser acceptance |
A sample preview combines applicable rules and global guidance. It produces no individual matched-rule attribution and does not execute the complete gateway path. |
| Input/output anonymization with optional restoration | Stable scoped aliases, irreversible masking or stateless authenticated AES-GCM tokens, optionally using RSA-OAEP public-key envelopes, carrying recoverable values; restoration requires rule permission and request opt-in. tests/unit/test_anonymization_tokens.py, tests/integration/test_asymmetric_anonymization.py, tests/unit/test_anonymization_reinspection.py, tests/integration/test_stateless_control.py |
Optional FFR2 recovery uses a trusted RSA-3072 public/private pair plus the issuer keyring; see the asymmetric example. One recipient pair is supported, so replacing it invalidates older FFR2 tokens. Exact complete tokens are required; model-altered tokens are not guessed. |
| OCR and multipage documents to model-readable Markdown | Local OCR, ordered multipage PDF processing and policy-checked Markdown; tests/integration/test_document_markdown.py, tests/unit/test_ocr_media.py, tests/unit/test_ocr.py |
No image/PDF editing or restored visual document is produced. Model and OCR prerequisites must be installed locally. |
| Authentication, scoped permissions and allowlists | Server-side provisioned identities and role/model/tool policy checks, separate management access; tests/integration/test_gateway.py, test_mcp.py, test_openai.py |
Not a turnkey external identity-provider service. Callers cannot provide trusted roles through request content. |
Deliverables and jury verification¶
The architecture diagram, policy and threshold reference, runnable examples, generated HTTP inventory and manual checklist cover the documented technical deliverables. Presentation work is excluded from the current implementation scope at the project owner's request.
For an ad hoc review, change a policy or feed with a higher version, send previously unseen allowed and prohibited inputs, and inspect active version, stage decisions, budgets and telemetry. Repeat through both REST and MCP. Confirm an invalid source update preserves the previous valid policy. Evaluate semantic accuracy and latency on the actual configured model; deterministic-only historical benchmarks do not represent the default Laya request path.
The supplied scoring documents disagree on the final two weights: 30/20/20/15/15 in the criteria document versus 30/20/20/20/10 in the rules document. Keep both versions visible until the organizer clarifies; this matrix does not assign a speculative total score. The earlier readiness checkpoint and independent security review retain their stated scope and dates.