Available on AWS Marketplace · Version 0.1.0
Development evaluation
Published 0.1.0 on AWS Marketplace. The final amd64 image passed an actual network-isolated SageMaker endpoint evaluation on ml.g6e.4xlarge and actual certification batch on ml.g6e.2xlarge. The table separates this AWS run from the earlier Thor diagnostics. These related development cases are not customer accuracy or a service-level guarantee.
Document-exact means the action segment/person/ISO-date metadata multiset matches the corrected gold labels. Rejected outputs count as incorrect.
| FP8 diagnostic |
English |
Spanish |
Portuguese |
Total exact |
Admitted |
False-action negative cases |
| v2, corrected labels |
18/20 |
17/20 |
17/20 |
52/60 |
59/60 |
1/21 |
| v3, Thor, corrected labels |
19/20 |
20/20 |
19/20 |
58/60 |
60/60 |
0/21 |
| v3, final AWS image, corrected labels |
19/20 |
20/20 |
19/20 |
58/60 |
60/60 |
0/21 |
For v3, metadata precision is 1.00 in each language; recall is 16/17 in English, 17/17 in Spanish and 16/17 in Portuguese. The two remaining misses are the venue's third-party lift-inspection promise in English and Portuguese. All three ambiguous incident-log examples match the included-task labels; excluding that stratum leaves 55/57 exact records.
v3 changes both the instructions and evidence protocol, so the comparison does not isolate either change's causal effect. The model selects segment IDs; the server copies the source quote. A previous 53/60 v2 score used superseded labels and is not the comparable baseline.
The actual Thor HTTP adapter also passed six positive/negative language fixtures and five input-boundary checks. Those six short requests took 12.70–27.60 seconds; they are not a recommended-instance p95. Summary factuality and semantic entailment remain unmeasured. See data scope.
Final-image AWS checks
Image: sha256:6d38d9384206b0b77181db4f47167dde5bd23fe9e4c7ff86075dfcc2b468c917, version 0.1.0, US East (N. Virginia). The 60 requests ran serially on one 4xlarge with network isolation and no API environment overrides. All 60 were admitted without request failures; nearest-rank end-to-end p95 was 33.16 seconds, below the provisional 55-second gate. Deployment, first-kernel initialization and preceding protocol probes are outside that latency measurement. It is not throughput or a latency guarantee for arbitrary documents.
Two valid protocol probes, six malformed/oversized/token-limit rejection probes and post-rejection recovery passed. The certified batch job returned three actual EN/ES/PT results; source quotations, Unicode offsets, named owners and literal dates were checked against the exact fictional inputs. Batch startup and these three outputs do not establish a batch latency percentile. The ECR basic scan completed with no HIGH/CRITICAL findings; this does not claim zero vulnerabilities or assess the AWS-managed host kernel.