Observatory ProtocolOpen developmentConformance programme
The conformance programme
Profiles, levels, the open conformance suite and how a claim is made, reviewed and listed.
- Version
- 0.1.0-draft
- Status
- Draft specification
- Specification
- CC BY 4.0
- Code
- Apache-2.0
- Steward
- GCRI Canada
- Source
- obsp-0.1-foundation
Profiles
Each profile names the functions an implementation of that kind provides. (OBSP 0.1 §7.1)
| Profile | For | Functions | Cases |
|---|---|---|---|
| OBSP-NODE | Node agent and device | join describe time observe qualify orchestrate exit | 19 |
| OBSP-SITE | Site cell controller | join describe time observe qualify orchestrate exit admit correct prove | 22 |
| OBSP-GRID | Cluster and grid control plane | join describe time observe qualify orchestrate exit admit correct prove federate exchange | 25 |
| OBSP-PROVIDER | Hardware, software, model and data provider | admit describe exit | 13 |
| OBSP-CONSUMER | Application reading products | exchange correct | 13 |
Levels
A level is reached only with the evidence it names; "conformant" describes a tested scope and is never written as certification. (OBSP 0.1 §7.2)
- Declared
- Signed self-declaration naming profile, version and scope.
- Self-tested
- Conformance report (obsp conformance run) produced by the participant, with digests of the tested artifacts.
- Lab-tested
- Conformance report (obsp conformance run) and lab test record from a qualified lab.
- Qualified
- Lab report plus field observation for the declared scope and a qualification-record.
- Suspended
- Suspension notice with reason.
- Withdrawn
- Withdrawal notice.
Conformance claims
How an implementation states its conformance, and what such a statement means. (Observatory Protocol policy, Conformance claims)
CNF-1- Conformance is attestation from evidence, never certification, accreditation or endorsement. Nobody may describe a claim as a certificate.
CNF-2- A claim names the implementation and version, the profiles, the level, the sovereignty overlay when claimed, and the digests of the evidence and of the conformance report.
CNF-3- A claim above DECLARED is reproducible: anyone can run the conformance suite on the published evidence and obtain the same report.
CNF-4- A claim lapses when the implementation changes, when the evidence expires or when a newer suite version fails it; the listing then says so.
CNF-5- Listing in IMPLEMENTATIONS.md is not endorsement, and the protocol name may be used only to state a claim that meets these rules.
Making a claim
The steps from a run of the suite to a listed claim. (Observatory Protocol, Conformance guide)
Conformance is attestation from evidence. It is never certification, accreditation or endorsement (CNF-1).
Levels
| Level | Evidence |
|---|---|
DECLARED | A conformance claim naming the profile and version |
SELF_TESTED | The claim, plus a conformance report from the open suite on published evidence, reproducible by anyone (CNF-3) |
LAB_TESTED | As above, run by an independent test lab on the tested artifacts |
QUALIFIED | As above, plus field qualification evidence named in the profile |
SUSPENDED and WITHDRAWN record a claim that no longer holds (CNF-4).
Steps
- Collect evidence documents for your profile. The cases in conformance/cases/ say which schemas each case needs. Example sets are in conformance/examples/.
- Run the suite: ``
sh obsp conformance run --profile OBSP-SITE --evidence ./evidence --out report.json`` - Write a conformance-claim with the digests of the report and the evidence. Validate it with
obsp validate. - Open a pull request adding a row to IMPLEMENTATIONS.md, or submit the claim to the gateway API (
POST /conformance-claims). - The conformance reviewers rerun the suite on your evidence. A claim that does not reproduce is not listed.
Manual cases (kind: manual) list the procedure and the evidence an assessor records. Without that evidence they report MANUAL, never PASS.
The conformance suite
How the suite turns the profiles into cases an implementer runs on their own evidence. (Observatory Protocol, Conformance suite)
The suite turns the conformance profiles (registers/conformance-profiles.yaml, specification section 10) into test cases an implementer can run on their own evidence. A run produces a ConformanceReport (schemas/json/conformance-report.schema.json). A participant then states what it claims in a ConformanceClaim (schemas/json/conformance-claim.schema.json), citing the report by digest. A conformance claim attests to a tested scope. It is never a certification. The level it states is bounded by the evidence that level requires.
Layout
| Path | Content |
|---|---|
suite.yaml | Suite version, the profiles and the overlay, and how requirement ids are formed |
cases/<PROFILE>/CT-<NAME>-NNN.yaml | One case per file |
examples/<PROFILE>/ | An evidence set that passes, built from fixtures/positive |
examples/negative/OBSP-NODE/ | An evidence set that must fail |
Requirements and coverage
For a profile P, the requirements are P:<function> for every function P lists, plus SOV-1 to SOV-10 when the run includes the operator sovereignty overlay (OBSP-SOV, specification section 12). A case names the functions or SOV ids it covers and the profiles it applies to (appliesTo): the node cases also run for site cells and grids. The suite check (obsp conformance check-suite) fails when any requirement of any profile has no case.
Cases
| Field | Meaning |
|---|---|
id | CT-<NAME>-NNN, also the file name |
profile | The profile that owns the case (its directory), or OBSP-SOV |
appliesTo | The profiles whose runs include the case |
requirements | Function ids of the profile, or SOV-n |
level | The lowest conformance level the case supports |
kind | schema: each named document is present and valid. invariant: the same, and the named cross-document invariants hold. manual: a procedure the implementer performs and attests to |
evidence | The schema names of the documents the implementer supplies |
invariants | Named checks implemented in sdk/python/src/obsp/conformance.py |
procedure | What a manual case requires, and what to record |
Evidence sets
Put one JSON document per file in a directory, named <schema-name>.<label>.json, for example time-quality.node01.json. A fixture wrapper (_schemaTitle, document) is also accepted. Manual cases are answered in attestations.yaml in the same directory:
CT-SOV-001:
result: PASS
statement: egress log of the default install over one hour shows no connection; log digest sha256:...Without an attestation, a manual case reports MANUAL and the run is INCOMPLETE. A run is PASS only when every case passes or is not applicable and every requirement is covered.
Running
pip install -e sdk/python
obsp conformance check-suite
obsp conformance run --profile OBSP-NODE --evidence conformance/examples/OBSP-NODE --out report.json
obsp conformance run --profile OBSP-NODE --sovereignty --evidence conformance/examples/OBSP-NODE
obsp conformance run --profile OBSP-NODE --evidence conformance/examples/negative/OBSP-NODE # exits 1The report lists the digest of every evidence document (RFC 8785 canonical form, SHA-256) and the digest of the whole set. Anyone holding the same evidence can repeat the run and get the same digests. --at fixes the timestamp, so a rerun produces an identical report.
Changing the suite
A new or changed case follows the OIP process when it changes what a profile requires. A case that only adds a check for a requirement that already exists is an editorial change. Bump version in suite.yaml on every release that changes a case.
Listed implementations
Implementations that claim a profile, with their evidence. Listing is not endorsement. (Observatory Protocol, Implementations)
Implementations that claim a conformance profile of the Observatory Protocol. Listing is not endorsement. A claim above DECLARED links the evidence named in registers/conformance-profiles.yaml.
| Implementation | Version | Profile | Level | Evidence |
|---|---|---|---|---|
| (none listed yet) |
To add a row, open a pull request with the declaration and, for SELF_TESTED or higher, the report and the digests of the tested artifacts.
References
Sources cited on this page, in order of first citation. Addresses were last checked on the date shown.
- OBSP 0.1 §7.1: Profiles Observatory Protocol 0.1 (draft), CC BY 4.0.
- OBSP 0.1 §7.2: Conformance levels Observatory Protocol 0.1 (draft), CC BY 4.0.
- Observatory Protocol policy, Conformance claims: registers/policies.yaml conformance-claims Observatory Protocol 0.1 (draft), CC BY 4.0.
- Observatory Protocol, Conformance guide: docs/guides/conformance.md Observatory Protocol 0.1 (draft), CC BY 4.0.
- Observatory Protocol, Conformance suite: conformance/README.md Observatory Protocol 0.1 (draft), CC BY 4.0.
- Observatory Protocol, Implementations: IMPLEMENTATIONS.md Observatory Protocol 0.1 (draft), CC BY 4.0.