Trace Provider Data Audit — Integration & API Reference
The single source of truth for the Trace data audit API: provider onboarding, auth, the Trace Schema, write and read endpoints, search, stats, scoped groups, limits, and validation.The API and Trace Schema are provider-agnostic — every provider integrates through the same endpoints, headers, and schema. Examples use Kled, a live provider; substitute your own provider identity wherever Kled appears.
Environments
Staging is where new provider integrations are onboarded and tested. Production is deployed but may not yet have provider data — check with the DATA Foundation team before sending production traffic.
Getting access
Write access is gated. Before integrating, contact the DATA Foundation team to be onboarded: we whitelist your provider identity, issue staging and production API keys, and assign the
X-Provider value your write requests must use. Until that is done, write requests are rejected. Read, search, and scoped-group endpoints are public audit views and do not require a key.Auth and provider scope
Every write request must include:- The API key must belong to your provider and match
X-Provider; use the key for the environment you’re targeting. - For backlog ingestion, add
X-Ingestion-Source: backlog. For live ingestion, omit the header — explicitliveis not accepted. - Read APIs are keyed by the global
data_id;provideris returned as a field and doubles as an optional filter.
Ingestion flow
1
Send a batch
The provider client POSTs to a webhook batch endpoint.
2
Asynchronous processing
Accepted items are enqueued and processed asynchronously by the DATA Foundation.
3
Read back
Records become available through the read and search endpoints.
records:batch response always includes per-item statuses:
202 Accepted— at least one item was accepted for asynchronous processing and no item conflicted.200 OK— every item was a duplicate, so no new async work was enqueued.409 Conflict— at least one item conflicted; anyaccepteditems in the same response were still enqueued.
X-Batch-Id.
Record lifecycle
Each record’s history is an append-only event log that Trace renders as a timeline. Each step maps to exactly one value; a step appears on the timeline when its value is present.
Steps 1–3 come from the registration payload’s metadata; steps 4–7 are audit events. Send each milestone as its own metadata update — one milestone, one timeline event.
The root
occurred_at on each batch item is required, provider-supplied, and dates that specific event — registration time on the DataRegistered event, update time on each MetadataUpdated event. Trace surfaces the Registered lifecycle step with its own ingested_at — the moment the DATA Foundation accepted the record, returned on per-record reads. timestamps.originated_at inside the attested payload carries the record’s origin moment: set it equal to the registration’s root occurred_at, and keep it fixed on later updates (each update’s root occurred_at dates the update itself). Records registered before this field was renamed carry the same value under the legacy name timestamps.occurred_at — readers should accept both. The only values the DATA Foundation adds are ingested_at and tx_hash.
Trace Schema v1.0
Populate the standardized Trace Schema fields directly and preserve your full original public payload underprovider_payload. The normalized fields are the portable contract the frontend and audit flows use; nothing provider-specific is lost.
- Use
schema_version: trace-v1.0ininitial_metadata_jsonandmetadata_json. The value is not validated — a missing or unparseableschema_versiondefaults totrace-v1.0, and whatever value is sent is folded into the event hash. - The DATA Foundation canonicalizes metadata JSON before computing internal event hashes, so object key order does not affect idempotency or conflict detection.
- Do not send a transaction hash in write payloads. The DATA Foundation owns
tx_hashand returns it on read responses for registration, metadata update, and search result rows — as an empty string until the DATA Foundation fills it after broadcast.
Canonical content hash (required)
Initial registrations and metadata updates must include one canonical content hash. Accepted fields areasset.hash, content_hash, file.content_sha256, or file.hashes.sha256; all normalize to sha256:<64-lowercase-hex>. If more than one alias is present, they must represent the same hash.
Recommended top-level shape
Searchable fields
Current exact-match searchable fields:
The only searchable hash is the canonical content SHA-256 — perceptual hashes (
phash64, dhash64, ahash64, keyframe_phashes) and md5 are stored but not indexed. The same goes for app.platform_name, app.legal_entity, file.behavior.*, file_specific.base.motion.*, and timestamps.payment_credited_at: send them when useful — stored fields can be indexed later without resending old data. Use /stats for distributions and provider as an optional query scope.
Field guidance
source_record_id— provider-owned stable public media ID (for examplekmf_...). The service trims surrounding whitespace, preserves case, rejects control characters, and accepts up to 512 bytes.contributor.anon_id— provider-owned public anonymized contributor ID.asset.collection_id,asset.customer_id,asset.task_id— optional provider-assigned grouping IDs: a collection or batch of records, the customer or campaign they were produced for, and the task they belong to. Each is exact-match searchable, and matching is case-sensitive likesource_record_id. Send them at the top level (as in the shape above) or insideprovider_payload— both are read. Omit any that don’t apply.contributor.kyc_status— recommended values:verified,pending,failed,unverified.contributor.kyc_country— ISO 3166-1 alpha-2 country code from KYC, if available. Country only; no address or GPS-derived country.contributor.tax_status— recommended values:submitted,not_submitted,not_applicable,unknown. For a boolean liketax_form_on_file, maptruetosubmittedandfalsetonot_submitted.contributor.account_verification_status— recommended values:verified,pending,failed,unverified.contributor.consent.tos_*andcontributor.consent.privacy_policy_*— the exact policy version, hash, and URI the contributor accepted for this record. Provider-level active policies are set separately throughPUT /webhook/v1/data-audit/provider-policy, never inside record payloads.attestation— signature is optional on staging. For production verification, sendpayload_hash,signature,key_id,key_url, andsigned_at_utc.app.legal_entity— legal counterparty information alongside the app/platform name.file.behavior— non-PII capture/upload behavior signals.file_specific.base.motion— shared motion signals that apply across media types.timestamps.payment_credited_at— optional UTC timestamp for when the provider finalized and credited the contributor’s payment. Omit when nothing was credited (or the payout was forfeited); if payment lands after registration, send it as a full-state metadata update. Stored-only.
Version constants
Version constants
Stored on durable and index rows. Future changes should dual-write/dual-read versions during transition rather than mutating history.
Write API
Set active provider policies
Use this endpoint when the provider publishes a new Terms of Service or Privacy Policy. Trace stores the active version, hash, and URI so the frontend can link to the policy and stats can compare records against it.Policy rows are audit-only — not indexed or aggregated. Record payloads still carry
contributor.consent.*; those record-level references are what /stats, scoped-group summaries, and policy-hash search use.tos and privacy_policy objects are required, each with version, hash, and uri; effective_at is optional. The hash accepts the same forms as asset hashes (sha256:<64-hex>, bare hex, 0x<64-hex>, 0x1220<64-hex>) and is stored canonically as sha256:<64-lowercase-hex>. The request body is capped at 1 MiB.
Read current and historical policy documents:
Register records
Use this endpoint for initial backlog and live registration batches. The provider sends its stablesource_record_id; the DATA Foundation generates the data_id and returns the mapping.
initial_metadata_json follows the recommended top-level shape above (abbreviated here):
initial_metadata_root should be the provider’s deterministic non-zero hash of the canonical Trace Schema v1.0 metadata JSON. The DATA Foundation stores this value as submitted; hash verification against initial_metadata_json is not enforced yet.
Successful response (202 when new items were enqueued, 200 when every item was a duplicate, 409 when any item conflicted):
data_id is the canonical ID for future metadata updates and reads. Re-sending the same X-Provider + source_record_id generates the same data_id.
Per-item statuses: an item already persisted with the exact same payload returns duplicate; the same source_record_id with different initial metadata returns conflict. Neither is (re-)enqueued, and other valid items in the batch are unaffected. An overlapping retry while the first request is still queued may return accepted — downstream ingestion stays idempotent.
If a caller already has DATA Foundation-assigned UUIDs, the lower-level POST /webhook/v1/data-audit/data-ids:batch endpoint exists. It requires data_id on every record and is not the recommended provider path.
Submit metadata updates
Use this endpoint for later corrections or mutable metadata changes.seq must be 1–100 per data_id (a hard cap of 100 updates per record, all environments). Changes that belong to the same provider-side revision can share one update and one seq.
metadata_root should be the provider’s deterministic non-zero hash of the canonical full updated Trace Schema v1.0 metadata JSON. prev_metadata_root must also be a non-zero root. The DATA Foundation stores these values as submitted; hash verification against metadata_json is not enforced yet.
Optional backlog file endpoints (NDJSON / CSV / TXT)
Optional backlog file endpoints (NDJSON / CSV / TXT)
The normal integration path is
application/json on /records:batch — that route accepts JSON only. For backlog tooling, these lower-level route variants also exist; data ID file routes require data_id in every record.- NDJSON requires
Content-Type: application/x-ndjsonandContent-Encoding: gzip. Each decompressed line is one JSON record with the same fields as the JSON endpoints. - CSV requires
Content-Type: text/csv.- Data ID columns:
data_id,source_record_id,initial_metadata_root,initial_metadata_json,occurred_at - Metadata update columns:
data_id,seq,prev_metadata_root,metadata_root,metadata_json,occurred_at initial_metadata_jsonandmetadata_jsonmust be valid JSON in a quoted CSV field and include one canonical content hash.
- Data ID columns:
- TXT requires
Content-Type: text/plainand accepts line-delimited JSON, a JSON array, or header-delimited comma/tab text using the same CSV columns.
Read API
Read endpoints are public audit views.provider is optional and acts as a narrowing filter.
Read model:
GET /data-ids/{data_id}returns the registration profile plus the latest raw metadata event — the exact payload of the highest stored sequence, not necessarily a diff. The response also carries a top-levelingested_at— the DATA Foundation-generated timestamp of the registration (seq 0) ingestion — omitted when unavailable.GET /data-ids/{data_id}/metadatasreturns the full append-only metadata history, including registration atseq: 0and later metadata updates.- Search, asset receipt lookup, and scoped-group summaries use the DATA Foundation’s normalized latest-state projection derived from those events. This projection powers fields such as MIME type, media category, KYC status, TOS/privacy versions, lifecycle status, and
tx_hash.
Get trace by data ID
List metadata history
tx_hash, initially as an empty string:
Search by indexed field
source_record_id, contributor.anon_id, collection_id, customer_id, and task_id; other fields match case-insensitively, and content-hash values are canonicalized before matching. Each match includes tx_hash (initially an empty string), and responses include next_cursor for pagination. limit defaults to 100 and is capped at 1000 — larger values are silently clamped, not rejected.
Typical flow: use /search to locate records by an exact field value, then /data-ids/{data_id} or /data-ids/{data_id}/metadatas for the canonical event payload used in audit verification. Use /stats for distribution counts, and /recent or /feed when the UI needs the newest records.
More examples:
Provider totals (/stats)
provider + contributor_anon_id values. Metadata updates change neither, but distributions and size totals follow the latest projection, so a full-state update can move a record between buckets. average_size_bytes is total_size_bytes / size_record_count, counting records with a positive size. shard_count is reserved (currently 0). The provider_* fields appear only with ?provider=, and active_* only when that provider has an active policy. App/platform fields are not stats scopes.
Recent ingestion feed
event_type, seq, data_id, source_record_id, asset_hash, occurred_at, and ingested_at. Use next_cursor to fetch older rows.
Recent registered records
next_cursor to fetch older rows.
Asset receipts by content hash
sha256:<64-hex>, plain <64-hex>, 0x<64-hex>, and 0x1220<64-hex>. The response returns the canonical sha256:<64-lowercase-hex> asset hash and all matched receipt rows.
Scoped groups
Scoped groups let the Trace frontend or a partner reviewer create a public snapshot over a review set. For provider review workflows, labs should submit the provider’ssource_record_id values; Trace computes the deterministic data_id, verifies every record exists, and only creates the group if the full set is valid. (Groups can also be created from content hashes via hashes / hashes_text — the two input kinds are mutually exclusive.) Creation is asynchronous: poll GET /scoped-groups/{group_id} until status is complete.
Create a group
Uploads for larger review sets
For larger source-record CSV/TXT inputs, request a presigned upload URL first:csv and txt. Upload the file bytes to the returned upload_url with the returned Content-Type header, then create the group with the returned upload_id.
provider in the create request only when the uploaded CSV has a provider column.
Read a group
GET /scoped-groups/{group_id} returns the group status and, once complete, aggregate metrics in summary. profile.status is one of pending, processing, complete, or failed; summary is only present once the group is complete.
/items returns one row per submitted source_record_id with the resolved data_id, asset_hash, and receipt summary. It is cursor-paginated and returns next_cursor when more rows exist. Both /items and /export.csv require the group to be complete and return 409 before that. For source-record groups, the CSV export columns are input_type,provider,source_record_id,data_id,asset_hash,status,mime_type,media_category,kyc_status,tos_version,privacy_policy_version,tx_hash; hash-manifest groups start with hash,status,data_id,... instead.
Limits and retry behavior
Retry guidance:
- Retry
502,503,504, network timeouts, and429with exponential backoff and jitter. (Only the public scoped-group create/upload endpoints emit429— 60 requests/minute per IP; webhook batch endpoints never do.) - Do not retry validation/auth
4xxuntil the request is fixed. - Keep
data_id, request body, andX-Batch-Idstable across retries. - Use
X-Ingestion-Source: backlogonly for backlog work; omit it for live work. - The write path is idempotent for the same
data_id, event key, and event hash. - If the same
data_idand event key are retried with different metadata, the record is treated as a conflict and rejected.
Validation rules
Delivery semantics
- Delivery is at least once, so duplicate submissions may occur.
- Duplicate submissions of the same event are treated idempotently.
- Same
data_id, same metadata sequence, and different event content is treated as a conflict. - Metadata updates may arrive before the initial data ID registration.
- Audit data is durable and does not expire.
- Event hashes are computed from canonicalized metadata JSON plus the event/hash/schema version fields.