Skip to main content

Trace Provider Data Audit — Integration & API Reference

The single source of truth for the Trace data audit API: provider onboarding, auth, the Trace Schema, write and read endpoints, search, stats, scoped groups, limits, and validation.
The API and Trace Schema are provider-agnostic — every provider integrates through the same endpoints, headers, and schema. Examples use Kled, a live provider; substitute your own provider identity wherever Kled appears.

Environments

Staging is where new provider integrations are onboarded and tested. Production is deployed but may not yet have provider data — check with the DATA Foundation team before sending production traffic.

Getting access

Write access is gated. Before integrating, contact the DATA Foundation team to be onboarded: we whitelist your provider identity, issue staging and production API keys, and assign the X-Provider value your write requests must use. Until that is done, write requests are rejected. Read, search, and scoped-group endpoints are public audit views and do not require a key.

Auth and provider scope

Every write request must include:
  • The API key must belong to your provider and match X-Provider; use the key for the environment you’re targeting.
  • For backlog ingestion, add X-Ingestion-Source: backlog. For live ingestion, omit the header — explicit live is not accepted.
  • Read APIs are keyed by the global data_id; provider is returned as a field and doubles as an optional filter.
Read, search, and scoped-group endpoints are public audit views. Do not put enterprise-only or sensitive fields into the public Trace payload. If a provider-only or partner-only read tier is ever needed, that is a separate API/product decision.

Ingestion flow

1

Send a batch

The provider client POSTs to a webhook batch endpoint.
2

Asynchronous processing

Accepted items are enqueued and processed asynchronously by the DATA Foundation.
3

Read back

Records become available through the read and search endpoints.
A records:batch response always includes per-item statuses:
  • 202 Accepted — at least one item was accepted for asynchronous processing and no item conflicted.
  • 200 OK — every item was a duplicate, so no new async work was enqueued.
  • 409 Conflict — at least one item conflicted; any accepted items in the same response were still enqueued.
Inspect item statuses, and retry transient request failures with the same payload and the same X-Batch-Id.

Record lifecycle

Each record’s history is an append-only event log that Trace renders as a timeline. Each step maps to exactly one value; a step appears on the timeline when its value is present. Steps 1–3 come from the registration payload’s metadata; steps 4–7 are audit events. Send each milestone as its own metadata update — one milestone, one timeline event. The root occurred_at on each batch item is required, provider-supplied, and dates that specific event — registration time on the DataRegistered event, update time on each MetadataUpdated event. Trace surfaces the Registered lifecycle step with its own ingested_at — the moment the DATA Foundation accepted the record, returned on per-record reads. timestamps.originated_at inside the attested payload carries the record’s origin moment: set it equal to the registration’s root occurred_at, and keep it fixed on later updates (each update’s root occurred_at dates the update itself). Records registered before this field was renamed carry the same value under the legacy name timestamps.occurred_at — readers should accept both. The only values the DATA Foundation adds are ingested_at and tx_hash.

Trace Schema v1.0

Populate the standardized Trace Schema fields directly and preserve your full original public payload under provider_payload. The normalized fields are the portable contract the frontend and audit flows use; nothing provider-specific is lost.
  • Use schema_version: trace-v1.0 in initial_metadata_json and metadata_json. The value is not validated — a missing or unparseable schema_version defaults to trace-v1.0, and whatever value is sent is folded into the event hash.
  • The DATA Foundation canonicalizes metadata JSON before computing internal event hashes, so object key order does not affect idempotency or conflict detection.
  • Do not send a transaction hash in write payloads. The DATA Foundation owns tx_hash and returns it on read responses for registration, metadata update, and search result rows — as an empty string until the DATA Foundation fills it after broadcast.

Canonical content hash (required)

Initial registrations and metadata updates must include one canonical content hash. Accepted fields are asset.hash, content_hash, file.content_sha256, or file.hashes.sha256; all normalize to sha256:<64-lowercase-hex>. If more than one alias is present, they must represent the same hash.

Searchable fields

Current exact-match searchable fields: The only searchable hash is the canonical content SHA-256 — perceptual hashes (phash64, dhash64, ahash64, keyframe_phashes) and md5 are stored but not indexed. The same goes for app.platform_name, app.legal_entity, file.behavior.*, file_specific.base.motion.*, and timestamps.payment_credited_at: send them when useful — stored fields can be indexed later without resending old data. Use /stats for distributions and provider as an optional query scope.

Field guidance

  • source_record_id — provider-owned stable public media ID (for example kmf_...). The service trims surrounding whitespace, preserves case, rejects control characters, and accepts up to 512 bytes.
  • contributor.anon_id — provider-owned public anonymized contributor ID.
  • asset.collection_id, asset.customer_id, asset.task_id — optional provider-assigned grouping IDs: a collection or batch of records, the customer or campaign they were produced for, and the task they belong to. Each is exact-match searchable, and matching is case-sensitive like source_record_id. Send them at the top level (as in the shape above) or inside provider_payload — both are read. Omit any that don’t apply.
  • contributor.kyc_status — recommended values: verified, pending, failed, unverified.
  • contributor.kyc_country — ISO 3166-1 alpha-2 country code from KYC, if available. Country only; no address or GPS-derived country.
  • contributor.tax_status — recommended values: submitted, not_submitted, not_applicable, unknown. For a boolean like tax_form_on_file, map true to submitted and false to not_submitted.
  • contributor.account_verification_status — recommended values: verified, pending, failed, unverified.
  • contributor.consent.tos_* and contributor.consent.privacy_policy_* — the exact policy version, hash, and URI the contributor accepted for this record. Provider-level active policies are set separately through PUT /webhook/v1/data-audit/provider-policy, never inside record payloads.
  • attestation — signature is optional on staging. For production verification, send payload_hash, signature, key_id, key_url, and signed_at_utc.
  • app.legal_entity — legal counterparty information alongside the app/platform name.
  • file.behavior — non-PII capture/upload behavior signals.
  • file_specific.base.motion — shared motion signals that apply across media types.
  • timestamps.payment_credited_at — optional UTC timestamp for when the provider finalized and credited the contributor’s payment. Omit when nothing was credited (or the payout was forfeited); if payment lands after registration, send it as a full-state metadata update. Stored-only.
Stored on durable and index rows. Future changes should dual-write/dual-read versions during transition rather than mutating history.

Write API

Set active provider policies

Use this endpoint when the provider publishes a new Terms of Service or Privacy Policy. Trace stores the active version, hash, and URI so the frontend can link to the policy and stats can compare records against it.
Policy rows are audit-only — not indexed or aggregated. Record payloads still carry contributor.consent.*; those record-level references are what /stats, scoped-group summaries, and policy-hash search use.
Both the tos and privacy_policy objects are required, each with version, hash, and uri; effective_at is optional. The hash accepts the same forms as asset hashes (sha256:<64-hex>, bare hex, 0x<64-hex>, 0x1220<64-hex>) and is stored canonically as sha256:<64-lowercase-hex>. The request body is capped at 1 MiB. Read current and historical policy documents:

Register records

Use this endpoint for initial backlog and live registration batches. The provider sends its stable source_record_id; the DATA Foundation generates the data_id and returns the mapping.
The request body is a JSON array. initial_metadata_json follows the recommended top-level shape above (abbreviated here):
initial_metadata_root should be the provider’s deterministic non-zero hash of the canonical Trace Schema v1.0 metadata JSON. The DATA Foundation stores this value as submitted; hash verification against initial_metadata_json is not enforced yet. Successful response (202 when new items were enqueued, 200 when every item was a duplicate, 409 when any item conflicted):
The returned data_id is the canonical ID for future metadata updates and reads. Re-sending the same X-Provider + source_record_id generates the same data_id. Per-item statuses: an item already persisted with the exact same payload returns duplicate; the same source_record_id with different initial metadata returns conflict. Neither is (re-)enqueued, and other valid items in the batch are unaffected. An overlapping retry while the first request is still queued may return accepted — downstream ingestion stays idempotent. If a caller already has DATA Foundation-assigned UUIDs, the lower-level POST /webhook/v1/data-audit/data-ids:batch endpoint exists. It requires data_id on every record and is not the recommended provider path.

Submit metadata updates

Use this endpoint for later corrections or mutable metadata changes. seq must be 1100 per data_id (a hard cap of 100 updates per record, all environments). Changes that belong to the same provider-side revision can share one update and one seq.
metadata_json must include a canonical content hash and be the full latest Trace metadata state — not a diff or patch. If only KYC changed, still include the unchanged file, app, consent, and provider payload fields, so each event verifies against metadata_root and the latest state can be rebuilt without merge rules.
Request body is a JSON array:
metadata_root should be the provider’s deterministic non-zero hash of the canonical full updated Trace Schema v1.0 metadata JSON. prev_metadata_root must also be a non-zero root. The DATA Foundation stores these values as submitted; hash verification against metadata_json is not enforced yet.
The normal integration path is application/json on /records:batch — that route accepts JSON only. For backlog tooling, these lower-level route variants also exist; data ID file routes require data_id in every record.
  • NDJSON requires Content-Type: application/x-ndjson and Content-Encoding: gzip. Each decompressed line is one JSON record with the same fields as the JSON endpoints.
  • CSV requires Content-Type: text/csv.
    • Data ID columns: data_id,source_record_id,initial_metadata_root,initial_metadata_json,occurred_at
    • Metadata update columns: data_id,seq,prev_metadata_root,metadata_root,metadata_json,occurred_at
    • initial_metadata_json and metadata_json must be valid JSON in a quoted CSV field and include one canonical content hash.
  • TXT requires Content-Type: text/plain and accepts line-delimited JSON, a JSON array, or header-delimited comma/tab text using the same CSV columns.

Read API

Read endpoints are public audit views. provider is optional and acts as a narrowing filter. Read model:
  • GET /data-ids/{data_id} returns the registration profile plus the latest raw metadata event — the exact payload of the highest stored sequence, not necessarily a diff. The response also carries a top-level ingested_at — the DATA Foundation-generated timestamp of the registration (seq 0) ingestion — omitted when unavailable.
  • GET /data-ids/{data_id}/metadatas returns the full append-only metadata history, including registration at seq: 0 and later metadata updates.
  • Search, asset receipt lookup, and scoped-group summaries use the DATA Foundation’s normalized latest-state projection derived from those events. This projection powers fields such as MIME type, media category, KYC status, TOS/privacy versions, lifecycle status, and tx_hash.

Get trace by data ID

List metadata history

Each metadata row includes tx_hash, initially as an empty string:

Search by indexed field

Search is exact-match on the searchable fields. Matching is case-sensitive for source_record_id, contributor.anon_id, collection_id, customer_id, and task_id; other fields match case-insensitively, and content-hash values are canonicalized before matching. Each match includes tx_hash (initially an empty string), and responses include next_cursor for pagination. limit defaults to 100 and is capped at 1000 — larger values are silently clamped, not rejected. Typical flow: use /search to locate records by an exact field value, then /data-ids/{data_id} or /data-ids/{data_id}/metadatas for the canonical event payload used in audit verification. Use /stats for distribution counts, and /recent or /feed when the UI needs the newest records. More examples:

Provider totals (/stats)

Record totals count each registered record once; contributor totals count distinct provider + contributor_anon_id values. Metadata updates change neither, but distributions and size totals follow the latest projection, so a full-state update can move a record between buckets. average_size_bytes is total_size_bytes / size_record_count, counting records with a positive size. shard_count is reserved (currently 0). The provider_* fields appear only with ?provider=, and active_* only when that provider has an active policy. App/platform fields are not stats scopes.

Recent ingestion feed

The feed returns recent audit events — both registrations and metadata updates — newest first by ingestion time. Rows include event_type, seq, data_id, source_record_id, asset_hash, occurred_at, and ingested_at. Use next_cursor to fetch older rows.

Recent registered records

Returns registered receipt rows only, newest first by ingestion time. Use next_cursor to fetch older rows.

Asset receipts by content hash

Accepted hash forms are sha256:<64-hex>, plain <64-hex>, 0x<64-hex>, and 0x1220<64-hex>. The response returns the canonical sha256:<64-lowercase-hex> asset hash and all matched receipt rows.

Scoped groups

Scoped groups let the Trace frontend or a partner reviewer create a public snapshot over a review set. For provider review workflows, labs should submit the provider’s source_record_id values; Trace computes the deterministic data_id, verifies every record exists, and only creates the group if the full set is valid. (Groups can also be created from content hashes via hashes / hashes_text — the two input kinds are mutually exclusive.) Creation is asynchronous: poll GET /scoped-groups/{group_id} until status is complete.

Create a group

Source-record groups are all-or-nothing:

Uploads for larger review sets

For larger source-record CSV/TXT inputs, request a presigned upload URL first:
Supported formats are csv and txt. Upload the file bytes to the returned upload_url with the returned Content-Type header, then create the group with the returned upload_id.
Omit provider in the create request only when the uploaded CSV has a provider column.

Read a group

GET /scoped-groups/{group_id} returns the group status and, once complete, aggregate metrics in summary. profile.status is one of pending, processing, complete, or failed; summary is only present once the group is complete.
/items returns one row per submitted source_record_id with the resolved data_id, asset_hash, and receipt summary. It is cursor-paginated and returns next_cursor when more rows exist. Both /items and /export.csv require the group to be complete and return 409 before that. For source-record groups, the CSV export columns are input_type,provider,source_record_id,data_id,asset_hash,status,mime_type,media_category,kyc_status,tos_version,privacy_policy_version,tx_hash; hash-manifest groups start with hash,status,data_id,... instead.

Limits and retry behavior

Retry guidance:
  • Retry 502, 503, 504, network timeouts, and 429 with exponential backoff and jitter. (Only the public scoped-group create/upload endpoints emit 429 — 60 requests/minute per IP; webhook batch endpoints never do.)
  • Do not retry validation/auth 4xx until the request is fixed.
  • Keep data_id, request body, and X-Batch-Id stable across retries.
  • Use X-Ingestion-Source: backlog only for backlog work; omit it for live work.
  • The write path is idempotent for the same data_id, event key, and event hash.
  • If the same data_id and event key are retried with different metadata, the record is treated as a conflict and rejected.

Validation rules

Delivery semantics

  • Delivery is at least once, so duplicate submissions may occur.
  • Duplicate submissions of the same event are treated idempotently.
  • Same data_id, same metadata sequence, and different event content is treated as a conflict.
  • Metadata updates may arrive before the initial data ID registration.
  • Audit data is durable and does not expire.
  • Event hashes are computed from canonicalized metadata JSON plus the event/hash/schema version fields.