# Data & Quantum — runtime 1.8

The public Site now has **07 Data / Quantum**. It controls your paired node; the Site is not the storage cluster or IBM account. CERN catalogue access requires no API key. Quantum hardware requires your IBM Cloud API key, an exact Quantum instance CRN and account access to an operational backend. Existing Kaska/n8n scientific orchestration remains separate from Apache Kafka event transport.

## Start

1. Update the runtime package. Run `Start.command` on Mac or `Start-server.sh` on Linux. Docker supplies the network-free worker. No cloud job runs at startup.
2. For IBM, install optional pinned dependencies on the node:
   `.venv/bin/python -m pip install -r requirements-quantum.txt`
   (Python 3.11+ runtime; Qiskit 2.5.2, qiskit-ibm-runtime 0.50.0.)
3. Issue a short-lived code on the node, using the exact UI origin:
   ```sh
   .venv/bin/python researchctl.py pair --origin https://zeroexternal-physics-runtime.collab-panterra.chatgpt.site --scopes read,research,review,export,connections
   ```
4. Pair through your HTTPS gateway, or use the local UI and a code for `http://127.0.0.1:8765`. Open **CERN / Quantum**. The new `connections` scope explicitly permits credentials/configuration, bounded downloads, agent assignment, metadata sharing and individually confirmed QPU submissions. Do not issue it to an independent resident or n8n grant. Sessions without it cannot operate these routes.
5. Secrets entered in this tab are sent to the paired node over the same authenticated gateway and cleared from the form. They are not put in browser storage, URLs, task text or graphs. The node's private secret store and its administrator remain trusted.

## CERN: start small, preserve the manifest

Use record **5203** for a 2,639,302-byte educational CSV containing 20,000 selected dimuon events. It is 2011 data, published in 2019; it is not a RAW detector stream or all CERN data.

Record **30563** is an example CMS NanoAOD manifest: the retrieved snapshot has 82 files / 140,061,944,181 bytes. Current metadata is read from `https://opendata.cern.ch/api/records/{id}` each time you discover a new snapshot. New snapshots pin a metadata hash, file checksums, version IDs, license, software context and source URL. Direct files and expanded `_file_indices` are parsed; unresolved indices are rejected. Available records vary; do not assume every catalogue entry is downloadable immediately.

Discovery is limited to 2 MiB and 10,000 resolved files per manifest. Missing declared file totals remain **unconfirmed**, not complete. File locations are returned in pages of 50 through `resources/files`; the UI offers the eight smallest candidates. Download uses fixed CERN HTTPS routes, no redirects/proxies, a wall deadline, declared byte size and source checksum. A transfer adds a local SHA-256. Cold/on-demand files must be staged through CERN; this client does not request tape staging automatically.

A normal node downloads **whole files up to 64 MiB**, within a **256 MiB sample cache**. The owner clicks a specific file. Failed receipts are retained and never retried automatically. Restart removes incomplete temporary files; completion requires a verified checksum and journal commit. No endpoint accepts arbitrary data URLs, shell commands or user Python.

For a verified CSV, **Profile first 10,000 rows** launches the fixed `dataset_csv` recipe in Docker with `network none`, read-only filesystem, non-root UID, bounded CPU/memory/time and exactly one read-only sample file. It computes finite counts, missing/nonfinite counts, means, standard deviations and extrema for at most 16 columns. The input hash, image/environment, row limit and result enter the graph as unreviewed evidence. This is descriptive profiling, not event selection, reconstruction or a new physical result. An isolated CSV parser can fail safely on malformed input. ROOT/CMSSW parsing is **not included** in this initial worker.

## Petabyte planning and languages

Python orchestrates manifest discovery, ROOT/uproot analysis recipes and Qiskit circuits. CMS reconstruction may require a specified C++/CMSSW release and GlobalTag. ROOT is a container format, while RAW/AOD/MiniAOD/NanoAOD describe data products. ksqlDB/SQL can process prepared Kafka streams; it does not replace ROOT or reconstruct detector events.

The plan divides the exact manifest into groups of whole files, pins the private research question, and records proposed worker count, network rate and storage. Oversized indivisible files are counted; no hidden ROOT-cluster splitting occurs. Its time estimate is only `bytes × 8 / bandwidth`, without staging, contention, compute or retries. Export contains the full manifest and private question: review it before sharing.

There is **no installed PB executor, storage-account connector, resumable XRootD mover or Dask/Spark cluster provisioner** in this package. A full run needs your selected dataset, available storage, network budget and a validated external executor. Do not submit a PB download to a laptop or publish raw event contents to Kafka. Scale by manifest/file (or validated ROOT-cluster) partitions, checkpointed external workers and compact verified aggregates. A QPU receives a small compiled circuit derived from a separately validated method, not petabytes of detector data.

## Agent work and Kaska

Creating a plan makes ordered work packages: catalogue → method → partition → analysis → optional quantum assessment → independent review. The owner assigns each package to a live resident admitted through the existing Habitat protocol. Assignment and reports use revision fencing. A lost agent can be replaced by reassigning the working step; its old revision no longer authorizes a report. Revoked/expired residents cannot read or write work. Prerequisite completion means only **a report was received**, not that its claim was accepted.

The **Create task in Kaska** button sends the research question, plan hash and work descriptions to your configured board, with a durable creation receipt. Ambiguous replies require inspecting the board; no silent replay. It does not claim to provision a new resident or wake an unconfigured external agent. Existing Kaska supervisor/n8n actions still run their already-supported fixed scientific recipes. An independent Kaska agent can poll the work API below after enrollment and assignment.

Use `resident-client.py` with its private credential file; all bodies below are JSON via stdin, never a master token:

```sh
printf '%s' '{}' | python3 resident-client.py work --token-file resident.json
printf '%s' '{"plan_id":"plan_ID","offset":0}' | python3 resident-client.py work-files --token-file resident.json
printf '%s' '{"work_id":"work_ID","expected_revision":2}' | python3 resident-client.py work-claim --token-file resident.json
printf '%s' '{"work_id":"work_ID","expected_revision":3,"request_key":"unique-report-key","summary":"Findings, evidence and limitations…"}' | python3 resident-client.py work-report --token-file resident.json
```

`work` returns assigned descriptions, source metadata, relevant sample profiles and prerequisite report bodies. `work-files` pages only manifests belonging to an assigned plan. The external agent supplies its own model/reasoning; the client is a transport. It has no provider configuration, download authority, arbitrary network, QPU-spend grant or claim-acceptance route. Reports are `unverified_agent_report` graph objects. No autonomous human recruitment or payment is triggered.

## Apache Kafka

Configure an operator-approved **Kafka REST Proxy v2 origin** and topic. A `host:9092` binary bootstrap address is not accepted as an HTTP API. Non-loopback origins need HTTPS and their exact hostname in `ZEROEXTERNAL_ALLOWED_ENDPOINT_HOSTS`; private IPs may be used only for this configured gateway. Authentication currently supports an optional Bearer token (no Confluent Cloud Basic/SASL adapter).

Click **Send plan metadata to Kafka**. The outbox binds its event ID, topic and connection revision before POST `/topics/{topic}`. It sends only IDs, hashes, size and partition count—not secrets, the question, event data or file contents. HTTP 200 alone is insufficient: each record needs a valid partition/offset acknowledgement with no error code. Uncertain delivery remains visible, with no automatic retry. This is not an exactly-once guarantee; consumers deduplicate by `event_id`. A configuration change cannot redirect an existing attempt.

ksqlDB can define a JSON stream over the topic after the operator provisions it:

```sql
CREATE STREAM ZEROEXTERNAL_PLANS (
  event_id VARCHAR, type VARCHAR, plan_id VARCHAR,
  dataset_id VARCHAR, partition_count BIGINT, total_bytes BIGINT
) WITH (KAFKA_TOPIC='zeroexternal.research', VALUE_FORMAT='JSON');
```

The event is a notification; it does not itself authorize a worker to download data or spend compute. The sample emits a plan-created event on explicit request, not every runtime status automatically.

## IBM Quantum

Supported provider: **IBM Quantum Platform**, us-east/eu-de, REST version `2026-04-15`. Supply the actual `crn:v1:bluemix:public:quantum-computing:us-east:a/ACCOUNT:INSTANCE::` (or eu-de). No saved ambient IBM account is loaded. A trusted read-only SDK subprocess lists devices, transpiles a fixed measured Bell circuit to the selected device ISA, and decodes results. The fixed provider adapter has network access; it is not the network-free physics cell. Arbitrary imported circuits and other providers are not supported yet.

1. Save account and budget (16–4096 shots; 1–10 submission attempts per UTC day across configurations).
2. Check access and select an operational backend.
3. Prepare the Bell circuit. The plan pins account configuration, backend, shots, SDK versions, QASM hash and ideal classical probabilities: 00/11 each 1/2. This step submits no hardware job.
4. Confirm that specific hashed plan and click **Send to QPU**. The trusted gateway obtains a short-lived IBM IAM token and sends exactly one REST job-creation POST; it never uses SDK auto-retries for submission. A durable receipt and unique correlation tag exist first. The 60-second QPU execution budget is not a price cap or queue timeout. Your provider plan controls billing.
5. Poll status explicitly; DONE yields actual counts and a `QuantumObservation`. Bell computational-basis counts alone cannot certify entanglement or quantum advantage and do not analyze CERN data.
6. If a response is lost, status is `submission_unknown`; never create another submission for the same plan. Find the workload in IBM Console by its correlation tag and provide its ID for verification of tag/backend before result retrieval. Cancellation sends a provider request and remains pending until status confirms it. Stopping your local runtime does not cancel a remote QPU job.

API key replacement creates a new configuration revision, invalidating old preparations. Existing jobs then require the original connection or IBM Console for inspection/cancellation. This is deliberate isolation, not seamless key rotation.

## Verification and scope

The pinned Qiskit SDK was installed and tested locally with 2/27/156-qubit synthetic backend targets and a real local StatevectorSampler result. This validates SDK/export compatibility, not access to a remote IBM QPU.

The implementation includes offline contracts with real public CERN metadata snapshots and synthetic transfer/provider responses. IBM paid submission is not tested with your account, Kafka is not connected to your broker, and Docker requires the installed local worker. Never label mocked provider counts as hardware results. The developer environment could not resolve CERN directly through the no-proxy egress path; live metadata had been inspected separately. Run the bounded 5203 workflow on your node as the first connectivity check.

Primary interfaces checked 2026-09-25:
- https://opendata.cern.ch/api/records/5203
- https://opendata.cern.ch/api/records/30563
- https://opendata.cern.ch/docs/colddata
- https://opendata.cern.ch/docs/cms-getting-started-nanoaod
- https://quantum.cloud.ibm.com/docs/en/guides/sampler-rest-api
- https://quantum.cloud.ibm.com/docs/en/api/qiskit-runtime-rest
- https://docs.confluent.io/platform/current/kafka-rest/api.html