Metadata-Version: 2.5
Name: abstract-client
Version: 0.3.1
Summary: Abstract's robot API from Python: action chunks from served robot models, sync and asyncio.
Project-URL: Homepage, https://abstract.inc
Project-URL: Documentation, https://docs.abstract.inc
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx2<3,>=2.12
Provides-Extra: quic
Requires-Dist: aioquic<2,>=1.3; extra == 'quic'
Description-Content-Type: text/markdown

# abstract.client

Abstract's robot API from Python. A `Client` holds your key and its connections and is safe to
share between threads; each `Policy` is one episode of one served robot model doing one task, one
action chunk per `predict()`.

```sh
uv add abstract-client
```

### Quickstart (0.3.0)

```python
import abstract

checkpoint = "allenai/molmoact2"
embodiment = "yam"
client = abstract.client(ABSTRACT_API_KEY, checkpoint, embodiment)

client.set_observation(frames, observation)
action = client.get_action()
client.reset_episode()
client.close()
```

`ABSTRACT_API_KEY` is your key, `frames` maps each camera to JPEG bytes, and `observation`
holds `prompt` (the task) and `state` (measured alongside the frames). The YAM camera and
state fields are shown below. The returned `Chunk` contains targets, never hardware motion;
reset ends the model's episode without repositioning the robot.

`abstract.client()` checks the account's unmetered model catalogue before configuring a
`RobotClient`. The checkpoint must be available and its `robot` must match `embodiment`.
An optional URDF layer converts poses to another arm's joint geometry; it does not establish
policy transfer. Omit `api_key` to use the environment or saved
CLI credentials; connection options such as `base_url` and `timeout` pass through to the
existing client. Context managers and the lower-level `abstract.Client` interface remain supported.

The `abstract.client(...)` factory has existed since 0.2.2. Version 0.3.0 moves the full SDK
into `abstract`, including the lower-level clients and policy types. Publish that version before
advertising the renamed advanced interfaces. `import atoms`, the 0.2 name, still resolves in 0.3
with a `DeprecationWarning`: its top-level names only, `AtomsClient` being `abstract.RobotClient`.

This is all a robot with its own stack needs. The `abstract` command and the robot station are
in `abstract-atoms`, built on it (`apps/atoms`).

```python
import abstract

client = abstract.Client()  # the key: api_key=, else ABSTRACT_API_KEY, else `abstract login`'s file

with client.connect("allenai/molmoact2", "fold the towel") as policy:
    frames = {"overhead": top_jpeg, "wrist_l": left_jpeg, "wrist_r": right_jpeg}
    while not done:
        state = {
            "left_eef_xyz": [x, y, z],  # metres, in the left arm's base frame
            "left_eef_rot6": rot6,  # the rotation matrix's first two columns
            "left_grip": [g],  # 0 closed, 1 open
            "right_eef_xyz": [x, y, z],
            "right_eef_rot6": rot6,
            "right_grip": [g],
        }
        chunk = policy.predict(frames, state)
        # Row k is a pose for t + k * chunk.dt; your arm's inverse kinematics turns each into
        # joint angles. The arm held still while predict() ran, so every row runs; a loop that
        # keeps the arm moving meanwhile skips the rows it ran, as the RobotClient loop below.
        for left, right in zip(chunk.left, chunk.right):
            robot.command(left, right)
        # The next call sends policy.frames_needed JPEGs per camera, oldest first.
        frames = latest_frames(policy.frames_needed)
# Leaving the block ends the episode now; the server would after 60 s idle.
```

A stateful model takes the video since its last observation: after the first call,
send `policy.frames_needed` frames per camera.

### Native robot checkpoints

These interfaces require `abstract-client` 0.2.0 or later.

`client.get_models()` is the availability source; an SDK adapter does not mean a checkpoint
is enabled on the account. These native contracts are separate from the YAM example:

| Model                             | Observation                                                                                                                                                                                                    | Returned commands                                                                                                                                      |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `verne/v0.5`                      | The YAM example's poses and cameras, one frame per camera and then four; not trained on the YAM                                                                                                                | 18 pose rows per arm at 30 Hz                                                                                                                          |
| `runway/praxis-1` | The YAM example's poses and cameras at 16:10, one frame per camera and then 21: one at the end of each row just run. Capture them yourself and call `policy.predict()`; `RobotClient`'s window also counts the pause between chunks. | 21 pose rows per arm at 15 Hz |
| `xiaomi/xiaomi-robotics-1`        | RoboDojo ARX X5 world-frame poses, normalized grippers (0 closed, 1 open), and `left_joint_position` / `right_joint_position` with six measured radians each. Cameras retain `overhead`, `wrist_l`, `wrist_r`. | `chunk.left` / `chunk.right`: 30 end-effector targets at 25 Hz, in the same RoboDojo world frame.                                                      |
| `black-forest-labs/flux-3-action` | DROID Franka `joint_position` (seven radians), `gripper_position` (one closed fraction: 0 open, 1 closed), and cameras `exterior_image_0_left`, `exterior_image_1_left`, `wrist_image_left`.                   | `chunk.joints`: 32 rows of seven absolute joint targets plus gripper at 15 Hz; `chunk.action_space == "joint_position"`. `left` and `right` are empty. |

Use `chunk.row_count` for either command format. `elapsed_rows`, sync/async policies, retries
and the `RobotClient` observation window work with both. Policies sharing one observation
window must use the same state and camera contract. The bimanual station does not drive these native contracts.

### `abstract.RobotClient`: Dream Labs' DreamClient on this API

`abstract.RobotClient` keeps that window for you and paces the requests, with the calls of Dream Labs'
`DreamClient` (Apache-2.0; see `NOTICE`): bind the key, choose the model, push every camera tick,
pull a chunk when it says so.

```python
from concurrent.futures import ThreadPoolExecutor

robot = abstract.RobotClient()  # the key, as abstract.Client finds it
robot.policy({"model": "allenai/molmoact2"})
with robot, ThreadPoolExecutor(1) as worker:  # the request runs while the robot moves
    chunk, row, pending = None, 0, None
    ran = ran_at_ask = 0  # rows the robot has run, and how many when the request went out
    while not done:
        frames = {"video.overhead": top, "video.wrist_l": left, "video.wrist_r": right}
        robot.set_observation(frames, {"prompt": "fold the towel", "state": read_state()})
        rows_left = len(chunk.left) - row if chunk else 0
        if pending is None and robot.refill_due(row, rows_left):
            pending, ran_at_ask = worker.submit(robot.get_action), ran
        if pending is not None and pending.done():
            chunk, pending = pending.result(), None
            # Skip the rows the robot ran while this chunk was computed (at most those already
            # due, chunk.elapsed_rows). A robot that held still meanwhile starts at row 0.
            row = min(chunk.elapsed_rows, ran - ran_at_ask)
        if chunk and row < len(chunk.left):
            robot.command(chunk.left[row], chunk.right[row])
            row, ran = row + 1, ran + 1
        wait_for_next_tick()
```

#### Several models on one robot

DreamClient runs one model. `RobotClient` runs as many as you open on the same camera ticks, each
with its own episode, frames (a stateful model takes several per camera, MolmoAct2 one), pacing and timing.
One drives the arms; the others answer the same observations, to compare them.

```python
robot.policies(
    {"molmo": {"model": "allenai/molmoact2"}, "pi": {"model": "physical-intelligence/pi0.5"}}
)
robot.set_observation(frames, observation)  # one tick, every model
answers = robot.get_actions()  # every model at once: {name: Chunk, or the error it raised}
drive = robot["molmo"]  # one model: get_action(), refill_due(), last, mode, last_timing
```

| DreamClient                                                                         | abstract.RobotClient                                                                                                                   |
| ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `DreamClient(api_key)`                                                              | `RobotClient(api_key)`, or `RobotClient(client=...)`                                                                                |
| `get_models()`                                                                      | `get_models()`: served models and your fine-tunes, with their rates                                                                 |
| `policy({"checkpoint": ...})`                                                       | `policy({"model": ...})`; `execution_horizon`, `action_hz`, `sync_above_ms`, `timeout_ms` as in DreamClient                         |
| `set_observation(frames, observation, timestamp=)`                                  | the same: `video.<camera>` frames, `prompt` and `state` (a dict or `state.<name>` keys)                                             |
| `get_action()` returns joint chunks                                                 | `get_action()` returns a `Chunk` in the selected model's action space: poses in `left` / `right`, native joint commands in `joints` |
| `reset_episode()`, `refill_due()`, `close()`                                        | the same                                                                                                                            |
| `mode`, `execution_horizon`, `action_hz`, `last_timing`, `checkpoint`, `session_id` | the same, `model` for `checkpoint`                                                                                                  |
| `obs_window`, `obs_fps`                                                             | none: each answer says how many frames the next observation carries                                                                 |
| `DreamSlowNetworkWarning`                                                           | `abstract.SlowNetworkWarning`                                                                                                          |
| one model per client                                                                | `policies({name: config})`, `get_actions()`, `robot[name]`                                                                          |

### Optional URDF FK and IK (0.3.0)

The solver is not in this package: install `abstract-atoms[ik]` after release, or
`uv pip install -e ./apps/atoms/client -e './apps/atoms[ik]'` from this checkout. Configure each physical arm's URDF, joint order, grasp point and frames:

```python
from abstract_robots.ik import RobotIK

ik = RobotIK.from_file("robot.json")
client = abstract.client(checkpoint="allenai/molmoact2", embodiment="yam", ik=ik)
client.set_joint_observation(frames, measured_joints, grippers, prompt="fold the towel")
chunk = client.get_action()
# Measure again after inference; solving is local and commands no hardware. The arm held still
# during this call; one that kept moving passes start= the rows it ran (at most elapsed_rows).
joints = client.to_joints(chunk, current_measured_joints)
```

See [the configuration and driver guide](../robots/IK.md) for full examples, limits, geometry checks and
validation scope. Different arm geometries/joint counts are supported locally. Model embodiment
checks remain; correct IK does not prove policy transfer. The Python station uses the same solver
through `--ik robot.json`, with `observe_joints()` / `command_joints()` drivers.

DreamClient's seam smoothing (`commanded`) is not implemented. Dataset and fine-tuning commands
and the station's Live View remain in `abstract-atoms`.

`abstract.AsyncClient` is native asyncio with the same arguments, rules and calls
(`await policy.predict(...)`). Cancelling a call cancels its request, but the server may already
be running it, so the next `predict()` asks for that answer first, as after a lost connection.

Connections are HTTPX2's (Pydantic's maintained HTTPX, as the OpenAI and Anthropic SDKs use).
Proxies come from `HTTPS_PROXY` and `NO_PROXY`, and TLS is checked against the operating system's
trust store, so a CA your IT installed works as is. For a private CA, a client certificate or a
proxy in code, configure HTTPX2 and pass it in; the client you pass stays yours to close:

```python
import ssl

import httpx2

tls = ssl.create_default_context(cafile="lab-ca.pem")
tls.load_cert_chain("robot.pem", "robot.key")
client = abstract.Client(http_client=httpx2.Client(verify=tls, proxy="http://proxy.lab:3128"))
```

### Direct sessions and the time profile

```python
with client.connect("allenai/molmoact2", "fold the towel") as policy:
    chunk = policy.predict(frames, state)
    chunk.stages  # this chunk's milliseconds: total, client, network, server and its parts
print(policy.profile)  # the episode: each stage's p50, p95 and max, and what happened on the way
```

Where the API names an engine for a model, an episode's chunks go straight to that engine and
no longer cross the API. There is nothing to set: the API holds a block of chunks on your
account and signs a ticket for it, and each `predict()` posts to the engine under that ticket.
The same `Chunk`, the same errors, an observation resent under its own id and never run or
billed twice. Your one choice stays `transport`:

- `"http"`: HTTPS to the engine's host.
- `"quic"`: the engine's own QUIC socket, which the client reaches by UDP hole punching. An
  episode's first chunk goes over HTTPS; the punch then runs in the background, and the chunks
  that follow travel over QUIC. Where a network blocks UDP or a NAT does not allow the punch,
  or the connection is later lost, the episode keeps to HTTPS without a word.

What happens on its own:

- The next ticket is asked for in the background while chunks remain, so `predict()` does not
  wait on it.
- An engine that fails twice in a row, or stops answering for 10 s after it has answered, is
  left: the API names the next one and the episode continues there, and when none is left it
  continues through the API. A model without episode memory moves unnoticed, its `step`
  restarting where the new engine is; a model that keeps its episode on the GPU raises
  `SessionExpiredError`, as it does when its replica is lost.
- A model the API names no engine for runs through the API as ever, and so do FLUX, whose
  content filters the API applies to every chunk, and your fine-tunes. A client asks about a
  model once, not once an episode. `AsyncClient` always goes through the API.

What to know:

- A direct episode's observations and answers are not stored with your account's trajectories.
- The engine's hosts end in `modal.run`: a firewall that allows `api.abstract.inc` alone sends
  the first episode back through the API after about four seconds, and the next ones at once.
- To punch, the client asks a public STUN server (`stun.cloudflare.com`, then
  `stun.l.google.com`) what address its UDP socket is seen at. That request carries nothing
  but the question; the server sees the machine's public address.

`policy.profile` is kept for every episode. `client` is the time to build the request, `server`
the engine's from receipt to answer (`decode`, `queue`, `prefill` and `denoise` are its parts,
`queue` counted from receipt), and `network` the rest: both ways, and the API when the chunk
crossed it. `policy.profile.summary()` returns the same as a dict, with which way each chunk
travelled (`api`, `https`, `quic`) and `events`: a ticket opened or renewed, a QUIC path punched
or given up for HTTPS, a move to another engine, a return to the API.

### Transports

`transport` chooses how the calls travel. `"http"` is the default and what everything above
describes. `"quic"` sends the same calls to the same routes over HTTP/3, on one QUIC connection
that every policy on the client shares; it is opt-in and needs the `quic` extra:

```sh
uv add 'abstract-client[quic]'
```

```python
client = abstract.Client(transport="quic")  # or ABSTRACT_TRANSPORT=quic; every client takes it
```

The API sees the same request either way, so chunks are billed the same and a call that gets no
answer is sent again under the same rules. What differs is the network's side:

- QUIC needs outbound UDP to the API on port 443. Where a network blocks it, calls fail with
  `APIConnectionError` or `APITimeoutError`; nothing switches to `"http"` on its own.
- No proxy can carry it. On a machine set to use one (`HTTPS_PROXY`, `ALL_PROXY` or the system's)
  the client raises `ConfigError` rather than go around the proxy, and `http_client=` cannot be
  combined with it.
- TLS is checked against certifi's CA bundle, not the operating system's trust store: a CA your
  IT installed is not trusted, and there is no setting for a private CA or a client certificate.
- A connection that hears nothing for 20 s is closed and the next call opens another. While a
  call waits for its answer the client pings every 5 s, so a model's cold start does not end it.

It runs aioquic's QUIC stack as it comes, with no tuning, and has not been measured against
`"http"`: choose it to compare the two on your own network, not on a promise that it is faster.
`client.transport` says which one a client uses.

`connect(model, task, scaling={"seed": 7})` (or `scaling` in a `RobotClient` config) sends the
API's `scaling` with every observation, so a seedable model replays its noise; the engine's reports
(`scaling`, `cache`, `context`) come back in `chunk.raw`.

Each chunk is billed and can wake a GPU. A call that gets no answer is sent again with the same
bytes and `observation_id`, so the server answers it from its record and never bills it twice.
Refusals the server marks retryable are sent again after its `Retry-After`, up to `max_retries`;
`timeout` bounds a whole call, resends included, and `connect_timeout` opening a connection.

Every error is an `abstract.AbstractError`. A refusal is an `abstract.APIStatusError` subclass named
for its status (`AuthenticationError`, `PaymentRequiredError`, `RateLimitError`, ...) with the
server's `code`, `retryable`, `retry_after` and `request_id`; a refused or answered
`get_action` also carries its reply's `trace_id`, which a replay keeps. `APIConnectionError` and
`APITimeoutError` mean no answer came: the next `predict()` asks for it first. The `abstract.client`
logger reports resends; it never logs the key.

Python 3.10 or later; its one dependency is `httpx2` (`aioquic` with the `quic` extra). Tests: `pytest apps/atoms` (the contract
tests need Bun and the engine's packages: `uv run --project apps/engine pytest apps/atoms`).

Version 0.3 replaces `import atoms` with `import abstract`; the buffered harness is
`abstract.RobotClient` (previously `AtomsClient`). `abstract.client(...)` is the recommended
entry point. The Python station uses this same implementation; there is no separate YAM runner.
