Unify the documentation layout and conventions

README.md

@@ -4,36 +4,38 @@
 
 # harp
 
+Python interface to [Harp](https://harp-tech.org/articles/what-is-harp.html) devices and their recorded data, implementing the [Harp binary protocol](https://harp-tech.org/protocol/BinaryProtocol-8bit.html).
+
+Harp is a standard for asynchronous real-time data acquisition and experimental control in neuroscience. Every command and event is hardware timestamped on the device. Devices sharing a clock line continuously self-synchronize, so events across a rig sit on one clock and need no post-hoc alignment.
+
 This project includes four main packages:
 
- - **harp-protocol**: Provides the core protocol definitions and utilities for the Harp protocol. See [Protocol API Documentation](https://harp-tech.org/python/api/protocol) for details.
+ - **harp-protocol**: Implements the Harp binary protocol in Python, with registers, messages, and payload parsing. See [Protocol API Documentation](https://harp-tech.org/python/api/protocol) for details.
 
- - **harp-serial**: Implements serial communication functionalities for generic Harp devices. See [Serial API Documentation](https://harp-tech.org/python/api/serial) for more information.
+ - **harp-device**: Implements the transport-agnostic `Device` interface and the core register set. See [Device API Documentation](https://harp-tech.org/python/api/device) for details.
 
- - **harp-device**: Implements the transport-agnostic `Device` interface, the common register map, and the shared registers and enums. See [Device API Documentation](https://harp-tech.org/python/api/device) for details.
+ - **harp-serial**: Connects to a `Device` over a serial COM or tty port. See [Serial API Documentation](https://harp-tech.org/python/api/serial) for details.
 
- - **harp-data**: Parses register binary dumps into pandas DataFrames. See [Data API Documentation](https://harp-tech.org/python/api/data) for more information.
+ - **harp-data**: Reads logged register files into pandas DataFrames. See [Data API Documentation](https://harp-tech.org/python/api/data) for details.
 
 ## Installation
 
-All packages are published to PyPI. The `harp` package is a metadata package with no code of its own. It depends on the four packages above, so it is the easiest way to get everything:
+All packages are published to PyPI. The `harp` package is a metapackage with no code of its own. It depends on the four packages above, so it is the easiest way to get everything:
 
 ```sh
 pip install harp
 ```
 
-```sh
-uv add harp
-```
+`uv add` substitutes for `pip install` throughout.
 
-To install only part of the stack, for example when parsing offline data dumps with no need for serial I/O, install the individual packages. Each one only pulls in what it actually depends on:
+To install only part of the stack, for example when reading recorded data with no need for serial I/O, install the individual packages. Each one only pulls in what it actually depends on:
 
-| Package | Provides | Depends on |
-| --- | --- | --- |
-| `harp-protocol` | Core protocol types: registers, messages, payload parsing | none |
-| `harp-device` | Transport-agnostic `Device` class, common register map | `harp-protocol` |
-| `harp-serial` | Serial COM or tty transport for `Device` | `harp-protocol`, `harp-device` |
-| `harp-data` | Parse register binary dumps into pandas DataFrames | `harp-protocol` |
+| Package | Depends on |
+| --- | --- |
+| `harp-protocol` | none |
+| `harp-device` | `harp-protocol` |
+| `harp-serial` | `harp-protocol`, `harp-device` |
+| `harp-data` | `harp-protocol` |
 
 ```sh
 pip install harp-protocol
@@ -56,7 +58,7 @@ from harp.device import behavior, core
 
 # Use "COMx" on Windows, "/dev/ttyUSBx" on Linux.
 with serial.open_device(behavior, port="COM3") as device:
-    print(device.read(core.WhoAmI).payload)         # a common register
+    print(device.read(core.WhoAmI).payload)         # a core register
     print(device.read(behavior.AnalogData).payload) # a device register
     device.write(
         core.OperationControl,
@@ -96,15 +98,15 @@ from pathlib import Path
 from harp.device import schema
 
 behavior = schema.create_device_module(Path("device.yml").read_bytes())
-AnalogData = behavior.AnalogData                 # registers are reached by name
+AnalogData = behavior.AnalogData                 # registers are accessed by name
 assert behavior.REGISTER_MAP[44] is AnalogData   # or by address
 ```
 
-See the [Examples](https://harp-tech.org/python/examples/) for the full walkthroughs, including subscribing to device events and working with custom interface-type converters.
+See the examples in the documentation for the full walkthroughs, including subscribing to device events and working with custom interface-type converters.
 
 ## Contributing
 
-harp is a [uv workspace](https://docs.astral.sh/uv/concepts/workspaces/): every package under `src/packages/` is its own distribution, plus the root `harp` metadata package. Bug reports and contributions are welcome, so please open an issue or pull request.
+harp is a [uv workspace](https://docs.astral.sh/uv/concepts/workspaces/): every package under `src/packages/` is its own distribution, plus the root `harp` metapackage. Bug reports and contributions are welcome, so please open an issue or pull request.
 
 Clone the repository and install everything with the `dev` dependency group: all workspace packages, editable, plus test and lint tooling.
 
@@ -124,7 +126,7 @@ uv run pytest --cov harp     # tests
 
 To add a new package, place it under `src/packages/<name>/` with its own `pyproject.toml` and add it to `[tool.uv.sources]` in the root `pyproject.toml`. If it should ship as part of `harp`, add it to the dependencies of the root package as well.
 
-## Building the documentation
+## Build the documentation
 
 Install the docs dependency group and run mkdocs through uv:
 

docs/articles/device-modules.md

@@ -0,0 +1,29 @@
+# Choose a device module
+
+A **device module** is what describes a Harp device to the library. It holds the register classes for that device at module level, a `REGISTER_MAP` keyed by address, and the payload and enum classes referenced by those registers. It is what `Device` needs to talk to hardware and what [`open_dataset`](../api/data.md) needs to read a recorded session.
+
+A device module comes from one of two places. A generated device package is a real Python module on disk, produced from a `device.yml` by the Harp code generators and installed as a dependency. A runtime module is built in memory by [`create_device_module`](../api/device.md), from a `device.yml` read at the moment it is needed.
+
+Both are the same kind of thing. Registers are accessed the same way in either case, by name as `behavior.AnalogData` or by address as `behavior.REGISTER_MAP[44]`. The names agree because both derive from the same schema under the same naming convention. Analysis code written against one lines up name for name against the other.
+
+The choice is therefore not about what the registers can do. It is about where the definitions come from, and what that costs.
+
+## Benefits of a generated package
+
+**Static typing and autocomplete.** The module exists on disk before the program runs, so an editor offers its register names and a type checker verifies them. A runtime module is built while the program runs, so neither can see it. It is not in `sys.modules` either, so it has to be bound to a name rather than imported.
+
+**A version that can be pinned.** A generated package is an ordinary dependency, so it can be pinned in a lock file and every install resolves the same register definitions. A runtime module is only as stable as its `device.yml`, so the same analysis code can see different field names once that file changes.
+
+**Converters for its own custom types.** A generated package includes the converters for any custom `interfaceType` declared by its schema. A runtime module has to be given them through `converters=`.
+
+## Benefits of a runtime module
+
+**No build step.** A `device.yml`, including one read straight off a device, becomes a working module in a single call. There is nothing to generate, install, or keep in step with the schema.
+
+**Coverage for any device.** No published package is needed, so unreleased, custom and one-off schemas work immediately. It is also what makes a recorded session readable when the only description of the device is the `device.yml` saved beside it.
+
+## Which to use
+
+For a widely used device with a published package, use the package. Better editor support, static typing and a version that can be pinned are worth a dependency for code that has to be maintained.
+
+Use `create_device_module` when no package exists, when the schema is still moving, or when a recorded session has to be read with nothing but the `device.yml` saved alongside it. See [Registers from a schema](../examples/registers-from-schema.md) for a worked example.

docs/articles/migrating_from_harp_python.md → docs/articles/harp-python-migration.md

@@ -1,44 +1,46 @@
-# Migrating from harp-python
+# Migrate from harp-python
 
-`harp-data` is the successor to `harp-python` for reading Harp binary data files into
-pandas DataFrames. The core concepts of device schemas, register maps and binary files
-are unchanged, but the API has been reorganized to separate data reading from device
-communication.
+`harp-data` is the successor to `harp-python` for reading Harp binary data files into pandas DataFrames. The core concepts of device schemas, register maps and binary files are unchanged, but the API has been reorganized to separate data reading from device communication.
 
 This guide covers the three workflows most users relied on in `harp-python`.
 
-## Swap the package
+## Installation
 
-Replace the old dependency:
+`harp-python` was a single distribution. Reading data now needs only `harp-data`, since serial transport and the device client are separate packages.
 
-```sh title="Before"
+**Before**
+
+```sh
 pip install harp-python
 ```
 
-```sh title="After"
+**After**
+
+```sh
 pip install harp-data
 ```
 
-For the full toolkit of serial transport, device client and data reading, install the
-umbrella package instead:
+For the full toolkit of serial transport, device client and data reading, install the metapackage instead:
 
 ```sh
 pip install harp
 ```
 
-## Loading a device schema at runtime
+## Load a device schema at runtime
 
-In `harp-python`, `harp.create_reader()` accepted a dataset folder and handled
-schema loading internally. `open_dataset` is the direct replacement, and it finds the
-`device.yml` inside the folder automatically:
+In `harp-python`, `harp.create_reader()` accepted a dataset folder and handled schema loading internally. `open_dataset` is the direct replacement, and it finds the `device.yml` inside the folder automatically.
 
-```python title="Before"
+**Before**
+
+```python
 import harp
 
 reader = harp.create_reader("session.harp")
 ```
 
-```python title="After"
+**After**
+
+```python
 from harp.data import open_dataset
 
 reader = open_dataset("session.harp")
@@ -50,35 +52,31 @@ If the schema lives outside the data folder, pass it explicitly:
 reader = open_dataset("session.harp", schema="/path/to/device.yml")
 ```
 
-### Finding out what a session holds
+### List the session contents
 
-`reader.contents` maps the name of every register with data in the folder to its address,
-in address order, which is the quickest way to see what was recorded:
+`reader.contents` maps the name of every register with data in the folder to its address, in address order, which is the quickest way to see what was recorded:
 
 ```python
 reader = open_dataset("session.harp")
 print(reader.contents)   # {'WhoAmI': 0, 'DigitalInputState': 32, ...}
 ```
 
-The reader also holds the compiled device module at `reader.device_module`, so a register
-class can be reached without keeping a separate variable:
+The reader also holds the compiled device module at `reader.device_module`, so a register class can be accessed without keeping a separate variable:
 
 ```python
 df = reader.read(reader.device_module.AnalogData)
 ```
 
 !!! note
-    The old `harp.read_schema()` had no direct equivalent that needed calling separately.
-    `open_dataset` handles schema loading in one step, matching the convenience
-    of the original API.
+    The old `harp.read_schema()` had no direct equivalent that needed calling separately. `open_dataset` handles schema loading in one step, matching the convenience of the original API.
+
+## Read a single register
 
-## Reading a single register
+The old API allowed reads through attribute access on the reader. The new API inverts this, so `reader.read()` takes the register class, its name, or its address as the argument.
 
-The old API allowed reads through attribute access on the reader. The new
-API inverts this, so `reader.read()` takes the register class, its name or
-its address as the argument.
+**Before**
 
-```python title="Before"
+```python
 # by attribute name
 df = reader.AnalogData.read()
 
@@ -89,7 +87,9 @@ df = reader.registers["AnalogData"].read()
 df = reader.registers[44].read()
 ```
 
-```python title="After"
+**After**
+
+```python
 # by register class (accessed through the reader)
 df = reader.read(reader.device_module.AnalogData)
 
@@ -100,55 +100,53 @@ df = reader.read("AnalogData")
 df = reader.read(44)
 ```
 
-Names resolve against the device address space rather than the module namespace, so a
-common register such as `reader.read("WhoAmI")` works even though a device module does
-not name it.
+Names resolve against the device address space rather than the module namespace, so access to a core register such as `reader.read("WhoAmI")` works even though a device module does not name it.
 
 ### Absolute timestamps
 
-The `epoch` parameter keeps its name and moves from `create_reader` to `open_dataset`.
-`harp-python` also allowed it per read. `harp-data` sets it once for the dataset, so
-every register is read on the same clock.
+The `epoch` parameter keeps its name and moves from `create_reader` to `open_dataset`. `harp-python` also allowed it per read. `harp-data` sets it once for the dataset, so every register is read on the same clock.
+
+**Before**
 
-```python title="Before"
+```python
 reader = harp.create_reader("session.harp", epoch=harp.REFERENCE_EPOCH)
 df = reader.AnalogData.read()
 ```
 
-```python title="After"
+**After**
+
+```python
 from harp.data import REFERENCE_EPOCH
 
 reader = open_dataset("session.harp", epoch=REFERENCE_EPOCH)
 df = reader.read(reader.device_module.AnalogData)
 ```
 
-### Reading the whole session at once
+### Read the whole session at once
 
-There is no `read_all()`. Whole-session loading is a comprehension over `contents`,
-which keeps the choice of what to load with the caller:
+There is no `read_all()`. Whole-session loading is a comprehension over `contents`, which keeps the choice of what to load with the caller:
 
 ```python
 everything = {name: reader.read(name) for name in reader.contents}
 ```
 
 ### Bitmask registers lose their per-flag columns by default
 
-This is the change most likely to break working code. `harp-python` always expanded a
-bitmask register into one boolean column per flag, so a script could select a flag by
-name. `harp-data` returns a single integer column instead, and expands the flags only
-when asked:
+This is the change most likely to break working code. `harp-python` always expanded a bitmask register into one boolean column per flag, so a script could select a flag by name. `harp-data` returns a single integer column instead, and expands the flags only when asked.
 
-```python title="Before"
+**Before**
+
+```python
 led = reader.DigitalOutputSet.read()["GP15"]
 ```
 
-```python title="After"
+**After**
+
+```python
 led = reader.read("DigitalOutputSet", demux_bit_masks=True)["GP15"]
 ```
 
-Group masks need no such flag. `harp-python` mapped each value to its member name, and
-`harp-data` decodes them by default, as a `pd.Categorical` rather than plain strings, so
-a comparison against a string still reads naturally.
+Group masks need no such flag. `harp-python` mapped each value to its member name, and `harp-data` decodes them by default, as a `pd.Categorical` rather than plain strings, so a comparison against a string still works.
 
 ### Parameter reference
 
@@ -163,36 +161,35 @@ a comparison against a string still reads naturally.
 
 ## Schemaless read
 
-For a raw `.bin` file with no schema, or a quick look at the data, the `read()`
-function works the same as before. Only the import path changes:
+For a raw `.bin` file with no schema, or a quick look at the data, the `read()` function works the same as before. Only the import path changes.
+
+**Before**
 
-```python title="Before"
+```python
 import harp
 
 df = harp.read("Behavior_44.bin")
 df = harp.read("Behavior_44.bin", keep_type=True)
 ```
 
-```python title="After"
+**After**
+
+```python
 from harp.data import read
 
 df = read("Behavior_44.bin")
 df = read("Behavior_44.bin", keep_type=True)
 ```
 
-Both functions infer the payload type and element count from the frame, so no register
-metadata is needed. The new one assumes timestamped data, which is what a device sends,
-and takes `time_index=False` for the rare buffer that is not. It also takes `epoch` per
-call, since a single file has no dataset to set one on.
+Both functions infer the payload type and element count from the frame, so no register metadata is needed. The new one assumes timestamped data, which is what a device sends, and takes `time_index=False` for the rare buffer that is not. It also takes `epoch` per call, since a single file has no dataset to set one on.
+
+## Static device packages
 
-## Going further: static device packages
+Loading a YAML at runtime is convenient. However, a generated device package gives the same interface without parsing a schema at startup, and its registers resolve under IDE autocompletion and a type checker.
 
-Loading a YAML at runtime is convenient, but for production workflows, or where IDE
-autocompletion and type checking matter, a generated device package gives the same
-interface without parsing a schema at startup.
+It also validates what it opens. The package declares its own `WHO_AM_I`, which the reader checks against the `device.yml` in the folder. A session recorded from a different device then fails on construction rather than decoding the files against the wrong register map. A module built from that same folder cannot catch this, since it agrees with the folder by definition.
 
-Such a package is an ordinary Python module under the `harp.device` namespace. Import
-it, pass it to `open_dataset`, and the rest of the API is identical:
+Such a package is an ordinary Python module under the `harp.device` namespace. Import it, pass it to `open_dataset`, and the rest of the API is identical:
 
 ```python
 from harp.device import behavior
@@ -204,7 +201,4 @@ reader = open_dataset("session.harp", behavior)
 df = reader.read(behavior.AnalogData)
 ```
 
-A generated module starts up faster and resolves under a type checker, which a module
-built from a schema at runtime cannot. See
-[Generating Registers from a Schema](../examples/create_device_module/create_device_module.md)
-for how device modules are structured.
+A generated module starts up faster and resolves under a type checker, which a module built from a schema at runtime cannot. See [Registers from a schema](../examples/registers-from-schema.md) for how device modules are structured.