wtf.petrichor.drinkme.measurement
Schema Diff
+1 -1
Compatibility Analysis
Backward Compatible
No changes detected.
1
1
{
2
2
"$type": "com.atproto.lexicon.schema",
3
3
"defs": {
4
4
"compression": {
5
5
"properties": {
6
6
"bitsPerWeight": {
7
7
"description": "Weight-weighted bits per weight as a decimal string: total encoded payload bits of the packed population over its total number of weights. The population is the Linears the codec packed — the compressed arm's swapped tensors, exactly the trunk `drinkme pack` writes — and nothing else: the raw remainder (embeddings, norms, biases, sub-1024 projections) and an MTP head are excluded, so this is not a whole-model storage multiplier.",
8
8
"type": "string"
9
9
},
10
10
"meanTensorBitsPerWeight": {
11
11
"description": "The unweighted mean of per-tensor bits per weight over the same packed population, as a decimal string — what a pack's meta.json calls meanBpw (/v1/models carries it under this name, as a number). A small projection and a large head count equally here; use bitsPerWeight for the model's bytes.",
12
12
"type": "string"
13
13
},
14
14
"packedBytes": {
15
15
"description": "The pack's bytes on disk (the npz sum), read off the loaded pack's meta.json — the number `drinkme pack` wrote. Present iff the compressed arm loaded a pack from disk (every mlx record; a torch record run with --pack-dir).",
16
16
"type": "integer"
17
17
},
18
18
"profile": {
19
19
"description": "The compression profile the compressed arm ran — `sip` (the default, faster) or `gulp` (smaller): exponent tiers, literal sign/mantissa, 1024-weight blocks. Read off the loaded arm or the loaded pack's meta.json `profile`, never typed into the bench; a pack is one profile, and a raw-fallback tensor beside the profile's tensors does not change it (docs/pack-format.md).",
20
20
"enum": [
21
21
"sip",
22
22
"gulp"
23
23
],
24
24
"type": "string"
25
25
},
26
26
"residentBytes": {
27
27
"description": "The pack's measured device footprint plus the raw remainder the loader streams, read off the loaded pack's meta.json — the number `drinkme pack` wrote. Present iff packedBytes is.",
28
28
"type": "integer"
29
29
}
30
30
},
31
31
"required": [
32
32
"profile",
33
33
"bitsPerWeight"
34
34
],
35
35
"type": "object"
36
36
},
37
37
"environment": {
38
38
"description": "The hardware the arms ran on, class-first: `deviceClass` is how owners self-identify ('Apple M3 Max', 'NVIDIA GeForce RTX 4090', or on a Linux APU the CPU brand string, 'AMD RYZEN AI MAX+ 395 w/ Radeon 8060S', because the GPU name identifies nothing retail), `memoryBytes` and `memoryKind` the pool the model lives in, `cpuInfo` and `gpuInfo` the detectors' verbatim strings so a normalization mistake is repairable without re-runs, `engine` and `platform` the runtime that ran the arms and the compute platform it ran on. The device's bandwidth is measured, not quoted: read_gb_s in metrics[].",
39
39
"properties": {
40
40
"cpuInfo": {
41
41
"description": "Verbatim.",
42
42
"maxLength": 512,
43
43
"type": "string"
44
44
},
45
45
"deviceClass": {
46
46
"description": "Normalized class string.",
47
47
"maxLength": 256,
48
48
"type": "string"
49
49
},
50
50
"engine": {
51
51
"description": "What ran the arms, one grammar, slash-separated segments each starting with its noun so the string is greppable by segment. After the version-carrying segments (drinkme, then the runtime), every kernel segment is `<noun> <impl>[ <version>]`, where <impl> names the implementation that ran or is `none`, and an impl that carries a version carries it in every segment that names it: `drinkme <semver> / torch <version>[+<accel tag>] / deltanet <fla <ver> | torch | none> / conv <fla <ver> | torch | none> / narrow-gemv <triton | torch> / stock-gemv <mv | linear | triton> / raw-gemv <twin | stock>[ / decode <eager | cudagraph>]` for the torch runtime (the runtime version is torch.__version__ verbatim, accelerator tag included; deltanet is the DeltaNet decode recurrence kernel every arm ran — fla's fused Triton kernel with fla's version, the torch recurrence, or none for a model without DeltaNet layers; conv is the prefill convolution kernel every arm ran — fla's time-major Triton kernel with fla's version, the torch conv1d fallback, or none, which must agree with deltanet; narrow-gemv is what ran the raw Linears under the codec's row threshold — triton for drinkme's Triton GEMV, torch for F.linear under DRINKME_NARROW_GEMV=0; stock-gemv is the one-row call of the raw Linears at or above it, every Linear of the stock arm — mv for torch.mv with the Lt BLAS library preferred for the call (hipBLASLt on ROCm), linear for F.linear on PyTorch's default library, triton for the twin arm's Triton GEMV over the raw BF16 weight; raw-gemv is the one-row call of the raw Linears at or above it that the compressed and twin arms hold (a tied lm_head) — twin for the twin arm's Triton GEMV over the raw BF16 weight, stock for the stock-gemv call; decode is the step path the decode figures ran on — cudagraph when every arm replayed a captured CUDA graph, eager otherwise; a record without it decoded eager), and `drinkme <semver> / mlx <version>` for the mlx runtime (environment.platform says metal). The kernel segments are read off the routing record the arms agreed on — arms that routed differently write no record — so one string names the runtime for every number in the record. Within a major version, new segments may be appended to the end of either form; existing segments do not change meaning. E.g. 'drinkme 1.0.0 / torch 2.12.0+rocm7.1 / deltanet fla 0.5.2 / conv fla 0.5.2 / narrow-gemv triton / stock-gemv mv / raw-gemv twin'.",
52
52
"maxLength": 256,
53
53
"type": "string"
54
54
},
55
55
"gpuInfo": {
56
56
"description": "Verbatim.",
57
57
"maxLength": 512,
58
58
"type": "string"
59
59
},
60
60
"memoryBytes": {
61
61
"description": "Capacity of the memory the model lives in, in bytes — the pool `memoryKind` names, as the detector read it.",
62
62
"type": "integer"
63
63
},
64
64
"memoryKind": {
65
65
"description": "`vram`: a discrete accelerator's own memory pool; `unified`: one pool shared with the CPU (Apple silicon, AMD APUs, GB10/Jetson).",
66
66
"enum": [
67
67
"vram",
68
68
"unified"
69
69
],
70
70
"type": "string"
71
71
},
72
72
"os": {
73
73
"maxLength": 256,
74
74
"type": "string"
75
75
},
76
76
"platform": {
77
77
"description": "The compute platform that executed the decode: `cuda` or `rocm` (the torch runtime), `metal` (the mlx runtime).",
78
78
"enum": [
79
79
"cuda",
80
80
"rocm",
81
81
"metal"
82
82
],
83
83
"type": "string"
84
84
}
85
85
},
86
86
"required": [
87
87
"deviceClass",
88
88
"memoryBytes",
89
89
"memoryKind"
90
90
],
91
91
"type": "object"
92
92
},
93
93
"main": {
94
94
"description": "One benchmark observation — a point on the curve — published to the participant's own repo; the DID that owns the repo is the identity. The record is self-contained: the hardware, the model, the compression and the measurements are inlined, so a consumer reads it without joining anything. The per-tensor bitwise round trip is a write-time gate in bench, never a field: a record whose packed tensors did not reconstruct their bf16 bytes exactly is not written.",
95
95
"key": "tid",
96
96
"record": {
97
97
"properties": {
98
98
"compression": {
99
99
"ref": "#compression",
100
100
"type": "ref"
101
101
},
102
102
"createdAt": {
103
103
"format": "datetime",
104
104
"type": "string"
105
105
},
106
106
"environment": {
107
107
"ref": "#environment",
108
108
"type": "ref"
109
109
},
110
110
"metrics": {
111
-
"description": "Self-describing metrics, each a name, a decimal-string value (the median when samples are present) and a unit. The canonical names are stock_decode_tok_s, compressed_decode_tok_s, stock_prefill_tok_s, compressed_prefill_tok_s, stock_ttft_s, compressed_ttft_s, read_gb_s, copy_gb_s, stock_weights_gb and compressed_weights_gb: decode is tok/s over the timed decode reps, prefill is tok/s for a single forward over a fixed 512-token prompt (raw.prefill_prompt_len carries the exact length), ttft is seconds of wall time from request start to the first generated token on that same prompt (one prefill forward plus one decode step), read_gb_s and copy_gb_s are a pure read and a STREAM-convention copy (read+write) of a probe buffer on the empty device, and stock_weights_gb/compressed_weights_gb are the arm's weight footprint. GB and GB/s are decimal — 10^9 bytes and 10^9 bytes per second; raw.bandwidth keeps the bytes. What the *_weights_gb metrics measure depends on environment.platform: cuda and rocm report the caching allocator's live allocation right after the arm loaded (torch.cuda.memory_allocated), metal reports the engine's own resident-bytes accounting (the weights it holds, packed or raw); neither is the process's total or a working set with KV. The stock_* names (stock_weights_gb among them) are present iff stock.outcome == 'measured'. On torch the twin arm (bench's diagnostic arm: the uncompressed BF16 weights through the compressed arm's own kernels) also publishes twin_decode_tok_s and twin_weights_gb when it ran; its outcome is in raw, and on mlx it publishes no metric. On torch each arm that ran also publishes <arm>_decode_read_gb (stock_decode_read_gb, twin_decode_read_gb, compressed_decode_read_gb): the bytes one decode step reads from that arm's weights (its Linears, packed or raw, and one row of the token embedding; not the vision tower or the other embedding rows, which count in *_weights_gb), the denominator of the arm's bandwidth bound read_gb_s / <arm>_decode_read_gb. When the checkpoint has an MTP head, bench on torch also times decode with speculation on, as drinkme serve runs it for such a model (--spec mtp, the head proposing): <arm>_spec_decode_tok_s_<prompt> for arm stock and compressed and prompt agent (an agent transcript), chat (a chat turn) and code (a code edit), each tok/s over 256 new tokens, the median of that prompt's timed reps after one untimed warm-up. Present iff the checkpoint has an MTP head, the arm was measured and the runtime speculates (mlx does not, so metal records carry none).",
111
+
"description": "Self-describing metrics, each a name, a decimal-string value (the median when samples are present) and a unit. The canonical names are stock_decode_tok_s, compressed_decode_tok_s, stock_prefill_tok_s, compressed_prefill_tok_s, stock_ttft_s, compressed_ttft_s, read_gb_s, copy_gb_s, stock_weights_gb and compressed_weights_gb: decode is tok/s over the timed decode reps, prefill is tok/s for a single forward over a fixed 512-token prompt (raw.prefill_prompt_len carries the exact length), ttft is seconds of wall time from request start to the first generated token on that same prompt (one prefill forward plus one decode step), read_gb_s and copy_gb_s are a pure read and a STREAM-convention copy (read+write) of a probe buffer on the empty device, and stock_weights_gb/compressed_weights_gb are the arm's weight footprint. GB and GB/s are decimal — 10^9 bytes and 10^9 bytes per second; raw.bandwidth keeps the bytes. What the *_weights_gb metrics measure depends on environment.platform: cuda and rocm report the caching allocator's live allocation right after the arm loaded (torch.cuda.memory_allocated), metal reports the engine's own resident-bytes accounting (the weights it holds, packed or raw); neither is the process's total or a working set with KV. The stock_* names (stock_weights_gb among them) are present iff stock.outcome == 'measured'. On torch the twin arm (bench's diagnostic arm: the uncompressed BF16 weights through the compressed arm's own kernels) also publishes twin_decode_tok_s and twin_weights_gb when it ran; its outcome is in raw, and on mlx it publishes no metric. Each arm that ran also publishes <arm>_decode_read_gb (stock_decode_read_gb, compressed_decode_read_gb, and on torch twin_decode_read_gb): the bytes one decode step reads from that arm's weights (its Linears, packed or raw, and one row of the token embedding; not the vision tower or the other embedding rows, which count in *_weights_gb), the denominator of the arm's bandwidth bound read_gb_s / <arm>_decode_read_gb. When the checkpoint has an MTP head, bench on torch also times decode with speculation on, as drinkme serve runs it for such a model (--spec mtp, the head proposing): <arm>_spec_decode_tok_s_<prompt> for arm stock and compressed and prompt agent (an agent transcript), chat (a chat turn) and code (a code edit), each tok/s over 256 new tokens, the median of that prompt's timed reps after one untimed warm-up. Present iff the checkpoint has an MTP head, the arm was measured and the runtime speculates (mlx does not, so metal records carry none).",
112
112
"items": {
113
113
"ref": "#metric",
114
114
"type": "ref"
115
115
},
116
116
"type": "array"
117
117
},
118
118
"model": {
119
119
"ref": "#model",
120
120
"type": "ref"
121
121
},
122
122
"raw": {
123
123
"description": "bench's own working detail (samples, bandwidth bytes, bytes per token, pack counts, the twin arm, device_source — how detect() named the device, detector — the detector's working notes (its weight budget in bytes, whether the memory kind was a guess, its evidence and version), the per-arm kernel routing as <arm>_routing — the record environment.engine's kernel segments were composed from, host_load — the host's 1-minute load average before and after each arm's timed passes and its CPU count, spec — the speculation pass: the proposer, tokens, reps and context, each prompt's sha256 and token count, and per arm and prompt the samples and the mean tokens each verify step accepted; or why it did not run (no MTP head, a runtime that does not speculate, --no-spec), and unpublishable — why a local record cannot be published); diagnostic, never aggregated.",
124
124
"type": "unknown"
125
125
},
126
126
"stock": {
127
127
"ref": "#stock",
128
128
"type": "ref"
129
129
},
130
130
"version": {
131
131
"description": "The drinkme semver that produced this record, e.g. '1.0.0'. The codec is the major (bug fixes = patch, kernel updates = minor, codec/format updates = major), so numbers are only comparable within a major.",
132
132
"type": "string"
133
133
}
134
134
},
135
135
"required": [
136
136
"model",
137
137
"compression",
138
138
"environment",
139
139
"metrics",
140
140
"stock",
141
141
"createdAt",
142
142
"version"
143
143
],
144
144
"type": "object"
145
145
},
146
146
"type": "record"
147
147
},
148
148
"metric": {
149
149
"properties": {
150
150
"name": {
151
151
"maxLength": 64,
152
152
"type": "string"
153
153
},
154
154
"samples": {
155
155
"description": "Every timed sample (bench runs three timed reps per arm after one untimed warm-up rep; raw.warmup_rep says whether the warm-up ran).",
156
156
"items": {
157
157
"type": "string"
158
158
},
159
159
"type": "array"
160
160
},
161
161
"unit": {
162
162
"maxLength": 32,
163
163
"type": "string"
164
164
},
165
165
"value": {
166
166
"description": "Decimal string (lexicon has no float); the median when samples are present.",
167
167
"type": "string"
168
168
}
169
169
},
170
170
"required": [
171
171
"name",
172
172
"value",
173
173
"unit"
174
174
],
175
175
"type": "object"
176
176
},
177
177
"model": {
178
178
"properties": {
179
179
"hfRepo": {
180
180
"maxLength": 256,
181
181
"type": "string"
182
182
},
183
183
"name": {
184
184
"maxLength": 128,
185
185
"type": "string"
186
186
},
187
187
"revision": {
188
188
"description": "The resolved HF commit the weights came from: the 40-hex sha of the one snapshot every arm and the tokenizer loaded, resolved by bench before any arm loaded (the pack's bound commit when the arm ran off a pack). Never the menu's coordinate, and never a local directory's structural digest — a checkpoint that is not a hub snapshot has no commit to name, so its record is written locally without this field and is not publishable (`drinkme publish` refuses it by name). Same model name at a different revision is a different cell.",
189
189
"maxLength": 64,
190
190
"type": "string"
191
191
}
192
192
},
193
193
"required": [
194
194
"name",
195
195
"hfRepo",
196
196
"revision"
197
197
],
198
198
"type": "object"
199
199
},
200
200
"stock": {
201
201
"description": "The stock BF16 arm's outcome, and the one fact that explains a non-measured one: the budget a skip was judged against, or the error a failed load raised.",
202
202
"properties": {
203
203
"budgetBytes": {
204
204
"description": "Present iff outcome == 'skipped_predicted_nonfit': the ceiling the fit check judged the stock arm's transient against, in bytes — physical memory on unified (environment.memoryBytes), the VRAM budget on discrete.",
205
205
"type": "integer"
206
206
},
207
207
"error": {
208
208
"description": "Present iff outcome == 'failed_load': the refusal or allocator message, verbatim, truncated to 2048 bytes.",
209
209
"maxLength": 2048,
210
210
"type": "string"
211
211
},
212
212
"outcome": {
213
213
"description": "`measured`: the pass ran and its stock_* metrics are present. `skipped_predicted_nonfit`: bench's stock-arm fit check (not the `drinkme check` verb) predicted a non-fit against budgetBytes and never attempted the load — a policy decision, not an observation; a different budget, loader or margin could fit. `failed_load`: the load was attempted and the live pre-allocation guard refused or the allocator raised out-of-memory. In both non-measured cases the stock_* metrics are omitted and the compressed arm still ran.",
214
214
"enum": [
215
215
"measured",
216
216
"skipped_predicted_nonfit",
217
217
"failed_load"
218
218
],
219
219
"type": "string"
220
220
}
221
221
},
222
222
"required": [
223
223
"outcome"
224
224
],
225
225
"type": "object"
226
226
}
227
227
},
228
228
"id": "wtf.petrichor.drinkme.measurement",
229
229
"lexicon": 1
230
230
}