Cs

Wait for the submitted CS fence, validate the signal sentinel, and copy the GPU-written row scores into `output`.

Retire the fence when the caller must abandon the result after submit.

Returns true when both PM4 submissions retired and the signal BO read back the expected sentinel value.

Open the canonical render node (`default_render_node`) and finish the full CS bring-up: context, BO list, IB / input / output / signal / shader buffers, all mapped into a low GPU VA range.

Open the given render node and bring up the full CS submission state.

Allocates an amdgpu context, creates GTT-backed BOs for the indirect buffer, input scratch (~2 MiB), output, signal and shader pages, maps each into a fixed low GPU VA so the kernel does not need to re-bind them per submission, uploads the gfx1201 PM4 kernels into the shader page, and creates a persistent BO list referencing all five BOs.

Tear down every kernel resource the `init*` paths created: destroy the BO list, free the amdgpu context, `munmap` each CPU mapping, and close the render-node file descriptor.

Round-trip one `u32` through the GPU as the simplest end-to-end gate: PM4 `COPY_DATA` from the input page to the output page, plus a `WRITE_DATA` of a per-submission sentinel into the signal page.

Alias for `copyU32` framed as the per-token decode pulse: prove the GPU produced a token by round-tripping its id through a real PM4 submission and fence wait.

Dispatch the gfx1201 top-2 argmax kernel and return the selected token.

Loads the argmax program into the compute SGPRs, packs the output VA, two ordered scores, and two token ids into `compute_user_data_2..7`, fires one workgroup, then waits on the signal sentinel before reading the kernel-chosen token from the output page.

Dispatch the gfx1201 ordered-score row-range argmax kernel.

Converts `scores` into sortable u32 keys, copies them into the shared input page, then lets the compute ring select the max row. The returned token id is absolute: `start_row + local_best`.

Dispatch the single-element gfx1201 final-RMS-norm kernel.

Stores `hidden0 * inv_rms * weight0` into `output_map[0]` via a real PM4 dispatch on the compute ring, with a signal sentinel verifying retirement.

Dispatch the gfx1201 row-range f32 dense matrix-vector kernel.

Copies the input vector and the row-major f32 weight block into the shared input page (64-byte aligned), records PM4 that points the kernel at the input/weights/output pages and the `rows`/`cols` arguments, and waits on the signal sentinel before reading `output`. This is the first row-oriented dense compute kernel the CS path runs; `cols` must be a multiple of 64 and `output` must hold at least `rows` elements.

Dispatch one F32 DMMV kernel over two packed row ranges sharing the same input vector.

This is used by the M1 SSM bridge to consume alpha and beta projection rows in one CS submission when both tensors are F32. The existing row-range shader sees one compact `rows_a + rows_b` matrix; output rows are written in that same order.

Dispatch the gfx1201 row-range Q4_0 matrix-vector kernel.

Copies the input vector and raw GGML Q4_0 rows into the shared input page, records PM4 for one serial workitem over `rows`, and reads back one f32 result per row. This intentionally validates real quantized model bytes through the native CS path while the full K-parallel DMMV kernel is still under construction.

Dispatch the wave-lane gfx1201 Q4_0 matrix-vector kernel for exactly 64 rows in parallel.

Stages the same source-format rows as `dmmvQ4_0RowRange`, but launches one wave64 workgroup where each lane computes one row. Intended for 64-row LM-head prefix/window ranges where the GPU row values participate in choosing the sampled token.

Dispatch the multi-workgroup TGID-delivery probe: `groups` workgroups, one workitem each, each storing its `workgroup_id_x` (SGPR s8) at `out[workgroup_id_x]`.

A correct dispatch yields `out[0..groups] = [0,1,...,groups-1]`; if TGID is not delivered, every workgroup collides at `out[0]`. Validates the grid-over-rows premise of the resident DMMV.

Dispatch the exhaustive TGID-slot probe (`tgid_dump`): `groups` workgroups scan candidate SGPRs s8..s19 and store each candidate value via a value-as-index trick.

Fills `out[0..out.len]` from the output BO; the caller decodes which candidate slot reads [0,1,...] across workgroups. `out[960]` reads 0x53475052 ("SGPR") iff the kernel ran.

Grid-over-rows Q4_0 dequant-matvec.

Stages `input` + `weights_q4_0` into the input BO, dispatches ceil(rows/64) workgroups (each picks its 64-row block from workgroup_id_x/ttmp9), and reads back `rows` results in ONE submit. This is the correctness path (weights staged per call); the perf path will instead point s[4:5] at a VRAM-resident weight BO.

One-shot check that a `size`-byte VRAM BO is CPU-mappable (large-BAR) and round-trips a host write/read.

Gates the resident-weight LM-head matvec, which only beats the CPU if the weight lives in VRAM (576 GB/s) rather than GTT (PCIe-bound). Leaks the probe BO (reclaimed at process exit).

Dispatch the wave64 Q4_0 single-row partial-sum kernel and reduce it on the host.

This lowers one real model row through a K-parallel CS kernel: lanes 0..31 each accumulate one column per Q4_0 block, the kernel writes 64 partial sums, and the host performs the final scalar reduction. The caller may pass `partials_out` to inspect the raw lane outputs.

Dispatch one or more 64-row wave-lane Q4_0 DMMV chunks in a single CS submission.

Each chunk runs the same source-format row-parallel shader as `dmmvQ4_0RowRangeParallel`; this helper stages a larger adjacent row window, emits one dispatch per 64-row chunk into the same IB, and waits once after the final release fence. The output slice receives all rows in order.

Stage a Q4_0 row window into the high end of the long-lived input BO.

The returned handle is valid until another caller overwrites the same resident region. Default M1 generation uses it for the first LM-head prefix proof, before any broad validation slices can reuse the scratch.

Dispatch a previously staged resident Q4_0 row window.

Only the activation vector is copied for this submission; PM4 points the shader at the resident weight VA in the shared input BO.

Submit one or more 64-row wave-lane Q4_0 chunks and return before waiting for the CS fence.

The shared input/output/signal maps and PM4 builder must not be reused until the returned dispatch is finished.

Dispatch two 64-row Q4_0 DMMV ranges that share one input vector in one CS submission.

The two ranges are staged back-to-back, then the existing 64-lane Q4_0 row-parallel kernel is dispatched twice in the same IB. The output slice receives A's 64 rows first and B's 64 rows second. This is used by the M1 forward bridge to consume a wider routed MoE gate/up slice without adding a second fence wait.

Dispatch four 64-row Q4_0 DMMV ranges that share one input vector in one CS submission.

This is the batched companion to `dmmvQ4_0TwoRowRangesParallel64` for decode-phase MoE gate/up validation across two routed experts. The output slice receives A, B, C, D ranges in order.

Dispatch Q4_0 DMMV for two arbitrary model rows staged back-to-back.

The caller supplies two individual source-format rows, which are packed into the shared staging page as a compact two-row matrix. This lets the current forward path obtain both LM-head top-2 scores from one real DMMV row-range submission even when the rows are not adjacent in vocab.

Dispatch the gfx1201 Q4_0 row-range DMMV kernel that performs argmax in the same submission.

The method stages the exact same source-format input and Q4_0 rows as `dmmvQ4_0RowRange`, but the kernel only stores the local best row and score. The forward path uses this for LM-head prefix/window candidates so a GPU-produced model value can directly participate in sampling without a follow-up direct argmax dispatch over copied logits.

Dispatch the gfx1201 row-range Q8_0 matrix-vector kernel.

Copies the input vector and raw GGML Q8_0 rows into the shared input page, records PM4 for one serial workitem over `rows`, and reads back one f32 result per row. This keeps source-format Q8_0 model-slice validation exact while the final K-parallel DMMV kernel is still under construction.

Dispatch one gfx1201 Q8_0 DMMV kernel over two adjacent logical row ranges that share the same input vector.

The method packs `weights_a` followed by `weights_b` into the staging page, then runs the same compact Q8_0 row-range kernel over `rows_a + rows_b` rows. The output slice receives A's rows first and B's rows second. This is used by the M1 bridge to consume paired SSM alpha/beta projections without paying two CS submissions for the same activation vector.

Dispatch one wave64 Q8_0 DMMV kernel over two packed row ranges totalling exactly 64 rows.

This is the row-parallel companion to `dmmvQ8_0TwoRowRanges` for the current SSM alpha+beta shape: 32 alpha rows plus 32 beta rows. Each lane computes one row from the packed staging block, eliminating the serial per-row loop of the scalar variant.