Cs

Returns true when both PM4 submissions retired and the signal BO read back the expected sentinel value.

Open the canonical render node (`default_render_node`) and finish the full CS bring-up: context, BO list, IB / input / output / signal / shader buffers, all mapped into a low GPU VA range.

Open the given render node and bring up the full CS submission state.

Allocates an amdgpu context, creates GTT-backed BOs for the indirect buffer, input scratch (~2 MiB), output, signal and shader pages, maps each into a fixed low GPU VA so the kernel does not need to re-bind them per submission, uploads the gfx1201 PM4 kernels into the shader page, and creates a persistent BO list referencing all five BOs.

Tear down every kernel resource the `init*` paths created: destroy the BO list, free the amdgpu context, `munmap` each CPU mapping, and close the render-node file descriptor.

Round-trip one `u32` through the GPU as the simplest end-to-end gate: PM4 `COPY_DATA` from the input page to the output page, plus a `WRITE_DATA` of a per-submission sentinel into the signal page.

Alias for `copyU32` framed as the per-token decode pulse: prove the GPU produced a token by round-tripping its id through a real PM4 submission and fence wait.

Dispatch the gfx1201 top-2 argmax kernel and return the selected token.

Loads the argmax program into the compute SGPRs, packs the output VA, two ordered scores, and two token ids into `compute_user_data_2..7`, fires one workgroup, then waits on the signal sentinel before reading the kernel-chosen token from the output page.

Dispatch the gfx1201 ordered-score row-range argmax kernel.

Converts `scores` into sortable u32 keys, copies them into the shared input page, then lets the compute ring select the max row. The returned token id is absolute: `start_row + local_best`.

Dispatch the single-element gfx1201 final-RMS-norm kernel.

Stores `hidden0 * inv_rms * weight0` into `output_map[0]` via a real PM4 dispatch on the compute ring, with a signal sentinel verifying retirement.

Dispatch the gfx1201 row-range f32 dense matrix-vector kernel.

Copies the input vector and the row-major f32 weight block into the shared input page (64-byte aligned), records PM4 that points the kernel at the input/weights/output pages and the `rows`/`cols` arguments, and waits on the signal sentinel before reading `output`. This is the first row-oriented dense compute kernel the CS path runs; `cols` must be a multiple of 64 and `output` must hold at least `rows` elements.

Dispatch the gfx1201 row-range Q4_0 matrix-vector kernel.

Copies the input vector and raw GGML Q4_0 rows into the shared input page, records PM4 for one serial workitem over `rows`, and reads back one f32 result per row. This intentionally validates real quantized model bytes through the native CS path while the full K-parallel DMMV kernel is still under construction.

Dispatch the wave-lane gfx1201 Q4_0 matrix-vector kernel for exactly 64 rows in parallel.

Stages the same source-format rows as `dmmvQ4_0RowRange`, but launches one wave64 workgroup where each lane computes one row. Intended for 64-row LM-head prefix/window ranges where the GPU row values participate in choosing the sampled token.

Dispatch Q4_0 DMMV for two arbitrary model rows staged back-to-back.

The caller supplies two individual source-format rows, which are packed into the shared staging page as a compact two-row matrix. This lets the current forward path obtain both LM-head top-2 scores from one real DMMV row-range submission even when the rows are not adjacent in vocab.

Dispatch the gfx1201 Q4_0 row-range DMMV kernel that performs argmax in the same submission.

The method stages the exact same source-format input and Q4_0 rows as `dmmvQ4_0RowRange`, but the kernel only stores the local best row and score. The forward path uses this for LM-head prefix/window candidates so a GPU-produced model value can directly participate in sampling without a follow-up direct argmax dispatch over copied logits.

Dispatch the gfx1201 row-range Q8_0 matrix-vector kernel.

Copies the input vector and raw GGML Q8_0 rows into the shared input page, records PM4 for one serial workitem over `rows`, and reads back one f32 result per row. This keeps source-format Q8_0 model-slice validation exact while the final K-parallel DMMV kernel is still under construction.

Dispatch one gfx1201 Q8_0 DMMV kernel over two adjacent logical row ranges that share the same input vector.

The method packs `weights_a` followed by `weights_b` into the staging page, then runs the same compact Q8_0 row-range kernel over `rows_a + rows_b` rows. The output slice receives A's rows first and B's rows second. This is used by the M1 bridge to consume paired SSM alpha/beta projections without paying two CS submissions for the same activation vector.

Dispatch one wave64 Q8_0 DMMV kernel over two packed row ranges totalling exactly 64 rows.

This is the row-parallel companion to `dmmvQ8_0TwoRowRanges` for the current SSM alpha+beta shape: 32 alpha rows plus 32 beta rows. Each lane computes one row from the packed staging block, eliminating the serial per-row loop of the scalar variant.