Building DAMASCUS: a language model from file to forward pass

Start with the boundaries of the problem

DAMASCUS targets one model, one stream, and one machine: Llama 3.2 3B Instruct, quantized as Q4_K_M GGUF, on an Apple M1 with 16 GB of unified memory. That scope makes the work inspectable. Each of the nine compartments has an explicit responsibility and a correctness check before it can become a performance experiment.

Those compartments cover platform setup, tensor storage, model loading, tokenization, compute kernels, transformer composition, sampling, the generation runtime, and quantization with AMX. The first six form the path from bytes on disk to next-token logits. Sampling and sustained generation come after that path is verified.

Read the model, then reproduce its operations

A GGUF loader must interpret the header, metadata, tensor directory, and quantized blocks consistently. Shapes and model configuration come from the file; a tensor with the right byte count can still be wrong if its dimensions or quantization layout are interpreted incorrectly. The Q4_K and Q6_K dequantizers are checked against the gguf library before their output feeds the transformer.

The tokenizer is a separate correctness boundary. Byte-level BPE must produce the same token IDs as the reference, and decoding must preserve the original text. A plausible-looking token sequence is insufficient: a different tokenization changes every later activation, making a forward-pass comparison meaningless.

Inside a decoder layer, RMSNorm precedes the QKV projections. RoPE supplies positional information, grouped-query attention reads the KV cache, and the attention output returns through a residual connection. A second RMSNorm and a SwiGLU feed-forward network complete the layer. After 28 layers, final normalization and the tied embedding matrix produce logits over 128,256 vocabulary entries.

A same-weights oracle makes the comparison useful

Comparing quantized inference to an unquantized model would mix implementation error with quantization error. Instead, the HuggingFace oracle loads the same Q4_K_M GGUF weights and dequantizes the same blocks. For the prompt “The capital of France is”, the documented CPU run reports per-layer cosine similarity of 1.0000000 and exact top-1 agreement. The next token is “ Paris”, at 68.65% in DAMASCUS and 68.63% in the oracle.

The absolute maximum exceeds the original 1e-3 acceptance threshold. The project documents that limitation: the largest differences occur on large activation values, while cosine similarity, exact next-token agreement, and top-five overlap support the CPU result. Publishing the error alongside the agreement is more informative than describing the result as bit-exact.

The README’s verified results and oracle harness contain the comparison details. The commands make cpu-check, make oracle-gguf, and make forward-check reproduce the tokenizer, kernel, and forward-pass checks with the required model and Docker environment.

Prefill and decode need different performance questions

Decode processes one token at a time. Its matrix-vector operations repeatedly stream model weights, so memory bandwidth can dominate. Prefill processes many prompt tokens together and uses matrix-matrix operations, giving more opportunities to reuse weights and occupy compute units. An optimization that helps one phase may barely affect the other.

The planned Metal path maps suitable operations to the GPU. The AMX experiment asks whether fused dequantization and matrix computation can reduce intermediate memory traffic during prefill, eventually alongside GPU work using unified memory. A 30% prefill improvement over llama.cpp is the target. It has not been established by the current results, and this article does not report a Metal or AMX speedup.

What a reproducible benchmark must record

A throughput claim needs a pinned model hash, quantization format, llama.cpp revision, hardware configuration, macOS version, prompt length, and generation length. Both implementations need the same workload and a controlled thermal state. The planned baseline measures 1,024-token prefill separately from 256-token decode, with warmup excluded and repeated runs rather than a single favourable measurement.

The project’s baseline protocol calls for ten-run medians, p95, and variation, as well as cold-start latency, peak resident memory, and sustained throughput after thermal load. Its results table is still unpopulated. Until those measurements exist, memory use and speed remain questions to measure, not numbers to infer from model file size or correctness tests.

Current limits and the next step

The published milestone is a verified CPU forward pass. The documented development environment used a Linux Docker guest to run the CPU checks when native binary execution was blocked; that environment cannot validate Metal or AMX. GPU validation, coherent generation, and reproducible performance measurements remain separate gates.

Follow the source repository and session write-ups for the evidence behind each change, or read more about my work.