# wav2vec2.cpp

**Repository Path**: AI2CG/wav2vec2.cpp

## Basic Information

- **Project Name**: wav2vec2.cpp
- **Description**: https://github.com/nabil6391/wav2vec2.cpp
- **Primary Language**: C++
- **License**: MIT
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-08-30
- **Last Updated**: 2026-09-05

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# wav2vec2.cpp

[![Build](https://github.com/nabil6391/wav2vec2.cpp/actions/workflows/build.yml/badge.svg)](https://github.com/nabil6391/wav2vec2.cpp/actions/workflows/build.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
![Platform](https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-blue)

Pure C++ CTC inference for any [`Wav2Vec2ForCTC`](https://huggingface.co/docs/transformers/model_doc/wav2vec2) model, using [ggml](https://github.com/ggml-org/ggml).

- No Python runtime
- No HuggingFace Transformers
- Single binary, zero dependencies beyond ggml (fetched automatically via CMake)
- Quantization down to 2-bit via `llama-quantize`
- Public C API (`wav2vec2.h`) — easy to embed in any application
- SIMD-accelerated: ARM NEON (Apple Silicon), AVX2 (x86-64), VSX (POWER)
- GPU backends: Metal, CUDA, Vulkan (via ggml backend flags)
- Voice Activity Detection (VAD) built-in
- Zero heap allocations during inference (pre-allocated buffers)

Inspired by [whisper.cpp](https://github.com/ggml-org/whisper.cpp). Supports both model variants:

- `feat_extract_norm="group"` — GroupNorm CNN (e.g. `wav2vec2-base`)
- `feat_extract_norm="layer"` — LayerNorm CNN (e.g. `wav2vec2-large`)

---

## Quick start

```bash
# 1. Build
git clone https://github.com/nabil6391/wav2vec2.cpp
cd wav2vec2.cpp
cmake -B build && cmake --build build -j

# 2. Convert a HuggingFace model to GGUF
pip install gguf transformers torch
python convert_wav2vec2_to_gguf.py \
    --model-dir /path/to/wav2vec2-model \
    --output     models/model-f16.gguf

# 3. (Optional) Quantize to 5-bit — 2.5× smaller, same accuracy
python quantize_gguf.py models/model-f16.gguf models/model-q5_0.gguf Q5_0

# 4. Transcribe
./build/wav2vec2 models/model-q5_0.gguf audio.wav
```

Output (Quranic Arabic example):
```
بسم الله الرحمن الرحيم الحمد لله رب العالمين الرحمن الرحيم مالك يوم الدين ...
```

---

## Files

| File | Description |
|------|-------------|
| `wav2vec2.h` | Public C API header — include this to embed the library |
| `wav2vec2.cpp` | Library implementation (SIMD, VAD, C API) |
| `main.cpp` | CLI binary — thin wrapper over the C API |
| `CMakeLists.txt` | Build config — static library + binary, all backend options |
| `convert_wav2vec2_to_gguf.py` | Export any `Wav2Vec2ForCTC` model → F16 GGUF |
| `quantize_gguf.py` | Quantize transformer weights to Q4_0 / Q5_0 / Q8_0 / MXFP4 |
| `eval_cer.py` | Measure CER and WER across model variants |
| `models/download-gguf-model.sh` | Download a GGUF from Hugging Face |
| `BLOG_POST.md` | Technical writeup — every bug hit building this from scratch |

---

## Build

Requires CMake ≥ 3.14 and a C++17 compiler. ggml is fetched automatically.

```bash
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
```

### GPU backends

```bash
# Apple Silicon (Metal)
cmake -B build -DWAV2VEC2_METAL=ON
cmake --build build -j

# NVIDIA (CUDA)
cmake -B build -DWAV2VEC2_CUDA=ON
cmake --build build -j

# Cross-platform (Vulkan)
cmake -B build -DWAV2VEC2_VULKAN=ON
cmake --build build -j

# CPU with BLAS/Accelerate (macOS)
cmake -B build -DWAV2VEC2_BLAS=ON
cmake --build build -j
```

Other available flags: `WAV2VEC2_OPENCL`, `WAV2VEC2_OPENVINO`, `WAV2VEC2_ASCEND`, `WAV2VEC2_MTHREADS`, `WAV2VEC2_COREML`.

To link against a system ggml instead of fetching:
```bash
cmake -B build -DWAV2VEC2_USE_SYSTEM_GGML=ON \
      -DGGML_INCLUDE_DIR=/path/to/ggml/include \
      -DGGML_LIBRARY=/path/to/libggml.a
cmake --build build -j
```

### CLI usage

```
wav2vec2 [options] model.gguf audio.wav

Options:
  --threads N       CPU threads (default: 1)
  --no-gpu          Disable GPU acceleration
  --timings         Print inference timing / RTF
  --vad             Run Voice Activity Detection
  --vad-threshold F VAD RMS threshold (default: 0.02)
  --sysinfo         Print build / SIMD / backend info and exit
```

### C API usage

```c
#include "wav2vec2.h"

struct wav2vec2_params p = wav2vec2_default_params();
p.print_timings = true;

wav2vec2_context *ctx = wav2vec2_init("model.gguf", p);

float *samples; int n_samples, sr;
wav2vec2_read_wav("audio.wav", &samples, &n_samples, &sr);

// Optional: check how much of the audio is speech
float speech = wav2vec2_pcm_to_vad(samples, n_samples, 0.02f, 400);

wav2vec2_full(ctx, samples, n_samples);
printf("%s\n", wav2vec2_full_get_text(ctx));

wav2vec2_pcm_free(samples);
wav2vec2_free(ctx);
```

---

## Converting a model

Any `Wav2Vec2ForCTC` model on Hugging Face can be converted:

```bash
python convert_wav2vec2_to_gguf.py \
    --model-dir /path/to/wav2vec2-model \
    --output    models/model-f16.gguf
```

The script writes transformer weights as F16 and CNN/bias/norm weights as F32. The resulting GGUF is self-contained — no Python needed at inference time.

---

## Quantization

### Python gguf library (Q4_0 · Q4_1 · Q5_0 · Q5_1 · Q8_0 · MXFP4)

```bash
python quantize_gguf.py models/model-f16.gguf models/model-q5_0.gguf Q5_0
```

Only the 2D transformer weight matrices (attention Q/K/V/O and FFN fc1/fc2) are quantized. CNN weights, biases, and LayerNorm parameters stay in their original dtype — they are 1D/3D or numerically sensitive.

### llama-quantize (K-series · IQ-series — `hidden_size` must be divisible by 256)

For `wav2vec2-large` (`hidden_size=1024`), K-series quantization is compatible.
Install [llama.cpp](https://github.com/ggml-org/llama.cpp) (`brew install llama.cpp` on macOS), then:

```bash
# Create large_tensor_types.txt to keep 3D tensors as F32
cat > large_tensor_types.txt <<'EOF'
cnn.0.conv.weight=f32
cnn.1.conv.weight=f32
cnn.2.conv.weight=f32
cnn.3.conv.weight=f32
cnn.4.conv.weight=f32
cnn.5.conv.weight=f32
cnn.6.conv.weight=f32
pos_conv.weight=f32
EOF

llama-quantize \
  --override-kv general.architecture=str:llama \
  --override-kv llama.context_length=int:1 \
  --override-kv llama.embedding_length=int:1024 \
  --override-kv llama.block_count=int:24 \
  --override-kv llama.attention.head_count=int:16 \
  --override-kv llama.attention.layer_norm_rms_epsilon=float:0.00001 \
  --override-kv tokenizer.ggml.model=str:none \
  --tensor-type-file large_tensor_types.txt \
  models/model-f16.gguf models/model-q4km.gguf Q4_K_M
```

> **Why pin the 3D tensors?** CNN conv weights (e.g. `[10, 1, 512]`) and `pos_conv.weight` (`[128, 64, H]`) are 3D — incompatible with block quantization. Without explicit pinning, llama-quantize silently converts them to F16/Q4_0 as a fallback, producing garbage inference output.

---

## Benchmarks

Measured on 41.4 s of audio, Apple M-series. CER/WER against known reference text.

> Single-file CER has high variance. Differences of ±2–3% between formats are within noise — only Q2_K shows a significant quality hit.

### wav2vec2-base (hidden=384, 10 layers)

K-series are **not compatible** — `hidden_size=384` is not divisible by 256. Use `quantize_gguf.py`.

| Format | Tool   | Size  | vs F16 | Speed | CER↓  | WER↓  |
|--------|--------|-------|--------|-------|-------|-------|
| F16    | —      | 51 MB | —      | 46s   | 16.8% | 55.2% |
| Q8_0   | Python | 35 MB | 1.5×   | 47s   | 16.8% | 55.2% |
| Q5_1   | Python | 30 MB | 1.7×   | 48s   | 16.8% | 58.6% |
| Q5_0   | Python | 29 MB | 1.8×   | 48s   | 16.8% | 51.7% |
| Q4_1   | Python | 28 MB | 1.8×   | 48s   | 15.4% | 55.2% |
| Q4_0   | Python | 27 MB | 1.9×   | 48s   | 16.1% | 58.6% |
| MXFP4  | Python | 26 MB | 2.0×   | 47s   | 19.6% | 58.6% |

**Recommended: Q5_0** — 1.8× smaller, identical CER to F16.

### wav2vec2-large (hidden=1024, 24 layers)

K-series **are compatible** — `hidden_size=1024` is divisible by 256. Use `llama-quantize`.

| Format  | Tool   | Size   | BPW   | vs F16 | Speed  | CER↓  | WER↓  |
|---------|--------|--------|-------|--------|--------|-------|-------|
| F16     | —      | 626 MB | 16.66 | —      | ~138s  | 17.5% | 55.2% |
| Q8_0    | llama  | 356 MB | 9.46  | 1.8×   | ~137s  | 17.5% | 55.2% |
| Q6_K    | llama  | 286 MB | 7.60  | 2.2×   | ~144s  | 17.5% | 55.2% |
| Q5_K_M  | llama  | 248 MB | 6.59  | 2.5×   | ~137s  | 16.8% | 51.7% |
| Q5_1    | Python | 266 MB | —     | 2.4×   | ~140s  | 19.6% | 58.6% |
| Q5_0    | Python | 248 MB | —     | 2.5×   | ~142s  | 16.8% | 55.2% |
| Q4_K_M  | llama  | 212 MB | 5.63  | 3.0×   | ~141s  | 18.9% | 58.6% |
| Q4_1    | Python | 230 MB | —     | 2.7×   | ~158s  | 18.9% | 58.6% |
| Q4_0    | Python | 212 MB | —     | 3.0×   | ~149s  | 18.2% | 62.1% |
| MXFP4   | Python | 203 MB | —     | 3.1×   | ~153s  | 17.5% | 55.2% |
| Q3_K_M  | llama  | 173 MB | 4.61  | 3.6×   | ~158s  | 17.5% | 48.3% |
| Q2_K    | llama  | 144 MB | 3.83  | 4.3×   | ~152s  | 25.2% | 79.3% |

**Recommended: Q5_K_M** — 2.5× smaller, same CER as F16.

**Speed note:** All formats run at essentially the same speed on Apple Silicon — ggml dequantizes weights to F32 before every matrix multiply. There is no native integer GEMM on Apple Silicon; the ±20s variance is thermal, not quantization overhead. Speedup from quantization requires x86+AVX or hardware with native integer GEMM.

---

## Evaluating accuracy

```bash
# Evaluate all *.gguf files in the current directory against a reference
python eval_cer.py --wav audio.wav --ref "reference transcription"

# Or use the built-in Al-Fatiha reference text (for Arabic Quran models):
python eval_cer.py --wav audio.wav --fatiha model-f16.gguf model-q5km.gguf

# Sort by CER:
python eval_cer.py --wav audio.wav --ref "..." --sort cer
```

---

## Memory usage

| Model | Disk | RAM (approx.) |
|-------|------|----------------|
| wav2vec2-base F16 | 51 MB | ~120 MB |
| wav2vec2-base Q5_0 | 29 MB | ~80 MB |
| wav2vec2-large F16 | 626 MB | ~900 MB |
| wav2vec2-large Q5_K_M | 248 MB | ~400 MB |

---

## Technical notes

- **CausalConv1d:** Applied during training but lost on `save_pretrained()`. The converted model uses plain `Conv1d(padding=0)` and the C++ inference uses `L_out = (L_in - K) / stride + 1`.
- **Global LayerNorm placement:** `Wav2Vec2EncoderStableLayerNorm` applies the global LayerNorm *after* all transformer layers, not before. A common source of incorrect ports.
- **feat_extract_norm:** The `wav2vec2.feat_extract_norm_type` GGUF metadata key controls CNN norm dispatch at inference time. `0` = GroupNorm/InstanceNorm (base), `1` = LayerNorm (large).
- **Full technical writeup:** See [BLOG_POST.md](BLOG_POST.md) for a detailed account of every bug encountered building this from PyTorch to C++.