Skip to content
Skills Library
Data & analytics · Learning & explanation

vector-quantization

/vector-quantization

Vector compression techniques: scalar quantization (int8, int4), binary quantization (bit vectors + Hamming), product quantization (PQ), optimized PQ, residual quantization. Covers rescoring workflow, storage math, accuracy tradeoffs, and Faiss/Qdrant/pgvector implementations. USE WHEN: user mentions "quantization", "int8 embeddings", "binary embeddings", "product quantization", "PQ", "OPQ", "residual quantization", "Hamming distance", "compress embeddings", "rescoring" DO NOT USE FOR: Matryoshka dimension truncation - use `matryoshka-embeddings`; ANN algorithm choice - use `ann-algorithms`

Licence
MIT
Last repository change
21.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 21.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Vector compression techniques: scalar quantization (int8, int4), binary quantization (bit vectors + Hamming), product quantization (PQ), optimized PQ, residual quantization. Covers rescoring workflow, storage math, accuracy tradeoffs, and Faiss/Qdrant/pgvector implementations. USE WHEN: user mentions "quantization", "int8 embeddings", "binary embeddings", "product quantization", "PQ", "OPQ", "residual quantization", "Hamming distance", "compress embeddings", "rescoring" DO NOT USE FOR: Matryoshka dimension truncation - use `matryoshka-embeddings`; ANN algorithm choice - use `ann-algorithms`

Triggers from the description

quantizationint8 embeddingsbinary embeddingsproduct quantizationOPQresidual quantizationHamming distancecompress embeddings
Original author description

Vector compression techniques: scalar quantization (int8, int4), binary quantization (bit vectors + Hamming), product quantization (PQ), optimized PQ, residual quantization. Covers rescoring workflow, storage math, accuracy tradeoffs, and Faiss/Qdrant/pgvector implementations. USE WHEN: user mentions "quantization", "int8 embeddings", "binary embeddings", "product quantization", "PQ", "OPQ", "residual quantization", "Hamming distance", "compress embeddings", "rescoring" DO NOT USE FOR: Matryoshka dimension truncation - use `matryoshka-embeddings`; ANN algorithm choice - use `ann-algorithms`

How it works

  1. Why Quantize
  2. Scalar Quantization (int8 / int4)
  3. Binary Quantization
  4. Product Quantization (PQ)
  5. Optimized PQ (OPQ)
  6. Residual Quantization (RQ)
  7. Rescoring Workflow (Universal Pattern)
  8. Storage Math Reference
  9. Accuracy vs Memory Curve (rough)
  10. Anti-Patterns
Original SKILL.md headings · 273 lines, 1,235 words
SKILL.md preview
---
name: vector-quantization
description: |
  Vector compression techniques: scalar quantization (int8, int4), binary
  quantization (bit vectors + Hamming), product quantization (PQ), optimized PQ,
  residual quantization. Covers rescoring workflow, storage math, accuracy
  tradeoffs, and Faiss/Qdrant/pgvector implementations.

  USE WHEN: user mentions "quantization", "int8 embeddings", "binary embeddings",
  "product quantization", "PQ", "OPQ", "residual quantization", "Hamming distance",
  "compress embeddings", "rescoring"

  DO NOT USE FOR: Matryoshka dimension truncation - use `matryoshka-embeddings`;
  ANN algorithm choice - use `ann-algorithms`
allowed-tools: Read, Grep, Glob, Write, Edit
---
# Vector Quantization

## Why Quantize

A float32 1024-dim embedding is 4 KB. 100M vectors = 400 GB. Quantization cuts this 4-128x with small recall cost — especially when paired with rescoring.

| Scheme | Bytes/vec (d=1024) | Reduction | Typical recall impact |
|---|---|---|---|
| Full float32 | 4096 | 1x | 0 |
| float16 / bfloat16 | 2048 | 2x | < -0.5% |
| int8 scalar | 1024 | 4x | -1 to -2% |
| int4 scalar | 512 | 8x | -2 to -5% |
| Binary (1 bit) | 128 | 32x | -5 to -15% (recover with rescore) |
| PQ (64 subvectors x 8 bits) | 64 | 64x | -2 to -10% |
| Binary + MRL (d=256) | 32 | 128x | -3 to -8% with rescore |

## Scalar Quantization (int8 / int4)

Linear mapping: each float32 dim scaled to an 8-bit or 4-bit integer with per-index or per-dim scale/zero-point.

```python
import numpy as np

def int8_quantize(x: np.ndarray):
    # Symmetric per-dim scale
    scale = np.max(np.abs(x), axis=0) / 127
    q = np.round(x / scale).clip(-128, 127).astype(np.int8)
    return q, scale

def int8_dot(q: np.ndarray, q_q: np.ndarray, s_a: np.ndarray, s_b: np.ndarray) -> np.ndarray:
    return (q.astype(np.int32) @ q_q.astype(np.int32).T) * (s_a * s_b).sum()
```

For production, use framework support (Faiss, pgvector `halfvec`/int8, Qdrant scalar quantizer, Elasticsearch `int8_hnsw`) — they handle numerical issues and SIMD.

### Faiss scalar quantization

```python
import faiss

quantizer = faiss.IndexFlatL2(d)
index = faiss.IndexIVFScalarQuantizer(
    quantizer, d, nlist=1024,
    qtype=faiss.ScalarQuantizer.QT_8bit, metric=faiss.METRIC_L2,
)
index.train(sample)
index.add(corpus)
index.nprobe = 32
```

## Binary Quantization

Each float becomes 1 bit: `bit = (value > 0)`. Distance becomes Hamming distance (XOR + popcount), which is 10-40x faster than cosine on modern CPUs.

```python
import numpy as np

def binary_quantize(vecs: np.ndarray) -> np.ndarray:
    return np.packbits((vecs > 0).astype(np.uint8), axis=1)   # (N, d/8)

def hamming_topk(qbits: np.ndarray, cbits: np.ndarray, k: int):
    # XOR + popcount via numpy
    xor = np.bitwise_xor(cbits, qbits)
    dists = np.unpackbits(xor, axis=1).sum(axis=1)
    topk = np.argpartition(dists, k)[:k]
    return topk[np.argsort(dists[topk])]
```

### Recovery via Rescoring

Binary alone loses 5-15% recall. Fix: oversample on binary, then rescore the shortlist with full float vectors.

```python
def search_binary_then_rescore(qvec: np.ndarray, corpus: np.ndarray,
                               cbits: np.ndarray, top_k: int, oversample: int = 20):
    qbits = binary_quantize(qvec[None])[0]
    shortlist = hamming_topk(qbits, cbits, top_k * oversample)
    # Rescore with full float32
    sub = corpus[shortlist]
    scores = sub @ qvec
    order = np.argsort(-scores)[:top_k]
    return shortlist[order], scores[order]
```

Typical result: ~99% of full-float recall at 5-10x lower search latency and 32x less stored vector data.

### Binary in pgvector

```sql
-- Column
embedding_bit bit(1024)

-- Populate
UPDATE chunks SET embedding_bit = binary_quantize(embedding::vector)::bit(1024);

-- Index (pgvector 0.7+)
CREATE INDEX ON chunks USING hnsw (embedding_bit bit_hamming_ops);

-- Search then rescore
WITH shortlist AS (
  SELECT id, embedding
  FROM chunks
  ORDER BY embedding_bit <~> binary_quantize($1::vector)::bit(1024)
  LIMIT 100
)
SELECT id, 1 - (embedding <=> $1::vector) AS score
FROM shortlist
ORDER BY embedding <=> $1::vector
LIMIT 10;
```

### Binary in Qdrant

```python
from qdrant_client import models

client.update_collection(
    "docs",
    quantization_config=models.BinaryQuantization(
        binary=models.BinaryQuantizationConfig(always_ram=True),
    ),
)
client.query_points(
    "docs",
    query=qvec.tolist(),
    using="dense",
    limit=10,
    search_params=models.SearchParams(
        quantization=models.QuantizationSearchParams(
            rescore=True, oversampling=3.0,
        ),
    ),
)
```

## Product Quantization (PQ)

Split each d-dim vector into `m` subvectors of size d/m. Quantize each subvector to one of `ksub = 2^nbits` centroids learned by k-means on a training sample. Stored code: `m * nbits` bits per vector.

### Faiss PQ

```python
import faiss

m, nbits = 64, 8                             # 64 bytes per vector
index = faiss.IndexPQ(d, m, nbits)
index.train(sample)                          # needs ~256 * 2^nbits points
index.add(corpus)
D, I = index.search(queries, 10)
```

### IVF + PQ (standard huge-scale combo)

```python
quantizer = faiss.IndexFlatL2(d)
index = faiss.IndexIVFPQ(quantizer, d, nlist=8192, m=64, nbits=8)
index.train(sample)
index.add(corpus)
index.nprobe = 64
```

Faiss computes query-to-centroid distances once per query (`m * 256` values) and
sums them per candidate, which is very SIMD-friendly.

## Optimized PQ (OPQ)

Apply a learned orthogonal rotation before PQ so subvectors are better balanced.
Gains ~1-3% recall at the same code size.

```python
import faiss
opq = faiss.OPQMatrix(d, m)
opq.train(sample)
rotated = opq.apply_py(corpus)
pq = faiss.IndexPQ(d, m, nbits)
pq.train(rotated)
pq.add(rotated)
# Or use the pre-built combo:
index = faiss.index_factory(d, f"OPQ{m}_64,IVF8192,PQ{m}x{nbits}")
```

## Residual Quantization (RQ)

Quantize the residual (original - first quantized approximation) with another codebook. Stack multiple stages for fine-grained quantization.

Use cases: where PQ at a given rate underfits — RQ gets closer to float32 at the
same bit budget. Implemented in Faiss as `IndexResidualQuantizer`, and underlies
ScaNN's anisotropic quantizer.

```python
index = faiss.IndexResidualQuantizer(d, nsplits=4, nbits=8)
index.train(sample)
index.add(corpus)
```

## Rescoring Workflow (Universal Pattern)

```
1. Store FULL precision vectors in cold storage (or in the same DB as a second column).
2. Index QUANTIZED vectors for ANN search.
3. Search quantized, retrieve `oversample * k` candidates.
4. Rescore candidates with full vectors (dot product / cosine).
5. Return true top-k.
```

This pattern recovers 99%+ of full-precision recall for binary/PQ and costs only
the oversample factor in extra candidates.

```python
def hybrid_search(qvec, k=10, oversample=20):
    candidates = quantized_index.search(qvec, k * oversample)  # fast
    full_vecs = full_store.get_many([c.id for c in candidates])
    scored = sorted(
        zip(candidates, full_vecs),
        key=lambda x: -np.dot(qvec, x[1]),
    )
    return scored[:k]
```

## Storage Math Reference

For N vectors of dimension d:

| Scheme | Bytes |
|---|---|
| float32 | N * d * 4 |
| float16 | N * d * 2 |
| int8 | N * d + N * 4 (per-vector scale) |
| binary | N * d / 8 |
| PQ(m, nbits) | N * m * nbits / 8 |
| IVFPQ(m, nbits, nlist) | N * m * nbits / 8 + nlist * d * 4 (centroids) |

Example: 100M x 1024-d

- float32: 400 GB
- float16: 200 GB
- int8: 102 GB
- binary: 12.5 GB
- PQ(64, 8): 6.4 GB
- IVFPQ(64, 8, 8192): 6.4 GB + 32 MB centroids

## Accuracy vs Memory Curve (rough)

- int8: keep 99%+ of float32 recall. Always safe default.
- PQ(d/16, 8): keep 95-98% with rescoring. Great for 10M+.
- Binary: keep 95-99% WITH rescoring, 85-92% without.
- PQ(d/64, 8): keep 85-92%, even with rescoring; use only at extreme scale.

Always measure on YOUR data. Quantization quality depends on embedding distribution.

## Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| Binary quantization without rescoring | Always oversample 2-5x and rescore with float |
| PQ with undersized training set | Train on at least `256 * 2^nbits` representative points |
| Mixing L2 and cosine across quantized / full vectors | Normalize everything; pick one metric |
| Quantizing without measuring recall on eval set | Sweep oversample + compare to brute-force baseline |
| Forgetting centroids need retraining on distribution shift | Retrain IVF / PQ centroids periodically |
| Int4 before int8 has been benchmarked | Try int8 first; only go lower if memory forces it |
| Storing full vectors only — no quantized mirror | Keep both: quantized for fast search, full for rescore |

## Production Checklist

- [ ] Recall@k measured against brute-force ground truth at chosen quantization
- [ ] Oversampling factor tuned (typically 2-10x)
- [ ] Rescoring pipeline implemented (full vectors retrievable by ID)
- [ ] Training sample size meets quantizer requirements
- [ ] Retraining schedule for IVF/PQ centroids documented
- [ ] Memory budget verified post-quantization
- [ ] Metric consistency (cosine vs L2) across quantized and full paths
- [ ] Fallback to float32 for critical queries if needed

Versions

The database has no other skills named vector-quantization.

Database snapshot: 13.11.2026 · package version f2124c567b0b (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100qdrant-advanced
    Advanced Qdrant features. Payload indexing, scalar/binary/product quantization, named vectors (multi-vector), sparse vectors, multi-tenancy, snapshots, sharding, gRPC vs HTTP, pre- vs post-filtering, and oversampling/rescoring. USE WHEN: user mentions "Qdrant", "named vectors",…
  • 100matryoshka-embeddings
    Matryoshka Representation Learning (MRL) for dimension truncation without retraining. Covers OpenAI text-embedding-3 MRL, Nomic, Jina v3, training with MatryoshkaLoss, binary quantization compatibility, and storage/quality tradeoffs. USE WHEN: user mentions "matryoshka", "MRL",…
  • 90qdrant
    Qdrant vector database: collections, points, payload filtering, indexing, quantization, snapshots, and Docker/Kubernetes deployment. Use when managing Qdrant collections, performing vector searches with payload filters, configuring HNSW indexes or quantization, or deploying…
  • 100vector-db
    Use when operating a vector store as a data layer — choosing or migrating between Pinecone, Qdrant, Weaviate and pgvector; designing a collection or index (distance metric, dimensions, HNSW parameters, named vectors); filtering on metadata; hybrid dense-plus-sparse search; and…
  • 95data-vector-databases
    Use this skill when the user says 'vector database', 'vector db', 'pgvector', 'qdrant', 'milvus', 'pinecone', 'weaviate', 'hnsw', 'ivf', 'product quantization', 'ann search', 'similarity search', 'embeddings storage', 'hybrid search'. Chooses a vector store, designs…
  • 100qdrant
    Qdrant is an open-source vector database written in Rust for similarity search over embeddings, with payload filtering, named and sparse vectors, hybrid search and quantization. Use when someone asks to set up Qdrant, build semantic search or RAG retrieval, store embeddings,…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.