Przejdź do treści
Biblioteka skilli
Dane i analityka · Nauka i wyjaśnianie

lance-format

/lance-format

Dostarcza szczegółowej wiedzy o kolumnowym formacie Lance v14, jego crate'ach w Rust i pylance: kodowaniach plików, formacie tabel, indeksach, ewolucji schematu i time travel.

Repozytorium
skills ↗
Licencja
MIT
Ostatnia zmiana repozytorium
24.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
65
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
3
Przestarzałość

Brak sygnałów. Repozytorium zmienione 24.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Dostarcza szczegółowej wiedzy o kolumnowym formacie Lance v14, jego crate'ach w Rust i pylance: kodowaniach plików, formacie tabel, indeksach, ewolucji schematu i time travel.

Oryginalny opis (EN)

Deep reference for Lance v14 columnar format, its Rust crates, and pylance - file encodings, table format, indexes, schema evolution, time travel. Use when building on the Lance crates or reading .lance datasets, not the LanceDB product.

Jak działa

  1. Lance vs LanceDB
  2. The crate workspace
  3. File format versions
  4. Version landscape
  5. Manifest feature flags
  6. The v13 final and v14 delta
  7. Earlier majors in brief
  8. Performance questions
  9. Official docs mirror
Sekcje z nagłówków SKILL.md, w oryginale · 356 linii, 3 708 słów
Podgląd SKILL.md
---
name: lance-format
description: Deep reference for Lance v14 columnar format, its Rust crates, and pylance - file encodings, table format, indexes, schema evolution, time travel. Use when building on the Lance crates or reading .lance datasets, not the LanceDB product.
metadata:
  version: "0.21.1"
  categories: "development, integrations"
  topics: "lance, columnar-format, vector-search, rust, lakehouse"
  upstream: "lance-format/lance@v14.0.0-beta.14"
  openclaw:
    homepage: https://github.com/tenequm/skills/tree/main/skills/lance-format
    emoji: "🗄️"
---

# Lance v14 reference

Lance is an open columnar format for multimodal AI - "a columnar data format that is 100x
faster than Parquet for random access." It is not one format but a stack of interoperating
specs: a **file format**, a **table format**, **index formats**, **catalog specs**, and a
**namespace client spec**. The Rust workspace at `lance-format/lance` implements all of them
plus Python (`pylance`) and Java bindings.

This skill tracks **`v14.0.0-beta.14`** (the `lance-format/lance` git tag), the current
development frontier; **`v13.0.0`** is the stable pin, released 2026-10-07. Pin against tags, not
`main` - Lance ships beta tags every few days and `next`-format encodings can change. Version
landscape below.

Three layers of reference, load what the task needs:

- **The deep reference** - any concrete schema, parameter, proto, or constraint. Split by topic:

  | File in `references/` | Covers | Sections |
  |------|--------|----------|
  | `format-file.md` | What Lance is, the 27 crates, file format, data types | 1-4 |
  | `format-table.md` | Dataset layout, manifests, fragments, schema evolution, versioning/tags/branches, row IDs, transactions + OCC, MemWAL | 5-10 |
  | `indexes.md` | Vector / scalar / FTS / geo indexes, distributed builds | 11-12 |
  | `ops.md` | Object store, capability matrix, source map | 13, 15, 16 |
  | `changelog-v7-v14.md` | The full v7 -> v14 delta | 14 |

  Cross-references written as "section N" resolve through `references/lance-reference.md`.
- `references/performance.md` - ALL performance guidance. Part A routes to the official text and
  adds the source-derived changes upstream has not documented; Part B is field-verified
  remote-storage practice. Load for any performance, tuning, maintenance-cost, or "why is this
  slow" question.
- `references/docs/` - a **verbatim mirror of the official docs** (`docs/src` at the tracked
  tag): every guide, quickstart, and format spec, unedited. Load when you need the full official
  text. Directory map below.

`references/maintenance.md` covers refreshing this skill against a new upstream tag.

## Lance vs LanceDB

These are two different things and conflating them produces wrong answers.

- **Lance** - the format and engine. The `lance-format/lance` repo; the `lance` /`lance-*`
  Rust crates; `pylance`. It gives you datasets, the file/table format, indexes, commits,
  scans. Consumed directly by DuckDB, Polars, Ray, Spark, PyTorch, DataFusion, or your own
  Rust/Python code. **This skill is about Lance.**
- **LanceDB** - a separate database *product* (`lancedb/lancedb`) built on top of Lance. It
  adds a query-builder API, an embedding registry, rerankers-as-API, multi-language SDK
  parity, and managed Cloud / Enterprise tiers. Not covered here.

**The wider ecosystem** (separate repos, own version lines, none covered here): Flink streaming
writes (`lance-flink`), PostgreSQL reads via `pglance`, a Cypher graph engine (`lance-graph`), a
dataset browser (`lance-data-viewer`), agentic context management (`lance-context`), and
namespace catalogs for Hive, Polaris, Gravitino, Unity Catalog, and AWS Glue.
The canonical docs site is **`lance.org`**. Generated per-language SDK docs live at
`lance-format.github.io/lance-python-doc` for Python and
[javadoc.io](https://www.javadoc.io/doc/org.lance/lance-core/latest/index.html) for Java - the
matching `lance-format.github.io/lance-java-doc` path 404s.

Linking the `lance` crate in `Cargo.toml` means you are using Lance directly - use this skill.
For LanceDB internals, the storage layer underneath is still Lance, so this skill remains the
authority for the format itself.

**The wrapper can hide format features.** LanceDB's `create_table` cannot enable stable row IDs;
only pylance's `write_dataset(enable_stable_row_ids=True)` can. If a format-level capability
matters to your design, check whether the wrapper exposes it before assuming the underlying
format settles the question - and reach for `pylance` directly when it does not.

## The crate workspace

27 crate directories under `rust/` at `v14.0.0-beta.14` - #9534 vendored the jsonb codec as
**`lance-jsonb`** (26 at `v13.0.0`). **`lance` is the public entry point** - `Dataset`, scanner,
indexes, commits; everything else (`lance-table`, `lance-file`, `lance-encoding`, `lance-index`,
`lance-io`, `lance-core`, `lance-datafusion`, `lance-linalg`, `lance-namespace*`, ...) is a layer
beneath it. Edition 2024, arrow 58, datafusion 54, pyo3 0.28; Python bindings need 3.10+. **MSRV
is 1.94.0** in the v14 line (#8997), up from 1.91.0 through `v13.0.0`. Full table with roles,
versions, and every workspace dep in `references/format-file.md` section 2.

**If you depend on anything below `lance`, expect moves.** v11 deleted `lance-encoding::version`
with no re-export (`LanceFileVersion` and `ConcreteFileVersion` live in `lance-file::version`),
removed `lance_io::encodings`, and gave each format its own `versions/v2_{0,1,2,3}` module
(#8020-#8026). The transaction code moved to a `rust/lance-table/src/transaction/` module tree
(#8053/#8054/#8056); a `lance::dataset::transaction` shim still re-exports `Operation`,
`Transaction`, `TransactionBuilder`, `RewriteGroup` and `UpdateMap`, so the common surface is
unbroken - but a symbol the shim omits, or a citation of the old path, needs retargeting.
Section 2.1.

## File format versions

The file format carries a single major.minor version. `data_storage_version` is set per dataset
at creation - but since `v12.0.0` it is **no longer fixed once the dataset exists**. It is the
*default* for writes that omit a target: "Create and overwrite establish this default; append,
update, merge-insert, and compaction do not change it." An existing V2 dataset can take `"2.0"`,
`"2.1"`, `"2.2"` or `"2.3"` per operation without rewriting its other files (#8582-#8585), so one
dataset can hold data files at several exact V2 versions. **V1 and V2 still cannot be mixed.**
Section 3.

| Version | Status | Notes |
|---------|--------|-------|
| `0.1` (`legacy`) | read-only | Original format; no longer writable |
| `2.0` | stable | Removed row groups; null support for lists/FSL/primitives |
| `2.1` | previous default | Adaptive structural encodings; better integer/string compression; nulls in struct fields; better nested random access. Was the default from Lance 5.0.0 until `v12.0.0-beta.15` |
| `2.2` | **current default** (`stable`) | Map type, Blob v2, `VariablePackedStruct`, larger mini-blocks. Required for Map and Blob v2 |
| `2.3` | unstable (`next`) | Sparse structural pages (auto-selected by the 2.3 writer), the generic block v5 codecs (#8325-#8333, all landed by `v14.0.0-beta.9`), and compressed mini-block offsets (#8331). Stable readers refuse it |

**`stable` resolves to 2.2** (#8657), and `2.2` is the enum `#[default]`, so a dataset created
without an explicit `data_storage_version` is written as 2.2 - Python's `write_dataset` inherits
it. **The docs still do not say so** - `format/file/versioning.md` only calls `stable` an "alias
for the default version" - so the code is the authority. `next` resolves to 2.3. Pin an explicit
number for deterministic behavior across builds: anything relying on `WriteParams::default()`
(tests, benches) silently changed format at the 2.2 switch.

On v12, a writer defaulting to 2.2 rejected legacy blob metadata outright. **Since `v13.0.0`**
(#9459) legacy blob metadata written to 2.2+ is converted to Blob v2, and a 2.0/2.1 table can
append Blob v2 without a rewrite. Blob v2 tiers by size: **64 KiB inline, 4 MiB dedicated**,
packed `.blob` sidecars between (the pre-v14 docs said 16 KiB / 2 MiB; the code never did).

The release *selectors* (`LanceFileVersion`) are a type distinct from the persisted identity
(`ConcreteFileVersion`). Details in `references/format-file.md` sections 3.1 and 3.6.

## Version landscape

The major is bumped by a bot, not a human: `ci/publish_beta.sh` re-roots at `MAJOR+1` whenever
any PR since the release root carries the GitHub `breaking-change` label - the marker is the
**label**, not a conventional-commit `!`. A major bump therefore means "some labeled breaking
change landed", not a redesign. It has fired on **five consecutive lines**: none of `v9.1.0`,
`v10.1.0`, `v11.1.0`, `v12.1.0` or `v13.1.0` was released. 13.1 got further than the rest -
`v13.1.0-beta.1` and `beta.2` were tagged (fury `13.1.0b1`/`b2`) before #9518 re-rooted it to 14
- so a 13.1 beta pin is a dead end, not an early 13.1.

Four recent lines **did** ship a final: `v10.0.0` (2026-08-08), `v11.0.0` (2026-08-30),
`v12.0.0` (2026-09-17) and `v13.0.0` (2026-10-07). Each sits on a stabilization branch that is
**not an ancestor of `main`** - normal for a Lance final. `v13.0.0` is the 14 release root plus
two release commits, so **every v13-final change is also in every v14 beta**.

| Major | Its breaking theme |
|-------|--------------------|
| **v14** (current, `v14.0.0-beta.14`) | `ScalarIndexPlugin::load_index` takes `index_version` (#9518, re-rooted the major); jsonb vendored as `lance-jsonb` (#9534); compressed 2.3 mini-block offsets (#8331). Unlabeled: inline transaction moved to manifest field 23 plus `Operation::Unknown` (#9482). Delta below |
| **v13** (`v13.0.0`, 2026-10-07) | `WriteParams.file_writer_options` (#9192); lazy page-metadata init with a per-page metadata cache key (#7465); `json_extract`/`json_get` stop routing to JSON indices (#9101); BM25F (#7905); JSON accessor API (#9479); `merge_columns` field ids (#9547); duplicate-pair output (#9551); covering-fields spec split (#9159). Net-new: tagged FRI supported, fragment metadata tree spec, MinHash LSH. Delta below |
| **v12** (`v12.0.0`, 2026-09-17) | `WrappingObjectStore::wrap_paginated` (no default); MemWAL `ShardManifestStore` narrowed; `lance-namespace` response objects; predecessor-conditioned external stores; namespace merge-insert key lists; caller-provided Writer / `open_part` removed (#9072). Unlabeled but bigger: `stable` -> 2.2 and the IVF_RQ 5-bit default. Net-new: mixed data-file versions |
| **v11** (`v11.0.0`, 2026-08-30) | Fragment ids became a dataset-lifetime high-water mark; `lance-file` / `lance-encoding` reorganization; transaction code moved to `lance-table`. Net-new: covering indexes, `merge_insert` `write_mode`, row-address prefilter |
| **v10** | Blob APIs preserve null selections; cache keys became opaque BLAKE3 digests; async `create_remapper`; MemWAL renamed generation -> SSTable |
| **v9.1** (never released; renamed into v10) | FTS/inverted creation took a `block_size` param. Net-new: Data Overlay Files (unstable, env-gated), sparse structural pages, `lance-index-core` |
| **v9** | Python 3.9 dropped; `alter_columns` fails fast when casting an indexed column; FM-Index proto rename; FTS defaults to on-disk format v2 |
| **v8** | All index builds unified onto one segment-based lifecycle. Net-new: `lance-derive`, FM-Index, multi-bit IVF_RQ, `approx_mode`, TOS + GooseFS |
| **v7** | MemWAL, branches, the geo/RTree index, the `lance-select` crate, ICU FTS |

**`v13.0.0` is the stable pin** and what GitHub Releases marks `Latest`. crates.io carries
**finals only** (newest `lance 13.0.0`); PyPI `pylance` is likewise at `13.0.0`. A beta pin
means a git dependency - beta wheels publish to fury.io (`https://pypi.fury.io/lance-format`),
which carries `pylance-14.0.0b1` through `b14`.

Full per-tag deltas with every PR citation: `references/changelog-v7-v14.md`.

## Manifest feature flags

A reader or writer refuses a manifest that sets a bit it does not support. The supported set is
`FLAG_UNKNOWN - 1` minus whatever the build masks out, so **the docs table is not the authority -
`rust/lance-table/src/feature_flags.rs` is**. Bits 0-6 are long-standing (deletion files, stable
row IDs, the deprecated v2 marker, table config, base paths, disabled transaction file, and the
env-gated data overlay files). Everything from bit 7 up arrived in the last four majors, and
`FLAG_UNKNOWN` moved `1 << 9` (v12.0.0) -> `1 << 14` (v13.0.0) -> `1 << 16` (v14 line):

| Bit | Flag | Status at `v14.0.0-beta.14` |
|-----|------|-----------------------------|
| 7 | `FLAG_COVERED_INDEX_METADATA` | Supported since v11 final (#8535) |
| 8 | `FLAG_MIXED_DATA_FILE_VERSIONS` | Supported since v12 final; sticky; a half-set manifest is a hard error |
| 9 | `FLAG_FRAG_REUSE_WITH_STABLE_ROW_IDS` | **Debug builds only** - release builds refuse it. Docs say reader/writer `Yes` |
| 10 | `FLAG_FRAGMENT_REUSE_INDEX` (tagged FRI) | Supported since `v13.0.0` (#9420); sticky |
| 11 | `FLAG_UNSTABLE_SPILLED_ROW_LINEAGE` | Unstable: debug builds or `LANCE_ENABLE_UNSTABLE_SPILLED_ROW_LINEAGE` |
| 12 | `FLAG_FRAGMENT_TREE` | Reserved; rejected until implemented, though the spec exists (#9060) |
| 13 | `FLAG_INDEPENDENT_COVERING_FIELDS` | Reserved and **masked out**. Docs present it as normative |
| 14 | `FLAG_MANAGED_BLOBS` | v14 only; sticky; opt-in via `lance.blob.managed=true` since beta.14 |
| 15 | `FLAG_NON_REUSABLE_FIELD_IDS` | v14 only (#8658); writer-only, sticky |

"Sticky" means the bit survives every later commit (`STICKY_PAIRED_FLAGS` / `STICKY_WRITER_FLAGS`),
so setting it once permanently raises the reader floor of the table.

**Two pin windows open a table they should refuse - or brick it for older readers.**

- **Do not pin `v11.0.0-beta.4` through `beta.17`.** Those builds read bit 7 as a retired MemWAL
  flag they support, so they *open* a covering-index dataset instead of refusing it - wrong
  neighbours, no error.
- **Do not pin `v14.0.0-beta.4` through `beta.13`.** Any ordinary Blob v2 write there sets
  `FLAG_MANAGED_BLOBS`, which is sticky, so `v13.0.0` can never open the table again. #9891
  (beta.14) made it explicit opt-in - "Ordinary Blob V2 writes currently activate
  `FLAG_MANAGED_BLOBS` and require newer readers" - but **the beta.14 docs still describe
  automatic activation**. Trust the code.

**Covering indexes** (#8535) - `IndexMetadata.covering_fields` (proto field 11) names the
columns an index *carries* values for, so a query projecting only those is answered without a
base-table take. Under the contract every released build implements, **the carried columns are
the trailing suffix of `fields`** in emission order, and a column the index is both keyed on and
carries appears twice: "`fields = [1, 1]` with `covering_fields = [1]` is keyed on `1` and
carries `1`." #9159 specified an independent contract behind bit 13 (`fields` holds only keys),
but the build masks that bit, so it is spec only. Invalidation is wide either way: **any** index
whose `fields` include the updated column. V3 IVF auxiliary files can physically carry columns
("a reader discovers carried columns by exclusion, not by position"), bound by a
`covering_field_ids` metadata key, and coverage is per-segment. Section 11.

## The v13 final and v14 delta

**161 commits** from `v13.0.0-beta.4` to the `v13.0.0` final, then **137** more on the v14 line
to `beta.14`. **8 `breaking-change`-labeled PRs**: #7905, #9479, #9551, #9547, #9159 in v13
final; #9518, #9534, #8331 in v14. Invariants: **27 crates** (v14), **17 transaction ops**
(`Operation::Unknown`, v14), `CommitConfig.num_retries` still 20 (in v14 the retry clock starts
after catch-up, #9631), arrow 58 / datafusion 54.

**The label missed the format break again.** #9482 (v14.0.0-beta.2) moved the inline
transaction from manifest field 21 to **field 23**, added `Operation::Unknown` so unknown
operations decode instead of failing, and turned `From` conversions into `TryFrom` - labeled
`format-change`, never `breaking-change`, so the bot did not count it. Exhaustive `match` on
`Operation` fails to compile.

- **`merge_columns` field ids come from the manifest** (#9547) - the Rust `max_field_id`
  argument is gone. The bug it fixed made merged columns **silently read back as null**.
- **BM25F cross-field search** (#7905) - `combined_fields` / `CombinedFieldsQuery` scores one
  query across several FTS-indexed columns as a single field. Section 11.
- **JSON API moved** (#9479, #9534) - one accessor (`JsonTextStream::try_new`), `jsonb_to_json`
  removed, and jsonb vendored as `lance-jsonb` with its own spec (`format/table/jsonb.md`). The
  v13 rule still stands: only `json_get_int` / `_float` / `_bool` / `_string` reach a JSON index
  (#9101); `json_extract` / `json_get` silently full-scan.
- **`ScalarIndexPlugin::load_index` takes `index_version`** (#9518) - every third-party scalar
  index plugin fails to compile.
- **Duplicate-pair enumeration** (#9493) changed output order and cosine values in #9551.
- **Default IVF probing moved, unlabeled.** #9184 leaves probes unset so queries get adaptive
  bounds, #9467 recalibrated dot-product profiles, and #9798 (beta.12) added `search_effort`
  (0 to 1, resolved to 0.5; it overrides a fixed `nprobes`) and dropped the Python binding's
  `DEFAULT_NPROBES = 1` minimum-probes floor. Latency and recall at default settings shift
  across these upgrades - re-benchmark rather than carry numbers forward.

**Net-new format surface:** tagged FRI is supported (bit 10, #9420), extending reuse to
reclustering via `StablePartition`; the **fragment metadata tree** spec (#9060 -
`fragment_metadata.proto`, `Manifest` field 22, a `_bt/` directory; bit 12 still rejected);
**spilled row lineage** replacing the retired `DataFragment.external_*` fields (#9253, hidden
columns at field ids -3/-4/-5, unstable); **non-reusable field ids** (#8658, bit 15,
`max_allocated_field_id`); and an experimental **MinHash LSH** index (type 12, #9076/#9114).

**Behavior changes without a label:** explicit cloud storage options now beat env vars (#9719);
`fast_search` no longer adds `_rowid` (#9699); `main` is reserved as the default-branch alias
(#7180); Restore conflicts with an UpdateConfig touching schema or field metadata (#9361); raw
deep-Clone commits are rejected (#9555); the DataFusion memory pool is 150MB per *effective*
partition - all cores when `target_partitions` is unset - with per-execution pools (#9183,
#9408).

**Fixes needing a rebuild, rewrite or repair, not just an upgrade:** rewrite data hit by #9018,
#9239 (NULL written as 0) or #9394; rebuild scalar indexes hit by #9421 (segment merge produced an
empty index), #9352 (LABEL_LIST) or #9203; repair or restore after #9218, #9219, #9573, #9447,
#9176, #9595, #9085; v14: #9760, #9524, #9620, #7706, and #9857 (MemWAL, beta.9-12 only).
Address-domain indexes - **ZoneMap, BloomFilter and FM-Index** - still lose coverage of
compacted fragments rather than claim them (v14 recalculates instead when a fragment reuse index
is present, #9514), and one damaged under v10 or earlier needs recreating. Conditions for each in `references/changelog-v7-v14.md`.

**Still not landed:** the arrow 59 / DataFusion 55 / pyo3 0.29 upgrade. #8997 merged only as
"prepare" ("Not in this PR: the dependency bumps and lockfiles"); the real bump is #9647, open and
blocked on upstream releases. It also gates PyO3 RUSTSEC-2026-0176/0177, still outstanding.

## Earlier majors in brief

**v13 betas** - `WriteParams.file_writer_options` (#9192, struct literals fail to compile);
**lazy page-metadata init** (#7465): `StructuralFieldScheduler::initialize` takes
`requested_ranges`, and caching moved from per-column `FieldDataCacheKey` to per-page
`PageDataCacheKey`, so **every warm or persisted metadata cache cold-misses** across the upgrade -
the payoff is that "a cold point/range read's metadata IO is invariant to the column's total page
count". The FRI got a versioned on-disk contract (#9136; stable row IDs and tagged FRI
mutually exclusive outside debug builds). `inline_optimization_enabled` flipped `true` -> `false`
(#9180). `hf://` object store (#9236).

**v12** - mixed data-file versions shipped (#8581-#8585); `stable` -> 2.2 (#8657); **IVF_RQ
defaults to 5 bits**, ~4.4x the 1-bit index size (#8936; set `num_bits=1` to opt out);
`WrappingObjectStore` implementors must add `wrap_paginated` - "There is deliberately no default"
- and paged listing via `read_dir_page` means walking until the token is `None`, never until a
page comes back short; namespace merge-insert keys have **arity-dependent NULL semantics** (a
composite key's NULL "matches nothing"); `lance-namespace` pins now differ by language (Rust
0.13.0, Java 0.12.0 since #8979, Python `>=0.13.0,<0.14` in v14 - re-check rather than carry forward).

**v11** - fragment ids are a dataset-lifetime high-water mark (#8206): overwrite no longer
restarts at 0, and datasets with duplicate ids read but no longer commit. `LanceFileVersion` lost
`Ord` (#8027/#8028). `Operation::Project` / `Merge` gained `preserves_nullability` (#8347). The
external-manifest protocol made object storage authoritative (#8499) - a retained ETag makes
readers reject a good manifest with `Manifest e_tag mismatch`. #8101 silently duplicated rows on
nullable primary keys under repeat `merge_insert`; existing duplicates must be removed by hand.
Rebuild HNSW indexes hit by #8834 (v11) or #9053 (v13 betas).

## Performance questions

For anything performance-shaped - slow scans or searches, remote/object-storage cost, index
maintenance cost, memory sizing, version bloat, benchmarking - load
`references/performance.md` first. Part A routes to the official guidance plus the undocumented
source-derived changes; Part B is field-verified practice against S3-compatible storage. The
governing rule stays **minimize remote calls** - fewer commits, fewer scans, fewer round trips -
because that is where the order-of-magnitude wins are. The official **"Tuning remote scans"**
section gives a starting point for cross-region or public-internet access, where the cloud
default of 64 concurrent requests is too aggressive: `LANCE_IO_THREADS=8`,
`fragment_readahead=1`, `batch_readahead=2`, `io_buffer_size=64MB`. It is a legitimate second
move once call volume is already minimized.

**AMX-FP16** (#8540) is the one performance change that alters *results*, not just speed: where
it engages, IVF partition assignment becomes **exact instead of approximate**, so recall improves
*and* assignments differ from an older build. It is shape-gated (`float16` + `dot`,
`dimension >= 32`, `num_centroids >= 32`). `LANCE_DISABLE_AMX=1` disables it and reverts
assignment to the approximate path too.

Two cache facts to know before tuning anything remote: Lance has **no resident data cache** (a
`Session` holds only index and metadata caches, never decoded values, so repeated point reads
re-pay object-store IO), and one `Arc<Session>` shared via `DatasetBuilder::with_session` lets
datasets share it. Cold first search is dominated by paging indexes in - `prewarm_index` is the
remedy, and for FTS it loads the whole inverted index into RAM. #7465 changed the metadata cache
key shape, so the first run after a v13 upgrade re-pays that paging even against a warm cache.

**Layout debt is version-dependent.** Binary-copy compaction of small appends preserves tiny
pages; a reader that pays per page (v12) turns that into tens of thousands of GETs per lookup.
Re-encode append-heavy tables, and benchmark on a fresh copy of the real store - details and
measurements in `references/performance.md` Part B.

**Time travel is not an archive mechanism.** Versions look like free history, but the default
cleanup reclaims anything older than 7 days and cleanup is part of routine optimize - so a design
that treats old versions as the durable record loses it on the first maintenance pass. Keep an
explicit archive if you need one. The reverse also holds: `delete` alone does not erase data -
old versions, unrewritten fragments, and indexes (deletes do not conflict with index creation)
all keep it until rewritten, cleaned up, and rebuilt.

## Official docs mirror

`references/docs/` mirrors `docs/src` of `lance-format/lance` at the tracked tag, verbatim -
49 markdown files plus 5 diagrams, all directly readable.

| Directory | Files | Covers |
|-----------|-------|--------|
| `guide/` | 14 | CRUD, performance, object store, distributed write + indexing, JSON, tokenizers, data types, data evolution, blob, arrays, tags/branches, migration, observability |
| `quickstart/` | 4 | First dataset, vector search, full-text search, versioning |
| `format/` | 1 | Spec-stack overview |
| `format/file/` | 3 | Container spec, structural encodings + compression, format versions |
| `format/table/` | 11 + 1 svg | Layout, schema, transactions (**conflict-resolution matrix**), versioning, row-id lineage, branch/tag, MemWAL, data overlay files, fragment metadata tree, JSONB |
| `format/index/` | 1 + 4 svg | Index lifecycle, fragment coverage, compaction interplay, covering contracts |
| `format/index/scalar/` | 11 | fts, fmindex, ngram, btree, bitmap, bloom_filter, label_list (`array_has_any/all`), zonemap, rtree, json, minhash_lsh |
| `format/index/vector/` | 1 | IVF / PQ / SQ / RQ / HNSW concepts and storage layout |
| `format/index/system/` | 2 | Fragment reuse index, MemWAL system index |
| `integrations/` | 1 | DataFusion SQL over Lance, incl. JSON functions |

**Not mirrored:** `docs/src/images/` and the `format/*.png` assets, so image links in the
mirrored pages do not resolve - the prose is self-contained, and the five `.svg` diagrams *are*
mirrored. Also out by design: `community/`, `examples/`,
`integrations/{index,pytorch,tensorflow}.md`; and the landing stubs and contributor files
(`format/AGENTS.md`).

**A whole tier of docs is not in this repo at all**, so it cannot be mirrored and cannot be
enumerated from a clone. `docs/make-full-website.sh` assembles `format/catalog`,
`format/namespace`, and the `integrations/{duckdb,huggingface,spark,ray,trino,context}` sections
at build time from six sibling repos with their own version lines - the checked-in
`integrations/index.md` links `spark/`, `duckdb` and `trino` as if they were local, but those
paths do not exist in the tree. **Lance Context** and the **HuggingFace** integration docs are
whole nav sections that exist only on the built site. For any of those, read `lance.org` rather
than this mirror. Protobuf message bodies are likewise expanded at build time from `protos/` by
`mkdocs_protobuf`, so the mirrored spec pages show `%%% proto.message.X %%%` placeholders where
the site shows a rendered schema.
Pliki w paczce (65)
  • CHANGELOG.md 62 KB
  • LICENSE.txt 9 KB
  • SKILL.md 26 KB
  • references/changelog-v7-v14.md 124 KB
  • references/format-file.md 47 KB
  • references/format-table.md 83 KB
  • references/indexes.md 71 KB
  • references/lance-reference.md 1 KB
  • references/maintenance.md 6 KB
  • references/ops.md 31 KB
  • references/performance.md 79 KB
  • references/docs/format/index.md 3 KB
  • references/docs/format/file/encoding.md 49 KB
  • references/docs/format/file/index.md 9 KB
  • references/docs/format/file/versioning.md 4 KB
  • references/docs/format/index/index.md 18 KB
  • references/docs/format/index/indices-compaction.drawio.svg 50 KB
  • references/docs/format/index/indices-fragment handling.drawio.svg 21 KB
  • references/docs/format/index/scalar_index.drawio.svg 8 KB
  • references/docs/format/index/starter-example.drawio.svg 14 KB
  • references/docs/format/index/scalar/bitmap.md 1 KB
  • references/docs/format/index/scalar/bloom_filter.md 5 KB
  • references/docs/format/index/scalar/btree.md 3 KB
  • references/docs/format/index/scalar/fmindex.md 5 KB
  • references/docs/format/index/scalar/fts.md 19 KB
  • references/docs/format/index/scalar/json.md 4 KB
  • references/docs/format/index/scalar/label_list.md 2 KB
  • references/docs/format/index/scalar/minhash_lsh.md 22 KB
  • references/docs/format/index/scalar/ngram.md 2 KB
  • references/docs/format/index/scalar/rtree.md 8 KB
  • references/docs/format/index/scalar/zonemap.md 3 KB
  • references/docs/format/index/system/frag_reuse.md 11 KB
  • references/docs/format/index/system/mem_wal.md 1 KB
  • references/docs/format/index/vector/index.md 21 KB
  • references/docs/format/table/branch_tag.md 5 KB
  • references/docs/format/table/data_overlay_file.md 17 KB
  • references/docs/format/table/fragment_metadata.md 22 KB
  • references/docs/format/table/fragment_metadata_tree.svg 3 KB
  • references/docs/format/table/index.md 11 KB
  • references/docs/format/table/jsonb.md 25 KB
  • references/docs/format/table/layout.md 10 KB
  • references/docs/format/table/mem_wal.md 36 KB
  • references/docs/format/table/row_id_lineage.md 18 KB
  • references/docs/format/table/schema.md 21 KB
  • references/docs/format/table/transaction.md 32 KB
  • references/docs/format/table/versioning.md 6 KB
  • references/docs/guide/arrays.md 6 KB
  • references/docs/guide/blob.md 22 KB
  • references/docs/guide/data_evolution.md 9 KB
  • references/docs/guide/data_types.md 14 KB
  • references/docs/guide/distributed_indexing.md 8 KB
  • references/docs/guide/distributed_write.md 11 KB
  • references/docs/guide/json.md 12 KB
  • references/docs/guide/migration.md 5 KB
  • references/docs/guide/object_store.md 35 KB
  • references/docs/guide/observability.md 4 KB
  • references/docs/guide/performance.md 31 KB
  • references/docs/guide/read_and_write.md 22 KB
  • references/docs/guide/tags_and_branches.md 5 KB
  • references/docs/guide/tokenizer.md 4 KB
  • references/docs/integrations/datafusion.md 4 KB
  • references/docs/quickstart/full-text-search.md 18 KB
  • references/docs/quickstart/index.md 3 KB
  • references/docs/quickstart/vector-search.md 10 KB
  • references/docs/quickstart/versioning.md 4 KB

Wersje

RepozytoriumWynikLicencja
Ta strona · tenequm/skills100MIT
tenequm/claude-plugins100MIT

Migawka bazy z 13.11.2026 · wersja paczki 59133edf3f47 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100lancedb
    Pomaga pracować z bazą wektorową LanceDB: przechowywanie oparte na Arrow, wersjonowanie, wyszukiwanie hybrydowe tekstowe i wektorowe, indeksy IVF_PQ i HNSW oraz integracja z pandas i polars.
  • 100lancedb
    Pomaga korzystać z wbudowanej bazy wektorowej LanceDB bez serwera: tabele, wyszukiwanie wektorowe i pełnotekstowe, filtry, indeksy oraz funkcje osadzeń, z klientami Node.js, Python i Rust.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.