Skip to content
Skills Library
Programming · Building & coding

tinygrad

/tinygrad

Deep learning framework development with tinygrad - a minimal tensor library with autograd, JIT compilation, and multi-device support. Use when writing neural networks, training models, implementing tensor operations, working with UOps/PatternMatcher for graph transformations, or contributing to tinygrad internals. Triggers on tinygrad imports, Tensor operations, nn modules, optimizer usage, schedule/codegen work, or device backends.

Author
av ↗
Repository
skills ↗
Licence
MIT
Last repository change
21.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
2
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 21.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Deep learning framework development with tinygrad - a minimal tensor library with autograd, JIT compilation, and multi-device support. Use when writing neural networks, training models, implementing tensor operations, working with UOps/PatternMatcher for graph transformations, or contributing to tinygrad internals. Triggers on tinygrad imports, Tensor operations, nn modules, optimizer usage, schedule/codegen work, or device backends.

Original author description

Deep learning framework development with tinygrad - a minimal tensor library with autograd, JIT compilation, and multi-device support. Use when writing neural networks, training models, implementing tensor operations, working with UOps/PatternMatcher for graph transformations, or contributing to tinygrad internals. Triggers on tinygrad imports, Tensor operations, nn modules, optimizer usage, schedule/codegen work, or device backends.

How it works

  1. Quick Reference
  2. Architecture Pipeline
  3. Training Loop Pattern
  4. Model Definition
  5. State Dict / Weights
  6. JIT Compilation
  7. Device Management
  8. Environment Variables
  9. Debugging
  10. UOp and PatternMatcher (Internals)
Original SKILL.md headings · 166 lines, 517 words
SKILL.md preview
---
name: tinygrad
description: Deep learning framework development with tinygrad - a minimal tensor library with autograd, JIT compilation, and multi-device support. Use when writing neural networks, training models, implementing tensor operations, working with UOps/PatternMatcher for graph transformations, or contributing to tinygrad internals. Triggers on tinygrad imports, Tensor operations, nn modules, optimizer usage, schedule/codegen work, or device backends.
---

# tinygrad

A minimal deep learning framework focused on beauty and minimalism. Every line must earn its keep.

## Quick Reference

```python
from tinygrad import Tensor, TinyJit, nn, dtypes, Device, GlobalCounters

# Tensor creation
x = Tensor([1, 2, 3])
x = Tensor.rand(2, 3)
x = Tensor.kaiming_uniform(128, 784)

# Operations are lazy until realized
y = (x + 1).relu().sum()
y.realize()  # or y.numpy()

# Training context
with Tensor.train():
  loss = model(x).sparse_categorical_crossentropy(labels).backward()
  optim.step()
```

## Architecture Pipeline

1. **Tensor** (`tinygrad/tensor.py`) - User API, creates UOp graph
2. **UOp** (`tinygrad/uop/ops.py`) - Unified IR for all operations
3. **Schedule** (`tinygrad/engine/schedule.py`) - Converts tensor UOps to kernel UOps
4. **Codegen** (`tinygrad/codegen/`) - Converts kernel UOps to device code
5. **Runtime** (`tinygrad/runtime/`) - Device-specific execution

## Training Loop Pattern

```python
from tinygrad import Tensor, TinyJit, nn
from tinygrad.nn.datasets import mnist

X_train, Y_train, X_test, Y_test = mnist()
model = Model()
optim = nn.optim.Adam(nn.state.get_parameters(model))

@TinyJit
@Tensor.train()
def train_step():
  optim.zero_grad()
  samples = Tensor.randint(512, high=X_train.shape[0])
  loss = model(X_train[samples]).sparse_categorical_crossentropy(Y_train[samples]).backward()
  return loss.realize(*optim.schedule_step())

for i in range(100):
  loss = train_step()
```

## Model Definition

Models are plain Python classes with `__call__`. No base class required.

```python
class Model:
  def __init__(self):
    self.l1 = nn.Linear(784, 128)
    self.l2 = nn.Linear(128, 10)
  def __call__(self, x):
    return self.l1(x).relu().sequential([self.l2])
```

**Available nn modules:** `Linear`, `Conv2d`, `BatchNorm`, `LayerNorm`, `RMSNorm`, `Embedding`, `GroupNorm`, `LSTMCell`

**Optimizers:** `SGD`, `Adam`, `AdamW`, `LARS`, `LAMB`, `Muon`

## State Dict / Weights

```python
from tinygrad.nn.state import safe_save, safe_load, get_state_dict, load_state_dict, get_parameters

# Save/load safetensors
safe_save(get_state_dict(model), "model.safetensors")
load_state_dict(model, safe_load("model.safetensors"))

# Get all trainable params
params = get_parameters(model)
```

## JIT Compilation

`TinyJit` captures and replays kernel graphs. Input shapes must be fixed.

```python
@TinyJit
def forward(x):
  return model(x).realize()

# First call captures, subsequent calls replay
out = forward(batch)
```

## Device Management

```python
from tinygrad import Device
print(Device.DEFAULT)  # Auto-detected: METAL, CUDA, AMD, CPU, etc.

# Force device
x = Tensor.rand(10, device="CPU")
x = x.to("CUDA")
```

## Environment Variables

| Variable | Values | Description |
|----------|--------|-------------|
| `DEBUG` | 1-7 | Increasing verbosity (4=code, 7=asm) |
| `VIZ` | 1 | Graph visualization |
| `BEAM` | # | Kernel beam search width |
| `NOOPT` | 1 | Disable optimizations |
| `SPEC` | 1-2 | UOp spec verification |

## Debugging

```bash
# Visualize computation graph
VIZ=1 python -c "from tinygrad import Tensor; Tensor.ones(10).sum().realize()"

# Show generated code
DEBUG=4 python script.py

# Run tests
python -m pytest test/test_tensor.py -xvs
```

## UOp and PatternMatcher (Internals)

UOps are immutable, cached graph nodes. Use PatternMatcher for transformations:

```python
from tinygrad.uop.ops import UOp, Ops
from tinygrad.uop.upat import UPat, PatternMatcher, graph_rewrite

pm = PatternMatcher([
  (UPat(Ops.ADD, src=(UPat.cvar("x"), UPat.cvar("x"))), lambda x: x * 2),
])
result = graph_rewrite(uop, pm)
```

**Key UOp properties:** `op`, `dtype`, `src`, `arg`, `tag`

**Define PatternMatchers at module level** - they're slow to construct.

## Style Guide

- 2-space indentation, 150 char line limit
- Prefer readability over cleverness
- Never mix functionality changes with whitespace changes
- All functionality changes must be tested
- Run `pre-commit run --all-files` before commits

## Testing

```bash
python -m pytest test/test_tensor.py -xvs
python -m pytest test/unit/test_schedule_cache.py -x --timeout=60
SPEC=2 python -m pytest test/test_something.py  # With spec verification
```
Files in this package (2)
  • README.md 1 KB
  • SKILL.md 5 KB

Versions

The database has no other skills named tinygrad.

Identical content appears in 1 other GitHub folders. We show one copy with the highest score.

Database snapshot: 13.11.2026 · package version 969a46872895 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100jax
    Expert JAX assistance covering Autograd, XLA compilation (`jit`), vectorization (`vmap`), and parallelization (`pmap`). Use when building high-performance numerical computing and cutting-edge deep learning research.
  • 100pytorch-deep-learning-fundamentals
    Use this skill when building and training custom neural network architectures using PyTorch tensors, autograd, and DataLoader modules.
  • 100jit-compilation
    JIT compilation skill for runtime code generation. Use when building LLVM ORC JIT, LLJIT, Cranelift JIT, inline caches, trampolines, or Rust dynasm codegen. Activates on queries about ORC JIT, LLJIT, Cranelift, ExecutionSession, inline cache, W^X, or dynasm.
  • 96cuda-streams
    Manage tensor lifetimes across CUDA streams and decide whether record_stream is avoidable. Use when using a tensor on a stream other than the one it was allocated on, adding or reviewing Tensor.record_stream / CUDACachingAllocator::recordStream, side streams, wait_stream, or…
  • 100jit-regression-test
    Extract a standalone JIT regression test case from a given GitHub issue and save it under a JIT regression runner folder. USE FOR: creating JIT regression tests, extracting repro code from dotnet/runtime issues, "write a test for this JIT bug", "create a regression test for…
  • 100pytorch
    Expert PyTorch deep learning assistance covering dynamic tensors, autograd, torch.compile, distributed training (DDP), and GPU acceleration. Use when training neural networks and deploying AI research.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.