parallel-processing

What it does and when to use it

Parallel processing with joblib for grid search and batch computations. Use when speeding up computationally intensive tasks across multiple CPU cores.

How it works

1. Basic Usage
2. Key Parameters
3. Grid Search Example
4. Pre-computing Shared Data
5. Performance Tips

---
name: parallel-processing
description: Parallel processing with joblib for grid search and batch computations. Use when speeding up computationally intensive tasks across multiple CPU cores.
---

# Parallel Processing with joblib

Speed up computationally intensive tasks by distributing work across multiple CPU cores.

## Basic Usage

```python
from joblib import Parallel, delayed

def process_item(x):
    """Process a single item."""
    return x ** 2

# Sequential
results = [process_item(x) for x in range(100)]

# Parallel (uses all available cores)
results = Parallel(n_jobs=-1)(
    delayed(process_item)(x) for x in range(100)
)
```

## Key Parameters

- **n_jobs**: `-1` for all cores, `1` for sequential, or specific number
- **verbose**: `0` (silent), `10` (progress), `50` (detailed)
- **backend**: `'loky'` (CPU-bound, default) or `'threading'` (I/O-bound)

## Grid Search Example

```python
from joblib import Parallel, delayed
from itertools import product

def evaluate_params(param_a, param_b):
    """Evaluate one parameter combination."""
    score = expensive_computation(param_a, param_b)
    return {'param_a': param_a, 'param_b': param_b, 'score': score}

# Define parameter grid
params = list(product([0.1, 0.5, 1.0], [10, 20, 30]))

# Parallel grid search
results = Parallel(n_jobs=-1, verbose=10)(
    delayed(evaluate_params)(a, b) for a, b in params
)

# Filter results
results = [r for r in results if r is not None]
best = max(results, key=lambda x: x['score'])
```

## Pre-computing Shared Data

When all tasks need the same data, pre-compute it once:

```python
# Pre-compute once
shared_data = load_data()

def process_with_shared(params, data):
    return compute(params, data)

# Pass shared data to each task
results = Parallel(n_jobs=-1)(
    delayed(process_with_shared)(p, shared_data)
    for p in param_list
)
```

## Performance Tips

- Only worth it for tasks taking >0.1s per item (overhead cost)
- Watch memory usage - each worker gets a copy of data
- Use `verbose=10` to monitor progress

Versions

Identical content appears in 1 other GitHub folders. We show one copy with the highest score.

Database snapshot: 13.11.2026 · package version b42f0762ee93 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

• 100parallel-grid-search
  How to parallelize hyperparameter grid search using joblib. Use this skill whenever the user needs to evaluate many hyperparameter combinations (e.g., DBSCAN epsilon, min_samples, shape_weight) efficiently across CPU cores. Covers joblib.Parallel, parameter grid generation with…
• 84processing-parallel-tasks
  Implements parallel processing using Parallel, PLINQ, and ConcurrentCollections in .NET. Use when processing CPU-bound tasks in parallel or improving multi-core utilization.
• 95get-available-resources
  This skill should be used at the start of any computationally intensive scientific task to detect and report available system resources (CPU cores, GPUs, memory, disk space). It creates a JSON file with resource information and strategic recommendations that inform computational…
• 100get-available-resources
  This skill should be used at the start of any computationally intensive scientific task to detect and report available system resources (CPU cores, GPUs, memory, disk space). It creates a JSON file with resource information and strategic recommendations that inform computational…
• 100web-workers-multithreading
  Use this skill when offloading CPU-intensive computations, cryptography, image processing, or data transformations from the main UI thread to Web Workers.
• 100media-batch
  Batch ffmpeg and media processing at scale with GNU parallel, xargs, and shell patterns: parallel transcode fleets, shard jobs across cores, log aggregation, retry logic, idempotent pipelines, find + parallel recipes, progress reporting. Use when the user asks to batch transcode…