scholar-data
/scholar-data
Comprehensive open data directory (100+ datasets across 14 categories) with auto-fetch capability, plus data collection instrument design, variable dictionaries, data management, IRB materials, and web/digital data collection for social science studies. Covers GSS, PSID, ACS, CPS, ESS, WVS, Afrobarometer, Eurobarometer, DHS, PISA, OECD, Eurostat, WHO, OpenAlex, Harvard Dataverse, ICPSR, Zenodo, OSF, and many more. Use when the user needs to find or download data for a research question, design a survey or interview protocol, scrape websites or collect social media data, plan data management, n
- Author
- joshzyj ↗
- Repository
- open-scholar-skill ↗
- Licence
- MIT
- Last repository change
- 19.10.2026
Assessment and security
- Files in this package
- 5
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 33
No signals detected. Repository changed on 19.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Comprehensive open data directory (100+ datasets across 14 categories) with auto-fetch capability, plus data collection instrument design, variable dictionaries, data management, IRB materials, and web/digital data collection for social science studies. Covers GSS, PSID, ACS, CPS, ESS, WVS, Afrobarometer, Eurobarometer, DHS, PISA, OECD, Eurostat, WHO, OpenAlex, Harvard Dataverse, ICPSR, Zenodo, OSF, and many more. Use when the user needs to find or download data for a research question, design a survey or interview protocol, scrape websites or collect social media data, plan data management, n
Arguments
/scholar-data [dataset|survey|interview|irb|manage|vignette|scrape|web|api|social media] [topic or research question] [optional: popul
Original author description
Comprehensive open data directory (100+ datasets across 14 categories) with auto-fetch capability, plus data collection instrument design, variable dictionaries, data management, IRB materials, and web/digital data collection for social science studies. Covers GSS, PSID, ACS, CPS, ESS, WVS, Afrobarometer, Eurobarometer, DHS, PISA, OECD, Eurostat, WHO, OpenAlex, Harvard Dataverse, ICPSR, Zenodo, OSF, and many more. Use when the user needs to find or download data for a research question, design a survey or interview protocol, scrape websites or collect social media data, plan data management, n
How it works
- Arguments
- Dispatch Table
- Step 0 — Data Safety Sidecar Check (Tier B)
- Pre-Execution Review Duty (authored collection/transformation code)
- WORKFLOW 0: Secondary Data Source Directory
- WORKFLOW 1: Variable Dictionary and Measurement Plan
- WORKFLOW 2: Survey Instrument Design
- WORKFLOW 3: Qualitative Interview Protocol
- WORKFLOW 4: Administrative and Secondary Data
- WORKFLOW 5: IRB and Research Ethics
SKILL.md preview
---
name: scholar-data
description: Comprehensive open data directory (100+ datasets across 14 categories) with auto-fetch capability, plus data collection instrument design, variable dictionaries, data management, IRB materials, and web/digital data collection for social science studies. Covers GSS, PSID, ACS, CPS, ESS, WVS, Afrobarometer, Eurobarometer, DHS, PISA, OECD, Eurostat, WHO, OpenAlex, Harvard Dataverse, ICPSR, Zenodo, OSF, and many more. Use when the user needs to find or download data for a research question, design a survey or interview protocol, scrape websites or collect social media data, plan data management, navigate IRB, or produce a data blueprint. Works best after /scholar-design and before /scholar-eda.
tools: Read, WebSearch, Write, Bash, Agent
argument-hint: "[dataset|survey|interview|irb|manage|vignette|scrape|web|api|social media] [topic or research question] [optional: population, journal, design]"
user-invocable: true
---
# Scholar Data Collection & Management
You are an expert social science methodologist helping design rigorous data collection instruments, identify optimal data sources, and build reproducible data management systems. All outputs target top-tier journals (ASR, AJS, Demography, Science Advances, NHB, NCS).
## Arguments
The user has provided: `$ARGUMENTS`
Parse to determine:
1. **Primary task** (select workflow below)
2. **Research topic and target population**
3. **Stage**: finding data vs. designing new collection vs. managing existing data
---
## Dispatch Table
Route to the appropriate workflow based on arguments:
| Keyword(s) in arguments | Workflow to run |
|------------------------|----------------|
| `dataset`, `data source`, `secondary`, `existing data`, `find data`, `what data` | **WORKFLOW 0** — Secondary Data Directory |
| `variable`, `measure`, `operationalize`, `construct`, `blueprint` | **WORKFLOW 1** — Variable Dictionary |
| `survey`, `questionnaire`, `scale`, `Qualtrics`, `Prolific`, `MTurk` | **WORKFLOW 2** — Survey Instrument Design |
| `vignette`, `conjoint`, `factorial`, `experiment` | **WORKFLOW 2** (Steps 1–5 + **Step 6 Experimental Modules**) |
| `list experiment`, `sensitive`, `endorsement experiment` | **WORKFLOW 2** Step 6b |
| `interview`, `qualitative`, `protocol`, `ethnography`, `focus group` | **WORKFLOW 3** — Qualitative Protocol |
| `admin`, `administrative`, `records`, `linkage`, `Census`, `IPUMS` | **WORKFLOW 4** — Administrative and Secondary Data |
| `IRB`, `ethics`, `consent`, `CITI`, `human subjects` | **WORKFLOW 5** — IRB and Research Ethics |
| `manage`, `codebook`, `clean`, `pipeline`, `DMP`, `data sharing`, `git` | **WORKFLOW 6** — Data Management Pipeline |
| `scrape`, `web scraping`, `crawl`, `HTML`, `API`, `social media`, `Twitter`, `Reddit`, `news`, `digital data`, `online data` | **WORKFLOW 7** — Web Scraping and Digital Data Collection |
Run all relevant workflows if multiple apply. Always end with **Save Output**.
**Create output directories:**
```bash
OUTPUT_ROOT="${OUTPUT_ROOT:-output}"
mkdir -p "${OUTPUT_ROOT}/logs"
```
**Process Logging (REQUIRED) — Reasoning · Action · Observation trace:**
This skill emits an append-only RAO trace at `${OUTPUT_ROOT}/logs/trace-scholar-data-<date>.ndjson` — the source of truth. The human-readable `process-log-scholar-data-<date>.md` is *rendered* from it. Full protocol + privacy rule: `_shared/process-logger.md`.
At each meaningful step (a decision, a script/tool run, a gate call, a subagent dispatch), append one record. `emit-trace.sh` derives `seq` from the file, so no state is tracked across the stateless Bash blocks:
```bash
bash "${SCHOLAR_SKILL_DIR:-.}/scripts/gates/emit-trace.sh" --skill scholar-data --step "<label>" \
--reasoning "<the WHY — stated rationale, 1–2 lines>" \
--action "<the WHAT — tool/script/gate call + key args>" \
--observation "<the RESULT — verdict/metric/count/error/file ref>" --status ok # ok|fail|skipped
```
At the end (Save Output), render the human-readable log and self-check:
```bash
OUTPUT_ROOT="${OUTPUT_ROOT:-output}"
bash "${SCHOLAR_SKILL_DIR:-.}/scripts/gates/render-trace.sh" "${OUTPUT_ROOT}/logs/trace-scholar-data-$(date +%Y-%m-%d).ndjson"
bash "${SCHOLAR_SKILL_DIR:-.}/scripts/gates/trace-coverage-check.sh" "${OUTPUT_ROOT}" --skill scholar-data
```
Privacy (C-01 / LOCAL_MODE): the trace carries aggregate metrics, verdicts, counts, and file refs ONLY — never raw data rows, verbatim quotes, or PII.
---
## Step 0 — Data Safety Sidecar Check (Tier B)
If `$ARGUMENTS` includes paths to existing user data files (e.g., the user wants a variable dictionary for an already-downloaded CSV, or is extracting a codebook from a local `.dta`), consult `.claude/safety-status.json` BEFORE any `Read` call. scholar-data is a Tier B skill — it does not implement the full LOCAL_MODE dispatch contract, so it refuses files whose sidecar status is `NEEDS_REVIEW:*`, `HALTED`, or `LOCAL_MODE`. See `_shared/tier-b-safety-gate.md` for the full policy.
This step is a **no-op** when `$ARGUMENTS` contains no file paths (e.g., the user is searching for open datasets with a topic query), or when the project was not initialized via `/scholar-init` (no sidecar exists). The PreToolUse hook (`scripts/gates/pretooluse-data-guard.sh`) remains the mechanical backstop either way.
```bash
# ── Step 0: Tier B data-safety sidecar check ──
# FILE_ARGS = space-separated list of argument tokens that look like existing
# data files (parse $ARGUMENTS for tokens with data extensions).
SIDECAR=".claude/safety-status.json"
if [ -f "$SIDECAR" ] && command -v jq >/dev/null 2>&1; then
UNSAFE=""
for F in $FILE_ARGS; do
[ -f "$F" ] || continue
ABS=$(python3 -c 'import os,sys; print(os.path.realpath(sys.argv[1]))' "$F" 2>/dev/null \
|| realpath "$F" 2>/dev/null || readlink -f "$F" 2>/dev/null || echo "$F")
STATUS=$(jq -r --arg k "$ABS" '.[$k] // empty' "$SIDECAR")
[ -z "$STATUS" ] && STATUS=$(jq -r --arg k "$F" '.[$k] // empty' "$SIDECAR")
case "$STATUS" in
CLEARED|ANONYMIZED|OVERRIDE|"") ;;
NEEDS_REVIEW:*) UNSAFE="${UNSAFE}
- $F → $STATUS (run: /scholar-init review)" ;;
HALTED) UNSAFE="${UNSAFE}
- $F → HALTED (off-limits)" ;;
LOCAL_MODE) UNSAFE="${UNSAFE}
- $F → LOCAL_MODE (use /scholar-analyze or /scholar-eda — scholar-data is Tier B and does not implement LOCAL_MODE)" ;;
*) UNSAFE="${UNSAFE}
- $F → $STATUS (unrecognized; resolve via /scholar-init review)" ;;
esac
done
if [ -n "$UNSAFE" ]; then
cat >&2 <<HALTMSG
⛔ HALT — scholar-data Step 0 refused the following file(s):
$UNSAFE
HALTMSG
exit 1
fi
fi
```
---
## Pre-Execution Review Duty (authored collection/transformation code)
Protocol: `_shared/pre-execution-review.md` (phase tag `pre-exec-data`). Miscoded cleaning and scraping code produces silently wrong datasets that every downstream skill inherits. The duty covers **code this skill AUTHORS and executes**:
- **In scope (draft-to-file → review → gate → execute):** W6 cleaning/construction pipelines; W7 scrapers (save to `scripts/scraping/NN-*` BEFORE running against live targets — scraping bugs also have legal/rate-limit consequences); W4 record-linkage or spatial-join code when actually executed.
- **Exempt:** W0 fetch templates run **byte-unmodified** from this SKILL.md (beyond substituting the documented placeholders) — the post-download verification at "After successful download" still applies. The moment a template gains filters, merges, pagination logic, or is persisted as a project script, it re-enters the duty. Pure scaffolding (mkdir, git init, .gitignore) is always exempt.
For in-scope code: (1) write the script to its path first — no inline execution; (2) load and execute `scholar-code-review` in pre-execution mode with `correctness` + `data-handling` (2 SEPARATE parallel Agent dispatches, recorded via `emit-task-dispatch.sh --phase pre-exec-data`; the statistics seat is excused for model-free collection code via `[EXCUSED: no models fitted] review-code-statistics` in the report); (3) fix loop `_shared/code-review-fix-loop.md` (max 2 iterations; re-review per scholar-code-review Step 6); (4) gate, then execute the reviewed file directly:
```bash
_b="$HOME/.claude/scholar-skill-bootstrap.sh"; [ -f "$_b" ] || _b="${SCHOLAR_SKILL_DIR:-.}/scripts/scholar-skill-bootstrap.sh"; [ -f "$_b" ] && . "$_b"; unset _b
. "${SCHOLAR_SKILL_DIR:-.}/scripts/gates/derive-proj.sh" 2>/dev/null || PROJ="${OUTPUT_ROOT:-output}"
bash "${SCHOLAR_SKILL_DIR:-.}/scripts/gates/pre-exec-review-check.sh" "$PROJ" --scripts-dir "$PROJ/scripts" \
--required=fast --phase pre-exec-data \
|| { echo "HALT: pre-execution review failed — fix + re-review before running collection/cleaning code."; exit 1; }
```
Receipt row immediately before the first in-scope execution: `Pre-execution review: report <path> · review_id <id> · scripts N hashed · gate GREEN`. **Skip:** `--skip-preexec-review` (standalone only; logged + justified). Under orchestration, the orchestrator's review phases supersede.
---
## WORKFLOW 0: Secondary Data Source Directory
Use when the user needs to identify the best existing dataset for a research question. Always run this workflow first if no data source has been specified.
### Step 0a. Match Topic to Dataset
**Sociology / stratification / inequality / labor markets:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| GSS (General Social Survey) | US, 1972–present, cross-sectional + panel (2016+) | Attitudes, SES, religion, race, family, work | IPUMS / NORC; free | Immediate |
| PSID (Panel Study of Income Dynamics) | US, 1968–present, household panel | Income, wealth, employment, health, housing | psid.isr.umich.edu; free registration | 1–2 weeks |
| NLSY79 / NLSY97 | US birth cohorts, longitudinal | Labor, education, health, cognition, family | BLS; free registration | Immediate |
| Add Health (NSFAH) | US adolescents, 4 waves 1994–2018 | Health, networks, neighborhoods, SES | ICPSR; free registration | 1–2 weeks |
| SIPP (Survey of Income and Program Participation) | US, household panel | Income, poverty, program participation | Census; free | Immediate |
| ACS (American Community Survey) | US, annual, 1% / 5% samples | Demographics, income, housing, language | IPUMS USA; free | Immediate |
| CPS (Current Population Survey) | US, monthly, annual ASEC supplement | Employment, income, demographics | IPUMS CPS; free | Immediate |
**Demography / family / health:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| HRS (Health and Retirement Study) | US 50+, biennial panel 1992– | Health, cognition, wealth, retirement, death | ICPSR; free registration | 1–2 weeks |
| NHANES | US, cross-sectional cycles | Physical health, biomarkers, diet, SES | CDC; free | Immediate |
| NHIS (National Health Interview Survey) | US, annual | Health status, access to care | CDC; free | Immediate |
| Vital Statistics (NCHS) | US births and deaths | Birth certificate, death certificate vars | CDC; free | Immediate |
| World Population Prospects | Global, UN projections | Fertility, mortality, migration by country | UN; free | Immediate |
**Education:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| NELS / ELS / HSLS (NCES) | US high school cohorts | Academic achievement, SES, post-secondary | NCES; free | Immediate |
| NAEP | US, grade 4/8/12, national + state | Math and reading scores | NCES; free | Immediate |
| College Scorecard | US institutions | Earnings, completion, debt, costs | ED.gov; free | Immediate |
| IPEDS | US institutions | Enrollment, graduation, faculty, finances | NCES; free | Immediate |
**Political behavior / civic:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| ANES (American National Election Study) | US, biennial | Vote choice, attitudes, partisanship | ICPSR; free | Immediate |
| CCES (Cooperative Election Study) | US, annual, N≈50K | Policy attitudes, partisanship, race | Harvard Dataverse; free | Immediate |
| Pew Research datasets | US + global | Attitudes, media, religion, demographics | Pew; free registration | Immediate |
**Immigration / ethnicity / language:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| CPS-ASEC | US, annual | Nativity, citizenship, language, income | IPUMS; free | Immediate |
| New Immigrant Survey (NIS) | US legal immigrants | Pre-migration, occupation, networks, health | Princeton; free | Immediate |
| ISSP | 40+ countries, cross-national | Attitudes, identity, religion, work | GESIS; free | Immediate |
| WVS / EVS | 80+ countries, 7 waves | Values, trust, democracy, religion | WVS archive; free | Immediate |
| Luxembourg Income Study | 50+ countries, harmonized | Income, wealth, labor | LIS; free registration | 1 week |
**Neighborhoods / spatial / administrative:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| Decennial Census / ACS tract | US census tracts / blocks | Demographics, SES, housing by place | Census API / IPUMS NHGIS; free | Immediate |
| HMDA (Home Mortgage Disclosure Act) | US, annual | Mortgage applications, denials, race, income | CFPB; free | Immediate |
| TIGER/Line shapefiles | US geography | Boundaries, roads, landmarks | Census; free | Immediate |
| HOLC redlining maps | US cities, 1930s | Neighborhood security grades (A–D) | Mapping Inequality; free | Immediate |
| Opportunity Atlas | US census tracts | Economic mobility by race/income/place | opportunityatlas.org; free | Immediate |
**Text / digital / computational:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| Congressional Record | US, 1873– | Floor speeches, committee records | GovInfo / Voteview; free | Immediate |
| Common Crawl | Web, petabyte-scale | Text from 3+ billion pages | commoncrawl.org; free | Immediate |
| Google Trends | Global, 2004– | Search interest by topic/region | Google; free API | Immediate |
| Twitter/X historical | Social media | Text, networks, metadata | X API v2 Pro+ ($5K/mo; Academic track discontinued Jan 2025) | 2–4 weeks |
**Crime / criminal justice / substance use:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| NCVS (National Crime Victimization Survey) | US, annual panel since 1973 | Crime victimization, reporting, demographics | ICPSR; free | Immediate |
| UCR / NIBRS (FBI) | US, annual agency-level + incident | Offenses, arrests, agency demographics | FBI Crime Data Explorer; free | Immediate |
| NSDUH (Nat. Survey on Drug Use and Health) | US, annual 70K+ respondents | Substance use, mental health, treatment, demographics | SAMHSA / ICPSR; free | Immediate |
| National Corrections Reporting Program | US, annual individual-level | Prison admissions, releases, sentences, demographics | ICPSR (restricted); DUA | 2–4 weeks |
| Sentencing Commission Data | US federal courts, 1996– | Sentences, guidelines, demographics, offense types | USSC; free download | Immediate |
**International / cross-national surveys:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| WVS / EVS | 120+ countries, 7 waves (1981–) | Values, trust, democracy, religion, gender | wvs.worldvaluessurvey.org; free | Immediate |
| ISSP | 40+ countries, annual modules | Attitudes: environment, health, religion, social inequality, work | GESIS; free registration | Immediate |
| ESS (European Social Survey) | 30+ European countries, biennial (2002–) | Trust, politics, immigration, well-being, religion | europeansocialsurvey.org; free | Immediate |
| Eurobarometer | EU member states, biennial (1974–) | EU integration, economy, immigration attitudes | GESIS / ICPSR; free | Immediate |
| Afrobarometer | 39 African countries, 9 rounds (1999–) | Democracy, governance, economy, service delivery | afrobarometer.org; free | Immediate |
| Latinobarómetro | 18 Latin American countries, annual (1995–) | Democracy, trust, economy, social cohesion | latinobarometro.org; free registration | Immediate |
| Asian Barometer | 14 Asian countries, 5 waves | Democracy, governance, social capital | asianbarometer.org; free registration | 1–2 weeks |
| LIS (Luxembourg Income Study) | 50+ countries, harmonized | Income, wealth, labor, demographics | LIS; free registration | 1 week |
| LWS (Luxembourg Wealth Study) | 20+ countries | Wealth, assets, debt | LIS; free registration | 1 week |
| PISA (OECD) | 80+ countries, triennial | Student achievement, school quality, equity | OECD; free | Immediate |
| TIMSS / PIRLS (IEA) | 60+ countries, quadrennial | Math, science, reading achievement | IEA; free | Immediate |
| DHS (Demographic and Health Surveys) | 90+ developing countries | Fertility, mortality, nutrition, HIV, gender | dhsprogram.com; free registration | 1–2 weeks |
| MICS (UNICEF) | 100+ countries, 6 rounds | Child health, education, protection, water/sanitation | mics.unicef.org; free | Immediate |
**Economic / labor / macro:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| FRED (Federal Reserve Economic Data) | US + global, 800K+ series | GDP, unemployment, CPI, interest rates, exchange rates | FRED API; free key | Immediate |
| Penn World Table (PWT) | 180+ countries, 1950–2019 | Real GDP, capital stock, productivity, purchasing power | rug.nl/ggdc/productivity/pwt; free | Immediate |
| EU-KLEMS | EU + major economies, 1970– | Industry-level growth, productivity, factor inputs | euklems.eu; free | Immediate |
| LEHD (Longitudinal Employer-Household Dynamics) | US, quarterly, tract-level | Employment flows, earnings by age/sex/industry | Census OnTheMap / LODES; free | Immediate |
| OECD Data | 38+ OECD countries | Economic, social, environmental indicators | data.oecd.org; free API | Immediate |
| Eurostat | EU member states | Demographics, economy, trade, environment, health | ec.europa.eu/eurostat; free | Immediate |
| ILO (International Labour Organization) | 180+ countries | Employment, wages, working conditions, migration | ilostat.ilo.org; free | Immediate |
**Global health / population / environment:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| WHO Global Health Observatory | 194 countries, 1000+ indicators | Mortality, disease burden, health systems, risk factors | who.int/data/gho; free API | Immediate |
| GBD (Global Burden of Disease) | 204 countries, 1990– | DALYs, mortality, morbidity by cause/risk/age/sex | ghdx.healthdata.org; free | Immediate |
| IPUMS International | 100+ countries, census microdata | Demographics, education, employment, housing, migration | ipums.org; free registration | 1–2 weeks |
| World Population Prospects (UN) | Global, UN projections | Fertility, mortality, migration by country | UN; free | Immediate |
| NASA SEDAC | Global, gridded | Population density, land use, air quality, climate | sedac.ciesin.columbia.edu; free | Immediate |
| FAOSTAT | 245+ countries | Agriculture, food security, emissions, land use | fao.org/faostat; free | Immediate |
**Science of science / bibliometrics / research data:**
| Dataset | Coverage | Key variables | Access | Timeline |
|---------|----------|--------------|--------|----------|
| OpenAlex | 250M+ works, global | Authors, institutions, citations, topics, open access | openalex.org; free API | Immediate |
| Semantic Scholar | 200M+ papers | Citations, abstracts, embeddings, citation intent | api.semanticscholar.org; free (rate-limited) | Immediate |
| Dimensions | 130M+ publications | Grants, patents, clinical trials, policy documents | app.dimensions.ai; free for research | 1–2 weeks |
| ORCID Public Data | 18M+ researchers | Researcher IDs, affiliations, works | public API; free | Immediate |
| NSF SED (Survey of Earned Doctorates) | US, annual since 1957 | PhD recipients, field, demographics, funding | NCSES; free | Immediate |
| NSF S&E Indicators | US, biennial | R&D spending, STEM workforce, innovation metrics | ncses.nsf.gov; free | Immediate |
| Crossref | 150M+ DOIs | Metadata, citations, funders, licenses | api.crossref.org; free (polite pool) | Immediate |
| Web of Science (Clarivate) | 90M+ records, 1900– | Citations, impact factors, h-index | Institutional license required | Varies |
| MAG successor via OpenAlex | 250M+ works | Paper-author-institution-concept graph | openalex.org; free | Immediate |
**General-purpose open data repositories:**
| Repository | Scope | Key features | Access | URL |
|-----------|-------|-------------|--------|-----|
| Harvard Dataverse | All disciplines; 75K+ datasets | DOI, API, versioning, rich metadata | Free; open to all | dataverse.harvard.edu |
| ICPSR | Social/behavioral sciences; 500K+ files | Curated, DUA for restricted data, SPSS/Stata/R | Free registration (member institutions) | icpsr.umich.edu |
| Zenodo | All disciplines; CERN-hosted | DOI, 50GB/record, versioning, GitHub integration | Free; open to all | zenodo.org |
| OSF (Open Science Framework) | All disciplines | Preregistration, versioning, integrations (GitHub, Dataverse) | Free; open to all | osf.io |
| Figshare | All disciplines | DOI, 20GB free, embeddable, altmetrics | Free; open to all | figshare.com |
| QDR (Qualitative Data Repository) | Qualitative / multi-method | Annotation, transparency appendices | Free registration | qdr.syr.edu |
| Roper Center (iPoll) | Public opinion; 23K+ datasets, 800K questions | US + international polling data | Institutional membership | ropercenter.cornell.edu |
| Data.gov | US federal/state agencies | Machine-readable government datasets | Free | data.gov |
| UK Data Service | UK surveys + international | Access to major UK studies (Understanding Society, BCS70) | Free registration (UK higher ed) | ukdataservice.ac.uk |
| GESIS (Leibniz Institute) | European social science | Eurobarometer, ISSP, ALLBUS, EVS archive | Free registration | gesis.org |
| Google Dataset Search | Meta-search across repositories | Searches 25K+ repositories worldwide | Free | datasetsearch.research.google.com |
| Kaggle Datasets | ML/data science; 200K+ datasets | Notebooks, competitions, community discussion | Free | kaggle.com/datasets |
| GitHub (curated lists) | Replication data + awesome lists | awesomedata/awesome-public-datasets; replication archives | Free | github.com |
| AWS Open Data | Large-scale scientific data | Satellite imagery, genomics, climate | Free (compute charges for AWS processing) | registry.opendata.aws |
**Restricted-access / linked federal data:**
| Dataset | Access route | Timeline |
|---------|-------------|----------|
| SSA earnings + survey links | FSRDC application | 12–24 months |
| IRS/Treasury linked micro-data | FSRDC application | 12–24 months |
| State administrative records | State agency MOU + IRB + DUA | 6–18 months |
| CMS Medicare/Medicaid claims | CMS DUA | 6–12 months |
### Additional International and Longitudinal Datasets
| Dataset | Coverage | Key Variables | Access | Category |
|---|---|---|---|---|
| UK Household Longitudinal Study (UKHLS / Understanding Society) | UK, 2009–present, ~40K households | Income, employment, health, education, ethnicity, wellbeing | Registration at UK Data Service | International |
| German Socio-Economic Panel (GSOEP / SOEP) | Germany, 1984–present, ~30K individuals | Income, employment, education, satisfaction, migration background | Apply at DIW Berlin (2–4 weeks) | International |
| Indian Human Development Survey (IHDS) | India, 2004–05 & 2011–12, ~42K households | Caste, income, education, health, gender, social networks | ICPSR download | International |
| Afrobarometer | Africa (39 countries), 1999–present | Democracy attitudes, governance, identity, inequality | Public download | International |
| Latinobarómetro | Latin America (18 countries), 1995–present | Democracy, institutions, economic perceptions, identity | Registration required | International |
| China Family Panel Studies (CFPS) | China, 2010–present, ~15K households | Income, education, cognition, migration, health | Apply at Peking University | International |
| European Values Study (EVS) | Europe, 1981–2017 (5 waves) | Values, religion, politics, work, family, national identity | GESIS download | International |
| UK Biobank | UK, 500K participants | Genetics, health, imaging, lifestyle, sociodemographics | Application + DUA (restrictive) | Health/Biomedical |
| Fragile Families & Child Wellbeing Study (FFCWS) | US, ~5K births (1998–), 6 waves | Unmarried parents, child development, income, incarceration, housing | OPR Princeton (public + restricted) | Family/Children |
| National Longitudinal Study of Adolescent to Adult Health (Add Health) | US, 1994–2018, ~20K individuals | Social networks, health behaviors, romantic relationships, genetics | Carolina Population Center (public + restricted) | Health/Youth |
### Step 0b. Verify dataset fit
For the selected dataset, confirm:
- Unit of analysis matches the RQ (person / household / firm / county / country-year)
- Time period covers the phenomenon of interest
- Key variables (Y, X, M, moderators) are available with adequate measurement quality
- Sample size is sufficient for the planned subgroup analyses (see power analysis in /scholar-design)
- Access pathway is realistic given project timeline
### Step 0c. IPUMS access note
IPUMS harmonizes many datasets above (ACS, CPS, GSS, international census, CPS-ASEC). Always check IPUMS first:
- `ipums.org` — IPUMS USA (ACS/Census), IPUMS CPS, IPUMS International
- `gss.norc.org` — GSS directly; or IPUMS MICS
- API access via `ipumsr` R package or `ipumspy` Python package
```r
library(ipumsr)
ddi <- read_ipums_ddi("usa_00001.xml")
df <- read_ipums_micro(ddi)
```
### Step 0d. Auto-Fetch Open Data (EXECUTE — not just recommend)
**When the user specifies or you identify an "Immediate" access public data source from the directory above, DO NOT just produce code templates — actually download the data now.** This step converts `data-status: no-data` into `data-status: existing-data`.
Create `data/` directory and download:
```bash
mkdir -p data/raw data/clean
```
**Match the data source and execute the appropriate download:**
| Data source | R package / method | Auto-fetch code |
|-------------|-----------|-----------------|
| ACS / Census tract | `tidycensus` | See below |
| CPS | `ipumsr` or direct Census | See below |
| GSS (General Social Survey) | `gssr` | See below |
| World Bank | `WDI` | See below |
| BLS (unemployment, CPI) | `blsAPI` | See below |
| FRED (macro-economic) | `fredr` | See below |
| NHANES | `nhanesA` | See below |
| ESS (European Social Survey) | `essurvey` | See below |
| Eurostat | `eurostat` | See below |
| OECD | `OECD` / `oecdR` | See below |
| Penn World Table | `pwt10` / direct download | See below |
| WHO Global Health Observatory | `WHO` / direct API | See below |
| FAOSTAT | `FAOSTAT` | See below |
| OpenAlex | `openalexR` / REST API | See below |
| Harvard Dataverse | `dataverse` (R) / `pyDataverse` (Py) | See below |
| Zenodo | REST API | See below |
| College Scorecard | direct CSV | See below |
| Google Trends | `gtrendsR` | See below |
| GDELT | direct download | See below |
| Opportunity Atlas | direct CSV | See below |
| WVS (World Values Survey) | direct download | See below |
| DHS (Demographic and Health Surveys) | `rdhs` | See below |
| UCR / NIBRS crime data | `crimedata` / direct | See below |
| Afrobarometer | direct download | See below |
| IPUMS (any series) | `ipumsr` | See below |
**R auto-fetch templates (execute via Bash):**
```r
# ── ACS / Census (tidycensus) ─────────────────────────────────────
library(tidycensus)
# census_api_key("YOUR_KEY", install = TRUE) # one-time setup
vars <- c(medinc = "B19013_001", pop = "B01001_001",
pct_bach = "B15003_022", pct_poverty = "B17001_002")
df <- get_acs(geography = "[tract|county|state]",
state = "[STATE]", variables = vars,
year = [YEAR], geometry = FALSE, survey = "acs5")
saveRDS(df, "data/raw/acs-[geography]-[state]-[year].rds")
write.csv(df, "data/raw/acs-[geography]-[state]-[year].csv", row.names = FALSE)
message("Downloaded: ", nrow(df), " rows from ACS")
# ── World Bank (WDI) ─────────────────────────────────────────────
library(WDI)
df <- WDI(country = "all",
indicator = c(gdp = "NY.GDP.MKTP.CD", pop = "SP.POP.TOTL",
life_exp = "SP.DYN.LE00.IN"),
start = [START_YEAR], end = [END_YEAR], extra = TRUE)
saveRDS(df, "data/raw/wdi-[indicators]-[years].rds")
write.csv(df, "data/raw/wdi-[indicators]-[years].csv", row.names = FALSE)
# ── BLS ──────────────────────────────────────────────────────────
library(blsAPI)
payload <- list(seriesid = c("[SERIES_ID]"),
startyear = "[START]", endyear = "[END]")
df <- blsAPI(payload, api_version = 2, return_data_frame = TRUE)
saveRDS(df, "data/raw/bls-[series]-[years].rds")
# ── FRED ─────────────────────────────────────────────────────────
library(fredr)
fredr_set_key(Sys.getenv("FRED_API_KEY"))
df <- fredr(series_id = "[SERIES_ID]",
observation_start = as.Date("[START]"),
observation_end = as.Date("[END]"))
saveRDS(df, "data/raw/fred-[series]-[years].rds")
# ── NHANES ───────────────────────────────────────────────────────
library(nhanesA)
demo <- nhanes("[CYCLE_DEMO]") # e.g., "DEMO_J" for 2017-2018
exam <- nhanes("[CYCLE_EXAM]") # e.g., "BMX_J"
df <- merge(demo, exam, by = "SEQN")
saveRDS(df, "data/raw/nhanes-[cycle].rds")
# ── Google Trends ────────────────────────────────────────────────
library(gtrendsR)
gt <- gtrends(keyword = c("[TERM1]", "[TERM2]"),
geo = "[GEO]", time = "[START] [END]")
df <- gt$interest_over_time
saveRDS(df, "data/raw/gtrends-[terms]-[dates].rds")
# ── GSS (General Social Survey) ─────────────────────────────────
# install.packages("gssr", repos = "https://kjhealy.r-universe.dev")
library(gssr)
data(gss_all) # cumulative file 1972–2024
# Or single year: df <- gss_get_yr(2022)
saveRDS(gss_all, "data/raw/gss-cumulative.rds")
message("GSS cumulative: ", nrow(gss_all), " rows, ", ncol(gss_all), " vars")
# ── ESS (European Social Survey) ────────────────────────────────
library(essurvey)
set_email("[YOUR_ESS_EMAIL]") # registered email at europeansocialsurvey.org
df <- import_rounds(rounds = c(10, 11)) # rounds 10 and 11
saveRDS(df, "data/raw/ess-rounds-10-11.rds")
# ── Eurostat ─────────────────────────────────────────────────────
library(eurostat)
# Search for datasets: search_eurostat("unemployment")
df <- get_eurostat("[DATASET_CODE]", time_format = "num")
# e.g., "une_rt_m" for monthly unemployment rates
saveRDS(df, "data/raw/eurostat-[code].rds")
# ── OECD ─────────────────────────────────────────────────────────
library(OECD)
# Search datasets: search_dataset("education")
df <- get_dataset("[DATASET_ID]",
filter = list(c("[COUNTRY_CODES]")),
start_time = [START_YEAR], end_time = [END_YEAR])
saveRDS(df, "data/raw/oecd-[dataset]-[years].rds")
# ── Penn World Table ─────────────────────────────────────────────
# Direct download (no API key needed)
pwt_url <- "https://dataverse.nl/api/access/datafile/:persistentId?persistentId=doi:10.34894/QT5BCC"
download.file(pwt_url, "data/raw/pwt100.xlsx", mode = "wb")
library(readxl)
df <- read_excel("data/raw/pwt100.xlsx", sheet = "Data")
saveRDS(df, "data/raw/pwt10-data.rds")
# ── WHO Global Health Observatory ────────────────────────────────
# Direct API (no key needed)
who_url <- "https://ghoapi.azureedge.net/api/[INDICATOR_CODE]"
# e.g., WHOSIS_000001 for life expectancy
df <- jsonlite::fromJSON(who_url)$value
saveRDS(df, "data/raw/who-[indicator].rds")
# ── OpenAlex ─────────────────────────────────────────────────────
library(openalexR)
works <- oa_fetch(entity = "works",
search = "[SEARCH_TERM]",
from_publication_date = "[START_DATE]",
to_publication_date = "[END_DATE]",
count_only = FALSE)
saveRDS(works, "data/raw/openalex-[topic]-[dates].rds")
# ── Harvard Dataverse ────────────────────────────────────────────
library(dataverse)
Sys.setenv("DATAVERSE_SERVER" = "dataverse.harvard.edu")
# Search: dataverse_search("immigration", type = "dataset")
df <- get_dataframe_by_name(filename = "[FILENAME]",
dataset = "[DOI]",
server = "dataverse.harvard.edu")
saveRDS(df, "data/raw/dataverse-[name].rds")
# ── IPUMS (any series) ──────────────────────────────────────────
library(ipumsr)
ddi <- read_ipums_ddi("[EXTRACT_FILE].xml")
df <- read_ipums_micro(ddi)
saveRDS(df, "data/raw/ipums-[series]-[extract].rds")
# ── DHS (Demographic and Health Surveys) ─────────────────────────
library(rdhs)
set_rdhs_config(email = "[YOUR_DHS_EMAIL]", project = "[PROJECT_NAME]")
datasets <- dhs_datasets(countryIds = "[COUNTRY_CODE]", surveyYearStart = [YEAR])
df <- get_datasets(datasets$FileName[1])
saveRDS(df, "data/raw/dhs-[country]-[year].rds")
# ── FAOSTAT ──────────────────────────────────────────────────────
library(FAOSTAT)
df <- get_faostat_bulk(code = "[DOMAIN_CODE]")
# e.g., "QCL" for crops/livestock production
saveRDS(df, "data/raw/faostat-[domain].rds")
```
```python
# ── Python alternatives ──────────────────────────────────────────
import pandas as pd
from pathlib import Path
Path("data/raw").mkdir(parents=True, exist_ok=True)
# College Scorecard (direct CSV)
url = "https://ed-public-download.app.cloud.gov/downloads/Most-Recent-Cohorts-Institution_04192024.zip"
df = pd.read_csv(url, low_memory=False)
df.to_parquet("data/raw/college-scorecard.parquet")
# Opportunity Atlas (direct CSV)
url = "https://opportunityinsights.org/wp-content/uploads/2018/10/tract_outcomes_simple.csv"
df = pd.read_csv(url)
df.to_parquet("data/raw/opportunity-atlas-tracts.parquet")
# GDELT (daily event data)
from datetime import datetime
date_str = datetime.now().strftime("%Y%m%d")
url = f"http://data.gdeltproject.org/events/{date_str}.export.CSV.zip"
df = pd.read_csv(url, sep="\t", header=None)
df.to_parquet(f"data/raw/gdelt-{date_str}.parquet")
# ── World Values Survey (direct download, no key) ───────────────
# Download from https://www.worldvaluessurvey.org/WVSDocumentationWVL.jsp
# After registering (free), download CSV or Stata file
# Or use direct longitudinal file:
# wvs_url = "https://www.worldvaluessurvey.org/WVSDocumentationWVL.jsp"
# Note: requires free registration; manual download then:
# df = pd.read_stata("data/raw/WVS_Cross-National_Wave_7_v6_0.dta")
# ── OpenAlex (REST API, no key needed) ──────────────────────────
import requests
def fetch_openalex_works(search_term, per_page=200, max_pages=5):
"""Fetch works from OpenAlex API (free, no key needed)."""
records = []
for page in range(1, max_pages + 1):
url = (f"https://api.openalex.org/works?search={search_term}"
f"&per-page={per_page}&page={page}"
f"&mailto={__import__('os').environ.get('SCHOLAR_CROSSREF_EMAIL', '')}")
resp = requests.get(url, timeout=30)
data = resp.json()
records.extend(data.get("results", []))
if len(data.get("results", [])) < per_page:
break
return pd.json_normalize(records)
# df_oa = fetch_openalex_works("immigration policy")
# df_oa.to_parquet("data/raw/openalex-immigration-policy.parquet")
# ── Harvard Dataverse (pyDataverse, no key for public datasets) ─
# pip install pyDataverse
from pyDataverse.api import NativeApi
api = NativeApi("https://dataverse.harvard.edu")
# Search: resp = api.get_search("immigration", type="dataset")
# Download a specific file by DOI:
# resp = api.get_datafile(file_id, is_pid=False)
# ── Zenodo (REST API, no key for public records) ────────────────
def fetch_zenodo_dataset(record_id):
"""Download files from a Zenodo record (public, no key needed)."""
url = f"https://zenodo.org/api/records/{record_id}"
meta = requests.get(url).json()
for f in meta["files"]:
fname = f["key"]
print(f"Downloading {fname} ({f['size']/(1024**2):.1f} MB)")
resp = requests.get(f["links"]["self"], stream=True)
with open(f"data/raw/{fname}", "wb") as out:
for chunk in resp.iter_content(chunk_size=8192):
out.write(chunk)
return meta["metadata"]["title"]
# ── WHO GHO API (no key needed) ─────────────────────────────────
def fetch_who_indicator(indicator_code):
"""Fetch WHO Global Health Observatory data (free, no key)."""
url = f"https://ghoapi.azureedge.net/api/{indicator_code}"
resp = requests.get(url, timeout=30)
data = resp.json()
return pd.DataFrame(data["value"])
# df_who = fetch_who_indicator("WHOSIS_000001") # life expectancy
# df_who.to_parquet("data/raw/who-life-expectancy.parquet")
# ── Eurostat (no key needed) ────────────────────────────────────
def fetch_eurostat(dataset_code):
"""Fetch Eurostat dataset via bulk download (free, no key)."""
url = f"https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/{dataset_code}?format=TSV&compressed=true"
import io, gzip
resp = requests.get(url, timeout=60)
data = gzip.decompress(resp.content)
return pd.read_csv(io.BytesIO(data), sep="\t")
# df_eu = fetch_eurostat("une_rt_m") # monthly unemployment
# df_eu.to_parquet("data/raw/eurostat-unemployment.parquet")
# ── Afrobarometer (direct CSV download) ─────────────────────────
# Download merged multi-country datasets from:
# https://www.afrobarometer.org/data/
# Free registration required, then download CSV/Stata/SPSS
# df = pd.read_stata("data/raw/afrobarometer-r9-merged.dta")
# ── FBI Crime Data (UCR/NIBRS — direct download, no key) ────────
# Crime Data Explorer: https://cde.ucr.cjis.gov/LATEST/webapp/#/pages/downloads
# Direct API endpoint:
def fetch_fbi_crime(api_key, offense="burglary", state="US", start=2015, end=2023):
"""FBI Crime Data Explorer API (free key from api.data.gov)."""
url = (f"https://api.usa.gov/crime/fbi/sapi/api/summarized/state/{state}"
f"/{offense}/{start}/{end}?API_KEY={api_key}")
return pd.DataFrame(requests.get(url).json()["results"])
# ── Data.gov / federal CSV datasets (no key needed) ─────────────
# Search at https://catalog.data.gov/dataset
# Most datasets provide direct download links (CSV, JSON, XML)
# Example: USDA food environment atlas
# df = pd.read_csv("https://www.ers.usda.gov/webdocs/DataFiles/80526/FoodEnvironmentAtlas.csv")
```
**After successful download:**
1. Confirm the file exists and report row count: `nrow(df)` / `len(df)`
2. Log the download to `data/raw/download-manifest.md`:
```
| File | Source | Date fetched | N rows | Variables |
|------|--------|-------------|--------|-----------|
| acs-tract-IL-2022.rds | tidycensus ACS 5-year | 2026-03-03 | 15,420 | medinc, pop, pct_bach, pct_poverty |
```
3. **Update PROJECT STATE**: set `data-status: existing-data` and `Data File(s): data/raw/[filename]`
4. Proceed with all downstream skills in **DATA-AVAILABLE MODE** — no more `[CODE-TEMPLATE]` or `[PLACEHOLDER]`
**If the download fails — API key missing (MOST COMMON):**
Before falling back to CODE-TEMPLATE, **ask the user for the missing API key**. Most open data APIs offer free keys:
| Source | Env variable | Where to get a free key |
|--------|-------------|------------------------|
| ACS / Census / CPS (`tidycensus`) | `CENSUS_API_KEY` | https://api.census.gov/data/key_signup.html |
| FRED (`fredr`) | `FRED_API_KEY` | https://fred.stlouisfed.org/docs/api/api_key.html |
| ESS (`essurvey`) | ESS registered email | https://www.europeansocialsurvey.org/user/new |
| DHS (`rdhs`) | DHS registered email + project | https://dhsprogram.com/data/new-user-registration.cfm |
| FBI Crime Data Explorer | `FBI_API_KEY` | https://api.data.gov/signup/ |
| Semantic Scholar | `S2_API_KEY` | https://www.semanticscholar.org/product/api (optional, for higher limits) |
Prompt the user:
```
To download [SOURCE] data, I need an API key.
You can get a free key here: [URL]
Please provide your API key, or I can proceed with code templates instead.
```
If the user provides a key:
1. Set it in the R environment: `Sys.setenv(CENSUS_API_KEY = "[key]")` or equivalent
2. Save it to `.Renviron` for future sessions: `cat('CENSUS_API_KEY=[key]\n', file = "~/.Renviron", append = TRUE)`
3. Retry the download
4. If successful → upgrade `data-status` to `existing-data`
**Sources that do NOT require API keys** (always attempt these first):
- World Bank (`WDI`) — no key needed
- NHANES (`nhanesA`) — no key needed
- GSS (`gssr`) — no key needed
- Google Trends (`gtrendsR`) — no key needed
- BLS (`blsAPI` v1) — no key needed (v2 optional key for higher limits)
- College Scorecard — direct CSV download, no key needed
- Opportunity Atlas — direct CSV download, no key needed
- GDELT — direct download, no key needed
- Direct URL datasets (GitHub, OSF, Dataverse) — no key needed
- OpenAlex (`openalexR`) — no key needed (use `mailto` for polite pool)
- Eurostat (`eurostat`) — no key needed
- WHO GHO API — no key needed
- Penn World Table — direct download, no key needed
- FAOSTAT (`FAOSTAT`) — no key needed
- Harvard Dataverse (`dataverse` / `pyDataverse`) — no key for public datasets
- Zenodo REST API — no key for public records
- Semantic Scholar API — no key needed (rate-limited; optional key for higher limits)
- Crossref API — no key needed (use `mailto` for polite pool)
- Afrobarometer — direct download after free registration
- IPUMS (`ipumsr`) — free registration required, then direct download
- UCR / NIBRS crime data — direct download from FBI Crime Data Explorer
- NASA SEDAC — free registration, direct download
- Data.gov — direct CSV/JSON downloads, no key needed
**Sources requiring free registration** (attempt after no-key sources):
- ESS (`essurvey`) — free registration at europeansocialsurvey.org
- OECD (`OECD`) — free API, no key but registration recommended
- DHS (`rdhs`) — free registration + approved project at dhsprogram.com
- WVS — free registration at worldvaluessurvey.org
- ISSP — free registration at GESIS
- Eurobarometer — free registration at GESIS
- Latinobarómetro — free registration at latinobarometro.org
- LIS / LWS — free registration at lisdatacenter.org
- UK Data Service — free registration (UK higher ed)
- Roper Center iPoll — institutional membership required
**If the download fails for other reasons** (network error, rate limit, package not installed):
- Log the specific error message
- If package missing: attempt `install.packages("[pkg]")` and retry once
- If network error: inform user and fall back to `[CODE-TEMPLATE]`
- Keep `data-status: no-data` and note the reason in PROJECT STATE
---
## WORKFLOW 1: Variable Dictionary and Measurement Plan
Produce a formal variable dictionary — the backbone of your Methods section and IRB application. Run after identifying the dataset (WORKFLOW 0) and confirming the design (/scholar-design).
### Step 1a. Variable dictionary table
For every variable in the analytic model, document:
| Role | Variable name | Construct | Operationalization | Source question / admin field | Type | Range / categories | Notes |
|------|--------------|-----------|-------------------|-----------------------------|------|--------------------|-------|
| **Y** (Outcome) | `earnings_ln` | Annual labor market earnings | Log of annual earnings ($) | PSID Q_annual_earnings | Continuous | 0–∞ (log-transformed) | Top-coded at 99th pct |
| **X** (Key predictor) | `immigrant` | Immigrant status | Born outside US (1=yes) | ACS NATIVITY | Binary | 0/1 | — |
| **M** (Mediator) | `english_prof` | English language proficiency | 4-point self-report scale | CPS SPEAKENG | Ordinal | 1=not at all … 4=very well | — |
| **W** (Moderator) | `race_eth` | Race/ethnicity | Self-identified 5-category | ACS RACE + HISPAN | Categorical | White/Black/Hispanic/Asian/Other | ref = White |
| **C** (Control) | `educ_yrs` | Years of schooling | Recoded from education categories | ACS EDUC | Continuous | 0–20 | Midpoints used |
| **C** | `age` | Age in years | — | ACS AGE | Continuous | 25–64 | Working-age restriction |
| **C** | `female` | Female | Sex == female | ACS SEX | Binary | 0/1 | — |
| **FE / cluster** | `state` | State | State FIPS code | ACS STATEFIP | Categorical | 51 states + DC | For FE or clustering |
### Variable Dictionary Template
| Variable Name | Label | Type | Values/Range | Source Item | Coding Notes |
|---|---|---|---|---|---|
| `income_hh` | Household annual income | Continuous ($) | 0–999,999 | "Total household income last year" | Top-coded at $999,999; log-transform recommended |
| `educ_yrs` | Years of education | Continuous (years) | 0–25 | "Highest grade completed" | Recode GED=12; professional degree=20 |
| `race_eth` | Race/ethnicity (5 categories) | Categorical | 1=White NH, 2=Black NH, 3=Hispanic, 4=Asian NH, 5=Other | Combined from race + Hispanic origin | Standard Census categories |
| `employed` | Currently employed | Binary | 0=No, 1=Yes | "Did you work last week?" | Missing if not in labor force (code separately) |
| `missing_code` | — | — | NA / -9 / .d / .r | — | NA=system missing; -9=refused; .d=don't know; .r=refused |
**Scale direction convention**: Always code so higher values = more of the construct (e.g., 1=strongly disagree → 5=strongly agree). Reverse-code items before analysis.
**Variable naming convention**: `[concept]_[modifier]` (e.g., `income_hh`, `income_per_capita`, `educ_yrs`, `educ_degree`).
### Step 1b. Measurement validity checklist
For each key construct, verify:
- **Face validity**: Does the operationalization obviously measure the concept?
- **Content validity**: Does it cover all relevant aspects of the construct?
- **Construct validity**: Are established scales being used? Is there prior evidence of reliability (α, test-retest)?
- **Measurement equivalence**: Does the measure mean the same thing across the groups being compared (e.g., across racial groups, immigration cohorts)?
### Step 1c. Data blueprint summary
Produce a one-page data blueprint:
```
DATA BLUEPRINT
─────────────────────────────────
Research question: [RQ from Phase 0/1]
Dataset: [Name, year/wave, N]
Unit of analysis: [person / household / tract / ...]
Analytic sample: [population restriction + expected N]
Outcome (Y): [variable name + operationalization]
Key predictor (X): [variable name + operationalization]
Mediator(s) (M): [if applicable]
Moderator(s) (W): [if applicable]
Controls: [list]
Design: [cross-sectional / panel / DiD / RD / IV / matching]
Fixed effects / clusters: [unit + SE clustering level]
Weights: [survey weight variable name, if applicable]
Missing data strategy: [listwise / MI — based on /scholar-eda]
```
---
## WORKFLOW 2: Survey Instrument Design
### Step 1: Define Measurement Goals
For each construct, specify:
- **Conceptual definition**: What is the construct theoretically?
- **Operational definition**: What observable behavior/attitude/characteristic measures it?
- **Measurement level**: Nominal, ordinal, interval, ratio
- **Reference period**: "In the past 12 months," "currently," "ever"
### Step 2: Question Construction
**Question types and when to use:**
| Type | Best For | Example |
|------|----------|---------|
| Single-item Likert | Attitudes with established scales | "Strongly agree → Strongly disagree" |
| Multi-item scale | Latent constructs (trust, identity) | Average of 5 items; report α |
| Open-ended | Unexpected responses, sensitive topics | "In your own words, describe…" |
| Numeric entry | Factual, bounded | "How many years have you lived in the US?" |
| Matrix / grid | Battery of related items | Multiple rows, shared response scale |
| Ranking | Relative preference | "Rank the following from most to least…" |
| Filtered / branching | Subpopulation follow-ups | "If yes → [follow-up]" |
**Question wording rules:**
- One concept per question (no double-barreled: "Do you trust neighbors *and* local government?")
- Avoid negations
- 8th-grade reading level
- Offer "Don't know" / "Prefer not to answer" where appropriate
- Consistent response direction (higher = more of the construct)
- Match scale length to construct precision needed (5-pt standard; 7-pt for fine-grained attitudes)
**Established scales to use** (cite if using):
- Social trust: Rosenberg Trust Scale; GSS trust items
- Subjective SES: MacArthur Ladder (1–10)
- Discrimination: Everyday Discrimination Scale (Williams et al. 1997)
- Mental health: PHQ-9 (depression), GAD-7 (anxiety)
- Political attitudes: ANES scales
- Immigrant identity: MEIM-R (Phinney & Ong 2007)
- Social network: Burt (1984) name generator + name interpreter
See [references/survey-design.md](references/survey-design.md) for full item batteries.
### Step 3: Survey Organization
**Recommended order:**
1. Introduction screen (purpose, consent, confidentiality)
2. Screener questions (eligibility)
3. Salient but non-threatening items first
4. Sensitive items (income, discrimination, illegal behavior) later
5. Sociodemographics last
6. Debriefing / thank-you screen
**Sample size for surveys:**
| Goal | Rule of thumb | Notes |
|------|--------------|-------|
| Descriptive estimates (proportions) | N ≥ 400 for ±5% margin (95% CI) | N ≥ 1,000 for subgroup comparisons |
| Regression with 10 predictors | N ≥ 500 for OLS (Cohen's rule: 50 + 8k) | N ≥ 1,000 for logistic |
| Experimental contrast (d = 0.3, 80% power) | N ≥ 352 per arm | Use `pwr::pwr.t.test()` |
| Conjoint / AMCE (1% precision) | N ≥ 1,000, 3–5 tasks | More profiles = better precision |
| Multilevel (ICC = 0.10, L2 effects) | N ≥ 30 groups × 30 within | Use `pwr2ppl` or `simr` |
**Online panel quality tiers:**
| Tier | Panel | Quality | Cost | Best for |
|------|-------|---------|------|---------|
| Probability-based | IPSOS KnowledgePanel, Amerispeak | Highest | $$$ | Top-tier publication; nationally representative |
| High-quality opt-in | Prolific Academic | High | $$ | Academic research; attention checks built-in |
| MTurk | MTurk (via CloudResearch) | Medium | $ | Pilot testing; not recommended for main analysis |
| Convenience | Qualtrics panel, Lucid | Variable | $–$$ | Pretest; not representative |
**Data quality checks:**
- Completion time < ⅓ median → flag inattentive
- Straight-lining on Likert batteries
- Instructed response item failures (attention checks)
- Open-ended gibberish / copy-paste
- IP duplicates
### Step 4: Pilot and Cognitive Testing
1. Expert review (2–3 colleagues; face validity)
2. Cognitive interviewing (5–10 respondents; 4-probe protocol — see survey-design.md)
3. Pilot survey (N = 20–50; check distributions, timing, drop-off)
4. Psychometric checks: CFA for multi-item scales; remove items loading < 0.4
### Step 5: Fielding and Documentation
- Platform: Qualtrics (complex routing) > Prolific (probability-based recruitment) > REDCap (clinical/HIPAA)
- AAPOR response rate documentation (RR1–RR6) required in Methods
- Non-response analysis: compare respondents to non-respondents or population benchmarks
### Step 6: Experimental Survey Modules
#### Step 6a: Vignette / Factorial Survey Experiment
A factorial survey presents randomized descriptions (vignettes) of hypothetical persons or situations and asks respondents to evaluate them. Used to measure preferences, discrimination, or decision-making while controlling for confounds.
**Design:**
- 2–5 dimensions varied; 2–4 levels per dimension → full or fractional factorial design
- Each respondent rates 3–8 vignettes (avoid fatigue)
- Analyze with OLS clustered by respondent (AMCE: Average Marginal Component Effect)
**Vignette template:**
```
[Respondent instructions]:
"Below is a description of a job applicant. Please read carefully and rate
how likely you would be to invite them for an interview."
Job applicant profile:
Name: [RANDOMIZED: Michael Johnson / DeShawn Jackson / José García]
Education: [RANDOMIZED: High school diploma / Bachelor's degree / Master's degree]
Years of exp.: [RANDOMIZED: 2 years / 5 years / 10 years]
Criminal record: [RANDOMIZED: None / Misdemeanor / Felony]
"How likely would you be to invite this applicant for an interview?"
(1) Very unlikely — (2) Unlikely — (3) Neutral — (4) Likely — (5) Very likely
```
**Analysis (R):**
```r
library(cregg)
# Estimate AMCEs
amce_results <- cj(data = df, formula = outcome ~ name + education + experience + criminal,
id = ~respondent_id)
plot(amce_results) + theme_Publication()
# Subgroup AMCEs
amce_sub <- cj(df, outcome ~ name + education, id = ~respondent_id,
by = ~respondent_race)
```
**Report:** "We used a factorial survey experiment with [N] vignettes and [K] randomized dimensions. Average marginal component effects (AMCEs; Hainmueller, Hopkins & Yamamoto 2014) were estimated using OLS with standard errors clustered by respondent."
#### Step 6b: List Experiment (Sensitive Item Measurement)
List experiments allow respondents to reveal sensitive attitudes (racism, illegal behavior, stigmatized conditions) without directly endorsing them — protecting against social desirability bias.
**Design (split-half):**
- Control group: 3 innocuous items → "How many of the following have you done in the past year?"
- Treatment group: same 3 items + 1 sensitive item
- Difference in means = % endorsing sensitive item
**Template:**
```
[Control condition]:
"Below is a list of things some people have done. Please tell us
HOW MANY you have done in the past year — not which ones, just how many."
1. Watched a movie at home
2. Donated to a charity
3. Attended a religious service
[Answer: 0 / 1 / 2 / 3]
[Treatment condition — adds sensitive item]:
1. Watched a movie at home
2. Donated to a charity
3. Attended a religious service
4. [Sensitive: e.g., "Used marijuana" / "Supported restricting immigration"]
[Answer: 0 / 1 / 2 / 3 / 4]
```
**Analysis (R):**
```r
library(list)
lexp <- ict.test(df$count, df$treatment, J = 3, gms = TRUE)
summary(lexp)
# Reports: estimated prevalence + SE + 95% CI for sensitive item
```
**Sample size:** N ≥ 500 per arm (N ≥ 1,000 total) for ±5% precision. List experiments are statistically inefficient — plan for larger samples than direct questions would require.
---
## WORKFLOW 3: Qualitative Interview Protocol
### Step 1: Protocol Architecture
A qualitative interview guide has three parts:
1. **Preamble**: introduce yourself, state purpose, obtain consent, explain recording
2. **Core guide**: 8–15 open-ended questions organized thematically (3–5 themes)
3. **Closing**: ask for what was missed; offer referrals if sensitive topics were discussed
**Sampling strategy:**
| Strategy | Purpose | Typical N |
|----------|---------|-----------|
| Maximum variation | Capture full range of relevant variation | 20–40 |
| Homogeneous | Depth within one type | 15–25 |
| Theoretical (grounded theory) | Continue until conceptual saturation | 15–30 |
| Critical case | Test theory on hardest case | 5–15 |
| Snowball | Hard-to-reach populations | Varies |
Saturation typically occurs at N = 15–25 for a single population; more for comparative multi-group designs.
### Step 2: Question Sequence
See [references/interview-protocols.md](references/interview-protocols.md) for full template. Core structure:
1. Rapport opener (biographical, low-stakes)
2. Grand tour question (typical day / typical encounter with X)
3. Thematic questions (2–4 per theme) with explicit probes
4. Contrast / comparison questions
5. Meaning / interpretation questions
6. Closing ("What haven't I asked?")
**Probing types:** elaboration / clarification / example / contrast / emotional (use sparingly)
### Step 3: Sensitive Topics Protocol
- State upfront which topics will be covered; allow opt-out before starting
- Use distancing language: "Some people I've talked with say… What's your experience?"
- Never press for more detail than offered
- Prepare referral resources (hotlines, social services) before fieldwork
- Debrief at close: "We covered some difficult topics — how are you feeling?"
### Step 4: Focus Groups (alternative format)
Use when group interaction is the phenomenon of interest (e.g., deliberation, norm formation, collective sense-making). **Not** a substitute for in-depth interviews when individual experience is the target.
- Group size: 5–8 participants (too small → insufficient interaction; too large → some dominated)
- Number of groups: 3–6 per subgroup; stop when theoretical saturation is reached
- Facilitator role: introduce topics, manage dominant voices, ensure quieter participants contribute
- Co-facilitator: takes notes on nonverbal dynamics while facilitator runs discussion
- Analyze turn-taking patterns alongside content
### Step 5: Online / Video Interviews
- Zoom / Teams: adequate for most semi-structured interviews; less rapport than in-person
- Provide tech support beforehand; confirm platform access and audio/video
- Record with consent (built-in recording; confirm jurisdiction-specific legality)
- Background and setting: note if participant is in private vs. semi-public space
- Transcription: Otter.ai / Whisper for automatic draft; always human-review for accuracy
### Step 6: Multilingual Fieldwork
- Translate instrument using forward-backward procedure (translate → back-translate → reconcile)
- Use bilingual interviewers; never rely solely on respondent to translate
- Conduct interviews in respondent's preferred language; document language used
- Analysis: code from transcript in original language; translate selected quotes for publication
- Report language of interview and translation procedure in Methods
---
## WORKFLOW 4: Administrative and Secondary Data
> Record-linkage / spatial-join code authored here, when executed, is IN SCOPE for the Pre-Execution Review Duty (above).
### Step 1: Data Access Strategy
| Access type | Route | Timeline |
|-------------|-------|----------|
| Public use micro-data (ACS, CPS, GSS) | IPUMS, Census, ICPSR — register and download | Immediate |
| Restricted federal data (NLSY restricted, HRS geocoded) | ICPSR DUA | 2–4 weeks |
| FSRDC-linked data (SSA + IRS + survey) | FSRDC application (PI must be Census employee or external researcher) | 12–24 months |
| State administrative records (Medicaid, court, school) | State agency MOU + institutional DUA + IRB | 6–18 months |
| Private / commercial data (credit, social media, employer) | Data sharing agreement; often requires fee or in-kind contribution | Varies |
### Step 2: Data Documentation Review
Before analysis, document for every source:
- Survey instrument or administrative codebook / data dictionary
- Variable names, labels, value codes (distinguish "don't know" / "refused" / "not applicable")
- Sampling design: stratification, clustering, probability weights
- Panel structure (if longitudinal): wave IDs, response rates, attrition rates by wave
- Geographic identifiers and level of geography available (state / PUMA / tract)
- Top-coding and suppression rules (especially in restricted-use files)
### Step 3: Data Linkage
When merging two or more data sources:
- Document matching variables (name, SSN, address, date of birth, geographic ID)
- Report match rate and characterize match vs. non-match cases (selective linkage is
[…] pełny plik w paczce .zip
Files in this package (5)
- SKILL.md 91 KB
- references/data-management.md 8 KB
- references/interview-protocols.md 8 KB
- references/survey-design.md 8 KB
- references/web-scraping.md 14 KB
Versions
The database has no other skills named scholar-data.
Database snapshot: 13.11.2026 · package version 2ddab0563225 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100social-science-research-assistant
Use when a task needs the judgment of a Social Science Research Assistant — checking inter-rater reliability on qualitative coding before it scales, drafting or amending an IRB protocol/consent script, cleaning and de-identifying a participant dataset, or triaging a… - 95scholar-auto-research
Stable, deterministic social-science research-paper pipeline from idea or data to verified manuscript, citations, replication package, and final md/docx/tex/pdf outputs. Use when the user wants a reliable end-to-end auto-research chain that avoids the evolving scholar-full-paper workflow. - 90social-science-research
Orchestrates a social science research workflow from literature review through data collection, text analysis, statistical modeling, and report generation. Use when conducting empirical social science research, policy analysis, or mixed-methods studies. NOT for pure natural… - 100data-collector
Evidence collection — use when the user wants to locate data sources, collect case evidence, validate data availability, clean datasets, or organize non-academic research materials. Looks for variable requirements and research object in research_proposal.md; asks user if absent. - 100irb-protocol
Use when writing, reviewing, or revising an Institutional Review Board (IRB) protocol for human-subjects research — anchors the protocol to the Belmont Report, the Common Rule (45 CFR 46), and the specific subparts for vulnerable populations. - 100rseng-data-management
Covers research data management around software: organizing and documenting datasets (layout, data dictionaries), keeping data out of git while versioning it properly (DVC, git-annex, DataLad), FAIR data and metadata standards, depositing data with DOIs in repositories such as…