Workshop · Use Case 3

Multi-Category Classification

A TaMPER example focusing on evaluation.

Compliance · Accuracy · Precision

►  Open the R notebook in Colab

Overview

Goals

1. Set up and run multi-category classification of Twitter bios.
2. Use Compliance, Accuracy, and Precision to evaluate the output.
3. Use those metrics to compare across
   • 3 different models
   • 2 different prompts
   • 2 different temperatures
   • Structured outputs vs no structured outputs

Task

Who is the author?

• Input: 39 synthetic Twitter bios about a city's bid to land a high-tech innovation center.
• Goal: label each author as one of four user types.
• Reference: every bio carries a human gold standard label (usertype_label) so we can score the model.

General Public Media Other Stakeholders Politicians / Government

The bio, not the tweet, is the unit classified.

Setup

Three open models, two temperatures

Prompt · full

Full prompt

Prompt · simple

Simple prompt

Same four categories — the only change is dropping the definitions and step-by-step instructions (and it asks for plain text, not JSON). We measure what that costs.

Run it

The experimental design

• All unique, model, prompt, temperature, and structured outputs are run 5 times.
• Default settings for each comparison: Model: Qwen 3.6, Prompt: Full prompt, Temperature: 0, Output: Structured.

Each comparison changes one factor and holds the rest at the defaults above. Every condition is run 5×.

Three Model Comparison

Output examples

Bio: “honored to serve as @bostonmayor & member of @nationalmayors. dedicated to enhancing the well-being and prosperity of all Bostonians. proud to call this vibrant city home.”  ·  Gold standard: Politicians/Government Accounts

1 · Compliance — the code

# % missing, % valid label
def compliance(recs):
    n = len(recs)
    miss  = sum(r["category"] in (None,"")
                for r in recs)
    valid = sum(r["category"] in CATEGORIES
                for r in recs)
    return {"pct_missing": 100*miss/n,
            "pct_valid":   100*valid/n}

# tidyverse
compliance_table <- function(runs) runs |>
  group_by(model, prompt) |>
  summarise(
    pct_missing = 100*mean(is.na(pred)),
    pct_valid   = 100*mean(pred %in% CATEGORIES),
    .groups = "drop")

1 · Compliance — results

Did the models emit valid labels?

reform prompt · temp 0 · structured · 39 bios × 5 runs. Near-perfect across the board; Gemma's small dip is a few transient API timeouts, not format failures.

2 · Accuracy — the code

# exact match vs gold standard
def accuracy(recs):
    pairs = [(canon(r["category"]),
              canon(r["gold"])) for r in recs]
    pairs = [(p,g) for p,g in pairs if p and g]
    return 100*mean(p==g for p,g in pairs)

# tidyverse
accuracy_table <- function(runs) runs |>
  group_by(model, prompt) |>
  summarise(
    accuracy = 100*mean(pred == gold,
                        na.rm = TRUE),
    .groups = "drop")

2 · Accuracy

Non-matching examples

2 · Accuracy — results

How close to the human gold standard?

Exact match vs human label · reform prompt · temp 0 · structured · 39 bios pooled over 5 runs. Tight band — all three within ~3 points.

2 · Accuracy — is the gap real?

McNemar's test

• Goal: the models' accuracies differ (83 vs 80 vs 82) — real difference, or run-to-run noise?
• Step 1. Score every prediction right/wrong against the human gold standard.
• Step 2. Pair two models on the same bio (and iteration).
• Step 3. Keep only the disagreements — bios one model gets right and the other wrong.
• Step 4. Exact binomial test on those discordant counts → a p-value.

Paired and per-item — the right test when both models see the exact same data.

2 · McNemar — the code

# statsmodels, exact binomial
from statsmodels.stats.contingency_tables \
     import mcnemar

df["correct"] = (df.pred == df.gold).astype(int)
a = df[df.model==m1].set_index(["id","it"]).correct
b = df[df.model==m2].set_index(["id","it"]).correct
tab = [[((a==1)&(b==1)).sum(), ((a==1)&(b==0)).sum()],
       [((a==0)&(b==1)).sum(), ((a==0)&(b==0)).sum()]]
mcnemar(tab, exact=True)        # .statistic .pvalue

# exact McNemar = binom test on discordants
a <- df$correct[df$model == m1]   # aligned by
b <- df$correct[df$model == m2]   # id + iteration
tab <- table(a, b)
binom.test(tab["1","0"],
           tab["1","0"] + tab["0","1"], 0.5)

2 · Accuracy — model comparison

Are the model gaps significant?

None are significant (all p > 0.05). The ~3-point accuracy gaps are noise — on this task, model choice doesn't move accuracy. (The prompt does — that's the matrix.)

3 · Precision — the code

# agreement + Krippendorff alpha
def precision(recs):
    items = defaultdict(list)
    for r in recs:
        items[r["id"]].append(canon(r["category"]))
    agree = [len(set(v))==1 for v in items.values()]
    return {"agreement_rate": 100*mean(agree),
            "kripp_alpha": kripp_alpha_nominal(items)}

# tidyverse + base-R alpha
precision_table <- function(runs) runs |>
  group_by(model, prompt) |>
  summarise(
    agreement_rate = 100*mean(all_agree(pred)),
    kripp_alpha = kripp_alpha_nominal(
                    pred, iteration),
    .groups = "drop")

3 · Precision

Imprecise examples

Identical settings, run 5× — two runs that disagreed:

3 · Precision — results

Same answer across 5 runs?

39 bios × 5 iterations · reform prompt · temp 0 · structured. Gemma perfectly reproducible; GPT-OSS the most variable.

Comparison · Prompt

Full vs simple prompt example

Comparison · Prompt

Full vs simple prompt

McNemar (accuracy): p = 0.002 — significant. The full prompt is meaningfully more accurate (+11 pts) and more consistent. Qwen 3.6 · temp 0 · structured · 39 bios × 5.

Comparison · Temperature

Temperature 0 vs 1 examples

Comparison · Temperature

Temperature 0 vs 1

McNemar (accuracy): p = 0.453 — not significant. Temperature barely touches accuracy, but it lowers precision (α 0.97 → 0.86). Qwen 3.6 · reform · structured · 39 bios × 5.

Comparison · Structured output

Schema-enforced vs free text

McNemar (accuracy): p = 1.000 — not significant. With the strong reform prompt Qwen is already reliable, so structure adds little here — its payoff is larger with weak prompts. Qwen 3.6 · reform · temp 0 · 39 bios × 5.