How Do You Lift a Classifier's Hardest Level From 4% to 77%?

Some classification problems are hard because the classes are subtle. This one was hard because of shape: a four-level taxonomy, 137 valid paths from root to leaf, and a distribution where a handful of paths dominate and a long tail of paths barely appear at all. The first honest measurement of the deepest level came back at roughly 4% accuracy — which is not "needs tuning." At 4%, the model has effectively memorized the majority classes and is guessing at everything else, and every downstream consumer of that level's output is better off ignoring it.

This post is about the two moves that fixed it — hierarchical decomposition and a deliberately layered imbalance toolkit — and about the production plumbing that keeps a many-model system honest once it's serving traffic. I can't show you the taxonomy or the data (client work; the specifics stay behind the wall). But the failure mode and the fix are completely general, and I've since seen the same pattern in open datasets, so the techniques are worth writing down properly.

Why flat classification collapses on deep taxonomies

The obvious first architecture is the one everyone tries: flatten the taxonomy into leaf classes, fine-tune one transformer with a 137-way softmax head, done. It's one model, one training job, one deployment. It also fails in a very characteristic way, and it's worth understanding why before reaching for anything fancier.

The label space is too wide for the evidence per decision. A 137-way softmax asks the model to make one giant decision with no structure. But the taxonomy has structure — leaves under the same parent are semantically close to each other and far from leaves under other parents. A flat head throws that away. The model has to relearn, from data alone, that mistaking one sibling for another is a near-miss while mistaking a leaf for something in a different branch entirely is a catastrophe. Cross-entropy over a flat softmax treats both errors identically.

Imbalance compounds across levels. Class frequency at the leaves is the product of frequencies down the path. If one branch holds most of the data and, inside it, one child holds most of that, the rare leaves in rare branches end up with vanishingly few examples relative to the head of the distribution. A flat model minimizing average loss can achieve a great-looking aggregate number by nailing the few dominant leaves and writing off the tail — which is precisely the ~4% story. Aggregate accuracy looked survivable; the deepest level, measured properly, was noise.

Errors are unconstrained. A flat model can — and does — predict leaf combinations that violate the hierarchy, assigning a record to a leaf whose implied ancestors contradict what the same model would say if you asked it about level 1 directly. There's no mechanism forcing its answers to be self-consistent, and inconsistent predictions are the fastest way to lose a human reviewer's trust.

Hierarchical decomposition: shrink the label space per decision

The fix is to make the taxonomy's structure load-bearing. Instead of one model making one 137-way decision, train one fine-tuned model per branch, and let each parent's prediction constrain which model runs next — and therefore which labels are even possible.

Every property the flat model lacked falls out of this decomposition:

• Each decision is small. No single model ever faces 137 options. Each one distinguishes only among the children of one node — a handful of semantically related choices, which is exactly the kind of fine-grained distinction a fine-tuned DeBERTa or RoBERTa encoder is good at when it isn't also being asked to do coarse routing in the same softmax.
• Consistency is structural, not learned. A leaf prediction can only be reached through its actual ancestors. Invalid paths aren't discouraged; they're unrepresentable.
• Imbalance becomes local and fixable. Instead of one distribution with a brutal global tail, each branch model sees its own smaller distribution. Some branches are nearly balanced and need nothing. Others are badly skewed and can be treated individually — which is what makes the toolkit in the next section practical.
• Errors localize. When the deepest level is wrong, you can see which decision failed. Half our early "level-4 failures" were actually level-2 routing mistakes, and no amount of leaf-level loss engineering fixes a record that arrived at the wrong subtree. Fixing routing first was worth more than any single loss change.

The cost is real: many models to train, version, evaluate, and serve, and errors cascade — a wrong parent makes the child's job impossible. That cascade is the argument people use against hierarchical decomposition, and it's a fair one. But upper levels are the easy levels — big classes, plenty of signal — so in practice you're spending accuracy where you have a surplus to buy accuracy where you're bankrupt. The trade was overwhelmingly worth it here.

The imbalance toolkit: three tools, three different jobs

Decomposition alone didn't get to 77%. Within the skewed branches, the rare classes still lost. Three techniques, layered deliberately, closed the rest of the gap. They're often presented as interchangeable "imbalance fixes." They're not — they attack different parts of the problem.

Class-weighted loss: pay more for rare mistakes

The baseline move: scale each class's contribution to cross-entropy by (roughly) inverse frequency, so one rare-class error costs as much as many majority-class errors. It's cheap, it's stable, and it stops the optimizer from rationally ignoring small classes. Its limit: it re-weights classes uniformly. Every rare-class example gets the same boost, whether the model already handles it or finds it hopeless. Weights raise the floor; they don't focus effort.

Focal loss: pay more for hard mistakes

Focal loss reshapes the loss per example rather than per class:

$FL(p_t) = -\alpha_t (1 - p_t)^\gamma \log(p_t)$

Here $p_t$ is the probability the model assigned to the true class, $\alpha_t$ is a per-class balancing weight, and $\gamma$ is the focusing exponent. In plain English: the $(1 - p_t)^\gamma$ factor measures how wrong the model was and scales the loss by it. When the model is already confident and correct ($p_t$ near 1), the factor collapses toward zero and the example contributes almost nothing. When the model is confused ($p_t$ low), the factor stays near 1 and the full loss applies.

Why this targets the rare-class problem specifically: in a skewed branch, the easy, abundant examples numerically dominate the gradient — thousands of small nudges that collectively drown out the tail. Focal loss mutes exactly those examples. The gradient budget gets spent where the model is struggling, and in an imbalanced branch, "where the model is struggling" is the rare classes. Class weights change what an error costs; focal loss changes which examples get the model's attention at all. With $\gamma = 0$ it degrades gracefully back to weighted cross-entropy, which also makes it easy to A/B honestly.

Branch oversampling: make rare classes exist at batch time

The subtlest failure needed a third tool. With few enough examples of a rare class, entire training batches can contain none of it — no loss function can weight an example that never appears. Oversampling duplicates (or resamples with augmentation) rare-class examples so every batch actually contains the tail. Because the hierarchy made imbalance local, this is done per branch, tuned to each branch's own skew — the nearly-balanced branches are left untouched, because oversampling a balanced branch just invites overfitting on duplicated text.

How they interact — and how they fight

These three multiply, and naively stacking them at full strength is a classic mistake: oversample a class 10×, weight it 10×, and add focal focusing on top, and you've effectively told the optimizer that class is 100×+ more important than the data says — the model starts hallucinating the rare class everywhere, and precision craters while recall "improves." The composition that worked: oversample first (moderately — enough that every batch sees the tail), then temper the class weights to account for the already-flattened batch distribution, then let focal loss handle the residual per-example difficulty. Sampling fixes presence, weights fix cost, focal fixes attention. Each tool at partial strength, aimed at the failure the others can't reach.

Here's the shape of it in HuggingFace terms — generic labels, invented branch names, nothing proprietary:

# One branch model among many. Labels are placeholders, not real taxonomy content.
labels = ["level3_branch_a", "level3_branch_b", "level3_branch_c", "level3_branch_d"]
 
model = AutoModelForSequenceClassification.from_pretrained(
    "microsoft/deberta-v3-base", num_labels=len(labels)
)
 
# 1) Branch oversampling: rare classes appear in every batch.
sample_weights = 1.0 / class_counts[train_label_ids]          # inverse frequency per example
sampler = WeightedRandomSampler(sample_weights, num_samples=len(train_label_ids))
 
# 2) Tempered class weights: soften because sampling already flattened the batches.
class_weights = (class_counts.sum() / class_counts) ** 0.5    # sqrt-tempered, not full inverse
 
class FocalTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels")
        logits = model(**inputs).logits
        ce = F.cross_entropy(logits, labels, weight=class_weights, reduction="none")
        p_t = torch.exp(-ce)                                   # prob assigned to the true class
        loss = ((1 - p_t) ** GAMMA * ce).mean()                # 3) focal focusing, gamma ~ 2
        return (loss, logits) if return_outputs else loss
 
    def _get_train_sampler(self, *args):
        return sampler

One model per branch means this recipe runs many times with per-branch hyperparameters — which is tedious, and exactly why the training loop had to become infrastructure (below) rather than a notebook.

Evaluation pitfalls at the deepest level

The number that matters is easy to fake, and I want to be precise about what "~4% to 77%+" means, because the honest version of this measurement is what made the problem visible at all.

Aggregate accuracy is a majority-class mirror. At the deepest level, overall accuracy is dominated by whichever paths are common. A model that's excellent on the head and useless on the tail posts a respectable aggregate. The metric that told the truth was per-path performance: evaluate each of the 137 paths separately, then look at the distribution — macro-style averaging and the worst-k paths, not the traffic-weighted mean. The ~4% figure is the deepest level measured honestly, end-to-end; so is the 77%+.

Conditional vs. end-to-end accuracy. A leaf model evaluated on records that arrived at the correct parent measures the model. Evaluated end-to-end — cascaded routing errors included — it measures the system. The second number is the only one users experience, and it is always worse. Report both; ship on the second.

Don't let oversampling leak into eval. Resampling belongs to training only. Evaluate on the natural distribution — and per-path so the natural distribution's skew doesn't hide the tail again. Duplicated rare examples must never straddle the train/eval split, or the rare-class numbers become fiction precisely where you most need them true.

Some paths can't be measured well. With very few held-out examples, a rare path's accuracy is a coin-flip estimate either way. The honest move is wide error bars and human spot-checks on those paths, not a confident decimal.

The MLOps loop: many models is an infrastructure problem

A hierarchy of fine-tuned models is only viable if retraining and rollout are boring. The production loop:

CI-triggered SageMaker fine-tuning. Training config lives in the repo; a merge triggers GPU fine-tuning jobs on SageMaker. No hand-run notebooks, no "which machine trained v3?" archaeology — every artifact traces to a commit, which matters enormously when there's one model per branch and any of them might need a targeted retrain.

EFS-versioned artifacts with symlink rollout. Every training run writes to a fresh versioned directory on EFS; a current symlink points at the blessed version. Rollout is flipping a symlink; rollback is flipping it back — near-instant either way, with every previous version still on disk. The serving containers on ECS Fargate resolve the symlink and stay ignorant of versioning entirely, and no rollout requires baking model weights into a container image.

Graceful degradation. With many models, "a model is unavailable" is a when, not an if — mid-rollout, a corrupt artifact, a branch intentionally pulled after a bad eval. The FastAPI service treats each branch model as optional: if a level's model can't be loaded, it returns the deepest prediction it could make, explicitly flagged as partial. A truncated-but-honest answer keeps the whole pipeline alive and keeps failures visible instead of silent — which is the property that made reviewers trust the system's outputs, including the 77%+ leaf level, in the first place.

What's hard, and what's next

Honesty section. What this system does not have solved:

Calibration at the leaves. 77%+ accuracy with confidence scores that mean something would let low-confidence predictions route to humans automatically. Rare-class probabilities coming out of an oversampled, focal-trained model are not naturally calibrated — the training tricks that fixed accuracy actively distort the probabilities. Post-hoc calibration per branch is the obvious next step, and per-branch calibration data for rare classes is scarce by definition. Unsolved.

The retraining tax. One model per branch means every architecture-level improvement multiplies across the tree. The CI/SageMaker loop makes it push-button, but not free — smarter change detection (retrain only branches whose data or upstream router shifted) is designed and not yet built.

The tail you can't measure. Some paths are rare enough that their eval numbers are anecdotes with decimal points. I don't have a better answer than error bars, human spot-checks, and refusing to report per-path precision I can't support. If you have a principled approach to evaluating classes with a handful of held-out examples, I genuinely want to hear it.

Cascade errors are still the ceiling. Hierarchical decomposition converts "impossible" into "bounded by the router," and the deepest level can never beat the levels above it. Soft routing — propagating top-k parents with probabilities instead of a hard argmax — is the known fix, trading the clean one-path story for accuracy. Measuring whether that trade is worth it is next on the list.

The headline arrow — ~4% to 77%+ — wasn't one clever trick. It was structure first (make each decision small and each error local), then three imbalance tools each doing the one job the others can't, then evaluation honest enough to prove it, then plumbing boring enough to keep it true in production. In my experience that ordering is the technique.