Ru-Jev 1.3 now understands unfamiliar category names — but not abbreviations

⚡ The gist in 5 seconds

• Ru-Jev 1.3 ships with a new augmentation: each label now has several name variants, and the model trains on a random one of them.
• Weights and code are MIT, the repo is besktech/ru-jev on Hugging Face, it runs on CPU and fits into a closed-loop setup.
• On abbreviations and codes the gain is zero, and temperature calibration on your own data will have to be redone via fit_temperature.py.

🔍 What happened

An author under the handle vladimirbesk published version 1.3 of the Ru-Jev model — an open model of typed decisions for Russian-language text (Habr). It doesn't write text. The input is a context and a set of options; the output is a probability for each option and one selected. There are three decision types: yes or no, a choice from a set, and an ordinal scale.

Under the hood is a pointwise cross-encoder. Each pair of "context plus option" goes through the mmBERT-base encoder in a separate pass: 307 million parameters, of which 110 million are outside the embedding matrix. Then a softmax over the options and temperature calibration on top. Two consequences follow: the order of options doesn't affect the answer, but runtime grows with their number — on 18 options the model is slower than those that read all options in a single pass.

What changed in 1.3. Version 1.1 saw each routing scenario in two forms — an English key and a Russian word. In 1.3, a label has several forms, and each training example takes one at random. For MASSIVE intents (60 labels) there are four: the English key, the Russian label, the key in Cyrillic via a transliteration rule, and a truncated form of that key. For MASSIVE scenarios (18 labels) — three. Toxicity has five formulations of two labels instead of one, review sentiment five, review rating four. Plus, the author fixed the validation split and raised the number of epochs from three to four, so this can't be called a clean ablation — he admits it himself.

He tested on the same 1000 questions from the validation split of MASSIVE scenarios. Only the scenario names changed. Result: on unfamiliar word names, native or borrowed, 1.3 is more accurate than 1.1. On abbreviations and codes there is no gain.

🧠 Our take

What hooks me here is not so much the gain as the framing itself. The author runs classifiers in a citizen-requests processing system where every prediction is checked by an operator. The bottleneck is not the model but that check. A confidence threshold lets some answers through automatically, and it works exactly as well as the probability is calibrated. So label-name augmentation is not cosmetics but an attempt to make it possible to rename a label without retraining everything from scratch.

The calibration numbers look decent. At the 50% most confident answers, the threshold is 0.85, accuracy 0.967, 33 errors per 1000. At 90% of answers, the threshold drops to 0.54, accuracy to 0.859, with 141 errors. That's 13,373 decisions, and the temperature was tuned on the other half of the validation set. But the same rows were used in epoch selection, so the numbers are optimistic — our estimate: the real ceiling will be lower, the only question is by how much.

Who needs this? Those who embed a classifier in a closed loop on ordinary hardware and don't want to run an LLM for "yes/no" and picking from a list. We believe the market for such small decision models is underrated: everyone is looking at generation, while the routine work is labeling requests, routing tickets, toxicity checks. The losers here, in my view, are cloud API wrappers: a 300-megabyte model on CPU with no network is a strong argument for the public sector and industry. And separately about the name: Ru-Jev echoes the commercial Jev from TypeSafe AI, but the author states outright — there is no connection, the model was not trained on Jev's outputs, and no API was called.

⚠️ What we don't know

The weak spot in the source is the methodology. The author honestly says: the difference is not confirmed by a strict ablation, because the validation split and the learning-rate schedule changed between versions. So part of the gain may not come from the augmentation. Portability is also concerning: on unfamiliar labels and new domains the shipped temperature doesn't work; it has to be retuned on a few hundred of your own rows. And one more thing: all measurements are a self-assessment by one person; there are no independent runs.