01 / AI · Analysis
Extra layers give mixed gains in Vietnamese punctuation test
The preprint fixes training at eight epochs and does not report how rankings shift once architecture-specific tuning is added.

The authors of "PhoPunct: A Large-Scale Benchmark of Transformer Architectures for Vietnamese Punctuation Restoration", a preprint not yet peer-reviewed, reported comparing 17 architectures for restoring punctuation to Vietnamese text (22.1 million labelled tokens, eight epochs, no architecture-specific tuning), and no independent replication exists yet. In an arXiv abstract dated 9 Nov 2026, they claimed that the effect of adding bidirectional long short-term memory (Bi-LSTM), a recurrent network that reads a sequence in both directions, and conditional random field (CRF), a layer that predicts a sequence of labels jointly, was inconsistent.
On that basis, the authors argued that gains from an architecture itself should be examined separately from gains due to tuning for that architecture. Their argument questions whether a score that rises after layers are added can be read directly as an architectural effect.
According to the abstract, the benchmark measures performance without model-by-model adjustment, and that design sets the limits on how the results can be read.
A training budget fixed at eight epochs
The authors reported evaluating the models on 22.1 million labelled tokens from News and Novels corpora (19.9M News, 2.2M Novels). The 17 architectures consist of four pretrained Transformer backbones with optional Bi-LSTM and CRF layers, together with PhoPunct. PhoPunct is PhoBERT-large with both layers added.
Every model was trained at the same learning rate for eight epochs, an epoch being one full pass through the training data, and architecture-specific tuning was left out, the paper reported. The score gaps are therefore better read as performance within a set training budget than as the best each architecture can do.
Mixed effects of added layers across eight pairs
Under the fixed budget, adding the two layers "does not reliably improve macro F1", the authors reported. Macro F1 is a metric that averages the scores for each punctuation type with equal weight.
The paper reported comparing eight pairs matched by domain and backbone (eight epochs, no architecture-specific tuning). In seven of them, a plain Transformer classifier "matches or outperforms" its Bi-LSTM-CRF counterpart. The abstract does not report the score differences for each pair.
The top score came from a different architecture in each domain, according to the paper. PhoPunct has the highest News result, a macro F1 of 61.35 (19.9M News tokens, eight epochs). XLM-R with a CRF layer has the highest Novels result at 50.18, ahead of PhoPunct at 48.97 (2.2M Novels tokens, eight epochs). Because PhoPunct, which carries both layers, leads in News, the results also do not support reading added layers as always a disadvantage.
A conclusion that tuning could change
The authors argued that the value of the two layers depends on optimisation conditions, and they limited their conclusion to the fixed-budget setting. The abstract does not report how the rankings change when tuning is added for each architecture.
The results have not been independently replicated. The abstract states that the source code is available in the GitHub repository aisia-research-lab/PhoPunct.
The paper notes that Vietnamese punctuation restoration is often evaluated on relatively small benchmarks, and it presents scale as the difference from that earlier work. What the larger comparison leaves behind is less a ranking of architectures than a measurement task: gauging the effects of architecture and tuning separately.