Training and validation losses on in-distribution and longer out-of-distribution sentences.
We built and trained an encoder–decoder Transformer for German-to-English neural machine translation, then studied how its behavior changed when sentence length moved beyond the training distribution. We compared our model with a pretrained translation system on an in-distribution split and a more challenging out-of-distribution split containing sentences of approximately 10–20 words.
We evaluated training behavior alongside BLEU, ChrF, output-length ratio, and bigram repetition. Our model learned the in-distribution task without severe overfitting, but BLEU fell from 0.294 to 0.096 on longer sentences and the output-length ratio dropped to 0.588. This sharp shortening exposed frequent under-translation that aggregate quality scores alone did not fully explain.
To understand these failures more directly, we manually categorized omissions, unsupported additions, truncations, fluency problems, and named-entity errors. Our model's annotated error count increased from 47 to 72 on the out-of-distribution split, with omissions and additions dominating. Cross-attention coverage helped identify translations worth inspecting, although specialized attention patterns made it too noisy to serve as a stand-alone quality measure.
Finally, we tuned decoding without retraining the model. A beam size of 8, length penalty of 1.5, and no-repeat trigram constraint improved every out-of-distribution metric, raising the length ratio from 0.587 to 0.628, but did not eliminate the generalization gap. Decoding changes reduced visible symptoms; stronger generalization will require better training coverage, objectives, or model design.
Baseline and tuned decoding results. ChrF is rescaled by 0.01 and repetition rate by 100 for comparison.