Language Agnostic Embeddings By Averaging Mutable Translations

Averaging embeddings across translations can produce a useful language-agnostic representation, but it is not a magic operation. It works best when three conditions hold simultaneously: the translations are genuinely...

Metadata

Field	Value
Source site	aiwikis.org
Source URL	https://aiwikis.org/
Canonical AIWikis URL	https://aiwikis.org/aiwikis/files/raw-system-archives-neurokinetic-agent-file-handoff-retired-source-archi-20663987/
Source reference	raw/system-archives/neurokinetic/agent-file-handoff/retired-source-archive-2026-06-13/2026-05-14-neurokinetic-redesign/Language-Agnostic Embeddings by Averaging Mutable Translations.md
File type	md
Content category	memory-file
Last fetched	2026-06-22T01:56:21.9510185Z
Last changed	2026-05-14T00:06:04.9666536Z
Content hash	sha256:2066398770ec3fbea1eaa7cb133ddc276ba557e5f852f587615a731cddf8e9a7
Import status	unchanged
Raw source layer	data/sources/aiwikis/raw-system-archives-neurokinetic-agent-file-handoff-retired-source-archive-2026-06-13-2026-05-14-2066398770ec.md
Normalized source layer	data/normalized/aiwikis/raw-system-archives-neurokinetic-agent-file-handoff-retired-source-archive-2026-06-13-2026-05-14-2066398770ec.txt

Current File Content

Structure Preview

• Language-Agnostic Embeddings by Averaging Mutable Translations
• Executive summary
• Definitions and scope
• Why averaging can work and when it breaks
• Methods landscape
• Evidence on averaging, alignment, and evaluation
• Implementation and experimental design
• Recommendations and research agenda

Raw Version

This public page shows a bounded preview of a large source file. The complete source remains in the raw and normalized source layers named in metadata, with the SHA-256 hash above for verification.

• Source characters: 29654
• Preview characters: 11844

# Language-Agnostic Embeddings by Averaging Mutable Translations

## Executive summary

Averaging embeddings across translations can produce a useful language-agnostic representation, but it is not a magic operation. It works best when three conditions hold simultaneously: the translations are genuinely semantically equivalent for the task at hand, the underlying embedding space is already aligned or has been aligned into a common geometry, and the vectors are normalized so that language-specific anisotropy or norm differences do not dominate the average. Under those conditions, averaging behaves like prototype construction: it suppresses language-specific noise and retains the shared semantic signal. This logic is strongly supported indirectly by cross-lingual word-mapping work, multilingual sentence encoders trained to pull translations together, and meta-embedding results showing that simple means can be surprisingly competitive once geometry is handled properly. citeturn18view0turn15view0turn21search0turn3view3turn3view4

The strongest practical recommendation is therefore conservative: if the “items” are words or short labels, use aligned word spaces such as MUSE or VecMap-style mappings; if the items are phrases, titles, or sentences, use a multilingual sentence encoder such as LASER, LaBSE, a multilingual Sentence-Transformer, multilingual-E5, or BGE-M3, then normalize each translation embedding, average them, and normalize again. For static spaces, add mean-centering, length normalization, orthogonal alignment, and CSLS-based retrieval. For raw pretrained language-model sentence vectors, consider mean-centering or whitening before aggregation because anisotropy is a documented problem. citeturn3view5turn3view2turn16search1turn18view0turn11search0turn11search1turn25search1turn31search2turn31search3

Simple mean is the baseline to beat. Weighted means are often better when translations vary in informativeness or noise level; classic examples include IDF/SIF-style weighting for bag-of-words sentence representations. By contrast, direct primary-source evidence for coordinate-wise medians, geometric medians, or other robust estimators specifically over multilingual translation sets is sparse. They are reasonable engineering options when some translations are noisy, idiomatic, non-literal, or sense-mismatched, but the literature surveyed here provides much stronger direct support for aligned means, weighted means, and learned cross-lingual encoders than for robust estimators over translation sets. citeturn18view0turn19search0turn15view0turn21search14

The main failure modes are not subtle. Averaging can blur polysemy, collapse literal and idiomatic readings, wash out word-order-sensitive contrasts, and over-smooth lexical or morphological distinctions that matter for the application. These risks are especially severe for single words without sense disambiguation, for distant language pairs where linear isomorphism is weak, and for “translations” that are really summaries, explanations, or localizations rather than meaning-preserving renderings. In short: averaging is most defensible when the translation set is sense-specific, clean, aligned, and task-matched. citeturn24search3turn27search4turn26search13turn26search17turn29search5turn23search6

## Definitions and scope

For this report, a **language-agnostic embedding** is a vector representation in which semantically equivalent items from different languages are intended to lie close enough that downstream comparison can be performed directly in embedding space, without translating everything into one pivot language at inference time. That is the design goal of multilingual word-alignment systems such as MUSE and VecMap, multilingual sentence encoders such as LASER and LaBSE, and more recent multilingual retrieval models such as multilingual-E5 and BGE-M3. citeturn3view5turn3view2turn3view3turn3view4turn31search2turn31search3

A **mutable translation** is best understood operationally, not philosophically. It is a surface realization in another language that may vary in wording, morphology, register, script, or local syntax while preserving the task-relevant content of an item. In this sense, “mutable” includes literal translations, close paraphrastic translations, alternative polite or informal renderings, and inflectional or orthographic variants, but it excludes cases where the rendering adds or drops propositions, resolves ambiguity in a way not present in the source, transforms an idiom into a non-literal explanation, or otherwise changes what a benchmark like STS, NLI, or bitext mining would treat as semantically equivalent. This task dependence is consistent with how XNLI operationalizes translation equivalence for inference pairs, how multilingual STS operationalizes graded similarity, and how paraphrase and idiom studies show that surface variation is not automatically semantics-preserving. citeturn7search6turn7search7turn24search3turn27search4

The phrase **same underlying meaning** should therefore be treated as a task-conditioned equivalence relation. At the strongest level, it means preservation of truth-conditional or entailment-relevant content. At a weaker level, it can mean preservation of retrieval intent or referential identity. Those are not identical requirements. For example, two product titles in different languages may be “the same item” for catalog retrieval even if one adds brand qualifiers, but two sentences in an NLI benchmark are not equivalent if one softens a quantifier or changes negation. This distinction matters because averaging across translations of different semantic granularity will often create a vector that represents none of them well. citeturn7search6turn7search7turn30search13

A practical implication follows. If the item is a **word**, especially a polysemous one, averaging across its possible translations is dangerous unless the sense is first fixed. If the item is a **sentence or short text unit** that already has a stable intended reading, averaging is much more defensible because modern multilingual sentence encoders were explicitly trained to make translations neighbors in a common space. citeturn24search3turn6search0turn3view3turn3view4

## Why averaging can work and when it breaks

The cleanest theoretical story is the prototype view. Suppose each translation embedding \(z_i\) can be decomposed into a shared semantic component \(s\) plus language-specific and lexical noise \(\epsilon_i\). If the embeddings already live in a common semantic space and the noise terms are roughly zero-mean, the arithmetic mean estimates \(s\) while reducing variance. This is the same broad intuition behind translation-ranking sentence encoders, cross-lingual distillation methods that force translations to occupy the same region, and meta-embedding work in which simple averaging becomes competitive once spaces are made comparable. citeturn3view4turn6search0turn15view0turn21search0

The literature provides three strong pieces of evidence for this prototype view. First, word-mapping work shows that orthogonal alignment plus careful similarity scaling can make bilingual spaces comparable enough for translation retrieval; Conneau et al. explicitly refine an adversarially initialized mapping with Procrustes and then retrieve with CSLS, and they report strong gains for both word translation and sentence translation retrieval using IDF-weighted sentence averages. citeturn18view0 Second, LaBSE learns cross-lingual sentence embeddings by combining MLM, TLM, dual-encoder translation ranking, and additive-margin softmax, and reports a large Tatoeba advantage over LASER, which is exactly what one would expect from a model trained to collapse translation variants toward a shared semantic point. citeturn4view2turn4view0 Third, Coates and Bollegala show that even across distinct source embeddings, arithmetic averaging can be surprisingly effective, while Jawanpuria et al. show that averaging is more amenable after learned orthogonal rotations and a common Mahalanobis metric scaling. citeturn15view0turn21search0

What breaks this story is geometry. Pretrained language-model sentence representations are often anisotropic: certain high-variance directions or language-specific means dominate the space. Li et al. show that vanilla BERT sentence embeddings form a non-smooth anisotropic semantic space, and Su et al. show that whitening can improve isotropy and semantics. Chang et al. further show, using XLM-R, that different languages occupy similar linear subspaces **after mean-centering**, while language-specific means encode language-sensitive information. That means naive averaging can accidentally average language identity, frequency bias, and other nuisance structure unless normalization is done first. citeturn11search0turn11search1turn25search1

What also breaks the story is non-compositionality and ambiguity. Contextual embedding analysis using paraphrases shows that BERT handles polysemous words better than static embeddings, but also gives synonyms surprisingly different representations in many cases and remains sensitive to word order. Idiomatic expressions are worse: their meaning is not inferable from constituent words, and dedicated idiom work such as MICE exists precisely because ordinary compositional assumptions fail. Consequently, averaging literal and non-literal translations, or mixing translations that resolve ambiguity differently, can produce a vector that is semantically blurred or even misleading. citeturn24search3turn27search4

Finally, averaging breaks when alignment assumptions are too weak. Mapping-based bilingual lexicon induction has long relied on approximate geometric isomorphism between spaces. That assumption is good enough for many related languages, but it degrades for distant pairs and for scenarios with significant domain or corpus mismatch. This is why later methods emphasize orthogonal constraints, whitening, self-learning, latent metric spaces, or nonlinear mappings, and why mBERT’s cross-lingual transfer is documented to work better for typologically similar languages and similar word orders. citeturn16search0turn16search1turn2search0turn23search6turn29search5turn26search13turn26search17

## Methods landscape

The multilingual-embedding literature falls into a few clear families. The table below compares the most relevant ones for translation-set averaging.

| Family | Representative methods | Training signal and unit | Best use case | Main limitation | Key sources |
|---|---|---|---|---|---|
| Linear mapping of static word spaces | MUSE, VecMap | Separately trained monolingual word embeddings mapped into a shared space with seed dictionaries, identical strings, or unsupervised/self-learning procedures | Bilingual lexicon induction, short labels, transparent alignment pipelines | Weak contextual semantics; needs approximate geometric compatibility | citeturn3view5turn3view2turn3view0turn16search1 |
| Joint multilingual sentence representation via MT training | LASER | Single encoder trained on parallel corpora with shared BPE vocabulary and translation objective | Sentence-level cross-lingual retrieval and transfer across many languages | Older architecture; less competitive than newer contrastive encoders on some retrieval tasks | citeturn4view4turn4view3turn0search1 |
| Multilingual masked or translation LM pretraining | mBERT, XLM, XLM-R | Joint pretraining across many languages with MLM and, for XLM, TLM | General-purpose multilingual contextual features | Raw sentence embeddings often need pooling/postprocessing; cross-lingual quality varies by language pair | citeturn26search13turn1search1turn0search7 |

Why This File Exists

This is a memory-system evidence file from aiwikis.org. It is shown here because AIWikis.org is demonstrating the real source files that make the UAIX / LLM Wiki memory system work, not only summarizing those systems after the fact.

Role

This file is memory-system evidence. It records source history, archive transfer, intake disposition, or another piece of provenance that should be retrievable without becoming an unsupported public claim.

Structure

The file is structured around these visible headings: Language-Agnostic Embeddings by Averaging Mutable Translations; Executive summary; Definitions and scope; Why averaging can work and when it breaks; Methods landscape; Evidence on averaging, alignment, and evaluation; Implementation and experimental design; Recommendations and research agenda. Those headings are retrieval anchors: a crawler or LLM can decide whether the file is relevant before reading every line.

Prompt-Size And Retrieval Benefit

Keeping this material in a separate file reduces prompt pressure because an agent can load this exact unit only when its role, source site, category, or hash is relevant. The surrounding index pages point to it, while this page preserves the full content for audit and exact recall.

How To Use It

• Humans should read the metadata first, then inspect the raw content when they need exact wording or provenance.
• LLMs and agents should use the source site, category, hash, headings, and related files to decide whether this file belongs in the active prompt.
• Crawlers should treat the AIWikis page as transparent evidence and follow the source URL/source reference for authority boundaries.
• Future maintainers should regenerate this page whenever the source hash changes, then review the explanation if the role or structure changed.

Update Requirements

When this source file changes, update the raw source layer, normalized source layer, hash history, this rendered page, generated explanation, source-file inventory, changed-files report, and any source-section index that links to it.

Related Pages

• Source overview
• Site file index
• Site report index
• UAI system index
• Source provenance
• Site directory
• Organization reports

Provenance And History

• Current observation: 2026-06-22T01:56:21.9510185Z
• Source origin: current-source-workspace
• Retrieval method: local-source-workspace
• Duplicate group: sfg-159 (primary)
• Historical hash records are stored in data/hashes/source-file-history.jsonl.

Machine-Readable Metadata

{
    "title":  "Language Agnostic Embeddings By Averaging Mutable Translations",
    "source_site":  "aiwikis.org",
    "source_url":  "https://aiwikis.org/",
    "canonical_url":  "https://aiwikis.org/aiwikis/files/raw-system-archives-neurokinetic-agent-file-handoff-retired-source-archi-20663987/",
    "source_reference":  "raw/system-archives/neurokinetic/agent-file-handoff/retired-source-archive-2026-06-13/2026-05-14-neurokinetic-redesign/Language-Agnostic Embeddings by Averaging Mutable Translations.md",
    "file_type":  "md",
    "content_category":  "memory-file",
    "content_hash":  "sha256:2066398770ec3fbea1eaa7cb133ddc276ba557e5f852f587615a731cddf8e9a7",
    "last_fetched":  "2026-06-22T01:56:21.9510185Z",
    "last_changed":  "2026-05-14T00:06:04.9666536Z",
    "import_status":  "unchanged",
    "duplicate_group_id":  "sfg-159",
    "duplicate_role":  "primary",
    "related_files":  [

                      ],
    "generated_explanation":  true,
    "explanation_last_generated":  "2026-06-22T01:56:21.9510185Z"
}