Bert2D: A 2D-Word Embedding Model for Morphologically Rich Languages

### Model description **Bert2D** is a novel transformer-based model that builds upon the `BertModel` architecture by introducing a two-dimensional word embedding system. This enhancement is specifically designed to improve performance on morphologically rich languages, such as Turkish, Finnish, and Hungarian. This model card describes the initial release, which includes the model implementation and a pretrained checkpoint for Turkish. This work is based on the research outlined in the paper **"Bert2D: A 2D-Word Embedding for Morphologically Rich Languages"**, which has been accepted by IEEE and is available at: [https://ieeexplore.ieee.org/document/10542953](https://ieeexplore.ieee.org/document/10542953). A pretrained model for Turkish, `Bert2D-cased-Turkish-128K-WWM-NSW2`, is available on the Hugging Face Hub at: [https://huggingface.co/yigitbekir/Bert2D-cased-Turkish-128K-WWM-NSW2](https://huggingface.co/yigitbekir/Bert2D-cased-Turkish-128K-WWM-NSW2) ## Model Description The primary innovation of **Bert2D** is its use of a 2D positional embedding mechanism to better capture the complex morphological structures present in agglutinative languages. Unlike standard BERT models that use a 1D positional embedding, Bert2D employs a dual system: 1. **Whole-Word Positional Embeddings (1st Dimension):** This captures the absolute position of each word in a sequence. 2. **Sub-word Relative Positional Embeddings (2nd Dimension):** This encodes the relative position of sub-words within each word, allowing the model to distinguish between the beginning, middle, and end of a word's sub-tokens. This two-dimensional approach provides a more nuanced representation of meaning by enabling the model to understand the relationships between words and their constituent morphemes. The model also incorporates **Whole Word Masking (WWM)**, a training technique where all sub-tokens corresponding to a single word are masked, encouraging the model to learn deeper contextual relationships. ##…