This is a type of architecture that most language models use today based on attention.
What is a Transformer?
The name for the process of asking a generative pre-trained language model to obtain an answer.
What is a prompt?
The name of the sequence of characters, bits, or letters that form smaller part of a larger word used in tokenization.
What is a subword?
A group of individual languages that are related to one another through descent from a common ancestral language.
What is a language family?
A mechanism in a neural architecture that allows to look at all tokens in parallel and decide on their weights.
What is attention?
The name of a type of language model that is trained to complete a given sequence.
What is a generative language model?
This is the name of a token that was used in traditional tokenization for tokens that are not in the vocabulary.
What is the unknown or UNK token?
The name of a method to jointly train word embeddings in two languages simultaneously.
What is Bilingual Skip-gram?
A type of architecture where one component takes the input and gives a compressed representation to the other component that creates the output.
What is an encoder-decoder?
A type of architecture for a pre-trained language model that scales well and is particularly well suited to for natural language generation.
What is a decoder-only model?
A subword tokenization algorithm that iteratively merges the most frequent adjacent pairs of symbols.
What is Byte-Pair Encoding?
A method for automatically aligning two monolingual vector spaces into one crosslingual vector space.
What is projection-based alignment?
A type of architecture that is designed to process sequential data in a serial manner with a kind of memory vector.
The name of a type of language model that hides words or sequences in a sentence during training.
What is a masked language model?
A tokenization algorithm that operates on raw UTF-8 bytes instead of Unicode characters.
What is byte-level Byte-Pair Encoding?
The rule that states that publications need to explicitly mention the languages they process, even if it is English.
What is the Bender Rule?