Neural Architectures
Pretraining
Tokenization
Surprise
100

This is a type of architecture that most language models use today based on attention.

What is a Transformer?

100

The name for the process of asking a generative pre-trained language model to obtain an answer.  

What is a prompt?

100

The name of the sequence of characters, bits, or letters that form smaller part of a larger word used in tokenization. 

What is a subword?

100

A group of individual languages that are related to one another through descent from a common ancestral language.

What is a language family?

200

A mechanism in a neural architecture that allows to look at all tokens in parallel and decide on their weights.

What is attention?

200

The name of a type of language model that is trained to complete a given sequence.  

What is a generative language model?

200

This is the name of a token that was used in traditional tokenization for tokens that are not in the vocabulary.

What is the unknown or UNK token?

200

The name of a method to jointly train word embeddings in two languages simultaneously.

What is Bilingual Skip-gram?

300

A type of architecture where one component takes the input and gives a compressed representation to the other component that creates the output.

What is an encoder-decoder?

300

A type of architecture for a pre-trained language model that scales well and is particularly well suited to for natural language generation.

What is a decoder-only model?

300

A subword tokenization algorithm that iteratively merges the most frequent adjacent pairs of symbols.

What is Byte-Pair Encoding?

300

A method for automatically aligning two monolingual vector spaces into one crosslingual vector space. 

What is projection-based alignment?

400

A type of architecture that is designed to process sequential data in a serial manner with a kind of memory vector. 

What is a Recurrent Neural Network (RNN)?
400

The name of a type of language model that hides words or sequences in a sentence during training.

What is a masked language model?

400

A tokenization algorithm that operates on raw UTF-8 bytes instead of Unicode characters.

What is byte-level Byte-Pair Encoding?

400

The rule that states that publications need to explicitly mention the languages they process, even if it is English. 

What is the Bender Rule?