cayleypy.models.GroupTokenizer

class cayleypy.models.GroupTokenizer(groups: Sequence[Sequence[int]])

Converts states to sequences of tokens, one token per group of consecutive elements of a state.

States of puzzles are usually encoded as permutations of sticker ids, where stickers of one piece occupy consecutive positions. Such state can be described more compactly by one token per piece: the token says which piece is in this slot and how it is oriented. For example, the Megaminx state has 120 elements: 20 corners with 3 stickers each, followed by 30 edges with 2 stickers each. It is described by 50 tokens (one per piece), and each token takes one of 60 values (20 corners times 3 orientations, or 30 edges times 2 orientations). Such tokenization is described by groups [[3, 20], [2, 30]], see cayleypy.models.ModelConfig.tokenizer_groups.

The i-th token is the value of the first element of the i-th group, counted from the beginning of the segment this group belongs to (a segment is all groups described by one [group_size, num_groups] pair). Therefore token ids of one segment are in range [0, group_size * num_groups), and token ids of different segments overlap. Models can tell segments apart by position, or by adding an embedding of token_type_ids to the embedding of tokens.

This encodes the state without loss of information as long as elements of one group are always stickers of one piece, listed in the same cyclic order - then the whole group is determined by its first element. Use verify() to check this for a particular graph.

Example

>>> from cayleypy.models import GroupTokenizer
>>> tokenizer = GroupTokenizer([[3, 20], [2, 30]])  # Megaminx.
>>> tokenizer.n_tokens, tokenizer.vocab_size
(50, 60)

__init__(groups: Sequence[Sequence[int]])

Creates tokenizer for states grouped as described by groups.

Parameters:

groups – Specification of groups, as a list of [group_size, num_groups] pairs. Groups are laid out in the state consecutively, in the order they are listed here.

Methods

__init__(groups)	Creates tokenizer for states grouped as described by groups.
from_config(config)	Creates tokenizer described by tokenizer_groups field of the given model config.
verify(graph_def)	Checks that states of the given graph are tokenized without loss of information.

Attributes

groups	Groups this tokenizer was created with, as a list of [group_size, num_groups] pairs.
state_size	Number of elements in the state this tokenizer expects.
n_tokens	Number of tokens one state is converted to.
n_token_types	Number of segments (i.e. of [group_size, num_groups] pairs describing this tokenization).
vocab_size	Number of distinct values a token can take (i.e. size of vocabulary needed to embed tokens).
token_type_ids	Index of the segment each token belongs to, of shape [n_tokens].

static from_config(config: ModelConfig) → GroupTokenizer

Creates tokenizer described by tokenizer_groups field of the given model config.

Parameters:

config – Config of a model that consumes tokens.

Returns:

Tokenizer described by this config.

groups: list[list[int]]

Groups this tokenizer was created with, as a list of [group_size, num_groups] pairs.

n_token_types: int

Number of segments (i.e. of [group_size, num_groups] pairs describing this tokenization).

n_tokens: int

Number of tokens one state is converted to.

state_size: int

Number of elements in the state this tokenizer expects.

token_type_ids: Tensor

Index of the segment each token belongs to, of shape [n_tokens].

verify(graph_def: CayleyGraphDef) → None

Checks that states of the given graph are tokenized without loss of information.

This holds when the central state lists stickers of every piece in one group, and every generator moves stickers of one piece to positions of one piece, preserving their cyclic order. Then every state reachable from the central state has one piece per group, so the group is determined by its first element (i.e. by its token).

Parameters:

graph_def – Definition of the graph whose states are going to be tokenized.

Raises:

ValueError – If states of this graph cannot be tokenized this way.

vocab_size: int

Number of distinct values a token can take (i.e. size of vocabulary needed to embed tokens).