Tokenizer usage examples

The Tokenizer class allows transforming arbitrary inputs into integer classes

from nfp.preprocessing import Tokenizer

tokenizer = Tokenizer()
tokenizer.train = True

The 0 and 1 classes are reserved for the <MASK> and missing labels, respectively.

[tokenizer(item) for item in ['A', 'B', 'C', 'A']]

[2, 3, 4, 2]

When train is set to False, unknown items are assigned the missing label

tokenizer.train = False
[tokenizer(item) for item in ['A', 'D']]

[2, 1]

The total number of seen classes is available from the num_classes property, useful to initializing embedding layer weights.

tokenizer.num_classes

4