nfp.preprocessing.tokenizer.Tokenizer

class Tokenizer

Bases: object

A class to turn arbitrary inputs into integer classes.

Methods

__call__(item)

Check to see if the Tokenizer has seen item before, and if so, return the integer class associated with it. Otherwise, if we’re training, create a new integer class, otherwise return the ‘unknown’ class.