tf.keras.layers.StringLookup

A preprocessing layer which maps string features to integer indices.

Inherits From: PreprocessingLayer, Layer, Module

tf.keras.layers.StringLookup(
    max_tokens=None,
    num_oov_indices=1,
    mask_token=None,
    oov_token='[UNK]',
    vocabulary=None,
    idf_weights=None,
    encoding=None,
    invert=False,
    output_mode='int',
    sparse=False,
    pad_to_max_tokens=False,
    **kwargs
)

This layer translates a set of arbitrary strings into integer output via a table-based vocabulary lookup.

The vocabulary for the layer must be either supplied on construction or learned via adapt(). During adapt(), the layer will analyze a data set, determine the frequency of individual strings tokens, and create a vocabulary from them. If the vocabulary is capped in size, the most frequent tokens will be used to create the vocabulary and all others will be treated as out-of-vocabulary (OOV).

There are two possible output modes for the layer. When output_mode is "int", input strings are converted to their index in the vocabulary (an integer). When output_mode is "multi_hot", "count", or "tf_idf", input strings are encoded into an array where each dimension corresponds to an element in the vocabulary.

The vocabulary can optionally contain a mask token as well as an OOV token (which can optionally occupy multiple indices in the vocabulary, as set by num_oov_indices). The position of these tokens in the vocabulary is fixed. When output_mode is "int", the vocabulary will begin with the mask token (if set), followed by OOV indices, followed by the rest of the vocabulary. When output_mode is "multi_hot", "count", or "tf_idf" the vocabulary will begin with OOV indices and instances of the mask token will be dropped.

For an overview and full list of preprocessing layers, see the preprocessing guide.

max_tokens Maximum size of the vocabulary for this layer. This should only be specified when adapting the vocabulary or when setting pad_to_max_tokens=True. If None, there is no cap on the size of the vocabulary. Note that this size includes the OOV and mask tokens. Defaults to None.
num_oov_indices The number of out-of-vocabulary tokens to use. If this value is more than 1, OOV inputs are hashed to determine their OOV value. If this value is 0, OOV inputs will cause an error when calling the layer. Defaults to 1.
mask_token A token that represents masked inputs. When output_mode is "int", the token is included in vocabulary and mapped to index 0. In other output modes, the token will not appear in the vocabulary and instances of the mask token in the input will be dropped. If set to None, no mask term will be added. Defaults to None.
oov_token Only used when invert is True. The token to return for OOV indices. Defaults to "[UNK]".
vocabulary Optional. Either an array of strings or a string path to a text file. If passing an array, can pass a tuple, list, 1D numpy array, or 1D tensor containing the string vocbulary terms. If passing a file path, the file should contain one line per term in the vocabulary. If this argument is set, there is no need to adapt the layer.
idf_weights Only valid when output_mode is "tf_idf". A tuple, list, 1D numpy array, or 1D tensor or the same length as the vocabulary, containing the floating point inverse document frequency weights, which will be multiplied by per sample term counts for the final tf_idf weight. If the vocabulary argument is set, and output_mode is "tf_idf", this argument must be supplied.
invert Only valid when output_mode is "int". If True, this layer will map indices to vocabulary items instead of mapping vocabulary items to indices. Default to False.
output_mode Specification for the output of the layer. Defaults to "int". Values can be "int", "one_hot", "multi_hot", "count", or "tf_idf" configuring the layer as follows:

  • "int": Return the raw integer indices of the input tokens.
  • "one_hot": Encodes each individual element in the input into an array the same size as the vocabulary, containing a 1 at the element index. If the last dimension is size 1, will encode on that dimension. If the last dimension is not size 1, will append a new dimension for the encoded output.
  • "multi_hot": Encodes each sample in the input into a single array the same size as the vocabulary, containing a 1 for each vocabulary term present in the sample. Treats the last dimension as the sample dimension, if input shape is (..., sample_length), output shape will be (..., num_tokens).
  • "count": As "multi_hot", but the int array contains a count of the number of times the token at that index appeared in the sample.
  • "tf_idf": As "multi_hot", but the TF-IDF algorithm is applied to find the value in each token slot. For "int" output, any shape of input and output is supported. For all other output modes, currently only output up to rank 2 is supported.
pad_to_max_tokens Only applicable when output_mode is "multi_hot", "count", or "tf_idf". If True, the output will have its feature axis padded to max_tokens even if the number of unique tokens in the vocabulary is less than max_tokens, resulting in a tensor of shape [batch_size, max_tokens] regardless of vocabulary size. Defaults to False.
sparse Boolean. Only applicable when output_mode is "multi_hot", "count", or "tf_idf". If True, returns a SparseTensor instead of a dense Tensor. Defaults to False.

Examples:

Creating a lookup layer with a known vocabulary

This example creates a lookup layer with a pre-existing vocabulary.

vocab = ["a", "b", "c", "d"]
data = tf.constant([["a", "c", "d"], ["d", "z", "b"]])
layer = tf.keras.layers.StringLookup(vocabulary=vocab)
layer(data)
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[1, 3, 4],
       [4, 0, 2]])>

Creating a lookup layer with an adapted vocabulary

This example creates a lookup layer and generates the vocabulary by analyzing the dataset.

data = tf.constant([["a", "c", "d"], ["d", "z", "b"]])
layer = tf.keras.layers.StringLookup()
layer.adapt(data)
layer.get_vocabulary()
['[UNK]', 'd', 'z', 'c', 'b', 'a']

Note that the OOV token "[UNK]" has been added to the vocabulary. The remaining tokens are sorted by frequency ("d", which has 2 occurrences, is first) then by inverse sort order.

data = tf.constant([["a", "c", "d"], ["d", "z", "b"]])
layer = tf.keras.layers.StringLookup()
layer.adapt(data)
layer(data)
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[5, 3, 1],
       [1, 2, 4]])>

Lookups with multiple OOV indices

This example demonstrates how to use a lookup layer with multiple OOV indices. When a layer is created with more than one OOV index, any OOV values are hashed into the number of OOV buckets, distributing OOV values in a deterministic fashion across the set.

vocab = ["a", "b", "c", "d"]
data = tf.constant([["a", "c", "d"], ["m", "z", "b"]])
layer = tf.keras.layers.StringLookup(vocabulary=vocab, num_oov_indices=2)
layer(data)
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[2, 4, 5],
       [0, 1, 3]])>

Note that the output for OOV value 'm' is 0, while the output for OOV value 'z' is 1. The in-vocab terms have their output index increased by 1 from earlier examples (a maps to 2, etc) in order to make space for the extra OOV value.

One-hot output

Configure the layer with output_mode='one_hot'. Note that the first num_oov_indices dimensions in the ont_hot encoding represent OOV values.

vocab = ["a", "b", "c", "d"]
data = tf.constant(["a", "b", "c", "d", "z"])
layer = tf.keras.layers.StringLookup(
    vocabulary=vocab, output_mode='one_hot')
layer(data)
<tf.Tensor: shape=(5, 5), dtype=float32, numpy=
  array([[0., 1., 0., 0., 0.],
         [0., 0., 1., 0., 0.],
         [0., 0., 0., 1., 0.],
         [0., 0., 0., 0., 1.],
         [1., 0., 0., 0., 0.]], dtype=float32)>

Multi-hot output

Configure the layer with output_mode='multi_hot'. Note that the first num_oov_indices dimensions in the multi_hot encoding represent OOV values.

vocab = ["a", "b", "c", "d"]
data = tf.constant([["a", "c", "d", "d"], ["d", "z", "b", "z"]])
layer = tf.keras.layers.StringLookup(
    vocabulary=vocab, output_mode='multi_hot')
layer(data)
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
  array([[0., 1., 0., 1., 1.],
         [1., 0., 1., 0., 1.]], dtype=float32)>

Token count output

Configure the layer with output_mode='count'. As with multi_hot output, the first num_oov_indices dimensions in the output represent OOV values.

vocab = ["a", "b", "c", "d"]
data = tf.constant([["a", "c", "d", "d"], ["d", "z", "b", "z"]])
layer = tf.keras.layers.StringLookup(
    vocabulary=vocab, output_mode='count')
layer(data)
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
  array([[0., 1., 0., 1., 2.],
         [2., 0., 1., 0., 1.]], dtype=float32)>

TF-IDF output

Configure the layer with output_mode="tf_idf". As with multi_hot output, the first num_oov_indices dimensions in the output represent OOV values.

Each token bin will output token_count * idf_weight, where the idf weights are the inverse document frequency weights per token. These should be provided along with the vocabulary. Note that the idf_weight for OOV values will default to the average of all idf weights passed in.

vocab = ["a", "b", "c", "d"]
idf_weights = [0.25, 0.75, 0.6, 0.4]
data = tf.constant([["a", "c", "d", "d"], ["d", "z", "b", "z"]])
layer = tf.keras.layers.StringLookup(output_mode="tf_idf")
layer.set_vocabulary(vocab, idf_weights=idf_weights)
layer(data)
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
  array([[0.  , 0.25, 0.  , 0.6 , 0.8 ],
         [1.0 , 0.  , 0.75, 0.  , 0.4 ]], dtype=float32)>

To specify the idf weights for oov values, you will need to pass the entire vocabularly including the leading oov token.

vocab = ["[UNK]", "a", "b", "c", "d"]
idf_weights = [0.9, 0.25, 0.75, 0.6, 0.4]
data = tf.constant([["a", "c", "d", "d"], ["d", "z", "b", "z"]])
layer = tf.keras.layers.StringLookup(output_mode="tf_idf")
layer.set_vocabulary(vocab, idf_weights=idf_weights)
layer(data)
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
  array([[0.  , 0.25, 0.  , 0.6 , 0.8 ],
         [1.8 , 0.  , 0.75, 0.  , 0.4 ]], dtype=float32)>

When adapting the layer in "tf_idf" mode, each input sample will be considered a document, and IDF weight per token will be calculated as log(1 + num_documents / (1 + token_document_count)).

Inverse lookup

This example demonstrates how to map indices to strings using this layer. (You can also use adapt() with inverse=True, but for simplicity we'll pass the vocab in this example.)

vocab = ["a", "b", "c", "d"]
data = tf.constant([[1, 3, 4], [4, 0, 2]])
layer = tf.keras.layers.StringLookup(vocabulary=vocab, invert=True)
layer(data)
<tf.Tensor: shape=(2, 3), dtype=string, numpy=
array([[b'a', b'c', b'd'],
       [b'd', b'[UNK]', b'b']], dtype=object)>

Note that the first index correspond to the oov token by default.

Forward and inverse lookup pairs

This example demonstrates how to use the vocabulary of a standard lookup layer to create an inverse lookup layer.

vocab = ["a", "b", "c", "d"]
data = tf.constant([["a", "c", "d"], ["d", "z", "b"]])
layer = tf.keras.layers.StringLookup(vocabulary=vocab)
i_layer = tf.keras.layers.StringLookup(vocabulary=vocab, invert=True)
int_data = layer(data)
i_layer(int_data)
<tf.Tensor: shape=(2, 3), dtype=string, numpy=
array([[b'a', b'c', b'd'],
       [b'd', b'[UNK]', b'b']], dtype=object)>

In this example, the input value "z" resulted in an output of "[UNK]", since 1000 was not in the vocabulary - it got represented as an OOV, and all OOV values are returned as "[UNK]" in the inverse layer. Also, note that for the inverse to work, you must have already set the forward layer vocabulary either directly or via adapt() before calling get_vocabulary().

is_adapted Whether the layer has been fit to data already.

Methods

adapt

View source

adapt(
    data, batch_size=None, steps=None
)

Fits the state of the preprocessing layer to the data being passed.

After calling adapt on a layer, a preprocessing layer's state will not update during training. In order to make preprocessing layers efficient in any distribution context, they are kept constant with respect to any compiled tf.Graphs that call the layer. This does not affect the layer use when adapting each layer only once, but if you adapt a layer multiple times you will need to take care to re-compile any compiled functions as follows:

  • If you are adding a preprocessing layer to a keras.Model, you need to call model.compile after each subsequent call to adapt.
  • If you are calling a preprocessing layer inside tf.data.Dataset.map, you should call map again on the input tf.data.Dataset after each adapt.
  • If you are using a tf.function directly which calls a preprocessing layer, you need to call tf.function again on your callable after each subsequent call to adapt.

tf.keras.Model example with multiple adapts:

layer = tf.keras.layers.experimental.preprocessing.Normalization(
    axis=None)
layer.adapt([0, 2])
model = tf.keras.Sequential(layer)
model.predict([0, 1, 2])
array([-1.,  0.,  1.], dtype=float32)
layer.adapt([-1, 1])
model.compile() # This is needed to re-compile model.predict!
model.predict([0, 1, 2])
array([0., 1., 2.], dtype=float32)

tf.data.Dataset example with multiple adapts:

layer = tf.keras.layers.experimental.preprocessing.Normalization(
    axis=None)
layer.adapt([0, 2])
input_ds = tf.data.Dataset.range(3)
normalized_ds = input_ds.map(layer)
list(normalized_ds.as_numpy_iterator())
[array([-1.], dtype=float32),
 array([0.], dtype=float32),
 array([1.], dtype=float32)]
layer.adapt([-1, 1])
normalized_ds = input_ds.map(layer) # Re-map over the input dataset.
list(normalized_ds.as_numpy_iterator())
[array([0.], dtype=float32),
 array([1.], dtype=float32),
 array([2.], dtype=float32)]

Arguments
data The data to train on. It can be passed either as a tf.data Dataset, or as a numpy array.
batch_size Integer or None. Number of samples per state update. If unspecified, batch_size will default to 32. Do not specify the batch_size if your data is in the form of datasets, generators, or keras.utils.Sequence instances (since they generate batches).
steps Integer or None. Total number of steps (batches of samples) When training with input tensors such as TensorFlow data tensors, the default None is equal to the number of samples in your dataset divided by the batch size, or 1 if that cannot be determined. If x is a tf.data dataset, and 'steps' is None, the epoch will run until the input dataset is exhausted. When passing an infinitely repeating dataset, you must specify the steps argument. This argument is not supported with array inputs.

compile

View source

compile(
    run_eagerly=None, steps_per_execution=None
)

Configures the layer for adapt.

Arguments
run_eagerly Bool. Defaults to False. If True, this Model's logic will not be wrapped in a tf.function. Recommended to leave this as None unless your Model cannot be run inside a tf.function. steps_per_execution: Int. Defaults to 1. The number of batches to run during each tf.function call. Running multiple batches inside a single tf.function call can greatly improve performance on TPUs or small models with a large Python overhead.

get_vocabulary

View source

get_vocabulary(
    include_special_tokens=True
)

Returns the current vocabulary of the layer.

Args
include_special_tokens If True, the returned vocabulary will include mask and OOV tokens, and a term's index in the vocabulary will equal the term's index when calling the layer. If False, the returned vocabulary will not include any mask or OOV tokens.

reset_state

View source

reset_state()

Resets the statistics of the preprocessing layer.

set_vocabulary

View source

set_vocabulary(
    vocabulary, idf_weights=None
)

Sets vocabulary (and optionally document frequency) data for this layer.

This method sets the vocabulary and idf weights for this layer directly, instead of analyzing a dataset through adapt. It should be used whenever the vocab (and optionally document frequency) information is already known. If vocabulary data is already present in the layer, this method will replace it.

Args
vocabulary Either an array or a string path to a text file. If passing an array, can pass a tuple, list, 1D numpy array, or 1D tensor containing the vocbulary terms. If passing a file path, the file should contain one line per term in the vocabulary.
idf_weights A tuple, list, 1D numpy array, or 1D tensor of inverse document frequency weights with equal length to vocabulary. Must be set if output_mode is "tf_idf". Should not be set otherwise.

Raises
ValueError If there are too many inputs, the inputs do not match, or input data is missing.
RuntimeError If the vocabulary cannot be set when this function is called. This happens when "multi_hot", "count", and "tf_idf" modes, if pad_to_max_tokens is False and the layer itself has already been called.
RuntimeError If a tensor vocabulary is passed outside of eager execution.

update_state

View source

update_state(
    data
)

Accumulates statistics for the preprocessing layer.

Arguments
data A mini-batch of inputs to the layer.

vocab_size

View source

vocab_size()

vocabulary_size

View source

vocabulary_size()

Gets the current size of the layer's vocabulary.

Returns
The integer size of the voculary, including optional mask and oov indices.