Skip to content

Instantly share code, notes, and snippets.

@stefanondisponibile

stefanondisponibile/tf2_TextVectorizationLayer_standardize.py

Last active January 20, 2020 16:28
Show Gist options
  • Save stefanondisponibile/9e5682087c6354974fcb2910b63a49b2 to your computer and use it in GitHub Desktop.
Save stefanondisponibile/9e5682087c6354974fcb2910b63a49b2 to your computer and use it in GitHub Desktop.
Download ZIP
TensorFlow TextVectorizationLayer.
Raw
tf2_TextVectorizationLayer_standardize.py
# No standardization.
vectorize_layer = TextVectorization(output_mode="int", max_tokens=5, standardize=None)
# Lowercase text and strip punctuation. [DEFAULT]
vectorize_layer = TextVectorization(output_mode="int", max_tokens=5, standardize="lower_and_strip_punctuation")
# Apply a custom standardization function.
def replace_foo_with_bar(input_data):
return tf.strings.regex_replace(input_data, "foo", "bar")
vectorize_layer = TextVectorization(output_mode="int", max_tokens=5, standardize=replace_foo_with_bar)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment