Skip to content

Instantly share code, notes, and snippets.

View hanneshapke's full-sized avatar

Hannes Hapke hanneshapke

View GitHub Profile
@hanneshapke
hanneshapke / tensorboard.sh
Last active February 12, 2023 18:27
TensorBoard Profiler + TensorFlow Serving
#!/bin/bash
TENSORFLOW_SERVING_VERSION=2.11.0
TENSORFLOW_SERVING_HOSTNAME=serving
TENSORFLOW_SERVING_MODEL_NAME=test_model
INTRA_OP_PARALLELISM=2
INTER_OP_PARALLELISM=2
TENSORBOARD_LOGDIR="/tmp/tensorboard"
DOCKER_PROFILER_TAG=tensorboard_profiler:latest
@hanneshapke
hanneshapke / install_gpu_driver_for_tf27.py
Last active April 2, 2022 17:03
Install Nvidia GPU and CUDA Drivers to support TF 2.7-2.8
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
@hanneshapke
hanneshapke / example-convolutional-neural-net-with-tf-keras.ipynb
Last active August 23, 2021 17:39
Example Convolutional Neural Net with tf.keras
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@hanneshapke
hanneshapke / parallelize.py
Created November 1, 2020 01:00
Parallelize data processing
# install job lib
from joblib import Parallel, delayed
import numpy as np
def myfun(p):
return np.linalg.norm(p[0]-p[1])
X = [1,2,3,4]
Y = [6,4,8,1]
@hanneshapke
hanneshapke / tfx_pipeline_for_bert_preprocessing_wo_tf-example.ipynb
Last active June 4, 2020 22:48
TFX_Pipeline_for_Bert_Preprocessing_wo_tf.Example.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
def _get_serve_tf_examples_fn(model, tf_transform_output):
model.tft_layer = tf_transform_output.transform_features_layer()
@tf.function
def serve_tf_examples_fn(serialized_tf_examples):
feature_spec = tf_transform_output.raw_feature_spec()
feature_spec.pop(_LABEL_KEY)
parsed_features = tf.io.parse_example(serialized_tf_examples, feature_spec)
transformed_features = model.tft_layer(parsed_features)
x = tf.keras.layers.Dense(256, activation='relu')(pooled_output)
dense = tf.keras.layers.Dense(64, activation='relu')(x)
pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)
model = tf.keras.Model(
inputs=[inputs['input_word_ids'],
inputs['input_mask'],
inputs['input_type_ids']],
outputs=pred
)
bert_layer = load_bert_layer()
pooled_output, _ = bert_layer(
[input_word_ids,
input_mask,
input_type_ids
]
)
feature_spec = tf_transform_output.transformed_feature_spec()
feature_spec.pop(_LABEL_KEY)
inputs = {
key: tf.keras.layers.Input(
shape=(max_seq_length),
name=key,
dtype=tf.int32)
for key in feature_spec.keys()
}
tf_transform_output = tft.TFTransformOutput(fn_args.transform_output)
train_dataset = _input_fn(fn_args.train_files, tf_transform_output, 32)
eval_dataset = _input_fn(fn_args.eval_files, tf_transform_output, 32)
...
model.fit(
train_dataset,
validation_data=eval_dataset,
...
)