Skip to content

Instantly share code, notes, and snippets.

@saravanabalagi
saravanabalagi / tf2_tracing_estimators.md
Created January 16, 2020 11:20
Profiling Tensorflow Estimator in tf2

Based on Summary Trace API,

device_name = tf.test.gpu_device_name()
if not tf.test.is_gpu_available():
    raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))
os.makedirs(os.path.join(args.exp_dir, 'plugins/profile'), exist_ok=True)
tf.summary.trace_on(graph=True, profiler=True)
tracing_params = params.copy()
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rzykov
rzykov / XgBoostRankSparkScala.scala
Last active March 31, 2022 22:43
XGboost Spark - ranking problem
import _root_.ml.dmlc.xgboost4j.scala.spark.XGBoost
import org.apache.spark.ml.feature.LabeledPoint
def encodeFeaturesToLabeledPoint(features: RDD[Feature], relevance: Option[RDD[Relevance]], workers: Int)
(implicit parallel: Int): (RDD[LabeledPoint], Seq[String], Seq[Seq[Int]]) = {
val missingValue = Double.NaN
val names = features
.map { _.name }
@smhanov
smhanov / description.md
Last active November 7, 2023 11:52
Pipeline multiprocessing for python with generators

Pipeline multiprocessing in Python with generators

Similar to mpipe, this short module lets you string together tasks so that they are executed in parallel.

The difference is that it lets you use generators, functions which yield results instead of returning them. Each yielded item gets passed to the next stage.

You can specify that one or more copies of the workers operate on the input queue.

Things that can be in a stage

@smhanov
smhanov / dawg.py
Last active April 26, 2023 16:01
Use a DAWG as a map
#!/usr/bin/python3
# By Steve Hanov, 2011. Released to the public domain.
# Please see http://stevehanov.ca/blog/index.php?id=115 for the accompanying article.
#
# Based on Daciuk, Jan, et al. "Incremental construction of minimal acyclic finite-state automata."
# Computational linguistics 26.1 (2000): 3-16.
#
# Updated 2014 to use DAWG as a mapping; see
# Kowaltowski, T.; CL. Lucchesi (1993), "Applications of finite automata representing large vocabularies",
# Software-Practice and Experience 1993
@amertkara
amertkara / aws_utils.py
Last active March 13, 2024 11:10
Amazon SNS Notification Verification with Python, M2Crypto. When the SNS pushes a notification, a receiver should verify the origin/integrity of the push notification (AWS) using the signature and certificate provided in the notification data. The function `verify_sns_notification` below takes the request object and verifies the origin/integrity…
# -*- coding: utf-8 -*-
import json
import urllib2
from M2Crypto import X509
from base64 import b64decode
from M2Crypto.Err import M2CryptoError
SNS_MESSAGE_TYPE_SUB_NOTIFICATION = "SubscriptionConfirmation"
SNS_MESSAGE_TYPE_NOTIFICATION = "Notification"
SNS_MESSAGE_TYPE_UNSUB_NOTIFICATION = "UnsubscribeConfirmation"
@redguardtoo
redguardtoo / .gitconfig
Last active April 25, 2023 07:51
my git setup
[apply]
whitespace = nowarn
[user]
name = Chen Bin
email = cb@myemail.com
[core]
# @see http://comments.gmane.org/gmane.comp.version-control.git/166098
filemode = true
ignorecase = false
autocrlf = false
@mtmtcode
mtmtcode / auto-rsync.el
Last active September 2, 2022 00:30
auto-rsync.el - Emacs minor mode to execute rsync automaticlly
;;; auto-rsync-mode -- minor mode for auto rsync
;;
;; Author: @l3msh0
;;
;;; Example
;;
;; (require 'auto-rsync)
;; (auto-rsync-mode t)
;; (setq auto-rsync-dir-alist