akhan619

Exploring Tokenizers from Hugging Face

Hugging Face (HF) has made NLP (Natural Language Processing) a breeze. In this post, we are going to take a look at tokenization using a hands on approach with the help of the Tokenizers library. We are going to load a real world dataset containing 10-K filings of public firms and see how to train a tokenizer from scratch based on the BERT tokenization scheme. In the process we will understand tokenization in detail and some gotchas to keep an eye out for.

Background on NLP (Optional)

If you already have an understanding of the NLP pipeline, you can safely skip this section.

For any NLP task, one of the first steps is pre-processing the data so that it can be fed into our NLP models. For those new to NLP, the general pipeline for any NLP task (text classification, question answering, etc.) is as follows:

manuelmazzuola

package main

import (
    "k8s.io/apimachinery/pkg/api/meta"
    "k8s.io/client-go/discovery"
    "k8s.io/client-go/discovery/cached/memory"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/restmapper"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/client-go/tools/clientcmd/api"

willprice

Install OpenCV 4.1.2 on Raspbian Buster

$ chmod +x *.sh
$ ./download-opencv.sh
$ ./install-deps.sh
$ ./build-opencv.sh
$ cd ~/opencv/opencv-4.1.2/build
$ sudo make install

mozillazg

from flask import Flask, render_template
from flask_paginate import Pagination, get_page_args


app = Flask(__name__)
app.template_folder = ''
users = list(range(100))


def get_users(offset=0, per_page=10):

simonw

How to recover lost Python source code if it's still resident in-memory

I screwed up using git ("git checkout --" on the wrong file) and managed to delete the code I had just written... but it was still running in a process in a docker container. Here's how I got it back, using https://pypi.python.org/pypi/pyrasite/ and https://pypi.python.org/pypi/uncompyle6

Attach a shell to the docker container

Install GDB (needed by pyrasite)

apt-get update && apt-get install gdb

joepie91

'use strict';

const parseExpression = require("./parse-expression");

function findAll(regex, target) {
	let results = [], match;
	
	while (match = regex.exec(target)) {
		results.push(match);
	}

markwallsgrove

package main

// https://godoc.org/github.com/inconshreveable/go-update

// This gist documents go-install using a SHA-256 checksum,
// elliptic curve (prime 256) encrypted signature & bsdiff
// formatted patch.

// These steps took me over a hour to figure. go-install currently
// doesn't include a tutorial or quick start guide, so I created this

shmup

Transmission Blocklist

The Transmission torrent client has an option to set a Blocklist, which helps protect you from getting caught and having the DMCA send a letter/email.

It's as simple as downloading and installing the latest client:

• https://transmissionbt.com/download/

abhishektomar

#!/bin/bash

# Checking whether user has enough permission to run this script
sudo -n true
if [ $? -ne 0 ]
    then
        echo "This script requires user to have passwordless sudo access"
        exit
fi

lmars

# create a redis app
flynn create --remote "" redis

# create a release using the latest (at the time of writing) Docker Redis image
flynn -a redis release add -f config.json "https://registry.hub.docker.com?name=redis&id=868be653dea3ff6082b043c0f34b95bb180cc82ab14a18d9d6b8e27b7929762c"

# scale the server to one process. This may time out initially as the server pulls the image, but watch "flynn -a redis ps" and should come up.
flynn -a redis scale server=1

# redis should now be running in the cluster at redis.discoverd:6379