Skip to content

Instantly share code, notes, and snippets.

andreasvc / longest non-taboo sequence.ipynb
Last active Feb 18, 2021
Find the longest sequence of tokens in a text without any taboo n-grams
View longest non-taboo sequence.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
andreasvc / Dockerfile
Created Sep 8, 2020
docker-compose example
View Dockerfile
# This is a comment
FROM ubuntu:20.04
MAINTAINER Andreas van Cranenburgh <>
RUN ln -fs /usr/share/zoneinfo/Europe/Amsterdam /etc/localtime
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
build-essential \
curl \
git \
python3 \
import random
from timeit import timeit
import re
import re2
re_ip = re.compile(br'\d+\.\d+\.\d+\.\d+')
re2_ip = re2.compile(br'\d+\.\d+\.\d+\.\d+')
lines = ['.'.join(str(random.randint(1, 255)) for _ in range(4)).encode('utf8')
for _ in range(16000)]
import datetime
def addseconds(timestamp, seconds):
"""Take timestamp as string and add seconds to it.
>>> addseconds('00:01:45,667', 1)
>>> addseconds('00:01:45,667', 0.5)
View detectedlangs_not_nl.tsv
filename lang confidence read_bytes
train/neg/3706_2.txt en 81.0 1268
train/neg/9466_1.txt en 99.0 1066
train/neg/6464_2.txt en 99.0 1248
train/neg/14850_2.txt en 99.0 1128
train/neg/4674_2.txt en 99.0 1306
train/neg/7036_1.txt fy 68.0 997
train/neg/7454_2.txt en 63.0 688
train/neg/4856_2.txt en 99.0 1363
train/neg/12096_2.txt en 99.0 1339
andreasvc /
Last active Oct 3, 2019
Apply polyglot language detection recursively
"""Apply polyglot language detection to all .txt files under current directory
(searched recursively), write report in tab-separated file detectedlangs.tsv.
import os
from glob import glob
from polyglot.detect import Detector
from polyglot.detect.base import UnknownLanguage
def main():
andreasvc /
Created Sep 17, 2019
More Python exercises
  1. Write a function char_freq() that takes a string and builds a frequency listing of the characters contained in it. Represent the frequency listing as a Python dictionary. Try it with something like char_freq("abbabcbdbabdbdbabababcbcbab").

  2. Write a function char_freq_table() that take a file name as argument, builds a frequency listing of the characters contained in the file, and prints a sorted and nicely formatted character frequency table to the screen.

  3. The third person singular verb form in English is distinguished by the suffix -s, which is added to the stem of the infinitive form: run -> runs. A simple set of rules can be given as follows:

    a. If the verb ends in y, remove it and add ies b. If the verb ends in o, ch, s, sh, x or z, add es c. By default just add s


Python exercises

  1. Define a function max() that takes two numbers as arguments and returns the largest of them. Use the if-then-else construct available in Python. (It is true that Python has the max() function built in, but writing it yourself is nevertheless a good exercise).

  2. Define a function max_of_three() that takes three numbers as arguments and returns the largest of them.

  3. Define a function that computes the length of a given list or string. (It is true that Python has the len() function built in, but writing it yourself is nevertheless a good exercise).

  4. Write a function that takes a character (i.e. a string of length 1) and returns True if it is a vowel, False otherwise.

andreasvc /
Last active Jul 8, 2020
A baseline Bag-of-Words text classification
"""A baseline Bag-of-Words text classification.
Usage: python3 <train.txt> <test.txt> [--svm] [--tfidf] [--bigrams]
train.txt and test.txt should contain one "document" per line,
first token should be the label.
The default is to use regularized Logistic Regression and relative frequencies.
Pass --svm to use Linear SVM instead.
Pass --tfidf to use tf-idf instead of relative frequencies.
Pass --bigrams to use bigrams instead of unigrams.
"""Prepare for use with fastText.
Divide train set into 90% train and 10% dev, balance positive and negative
rewiews, and shuffle. Write result in fastText format."""
import os
import re
import random
import glob
from syntok.tokenizer import Tokenizer