tuan3w

Datasets

I had such a bad time trying to create english-vietnamese parallel corpus from bilingual stories, but it sucks. It just wastes a lot of time. So I try to find out as much corpora as possible throughout the internet. My final dataset consists of about 2.5M pair of sentences. You can find all corpora here: link

Model

I use OpenNMT to train my nmt model. Thanks Systran and HavardNLP for open source this project. It will help me and many others to understand how a industral translation system might work. The parameters of my model are as follow:

• Preprocesssing: Using aggressive tokenizer provided by OpenNMT

tuan3w

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *

tuan3w

# eth0 is public network interface

iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT

# Allow connection RELATED,ESTABLISHED on eth0
iptables -t filter -A INPUT -i eth0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Prevent DOCKER chain
iptables -t nat -I PREROUTING -m addrtype --dst-type LOCAL -j RETURN

tuan3w

worker_processes  auto; #should be 1 for Windows, for it doesn't support Unix domain socket

worker_rlimit_nofile 100000;


error_log logs/error.log error;

#if the module is compiled as a dynamic module and features relevant
#to RTMP are needed, the command below MUST be specified and MUST be
#located before events directive, otherwise the module won't be loaded

tuan3w

import torch
import torch.nn as nn
from torch.nn import Parameter
import math
import torch.nn.functional as F


class Scale(nn.Module):
    def __init__(self, channels):
        super(Scale, self).__init__()

tuan3w

ffmpeg -f video4linux2 -input_format mjpeg -s 1280x720 -i /dev/video0 \
-vf "drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf: \
text=\'%{pts\:gmtime\:0\:%M\\\:%S}\': fontcolor=white@0.8: x=7: y=7: fontsize=24" -vcodec libx264 \
-preset veryfast -f mp4 -pix_fmt yuv420p -y output.mp4

tuan3w

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# vim:fenc=utf-8
#
# Copyright © 2017 zc <zc@www>
#
# Distributed under terms of the MIT license.

"""
Service registrator agent

tuan3w

(dev3) ❯ python synthesis.py --hparams="builder=deepvoice3,preset=deepvoice3_ljspeech" checkpoints_deepvoice3/checkpoint_step000390000.pth test.txt samples
Command line args:
 {'--checkpoint-postnet': None,
 '--checkpoint-seq2seq': None,
 '--file-name-suffix': '',
 '--help': False,
 '--hparams': 'builder=deepvoice3,preset=deepvoice3_ljspeech',
 '--max-decoder-steps': '500',
 '--output-html': False,
 '--replace_pronunciation_prob': '0.0',

tuan3w

# --------------------------------------------------------
# Fast R-CNN
# Copyright (c) 2015 Microsoft
# Licensed under The MIT License [see LICENSE for details]
# Written by Ross Girshick
# --------------------------------------------------------

import os
from os.path import join as pjoin
import numpy as np

tuan3w

import com.soundcloud.prometheus.hystrix.HystrixPrometheusMetricsPublisher;
import com.vcc.bigdata.micro.cmd.FailCommand;

import io.vertx.circuitbreaker.CircuitBreaker;
import io.vertx.circuitbreaker.CircuitBreakerOptions;
import io.vertx.circuitbreaker.HystrixMetricHandler;
import io.vertx.core.AbstractVerticle;
import io.vertx.core.http.HttpServer;
import io.vertx.core.http.HttpServerOptions;
import io.vertx.ext.web.Router;