kuenishi

Apache Ozone Multipart Upload

Protocol

InitiateMultipartUpload -> UploadPart (xN) -> CompleteMultipartUpload
                                 +-> AbortMultipartUpload

• Limits https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html

kuenishi

OM Client Initialization

• OzoneClient は EndpointBase でInjectされている https://github.com/apache/ozone/blob/ozone-1.1.0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/EndpointBase.java#L43-L44
• OMClientProducer に bind されている https://github.com/apache/ozone/blob/ozone-1.1.0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java#L70
• OzoneClientFactory.getRpcClient(...) 内で omTransport を作っている
• まず getClientProtocol(config, serviceId) している https://github.com/apache/ozone/blob/f2406c4b4eb2b1295d06ec379cfe684ed9a16637/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneClientFactory.java#L243
• これが new RpcClient(config, serviceId) している https://github.com/apache/ozone/blob/f2406c4b4eb2b1295d06ec379cfe684ed9a16637/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java#L154

    OmTransport omTransport = OmTransportFactory.create(conf, ugi, omServiceId);

kuenishi

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0

kuenishi

#!/usr/bin/python
# @lint-avoid-python-3-compatibility-imports
#
# zfsslower  Trace slow ZFS operations.
#            For Linux, uses BCC, eBPF.
#
# USAGE: zfsslower [-h] [-j] [-p PID] [min_ms]
#
# This script traces common ZFS file operations: reads, writes, opens, and
# syncs. It measures the time spent in these operations, and prints details

kuenishi

Wrapper shell script:

$ cat ./riak_cs_inspector.sh
#!/bin/bash
ERL_LIBS=dev/dev1/lib dev/dev1/erts-5.9.3.1/bin/escript \
    dev/dev1/lib/riak_cs/priv/tools/internal/riak_cs_inspector.erl $*

Usage:

kuenishi

import sys

def char(i):
    '''
    0-9A-Z => sp(decimal)
    '''
    if i >= 0 and i <= 9:
        return chr(ord('0')+i)
    elif i < 36:
        return chr(ord('A')+i-10)

kuenishi

#!/usr/bin/env python3

import re
import sys

filename = sys.argv[1]
pattern = r"(https://youtu\.be/[a-zA-Z0-9_-]*)"
with open(filename, 'r') as fp:
    for line in fp.readlines():
        for match in re.findall(pattern, line):

kuenishi

#!/usr/bin/env python3
import argparse
import difflib
import json
import os
import shutil


class LBL:
    """File handler to handle text files with line-by-line comparison"""

kuenishi

Howto

1. install brasero (gnome automatically installs it)
2. install libdvdcss
3. start brasero
4. install disk
5. set region code

kuenishi

#!/bin/sh

set -eux

files=$@

cwd=`pwd`
out=`pwd`/output.mp4

echo $@