Skip to content

Instantly share code, notes, and snippets.

View portante's full-sized avatar

Peter Portante portante

86 followers · 33 following
  • Red Hat, Inc.
View GitHub Profile
@portante
portante / PUT_sequence.txt
Last active December 13, 2015 19:58
PUT time budget.
[pid 30268] 01:28:38.185609 accept(8, {sa_family=AF_INET, sin_port=htons(42975), sin_addr=inet_addr("127.0.0.1")}, [16]) = 14
[pid 30268] 01:28:38.185682 fcntl(14, F_GETFL) = 0x2 (flags O_RDWR)
[pid 30268] 01:28:38.185718 fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0
= 109us
# Discussion: fcntl() calls are overhead cost of eventlet
[pid 30268] 01:28:38.185763 fcntl(14, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 30268] 01:28:38.185808 fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 30268] 01:28:38.185897 fcntl(14, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 30268] 01:28:38.185923 fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0
@portante
portante / gist:4998794
Created February 20, 2013 19:52
Thread per Directory
Finished - Total( 3211328599 bytes 10323 files 24.3s 126.24 MB/s 425.58 Files/s) - Interval ( 3211328599 bytes 10323 files 24.3s 126.24 MB/s 425.58 Files/s)
Finished - Total( 3992640781 bytes 20646 files 38.7s 98.34 MB/s 533.29 Files/s) - Interval ( 781312182 bytes 10323 files 14.5s 51.53 MB/s 713.98 Files/s)
Finished - Total( 4516426858 bytes 30969 files 51.3s 83.99 MB/s 603.89 Files/s) - Interval ( 523786077 bytes 10323 files 12.6s 39.70 MB/s 821.37 Files/s)
Finished - Total( 5029685030 bytes 41292 files 64.7s 74.09 MB/s 637.90 Files/s) - Interval ( 513258172 bytes 10323 files 13.4s 36.36 MB/s 767.58 Files/s)
Finished - Total( 5445423223 bytes 51615 files 79.2s 65.60 MB/s 652.03 Files/s) - Interval ( 415738193 bytes 10323 files 14.4s 27.44 MB/s 715.44 Files/s)
Finished - Total( 5935228269 bytes 61938 files 91.7s 61.70 MB/s 675.16 Files/s) - Interval ( 489805046 bytes 10323
@portante
portante / annotated_strace_proxy_object_glusterfs.txt
Last active December 14, 2015 04:39
strace of proxy-server, object-server, glusterfs with posix translator only.
0.000115 [proxy -server] 1361794636.177603 accept(4, {sa_family=AF_INET, sin_port=htons(56708), sin_addr=inet_addr("172.17.10.19")}, [16]) = 11
0.000058 [proxy -server] 1361794636.177894 fcntl(11, F_GETFL) = 0x2 (flags O_RDWR)
0.000028 [proxy -server] 1361794636.178021 fcntl(11, F_SETFL, O_RDWR|O_NONBLOCK) = 0
0.000172 [proxy -server] 1361794636.178319 recvfrom(11, "PUT /v1/AUTH_del0/gprfc019.pw32."..., 8192, 0, NULL, NULL) = 309
0.000013 [proxy -server] 1361794636.178717 getsockname(11, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("172.17.40.13")}, [16]) = 0
0.000014 [proxy -server] 1361794636.179056 read(12, "|R\242x\335\250\227\36P\352\221/\25\207\276\34", 16) = 16
0.000008 [proxy -server] 1361794636.179097 gettid() = 10464
0.000108 [proxy -server] 1361794636.179306 stat("/etc/swift/container.ring.gz", {st_mode=S_IFREG|0644, st_size=741, ...}) = 0
0.000068 [proxy -server] 1361794636.179553 sendto(13, "get 18f487541b6b4e33f36f1165f5ef"..., 38, 0, NULL, 0) = 38
0.000018 [proxy -ser
@portante
portante / inode_star.diffs
Last active December 15, 2015 09:49
gluster internal inode and dirent hashtable increases
diff -rcw boo/glusterfs-3.3.0catalyst3/libglusterfs/src/inode.c foo/glusterfs-3.3.0catalyst3p/libglusterfs/src/inode.c
*** boo/glusterfs-3.3.0catalyst3/libglusterfs/src/inode.c 2012-10-16 13:32:42.000000000 -0400
--- foo/glusterfs-3.3.0catalyst3p/libglusterfs/src/inode.c 2013-03-25 17:53:54.172430688 -0400
***************
*** 49,54 ****
--- 49,56 ----
void
fd_dump (struct list_head *head, char *prefix);
+ #define NAME_HASHSIZE 999983 /* Was 14057 */
@portante
portante / DiskFile Class Usage
Last active December 16, 2015 19:50
OpenStack Swift DiskFile class API and usage summary comparison for DiskFile Refactoring. This is intended to be viewed "raw" so that one can see the three columns allowing for a visual comparison of the behaviors. Like methods are lined up to show as much parallelism in behavior as possible.
DiskFile Class API Comparison
=============================
(as of master bfcf72a Merge "Remove keep_data_fp argument from (https://review.openstack.org/35381, Patch Set 16) (https://review.openstack.org/43973, Patch Set 1)
DiskFile constructor")
---------------------------------------------------------------+----------------------------------------------------------------+----------------------------------------------------------------
*DiskFileManager(conf)
Methods:
.pickle_async_update()
@portante
portante / TableOfSwiftObjServerAPIMarshalling.txt
Last active December 17, 2015 08:18
Table of actions taken in Swift swift/obj/server.py module for the ObjectController for all HTTP methods. Each action is representative of the steps taken to validate a request up to the point where a DiskFile object is instantiated.
Operations | POST | PUT | GET | HEAD | DELETE | REPLICATE
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
split_path | split_path | split_path | split_path | split_path | split_path
validate_device_partition | validate_device_partition | validate_device_partition | validate_device_partition | validate_device_partition | validate_device_partition
X-timestamp check | X-timestamp check | | | X-timestamp check |
| check_object_creation | | | |
Delete-at check
@portante
portante / TableOfSwiftObjServerParameterUsage.txt
Last active December 17, 2015 09:18
Table of Swift Object Server parameter usage for device, partition, account, container and obj values. It does not include the device/partition validation code.
Parameter
Operations | POST | PUT | GET | HEAD | DELETE | REPLICATE
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
device | mount_check | mount_check | mount_check | mount_check | mount_check | mount_check
| DiskFile | DiskFile | DiskFile | DiskFile | DiskFile |
| | Log-transfer-rate | | | |
| delete_at_update | delete_at_update | | | delete_at_update |
| | contain
@portante
portante / b.py
Created September 19, 2013 13:21
Run on a ThinkPad T520. See the "lscpu" output below for hardware details. The first result was run without "numctl -C 0", the second with. The results seem to show that when threads start yielding between them, things slow down tremendously. This is based on Sam Merritt's original results, found at: https://gist.github.com/smerritt/6619709
#!/usr/bin/env python
import benchmark
import Queue
import threading
import os
from time import sleep
class BenchQueueStuff(benchmark.Benchmark):
@portante
portante / 1cpu.txt
Created September 19, 2013 20:03
Python threading comparison with listdir. Refactored to pull out the threading stuff from the measurements. This work also tries to show when one blocks between put/get it can really slow things down, and it gets worse across multiple CPUs. Run on a ThinkPad T520
$ numactl -C 0 python b.py
Benchmark Report
================
BenchQueueStuff
---------------
name | rank | runs | mean | sd | timesBaseline
-----------------------------|------|------|---------|----------|--------------
@portante
portante / proxy_normalize_timestamp_callers.txt
Created September 24, 2013 04:42
Callers of normalize_timestamp() in the proxy server. For each module, each method listed indented 4 spaces are direct callers of normalize_timestamp(). Each subsequently indented method indicates the callers of that method. Those lines beginning with "*" indicate additional ways that method uses it.
swift/proxy/controllers/base.py
generate_request_headers() -- time.time()
GETorHEAD_base()
autocreate_account() -- time.time()
swift/proxy/controllers/account.py
generate_request_headers() -- time.time()
PUT()
POST()
DELETE()