Skip to content

Instantly share code, notes, and snippets.

View steveloughran's full-sized avatar
🌴
offline until 2025

Steve Loughran steveloughran

🌴
offline until 2025
465 followers · 2 following
View GitHub Profile
@steveloughran
steveloughran / gist:c8b39a7b87a9bd63d7a383bda8687e7e
Created August 23, 2016 15:20
log of a streaming against s3 test
2016-08-23 16:17:51,877 INFO server.Server (Server.java:doStart(379)) - Started @4407ms
2016-08-23 16:17:51,878 INFO util.Utils (Logging.scala:logInfo(54)) - Successfully started service 'SparkUI' on port 4040.
2016-08-23 16:17:51,881 INFO ui.SparkUI (Logging.scala:logInfo(54)) - Bound SparkUI to 0.0.0.0, and started at http://192.168.1.7:4040
2016-08-23 16:17:51,953 INFO executor.Executor (Logging.scala:logInfo(54)) - Starting executor ID driver on host localhost
2016-08-23 16:17:51,984 INFO util.Utils (Logging.scala:logInfo(54)) - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 61294.
2016-08-23 16:17:51,985 INFO netty.NettyBlockTransferService (Logging.scala:logInfo(54)) - Server created on 192.168.1.7:61294
2016-08-23 16:17:51,986 INFO storage.BlockManagerMaster (Logging.scala:logInfo(54)) - Registering BlockManager BlockManagerId(driver, 192.168.1.7, 61294)
2016-08-23 16:17:51,991 INFO storage.BlockManagerMasterEndpoint (Logging.scala:logInfo(54)) -
@steveloughran
steveloughran / testlog.txt
Created June 27, 2017 15:07
- listFiles
2017-06-27 15:47:02,289 [ScalaTest-main-running-S3ACSVReadSuite] INFO s3.S3ACSVReadSuite (Logging.scala:logInfo(54)) - ReadBytesReturned
Read in blocks and assess their size and duration.
This is to identify buffering quirks.
-------------------------------------------
2017-06-27 15:47:03,003 [ScalaTest-main-running-S3ACSVReadSuite] INFO s3a.S3AFileSystem (S3AFileSystem.java:initialize(276)) - Magic committer is enabled
2017-06-27 15:47:05,121 [ScalaTest-main-running-S3ACSVReadSuite] WARN internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-06-27 15:47:06,730 [ScalaTest-main-running-S3ACSVReadSuite] WARN internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream, aborting
@steveloughran
steveloughran / faslane-synchrotron.gpx
Created January 29, 2018 03:47
Faslane Synchrotron raw GPX trace
<?xml version="1.0" ?>
<gpx xmlns="http://www.topografix.com/GPX/1/1">
<trk>
<name>
faslane synchrotron
</name>
<trkseg>
<trkpt lat="56.0666066942" lon="-4.8184543848"/>
<trkpt lat="56.0664389978" lon="-4.8188728094"/>
<trkpt lat="56.0664389978" lon="-4.8188728094"/>
@steveloughran
steveloughran / faslane-synchrotron-fake-timings
Created January 29, 2018 03:49
<?xml version="1.0" ?>
<gpx xmlns="http://www.topografix.com/GPX/1/1">
<trk>
<name>
faslane synchrotron
</name>
<trkseg>
<trkpt lat="56.0666066942" lon="-4.8184543848">
<time>2018-01-28T07:46:58Z</time>
</trkpt>
@steveloughran
steveloughran / banyoles.gpx
Last active January 30, 2018 06:24
Full GPX trace of a ride around Banyoles, Catalunia, [Spain].
<?xml version="1.0" encoding="UTF-8"?>
<gpx creator="StravaGPX" version="1.1" xmlns="http://www.topografix.com/GPX/1/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.topografix.com/GPX/1/1 http://www.topografix.com/GPX/1/1/gpx.xsd http://www.garmin.com/xmlschemas/GpxExtensions/v3 http://www.garmin.com/xmlschemas/GpxExtensionsv3.xsd http://www.garmin.com/xmlschemas/TrackPointExtension/v1 http://www.garmin.com/xmlschemas/TrackPointExtensionv1.xsd" xmlns:gpxtpx="http://www.garmin.com/xmlschemas/TrackPointExtension/v1" xmlns:gpxx="http://www.garmin.com/xmlschemas/GpxExtensions/v3">
<metadata>
<time>2017-07-21T14:12:25Z</time>
</metadata>
<trk>
<name>Farm Trails to town and a cafe largo</name>
<trkseg>
<trkpt lat="42.1775050" lon="2.7566040">
<ele>183.0</ele>
@steveloughran
steveloughran / pannisac.gpx
Created January 30, 2018 06:40
Pannisac GPX trace
<?xml version="1.0" encoding="UTF-8"?>
<gpx creator="StravaGPX iPhone" version="1.1" xmlns="http://www.topografix.com/GPX/1/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.topografix.com/GPX/1/1 http://www.topografix.com/GPX/1/1/gpx.xsd">
<metadata>
<time>2017-07-17T17:42:03Z</time>
</metadata>
<trk>
<name>Panissac</name>
<trkseg>
<trkpt lat="46.0870610" lon="1.0791790">
<ele>275.9</ele>
@steveloughran
steveloughran / terasort.java
Created January 22, 2019 11:01
@Test
public void test_140_teracomplete() throws Throwable {
terasortDuration.get().close();
final StringBuilder results = new StringBuilder();
results.append("\"Operation\"\t\"Duration\"\n");
// this is how you dynamically create a function in a method
// for use afterwards.
// Works because there's no IOEs being raised in this sequence.
@steveloughran
steveloughran / ITestS3AFileOperationCost.java
Last active December 16, 2020 18:20
S3A IOStatistics Dump from an ITestS3AOperationCost run (remote)
Aggregate FileSystem Statistics
counters=((action_executor_acquired=68)
(action_executor_acquired.failures=0)
(action_http_get_request=4)
(action_http_get_request.failures=0)
(action_http_head_request=284)
(action_http_head_request.failures=0)
(committer_bytes_committed=0)
(committer_bytes_uploaded=0)
(committer_commit_job=0)
@steveloughran
steveloughran / AWS S3 Log.txt
Created May 17, 2021 13:27
AWS S3 log with S3A auditing enabled during terasort test; note job ID propagation...this is only in store actions during task/job commit, not worker task R/W
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [13/May/2021:13:17:51 +0000] 109.157.171.170 arn:aws:iam::152813717728:user/stevel-dev VFFQ3CDMPHBBB42W REST.GET.BUCKET - "GET /?list-type=2&delimiter=%2F&max-keys=2&prefix=terasort-magic%2Fvalidate%2F__magic%2Fjob-job_1620911577786_0006%2Ftasks%2Fattempt_1620911577786_0006_m_000000_0%2F__base%2F&fetch-owner=false HTTP/1.1" 200 - 375 - 28 28 "https://audit.example.org/op_mkdirs/fc311a70-b166-423a-9345-ccd17f38ff5c-00000006/?op=op_mkdirs&p1=terasort-magic/validate/__magic/job-job_1620911577786_0006/tasks/attempt_1620911577786_0006_m_000000_0/__base&pr=stevel&ps=c765f30e-9f74-488b-b536-dc1714a90245&id=fc311a70-b166-423a-9345-ccd17f38ff5c-00000006&t0=1&fs=fc311a70-b166-423a-9345-ccd17f38ff5c&t1=1&ji=job_1620911577786_0006&ts=1620911871265" "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.16 OpenJDK_64-Bit_Server_VM/25.282-b08 java/1.8.0_282 vendor/AdoptOpenJDK" - 0X+S2Iv0LI8xRYuM11e3aiPEIFkg2t2d4xLMZzLAf+p4gJGDifQGItg4qvPZLgG
@steveloughran
steveloughran / gist:664d30cef20f605f3164ad01f92a458a
Created June 3, 2021 19:12
cloudstore dux
~/P/R/hadoop-3.3.1 time bin/hadoop jar $CLOUDSTORE dux -threads 64 -limit 1000 -verbose $BUCKET
Listing files under s3a://stevel-london/ with thread count 64
=============================================================
2021-06-03 20:10:37,994 [main] INFO commands.ExtendedDu (StoreDurationInfo.java:<init>(53)) - Starting: List files under s3a://stevel-london/
2021-06-03 20:10:39,932 [main] INFO commands.ExtendedDu (StoreDurationInfo.java:<init>(53)) - Starting: Initial list of path s3a://stevel-london/
2021-06-03 20:10:40,288 [main] INFO commands.ExtendedDu (StoreDurationInfo.java:close(100)) - Initial list of path s3a://stevel-london/: duration 0:00:356
2021-06-03 20:10:40,290 [pool-3-thread-1] INFO commands.ExtendedDu (StoreDurationInfo.java:<init>(53)) - Starting: List s3a://stevel-london/cloud-integration