djmitche

Requirements

A reliable, secure system for downloading large artifacts used during the build process.

Automated build processes and developers (with credentials, if required) download sets of related files using a "manifest" that specifies the sha512 digest of each required file along with the destination filename. Files are cached locally, and only downloaded if not present locally. Digests are verified on every run. Last-accessed times are kept for all files.

Each file has a visibility level -- public or internal. public files are available for anyone to download. internal files require identification as an employee or an internall automated build process. The name "internal" is chosen to correspond to the data-security level of this data.

Authorized users can upload new files in batches tagged with a username and a message (similar to a commit message). Each file in such a batch is tagged with its visiblity level. Storage is deduplicated, so an upload of a batch with a file that alrea

djmitche

4:03 IP laptop.60061 > trac.http: Flags [S], seq 833594151, win 29200, options [mss 1460,sackOK,TS val 17271196 ecr 0,nop,wscale 7], length 0
4:03 IP trac.http > laptop.60061: Flags [S.], seq 1018352583, ack 833594152, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 4013412119 ecr 17271196], length 0
4:03 IP laptop.60061 > trac.http: Flags [.], ack 1, win 229, options [nop,nop,TS val 17271280 ecr 4013412119], length 0
4:03 IP laptop.60061 > trac.http: Flags [.], seq 1:1449, ack 1, win 229, options [nop,nop,TS val 17271280 ecr 4013412119], length 1448
4:03 IP laptop.60061 > trac.http: Flags [.], seq 1449:2897, ack 1, win 229, options [nop,nop,TS val 17271280 ecr 4013412119], length 1448
4:03 IP laptop.60061 > trac.http: Flags [P.], seq 2897:3681, ack 1, win 229, options [nop,nop,TS val 17271282 ecr 4013412119], length 784
4:04 IP trac.http > laptop.60061: Flags [.], ack 2897, win 1018, options [nop,nop,TS val 4013412205 ecr 17271280], length 0
4:04 IP trac.http > laptop.60061: Flags [.], ack 3681, win 

djmitche

Goal

"Drive Android builds out of taskcluster" .. meaning to run builds of Firefox for Android (we don't build Android) within taskcluster.

The idea is to get something non-b2g from releng running in taskcluster, both as a learning opportunity and as a proof of concept.

Task List

• build a docker image for building Android using the tools in alder
• build Android by hand / shell script in that docker image
• do a basic Android build in a simple, docker-compatible way
• handle tooltool cache

djmitche

// $FreeBSD: head/dns/bind910/files/named.conf.in 369042 2014-09-23 11:22:40Z mat $
//
// Refer to the named.conf(5) and named(8) man pages, and the documentation
// in /usr/local/share/doc/bind for more details.
//
// The named.conf as it came with installation of BIND 9.10 is split so each
// part could be easily read and understood:
// * named.conf.options -- all options
// * named.conf.rfcs -- all RFC related zones (these are more or less static)
// * named.conf.zones -- all zone definitions (except for RFC related ones)

djmitche

Problem Summary

Here's what is currently running on the releng web cluster, and needs to be migrated:

• https://api.pub.build.mozilla.org (RelengAPI)
  • DBs (local and remote)
  • Worker cluster
  • Badpenny system of some sort
• http://builddata.pub.build.mozilla.org
• Static data

djmitche

Not clear where to send syslog data, and anyway we'd probably want to filter somewhat first.

Installed Heka directly from https://github.com/mozilla-services/heka/releases/download/v0.8.0/heka-0_8_0-linux-amd64.rpm on hp1

Heka can't do syslog-over-UDP at all, so it has to be fronted by something which dumps to disk

Overall plan:
 - get rsyslog to log something reasonable to disk - RSYSLOG_FileFormat seems close
   - maybe add fromhost?
   - maybe add severity, facility

djmitche

[root@relabs-puppet2.relabs.releng.scl3.mozilla.com custom]# git init
Initialized empty Git repository in /tmp/git-annex-stuff/custom/.git/

# use fqdn for the name
[root@relabs-puppet2.relabs.releng.scl3.mozilla.com custom]# git annex init $(facter fqdn)
init relabs-puppet2.relabs.releng.scl3.mozilla.com ok
(Recording state in git...)

# then added and committed existing files
git annex add .

djmitche

Ubuntu:
 choose java
  openjdk?

 updating repos frequently - monthly?
  sec updates, testing against newest versions
  including flash

Need to update flash, java frequently

djmitche

fw1_releng_scl3:
    type: srx
    output: /opt/fwunit/releng/data/fw1_releng_scl3.json
    firewall: ...
    ssh_username: ...
    ssh_password: ...
    application-map:
        # drop the superfluous 'junos-' prefix, but otherwise leave unchanged:
        junos-ssh: ssh
        junos-http: http

djmitche

{
  "url": "https://api.github.com/repos/buildbot/buildbot/hooks/25722",
  "test_url": "https://api.github.com/repos/buildbot/buildbot/hooks/25722/test",
  "id": 25722,
  "name": "irc",
  "active": true,
  "events": [
    "push",
    "pull_request",
    "pull_request_review_comment",