tuxdna

Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law. https://en.wikipedia.org/wiki/Hofstadter's_law

f(x) = g(x) + delta + lambda*f(x)

where 

tuxdna

# find closest pair of points in 2d space in O(nlogn)

import sys
import math


def euclidean_distance(p1, p2):
    x1, y1 = p1
    x2, y2 = p2
    return math.sqrt((x2-x1)**2 + (y2 - y1)**2)

tuxdna

Estimating hard to invert functions

Works for

• n log(n)
• n! ( Factorial function )

$ python function_estimation.py 

tuxdna

How to run?

Create config.py from config.py.template and fill in your AUTH token, and the target URL.

Then invoke the grpah sync:

$ python3 perform_graph_sync.py package input-file.json maven

Invoking url: http://.../api/v1/graphsync/sync_all/maven/HTTPClient%3AHTTPClient

tuxdna

Create a disk file with random data

$ truncate -s 50G encrypted_backup.disk

Setup encryption for that device file

$ cryptsetup --verbose --verify-passphrase -s 512 luksFormat encrypted_backup.disk

tuxdna

{
    "requestId": "d3c056d7-428e-42f8-ba43-cbc0e1413970",
    "status": {
        "message": "",
        "code": 200,
        "attributes": {}
    },
    "result": {
        "data": [
            {

tuxdna

$ find . -name requirements.txt -exec ls "{}" \; 
./integration-tests/data/requirements.txt
./integration-tests/requirements.txt
./dashboard/requirements.txt
./fabric8-analytics-firehose-fetcher/requirements.txt
./fabric8-analytics-firehose-fetcher/integration_tests/requirements.txt
./fabric8-analytics-license-analysis/requirements.txt
./fabric8-analytics-tagger/requirements.txt
./perf-tests/requirements.txt

tuxdna

http://localhost:32000/api/v1/component-analyses/go/github.com%2fgorilla%2fmux/master

data-model-importer_1   | --------------------------------------------------------------------------------
data-model-importer_1   | INFO in rest_api [/src/rest_api.py:80]:
data-model-importer_1   | Selective Ingestion with payload - {"select_ingest": ["libraries_io", "git_stats"], "package_list": [{"ecosystem": "go", "version": null, "name": "github.com/gorilla/context"}]}
data-model-importer_1   | --------------------------------------------------------------------------------
data-model-importer_1   | --------------------------------------------------------------------------------
data-model-importer_1   | INFO in rest_api [/src/rest_api.py:88]:

tuxdna

Document describing what kind of dependencies go list reports, and whether or not it satisfies the requirements of ingestion.

An alternative approach ( as suggested on mailing list) could be to use go list command line tool to find all the dependencies of a Go project.

For example:

$ go get github.com/mattermost/platform
$ go list -f="{{.ImportPath}} {{.Imports}}" github.com/mattermost/platform | python2 -c 'import sys; import re; import json; print json.dumps({x[0]: x[1] for x in map(lambda s: (re.match("(\S+)\s\[(.*?)\]", s).groups()[0], re.match("(\S+)\s\[(.*?)\]", s).groups()[1].split()), sys.stdin.readlines())})'  | python -mjson.tool

tuxdna

import requests
import functools
# data = json.load(open("top-go-packages.json"))
r = requests.get("http://go-search.org/api?action=tops&len=100")
data = r.json()
kinds = [d for d in data if d["Name"] != "Sites"]
concatenated = functools.reduce(lambda x, y: x + y["Items"], kinds, [])
top_list = sorted(set([(x["Name"], x["Package"]) for x in concatenated]))

for i,d in enumerate(top_list):