Skip to content

Instantly share code, notes, and snippets.

Robin Palotai robinp

  • Budapest, Hungary
View GitHub Profile
@robinp
robinp / index-kythe.sh
Last active Apr 11, 2019
Indexing Kythe Go parts
View index-kythe.sh
bazel build kythe/go/languageserver/... --experimental_action_listener kythe/go/extractors/cmd/bazel:extract_kzip_go
./bazel-bin/kythe/go/indexer/cmd/go_indexer/go_indexer $(find bazel-out/k8-fastbuild/extra_actions -name '*.go.kzip') > go.ents
./bazel-bin/kythe/go/serving/tools/write_tables/write_tables --experimental_beam_pipeline --entries go.ents --out /data/kythe/beamed
# To serve it:
# Note: public_resources set to /tmp since not needed when only API is used.
./bazel-bin/kythe/go/serving/tools/http_server/http_server --serving_table /data/kythe/beamed --listen 0.0.0.0:8080 --public_resources /tmp
@robinp
robinp / Streaming.hs
Created Mar 31, 2019
LevelDB Stream merging and grouping
View Streaming.hs
{- Note: originally for treetide.com -}
{- Releasing under Apache 2.0 licence (https://www.apache.org/licenses/LICENSE-2.0), feel free to reuse accordingly. -}
{-# LANGUAGE TupleSections #-}
module Database.LevelDB.Streaming.Extended
( module X
, mergeStreams
, groupStream
)
where
@robinp
robinp / github-vertical-unspace.user.css
Created Feb 25, 2019
Stylus script for to make github code review vertically more compact.
View github-vertical-unspace.user.css
body {
font-size: 12px;
}
.blob-num {
line-height: normal;
}
.blob-code {
line-height: normal;
@robinp
robinp / run.bat
Created Feb 10, 2019
Converting image dirs to pdf using imagemagick on Windows
View run.bat
for /D %%f in (*) do (
cd %%f
mkdir annotated
magick mogrify -path annotated -set filename:xxx %%t -fill black -undercolor white -gravity southwest -pointsize 24 -annotate +0+0 "%%[filename:xxx]" *png *jpg
magick convert annotated/* ../%%~nf.pdf
rm -r annotated
cd ..
)
@robinp
robinp / Bazel build error
Created Jan 17, 2019
Bazel action_env gets lost somewhere
View Bazel build error
$ sh invoke-build.sh kythe/cxx/extractor/...
...
SUBCOMMAND: # //kythe/cxx/extractor:index_pack [action 'Compiling kythe/cxx/extractor/index_pack.cc [for host]']
(cd /home/ron/.cache/bazel/_bazel_ron/084c77e61dfc73e2d0fc7fb1fb9a9252/execroot/io_kythe && \
exec env - \
PATH=/nix/store/vs6d2fjkl4kb3jb7rwibsd76k9v2n4xy-bash-4.4-p23/bin:/nix/store/lvhndwdy2q09fhwgzykjjigz7yxq5yiv-coreutils-8.30/bin:/nix/store/5c2428hk25dwsgnifcmn5dvgrwvvgvrx-findutils-4.6.0/bin:/nix/store/9jjxqhwak78zi94m7gvwdw3gqzd1hr1b-gawk-4.2.1/bin:/nix/store/f7c1ijdv5czqb0dxi4mi5wkfvi1pc7rn-gnugrep-3.3/bin:/nix/store/4lvbww6zrphadhcbqidx8qskc0rsblg6-gnused-4.7/bin:/nix/store/mwc5mwvw080nbahpdc0xr1rk3mhkcdp1-which-2.21/bin:/nix/store/c39nikyab2qfifbqzy7hqqkhvk7k05ca-unzip-6.0/bin \
PWD=/proc/self/cwd \
/nix/store/isg8rxaxkijl9x3hr2gzsf8pqfnqxg3k-gcc-wrapper-7.4.0/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/nix/store/isg8rxaxkijl9x3hr2gzsf8pqfnqxg3k-gcc-wrapper-7.4.0/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-n
View Kythe-LSIF.txt
I'm not very familiar with LSP/LSIF so far, but gave a quick read and here's a summary of LSP/LSIF vs Kythe:
- Documentation: LSP/LSIF protocol seems well documented. Kythe schema is a bit more dense, protocol needs digging around in .proto files (which are OK though).
- Generally, Kythe pipeline needs more implicit knowledge to use - some online posts might address these though.
- Windows: Kythe serving tools run on Linux, though some Docker magic might be available.
- In Kythe, the storage format and the serving protocol are more separated, while LSIF tries to maintain serialized LSP responses.
- In fact, Kythe has no standard storage format (the reference implementation uses some columnar protobufs AFAIK)
@robinp
robinp / graph-bashing.sh
Last active Sep 4, 2017
Extract callgraph edges with awk
View graph-bashing.sh
gunzip -c all.nq.gz | grep 'node/kind" "package' > packages.nq
gunzip -c all.nq.gz | awk 'NR==FNR { a[$1]; next } /childof/{if ($3 in a) print $1}' packages.nq - > pkgchildren.txt
gunzip -c all.nq.gz | awk 'NR==FNR { a[$1]; next } /childof/{if ($3 in a) print $1" "$3}' pkgchildren.txt - | gzip - > topchildren.txt.gz
gunzip -c all.nq.gz | awk 'NR==FNR { a[$1]=$2; mx=FNR; next } (NR-mx==FNR) { b[$1] ; next } /kythe\/edge\/ref/{if (($1 in a) && ($3 in b)) print a[$1]" "$3}' <(gunzip -c topchildren.txt.gz) pkgchildren.txt - | uniq | gzip - > callgraph.txt.gz
@robinp
robinp / ghc-api-reuse.md
Last active Apr 2, 2018
Calling GHC API multiple times: choose your ExitFailure
View ghc-api-reuse.md

GHC API lets you process Haskell sources, and (among other) specify code generation and linking level. For example:

  • (1) no codegen & no link
  • (2) bytecode generation & link in memory
  • (3) machine code generation & linking output binaries

For code analysis purposes, based on generating the typechecked AST, option (1) suffices most of the time. There are some situations in which it doesn't:

  • TemplateHaskell (TH) splice needs to execute code (at compile time) from an imported module: the imported module must be available in compiled form, so either (2) or (3) is needed. Example: in $([|$(foo)|]), foo will be evaluated at compile-time.
  • Code uses FFI imports. For this one would expect that (2) is needed (see checkCOrAsmOrLlvmOrInterp in TcForeign.hs), but actually unless it is used (say by TH, see below), even (1) works too (see the wrapper checkCg).
@robinp
robinp / GhcWrapper.hs
Created Jul 13, 2017
Random GHC wrapper script.
View GhcWrapper.hs
import GHC.Paths
import System.Posix.Process
import System.Environment
import System.IO
import Data.List (intercalate)
import Control.Monad (when)
lg = appendFile "/tmp/fake.lg" . (++ "\n")
main = do
@robinp
robinp / ghc_wrapper.sh
Created Jun 30, 2017
Indexing GHC source
View ghc_wrapper.sh
#!/bin/bash
# GHC wrapper for indexing Haskell packages.
# Note that variables INDEXER_OUTPUT_DIR and REALGHC are set outside this script.
log() {
echo "$1" >> "$INDEXER_OUTPUT_DIR/$PKG.log"
}
log "GHC $*"
You can’t perform that action at this time.