Skip to content

Instantly share code, notes, and snippets.

Robin Palotai robinp

  • Budapest, Hungary
View GitHub Profile
robinp /
Last active Apr 11, 2019
Indexing Kythe Go parts
bazel build kythe/go/languageserver/... --experimental_action_listener kythe/go/extractors/cmd/bazel:extract_kzip_go
./bazel-bin/kythe/go/indexer/cmd/go_indexer/go_indexer $(find bazel-out/k8-fastbuild/extra_actions -name '*.go.kzip') > go.ents
./bazel-bin/kythe/go/serving/tools/write_tables/write_tables --experimental_beam_pipeline --entries go.ents --out /data/kythe/beamed
# To serve it:
# Note: public_resources set to /tmp since not needed when only API is used.
./bazel-bin/kythe/go/serving/tools/http_server/http_server --serving_table /data/kythe/beamed --listen --public_resources /tmp
robinp / Streaming.hs
Created Mar 31, 2019
LevelDB Stream merging and grouping
View Streaming.hs
{- Note: originally for -}
{- Releasing under Apache 2.0 licence (, feel free to reuse accordingly. -}
{-# LANGUAGE TupleSections #-}
module Database.LevelDB.Streaming.Extended
( module X
, mergeStreams
, groupStream
robinp / github-vertical-unspace.user.css
Created Feb 25, 2019
Stylus script for to make github code review vertically more compact.
View github-vertical-unspace.user.css
body {
font-size: 12px;
.blob-num {
line-height: normal;
.blob-code {
line-height: normal;
robinp / run.bat
Created Feb 10, 2019
Converting image dirs to pdf using imagemagick on Windows
View run.bat
for /D %%f in (*) do (
cd %%f
mkdir annotated
magick mogrify -path annotated -set filename:xxx %%t -fill black -undercolor white -gravity southwest -pointsize 24 -annotate +0+0 "%%[filename:xxx]" *png *jpg
magick convert annotated/* ../%%~nf.pdf
rm -r annotated
cd ..
robinp / Bazel build error
Created Jan 17, 2019
Bazel action_env gets lost somewhere
View Bazel build error
$ sh kythe/cxx/extractor/...
SUBCOMMAND: # //kythe/cxx/extractor:index_pack [action 'Compiling kythe/cxx/extractor/ [for host]']
(cd /home/ron/.cache/bazel/_bazel_ron/084c77e61dfc73e2d0fc7fb1fb9a9252/execroot/io_kythe && \
exec env - \
PATH=/nix/store/vs6d2fjkl4kb3jb7rwibsd76k9v2n4xy-bash-4.4-p23/bin:/nix/store/lvhndwdy2q09fhwgzykjjigz7yxq5yiv-coreutils-8.30/bin:/nix/store/5c2428hk25dwsgnifcmn5dvgrwvvgvrx-findutils-4.6.0/bin:/nix/store/9jjxqhwak78zi94m7gvwdw3gqzd1hr1b-gawk-4.2.1/bin:/nix/store/f7c1ijdv5czqb0dxi4mi5wkfvi1pc7rn-gnugrep-3.3/bin:/nix/store/4lvbww6zrphadhcbqidx8qskc0rsblg6-gnused-4.7/bin:/nix/store/mwc5mwvw080nbahpdc0xr1rk3mhkcdp1-which-2.21/bin:/nix/store/c39nikyab2qfifbqzy7hqqkhvk7k05ca-unzip-6.0/bin \
PWD=/proc/self/cwd \
/nix/store/isg8rxaxkijl9x3hr2gzsf8pqfnqxg3k-gcc-wrapper-7.4.0/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/nix/store/isg8rxaxkijl9x3hr2gzsf8pqfnqxg3k-gcc-wrapper-7.4.0/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-n
View Kythe-LSIF.txt
I'm not very familiar with LSP/LSIF so far, but gave a quick read and here's a summary of LSP/LSIF vs Kythe:
- Documentation: LSP/LSIF protocol seems well documented. Kythe schema is a bit more dense, protocol needs digging around in .proto files (which are OK though).
- Generally, Kythe pipeline needs more implicit knowledge to use - some online posts might address these though.
- Windows: Kythe serving tools run on Linux, though some Docker magic might be available.
- In Kythe, the storage format and the serving protocol are more separated, while LSIF tries to maintain serialized LSP responses.
- In fact, Kythe has no standard storage format (the reference implementation uses some columnar protobufs AFAIK)
robinp /
Last active Sep 4, 2017
Extract callgraph edges with awk
gunzip -c all.nq.gz | grep 'node/kind" "package' > packages.nq
gunzip -c all.nq.gz | awk 'NR==FNR { a[$1]; next } /childof/{if ($3 in a) print $1}' packages.nq - > pkgchildren.txt
gunzip -c all.nq.gz | awk 'NR==FNR { a[$1]; next } /childof/{if ($3 in a) print $1" "$3}' pkgchildren.txt - | gzip - > topchildren.txt.gz
gunzip -c all.nq.gz | awk 'NR==FNR { a[$1]=$2; mx=FNR; next } (NR-mx==FNR) { b[$1] ; next } /kythe\/edge\/ref/{if (($1 in a) && ($3 in b)) print a[$1]" "$3}' <(gunzip -c topchildren.txt.gz) pkgchildren.txt - | uniq | gzip - > callgraph.txt.gz
robinp /
Last active Apr 2, 2018
Calling GHC API multiple times: choose your ExitFailure

GHC API lets you process Haskell sources, and (among other) specify code generation and linking level. For example:

  • (1) no codegen & no link
  • (2) bytecode generation & link in memory
  • (3) machine code generation & linking output binaries

For code analysis purposes, based on generating the typechecked AST, option (1) suffices most of the time. There are some situations in which it doesn't:

  • TemplateHaskell (TH) splice needs to execute code (at compile time) from an imported module: the imported module must be available in compiled form, so either (2) or (3) is needed. Example: in $([|$(foo)|]), foo will be evaluated at compile-time.
  • Code uses FFI imports. For this one would expect that (2) is needed (see checkCOrAsmOrLlvmOrInterp in TcForeign.hs), but actually unless it is used (say by TH, see below), even (1) works too (see the wrapper checkCg).
robinp / GhcWrapper.hs
Created Jul 13, 2017
Random GHC wrapper script.
View GhcWrapper.hs
import GHC.Paths
import System.Posix.Process
import System.Environment
import System.IO
import Data.List (intercalate)
import Control.Monad (when)
lg = appendFile "/tmp/fake.lg" . (++ "\n")
main = do
robinp /
Created Jun 30, 2017
Indexing GHC source
# GHC wrapper for indexing Haskell packages.
# Note that variables INDEXER_OUTPUT_DIR and REALGHC are set outside this script.
log() {
echo "$1" >> "$INDEXER_OUTPUT_DIR/$PKG.log"
log "GHC $*"
You can’t perform that action at this time.