jrudolph

Building (parts of) yugabyte locally

Yugabyte is best built inside of a docker container for the correct runtime environment mimicking the one the actual release is built with itself.

I found a suitable build environment by looking at the yb's CI scripts: https://github.com/yugabyte/yugabyte-db/blob/master/.github/workflows/build.yml.

Steps

Check out the source code.

jrudolph

Volume Cartographer PPM file format

PPM files map U/V coordinates from flattened surfaces back to the original 3D volume coordinates x/y/z and also provide a normal for every point.

The file has a small header and is otherwise a huge array of double values (in the common case, but see header).

Header

jrudolph

# use persofoto.de to crop a foto to the right dimensions

# then use this script to lay it out on a 1600x2312 jpg to print as 9x13
montage mein-passfoto.jpg mein-passfoto.jpg mein-passfoto.jpg mein-passfoto.jpg -geometry 622x898+89+129 out.jpg

jrudolph

Freiburg im Breisgau

The town of Freiburg is located in south-western Germany, in the state of Baden-Württemberg, 150 km (93 mi) south-west of Karlsruhe, 110 km (68 mi) west of Strasbourg, France, and 130 km (81 mi) east of the Alpine passes of the Alps.

Geography

Freiburg lies in the extreme south of Baden-Württemberg and is bordered on the west by France. It lies 500 m (1,600 ft) above sea level, within the southern part of the northern Black Forest at the foothills of the Alps, near the confluence of the rivers Singold and Dreisam, in the historical region of Sundgau, locally referred to as Breisgau. Freiburg is in the traditional wine-growing region of Germany and hosts one of the country's largest wine festivals, the Freiburger Weihnachtsmarkt, which takes place from the last weekend in November until the first weekend in December.

Subdivisions

jrudolph

Model from https://huggingface.co/TheBloke/NewHope-GGML

llama2JVM ### Instruction:
llama2JVM Generate scala code that find the the top-p (for nucleus sampling) elements, i.e. the smallest set of elements that reaches a given level `p` of cumulative probability, of an unsorted array of probabilities summing to 1. Expect a power-law distribution, i.e. the number of elements found will be low (< 10). Do not use any data structures besides arrays (in particular, expect that inserting into apriority queue will be too slow). Avoid sorting.
llama2JVM
llama2JVM ### Response:
llama2JVM Here's a possible implementation in Scala that uses binary search and a priority queue to solve this problem.
llama2JVM
llama2JVM ```scala
llama2JVM import scala.annotation.tailrec

jrudolph

package akka.http.scaladsl

import java.io.File

import akka.http.scaladsl.unmarshalling.Unmarshal
import akka.util.ByteString

import scala.concurrent.duration._

import akka.actor.ActorSystem

jrudolph

Yugabyte Debugging Case Study

Problem

Under load, the yugabyte clusters showed a latency spike ~ every 30 seconds. Latencies went up to 4 seconds.

Analysis

Aside from the main problem, we found that one of the servers in the cluster showed CPU saturation during those spikes. Since the spikes were frequent enough, top could be used to identify the yb-master process as the

jrudolph

import akka.actor.ActorSystem
import akka.stream.{ Attributes, FlowShape, Inlet, Outlet, OverflowStrategy }
import akka.stream.scaladsl.{ Compression, FileIO, JsonFraming }
import akka.stream.stage.{ GraphStage, GraphStageLogic, InHandler, OutHandler }
import akka.util.ByteString

import java.io.{ File, FileOutputStream }
import spray.json._

import scala.concurrent.Future

jrudolph

As of version 2.12.1.0

https://docs.yugabyte.com/preview/architecture/ has some information but too little about the actual implementation details.

The whole project is written in C++.

Components

• master(s): provide metadata services and are involved in all kinds of metadata changes, manage the set of tservers, disitrbute tablets to tservers. There's a cluster of masters and RAFT is used to determine a leader. It seems that,

jrudolph

import spray.json._

object TestJsonClassHierarchyFormat extends App {
  sealed trait Animal extends Product
  case class Dog(name: String, age: Int) extends Animal
  case class Cat(name: String, catchesBirds: Boolean)extends Animal

  import DefaultJsonProtocol._

  implicit val dogFormat = jsonFormat2(Dog)