Skip to content

Instantly share code, notes, and snippets.

@rain-1
rain-1 / llama-home.md
Last active February 22, 2024 05:52
How to run Llama 13B with a 6GB graphics card

This worked on 14/May/23. The instructions will probably require updating in the future.

llama is a text prediction model similar to GPT-2, and the version of GPT-3 that has not been fine tuned yet. It is also possible to run fine tuned versions (like alpaca or vicuna with this. I think. Those versions are more focused on answering questions)

Note: I have been told that this does not support multiple GPUs. It can only use a single GPU.

It is possible to run LLama 13B with a 6GB graphics card now! (e.g. a RTX 2060). Thanks to the amazing work involved in llama.cpp. The latest change is CUDA/cuBLAS which allows you pick an arbitrary number of the transformer layers to be run on the GPU. This is perfect for low VRAM.

  • Clone llama.cpp from git, I am on commit 08737ef720f0510c7ec2aa84d7f70c691073c35d.
using OpenTelemetry;
using OpenTelemetry.Logs;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
var appResourceBuilder = ResourceBuilder.CreateDefault()
.AddService(serviceName: Telemetry.ServiceName, serviceVersion: Telemetry.ServiceVersion);
#!/usr/bin/env bash
# on ubuntu 18.04
cat << EOF >/etc/apt/source.list
deb http://archive.ubuntu.com/ubuntu bionic main multiverse restricted universe
deb http://archive.ubuntu.com/ubuntu bionic-security main multiverse restricted universe
deb http://archive.ubuntu.com/ubuntu bionic-updates main multiverse restricted universe
EOF
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
cat << EOF > /etc/apt/sources.list.d/clang.list
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\SOFTWARE\Microsoft\PCHC]
"UpgradeEligibility"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\Setup\LabConfig]
"BypassTPMCheck"=dword:00000001
"BypassSecureBootCheck"=dword:00000001
"BypassRAMCheck"=dword:00000001
"BypassStorageCheck"=dword:00000001
@davidfowl
davidfowl / .NET6Migration.md
Last active April 11, 2024 02:02
.NET 6 ASP.NET Core Migration
@ejlp12
ejlp12 / eks_best_practice.md
Last active February 8, 2024 18:42
EKS Best Practice

Please check https://aws.github.io/aws-eks-best-practices/ for more comprehensive EKS best practice!

Architecture

  • Think about multi-tenancy, isolation for different environment or different workload
    • Isolation at account level using AWS organization
    • Isolation at the network layer ie. different VPC & different cluster
    • Use different Nodes Group (Node pool) for different purpose/category e.g. create dedicated node groups for Operational tools such as CI/CD tool, Monitoring tool, Centralize logging system.
    • Separate namespace for different workload

Reliability | Principles

@yellow1912
yellow1912 / nginx.toml
Last active March 26, 2024 16:34
Sample Vector configuration for nginx
# be careful, you will gets tons of logs
# you may want to skip access logs or use sampling only here
[sources.nginx_access_logs]
type = "file"
include = ["/var/log/sites/*/*/*/*/nginx/access.log"] # supports globbing
ignore_older = 86400 # 1 day
# I'm configuring my nginx to use json logs, if you use another logging format you need to change this
[transforms.nginx_access_logs_json]
type = "json_parser"
@ArturKlauser
ArturKlauser / check-qemu-binfmt.sh
Last active February 13, 2023 16:53
Building Multi-Architecture Docker Images With Buildx
#!/bin/bash
# (c) 2020 Artur.Klauser@computer.org
# SPDX-License-Identifier: Apache-2.0 OR MIT
#
# This script checks if all software requirements are met in a Linux environment
# in order to use 'docker buildx' to build multi-architecture images.
# For more information see:
# https://nexus.eddiesinentropy.net/2020/01/12/Building-Multi-architecture-Docker-Images-With-Buildx/
function error() {
@MichaelSimons
MichaelSimons / RetrievingDockerImageSizes.md
Last active March 25, 2024 11:14
Retrieving Docker Image Sizes

Retrieving Docker Image Sizes

There are two metrics that are important to consider when discussing the size of Docker images.

  1. Compressed size - This is often referred to as the wire size. This affects how fast/slow images can be pulled from a registry. This impacts the first run experience on machines where images are not cached.
  2. Uncompressed size - This is often referred to as the size on disk. This affects how much local storage is required to support your Docker workloads.

The example commands shown below will work on Windows, MacOS, and Linux.

How to Measure the Compressed Size