Skip to content

Instantly share code, notes, and snippets.

View hardeepnarang10's full-sized avatar
:octocat:

Hardeep Narang hardeepnarang10

:octocat:
24 followers · 77 following
View GitHub Profile
@hardeepnarang10
hardeepnarang10 / copy.go
Created April 3, 2024 06:42 — forked from ricardobranco777/copy.go
Copy file from Docker image to standard output
package main
import (
"archive/tar"
"fmt"
"io"
"io/ioutil"
"os"
"github.com/docker/docker/api/types"
@hardeepnarang10
hardeepnarang10 / main.go
Created April 1, 2024 17:08 — forked from ayubmalik/main.go
Golang AWS SSO login from the command line. Will start a new AWS SSO session and retrieve valid access token.
package main
import (
"context"
"log"
"strings"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
@hardeepnarang10
hardeepnarang10 / llm-training.md
Created March 17, 2024 13:17 — forked from btbytes/llm-training.md
Copy of https://rentry.org/llm-training. Updated 2024-03-03

-> lmg-train.png<-

The Novice's LLM Training Guide

->Written by Alpin<- ->Inspired by /hdg/'s LoRA train rentry<- !!!warning This guide is being slowly updated. We've already moved to the axolotl trainer.

[TOC2]

@hardeepnarang10
hardeepnarang10 / TL_README.md
Created October 26, 2023 06:17 — forked from silveimar/TL_README.md

2020 - September

Why a Tech Lead Readme?

I wrote this document following my deeply believe that clarity and honesty can help build and strengthen relationships. So I aim to provide a clear understanding of how I can help you and how we can interact better.

TL;DR: I’m here to serve you, not the other way around.

My role

  • Help you to succeed and be happy as part of the team.
  • Enforce best practices in the team, technical, communications, development workflow.
  • Help the team to deliver quality technical solutions in alignment with product and business necessities.
  • Help to solve any internal or external technical blocking that may affect the team roadmap.
@hardeepnarang10
hardeepnarang10 / graceful_stop_channels.go
Created September 19, 2023 06:42
A practical graceful stop implementation in go, one illustrated with channels, and a second, arguably better with a local context. Channels implementation expects an incoming context; recommended to pass a notify context. Grace duration is independent of the incoming context.
package main
import (
"context"
"fmt"
"log"
"time"
)
func gracefulStop(ctx context.Context, graceDuration time.Duration, stop func() error) error {
@hardeepnarang10
hardeepnarang10 / AWS auto-scalable web deployment guide.md
Last active March 7, 2021 20:12
Guide for deploying an auto-scalable back-end solution for a high-availability website on AWS platform

Domain Notes:

  • Names are arbitrary values. Specified for our use case because CA performs component checks using name as identifiers.
  • Component creation is sequenced in a logical order based on dependency.
  • Launch Template (LTs) are templates and Launch Configuration (LCs) & Auto-Scaling Group (ASGs) are implementation of the said template.

Target Group:

  • Type: Instances.
  • Name: website-tg.
@hardeepnarang10
hardeepnarang10 / VNC Setup.md
Last active October 31, 2020 05:16
My minimal VNC server (host) setup script running gnome environment on cloud VM instance.
  • Create swap space. If RAM ~ 1G, 4G would be an overkill in most cases and should be replaced with 2G.
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo echo "/swapfile swap swap defaults 0 0" >> /etc/fstab
  • Update official repo & packages. Install modules for gnome-desktop. Install VNCServer.
@hardeepnarang10
hardeepnarang10 / Transcoder.md
Last active October 31, 2020 05:16
Host a Transcoder service on VM instance/cluster.

  • Mass convert large media files on cloud VM instance/cluster into smaller size. Output is generally 10X+ smaller on a 1080P or higher format and almost visually lossless. Can be further down-sized using UHARC if you have many files. Also, way cheaper if you're using cloud VMs for scraping as well.

  • Quad-core CPU-optimized instances work best in terms of resource vs cost. I've experimented with octa-core instances but the gains aren't linear and rank lower on cost vs time gains.

  • Use snap for latest ffmpeg release (possibly improved compression algorithm). If you're using ffplay for testing installed through apt, then ffmpeg command here may cause clash.

sudo snap install ffmpeg
echo "#!/bin/bash" >> Processor.sh
echo "for file in ./*.mp4; do ffmpeg -i \"\$file\" -c:v libx265 -preset fast -c:a copy \"H265/\$(basename \"\$file\" .mp4)_x265.mp4\" done" &gt;&gt; Processor.sh
@hardeepnarang10
hardeepnarang10 / Transcode Challenges.md
Last active October 31, 2020 05:17
Cluster-optimized HLS transcoder command (ffmpeg).

Transcode.sh

ffmpeg -i <input file> -hide_banner -y -c:a aac -ar 48000 -c:v h264 -profile:v baseline -crf 20 -sc_threshold 0 -pix_fmt yuv420p -hls_time 10 -strict -2 -vsync 2 -vf scale=w=XXX:h=XXX -b:v 800k -maxrate XXXk -bufsize 1200k -b:a 96k -hls_time 10 -f hls -hls_list_size 0 <output file>

Replace XXXs for scale settings to match your mediaplayer. Preferable to use an autoscaler on upper-medium settings first and let it process heavier resolutions in background. Maxrate of 1000k-1200k should work fine in most cases - keeping note that bufsize is set to 1200k.

Challenges with Transcoders

Can get stuck every now and then. Use health monitor to pass the following checks and have Kubernetes restart container:

@hardeepnarang10
hardeepnarang10 / config.fish
Created October 23, 2020 05:18
alias cls="clear"
alias python="python3"
alias py="python3"
alias pip="pip3"
for file in /etc/profile.d/*.sh
bass source $file
end