Skip to content

Instantly share code, notes, and snippets.

View wolfeidau's full-sized avatar
🐺
Building data science projects

Mark Wolfe wolfeidau

🐺
Building data science projects
View GitHub Profile
@lewtun
lewtun / sft_trainer.py
Last active April 12, 2024 14:51
Fine-tuning Mistral 7B with TRL & DeepSpeed ZeRO-3
# This is a modified version of TRL's `SFTTrainer` example (https://github.com/huggingface/trl/blob/main/examples/scripts/sft_trainer.py),
# adapted to run with DeepSpeed ZeRO-3 and Mistral-7B-V1.0. The settings below were run on 1 node of 8 x A100 (80GB) GPUs.
#
# Usage:
# - Install the latest transformers & accelerate versions: `pip install -U transformers accelerate`
# - Install deepspeed: `pip install deepspeed==0.9.5`
# - Install TRL from main: pip install git+https://github.com/huggingface/trl.git
# - Clone the repo: git clone github.com/huggingface/trl.git
# - Copy this Gist into trl/examples/scripts
# - Run from root of trl repo with: accelerate launch --config_file=examples/accelerate_configs/deepspeed_zero3.yaml --gradient_accumulation_steps 8 examples/scripts/sft_trainer.py
@dongri
dongri / uninstall-netskope.sh
Last active February 16, 2024 15:23
uninstall netskope
#!/bin/sh
sudo ps aux | grep Netskope | grep -v grep | awk '{ print "kill -9", $2 }' | sudo sh
echo '[✓] Kill Netskope Process'
sudo rm -rf /Applications/Remove\ Netskope\ Client.app
echo '[✓] Removed Remove Netskope Client.app'
sudo rm -rf /Library/Application\ Support/Netskope
echo '[✓] Removed Agent of Netskope Client.app'
@andif888
andif888 / playbook.yml
Created April 13, 2019 04:17
Ansible Playbook for Ubuntu Desktop minimal
---
- hosts: localhost
tasks:
- name: Add Apt-Keys
apt_key:
url: '{{item.name}}'
state: present
with_items:
@stevenringo
stevenringo / reinvent-2017-youtube.md
Created December 3, 2017 23:01
Links to YouTube recordings of AWS re:Invent 2017 sessions

| Title | Description

@kartiksura
kartiksura / main.go
Last active June 7, 2023 16:38
Querying AWS Athena using Golang SDK
package main
import (
"fmt"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/athena"
)
@sapessi
sapessi / README.md
Last active September 17, 2021 21:32
continuous deployment of Golang Gin application in AWS Lambda and Amazon API Gateway with CodePipeline/CodeBuild

You can use CodePipeline and CodeBuild to create a continuous deployment/integration pipeline for serverless applications build on AWS Lambda and Amazon API Gateway. This sample application is written in Go with the Gin framework and uses the eawsy API Gateway proxy shim: https://github.com/eawsy/aws-lambda-go-net

We initially detailed our methodology in this blog post: https://aws.amazon.com/blogs/compute/continuous-deployment-for-serverless-applications/

We have used the shim technology created by eawsy to run Golang applications inside AWS Lambda (https://github.com/eawsy/aws-lambda-go-shim) and created a container that can be used with CodeBuild as part of our original pipeline template.

The container is available on DockerHub and is called sapessi/aws-lambda-go18-codebuild:latest. To use this container, simply change the Image property of the CodeBuild project environment.

The pipeline template, sample app, buildspec and SAM files are attached to this gist.

@17twenty
17twenty / xForwardTest.go
Created January 18, 2017 04:45
Extracting X-Forwarded-For from connections in Golang
package main
import (
"fmt"
"log"
"net/http"
"strings"
)
func main() {
@jcloutz
jcloutz / .golang-example-gitlab-ci.yml
Last active December 6, 2022 11:20
Example Gitlab CI setup for Go using the official golang docker image
image: golang:1.7
stages:
- build
- test
before_script:
- go get github.com/tools/godep
- cp -r /builds/user /go/src/github.com/user/
- cd /go/src/github.com/user/repo
@wapa5pow
wapa5pow / gist:51eb79cf3b258a71e0934bfcb1fee463
Created August 15, 2016 05:25
Try to update ca for Amazon Linux
[root@ip-10-0-11-111 anchors]# curl -o /etc/pki/ca-trust/source/anchors/GTEGlRoot.crt https://www.cybertrust.ne.jp/sureserver/download/root_ca/GTEGlRoot.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 890 100 890 0 0 6885 0 --:--:-- --:--:-- --:--:-- 6899
[root@ip-10-0-11-111 anchors]# ls
GTEGlRoot.crt
[root@ip-10-0-11-111 anchors]# update-ca-trust
p11-kit: duplicate 'GTE CyberTrust Global Root' certificate found in: ca-bundle.legacy.crt
p11-kit: duplicate 'GTE CyberTrust Global Root' certificate found in: ca-bundle.legacy.crt
p11-kit: duplicate 'GTE CyberTrust Global Root' certificate found in: ca-bundle.legacy.crt