Skip to content

Instantly share code, notes, and snippets.

@HughPH
HughPH / jserv_hf_fast.py
Last active Sep 9, 2021 — forked from abodacs/jserv_hf_fast.py
Run HuggingFace converted GPT-J-6B checkpoint using FastAPI and Ngrok on local GPU (3090 or Titan)
View jserv_hf_fast.py
# So you want to run GPT-J-6B using HuggingFace+FastAPI on a local rig (3090 or TITAN) ... tricky.
# special help from the Kolob Colab server https://colab.research.google.com/drive/1VFh5DOkCJjWIrQ6eB82lxGKKPgXmsO5D?usp=sharing#scrollTo=iCHgJvfL4alW
# Conversion to HF format (12.6GB tar image) found at https://drive.google.com/u/0/uc?id=1NXP75l1Xa5s9K18yf3qLoZcR6p4Wced1&export=download
# Uses GDOWN to get the image
# You will need 26 GB of space, 12+GB for the tar and 12+GB expanded (you can nuke the tar after expansion)
# HPPH: Not sure where you'll find this file, the links I found didn't work and the GDOWN was returning unauthorised errors. Maybe I'll make it a torrent.
# HPPH: I also dumped the kobold endpoint. And added one for getting token counts so you can prune your prompt if necessary.
# HPPH: And finally... Now the prompt goes in the POST body, which simplifies matters significantly.
# Near Simplest Language model API, with room to expand!
View gist:1315b6d10218aa810910f015749b601c
Callstack Function(desc) Line Bytes File Process Address
--------------------------- ---- ----- ---- ------- -------
_IO_fwrite() ... + 0x3a3ba8e0 ?
Register Info
---------------------------
EDI: 0x00007F47440024E0 ESI: 0x0000000044000080 EAX: 0x0000000000000000
EBX: 0x0000000000000000 ECX: 0x00007F47440024C0 EDX: 0x0000000000000001
EIP: 0x00007F483A3BA8E0 EBP: 0x0000000044000080 SegCs: 0x0000000000000033
EFlags: 0x0000000000010246 ESP: 0x00007F465289F4A0 SegSs: 0x000000000000002B
View gist:392a3c7d119a6c78a60899689154bb81
2604.421:0008:0031:trace:seh:NtRaiseException code=40010006 flags=0 addr=0x7b44abbc ip=7b44abbc tid=0031
2604.421:0008:0031:trace:seh:NtRaiseException info[0]=0000000000000016
2604.421:0008:0031:trace:seh:NtRaiseException info[1]=0000000001280f30
2604.421:0008:0031:trace:seh:NtRaiseException rax=0000000001280590 rbx=00007fffffea4000 rcx=0000000001280590 rdx=00000000012805b0
2604.421:0008:0031:trace:seh:NtRaiseException rsi=0000000001280690 rdi=00000000012805c0 rbp=0000000001280640 rsp=0000000001280570
2604.421:0008:0031:trace:seh:NtRaiseException r8=0000000000000002 r9=0000000001280680 r10=00000000ffffffe8 r11=0000000000000000
2604.421:0008:0031:trace:seh:NtRaiseException r12=0000000000000000 r13=0000000000000001 r14=00007f3df201b3d0 r15=0000000000000001
2604.421:0008:0031:trace:seh:dwarf_virtual_unwind function 7b44abbc base 0x7b44ab50 cie 0x7b6351d8 len 14 id 0 version 1 aug 'zR' code_align 1 data_align -8 retaddr %rip
2604.421:0008:0031:trace:seh:execute_cfa_instructions 7b44ab50: DW_CFA_def_cfa %r
View gist:94701afc757803c72bb6c9cb03c26d1a
Computer Information:
Manufacturer: Unknown
Model: Unknown
Form Factor: Laptop
No Touch Input Detected
Processor Information:
CPU Vendor: GenuineIntel
CPU Brand: Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz
CPU Family: 0x6