Skip to content

Instantly share code, notes, and snippets.

View dtaivpp's full-sized avatar

David Tippett dtaivpp

77 followers · 40 following
View GitHub Profile
@dtaivpp
dtaivpp / local_semantic_search.md
Last active May 16, 2024 16:03
Shows how to use semantic search locally on OpenSearch using Distillibert.

Local Semantic Search in OpenSearch!

If you'd like to watch this demo through it's available on YouTube or if you prefer reading there is a walkthrough on my blog.

Cluster settings for Amazon OpenSearch and Locally running

PUT /_cluster/settings
{
    "persistent": {
        "plugins.ml_commons.only_run_on_ml_node": false,
@dtaivpp
dtaivpp / opensearch-compose-generate.sh
Last active May 14, 2024 17:17
This is a shell script for setting up a single node cluster for OpenSearch. This is something you can use in local development. It creates a secret password that is random every time.
#!/bin/bash
# 1. To run this script curl it locally:
# curl https://gist.githubusercontent.com/dtaivpp/c587d99a2cab441eba0314534ae87c86/raw -o opensearch-compose-bootstrap.sh
# 2. Change it to be executable:
# chmod +x opensearch-compose-generate.sh
# 3. Run it:
# ./opensearch-compose-generate.sh
#
# This will create:
# - docker-compose.yml file for OpenSearch
@dtaivpp
dtaivpp / cohere_semantic_search.md
Last active May 15, 2024 18:47

Semantic Search with OpenSearch and Cohere

Cluster Settings:

PUT /_cluster/settings
{
    "persistent": {
        "plugins.ml_commons.allow_registering_model_via_url": true,
        "plugins.ml_commons.only_run_on_ml_node": false,
        "plugins.ml_commons.connector_access_control_enabled": true,
@dtaivpp
dtaivpp / share-containers.md
Created April 27, 2023 16:01
This is a workflow to share docker containers/images by exporting and then importing them.

Find the image to save

docker images

Save to destination

docker save -o /path/to/output.tar org/container

Import

docker docker load -i /path/to/output.tar

@dtaivpp
dtaivpp / old-opensearch-docker-compose.yml
Last active May 13, 2024 16:18
A single node OpenSearch and OpenSearch dashboards docker compose.
services:
opensearch:
image: opensearchproject/opensearch:${OPENSEARCH_VERSION:-2.11.1}
container_name: opensearch
environment:
discovery.type: single-node
node.name: opensearch
OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
volumes:
- opensearch-data:/usr/share/opensearch/data
@dtaivpp
dtaivpp / logging_demo.py
Last active December 3, 2023 14:42
An introduction to logging in Python!
# Download this file with:
# curl -L https://gist.githubusercontent.com/dtaivpp/a9b00957aa7d9cfe33e92aff8d50c835/raw/logging_demo.py -o logging_demo.py
import logging
#
# Named loggers
#
logger = logging.getLogger(__name__)
print(logger.name)
@dtaivpp
dtaivpp / nvidia-containerd-config.toml
Created July 25, 2022 14:40
Containerd config.toml for Nvidia-Container-Runtime
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "nvidia"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
privileged_without_host_devices = false
runtime_engine = ""
@dtaivpp
dtaivpp / GHDorkerUsage
Last active November 3, 2021 23:14
usage: ghdorker [-h] [-v] [-s {repo,user,org}] [-d DORKS] [--debug] [-o OUTPUT_FILENAME] [--options INPUT_OPTION [INPUT_OPTION ...]] search
Search github for github dorks
positional arguments:
search The repo, username, or organization you would like to search
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
@dtaivpp
dtaivpp / terraform_resource_scraper.py
Created September 28, 2021 15:14
This is a quick script for scraping over a terraform directory and extracting the resources used
import yaml
import os
yaml_files = []
rootDir = '.'
for dirName, subdirList, fileList in os.walk(rootDir):
for fname in fileList:
if fname[-4:] == 'yaml':
yaml_files.append( dirName + "/" + fname)
@dtaivpp
dtaivpp / devcontainer.json
Created September 5, 2021 18:12
{
"name": "Python 3",
"build": {
"dockerfile": "Dockerfile",
"context": "..",
"args": {
// Update 'VARIANT' to pick a Python version: 3, 3.6, 3.7, 3.8, 3.9
"VARIANT": "3",
// Options
"INSTALL_NODE": "true",