Skip to content

Instantly share code, notes, and snippets.

View densumesh's full-sized avatar

DNS densumesh

  • Devflow, Inc
  • 12:39 (UTC -07:00)
View GitHub Profile
@densumesh
densumesh / splade.rs
Created December 14, 2023 19:26
Rust Splade Embeddings generation
pub fn get_splade_vector(
input: String,
model: Model,
tokenizer: &Tokenizer,
) -> Result<Vec<f32>, ServiceError> {
let tokenized_inputs = tokenizer.encode(input, false).unwrap();
let tokens = tokenized_inputs.get_ids().to_vec();
let token_ids = Tensor::new(tokens.as_slice(), &candle_core::Device::Cpu)
.map_err(|e| ServiceError::BadRequest(format!("Could not create tensor: {}", e)))?;
@densumesh
densumesh / qdrant_upsert.rs
Created December 15, 2023 16:38
qdrant_upsert.rs
pub async fn create_new_qdrant_point_query(
point_id: uuid::Uuid,
embedding_vector: Vec<f32>,
private: bool,
card_metadata: CardMetadata,
splade_models: web::Data<SpladeModels>,
author_id: Option<uuid::Uuid>,
dataset_id: uuid::Uuid,
) -> Result<(), actix_web::Error> {
let qdrant_collection = get_env!(
import { sleep } from "bun";
const secure_the_border_act_path = "./secure_the_border_act.txt";
const tax_relief_act_path = "./tax_relief_for_american_families_act.txt";
/// Here are two bills that we want to upload to Trieve
const bills = [
await Bun.file(secure_the_border_act_path).text(),
await Bun.file(tax_relief_act_path).text(),
];
We can't make this file beautiful and searchable because it's too large.
Chunk HTML,Link,Tag Set,Tracking ID,Metadata
<div><h1>Alacrity</h1><h3>AI Based Account Takeover Prevention Platform</h3><p>We are building a safer internet. Account Takeovers remain a problem - despite 2FA and all the fraud prevention platforms out there today. We are building the next generation of fraud prevention and identity tools. Having fought fraud at the largest scale at some of the biggest companies in the world; we are now building what we wish we had and giving those tools to everyone.</p><p>Located in San Francisco; US and founded in 2024</p></div>,/companies/alacrity,W24|identity|fraud-prevention|fraud-detection|san-francisco-bay-area,29533,{"company_name":"Alacrity";"company_one_liner":"AI Based Account Takeover Prevention Platform";"company_long_description":"We are building a safer internet. Account Takeovers remain a problem - despite 2FA and all the fraud prevention platforms out there today. We are building the next generation of fraud prevention and identity tools. Having fought fraud at
import os
import dotenv
import base64
import mimetypes
import trieve_py_client
import trieve_py_client.api
import trieve_py_client.api.file_api
def to_base64(file):
import * as fs from "fs";
import * as readline from "readline";
import * as path from "path";
import {
ChunkApi,
Configuration,
CreateChunkData,
} from "@devflowinc/trieve-js-ts-client";
import { it } from "node:test";
import * as fs from "fs";
import * as readline from "readline";
const trieveApiKey = Bun.env.TRIEVE_API_KEY ?? "";
const trieveDatasetId = Bun.env.TRIEVE_DATASET_ID ?? "";
const trieveBaseUrl = Bun.env.TRIEVE_BASE_URL ?? "https://api.trieve.ai";
interface Address {
City?: string;
Country?: string;
We can't make this file beautiful and searchable because it's too large.
group_tracking_ids|tracking_id|chunk_html|time_stamp|link|episode_number|episode_title
https://www.philosophizethis.org/podcast/anarchism-part-one-26l4k-pj5c2-yehbw-jebes-cwz68-y6cd4|https://www.philosophizethis.org/podcast/anarchism-part-one-26l4k-pj5c2-yehbw-jebes-cwz68-y6cd4-0| Hello everyone! I’m Stephen West. This is Philosophize This. So I got a message a couple days ago from someone on Patreon. His name was Andrew. He was from Australia. Shout out to Australia, by the way. But he was responding to that question of what would you ask Zizek if you could ask him anything. And what he said was something to the effect of okay, Zizek. I get it. My worldview is ideology. I’m but a FEEBLE human mind… that NEEDS this ideology you’re talking about…to give me SHORTCUTS.Shortcuts that allow me to UNDERSTAND my reality…because NO human being has the mental resources to understand the FULL complexity of reality on it’s own terms. I get it. But isn’t it ALSO the case that whenever ANYONE out there does something GOO
import psycopg2
import os
import dotenv
import uuid
dotenv.load_dotenv()
# Connect to the PostgreSQL database
# Get the PostgreSQL connection details from environment variables
conn = psycopg2.connect(
os.getenv("DATABASE_URL"),