rjurney

#!/usr/bin/env bash
: '
@echo off
powershell -ExecutionPolicy Bypass -Command "$ErrorActionPreference='Stop'; $ProgressPreference='SilentlyContinue'; 
$output_file = 'data/full-oldb.LATEST.zip'
$extract_dir = 'data'

Write-Host "`nDownloading the ICIJ Offshore Leaks Database to $output_file`n"
Invoke-WebRequest -Uri 'https://offshoreleaks-data.icij.org/offshoreleaks/csv/full-oldb.LATEST.zip' -OutFile $output_file

rjurney

import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

class CosineSentenceBERT(nn.Module):
    def __init__(self, model_name=SBERT_MODEL, dim=384):
        super().__init__()
        self.model_name = model_name
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)

rjurney

#!/bin/bash

#
# Quickly extract all unique address, person and company name records from pairs.json: https://www.opensanctions.org/docs/pairs/
# Note: non-commercial use only, affordable licenses available at https://www.opensanctions.org/licensing/
#

# Get the data
wget https://data.opensanctions.org/contrib/training/pairs.json -O data/pairs.json

rjurney

class SentenceBERT(torch.nn.Module):
    def __init__(self, model_name=SBERT_MODEL, dim=384):
        super().__init__()
        self.model_name = model_name
        self.tokenizer = AutoTokenizer.from_pretrained("data/fine-tuned-sbert-paraphrase-multilingual-MiniLM-L12-v2-original/checkpoint-2400/")
        self.model = AutoModel.from_pretrained("data/fine-tuned-sbert-paraphrase-multilingual-MiniLM-L12-v2-original/checkpoint-2400/")
        self.ffnn = torch.nn.Linear(dim*3, 1)
        
        # Freeze the weights of the pre-trained model
        for param in self.model.parameters():

rjurney

System: I need your help with a data science, data augmentation task. I am fine-tuning a sentence transformer paraphrase model to match pairs of addresses. I tried several embedding models and none of them perform well. They need fine-tuning for this task. I have created 27 example pairs of addresses to serve as training data for fine-tuning a SentenceTransformer model. Each record has the fields Address1, Address2, a Description of the semantic they express (ex. 'different street number') and a Label (1.0 for positive match, 0.0 for negative).

The training data covers two categories of corner cases. The first is when similar addresses in string distance aren't the same. The second is the opposite: when dissimilar addresses in string distance are the same. Your task is to read a pair of Addresses, their Description and their Label and generate 100 different examples that express a similar semantic. Your job is to create variations of these records. For some of the records, implement the logic in the Descript

rjurney

import numpy as np
import pk
import seaborn as sns


drug = pk.Drug(hl=8, t_max=1)
# 5 day simulation
conc = drug.concentration(
    60,
    1, 

rjurney

All vs Paraphrase Mini-LM Model Comparisons

This experiment compares multiple methods of sentence encoding on people's names - including across character sets - using the following models:

• sentence-transformers/all-MiniLM-L12-v2
• sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

Notes

Compared to the names, JSON tends to compress scores together owing to overlapping text in formatting: field names, quotes and brackets. You can see in the name pairs name length is a source of error. The dates behave well in the JSON records.

rjurney

import gspread
from gspread_dataframe import set_with_dataframe
import pandas as pd

# Assume df_users and df_companies are your DataFrames
df_users = pd.DataFrame({'Name': ['Alice', 'Bob'], 'Profile': ['alice123', 'bob456']})
df_companies = pd.DataFrame({'Company': ['TechCorp', 'BizInc'], 'Industry': ['Tech', 'Finance']})

# Step 1: Authenticate to Google Sheets API
# (You'll need to follow the gspread authentication steps which involve creating a service account and obtaining a JSON credentials file)

rjurney

from pyspark.sql.types import StructField, IntegerType, LongType, StringType, TimestampType


def add_missing_columns(df, all_columns):
    """Add any missing columns from any DataFrame among several we want to merge."""
    for col_name, schema_field in all_columns:
        if col_name not in df.columns:
            df = df.withColumn(col_name, F.lit(None).cast(schema_field.dataType))
    return df
 

rjurney

version: "3.8"

services:

  opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearchproject/opensearch:latest # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container