rjurney

import numpy as np
import pk
import seaborn as sns


drug = pk.Drug(hl=8, t_max=1)
# 5 day simulation
conc = drug.concentration(
    60,
    1, 

rjurney

version: "3.8"

services:

  opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearchproject/opensearch:latest # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container

rjurney

All vs Paraphrase Mini-LM Model Comparisons

This experiment compares multiple methods of sentence encoding on people's names - including across character sets - using the following models:

• sentence-transformers/all-MiniLM-L12-v2
• sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

Notes

Compared to the names, JSON tends to compress scores together owing to overlapping text in formatting: field names, quotes and brackets. You can see in the name pairs name length is a source of error. The dates behave well in the JSON records.

rjurney

import pandas as pd
import numpy as np
import glob

pd.set_option('display.max_columns', 500)

all_files = glob.glob('../data/patent_applications/2019-04-07.jsonl.gz/part-*.json.gz')
li = []

for filename in all_files:

rjurney

# Adapted for MrJob from Joe Stein's example at:
# http://allthingshadoop.com/2011/12/16/simple-hadoop-streaming-tutorial-using-joins-and-keys-with-python/

import sys, os, re
from mrjob.job import MRJob

class MRJoin(MRJob):
  
  SORT_VALUES = True
  

rjurney

import gspread
from gspread_dataframe import set_with_dataframe
import pandas as pd

# Assume df_users and df_companies are your DataFrames
df_users = pd.DataFrame({'Name': ['Alice', 'Bob'], 'Profile': ['alice123', 'bob456']})
df_companies = pd.DataFrame({'Company': ['TechCorp', 'BizInc'], 'Industry': ['Tech', 'Finance']})

# Step 1: Authenticate to Google Sheets API
# (You'll need to follow the gspread authentication steps which involve creating a service account and obtaining a JSON credentials file)

rjurney

from pyspark.sql.types import StructField, IntegerType, LongType, StringType, TimestampType


def add_missing_columns(df, all_columns):
    """Add any missing columns from any DataFrame among several we want to merge."""
    for col_name, schema_field in all_columns:
        if col_name not in df.columns:
            df = df.withColumn(col_name, F.lit(None).cast(schema_field.dataType))
    return df
 

rjurney

I have run the following code to compute dimension reduction with unlabeled UMAP and DBScan for clustering to group dissimilar names for the same academic journals into clusters representing each journal.

The UMAP code is:

# Step 2: Dimension Reduction with UMAP
reducer = umap.UMAP()
reduced_embeddings = reducer.fit_transform(scaled_embeddings)

rjurney

// Place your key bindings in this file to override the defaultsauto[]
[
    {
        "key": "cmd+l",
        "command": "workbench.action.gotoLine"
    },
    {
        "key": "ctrl+g",
        "command": "-workbench.action.gotoLine"
    },

rjurney

    #
    # Create a pd.DataFrame of the nodes for analysis in a notebook
    #

    # Extract nodes and their attributes into a list of dictionaries
    node_data = [{**{"node": node}, **attr} for node, attr in G.nodes(data=True)]

    # Convert the list of dictionaries into a DataFrame
    node_df = pd.DataFrame(node_data)