Skip to content

Instantly share code, notes, and snippets.

View thesteve0's full-sized avatar

Steven Pousty thesteve0

  • Tech Raven Consulting
  • Santa Cruz, CA
  • X @thesteve0
View GitHub Profile
@thesteve0
thesteve0 / omf-transpo-schema.json
Last active March 14, 2024 02:34
OMF transportation schema
{
"$schema": "http://json-schema.org/draft-06/schema#",
"$ref": "#/definitions/Welcome7",
"definitions": {
"Welcome7": {
"type": "object",
"additionalProperties": false,
"properties": {
"$schema": {
"type": "string",
@thesteve0
thesteve0 / generate_image_embeddings.py
Created October 30, 2023 00:26
Python code to generate image embeddings
import numpy as np
import torch
from thingsvision import get_extractor
from thingsvision.utils.storing import save_features
from thingsvision.utils.data import ImageDataset, DataLoader
source = 'custom'
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model_name = 'clip'
model_parameters = {
@thesteve0
thesteve0 / function.sql
Created September 15, 2020 06:34
A generic function for standardizing and centering explanatory variables in a table
create or replace function final.initial_center_standardize(schema_name text, table_name text, prefix text, column_names text[], pkey text)
returns text as
$$
DECLARE
col text;
new_col_name text;
debug_string text;
BEGIN
-- loop through column names
foreach col in ARRAY column_names
psql -h localhost -U groot -f /data/crunchy_demo_data/boundaries/county_boundaries.ddl.sql workshop
\COPY county_geometry (statefp, countyfp, countyns, geoid, county_name, namelsad, funcstat, aland, awater, interior_pnt, the_geom) from '/data/crunchy_demo_data/boundaries/county_boundaries_copy.txt' WITH CSV QUOTE '"';
pache Maven 3.5.2 (Red Hat 3.5.2-5)
Maven home: /usr/share/maven
Java version: 1.8.0_171, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-4.b10.fc28.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.16.7-300.fc28.x86_64", arch: "amd64", family: "unix"
[DEBUG] Created new class realm maven.api
[DEBUG] Importing foreign packages into class realm maven.api
[DEBUG] Imported: javax.annotation.* < plexus.core
[DEBUG] Imported: javax.enterprise.inject.* < plexus.core
18/05/14 15:37:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: io/thorntail/Thorntail
at com.molw.app.BirdAnalysis.main(BirdAnalysis.java:14)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
exec java -javaagent:/opt/agent-bond/agent-bond.jar=jolokia{{host=0.0.0.0}},jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -Xmx768m -XX:ParallelGCThreads=1 -XX:ConcGCThreads=1 -Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -XX:CICompilerCount=2 -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp .:/maven/* io.thorntail.Thorntail
I> No access restrictor found, access to any MBean is allowed
Jolokia: Agent started with URL http://10.1.2.138:8778/jolokia/
2018-05-14 05:11:19 INFO core:170 - THORN-000001: Thorntail - version 1.0.0-SNAPSHOT
2018-05-14 05:11:19 INFO core:172 - THORN-000010: Thorntail starting
2018-05-14 05:11:19 INFO Version:145 - WELD-000900: 3.0.3 (Final)
2018-05-14 05:11:19 INFO Bootstrap:49 - WELD-ENV-000020: Using jandex for bean discovery
2018-05-14 05:11:21 INFO Bootstrap:219 - WELD-000101: Transactional services not available. Injection of @Inject UserTransaction no
oshinko v0.5.1
Default spark image: radanalyticsio/openshift-spark:2.3-latest
Found cluster sparky
Using shared cluster sparky
Waiting for spark master http://sparky-ui:8080 to be available ...
Waiting for spark workers (1/1 alive) ...
All spark workers alive
spark-submit --class bird-analysis-0.1-SNAPSHOT.jar --master spark://sparky:7077 /opt/app-root/src/bird-analysis-0.1-SNAPSHOT.jar
18/05/14 04:56:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.ClassNotFoundException: bird-analysis-0.1-SNAPSHOT.jar
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[WARNING] 'build.plugins.plugin.version' for io.thorntail:thorntail-maven-plugin is missing. @ line 38, column 17
[ERROR] Non-resolvable import POM: Could not find artifact io.thorntail:thorntail-bom:pom:1.0.0-SNAPSHOT @ line 15, column 25
[ERROR] 'dependencies.dependency.version' for io.thorntail:thorntail-jaxrs:jar is missing. @ line 26, column 21
[ERROR] 'dependencies.dependency.version' for io.thorntail:thorntail-devtools:jar is missing. @ line 30, column 21
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project com.molw:bird-analysis:0.1-SNAPSHOT (C:\Users\steve\git\molw\birdanalysis\pom.xml) has 3 errors
[ERROR] Non-resolvable import POM: Could not find artifact io.thorntail:thorntail-bom:pom:1.0.0-SNAPSHOT @ line 15, column 25 -> [Help 2]
"C:\Program Files\DevelopmentSuite\jdk8\bin\java.exe" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2018.1\lib\idea_rt.jar=52395:C:\Program Files\JetBrains\IntelliJ IDEA 2018.1\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\DevelopmentSuite\jdk8\jre\lib\charsets.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\access-bridge-64.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\cldrdata.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\dnsns.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\jaccess.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\localedata.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\nashorn.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\sunec.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\sunjce_provider.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\sunmscapi.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\sunpkcs11.jar;C:\Program Files\DevelopmentSuite\jdk8\jre\lib\ext\zipfs.jar;C:\Program Files\DevelopmentS