kmcelwee

kmcelwee

const puppeteer = require('puppeteer');

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

(async () => {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();

kmcelwee

# goal is to map from
# /library/levi-strauss-anthropologie-structurale-1958/gallery/images/front-cover/iiif/
# ..to..
# https://iiif-cloud.princeton.edu/iiif/2/ee%2F09ce512bce02%2Fintermediate_file

import json
from django.urls import reverse
from derrida.books.models import Instance
from derrida.common.utils import absolutize_url

kmcelwee

Web Scraping & Data Analysis Cheat Sheet

Case Study

• Fortune 100 & BLM
• Coding notebook example (If it has trouble loading, try refreshing)

Definitions

kmcelwee

#!/bin/sh
#
# Check for ruby style errors and autocorrect.

red='\033[0;31m'
green='\033[0;32m'
yellow='\033[0;33m'
NC='\033[0m'

# Get only the staged files

kmcelwee

kmcelwee

gh issue create --title "Import African American Studies theses to dataspace prod" --label "senior theses" --body "When completed, don't forget to update [Lynn's spreadsheet](https://docs.google.com/spreadsheets/d/1NUUN0B1pc6iWxQoTYUW_WE0AwGCMt_qqC-e3CVlG-P0/edit#gid=0)";
sleep 3;
gh issue create --title "Import Anthropology theses to dataspace prod" --label "senior theses" --body "When completed, don't forget to update [Lynn's spreadsheet](https://docs.google.com/spreadsheets/d/1NUUN0B1pc6iWxQoTYUW_WE0AwGCMt_qqC-e3CVlG-P0/edit#gid=0)";
sleep 3;
gh issue create --title "Import Architecture School theses to dataspace prod" --label "senior theses" --body "When completed, don't forget to update [Lynn's spreadsheet](https://docs.google.com/spreadsheets/d/1NUUN0B1pc6iWxQoTYUW_WE0AwGCMt_qqC-e3CVlG-P0/edit#gid=0)";
sleep 3;
gh issue create --title "Import Art and Archaeology theses to dataspace prod" --label "senior theses" --body "When completed, don't forget to update [Lynn's spreadsheet](https://docs.google.com/s

kmcelwee

kmcelwee

{
    "created_at": "Wed Nov 18 19:02:24 +0000 2020",
    "id": 1329137902199005184,
    "id_str": "1329137902199005184",
    "full_text": "@2legit2dunk https://t.co/4lx6Z4wqAp",
    "truncated": false,
    "display_text_range": [
        13,
        36
    ],

kmcelwee

PostgreSQL & other queries in Dataspace

To enter the Postgres command line, you need to be the dspace user (sudo su - dspace). The command is psql. Here is a link to the database diagram for DSpace 5.

It sometimes may be quicker to use the REST API than creating a complicated query. And the JRuby DSpace wrapper (documentation) may be simpler as well.

Useful commands:

• \dt: describe all tables
• \d {TABLE}: describe the given table
• \copy ({query}) to '{filename}' as CSV HEADER: saves the query to a CSV with a header