Skip to content

Instantly share code, notes, and snippets.

View EdWarga's full-sized avatar

Ed Warga EdWarga

  • Austin, TX
View GitHub Profile
# Read parent CSV
$InputFilename = Get-Content 'C:\working\splitCSV\testSplit.csv'
$OutputFilenamePattern = 'output_done_'
$LineLimit = 2000
# Initialize
$line = 0
$i = 0
$file = 0
$start = 0
@EdWarga
EdWarga / getDivs.py
Created May 25, 2018 20:55
python bits!
from bs4 import BeautifulSoup
import requests
url = "https://www.wunderground.com/weather/us/tx/corpus-christi"
r = requests.get(url)
data = r.text
@EdWarga
EdWarga / GetOAI-PMH.xqy
Last active June 6, 2018 15:28
Code Examples to support proposal for the 2018 Open Repositories conference. See files below for example scripts created for repository management activities. These are simple ad hoc examples created by a non-developer library professional. They have been coded using BaseX software and may require that software to run.
(:code snippet originally created for: Anderson, C., Stringer-Hye, & Warga, E. (2017). graphs-without-ontologies: Data and Code for 2015 VIVO Conference Presentation. XQuery, Heard Library. Retrieved from https://github.com/HeardLibrary/graphs-without-ontologies
copied from: https://github.com/HeardLibrary/graphs-without-ontologies/blob/master/XQuery/get-OAI-data.xq
This script will harvest metadata from an OAI-PMH content provider. Update the base URL, setSpec, and metadata format to target the repository, collection, and particular metadata format desired.
I have been using this script to harvest the DSpace METS records for items in test collections to perform quality control checks. See the qualityCheckQueries.xqy file to see queries to check certain aspects of item metadata stored in the PREMIS section of the DSPace METS documents.
:)
xquery version "3.1";
@EdWarga
EdWarga / CFRegisterToCSV.xq
Created December 11, 2015 17:48
Convert CFRegister Data to CSV
xquery version "3.1";
declare namespace xpf = "http://www.w3.org/2005/xpath-functions";
(: Connects to the Comédie Française registers, converts JSON data to XML, and (optionally) persists documents to BaseX database :)
(:Based on https://gist.github.com/CliffordAnderson/d8ee47849964f75030e9:)
declare function local:convert-play($map as element()) as element()
{
element play {
attribute id {$map/xpf:number[@key="id"]/text()},
xquery version "3.1";
(: Produces facets dynamically by applying a config file ('facets.xml')
to a book list ('books.csv') :)
declare function local:build-facet ($books as element()*, $path as xs:string, $name as xs:string) as element(facet)?
{
let $facets :=
for $book in $books
group by $facet :=
<terms>
<term>
<path>date</path>
<facet>date</facet>
</term>
</terms>
TEI
teiHeader
fileDesc
titleStmt
title
author
respStmt
resp
name
sponsor
xquery version "3.1";
fetch:text("
http://www.worldcat.org/webservices/catalog/search/opensearch?q=[SEARCH TERMS];wskey=[KEY]")
We can make this file beautiful and searchable if this error is corrected: It looks like row 5 should actually have 8 columns, instead of 4. in line 4.
recordIdentifier,creator,date,title,subject,publisher,language,type
oai:DiscoverArchive.Vanderbilt.edu:1803/5594,"Thomas, Randall S., 1955-; Palmiter, Alan R.; Cotter, James F.",2012,Dodd-Frank's Say on Pay: Will It Lead to a Greater Role for Shareholders in Corporate Governance?,Dodd-Frank; corporate governance; executive compensation,Cornell Law Review; http://www.lawschool.cornell.edu/research/cornell-law-review/,en_US,Article
oai:DiscoverArchive.Vanderbilt.edu:1803/5596,"Bressman, Lisa Schultz",2009,Chevron's Mistake,Canons of construction; Statutory interpretation; Judicial Review; Delegation; Administrative law,Duke Law Journal,en_US,Article
oai:DiscoverArchive.Vanderbilt.edu:1803/5597,"Gervais, Daniel J., 1963-",2005,"Intellectual Property, Trade & Development: The State of Play",intellectual property rights; international trade; economic development; TRIPS agreement,Fordham Law Review,en_US,Article
oai:DiscoverArchive.Vanderbilt.edu:1803/5599,"Maroney, Terry A.",2011,Adolescent Brain Science After Gra
xquery version "3.1";
declare namespace xsi = "http://www.w3.org/2001/XMLSchema-instance";
declare namespace oai = "http://www.openarchives.org/OAI/2.0/";
declare namespace oai_dc = "http://www.openarchives.org/OAI/2.0/oai_dc/";
declare namespace dc = "http://purl.org/dc/elements/1.1/";