Skip to content

Instantly share code, notes, and snippets.

Avatar

Joe Wicentowski joewiz

  • Arlington, Virginia
View GitHub Profile
@joewiz
joewiz / post-mortem.md
Last active Aug 11, 2022
Recovery from nginx "Too many open files" error on Amazon AWS Linux
View post-mortem.md

On Tue Oct 27, 2015, history.state.gov began buckling under load, intermittently issuing 500 errors. Nginx's error log was sprinkled with the following errors:

2015/10/27 21:48:36 [crit] 2475#0: accept4() failed (24: Too many open files)

2015/10/27 21:48:36 [alert] 2475#0: *7163915 socket() failed (24: Too many open files) while connecting to upstream...

An article at http://www.cyberciti.biz/faq/linux-unix-nginx-too-many-open-files/ provided directions that mostly worked. Below are the steps we followed. The steps that diverged from the article's directions are marked with an *.

  1. * Instead of using su to run ulimit on the nginx account, use ps aux | grep nginx to locate nginx's process IDs. Then query each process's file handle limits using cat /proc/pid/limits (where pid is the process id retrieved from ps). (Note: sudo may be necessary on your system for the cat command here, depending on your system.)
  2. Added fs.file-max = 70000 to /etc/sysctl.conf
@joewiz
joewiz / entities.xqm
Last active Aug 9, 2022
Look up HTML character entities and convert references to XML-compatible characters in XQuery
View entities.xqm
xquery version "3.1";
(:~ A module containing a lookup table, in the form of a map, for HTML entities
: that aren't defined in XML, and a function for converting HTML entities in
: strings to the appropriate characters. Look up characters via map lookup:
:
: $entities:entities?ograve?character => ò
:
: @author Joe Wicentowski
: @see https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references#Character_entity_references_in_HTML
@joewiz
joewiz / yaml-to-xml.xq
Created Aug 22, 2016
Convert YAML to XML, with XQuery
View yaml-to-xml.xq
xquery version "3.0";
(: doesn't support YAML indentation yet - just a start :)
declare function local:process-yaml-value($value) {
let $single-quote := "^'(.+)'$"
let $double-quote := '^"(.+)"$'
return
if (matches($value, $single-quote) or matches($value, $double-quote)) then
let $pattern := "^['""](.+)['""]$"
@joewiz
joewiz / 01-levenshtein-distance.xq
Last active Jul 5, 2022
Calculate Levenshtein Distance, using XQuery
View 01-levenshtein-distance.xq
xquery version "3.1";
(:~
Calculate Levenshtein Distance, using XQuery
@author Guillaume Mella
@see http://apps.jmmc.fr/~mellag/xquery/levenshtein/2018/01/19/xquery-levenshtein-distance/
:)
declare function local:levenshtein-distance($string1 as xs:string?, $string2 as xs:string?)
as xs:integer
@joewiz
joewiz / README.md
Last active Jun 24, 2022
A report showing function signature mismatches in eXist's implementation of the XPath & XQuery functions & operators spec
View README.md
@joewiz
joewiz / 01-generate-function-signatureTests.xq
Last active Jun 24, 2022 — forked from dizzzz/functionSignatureTests.xquery
A script to generate function-signature-tests for xquery functions in eXist-db
View 01-generate-function-signatureTests.xq
xquery version "3.1";
declare namespace fos="http://www.w3.org/xpath-functions/spec/namespace";
declare function local:writeHeader() as xs:string {
'
xquery version "3.1";
module namespace xqfunctions="http://exist-db.org/xquery/test/xqfunctions";
@joewiz
joewiz / tokenize-sentences-nlp.xq
Last active Apr 20, 2022
Split (or "tokenize") a string into "sentences", with XQuery. See https://gist.github.com/joewiz/5889711
View tokenize-sentences-nlp.xq
xquery version "3.1";
(: Use the eXist Stanford NLP package for sentence tokenization.
: Compared to my original "naïve" approach, this approach takes a quarter the number of lines of XQuery code.
: See https://gist.github.com/joewiz/5889711 :)
import module namespace nlp="http://exist-db.org/xquery/stanford-nlp";
declare function local:tokenize-sentences($text as xs:string) {
local:tokenize-sentences($text, map{})
@joewiz
joewiz / saxon-homebrew-macos-xquery.md
Last active Apr 7, 2022
Install saxon via homebrew for command line XQuery on macOS
View saxon-homebrew-macos-xquery.md
@joewiz
joewiz / an-introduction-to-recursion-in-xquery.md
Last active Apr 7, 2022
An introduction to recursion in XQuery
View an-introduction-to-recursion-in-xquery.md

An introduction to recursion in XQuery

  • Created: Nov 28, 2017
  • Updated: Nov 29, 2017: Now covers transformation of XML documents

Recursion is a powerful programming technique, but the idea is simple: instead of performing a single operation, a function calls itself repeatedly to whittle through a larger task. In XQuery, recursion can be used to accomplish complex tasks on data that a plain FLWOR expression (which iterates through a sequence) cannot, such as transforming an entire XML document from one format into another, like TEI or DocBook into HTML, EPUB, LaTeX, or XSL-FO. Transforming a document is well-suited to recursion because each of the document's nodes may need to be examined and manipulated based on the node's type, name, and location in the document; and once a node has been processed, the transformation must continue processing the nodes' children and descendants until the deepest leaf node has been processed. But learning the technique of recursion is often hard for a beginning program

@joewiz
joewiz / chmod-recursive.xq
Created Jun 3, 2021
Batch change permissions on resources and collections in eXist-db
View chmod-recursive.xq
xquery version "3.1";
import module namespace dbutil="http://exist-db.org/xquery/dbutil";
dbutil:scan(
xs:anyURI("/db/apps/airlock-data"),
function($col, $res) {
if ($res) then
(: Set permissions on resources here :)
(