Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
How to programmatically write page content and PHP commands to create multiple new Wordpress pages, each linking to multiple documents.
id Title Year Department Description Directory
A01-lantel-OEM Lantel at the Boston Office of Emergency Management 2019 Boston Office of Emergency Management Since 2003, the City of Boston and surrounding cities and towns have been designated as an “Urban Areas Security Initiative” (UASI) region by the federal Department of Homeland Security. This UASI designation enables the City and the larger Metro Boston Homeland Security Region (MBHSR) to apply for and obtain federal grants for information sharing, surveillance, emergency preparedness, police and first responder communications, and other technologies and staffing. The cities and towns included in the MBHSR are Boston, Brookline, Cambridge, Chelsea, Everett, Quincy, Revere, Somerville and Winthrop. Since 2003, the region has received hundreds of millions of dollars through the UASI grant program. These funds are distributed through the Office of Emergency Management in Boston, which also coordinates regional planning meetings, technology acquisitions and development, and regional training exercises, among other efforts. In July 2019, the ACLU sent an information request to the Office of Emergency Management, to learn more about region-wide surveillance networks. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A01-lantel-OEM/
A02-lantel-EOHHS Lantel at the Executive Office of Health and Human Services 2019 Executive Office of Health and Human Services In 2019, we filed a public records request with the state Executive Office of Health and Human Services seeking information related to the agency's procurement and management of surveillance cameras and video analytics technology, including facial recognition. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A02-lantel-EOHHS/
A03-lantel-EOPSS Lantel at the Executive Office of Public Safety and Security 2019 Executive Office of Public Safety and Security In 2019, we filed a public records request with the state Executive Office of Public Safety and Security seeking information related to the agency's procurement and management of surveillance cameras and video analytics technology, including facial recognition. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A03-lantel-EOPSS/
A04-lantel-EOAF Lantel at the Executive Office for Administration and Finance 2019 Executive Office for Administration and Finance In 2019, we filed a public records request with the state Executive Office for Administration and Finance seeking information related to the agency's procurement and management of surveillance cameras and video analytics technology, including facial recognition. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A04-lantel-EOAF/
A05-lantel-MBTA Lantel at the Massachusetts Bay Transit Authority (MBTA) 2019 Massachusetts Bay Transit Authority (MBTA) In 2019, we filed a public records request with the Massachusetts Bay Transit Authority (MBTA) seeking information related to the agency's procurement and management of surveillance cameras and video analytics technology, including facial recognition. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A05-lantel-MBTA/
A06-lantel-MSP Lantel at the Massachusetts State Police 2019 Massachusetts State Police In 2019, we filed a public records request with the Massachusetts State Police seeking information related to the department's procurement and management of surveillance cameras and video analytics technology, including facial recognition. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A06-lantel-MSP/
A07-dataworks-EOPSS DataWorks at the Executive Office of Public Safety and Security 2019 Executive Office of Public Safety and Security In 2019, we filed a public records request with the state Executive Office of Public Safety and Security seeking information related to the agency's procurement of technology from the company DataWorks Plus. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A07-dataworks-EOPSS/
A08-morpho-EOHSS Morpho at the Executive Office of Health and Human Services 2019 Executive Office of Health and Human Services In 2019, we filed a public records request with the state Executive Office of Health and Human Services seeking information related to the agency's procurement of technology from the company MorphoTrak. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A08-morpho-EOHSS/
A09-IBM IBM at the Springfield Police Department 2020 Springfield Police Department In 2020, we filed a public records request with the Springfield Police Department, seeking information about its contracts with IBM. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A09-IBM/
A10-briefcam Briefcam at the City of Boston (2019) 2019 City of Boston In 2019, we filed a public records request with the City of Boston, seeking information about its contracts with the company BriefCam, a manufacturer of video analytics technology. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A10-briefcam/
A11a-solicitations MA Police Departments Receiving Solicitations from Clearview AI 2020 Various Police Departments In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The following departments provided records detailing that they had received solicitations from the company: /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11a-solicitations/
A11b-unresponsive MA Police Departments with No Records from Clearview AI 2020 Various Police Departments In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The following departments informed us that they had no responsive records: /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11b-unresponsive/
A11c-agawam Clearview AI at the Agawam PD 2020 Agawam PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Agawam Police Department provided records indicating an officer, Dan Bonafilia, received a solicitation and activated an account with Clearview. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11c-agawam/
A11d-andover Clearview AI at the Andover PD 2020 Andover PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Andover Police Department provided records indicating discussions with a Clearview representative. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11d-andover/
A11e-boston Clearview AI at the Boston PD 2020 Boston PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Boston Police Department has yet to provide responsive documents; we are awaiting appeal. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11e-boston/
A11f-beverly Clearview AI at the Beverly PD 2020 Beverly PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Beverly Police Department provided records indicating emails with solicitations from different distribution lists where Clearview was present. They also showed an email from a PD in Indiana that actively seeks information from Clearview. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11f-beverly/
A11g-marlborough Clearview AI at the Marlborough PD 2020 Marlborough PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Marlborough Police Department provided records indicating patrol officer Christopher Leduc activated the Clearview link and has an active account. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11g-marlborough/
A11h-salem Clearview AI at the Salem PD 2020 Salem PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Salem Police Department provided records indicating Captain Federik Ryan attended a webinar related to facial recognition provided by Signet. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11h-salem/
A11i-somerset Clearview AI at the Somerset PD 2020 Somerset PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Agawam Police Department provided records indicating an officer received a solicitation and used a pilot of Clearview. Using the product, they helped an NYPD officer after an officer saw a post on “crimedex.” After the free trial ended, they received a pricing proposal for an annual subscription rate of $10,000. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11i-somerset/
A11j-wellesley Clearview AI at the Wellesley PD 2020 Wellesley PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Wellesley Police Department provided records indicating Officer Brian Shore signed up for a free trial for Clearview. After the free trial expired they sent an offer for a paid subscription which Officer Shore did not respond to. Responsive records also include a slide presentation from Clearview that outlines how the product works, a Kirkland & Ellis memo that alleges its legality, a Clearview “search tips” document, and information regarding a training hosted by the Wellesley PD on facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11j-wellesley/
A11k-yarmouth Clearview AI at the Yarmouth PD 2020 Yarmouth PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Yarmouth Police Department provided records indicating Detective David Schneeweis activated the link sent by Clearview AI. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11k-yarmouth/
A11l-bourne Clearview AI at the Bourne PD 2020 Bourne PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Bourne Police Department provided records indicating that the Bourne PD requested information about Clearview but never used it.. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11l-bourne/
A11m-dalton Clearview AI at the Dalton PD 2020 Dalton PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Dalton Police Department provided records indicating how the Dalton PD uses the Fusion Center and Coplink face recognition software. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11m-dalton/
A11n-falmouth Clearview AI at the Falmouth PD 2020 Falmouth PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Falmouth Police Department provided records indicating Detective Massi activated the account, tried it to make some searches, and decided that it was ill-suited for law enforcement purposes. Those searches were not recorded. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11n-falmouth/
A11o-great barrington Clearview AI at the Great Barrington PD 2020 Great Barrington PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Great Barrington Police Department provided records indicating that they communicated about the pandemic and the possibility of an ACLU initiative to advocate for incarcerated people. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11o-great barrington/
A11p-harvard Clearview AI at the Harvard PD 2020 Harvard PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Harvard Police Department provided records indicating training materials related to Coplink, a different facial recognition software. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11p-harvard/
A11q-norfolk-sheriff Clearview AI at the Norfolk Sheriff's Office 2020 Norfolk Sheriff's Office In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Norfolk Sheriff's Department provided records indicating an officer activated an account but did not use it. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11q-norfolk-sheriff/
signet Clearview AI at the Oxford PD 2020 Oxford PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Oxford Police Department provided records indicating the Oxford PD activated a Clearview account, though it is unclear whether they continued to use it after the pilot ended. Responsive records also include some screen captions of Clearview. signet
A11s-sudbury Clearview AI at the Sudbury PD 2020 Sudbury PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Sudbury Police Department provided records indicating solicitations from Clearview and a training manual related to face recognition software. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11s-sudbury/
A11t-west bridgewater Clearview AI at the West Bridgewater PD 2020 West Bridgewater PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The West Bridgewater Police Department provided records indicating that they use Hikvision for license plate reader software, but do not have an interest in facial recognition. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11t-west bridgewater/
A11u-foxborough Clearview AI at the Foxbourugh PD 2020 Foxbourugh PD In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Foxborough Police Department provided records indicating the Foxborough PD has multiple Clearview accounts. Responsive records also include solicitations and Crimedex files showing Clearview searches in other parts of the country. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11u-foxborough/
A11v-msp Clearview AI at the Massachusetts State Police (MSP) 2020 Massachusetts State Police (MSP) In 2020, we filed public records requests with various police departments throughout Massachusetts, seeking information about Clearview AI, a facial recognition company. The Massachusetts State Police provided records indicating an inquiry from the Richmond PD regarding Clearview. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A11-clearview/A11v-msp/
A12-briefcam2020 Briefcam at the City of Boston (2020) 2020 City of Boston In 2020, we filed a public records request with the City of Boston, seeking information about its contracts with the company BriefCam, a manufacturer of video analytics technology. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A12-briefcam2020/
A13-COPLINK-MSP COPLINK Facematch at the Massachusetts State Police 2019-2020 Massachusetts State Police (MSP) In 2020, we filed a public records request with the Massachusetts State Police seeking information about the agency's use of the FaceMatch facial recognition feature in the COPLINK intelligence database. /Users/lchambers/D4J/dir_for_d4j/A-privatecompanies/A13-COPLINK-MSP/
B01-EOPSS EOPSS Grants for Surveillance Tech in MA Schools 2019 Executive Office of Public Safety and Security In 2019, we filed a public records request with the Executive Office of Public Safety and Security seeking information about grants the agency provided to schools throughout the state for the purchase of surveillance related technologies. /Users/lchambers/D4J/dir_for_d4j/B-schools/B01-EOPSS/
B02-FRTAugust2019 Face Recognition in MA Schools 2019 Various School Districts In 2019, we filed numerous public records requests with public school districts across the state seeking information about facial recognition technology. Only Boston returned any responsive documents. /Users/lchambers/D4J/dir_for_d4j/B-schools/B02-FRTAugust2019/
B03-revere Face Recognition in Revere Public Schools 2019 Revere Public Schools In 2019, we filed a public records request with the Revere Public Schools seeking information pertaining to the use of facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/B-schools/B03-revere/
C01-dcjis-frt Face Recognition at the Department of Criminal Justice Information Services (DCJIS) 2019 Department of Criminal Justice Information Services (DCJIS) In 2019, we filed a public records request with the Department of Criminal Justice Information Services (DCJIS) seeking information about facial recognition and the use of facial images under the state's control. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C01-dcjis-frt/
C03-eopss-frt Face Recognition at the Executive Office of Public Safety and Security 2019 Executive Office of Public Safety and Security In 2019, we filed a public records request with the Executive Office of Public Safety and Security seeking information about facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C03-eopss-frt/
C05-msp-frt-2018 Face Recognition at the Massachusetts State Police (MSP) 2018 Massachusetts State Police (MSP) In 2018, we filed a public records request with the Massachusetts State Police seeking information about the agency's use of facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C05-msp-frt-2018/
C06-msp-frt-2019 Known Use of Face Recognition at the Massachusetts State Police (MSP) 2019 Massachusetts State Police (MSP) In 2019, we filed a public records request with the Massachusetts State Police seeking information about the agency's known use of facial recognition technology and related communications with other public entities. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C06-msp-frt-2019/
C07-msp-rmv Massachusetts State Police (MSP) Use of Face Recognition at the Registry of Motor Vehicles (RMV) 2019 Massachusetts State Police (MSP) In 2019, we filed a public records request with the Massachsuetts State Police seeking information about the agency's use of the Registry of Motor Vehicles facial recognition system. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C07-msp-rmv/
C08-masheriffassn Face Recognition at the MA Sheriff's Association [2019] 2019 MA Sheriff's Association In 2019, we filed a public records request with the Massachsuetts Sheriff's Association seeking information about the use of facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C08-masheriffassn/
C09-masheriffassn-frt Face Recognition at the MA Sheriff's Association [2018] 2018 MA Sheriff's Association In 2018, we filed a public records request with the Massachsuetts Sheriff's Association seeking information about the use of facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C09-masheriffassn-frt/
C10-mbta-frt Face Recognition at the Massachusetts Bay Transit Authority (MBTA) 2018-2019 Massachusetts Bay Transit Authority (MBTA) In 2018 and 2019, we filed a public records request with the Massachusetts Bay Transit Authority seeking information related to facial recognition technology. After they failed to respond, we filed an appeal with the State Supervisor of Records. The MBTA has still yet to provide responsive records. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C10-mbta-frt/
C11-dot-frt Face Recognition at the Department of Transportation (DOT) 2019 Department of Transportation (DOT) In 2019, we filed two public records requests with the Department of Transportation seeking information related to the use of facial recognition technology at the Registry of Motor Vehicles. The DOT did not respond to our requests, so we filed suit. Ultimately, as a result of the lawsuit, the agency disclosed thousands of pages of documents, revealing that the RMV had for years been performing facial recognition searches for local, state, and federal policing and immigration agencies, hundreds of times per year. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C11-dot-frt/
C12-eohhs-frt Face Recognition at the Executive Office of Health and Human Services 2019 Executive Office of Health and Human Services In 2019, we filed a public records request with the Executive Office of Health and Human Services seeking information related to facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C12-eohhs-frt/
C13-dcjis-frtproducts Face Recognition at the Department of Criminal Justice Information Services (DCJIS) 2019 Department of Criminal Justice Information Services (DCJIS) In 2019, we filed a public records request with the state Department of Criminal Justice Information Services (DCJIS) seeking information about the use of facial recognition technology and the agency's use of facial images. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C13-dcjis-frtproducts/
C14-MSP-booking-database Massachusetts State Police Access to Fusion Center Booking Database 2017 Massachusetts State Police (MSP) In 2017, we filed a public records request with the Massachusetts State Police seeking information about MSP's access to a 'booking photo database' via the Commonwealth Fusion Center. The database was reported on in the Boston Globe, and allegedly contains over 2.6 million individuals' photos and accessible by thousands of analysts at hundreds of law enforcement agenices. /Users/lchambers/D4J/dir_for_d4j/C-stateagencies/C14-MSP-booking-database/
D01-unresponsive MA Police Departments with No Records on Face Recognition (2018-2019) 2018-2019 Various Police Departments In the July 2018 and March 2019, we filed numerous public records requests with over 40 police departments and law enforcment agencies throughout Massachusetts seeking information about facial recognition technology. The following Departments told us they had no responsive records. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D01-unresponsive/
D02-cambridge Facial Recognition at the Cambridge PD 2018 Cambridge PD In 2018, we filed a public records request with the Cambridge Police Department seeking information related to facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D02-cambridge/
D03-medford Facial Recognition at the Medford PD 2019 Medford PD In 2019, we filed a public records request with the Medford Police Department seeking information related to facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D03-medford/
D04-newbedford Facial Recognition at the New Bedford PD 2018 New Bedford PD In 2018, we filed a public records request with the New Bedford Police Department seeking information related to facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D04-newbedford/
D05-northampton Facial Recognition & COPLINK at the Northampton PD 2019 Northampton PD In 2019, we filed a public records request with the Northampton Police Department seeking information about facial recognition technology and the Department's use of the statewide COPLINK intelligence database. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D05-northampton/
D06-plymouth2018 Suspect Technologies and the Plymouth PD 2018 Plymouth PD In July 2018, the ACLU of Massachusetts filed requests with dozens of police departments to learn about how they use face surveillance technology. The Plymouth Police Department provided hundreds of emails in response to that request. The emails contain a years long correspondence between a billionaire-backed face surveillance start-up called “Suspect Technologies” and representatives for the Plymouth Police Department. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D06-plymouth2018/
D07-plymouth2019 Facial Recognition at the Plymouth PD 2019 Plymouth PD In 2019, we filed a public records request with the Plymouth Police Department seeking information related to facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D07-plymouth2019/
D08-revere Facial Recognition at the Revere PD 2018 Revere PD In 2018, we filed a public records request with the Revere Police Department seeking information related to facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D08-revere/
D09-salem Facial Recognition at the Salem PD 2019 Salem PD In 2019, we filed a public records request with the Salem Police Department seeking information related to facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D09-salem/
D10-springfield Facial Recognition at the Springfield PD 2019 Springfield PD In 2019, we filed a public records request with the Springfield Police Department seeking information related to facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D10-springfield/
D11-peabody Facial Recognition at the Peabody PD 2019 Peabody PD In 2019, we filed a public records request with the Peabody Police Department seeking information related to facial recognition technology. /Users/lchambers/D4J/dir_for_d4j/D-town&city/D11-peabody/
E01-camera-locs Location of Surveillance Cameras Across Boston 2019 City of Boston In 2019, we filed a public records request with the City of Boston seeking information revealing the locations of all city-owned surveillance cameras. /Users/lchambers/D4J/dir_for_d4j/E-boston/E01-camera-locs/
E02-videofootage Video Surveillance Data Sharing in Boston 2019 City of Boston In 2019, we filed a public records request with the City of Boston seeking information about the City's agreements with external entities to share video surveillance data. /Users/lchambers/D4J/dir_for_d4j/E-boston/E02-videofootage/
E03-camerastrategymeetings Boston Security Camera Strategy Meetings 2019 City of Boston In 2019, we filed a public records request with the City of Boston seeking information related to the development and planning of the City's Critical Infrastructure Monitoring System (CIMS), the regional surveillance camera network. /Users/lchambers/D4J/dir_for_d4j/E-boston/E03-camerastrategymeetings/
F01-dea Facial Recognition at the Drug Enforcement Administration (DEA) 2019 Drug Enforcement Administration (DEA) In 2019, the ACLU of Massachusetts and the National ACLU's Speech, Privacy, and Technology Project filed a federal FOIA request with the Drug Enforcement Administration (DEA) and the Federal Bureau of Investigation (FBI) seeking information about how these agencies use facial recognition and other remote biometric monitoring technologies. /Users/lchambers/D4J/dir_for_d4j/F-federal/F01-dea/
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"toc": true
},
"source": [
"<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
"<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Set-up\" data-toc-modified-id=\"Set-up-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Set up</a></span><ul class=\"toc-item\"><li><span><a href=\"#Import-necessary-packages\" data-toc-modified-id=\"Import-necessary-packages-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Import necessary packages</a></span></li><li><span><a href=\"#Use-pandas-to-load-spreadsheet-as-dataframe\" data-toc-modified-id=\"Use-pandas-to-load-spreadsheet-as-dataframe-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Use <code>pandas</code> to load spreadsheet as dataframe</a></span></li><li><span><a href=\"#Define-the-root-URL-for-where-your-documents-are-on-the-server\" data-toc-modified-id=\"Define-the-root-URL-for-where-your-documents-are-on-the-server-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Define the root URL for where your documents are on the server</a></span></li></ul></li><li><span><a href=\"#Write-Wordpress-page-content-for-each-page\" data-toc-modified-id=\"Write-Wordpress-page-content-for-each-page-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Write Wordpress page content for each page</a></span><ul class=\"toc-item\"><li><span><a href=\"#Define-page-content-template,-using-string-formatting-with-replacement-fields\" data-toc-modified-id=\"Define-page-content-template,-using-string-formatting-with-replacement-fields-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Define page content template, using string formatting with replacement fields</a></span></li><li><span><a href=\"#Map-URLs-to-.zip-files-for-pages-with-a-ton-of-documents\" data-toc-modified-id=\"Map-URLs-to-.zip-files-for-pages-with-a-ton-of-documents-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Map URLs to .zip files for pages with a ton of documents</a></span></li><li><span><a href=\"#Function-to-turn-spreadsheet-row-into-page-content\" data-toc-modified-id=\"Function-to-turn-spreadsheet-row-into-page-content-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Function to turn spreadsheet row into page content</a></span></li><li><span><a href=\"#Use-.apply()-to-write-content-for-all-pages\" data-toc-modified-id=\"Use-.apply()-to-write-content-for-all-pages-2.4\"><span class=\"toc-item-num\">2.4&nbsp;&nbsp;</span>Use <code>.apply()</code> to write content for all pages</a></span></li></ul></li><li><span><a href=\"#Write-PHP-snippets-for-all-pages\" data-toc-modified-id=\"Write-PHP-snippets-for-all-pages-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Write PHP snippets for all pages</a></span><ul class=\"toc-item\"><li><span><a href=\"#Map-parent-page-IDs\" data-toc-modified-id=\"Map-parent-page-IDs-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Map parent page IDs</a></span></li><li><span><a href=\"#Function-to-turn-spreadsheet-row-into-PHP-command\" data-toc-modified-id=\"Function-to-turn-spreadsheet-row-into-PHP-command-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Function to turn spreadsheet row into PHP command</a></span></li><li><span><a href=\"#Use-.apply()-to-write-PHP-command-for-all-pages\" data-toc-modified-id=\"Use-.apply()-to-write-PHP-command-for-all-pages-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Use <code>.apply()</code> to write PHP command for all pages</a></span></li><li><span><a href=\"#Write-all-of-the-PHP-commands-out-to-a-txt-file\" data-toc-modified-id=\"Write-all-of-the-PHP-commands-out-to-a-txt-file-3.4\"><span class=\"toc-item-num\">3.4&nbsp;&nbsp;</span>Write all of the PHP commands out to a txt file</a></span></li></ul></li></ul></div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Programmatically create WP pages with Python\n",
"\n",
"How to programmatically write page content and PHP commands to create multiple new Wordpress pages, each linking to multiple documents.\n",
"\n",
"This notebook assumes you have:\n",
"1. A directory structure containing all the documents to link to, organized such that there is one directory per page\n",
"1. Those same documents uploaded to Wordpress, preserving directory structure\n",
"1. A spreadsheet listing each desired page, the path to its corresponding folder in the directory structure, and any desired metadata to include\n",
"\n",
"---\n",
"Feel free to contact [lchambers@aclum.org](mailto:lchambers@aclum.org) with any questions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set up\n",
"### Import necessary packages"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# Standard library\n",
"import glob\n",
"import os\n",
"import re\n",
"import urllib\n",
"\n",
"# Third party packages\n",
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Use `pandas` to load spreadsheet as dataframe"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>Title</th>\n",
" <th>Year</th>\n",
" <th>Department</th>\n",
" <th>Description</th>\n",
" <th>Files Transferred from Dropbox?</th>\n",
" <th>Numbered requests &amp; responses?</th>\n",
" <th>Directory</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>A01-lantel-OEM</td>\n",
" <td>Lantel at the Boston Office of Emergency Manag...</td>\n",
" <td>2019</td>\n",
" <td>Boston Office of Emergency Management</td>\n",
" <td>Since 2003, the City of Boston and surrounding...</td>\n",
" <td>Yes</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/A-privatecomp...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>A02-lantel-EOHHS</td>\n",
" <td>Lantel at the Executive Office of Health and H...</td>\n",
" <td>2019</td>\n",
" <td>Executive Office of Health and Human Services</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/A-privatecomp...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>A03-lantel-EOPSS</td>\n",
" <td>Lantel at the Executive Office of Public Safet...</td>\n",
" <td>2019</td>\n",
" <td>Executive Office of Public Safety and Security</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/A-privatecomp...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>A04-lantel-EOAF</td>\n",
" <td>Lantel at the Executive Office for Administrat...</td>\n",
" <td>2019</td>\n",
" <td>Executive Office for Administration and Finance</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/A-privatecomp...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>A05-lantel-MBTA</td>\n",
" <td>Lantel at the Massachusetts Bay Transit Author...</td>\n",
" <td>2019</td>\n",
" <td>Massachusetts Bay Transit Authority (MBTA)</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/A-privatecomp...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>59</td>\n",
" <td>D11-peabody</td>\n",
" <td>Facial Recognition at the Peabody PD</td>\n",
" <td>2019</td>\n",
" <td>Peabody PD</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>Yes</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/D-town&amp;city/D...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>60</td>\n",
" <td>E01-camera-locs</td>\n",
" <td>Location of Surveillance Cameras Across Boston</td>\n",
" <td>2019</td>\n",
" <td>City of Boston</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Waiting for response</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/E-boston/E01-...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>61</td>\n",
" <td>E02-videofootage</td>\n",
" <td>Video Surveillance Data Sharing in Boston</td>\n",
" <td>2019</td>\n",
" <td>City of Boston</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>Yes</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/E-boston/E02-...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>62</td>\n",
" <td>E03-camerastrategymeetings</td>\n",
" <td>Boston Security Camera Strategy Meetings</td>\n",
" <td>2019</td>\n",
" <td>City of Boston</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>Yes</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/E-boston/E03-...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>63</td>\n",
" <td>F01-dea</td>\n",
" <td>Facial Recognition at the Drug Enforcement Adm...</td>\n",
" <td>2019</td>\n",
" <td>Drug Enforcement Administration (DEA)</td>\n",
" <td>In 2019, the ACLU of Massachusetts and the Nat...</td>\n",
" <td>Waiting for response</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/F-federal/F01...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>64 rows × 8 columns</p>\n",
"</div>"
],
"text/plain": [
" id \\\n",
"0 A01-lantel-OEM \n",
"1 A02-lantel-EOHHS \n",
"2 A03-lantel-EOPSS \n",
"3 A04-lantel-EOAF \n",
"4 A05-lantel-MBTA \n",
".. ... \n",
"59 D11-peabody \n",
"60 E01-camera-locs \n",
"61 E02-videofootage \n",
"62 E03-camerastrategymeetings \n",
"63 F01-dea \n",
"\n",
" Title Year \\\n",
"0 Lantel at the Boston Office of Emergency Manag... 2019 \n",
"1 Lantel at the Executive Office of Health and H... 2019 \n",
"2 Lantel at the Executive Office of Public Safet... 2019 \n",
"3 Lantel at the Executive Office for Administrat... 2019 \n",
"4 Lantel at the Massachusetts Bay Transit Author... 2019 \n",
".. ... ... \n",
"59 Facial Recognition at the Peabody PD 2019 \n",
"60 Location of Surveillance Cameras Across Boston 2019 \n",
"61 Video Surveillance Data Sharing in Boston 2019 \n",
"62 Boston Security Camera Strategy Meetings 2019 \n",
"63 Facial Recognition at the Drug Enforcement Adm... 2019 \n",
"\n",
" Department \\\n",
"0 Boston Office of Emergency Management \n",
"1 Executive Office of Health and Human Services \n",
"2 Executive Office of Public Safety and Security \n",
"3 Executive Office for Administration and Finance \n",
"4 Massachusetts Bay Transit Authority (MBTA) \n",
".. ... \n",
"59 Peabody PD \n",
"60 City of Boston \n",
"61 City of Boston \n",
"62 City of Boston \n",
"63 Drug Enforcement Administration (DEA) \n",
"\n",
" Description \\\n",
"0 Since 2003, the City of Boston and surrounding... \n",
"1 In 2019, we filed a public records request wit... \n",
"2 In 2019, we filed a public records request wit... \n",
"3 In 2019, we filed a public records request wit... \n",
"4 In 2019, we filed a public records request wit... \n",
".. ... \n",
"59 In 2019, we filed a public records request wit... \n",
"60 In 2019, we filed a public records request wit... \n",
"61 In 2019, we filed a public records request wit... \n",
"62 In 2019, we filed a public records request wit... \n",
"63 In 2019, the ACLU of Massachusetts and the Nat... \n",
"\n",
" Files Transferred from Dropbox? Numbered requests & responses? \\\n",
"0 Yes NaN \n",
"1 Yes NaN \n",
"2 Yes NaN \n",
"3 Yes NaN \n",
"4 Yes NaN \n",
".. ... ... \n",
"59 Yes Yes \n",
"60 Waiting for response NaN \n",
"61 Yes Yes \n",
"62 Yes Yes \n",
"63 Waiting for response NaN \n",
"\n",
" Directory \n",
"0 /Users/lchambers/D4J/dir_for_d4j/A-privatecomp... \n",
"1 /Users/lchambers/D4J/dir_for_d4j/A-privatecomp... \n",
"2 /Users/lchambers/D4J/dir_for_d4j/A-privatecomp... \n",
"3 /Users/lchambers/D4J/dir_for_d4j/A-privatecomp... \n",
"4 /Users/lchambers/D4J/dir_for_d4j/A-privatecomp... \n",
".. ... \n",
"59 /Users/lchambers/D4J/dir_for_d4j/D-town&city/D... \n",
"60 /Users/lchambers/D4J/dir_for_d4j/E-boston/E01-... \n",
"61 /Users/lchambers/D4J/dir_for_d4j/E-boston/E02-... \n",
"62 /Users/lchambers/D4J/dir_for_d4j/E-boston/E03-... \n",
"63 /Users/lchambers/D4J/dir_for_d4j/F-federal/F01... \n",
"\n",
"[64 rows x 8 columns]"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"all_pages_df = pd.read_csv(\"db_face_rec.csv\")\n",
"all_pages_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Define the root URL for where your documents are on the server"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"D4J_root_upload_url = \"https://data.aclum.org/wp-content/uploads/2021/02/\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Write Wordpress page content for each page"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Define page content template, using string formatting with replacement fields\n",
"\n",
"Here we mix raw HTML with the syntax for the Visual Composer editor in Wordpress to define a template that contains the common elements and structure of all pages.\n",
"\n",
"The replacement fields I include here include:\n",
"- `description` - Text describing the current page\n",
"- `agency` - Agency that received the public records request\n",
"- `category` - Topic of request (e.g. surveillance, law enforcement)\n",
"- `year` - Date of request\n",
"- `n_docs` - Number of responsive documents returned\n",
"- `request_li` - HTML with list of \"request\" documents\n",
"- `response_li` - HTML with list of \"responses\" documents\n",
"- `download_all_files` - button to download multiple files in a `.zip`\n",
"- `docs_table` - HTML with table of all responsive documents"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"page_content_template = '''\n",
"[vc_row]\n",
" [vc_column column_width_percent=\"100\" gutter_size=\"3\" style=\"light\" overlay_alpha=\"50\" shift_x=\"0\" shift_y=\"0\" shift_y_down=\"0\" z_index=\"0\" medium_width=\"0\" mobile_width=\"0\" width=\"2/3\"][vc_column_text]\n",
" <h2><span style=\"color: #000000;\">Background</span></h2>\n",
" <span style=\"color: #777777;\">\n",
" {description}\n",
" </span>\n",
" [/vc_column_text][/vc_column]\n",
" \n",
" [vc_column column_width_percent=\"100\" position_vertical=\"middle\" gutter_size=\"3\" overlay_alpha=\"50\" shift_x=\"0\" shift_y=\"0\" shift_y_down=\"0\" z_index=\"0\" medium_width=\"0\" mobile_width=\"0\" width=\"1/3\"][vc_column_text border_color=\"transparent\"]\n",
" <table style=\"height: 72px; width: 99.8992%; border: none;\">\n",
" <tbody>\n",
" <tr style=\"height: 24px; background-color: #f2f2f2;\">\n",
" <td style=\"width: 50%; height: 24px;\"><strong><span style=\"color: #808080; font-size: 16px;\"><span style=\"font-family: Poppins;\">Request Submitted To</span></span></strong></td>\n",
" <td style=\"width: 50%; height: 24px; font-weight: 200 !important;\"><span style=\"color: #808080; font-size: 16px;\"><span style=\"font-family: Poppins;\">\n",
" {agency}\n",
" </span></span></td>\n",
" </tr>\n",
" <tr style=\"height: 24px;\">\n",
" <td style=\"width: 50%; height: 24px;\"><strong><span style=\"color: #808080; font-size: 16px;\"><span style=\"font-family: Poppins;\">Category</span></span></strong></td>\n",
" <td style=\"width: 50%; height: 24px; font-weight: 200 !important;\"><span style=\"color: #808080; font-size: 16px;\"><span style=\"font-family: Poppins;\">\n",
" {category}\n",
" </span></span></td>\n",
" </tr>\n",
" <tr style=\"height: 24px; background-color: #f2f2f2;\">\n",
" <td style=\"width: 50%; height: 24px;\"><strong><span style=\"color: #808080; font-size: 16px;\"><span style=\"font-family: Poppins;\">Year Filed</span></span></strong></td>\n",
" <td style=\"width: 50%; height: 24px; font-weight: 200 !important;\"><span style=\"color: #808080; font-size: 16px;\"><span style=\"font-family: Poppins;\">\n",
" {year}\n",
" </span></span></td>\n",
" </tr>\n",
" <tr>\n",
" <td style=\"width: 50%;\"><strong><span style=\"color: #808080; font-size: 16px;\"><span style=\"font-family: Poppins;\">Number of Documents Received</span></span></strong></td>\n",
" <td style=\"width: 50%; font-weight: 200 !important;\"><span style=\"color: #808080; font-size: 16px;\"><span style=\"font-family: Poppins;\">\n",
" {n_docs}\n",
" </span></span></td>\n",
" </tr>\n",
" </tbody>\n",
" </table>\n",
" [/vc_column_text][/vc_column]\n",
"[/vc_row]\n",
"\n",
"[vc_row]\n",
" [vc_column width=\"1/2\"][vc_column_text]\n",
" <h2>We said...</h2>\n",
" <ul>\n",
" {request_li}\n",
" </ul>\n",
" [/vc_column_text][/vc_column]\n",
"\n",
" [vc_column width=\"1/2\"][vc_column_text]\n",
" <h2>They said...</h2>\n",
" <ul>\n",
" {response_li}\n",
" </ul>\n",
" [/vc_column_text][/vc_column]\n",
"[/vc_row]\n",
"\n",
"[vc_row][vc_column width=\"1/1\"][vc_column_text]\n",
" <h2>Responsive Documents</h2>\n",
" {download_all_files}\n",
" {docs_table}\n",
"[/vc_column_text][/vc_column][/vc_row]\n",
"\n",
"'''"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Map URLs to .zip files for pages with a ton of documents\n",
"For pages with many documents, I manually compressed all documents into a .zip file and uploaded to Google Drive. Here we map the URLs to the file downloads to different page IDs."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"zipped_doc_pages = {\n",
" \"A01\": \"https://drive.google.com/u/1/uc?id=1Cd6sY5J3oyjL4ydj90tXL4gsTsA0LJUe&export=download\",\n",
" \"A06\": \"https://drive.google.com/u/1/uc?id=14G6MtdDB9sCH0ujYlCVbpj5EtvWf-zgr&export=download\", \n",
" \"A11a\": \"https://drive.google.com/u/1/uc?id=1c6DDgkfue-GCFHcZlIC6gob50uRKt9lg&export=download\", \n",
" \"A11b\": \"https://drive.google.com/u/1/uc?id=1j9jh6iJo-o2KJssiQbUkkajwxYcmOGTS&export=download\", \n",
" \"A11t\": \"https://drive.google.com/u/1/uc?id=1BPRgoVsJbgit35kaYhuW2xq14bCAxZkD&export=download\", \n",
" \"C11\": \"https://drive.google.com/u/1/uc?id=1uzHJ4WAfmnD5lsuRzT-9xTYr0BwDwsjM&export=download\", \n",
" \"D01\": \"https://drive.google.com/u/1/uc?id=19hj5afd9pWH7-zevXa52TA4Tpg4zAman&export=download\", \n",
" \"D05\": \"https://drive.google.com/u/1/uc?id=1Gn1BpII8Rm3BhYwGFu8gBVohxvRXgnMt&export=download\", \n",
" \"D10\": \"https://drive.google.com/u/1/uc?id=1XlWZh16DeCyZYj1eTjcAJCn9xSeMdTUp&export=download\", \n",
"}\n",
"\n",
"regex_zipped_docs = '|'.join(zipped_doc_pages.keys())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Function to turn spreadsheet row into page content"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"def write_wordpress_page(row):\n",
" page_metadata = row\n",
" \n",
" if page_metadata[\"id\"] in [\"A11a-solicitations\", \"A11b-unresponsive\", \"D01-unresponsive\"]:\n",
" # In reality, use custom functions to write tricky pages.\n",
" # In this example, just skip.\n",
" return \"\"\n",
" \n",
" # Basic metadata\n",
" description = page_metadata[\"Description\"]\n",
" agency = page_metadata[\"Department\"]\n",
" category = \"Surveillance\"\n",
" year = page_metadata[\"Year\"]\n",
" page_dir = page_metadata[\"Directory\"]\n",
" \n",
" # Create li tag for each request doc\n",
" requests = glob.glob(os.path.join(page_dir, \"request\", \"**\"), recursive=True)\n",
" requests = sorted([r for r in requests if os.path.isfile(r)])\n",
" \n",
" requests_li = \"\" \n",
" for r in requests:\n",
" url = D4J_root_upload_url + \\\n",
" r.split(\"dir_for_d4j/\")[-1].replace(\" \", \"%20\").replace(\"#\", \"\").replace(\"''\", \"\")\n",
" title = os.path.basename(r)\n",
" title = os.path.splitext(title)[0]\n",
" requests_li += '<li><a href=\"{url}\">{title}</a></li>'.format(url=url, title=title)\n",
" \n",
" # Create li tag for each response doc\n",
" responses = glob.glob(os.path.join(page_dir, \"response\", \"**\"), recursive=True)\n",
" responses = sorted([r for r in responses if os.path.isfile(r)])\n",
"\n",
" if len(responses) == 0:\n",
" responses_li = \"<i>No formal written acknowledgement or response was received.</i>\"\n",
" else:\n",
" responses_li = \"\"\n",
" for r in responses:\n",
" url = D4J_root_upload_url + \\\n",
" r.split(\"dir_for_d4j/\")[-1].replace(\" \", \"%20\").replace(\"#\", \"\").replace(\"''\", \"\")\n",
" title = os.path.basename(r)\n",
" title = os.path.splitext(title)[0]\n",
" responses_li += '<li><a href=\"{url}\">{title}</a></li>'.format(url=url, title=title)\n",
" \n",
" # Count total number of documents\n",
" docs = glob.glob(os.path.join(page_dir, \"documents\", \"**\"), recursive=True)\n",
" n_docs = len([d for d in docs if os.path.isfile(d)])\n",
" \n",
" # Add button to download ZIP of all docs, if applicable\n",
" zipped_id = re.findall(regex_zipped_docs, page_metadata[\"id\"])\n",
" if zipped_id:\n",
" url = zipped_doc_pages[zipped_id[0]]\n",
" url = urllib.parse.quote_plus(url)\n",
" download_all_files = '[vc_button border_width=\"0\" link=\"url:{}|||\"]Download all {} docs from Google Drive[/vc_button]'.format(url, n_docs)\n",
" else: \n",
" download_all_files = \"\"\n",
" \n",
" # Create tr for each document\n",
" if n_docs == 0:\n",
" docs_table = \"<i>No responsive documents returned.</i>\"\n",
" else:\n",
" docs_table = \"\"\n",
" for i, d in enumerate([d for d in docs if os.path.isfile(d)]):\n",
"\n",
" row_color = \" background-color: #f2f2f2;\" if i%2 == 1 else \"\"\n",
"\n",
" url = D4J_root_upload_url + \\\n",
" d.split(\"dir_for_d4j/\")[-1].replace(\" \", \"%20\").replace(\"#\", \"\").replace(\"''\", \"\")\n",
"\n",
" filename = os.path.basename(d)\n",
" title = os.path.splitext(filename)[0]\n",
" extension = os.path.splitext(filename)[1].upper()[1:]\n",
"\n",
" size_bytes = os.path.getsize(d)\n",
" if size_bytes < 1e6:\n",
" size = \"{:.1f} KB\".format(size_bytes / 1e3)\n",
" elif size_bytes < 1e9:\n",
" size = \"{:.1f} MB\".format(size_bytes / 1e6)\n",
" elif size_bytes >= 1e9:\n",
" size = \"{:.1f} GB\".format(size_bytes / 1e9)\n",
"\n",
" docs_table += '''\n",
" <tr style=\"height: 24px;{row_color}\">\n",
" <td style=\"width: 5.40787%; height: 24px;\">\n",
" <span style=\"font-family: Poppins; font-size: 16px;\">\n",
" <strong><span style=\"color: #808080;\">{i_doc}</span></strong>\n",
" </span>\n",
" </td>\n",
" <td style=\"width: 40.8948%; height: 24px; font-weight: 200 !important;\">\n",
" <span style=\"color: #006cff; font-family: Vollkorn; font-size: 16px;\">\n",
" <a style=\"color: #006cff;\" href=\"{url}\">{title}</a>\n",
" </span>\n",
" </td>\n",
" <!--<td style=\"width: 26.8486%; font-weight: 200 !important; height: 24px;\">\n",
" <span style=\"color: #808080; font-size: 16px; font-family: Vollkorn;\">Description</span>\n",
" </td>-->\n",
" <td style=\"width: 13.4243%; font-weight: 200 !important; height: 24px;\">\n",
" <span style=\"color: #808080; font-size: 16px; font-family: Vollkorn;\">{extension}</span>\n",
" </td>\n",
" <td style=\"width: 13.4243%; font-weight: 200 !important; height: 24px;\">\n",
" <span style=\"color: #808080; font-size: 16px; font-family: Vollkorn;\">{size}</span>\n",
" </td>\n",
" </tr>\n",
" '''.format(row_color=row_color, i_doc=i+1, url=url, title=title,\n",
" extension=extension, size=size)\n",
"\n",
" docs_table = docs_table.replace(\"\\n\", \"\")\n",
" \n",
" docs_table = '''\n",
" <table style=\"height: 168px; width: 99.8992%; border: none;\">\n",
" <tbody>\n",
" <tr style=\"height: 24px;\">\n",
" <td style=\"width: 5.40787%; height: 24px;\">\n",
" <span style=\"font-size: 16px;\"><strong>\n",
" <span style=\"color: #808080;\">\n",
" <span style=\"font-family: Poppins;\">#</span>\n",
" </span>\n",
" </strong></span>\n",
" </td>\n",
" <td style=\"width: 40.8948%; height: 24px; font-weight: 200 !important;\">\n",
" <span style=\"font-size: 16px;\"><strong>\n",
" <span style=\"color: #808080;\">\n",
" <span style=\"font-family: Poppins;\">File Name</span>\n",
" </span>\n",
" </strong></span>\n",
" </td>\n",
" <!--<td style=\"width: 26.8486%; font-weight: 200 !important; height: 24px;\">\n",
" <span style=\"font-size: 16px;\"><strong>\n",
" <span style=\"color: #808080;\">\n",
" <span style=\"font-family: Poppins;\">Description</span>\n",
" </span>\n",
" </strong></span> \n",
" </td>-->\n",
" <td style=\"width: 13.4243%; font-weight: 200 !important; height: 24px;\">\n",
" <span style=\"font-size: 16px;\"><strong>\n",
" <span style=\"color: #808080;\">\n",
" <span style=\"font-family: Poppins;\">File Type</span>\n",
" </span>\n",
" </strong></span>\n",
" </td>\n",
" <td style=\"width: 13.4243%; font-weight: 200 !important; height: 24px;\">\n",
" <span style=\"font-size: 16px;\"><strong>\n",
" <span style=\"color: #808080;\">\n",
" <span style=\"font-family: Poppins;\">File Size</span>\n",
" </span>\n",
" </strong></span>\n",
" </td>\n",
" </tr>\n",
" {}\n",
" </tbody>\n",
" </table>\n",
" '''.format(docs_table)\n",
" \n",
" # Put it all together!\n",
" page_content = page_content_template.format(\n",
" description=description, agency=agency, category=category, \n",
" year=year, n_docs=n_docs, request_li=requests_li, \n",
" response_li=responses_li, download_all_files=download_all_files, \n",
" docs_table=docs_table)\n",
"\n",
" return page_content.replace(\"\\n\", \"\").replace(\"\\t\", \"\").replace('\"', r'\\\"')\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Use `.apply()` to write content for all pages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"all_page_content = all_pages_df.apply(lambda row: write_wordpress_page(row), axis=1)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>Title</th>\n",
" <th>Year</th>\n",
" <th>Department</th>\n",
" <th>Description</th>\n",
" <th>Files Transferred from Dropbox?</th>\n",
" <th>Numbered requests &amp; responses?</th>\n",
" <th>Directory</th>\n",
" <th>page_content</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>A01-lantel-OEM</td>\n",
" <td>Lantel at the Boston Office of Emergency Manag...</td>\n",
" <td>2019</td>\n",
" <td>Boston Office of Emergency Management</td>\n",
" <td>Since 2003, the City of Boston and surrounding...</td>\n",
" <td>Yes</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/A-privatecomp...</td>\n",
" <td>[vc_row][vc_column column_width_percent=\\\"100\\...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>A02-lantel-EOHHS</td>\n",
" <td>Lantel at the Executive Office of Health and H...</td>\n",
" <td>2019</td>\n",
" <td>Executive Office of Health and Human Services</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/A-privatecomp...</td>\n",
" <td>[vc_row][vc_column column_width_percent=\\\"100\\...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>A03-lantel-EOPSS</td>\n",
" <td>Lantel at the Executive Office of Public Safet...</td>\n",
" <td>2019</td>\n",
" <td>Executive Office of Public Safety and Security</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/A-privatecomp...</td>\n",
" <td>[vc_row][vc_column column_width_percent=\\\"100\\...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>A04-lantel-EOAF</td>\n",
" <td>Lantel at the Executive Office for Administrat...</td>\n",
" <td>2019</td>\n",
" <td>Executive Office for Administration and Finance</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/A-privatecomp...</td>\n",
" <td>[vc_row][vc_column column_width_percent=\\\"100\\...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>A05-lantel-MBTA</td>\n",
" <td>Lantel at the Massachusetts Bay Transit Author...</td>\n",
" <td>2019</td>\n",
" <td>Massachusetts Bay Transit Authority (MBTA)</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/A-privatecomp...</td>\n",
" <td>[vc_row][vc_column column_width_percent=\\\"100\\...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>59</td>\n",
" <td>D11-peabody</td>\n",
" <td>Facial Recognition at the Peabody PD</td>\n",
" <td>2019</td>\n",
" <td>Peabody PD</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>Yes</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/D-town&amp;city/D...</td>\n",
" <td>[vc_row][vc_column column_width_percent=\\\"100\\...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>60</td>\n",
" <td>E01-camera-locs</td>\n",
" <td>Location of Surveillance Cameras Across Boston</td>\n",
" <td>2019</td>\n",
" <td>City of Boston</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Waiting for response</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/E-boston/E01-...</td>\n",
" <td>[vc_row][vc_column column_width_percent=\\\"100\\...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>61</td>\n",
" <td>E02-videofootage</td>\n",
" <td>Video Surveillance Data Sharing in Boston</td>\n",
" <td>2019</td>\n",
" <td>City of Boston</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>Yes</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/E-boston/E02-...</td>\n",
" <td>[vc_row][vc_column column_width_percent=\\\"100\\...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>62</td>\n",
" <td>E03-camerastrategymeetings</td>\n",
" <td>Boston Security Camera Strategy Meetings</td>\n",
" <td>2019</td>\n",
" <td>City of Boston</td>\n",
" <td>In 2019, we filed a public records request wit...</td>\n",
" <td>Yes</td>\n",
" <td>Yes</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/E-boston/E03-...</td>\n",
" <td>[vc_row][vc_column column_width_percent=\\\"100\\...</td>\n",
" </tr>\n",
" <tr>\n",
" <td>63</td>\n",
" <td>F01-dea</td>\n",
" <td>Facial Recognition at the Drug Enforcement Adm...</td>\n",
" <td>2019</td>\n",
" <td>Drug Enforcement Administration (DEA)</td>\n",
" <td>In 2019, the ACLU of Massachusetts and the Nat...</td>\n",
" <td>Waiting for response</td>\n",
" <td>NaN</td>\n",
" <td>/Users/lchambers/D4J/dir_for_d4j/F-federal/F01...</td>\n",
" <td>[vc_row][vc_column column_width_percent=\\\"100\\...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>64 rows × 9 columns</p>\n",
"</div>"
],
"text/plain": [
" id \\\n",
"0 A01-lantel-OEM \n",
"1 A02-lantel-EOHHS \n",
"2 A03-lantel-EOPSS \n",
"3 A04-lantel-EOAF \n",
"4 A05-lantel-MBTA \n",
".. ... \n",
"59 D11-peabody \n",
"60 E01-camera-locs \n",
"61 E02-videofootage \n",
"62 E03-camerastrategymeetings \n",
"63 F01-dea \n",
"\n",
" Title Year \\\n",
"0 Lantel at the Boston Office of Emergency Manag... 2019 \n",
"1 Lantel at the Executive Office of Health and H... 2019 \n",
"2 Lantel at the Executive Office of Public Safet... 2019 \n",
"3 Lantel at the Executive Office for Administrat... 2019 \n",
"4 Lantel at the Massachusetts Bay Transit Author... 2019 \n",
".. ... ... \n",
"59 Facial Recognition at the Peabody PD 2019 \n",
"60 Location of Surveillance Cameras Across Boston 2019 \n",
"61 Video Surveillance Data Sharing in Boston 2019 \n",
"62 Boston Security Camera Strategy Meetings 2019 \n",
"63 Facial Recognition at the Drug Enforcement Adm... 2019 \n",
"\n",
" Department \\\n",
"0 Boston Office of Emergency Management \n",
"1 Executive Office of Health and Human Services \n",
"2 Executive Office of Public Safety and Security \n",
"3 Executive Office for Administration and Finance \n",
"4 Massachusetts Bay Transit Authority (MBTA) \n",
".. ... \n",
"59 Peabody PD \n",
"60 City of Boston \n",
"61 City of Boston \n",
"62 City of Boston \n",
"63 Drug Enforcement Administration (DEA) \n",
"\n",
" Description \\\n",
"0 Since 2003, the City of Boston and surrounding... \n",
"1 In 2019, we filed a public records request wit... \n",
"2 In 2019, we filed a public records request wit... \n",
"3 In 2019, we filed a public records request wit... \n",
"4 In 2019, we filed a public records request wit... \n",
".. ... \n",
"59 In 2019, we filed a public records request wit... \n",
"60 In 2019, we filed a public records request wit... \n",
"61 In 2019, we filed a public records request wit... \n",
"62 In 2019, we filed a public records request wit... \n",
"63 In 2019, the ACLU of Massachusetts and the Nat... \n",
"\n",
" Files Transferred from Dropbox? Numbered requests & responses? \\\n",
"0 Yes NaN \n",
"1 Yes NaN \n",
"2 Yes NaN \n",
"3 Yes NaN \n",
"4 Yes NaN \n",
".. ... ... \n",
"59 Yes Yes \n",
"60 Waiting for response NaN \n",
"61 Yes Yes \n",
"62 Yes Yes \n",
"63 Waiting for response NaN \n",
"\n",
" Directory \\\n",
"0 /Users/lchambers/D4J/dir_for_d4j/A-privatecomp... \n",
"1 /Users/lchambers/D4J/dir_for_d4j/A-privatecomp... \n",
"2 /Users/lchambers/D4J/dir_for_d4j/A-privatecomp... \n",
"3 /Users/lchambers/D4J/dir_for_d4j/A-privatecomp... \n",
"4 /Users/lchambers/D4J/dir_for_d4j/A-privatecomp... \n",
".. ... \n",
"59 /Users/lchambers/D4J/dir_for_d4j/D-town&city/D... \n",
"60 /Users/lchambers/D4J/dir_for_d4j/E-boston/E01-... \n",
"61 /Users/lchambers/D4J/dir_for_d4j/E-boston/E02-... \n",
"62 /Users/lchambers/D4J/dir_for_d4j/E-boston/E03-... \n",
"63 /Users/lchambers/D4J/dir_for_d4j/F-federal/F01... \n",
"\n",
" page_content \n",
"0 [vc_row][vc_column column_width_percent=\\\"100\\... \n",
"1 [vc_row][vc_column column_width_percent=\\\"100\\... \n",
"2 [vc_row][vc_column column_width_percent=\\\"100\\... \n",
"3 [vc_row][vc_column column_width_percent=\\\"100\\... \n",
"4 [vc_row][vc_column column_width_percent=\\\"100\\... \n",
".. ... \n",
"59 [vc_row][vc_column column_width_percent=\\\"100\\... \n",
"60 [vc_row][vc_column column_width_percent=\\\"100\\... \n",
"61 [vc_row][vc_column column_width_percent=\\\"100\\... \n",
"62 [vc_row][vc_column column_width_percent=\\\"100\\... \n",
"63 [vc_row][vc_column column_width_percent=\\\"100\\... \n",
"\n",
"[64 rows x 9 columns]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Add as new column to existing datarame\n",
"all_pages_df[\"page_content\"] = all_page_content\n",
"all_pages_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Write PHP snippets for all pages"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To publish the page content written above, we can use Wordpress' `wp_insert_post()` function. Similarly, we'll use Python to customize a template PHP function call for each page.\n",
"\n",
"We're going to write the following PHP command for each page:\n",
"\n",
"```PHP\n",
"// Add page after wp_loaded\n",
"add_action( 'wp_loaded', function () {{ \n",
"\t\n",
"\t$post_content = \"{content}\";\n",
" \n",
" $title = \"{title}\";\n",
"\n",
"\t$new_page = array(\n",
"\t\t\t'comment_status' => 'close',\n",
"\t\t\t'ping_status' => 'close',\n",
"\t\t\t'post_title' => $title,\n",
"\t\t\t'post_name' => strtolower(str_replace(' ', '-', trim( $title ))),\n",
"\t\t\t'post_status' => 'publish',\n",
" 'post_password' => 'password', // Don't use this!\n",
"\t\t\t'post_content' => $post_content,\n",
"\t\t\t'post_type' => 'page',\n",
"\t\t\t'post_parent' => '{parent_id}',\n",
"\t\t\t);\n",
"\n",
" // Check if the page already exists\n",
"\t$check_page_exist = get_page_by_title( $title, 'OBJECT', 'page');\n",
"\n",
"\tif(empty($check_page_exist)) {{\n",
"\t\t$page_id = wp_insert_post( $new_page );\n",
"\n",
"\t\t// Add custom-auto category\n",
"\t\twp_set_object_terms($page_id, array(\"custom-auto\"), \"page_category\");\n",
"\t}} else {{\n",
"\t\terror_log($title . \" page already exists; not inserting new post\");\n",
"\t}}\n",
"}});\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"php_snippet_template = '''\n",
"// Add page after wp_loaded\n",
"add_action( 'wp_loaded', function () {{ \n",
"\t\n",
"\t$post_content = \"{content}\";\n",
"\n",
"\t$title = \"{title}\";\n",
"\n",
"\t$new_page = array(\n",
"\t\t\t'comment_status' => 'close',\n",
"\t\t\t'ping_status' => 'close',\n",
"\t\t\t'post_title' => $title,\n",
"\t\t\t'post_name' => strtolower(str_replace(' ', '-', trim( $title ))),\n",
"\t\t\t'post_status' => 'publish',\n",
" 'post_password' => 'password', // Don't use this!\n",
"\t\t\t'post_content' => $post_content,\n",
"\t\t\t'post_type' => 'page',\n",
"\t\t\t'post_parent' => '{parent_id}',\n",
"\t\t\t);\n",
"\n",
"\t// Check if the page already exists\n",
"\t$check_page_exist = get_page_by_title( $title, 'OBJECT', 'page');\n",
"\n",
"\tif(empty($check_page_exist)) {{\n",
"\t\t$page_id = wp_insert_post( $new_page );\n",
"\n",
"\t\t// Add custom-auto category\n",
"\t\twp_set_object_terms($page_id, array(\"custom-auto\"), \"page_category\");\n",
"\t}} else {{\n",
"\t\terror_log($title . \" page already exists; not inserting new post\");\n",
"\t}}\n",
"}});\n",
"'''"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Map parent page IDs\n",
"We wanted certain pages to be child pages of other pages, so we mapped the page ID to the alphabetic code (A-F) here."
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"# Define IDs of parent pages (dummy IDs)\n",
"parent_id_key = {\n",
" 'A-privatecompanies': 1,\n",
" 'A11-clearview': 2,\n",
" 'B-schools': 3,\n",
" 'C-stateagencies': 4,\n",
" 'D-town&city': 5,\n",
" 'E-boston': 6,\n",
" 'F-federal': 7 \n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Function to turn spreadsheet row into PHP command"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"def create_php_snippet(row):\n",
" # Determine alphabetic code for parent page ID from directory path\n",
" parent_dir = os.path.dirname(os.path.dirname(row[\"Directory\"]))\n",
" parent_dir = os.path.basename(parent_dir)\n",
" parent_id = parent_id_key[parent_dir]\n",
" \n",
" # Fill in PHP command with content, title, and parent ID for each page\n",
" php_snippet = php_snippet_template.format(\n",
" content=row[\"page_content\"],\n",
" title=row[\"Title\"],\n",
" parent_id=parent_id\n",
" )\n",
" \n",
" return php_snippet"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Use `.apply()` to write PHP command for all pages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"all_snippets = all_pages_df.apply(lambda row: create_php_snippet(row), axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Write all of the PHP commands out to a txt file"
]
},
{
"cell_type": "code",
"execution_count": 87,
"metadata": {},
"outputs": [],
"source": [
"# Write them!!!\n",
"with open(\"insert_all_pages_php_snippet.txt\", \"w+\") as fo:\n",
" for s in all_snippets:\n",
" fo.write(s)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": true,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": true,
"toc_position": {
"height": "calc(100% - 180px)",
"left": "10px",
"top": "150px",
"width": "272.391px"
},
"toc_section_display": true,
"toc_window_display": true
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment