Create a gist now

Instantly share code, notes, and snippets.

Embed
What would you like to do?
My notes of the NYCC Tech Committee meeting on the Algorithmic Transparency Bill, 16-96

These notes may have errors and omissions. I couldn’t get the names of a lot of the speakers and there are some places where I was thinking or distracted. I make no claims as to the completeness of this information

Algorithmic transparency legislation hearing 10/16/17

James Vaca, Chair of NYCC committee on technology

16-96 2017 Measures of transparency when NYC uses algorithms to impose penalties, police persons

  • Requires publication of source code and querying systems with sample data

  • If left unchecked, algorithms can have negative repercussions
  • Algorithms are a way of encoding assumptions
  • the very data behind them can be biased
  • Despite importance to governamce and their problems, they tend to be hidden from public view
  • Unclear what assumptions they are based upon and what
  • Hence, a lack of transparency
  • What is considered to be most efficient?
  • More dificult for members of the city council to advocate for their citizens if variables are tied up in algorithms
  • When there appears to be inequities or shortages in services, we should find out why
  • Vaca seems to be worked up that his district does not have as much police manpower as other districts
  • The ability to make government accountable is obscured by algorithms and democracy is undermined
  • To ensure that city agencies, when utilizing cutting-edge tools, are accountable
  • First city, first legislative body to undertake the issue in the US

Sunderland – City enterprise software IT, joined by Craig Kamble mayor’s office of data & anaytics

  • City services heavily rely on computer programs
  • Notify NYC app: in-source team
  • Several positions that the city may not hire on their own
  • Not making policy, making apps.
  • 16-96 presents significant operational concerns

security concerns and problems

  • Roadmap for bad actors to exploit and abuse
  • meaningful risk to divulge software
  • Scope is all-encompassing, intentionally targets all programs
  • Releasing proprietary code, releasing old source code
  • Testing is not possible (?)
  • IT departments would have to create a new body of software

Unintended consequences

  • Deliver a deluge of information, most of it unrelated to most interesting city services
  • Users could fabricate data to get the responses they want
  • Code is a small part of decision-making
  • Algorithms supplement, rather than supersede decision making progress

Mayor’s office

  • Open data includes 3 recent projects. Reviewing a backlog
  • Motivation to create project library closely aligned with legislation
  • Mayor’s office vision aligns somewhat
  • Much legislation about transparency, but decisions are cloaked in opaque algorithms
  • MODA works on specific focus areas: agency projects on priority of the mayor, or legislative mandate
  • MODA’s goal is to not own any analytics projects long-term, but for specific projects

Questions

  • RAND formula: always opaque, always used, public does not have a right to know
  • Don’t know if it’s update in 20 years
  • No comprehensive list of data analytics teams in NYC
  • Why doesn’t the mayor know about who’s using data and analytics
  • Is there no oversight over which agencies employ advanced data analytics
  • At what level is there an understanding of other agencies’ use of data and analytics
  • Never been approached with an agency seeking to implement more transparency
  • Open source: thoroughly vetted
  • Most city systems would divulge system architechture details
  • We don’t believe in transparency because we’re not doing anything
  • It was a new topic for the Enterprise IT (transparency)
  • HRA employs algorithms to detect benefits fraud, no level of human review
  • Human rights commission: studying decision-making (not algorithms)
  • EIT personally does not know about information rendered without human input
  • How does someone get an apartment in public housing? Strictly by computer, with one appeal
  • What are the bases of those decisions?
  • Feedback loops:
  • Officers stationed in places which have lots of nuisnace calls/arrests which generate nuisance arrests
  • Agencies are not watching their own algorithms
  • Open to creation of commission-based body to oversee the use of algorithms?

My own notes

  • EIT architect seems to be work-shy or worried about mass-reimplimentation of software, renegotiation of contracts
  • Mayor’s office analyst seems young & that his boss probably told him what to say, idk
  • If NYC funds are paying for NYC software, well, why isn’t that open source/transparent?

Witnesses in testimony

(name unknown)

  • legislation reinforces the core of a responsible, equitable government
  • worried about tech companies entering the public spheres
  • we are outsourcing our government to the unknown

Sheena richardson NYCLU

  • CLU stuff. Much in favor, full testimony submitted
  • (had to answer a couple texts, incomplete notes)

Research fellow at Cornell tech

  • multi-year NSF funding in decision making in algorithmic systems
  • bold proposal, exciting
  • does not reach critical threshold in research
  • privacy implications
  • no guarantees of accuracy or fitness of use
  • any proprietary claims no matter how broad, will fall short of the law
  • black box: administratively burdensome, testing usually takes thousands of queries

Rachel – Brennan center for justice

  • restoring proper flow of information from government to people
  • filed FOIA to NYPD over predictive policing technologies
  • NYPD expected to spend 45 million on predictive policing software over 5 years
  • 0 records returnd from FOIA, followed suit
  • no disclosure of source code
  • had a hearing in August, and now is before a judge
  • lawsuit filed in July, suit in December, hearing in January
  • striking, the number of exemptions that were brought
  • the NYPD strategy was to wait for a lawsuit to force documents

Alex Kroff

  • no certifications required to create algorithms for city government, but cosmetology licenses required. Seems backwards.

Bronx defenders

  • want to bring to attention of the public a specific algorithm and to make sure that hey are just and fair:
  • NYC developing with private contractor to predict defendent’s likelihood in appearing in court
  • would be deployed in bail hearing
  • would increase pre-trial detention in NYC
  • can’t predict, but attempts to do so would run aground of bail reform
  • concerned about racial justice aspects of algorithms, exacerbate existing racial disparities
  • one reccommendations: transparency & accountability are good first steps
  • before city algorithms are applied (in courts) the city would be required to perform equity assessments

Brooklyn defenders

  • bail reform & algorithmic transparency, ore algorithms, more policing, more CRJ
  • multinational surety companies have popped up to take advantage of overpolicing
  • RAI’s: discriminatory masures like priors, homelessness, education, etc.
  • investigation found accuracy only slightly more accurate than a coin flip
  • RAI’s bypass an individual’s right to due process
  • underlying data is not transparent, even though engineers say that they are

Security concerns

  • constitutional protections vs security risks: const. protections must take precedence
  • “post-equifax world”
  • people are not consenting to their data being used for overpolicing

Robert Wallace - Research scientist - div of epidemiology

  • algorithms for model systems
  • Rand models that nobody can see & damage data
  • response time is a good index for ambulance
  • for a fire you ‘have to build a hospital around apatient’
  • empirical damage measures must be used to determine fire policy
  • you will target tenements for reduction of fire services in high fire neighborhoods
  • models of rand’s quality, you wouldn’t use for fish! But they are for humans
  • rand hasn’t changed since 1970’s
  • cuts to city services based on models with questionable validity
  • e.g. city island
  • with global warming, more hurricanes, still using ancient models

Selene - Policy director for Tech NYC

  • nonprofit trade group, increased engagement from tech companies in politics (??)
  • we believe in transparency, treating residents fairly
  • imposing protocols to publish sensitive source code is bad
  • chilling effect on companies publishing code online
  • protecting confidential data
  • (basically against legislation – so why are you “for transparency” again?)

Josh - Staff attorney for Decarceration, legal aid society

  • brunt of new algorithms have been shoulderd by constituents in
  • may result in wrongful convictions
  • hidden from public view
  • 6 areas where algorithms are used in criminal justice
  • bail, predicitve policing, DNA, family court, parole proceedings, sex offender registration
  • 2 algorithms in bail
  • 17 million dollar prgram with 3000 spots, determined by algorithm

Julie - DNA unit of egal aid society

  • dna evidence & challenges dna certification software in the courtroom
  • fst: probabilitic gene interpreting algorithm
  • no idea how fst calculations are formed, no way to verify soundness of conclusions
  • people went to prison, lost their child from fst
  • fst’s source code contained an error
  • hope is that the entire source code
  • strmix - 2 verified errors
  • different algorithms will get different answers in the same case
  • only way to verify that questionable forensic software is not used is to go open source

Center for information technology policy

  • NSF funded research institute
  • bill requires significant changes
  • algo transparency cannot be achieved without data transparency
  • results must be interpretable
  • making source code available is a good 1st step, but must be readable and complete
  • The same algorithm may exhibit two different results with two different training sets
  • Scoring methods for schools and students are different
  • propose following interpretation of transparency:
  • agents must make available details of data collection, summaries of statistical collections of the data
  • privacy-preserving synthetic data
  • 1st mention of EU legislation

Charlie Moffett - NYU graduate student

  • conducted research on behalf of data officers in SF, but also countrywide
  • most research echo what has already been said here today
  • extra recommendations to that committee:
  • regarding publishing source code, often most folks won’t understand it
  • understanding outcomes, not just the process
  • algorithms must be clear about confidence and data quality
  • addressing explainability
  • question the use of an algorithm at all, if they can’t be explained to the public
  • burden should fall on the vendor
  • reactive auditing: not good
  • leveraging this position when contracting with vendors
  • plan in place for when algorithms go wrong or if mistakes go wrong, what is the redress?
  • transparency must be understandable, and should be inclusive

William Banfield - Tech Worker

  • value of open source
  • 2013 company had incorrect implementation of RAFT protocol. Users downloaded, provided fixes, publicly
  • private company’s software could have resulted in data loss
  • regarding security: not a practical way to enforce it
  • openSSL is a public project, is the standard of TLS, government
  • keeping things a secret to improve security is silly.

Sumana - Tech Worker / Recurse Alum

  • Speaking as consultant programmer, citizen
  • 1: tech NYC does not speak for me
  • open source and transparency are the way to better security
  • if there are businesses that make money from citizens data, we need to hold them accountable
  • phrasings of analytics, data, etc require more scrutiny in the context in the law
  • trade secrets, proprietary code: must fix procurement process.
  • code you write with taxpayer money should be made public
  • food report cards: easily understandable
  • if they want to understand deeper, resources are available

Bert - Google, speaking as a citizen

  • objections raised are about existing programs
  • not as many concerns about security models or engineers rehashing old code
  • new development can be held to a high standard of transparency by default
  • centralizing of information and review
  • people of new york should be able to obtain a list of all programs that police persons, penalize, etc

Alex Rich - Data scientist at NYU

  • on bail decisions:
  • open source can be quite understandable
  • new systems can allow people to understand how decisions are made for people

Other thoughts

  • appeal is based on pleading, but you don’t have that kind of information
  • people need to know what government decisions are made and on what basis
  • introducing the need for transparency ends up exposing tons of problems, inefficiencies and problems, as a side-efect
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment