These notes may have errors and omissions. I couldn’t get the names of a lot of the speakers and there are some places where I was thinking or distracted. I make no claims as to the completeness of this information
James Vaca, Chair of NYCC committee on technology
- Requires publication of source code and querying systems with sample data
- If left unchecked, algorithms can have negative repercussions
- Algorithms are a way of encoding assumptions
- the very data behind them can be biased
- Despite importance to governamce and their problems, they tend to be hidden from public view
- Unclear what assumptions they are based upon and what
- Hence, a lack of transparency
- What is considered to be most efficient?
- More dificult for members of the city council to advocate for their citizens if variables are tied up in algorithms
- When there appears to be inequities or shortages in services, we should find out why
- Vaca seems to be worked up that his district does not have as much police manpower as other districts
- The ability to make government accountable is obscured by algorithms and democracy is undermined
- To ensure that city agencies, when utilizing cutting-edge tools, are accountable
- First city, first legislative body to undertake the issue in the US
- City services heavily rely on computer programs
- Notify NYC app: in-source team
- Several positions that the city may not hire on their own
- Not making policy, making apps.
- 16-96 presents significant operational concerns
- Roadmap for bad actors to exploit and abuse
- meaningful risk to divulge software
- Scope is all-encompassing, intentionally targets all programs
- Releasing proprietary code, releasing old source code
- Testing is not possible (?)
- IT departments would have to create a new body of software
- Deliver a deluge of information, most of it unrelated to most interesting city services
- Users could fabricate data to get the responses they want
- Code is a small part of decision-making
- Algorithms supplement, rather than supersede decision making progress
- Open data includes 3 recent projects. Reviewing a backlog
- Motivation to create project library closely aligned with legislation
- Mayor’s office vision aligns somewhat
- Much legislation about transparency, but decisions are cloaked in opaque algorithms
- MODA works on specific focus areas: agency projects on priority of the mayor, or legislative mandate
- MODA’s goal is to not own any analytics projects long-term, but for specific projects
- RAND formula: always opaque, always used, public does not have a right to know
- Don’t know if it’s update in 20 years
- No comprehensive list of data analytics teams in NYC
- Why doesn’t the mayor know about who’s using data and analytics
- Is there no oversight over which agencies employ advanced data analytics
- At what level is there an understanding of other agencies’ use of data and analytics
- Never been approached with an agency seeking to implement more transparency
- Open source: thoroughly vetted
- Most city systems would divulge system architechture details
- We don’t believe in transparency because we’re not doing anything
- It was a new topic for the Enterprise IT (transparency)
- HRA employs algorithms to detect benefits fraud, no level of human review
- Human rights commission: studying decision-making (not algorithms)
- EIT personally does not know about information rendered without human input
- How does someone get an apartment in public housing? Strictly by computer, with one appeal
- What are the bases of those decisions?
- Feedback loops:
- Officers stationed in places which have lots of nuisnace calls/arrests which generate nuisance arrests
- Agencies are not watching their own algorithms
- Open to creation of commission-based body to oversee the use of algorithms?
- EIT architect seems to be work-shy or worried about mass-reimplimentation of software, renegotiation of contracts
- Mayor’s office analyst seems young & that his boss probably told him what to say, idk
- If NYC funds are paying for NYC software, well, why isn’t that open source/transparent?
- legislation reinforces the core of a responsible, equitable government
- worried about tech companies entering the public spheres
- we are outsourcing our government to the unknown
- CLU stuff. Much in favor, full testimony submitted
- (had to answer a couple texts, incomplete notes)
- multi-year NSF funding in decision making in algorithmic systems
- bold proposal, exciting
- does not reach critical threshold in research
- privacy implications
- no guarantees of accuracy or fitness of use
- any proprietary claims no matter how broad, will fall short of the law
- black box: administratively burdensome, testing usually takes thousands of queries
- restoring proper flow of information from government to people
- filed FOIA to NYPD over predictive policing technologies
- NYPD expected to spend 45 million on predictive policing software over 5 years
- 0 records returnd from FOIA, followed suit
- no disclosure of source code
- had a hearing in August, and now is before a judge
- lawsuit filed in July, suit in December, hearing in January
- striking, the number of exemptions that were brought
- the NYPD strategy was to wait for a lawsuit to force documents
- no certifications required to create algorithms for city government, but cosmetology licenses required. Seems backwards.
- want to bring to attention of the public a specific algorithm and to make sure that hey are just and fair:
- NYC developing with private contractor to predict defendent’s likelihood in appearing in court
- would be deployed in bail hearing
- would increase pre-trial detention in NYC
- can’t predict, but attempts to do so would run aground of bail reform
- concerned about racial justice aspects of algorithms, exacerbate existing racial disparities
- one reccommendations: transparency & accountability are good first steps
- before city algorithms are applied (in courts) the city would be required to perform equity assessments
- bail reform & algorithmic transparency, ore algorithms, more policing, more CRJ
- multinational surety companies have popped up to take advantage of overpolicing
- RAI’s: discriminatory masures like priors, homelessness, education, etc.
- investigation found accuracy only slightly more accurate than a coin flip
- RAI’s bypass an individual’s right to due process
- underlying data is not transparent, even though engineers say that they are
- constitutional protections vs security risks: const. protections must take precedence
- “post-equifax world”
- people are not consenting to their data being used for overpolicing
- algorithms for model systems
- Rand models that nobody can see & damage data
- response time is a good index for ambulance
- for a fire you ‘have to build a hospital around apatient’
- empirical damage measures must be used to determine fire policy
- you will target tenements for reduction of fire services in high fire neighborhoods
- models of rand’s quality, you wouldn’t use for fish! But they are for humans
- rand hasn’t changed since 1970’s
- cuts to city services based on models with questionable validity
- e.g. city island
- with global warming, more hurricanes, still using ancient models
- nonprofit trade group, increased engagement from tech companies in politics (??)
- we believe in transparency, treating residents fairly
- imposing protocols to publish sensitive source code is bad
- chilling effect on companies publishing code online
- protecting confidential data
- (basically against legislation – so why are you “for transparency” again?)
- brunt of new algorithms have been shoulderd by constituents in
- may result in wrongful convictions
- hidden from public view
- 6 areas where algorithms are used in criminal justice
- bail, predicitve policing, DNA, family court, parole proceedings, sex offender registration
- 2 algorithms in bail
- 17 million dollar prgram with 3000 spots, determined by algorithm
- dna evidence & challenges dna certification software in the courtroom
- fst: probabilitic gene interpreting algorithm
- no idea how fst calculations are formed, no way to verify soundness of conclusions
- people went to prison, lost their child from fst
- fst’s source code contained an error
- hope is that the entire source code
- strmix - 2 verified errors
- different algorithms will get different answers in the same case
- only way to verify that questionable forensic software is not used is to go open source
- NSF funded research institute
- bill requires significant changes
- algo transparency cannot be achieved without data transparency
- results must be interpretable
- making source code available is a good 1st step, but must be readable and complete
- The same algorithm may exhibit two different results with two different training sets
- Scoring methods for schools and students are different
- propose following interpretation of transparency:
- agents must make available details of data collection, summaries of statistical collections of the data
- privacy-preserving synthetic data
- 1st mention of EU legislation
- conducted research on behalf of data officers in SF, but also countrywide
- most research echo what has already been said here today
- extra recommendations to that committee:
- regarding publishing source code, often most folks won’t understand it
- understanding outcomes, not just the process
- algorithms must be clear about confidence and data quality
- addressing explainability
- question the use of an algorithm at all, if they can’t be explained to the public
- burden should fall on the vendor
- reactive auditing: not good
- leveraging this position when contracting with vendors
- plan in place for when algorithms go wrong or if mistakes go wrong, what is the redress?
- transparency must be understandable, and should be inclusive
- value of open source
- 2013 company had incorrect implementation of RAFT protocol. Users downloaded, provided fixes, publicly
- private company’s software could have resulted in data loss
- regarding security: not a practical way to enforce it
- openSSL is a public project, is the standard of TLS, government
- keeping things a secret to improve security is silly.
- Speaking as consultant programmer, citizen
- 1: tech NYC does not speak for me
- open source and transparency are the way to better security
- if there are businesses that make money from citizens data, we need to hold them accountable
- phrasings of analytics, data, etc require more scrutiny in the context in the law
- trade secrets, proprietary code: must fix procurement process.
- code you write with taxpayer money should be made public
- food report cards: easily understandable
- if they want to understand deeper, resources are available
- objections raised are about existing programs
- not as many concerns about security models or engineers rehashing old code
- new development can be held to a high standard of transparency by default
- centralizing of information and review
- people of new york should be able to obtain a list of all programs that police persons, penalize, etc
- on bail decisions:
- open source can be quite understandable
- new systems can allow people to understand how decisions are made for people
- appeal is based on pleading, but you don’t have that kind of information
- people need to know what government decisions are made and on what basis
- introducing the need for transparency ends up exposing tons of problems, inefficiencies and problems, as a side-efect