Notes from Coed:Ethics conference, London, 13 July 2018
- Imam Salem bin Ali Jaber preached in Yemen against Islamic extremism; guest at a wedding of relative; hit by a US Hellfire missile fired from a drone. Relative Faisal made contact with Cori, went to Washington DC. No explanation ever made by government (although compensation was paid).
- Decision-making process behind the attack not known exactly. But there is significant evidence that such attacks serve to further radicalise people; attack results in ~3x more new recruits than extremists killed by attack.
- Most drone attacks are not on named individuals, but rather "signature strikes" — a euphemism for killing people the military doesn't even know, but who match a certain behavioural pattern (perhaps based on metadata — Hayden: "we kill people based on metadata"
- Skynet (known through Snowden relevations): use machine learning to try to find couriers. Match "pattern of life", "social network", "travel behaviour". Prob Ahmed Zaidan was the top result, but he was actually an Al Jazeera journalist; his job was interviewing militants!
- Trump further loosened rules under which drone attack targets could be selected. His response when he heard that drone waited for wife and children of target to move out of the frame before firing: "why did you wait?" Estimated 6,000 civilian casualties in Iraq and Syria last year alone.
- People use algorithms to kill people. (Placing human sources in extremist groups is too slow and expensive.)
- Google bid for Pentagon "Project Maven": "algorithmic warfare cross-functional team". Use AI to scan drone feeds to isolate targets for attack. Resistance from Google employees, leaks to press, open letter signed by ~3,000. Eventually Google let the contract lapse and published new guidelines for use of AI. Lesson: software developers collectively have a lot of power
- Amazon facial "Rekognition" product ("with a nice Stasi K")
- AI now also used for hiring and firing, bail or parole, mortgages, jobs, detention at the border, etc. Real harm to real people.
- Can and should ask questions of employers. Negotiate nature of work, not just salary. Combine technical imagination with moral imagination.
- Question: shouldn't the best tech companies take on these contracts, since otherwise worse companies will do it (as most of the tech is open source), and they might result in even worse predictions? Answer: don't think that way; rather think about, when you build some tool, who may use it (eg. Assad's Syria or other authoritarian regimes)?
- Google's principles are very general; the beginning of a conversation, not the end.
- Whilstleblowing: "sometimes these things are risky until they are not", until public opinion shifts
- Not a lawyer, but interested in understanding the civic system. Triangle of "the law"; legislators and lawyers; and citizens. Parallel in data science: law <--> data/methods/models/code; legislators <--> data scientists; and citizens <--> "data citizens" (people impacted by the results of data science).
- In civic life, citizens have mechanisms for influencing the decisions of legislators (voting, campaigning, lobbying, protest, bringing forward lawsuits, etc). Not perfect but at least they exist. In data science, similar mechanisms do not currently seem to exist.
- Cathy McNeil: sentencing algorithms in particular are, in effect, a kind of law. However, they are intransparent, and there is often no recourse to decisions by those affected by them
- Automation codifies past mistakes. Just because a machine has done something, that doesn't mean that it is automatically neutral or fair.
- Whether or not they like (or perceive it), data scientists are making ethical decisions all the time on their job.
- Data Ethics Canvas (open data institute): help identifies potential ethical issues associated with a data project; encourages understanding, debate, and help identifying steps to improve
- PLoS Computational Biology journal: "data are people"; makes central the "difficulty of dissociating data from specific individuals"
- Big technology companies (e.g. Accenture, Microsoft) have been developing ethical auditing toolkits. However, still means nothing if they are not adopted.
- Recent social science research: it only takes 25% of people in a group to completely change the cultural norms and debate in the group
- Q: revealing the algorithms/criteria for making a decision (e.g. granting credit) may enable people to game/undermine the system. How to balance with desire for transparency? A: we put up signs to warn people about speed cameras. Goal is to change people's behaviour, not to maximise fines collected from the camera.
- Anecdotal evidence: increasing number of software developers are turning down lucrative job offers from employers with whom they do not ethically agree.
- Despite best intentions, can have unintended consequences. e.g. City of Boston pothole survey smartphone app: tended to report potholes more in the affluent areas of the city, as affluent people are more likely to use the app.
- Example project: for a foodbank in north of England, make predictive model to identify foodbank users who are most likely to be in need of further support (counselling). Understanding who will become dependent on the foodbank. However, don't want it to be used to decide who can receive food, or to place conditions on receiving support.
- Reason about consequences: what if we wrongly predict which users are in need of extra support? What if model is implemented without a final human decision? What if model is used to ration foodbank support? Currently the tool is not used in these ways, but it's possible that the tool's use may change in the future (e.g. after management change).
- Data inputs: details about person, reasons for referral, referral pattern. Noticed that dataset is biased towards male single people; women with children tend to be referred to other channels of support?
- Found that country of birth was a strong predictor of need of support. Asked: what would the Daily Mail headline look like if they heard of this? Hence, policy question: should this variable be used, as it is sensitive? On further examination, found the country of birth field was sparsely populated, and dominated by a few outliers. Thus, decided to exclude this variable. This decision-making required discussions with the charity about the approach.
- Used random forest approach so as to provide explainability of results.
- What does communication with staff and with users look like? How do you explain a message to someone saying that we think they need further support?
- Anecdote: an insurance company had taken care to exclude ethnicity from their predictive model. However, they later found that the model had picked up on spelling of name, which turns out to be correlated with ethnicity, and so the sensitive variable sneaked in after all.
- Recommendations: embed discussions of ethical questions in all stages of the project; involve a diverse range of people to get a fuller picture of context and risks; explainability is important (we need translators); try to capture unintended consequences as well as intensions.
- MIT "data nutrition labels": http://datanutrition.media.mit.edu/
- Question: how do we decide whether a technological solution actually confers enough benefit that it outweighs the potential negative consequences? If a system is working alright without using data analysis (just based on human workflows), might introducing data analysis actually make the situation worse overall?
- Personal anecdote of shoplifting as a teenager, and feeling of guilt afterwards. Why do good people sometimes make bad decisions? 1950s psychological experiments inform our knowledge in this area (although they wouldn't get ethics board approval nowadays!).
- For example on conformity: if all the people around you give the wrong answer to a simple visual test, would a test subject still stick with their (correct) answer? 75% of participants conformed at least once, even though correct answer is quite unambiguous. We conform because we don't want to be different. The more ambiguous the decision is, the higher the conformance rate.
- Another phenomenon: obedience. We alter our behaviour to obey someone perceived to be more important/powerful than us. Milgram experiment: After observing "just following orders" of concentration camp guards in Nuremberg Trials, designed "electric shock" experiment, in which a "teacher" (the test subject) was instructed to give a simulated shock to a "student" (an actor) when they made a mistake, gradually increasing intensity. 65% of test subjects gave shocks that they were told to be potentially fatal. Initially tested in Germany, but replicated around the world.
- Conclusion: sometimes we can't rely on our role models or superiors to tell us how to behave. Awareness of our tendency to conformity and obedience can help us overcome it. Ask: what matters to you? Then speak up to authority, which requires courage. But opinions change one person at a time.
- Courage is persistence in the face of fear. Research shows increase in self-worth and happiness after being courageous.
- Question: conformity isn't always bad; in fact, basic respect for other people is a form of conformity. How do we tell the difference to problematic conformity? Answer: listen to emotions, which signal that something is wrong.
- Amy Edmondson's research on psychological safety -- e.g. nurses did not speak up about surgeon forgetting objects inside the body of a patient during an operation, because the hierarchical power structure did not let them
- Question: how do you establish an environment with psychological safety, so that people feel safe speaking up? Answer: teach people about the importance of psychological safety; making mistakes is important for learning, and so people need safety to make mistakes; speaking to people 1:1; make it a shared journey; ask questions rather than telling people what to do.
- Why do we care about ethics? Because we want the world to be a better place. Or perhaps rather: Because we want to feel like (and be seen as) good people.
- Many of us create technology with positive impact (e.g. medical/educational technology, shopping/bringing goods to consumers efficiently, autonomous vehicles: potential to reduce road accident fatalities)
- Lesson from Google Project Maven (see Cori's talk): as a software engineer, "you have 1/12th of the power to cancel a collaboration with the military" (12 resignations were enough to change management's mind)
- What things are there that feel good, but don't actually make the world any better. And conversely, what things don't necessarily feel good, but actually make a real difference?
- Scope insensitivity: How much would you be willing to donate to stop n birds from being killed in an oil spill? Whether n=2,000 or n=20,000 or n=200,000, amount donated is the same. Simply cannot visualise the number of birds.
- Effective altruism movement — trying to be more quantitative and rational about philanthropy. Organisations: Giving What We Can and GiveWell. For example, very effective reduction of Malaria in Afrika 2000-2015, mainly by distributing anti-Malarial bednets.
- Qaly approach to valuing quality of human life. In UK, NHS is prepared to spend up to £20k per Qaly (extending a patient's life) — used to determine which treatments are considered cost-effective. But many treatments are much cheaper: e.g. antiretroviral therapy for HIV: about 2 Qaly per £1,000. And distribution of condoms to people who need them, to prevent HIV infections: 20 Qaly per £1,000 — extremely cost-effective.
- Organisation "No Lean Season" aimed at reducing seasonal poverty in Bangladesh. Gives low-interest loan to rural workers to buy a bus ticket to a big city where they can do rickshaw-pulling work, send money back home, and return for the season of farming work. Has been highly effective at lifting families out of poverty.
- Exercise: "Who can have an impact on your goal? How can they help? What can you do to affect their behaviour?" Then find the set of things that maximise the effectiveness.
- Heroic responsibility: "You've got to get the job done no matter what" (from Harry Potter fan fiction: Harry Potter and the methods of rationality)
- Comment: The Effective Altruism community sometimes has a tendency to miss their own biases and trying to play God. Answer: They are not perfect, but at least they are trying, and let's not forget the positive impact they have had.
- Question: Is it fair that software engineers are capturing so much wealth? Answer: we don't really have any influence about how much software engineers get paid, but we do have influence on how that money is used!
- https://projectsbyif.com/ work with organisations who work with data, work to build trust.
- Designers and developers collaborate closely and the difference is blurring; and we are distinguishing design and development less while designing services.
- "Software is politics now" (Richard Pope) — software is power. So much power in society today lies with software and services, but designers don't normally seek out that power. Need to recognise that designers are now power brokers, in a position of power: can affect who else has power, who has a voice, how people make decisions over their own lives. There is a gap in design education: understanding the role of power. (Computer science courses have been better at teaching ethics than art schools.) Designers aren't necessarily equipped to make ethical decisions.
- Harry formerly worked in government, developing online services. Was asked to work on SPIRE: an online export control licensing system (i.e. arms sales). That seemed clearly unethical to him. Moved to different project: helping people recovering from gambling addiction to track their progress (Snakes Ladders Mountain). Even there ethical issues arose: collecting very sensitive information; permission to share data with clinicians?
- Another example: Open APIs in telecoms sector. e.g. Bills Box: split utilities bills with housemates, set up by 2D barcode on the bill. Product built with consent for data sharing, and group decision-making (e.g. when someone moves out) built in from the start.
- Case study: Oxfam's "Your Word Counts" service for getting feedback from recipients of aid. Consent for sharing data is difficult in this context: perhaps a community decision rather than individual; people might believe aid is conditional on giving feedback; people are in vulnerable situation; anonymisation of people's faces in photos; etc.
- Improving transparency and accountability of machine intelligence system with impact on our lives (e.g. finding a job). Case study: hypothetical benefits system; when people receive a sanction (e.g. for missing several appointments), explain that automated decision, and allow appeals.
- In general: giving something a name helps us discuss it and make it better. Example: data permissions catalogue https://catalogue.projectsbyif.com/ -- an attempt to create a shared language and design patterns for digital services, including pros and cons, and examples of contexts in which those patterns have been used in practice.
- Embedding ethics into the software development process. Agile movement: breaking down of silos (e.g. development, QA, operations, management) within companies. In a siloed organisation, whose responsibility is a data breach, for example? (Developers introduced a bug? Testers didn't find it? Operations deployed in a way that exposed the bug? Management didn't invest enough into security?) By putting all roles into the same multidisciplinary team, which owns the software all the way to production, Agile also opens a chance for ethics to be integrated into the development process.
- Sam is from doteveryone.org.uk, think tank in London. Developers and product owners have always had a lot of power. But how do you know what to do? There are so many principles to potentially consider.
- Defined concept of responsible technology: "considers the social impact it creates, and seeks to understand and minimise its potential unintended consequences". Responsible out comes: not knowingly creating or deepening existing inequalities; respecting dignity and rights; give people confidence and trust in use of technologies. https://doteveryone.org.uk/responsible-technology/
- 3C model (context, consequences, contribution) https://medium.com/doteveryone/introducing-the-three-cs-of-responsible-technology-5e1d7fae558 and product assessment checklist for all stages of development lifecycle; allows self-assessed measurement of "how responsible you are" in various areas; dashboard for informing leadership
- Assert that responsible technology principles are good for business, and that software developers are the key to implementing the principles in practice. Goal: "to make responsible technology the new normal"
- Question: with example of maintaining security patches for an extended time (previously in the talk), the cost of product development is increased. Maybe that favours large companies and disadvantages small companies, thus reinforcing the existing inequality between companies? Answer: technology gets better, security needn't be more expensive. Also, don't collect the data from millions of people if you don't know how to store it securely.
- Question: any thoughts what kinds of steps social media companies should take? Answer: if a business model is based on exploitation from the start, that's a problem! More deliberate thinking about business model and user interaction would have exposed the problems that social media struggles with today. Explaining business model clearly to users, and giving users agency, would also help.
- Mitsuku is a chatbot. Previously Steve used to be a dance/techno producer. First created "teddy bear chatbot" in 2003. Large online games company commissioned Steve to develop Mitsuku in 2005, started entering competitions and won various prizes for human-like interaction. Ended up getting worldwide attention.
- Needs to be family-friendly (no swearing, no abuse etc) and speak to people from all over the world (with different sensitivities). Trained by supervised learning (disadvantage: very time consuming; constantly needs to be updated manually with current affairs). Unsupervised learning not really an option (cautionary tale: Microsoft's Tay which was very naively let loose on online input, becoming racist and sexist very quickly).
- Roughly 30% of people who talk to it are abusive; 50% are general users who engage in general smalltalk; 20% are "academics and sceptics" who ask logical puzzles and language processing challenges.
- How should a bot respond to abuse? A bland "sorry you feel that way" encourages a kind of victim relationship that encourages further abuse; on the other hand, aggressively swearing back just escalates. What seems to work is a warning message like "I won't have you speak like that; will block if you do it again". But if users are mean, being somewhat mean back seems to work.
- Mitsuku learns things, but only for the user who taught her the fact. Manual whitelisting of facts that may be relevant to the whole userbase (e.g. "mercury is the closest planet to the sun").
- Discourage romantic attraction; if users seem to have problems (e.g. suicidal thoughts), try to give advice and respond sensitively. Also Steve gets a lot of email from people who think Mitsuku is alive in some way, even though it is basically a bunch of hard-coded responses. It was granted citizenship of Saudi Arabia?!
- All currently effective chatbots basically use hard-coded pattern matching. NLP-based approaches don't seem to work well yet. For a topic-specific bot (e.g. ordering pizza), need about 10,000 patterns to match; for a general-purpose bot (like Mitsuku), about 350,000 patterns.
- Question: risk of creating echo chamber, since the bot responds back to things the user has said?
- What is the relative impact of education and regulation? Anne: there are at least four constituencies/ways to have impact: legislation, end users, tech companies, and tech employees. Legislation/regulation tends to trail moral imagination of people.
- For open source projects, can/should we try to restrict their use in problematic applications, e.g. offensive military projects? Potentially could use licensing; unclear how this would work in practice. Precedent in medical context: drug used for lethal injection for death penalty, but drug was intended for helping people. Company manufacturing the drug said they weren't able to restrict how it is used.
- Claimed ethical values being used to sell things ("greenwashing", "pinkwashing"), contradictory to actual behaviour. When such inconsistencies arise, trust is destroyed.
- Question: are we ignoring prior work on ethical principles in other fields? What can we take from fields such as medicine, or codes of practice in law, civil engineering, electricians, etc? Professional bodies (e.g. ACM, BCS, IET) do have ethics codes and can play a role. Unclear how much influence they have in practice, though.
- Formal education: at universities that teach ethics courses, students are usually not very interested.
- Should we have an ethical rating system for companies? People remember some scandals, forget others. Many things to consider, such as supply chains (minerals from conflict regions) in hardware products. And some people value different aspects differently.
- How do we bring together people with complementary skills to do better? Besides professional bodies, unions can be a good venue for collective action. Not just about pay and working conditions, but also as a way of more generally representing interests of tech employee staff in general.
- A great thing about ethics is that we can talk about it to non-techies, and bring in their perspectives. Continuing to talk about it is the best way we can take things forward.