You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Appendix A contains code snippets from the "cookbook" conversational AI skill developed in MIT App Inventor.
Appendix B contains links to the conversational AI curriculum (slides, tutorials, etc.) we taught students in the workshops.
Appendix C contains the questions asked on the questionnaires, codes we used for the thematic analysis (and example quotes), and visualizations of the results.
The EAAI (learning outcome) study can be cited as follows:
Van Brummelen, J., Heng, T., & Tabunshchyk, V. (2021). Teaching Tech to Talk: K-12 Conversational Artificial Intelligence Literacy Curriculum and Development Tools. In 2021 AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI).
or with BibTeX:
@INPROCEEDINGS{vanbrummelen-teaching,
author={{Van Brummelen}, Jessica and Heng, Tommy and Tabunshchyk, Viktoriya},
booktitle={2021 AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI)},
title={Teaching Tech to Talk: K-12 Conversational Artificial Intelligence Literacy Curriculum and Development Tools},
series = {EAAI '21},
publisher = {Association for the Advancement of Artificial Intelligence},
year={2021},
}
The IDC (Alexa perceptions) study can be cited as follows:
Van Brummelen, J., Tabunshchyk, V., & Heng, T. (2021). "Alexa, Can I Program You?": Student Perceptions of Conversational Artificial Intelligence Before and After Programming Alexa. In 2021 Conference on Interaction Design and Children (IDC).
or with BibTeX:
@INPROCEEDINGS{vanbrummelen-alexa,
author={{Van Brummelen}, Jessica and Tabunshchyk, Viktoriya and Heng, Tommy},
title={``Alexa, Can I Program You?'': Student Perceptions of Conversational Artificial Intelligence Before and After Programming Alexa},
booktitle = {Proceedings of the 2021 Conference on Interaction Design and Children (IDC)},
series = {IDC '21},
publisher = {Association for Computing Machinery},
year={2021},
}
We created a sample skill to show the motivation behind the conversation AI interface and demonstrate how it may be used by students to build their own projects.
This skill is a talking cookbook: You can ask your skill what ingredients you'll need for a recipe, and you can ask for step-by-step instructions on how to cook or bake a given dish. With each new step, the skill also notifies the accompanied app to display a visual aid to the user that shows them how to do the step they're currently on. Below, you will see a more detailed explanation of how to app works and how the interface facilitates this project using block-based coding.
Figure A: The Designer page on the main App Inventor interface. Here, a user can drag in Components that they would like to include in their app, which in this case is a Label, an Image, and a CloudDB component. CloudDB is a redis server hosted by App Inventor on the cloud. Users can also upload files to use in their app, such as the different images that they would like to use to show the user how to follow different recipes.
The app side of the code stores two dictionaries with predetermined recipes that the programmer included in the project while building it: one for the steps of each recipe, and another that stores the names of the image files that demonstrate each step. Next is a CloudDB.DataChanged block that is evoked any time some data changes in the user's CLoudDB. This data will come from the skill. The app will then store the data changed in global variables within the app, and if the data changed happened to be a new step_number, then the app updates a label and image object to represent the recipe and step that the user is currently on.
Figure B: The blocks for the MIT App Inventor app that accompanies this skill.
The skill side of the code defines the Voice User Interface for the programmers's skill, as well as the endpoint code for each intent. Here, you can see that the user has defined three intents: firstStep, nextStep, and ingredients. These intents include a slot of type Food to store information about which recipe the user is interested in. To the right are the endpoint blocks for these three intents: each when intent spoken block will be translated by the interface into a JavaScript function that Alexa can understand and execute. The ingredients intent simply finds the appropriate list of ingredients in the global ingredients_dict, translates it into a csv string, and speaks it with some extra text at the beginning to turn it into a full sentence. The firstStep intent stores the value of the recipe slot spoken by the user in a variable, sets the current step number to 1 (App Inventor uses 1-based indexing), and sends both of these values to the user's CloudDB to notify the app that it needs to update its display to the current step. Next, it finds the first step for the given recipe in the global steps_dict, and speaks it to the user. Finally, the nextStep intent is very similar to firstStep, the only difference being that it increments the current step number by 1, and does not store any information about the recipe because it assumes that it has already been stored when the user asked for the first step.
Figure C: The blocks for the Alexa skill for Sheila's Cookbook project.
This document was sent to teachers whose students were participating in the workshops. By completing the tutorial, students can ensure all of the technical requirements for the workshops (i.e., computer with internet connection, and a method to test apps and skills) are satisfied.
This document is a comprehensive guide for teachers who wish to replicate the five-day workshop. It includes links to all of the resources described here and where in the curriculum each resource was used.
This document gives instructions on creating the Codi Clicker app in App Inventor. This tutorial introduces the App Inventor programming interface, UI-design, event-handling, and variables.
This document gives instructions on creating the Talk With Me app in App Inventor. This tutorial introduces if-statements, string manipulation, and creating a simple rule-based conversational agent.
This presentation accompanies the activity on Day 2 where the students are introduced to the Big 5 AI Ideas and participate in an activity where the class decides whether a given device uses AI or not.
This presentation accompanies the discussion on AI Ethics. It begins with a clip of the Google Duplex demo, an example of incredibly advanced AI and some of the moral consequences of using it. The presentation also has some questions where students can answer and debate freely.
This document gives instructions on creating the Hello World skill in App Inventor. This tutorial will familiarize students with the steps and vocabulary involved with creating an Alexa skill by making a simple skill that says hello or good-bye when the user says a greeting or farewell respectively.
This presentation accompanies the lesson describing the role that machine learning plays in creating the Alexa skill. It describes at a high level that AI needs a large amount of data to be able to generalize meaning in language and mentions that transfer learning is likely another factor in training each skill's intent classifier.
This document gives instructions on creating the My Calculator project in App Inventor. This tutorial will teach students how to extract information from the user's utterance by using "slots."
This presentation accompanies the lesson describing at a high level the process of encoding data, getting output from the model, and changing the weights of a neural network. The goal here is to demystify the "thinking" and "learning" processes of AI. mentioned in the previous slides. Although details such as gradient descent and activation functions are omitted from the description of neural networks, we hope that students will still take away that machine learning algorithms are math functions with tuned weights.
This presentation accompanies the lesson describing the CloudDB Component in App Inventor. In short, the CloudDB component allows devices (be it a mobile device or an Alexa-enabled device) to connect to an MIT-owned REDIS database. Using the CloudDB component, these devices can change values on CloudDB, read values from CloudDB with a given tag, and even detect when another device changes a value on the CloudDB.
This document gives instructions on creating the Read the Text project in App Inventor. This tutorial uses CloudDB to facilitate a degree of communication between a mobile app and an Alexa skill made in the App Inventor interface. Students can type information into the app they create and then ask Alexa to read the inputted text.
This presentation accompanies the brainstorming activity at the end of the fourth day. This presentation lists a number of themes that students can propose ideas for their final projects to attempt to address.
Padlet Brainstorming Figures
For the brainstorming activity, we used Padlet. Below are some figures which may be of interest for the first and second week.
Through a reflexive, open-coding approach to thematic analysis, we identified codes for each of the study's questions. Representative quotations from each of the codes and histograms of the data follow.
Day 2, 3 & 4: What was the most interesting thing you did or learned in today's class?
AI-related
"Certain qualities that makes a machine classified as artificial intelligence such as visual perception."
"I learned about machine learning AI and rule based AI"
"The concept of an AI’s ‘brain’ was super interesting, and I became really fascinated with the idea of memory in AI systems."
Group activities/interactions
"The brainstorming of ideas for what the robots could do was interesting to me."
"It was the AI Dungeon, very interesting tutorial."
"Everyone’s ideas for functional apps that could easily improve peoples lives were really interesting and showed me all o lot of possibilities."
Making Alexa skills
"The most interesting thing i did was make a skill for alexa that can speak to us."
"The most interesting thing i did today was properly seeing how the blocks we use affect the echo dot."
"The most interesting thing I did was program my own calculator with the MIT App Inventor software."
Programming
"I learned tags and tag values are and how information goes to the cloud and then back to the user"
"About how you can make a sound with a phone"
"The most interesting thing I have learned is about cloudDB and how it stores information for later use."
Real-world examples
"The google duplex and how it sounds like a human voice"
"The most interesting thing I learned is how google programmed something to talk and respond to calls like a human."
"I also really like Mr. Tommy’s presentation of the Google assistant integrating itself ion everyday life, and we didn’t even know it."
Societal impact
"The Ethics of AI. It is an important topic to talk about. How far is too far and where do you draw a line of AI usage."
"I realized that it would very hard for AI devices to be perfectly unbiased considering that humans tend to control the input and knowledge of the devices, and humans are biased."
"Coming up with ideas for AI to improve living situations."
Day2:
Day 3:
Day 4:
Day 2, 3 & 4: In what way did you demonstrate the above maker skills?
For context: Maker skills include risk-taking, collaboration, connecting knowledge, and persistence:
Risk-taking. Students tried ideas outside of their comfort zones and learned from failures.
Collaborating. Students actively participated in the class and supported others’ learning.
Connecting. Students connected their experiences, culture, and knowledge to content in the class and shared them with others.
Persisting. Students persevered to solve problems and iterated on their designs and ideas.
Debugging
"I kept trying to fix the coding if it was broken"
"I had trouble with my skills and figured it out on my own, and I have been following everything no matter what"
"I had a problem where the text look funny when type and spoken so i when to the blocks section again retype the problem but add space before and after the word and final it work and i both test on the type and on Alexa herself."
Extending materials
"I tried to add some of my own sayings and utterances into the block code."
"I showed Risk taking by adding other functions to my calculator skill, such as subtraction, division, exponential functions, etc."
"I try to see if Alexa could speak Spanish back when I speak to it in Spanish like "Alexa, adios" "Goodbye". Still need work but it was great."
Helping others
"I showed collaboration by sharing my code and ideas to help others who were having errors"
"A couple of my peers had run into a few problems that were similar to mine, so we helped one another out before pushing on."
"I have a cousin who was working with us and needed help so I went over and decided to help"
Speaking up
"I also participated and was able to offer ideas for apps today."
"I opened up to answer a question that I wasn’t sure to have the right answer. I chime in when being ask for my input on the subject."
"I made sure to ask questions when I could an tried to participate when I could."
Didn't demonstrate
"I didn’t really say anything but mostly listened to others ideas"
"I was mainly following the instructions and paying attention in class."
"I worked independently"
Day 2:
Day 3:
Day 4:
Perception of AI Question 1: Describe in your own words what AI is.
Perception:
"AI is a computer program that can perceive the outside world and needs to be gradually fixed so it can almost always be right."
"AI is a computer program that can perceive the outside world and needs to be gradually fixed so it can almost always be right."
Representation and reasoning
"AI is artificial intelligence that is able to perceive its surroundings and obtain huge amounts of data and be able to organize them. It also is able to understand this data and continuously learn and adapt."
"Artificial intelligence that can perceive, learn, and output an answer."
Learning
"Computers learning from their experience how to do new tasks"
"[...] AI is constantly improving with the data it is collecting from users.
AI is a system made to by engineers to think and learn."
Natural interaction (e.g., human language)
"It is like a smart computer that can talk to you"
"I think artificial intelligence is programmed to act, talk, and think like humans."
Explanation of acronym
"I think its artificial intelligence"
Mimics humans (e.g., human-like, personification of AI, etc.)
"AI stands for artificial intelligence, which refers to the active machine learning of the given mechanism, usually displaying intelligence that can be compared to that of the human mind."
"AI is technology that was fed information and learnt enough to at least have some sort of judgement and can do things on its own. It's basically machines programmed to kind of mimic human intelligence in a way."
Helps humans
"AI is where you use programing to make things like robots or technology to be able to communicate with humans to make life more easier."
"It can also be used to control certain functions such as music, lights, and television."
Computer control or programming
"Al is something to let you control of something. [sic]"
"Artificial Intelligence that is created by a kind of programming to perform a task."
Real-world examples
"Artificial Intelligence is a concept that describe a platform (Siri, Alexa, Google Assistant) that is able to think, talk, "behave" like a human or at least, it tries it best to be."
"[...] Examples include siri or Amazon Alexa."
Vague/shallow answers
"Intelligence"
Perception of AI Question 2: Describe in your own words what conversational AI is (e.g., chatbots, like Alexa or Google Home, use conversational AI).
Pre-programmed responses
"They have been programmed to respond to most possible outcomes."
"A conversational AI is an AI that is given phrases or has default phrases to say"
Purely Natural language response (i.e., system talking to human)
"I think its a AI that knows how to respond to someones questions."
Purely Natural language understanding (i.e., human talking to system)
"ask it something you want it to accomplish like 'Alexa play music'"
Natural language response and understanding
"conversational AI is when you talk to an AI and they respond back."
Representation and reasoning
"Conversational AI is the same as an AI except it is also able to understand human language and respond in that human language after interpreting what was said."
"An AI that speaks back to you and analyzes."
Learning
"Conversational AI is AI that slowly understands human speech and interactions through a long learning process."
"Conversational AI is complex technology that is able to understand its enviornment [sic] and continue to learn new things over time"
Related to robots
"It is a robot programmed to send realistic responses to questions asked to it."
"A robot that responds to certain commands and requests "
Helps Humans
"Conversational AI is a tool that programmers created so people can get through their day to day lives."
"A conversational AI is device or program that is used to help others."
Real-World Examples
"This included shopping, alarms, etc. Its very beneficial."
"[...] such as the Alexa always saying that its favorite color is ultraviolet"
Vague/shallow answers
"that follow their line of code"
Perception of AI Question 3: Describe in your own words how conversational agents (like a chatbot, Alexa or Google Home) decide what to say.
Perception
"I think the Alexa works by listening to your question with a microphone built into it and a text to speech to program it to talk to you."
"I think conversational agents decide what to say by recording the speech input they receive from the user, converting it to text"
Representation and Reasoning
"I think it record our questions, then analyzes what the question is asking, uses the data that it has in its database and answers the question."
Steps of ML
"Conversational agents decide what to say in many ways/steps. When the agent is first being built, it is trained by being shown lots of samples. After the initial training is completed, the agent goes through testing of these skills. After this initial input of data, the conversation agent is able to decide what to say by using complex and fast math to derive an output from an input. Conversational agents can also use memory to help them decide what to say."
Natural Interaction
"I believe that conversational agents copy human conversation and decipher the speech and text which allows them to know what to respond."
"I think it first process what the person says or the question given before it replies. It engages in contextual dialogue using natural language processing which i believe helps with what it says."
Collects/uses data
"Based on how the user interacts and also on data already input previously or that it remembers."
"a Personal assistant that logs your information and stores it. It answers questions based off data."
Accesses the internet
"I think that alexa works by taking the user's command(s) and uploading it to Amazon's servers to find an answer."
"she will look up the phrase you're talking about"
Purely Learned
"They decide what to say from experiences in development or usage."
"They are taught and fed information which with time helps them decipher ways to respond."
Purely Hard-Coded
"There are also common facts and truth that is implemented in these agents in advance to make sure it can understand basic languages and common sense."
"They are programed to respond to questions with a specific answer."
Part Learned, Partly Hard-Coded
Listens for keywords
"Conversational agents decide what to say by listening out for key words or phrases used by the speaker, then interpreting this and promptly responding with the information at its disposal."
"They listen to key components and check to see what the user is asking for."
Vague/shallow answers
"Conversational agents decide what to say based on the very complex codes and algorithm that are implemented into them."
"Conversational AI" [sic]
Day 2 Question 1 a.: In your own words, what questions might you ask your friend to figure out whether the agent is actually AI?
For context: Your friend says that there’s an AI integrated into her fridge. Apparently, the AI knows what’s in the fridge, and your friend can use it to order food that’s not already there.
Learning:
"Is the AI constantly learning the different kinds of food?"
"if it gets smarter and smarter every time"
"Does the fridge already have a database of what it is seeing, or does it learn what its contents are through trial and error?"
Natural Interaction:
"A question that i may ask is if it is interacting with a human"
"If they have emotion or can respond to complicated phrases/sarcasm."
"Does it interact and respond to the environment?"
Perception:
"whenever something new is added or taken away will it know?"
"I would ask if the agent can perceive the outside world"
"Is it understanding or just observing the contents of the fridge and what is needed?"
Representation & Reasoning:
"Is it following rules set by a human, or venturing off into some thinking of its own?"
"show problem-solving traits that relate to the human mind"
"Does the fridge make the choices by itself?"
Shallow/Unrelated to Big Five:
"How do my Walmart slide door open."
"What the agent does."
"Ask what exactly is in the fridge"
Day 2 Question 1. b.: In your own words, what questions might you ask your friend to figure out whether the agent is conversational AI?
For context: Your friend says that there’s an AI integrated into her fridge. Apparently, the AI knows what’s in the fridge, and your friend can use it to order food that’s not already there.
Natural Language Response
"I would ask my friend if it answers back to his/her commands."
"Does the fridge talk back to you like if your having a conversation?"
"Does the fridge communicate to us through natural language?"
Natural Language Understanding
"Do you talk to it to figure out what food is inside?"
Natural Language Understanding & Response
"Ask it questions about the food and see if it will respond and have a conversation about the food"
"I would ask about the contents of the fridge and sees if it talks back to me or not"
"I would ask it it can actually talk to humans understand their language and respond to what a human says."
Questions about AI, but not Conversational AI
"A question I may ask in it following a rule"
"Can the AI identify different kinds of food, different sizes and colors without her input? Is the AI constantly learning the different kinds of food?"
"Can it interact whit you."
Real-World Examples
"Do u speak to Siri or Alexa before"
Doesn't show deep understanding of Conversational AI
"What food should I order?"
"How dose to know what is work"
"Say "how was your morning" and if it responds somewhat closely to the question then it’s deemed conversationally capable."
Day 3 Question 1: Describe differences between rule-based AI and machine learning based AI.
ML: Learning
"Machine learning allows AI to learn from its mistakes and improve"
"While machine learning is where the AI is learning something new it never had in it system."
"machine learning AI has to be tweaked a lot to get the closest answer."
Rule-based: Limited in scope
"Rule based AI can only do so much, and their response are very limited."
"Rule-based AI is programmed by a person, which makes it more limited and bound to the specific rules it is fed."
"a rule based Ai folles the code but the learning one can adapt to many things"
Rule-based: Programmed rules
"ruble-based AI are rules that have been installed into the AI before."
"Rule based AI follows strict rules to do task"
"Rule-based AI is pre-programmed rules"
Shallow/Unrelated to Big Five
"I feel like they have different way of working"
Day 4 Question 1 b.: Why would or why wouldn’t Alexa recognize “Give me a math joke”?
For context: Let’s say you gave Alexa three phrases for an intent,
“Say a math joke”,
“Give me a numbers joke”, and
“Tell me a joke with numbers”
and you programmed Alexa to respond with, “Why was 6 afraid of 7? Because 7, 8, 9!”.
Doesn't match training data exactly
"I don’t think Alexa would recognize this phrase because it is not exactly what was inputed."
"Thee three phrases does not include "Give me a Math joke""
"Give me a math joke isn’t one of the phrases listed so it wont see it as the same questions"
Similar to training data
"Because Alexa is a learning based AI so she would learn that it is close to one of the phrases programmed in it."
"Because it somewhat similar to the word say "say a math joke" that was implanted in Alexa"
"But if advanced, Alexa would be able to depict the similar words from her known phrases."
Vague/unrelated
"I think Alexa would recognize a math joke if it was programmed to do so like we did with the book worm skil."
"what is 5 time 59"
"She is programmed to give a regular joke so why not a math joke."
Day 4 Question 1 c.: What do you think Alexa would do if you said “Give me a math joke”?
For context: Let’s say you gave Alexa three phrases for an intent,
“Say a math joke”,
“Give me a numbers joke”, and
“Tell me a joke with numbers”
and you programmed Alexa to respond with, “Why was 6 afraid of 7? Because 7, 8, 9!”.
Joke programmed in
"Alexa would respond with this: “Why was 6 afraid of 7? Because 7, 8, 9!” because she was programmed to respond in that way."
"Alexa would say "Why was 6 afraid of 7? Because 7, 8, 9!""
"She would say the joke."
Math joke (no specification)
"Alexa would tell me a math joke."
"She would respond with a math joke of some sort."
"I think she would give you a math joke"
Some other math joke
"I think it will respond with a math joke or maybe search up math jokes"
"I think she would associate it with the number’s joke, And would not make it a number’s joke but instead a Pi joke for example."
"She would give me a math joke from either the web or with the code I put."
Nothing/Null/Error
"Alexa would say that the intent phrase is wrong and try again."
"Probably that she didn’t understand or "null""
"She would not answer."
Vague/unrelated
"i think Alexa will get me the answer"
"I don’t know"
Post-questionnaire Question 1 a.: What are some reasons why creating a conversational agent to help your friend with math might be helpful or a good idea? (E.g., What are conversational agents good at/for?)
Saves time
"The agent will the person so you don’t need to take time to help them."
"It can help your friend instead of you helping him.Its a good idea cause you can do something else and the agent can help him"
"It could help him complete it faster."
Personalized to the user
"It could be helpful because it can teach at the persons pace"
"Specific intervention and help students with social anxiety/insecurity"
Automates tedious/difficult task
"it could tell him answers to a question he is struggling on"
"Creating a conversational AI would make problems a bit easier, as the only input required is his voice; rather than manually putting in every single part of the equation."
"They can do calculations that might be difficult for some people."
Clear/correct answer
"The conversational agent could answer specific questions that your friend might have that you might not know the answer to."
"[sic] there wouldn’t be a difference of opinions or methods, which can be confusing."
"Creating a conversational agent to help you with math can possibly explain to you how to do a math problem step by step."
Access to resources
"[sic] because they have lots of different resources he can use if he needs additional help."
"Conversational agents have access to the internet"
Always available
"It could help them because a conversational agent can be used at anytime"
"He will stop asking me. If my phone dies, he can depend on Conversational agents."
"[sic] to help them if I am busy with other work or I am just not there."
Conversation to explain the answer
"Also they are able to go back and forth with the questions and answers."
"The conversational AI can help your friend understand each step by explaining what to do step by step and if the friend asks a specific question about the problem like how does this become this."
"it could be like a personal tutor programmed to solve math questions and make conversation."
Vague/shallow
"its so they can under stand what they are doing"
"It would be a good idea so you can help hin out"
"It might be helpful because it will be able to get your words and read it out to them if they need help."
Post-questionnaire Question 1 b. What are some reasons why creating a conversational agent to help your friend might not be a good idea, or why it might be easier or better to help him directly? (E.g., What are conversational agents not good at/for?)
Bad at understanding/making connections
"Also the friend may not be able to properly articulate what they’re confused about so it would be hard to tell an AI"
"[sic] might not be a good idea because AIs have no emotions or guilt"
"Creating one might not be good at word problems,that would be better interpreted by a human in my opinion"
Can give the wrong answer
"One the person might put the wrong answer in the system"
"The AI might be wrong or he never learns"
"[sic] the conversational agent could go off topic or tell him the wrong way on how to do something"
Ethics
"There may also be run ins with ethical issues and biases against the user."
"The issue with making this conversational agent is because they could just use it to give themselves answers instead of using it to learn like its intended for. It might be easier to help him directly because you won’t be helping him cheat, whereas a program won’t know how it’s being used."
"It may not be a good idea because sometimes you also don’t know what to do and when you tell them they might be upset."
Humans are better teachers/adaptable
"An AI may not be clear with its instructions. The personal would be advised because you can understand your friend better than the AI and you probably can explain it better than the AI"
"The agent might only provide the answers and might not be to explain or describe the set up of each problem."
"It would easier to help my friend directly because I can attend solely on what they are confused on and help them specifically in one area."
Limited answers
"It might not be a good idea because conversational agents can only handle level 1 questions and not many things complex."
"Conversational agents are limited in their responses since they are programmed with limited response banks."
"[sic] the agent will only be able to help with what is programmed into it."
Time to build the agent
"It might take more time to code than to actually help him."
"Furthermore, it would take less to help the friend than to create a conversational agent"
"There are many variable in helping him.And by the time you finish programming it, you might have already finished helping him if you helped him."
Vague/Shallow
"to help people understand what to do"
"he needs to figure it out on his own"
Post-questionnaire Question 2: Name a few decisions (2-3) you made when designing your conversational agent. What are some other decisions (1-2) other developers might think about when creating AI systems?
Societal impact
"Other developers are likely to think about what is best for their users as well as making sure the device is able to recognize a variety of voive types (for example, male and female) to minimize biases."
"I had to see if asking for their allergies and medicine would be to personal since the app is for in case of emergencies and basically a web md but alexa form."
Natural interaction (e.g., usability, voice user interface design)
"I had to decide how the user would directly interact with the AI and how they would input the information and how they would do that. For example what buttons lead to what. Some developers may need to question how easily understandable the visual information may be to a new user. "
"[...] Other developers might have to decide what their AI will sound like."
Learning (e.g., training the ML)
"[...] They would also have to consider how to maybe make the conversational AI be able to learn while it has conversations."
Specific functions and programming
"[...] Some decisions other developers might think of when creating AI systems is what program the AI system to do and what specific functions can they add."
"When It displays a color, I need the agent to ask what the color is. I then need the conversational agent to take whatever there reply was in the text box and say if it is right or wrong."
Post-questionnaire Question 3: If your Alexa skill started to be used by millions of people around the world, what effect would this have? What would be some positive effects? What would be some negative effects? Who might be affected?
(Positive) Usefulness
"Some positive effects to world wide use of Alexa skills would be more convenience for people, improvement on certain aspects of a person’s daily life."
"It will help others keep a schedule and avoid stress."
(Positive) Environmental
"One effect it would have is that the trash in recycling centers will be reduced since people are using my alexa skill."
"A positive impact would be that this can possible save a lot of paper waste. My Alexa skill can replace paper flashcards."
(Positive) Improves Communication
"If this skill were used by millions across the world, it would aid in the communication between millions of people, regardless of languages, making it easy for travelers, or those learning a second language, to communicate. [...]"
"If my Alexa skill started to be used by millions of people around the world, it would help people who have a hard time learning or remembering a language. It would also help tourists who are visiting foreign countries."
(Positive) Educational
"For my Alexa skill to be used, it will benefit foreign speakers and allow to people to better understand languages in a short amount of time. It can make many people bilingual and make it easier for them to get jobs."
"If my Alexa started to be used by millions of people around the world I think it would help kids learn their colors faster or at an earlier age."
(Positive) Mental / Physical Health
"My skill would have positive effects because Alexa would be able to assist those struggling with mental health issues like depression, loneliness, sadness, and anxiety. This skill would help variety of age ranges. Anyone struggling with these problems would find comfort in this skill."
"If my project was used by the masses fitness could potentially be more easily accessible. Thus, users would be able to stay in shape better."
"If my Alexa skill was used by millions of people around the world there would be some positive effects because it just generates random jokes and memes"
(Positive) Developer fame
"If my Alexa skill was used worldwide then this would make the skill more known. Positive effects would I would be more known as a developer [...]"
"The positive affects would be that now I'm probably making some money off the app from the downloads and company sponsors and being a popular developer. [...]"
(Negative) Overreliance on AI
"I think some negative effects can be that kids become very reliant on AI systems."
"One negative effect is that people will rely more on an AI than themselves."
(Negative) Affects Jobs
"Fitness trainers could possibly lose business as the Alexa could take their places."
"It may also negatively imoact [sic] teaching jobs or methods."
(Negative) Technical Issues (e.g., bugs, need for additional features)
"A negative effect is that it might be a little delayed or not functioning correctly. "
"The negative effects are that she can have errors."
(Negative) End-user ethical concerns (e.g., used to cheat on tests)
"However, this app could be used by students learning a second language to cheat on home-assignments."
"some negative effects if maybe some memes that are not funny and more rude that could lower a person's mood."
"It may have negative affects because some memes are offensive to some people (like the Karen meme). I think all kinds of people would be affected."
(Negative) Privacy/security
"[...] One concern that it might brought up is the privacy issues with my program."
(Negative) Threaten the developer
"[...] Negative effects would be that they're would be people who despise my app so much they send threats to me or try to take my app down from the market. I would get affected from all of this."
Vague/shallow answers
"It would be good"
"It would positively impact people because it could help them in many ways."
Post-questionnaire Question 7. If you could design the next version of Amazon Alexa, how would you change it? What features would you add? How could it be improved? (Feel free to think outside the box!)
Emotion Recognition
"Ideally Alexa would be able to pick-up on and understand emotions/expressions by the user."
"I would give it the feature to bond with the user. Perhaps they can find the usual tone in your voice and make guesses from that."
Ethics
"I would add a better security system for Alexa
"[sic] but that brings up the subject of what kind of face to choose, gender, color and race are very sensitive topics."
"Some features that I would add is the ability to add voice password so if someone other than you says "hey Alexa" she wouldn’t respond."
Futuristic features
"I would add maybe a visual component like a hologram sort of thing so if you ask what is an apple she can say its a fruit etc. and then also display an apple."
Graphical UI
"Maybe facial expression, like a wink, or a smile which you can see and interact [sic]"
"I will add pictures."
"Maybe adding in a friendly user interface if the next Alexa had a screen. Take for example, putting in a simulated person who would have face to face communication with you to help you feel more at ease with the program."
Improve Speech Recognition
"I would make her better understand accents and answer questions better. I feel like a lot of times i’ve asked alexa a questions she answers it wrong."
"I think Alexa is fine, the only thing I would change is for her to better understand speech and requests [sic]"
"If I could change Alexa, I would make it so she could understand what I am saying better. She could be improved because when I tell her a command she does not understand most of the time, what I am saying"
More Languages
"I would make it so that you can change the language of alexa just with a voice command so that anybody can easily change it."
"I would make it have a translator into different languages"
"if i could add anything new i would create a system where it can scan any language and translate it to english or make it speak a language they are comfortable with."
New skill/Programmed feature
"i would add a calculator feature and a translator feature since those features will help a lot of people."
"I would add more helpful features"
"Also her database needs to go beyond the regular old stuff. She needs to know the top 10 advice you can give someone, or the best meme of the year, or top 5 recommended songs based on your mood."
Physical UI
"I would make more of a portable alexa or one that can work offline."
"If I could design the next version of alexa I think i would make a variety of shapes instead of just a cylinder just to make it look more fun"
"If I were to design a next version of Amazon’s Alexa, I would make it modular so you can bring Alexa anywhere. You can make Alexa modular by connecting Alexa with your phone, or you can design a earpiece with Alexa’s AI embedded so you can ask Alexa things on the go."
VUI
"I would try to make it more life like"
"If I could design the next Alexa, I would improve her character. She needs to be more of a friend than an assistance."
"If I could design the next version of Alexa, I would add more of a personality to it by having it speak in a different tone or have a different reaction based on what the user said. This way, it’ll be like you are talking to a real person(almost a real person)."
Post-Questionnaire Question 7. What was your favorite activity and what did you enjoy about the workshops? What was helpful for learning? (As well as a question on students' final project slides.)
Instructors/teaching
"[...] the staff members are all nice and very helpful."
"[...] I really enjoyed how kind the speakers were, and how they would always help if you needed it. [...]"
Organizational features
"[...] Things that were helpful for learning were the pace and the pdf. I looked at the pdf for the last two workshops because I liked to go ahead for a bit and re-listen to why specific coding was used."
"[...] the powerpoints and demos was really helpful for learning since I would sometimes get confused on certain concepts."
AI
"[...] I really also enjoyed learning about how AI actually worked [...]"
"I really enjoyed the workshops because [...] teach me how AI is. [...]"
Projects
"My favorite activity was when we got to make our own projects that was really cool. [...]"
"My favourite activity from the workshops was the final project, in which we each were allowed to let our creativity soar. [...]"
Codi Clicker Tutorial
"I enjoyed the codi clicker the most because of how easy and fun it can be just as a time killer."
"The Kodi [sic] Clicker because I liked that it fun and a learning experience at the same time."
Read The Text Tutorial
"My favorite activity was the Text Reader when Alexa could actual [sic] read what you wrote in the text box. It was really enjoyable."
My Calculator Tutorial
"My favorite activity was the calculator one"
Group activities (e.g., AI Dungeon, brainstorming)
"I favorite [activity] is when we have any exciting ideas on what to do."
"My favorite activity was the AI Dungeon! [...]"
Programming in general
"[...] I enjoyed learning how to code certain functions such as audio wise and telling Alexa commands for her to respond to based on what you put in your code."
"[...] The coding blocks are good way to help and learn coding [...]"
Seeing code working / on actual device
"[...] My favorite activity was learning how to connect the app inventor to my phone and then transfer the data to Alexa."
"I liked learning how to program skills and when it actually worked [...]"
Post-Questionnaire:
Final Project Slides:
Post-questionnaire Question 8. What was the most difficult part of the workshops? How could we improve the workshops?
Technical difficulties
"I think the most difficult part is when updating our skill because not all the time it worked or go through"
"It was difficult to run the emulator, so I would figure out a different way to test the app, like make it for iPhone users."
"The most difficult parts were figuring out where you went from when the MIT App inventor would no work."
Learning to program
"The most difficult part was programming my own app from my own idea since I was basically on my own."
"when making your own skill, it doesn’t exactly work the way you wanted."
"I think the most difficult part was making our own project. You could add more tutorials to learn more about the amazon skill."
More time for projects
"The most difficult part was coding on my own, but allotting more time for students to be able to code on their own would be better"
"[sic] Something you can do is give us a day to imagine what our project would be like so I can do the coding easier."
Talking out loud
"the talking part was hard"
"The most difficult part was probably talking because there was a whole lot of people but that was since i’m shy i enjoyed the programming and loved learning about ai and ml :)"
Increase pace
"Nothing was too difficult, it just felt a bit slow at times."
"The difficult part was on some day it was slow but other than that it was good."
User Interface
"The most difficult part was finding some of the blocks when we worked on the MIT app inventor."
"The difficult part is that you must remember what app you’re using and the blocks."
Nothing to improve
"I think the work shop was good I can’t think of how to improve it"
"There was not any particular difficult part of the workshop."
"I didn’t really find any part of the workshops difficult though because we were able to ask for help when needed."
If you did get to use a physical Alexa device, what was it like? Did you enjoy using it? If you didn't, would you have wanted to try your skill on a physical Alexa device? Why or why not?
"I prefer to actually use the device because i feel like its more hands on"
"I liked using my device to test the skills, I haven't been able to test the one I made because it is having trouble sending updates to Amazon. Testing skills on the actual Alexa is very rewarding because I can see skills that I coded working in the same way that Amazons skills work."
Easy to set up
"Using the physical Alexa device was fun and intuitive. It was easy to set up [...]"
Enjoyed exploring device features
"I really enjoyed using it and getting to explore all the features available on it."
"I did enjoy using the Alexa device because it does let you personalize it and you can always ask it questions."
Wanted to see programs on-device
"I wanted too but the only alexa device I have doesnt let me connect to an amazon account :/ or at least I couldn’t figure it out."
"I would have loved to use my skills on a physical Alexa device because I would enjoy to see the difference. Also it seems as if it might be easier to operate. "
Device didn't seem necessary
"I feel like the experience would’ve been the same"
"I did not care if I could try it on a physical Alexa because the simulator was fine."
Teacher Experience Question 1. Did the material in this course change your opinion about AI, conversational AI or teaching AI to students in any way?
Made understanding/teaching AI seem more feasible
"It made me more confident that it can be presented on a Middle School Level."
"Yes, I know now it is more than what I see in movies as far as my understanding"
"Yes it did. In my opinion teaching AI is daunting, however with the materials presented and the step by step tutorials and explanations help me visualize how I can teach and differentiate AI in class"
Teacher Experience Question 2. Do you feel like you gained a better understanding of conversational AI through the workshops? If so, in what ways?
Better understanding of AI in general
"Conversational agents, coding, and application of AI."
"[sic] With the help of tutorial lesson and the explanation of the facilitators, I understand how AI worked."
"Conversational agents, coding, and application of AI."
Yes
"Yes. Understanding the logic behind how it is structured"
"I had no clue how conversational AI works at all, now I have some idea. I liked that math is the biggest part of coding AI"
"Yes I had some previous background in AI and robotics however this activity strengthend me in conversational AI"
Teacher Experience Question 3. What is something new that you bring to this course or would like to add to it?
Extended time/content
"More short tutorials or practice activities that students can do outside of the workshop that will not require more than 15-20 minutes to do"
"If the course was longer in time you can add a more detailed explanation of the blocks while coding, and dig a little deeper into the concepts during the presentations."
More engagement/collaboration
"My students seemed more engaged when the curriculum was hands on and required them to participate in real time."
"more student engagement, interaction which we did"
"More collaboration opportunities with feedback from multiple students."
Teacher Experience Question 4. How engaged were students in the course material? Were there particularly engaging portions of the course?
Doing hands-on work
"When it came to hands on coding they were excited to work after the initial presentations of the daily lessons"
"Students were engaged during collaboration and tutorials (hands-on)"
"[quote given from student] As I was creating my app, I first designed the actually app. Then I went to the blocks. At first I was unsure on how to execute my plan, but after looking through the other projects we had previously worked on, I had an idea about what to do. My app’s purpose is to translate different languages. With my app, I just translated English to Spanish and Spanish to English."
Students were engaged
"Students seemed engaged and excited"
"My students were 100% engaged in the course. They loved coding and the fact that by the end of each day they had a working app made it especially interesting"
"I thought the students were engaged and it only got better as the workshop went on"
Teacher Experience Question 6. What were the most important skills and ideas that your students learned in this course?
Programming
"Programming logic"
"they were introduced to block coding in the least intimidating way"
"they know how machines are programmed and they can program too."
How AI works
"Rule Based AI versus Machine Learning"
"They know more about the … gut of Alexa"
"very difficult concepts(ex: how AI is learning) were presented in a simple and yet not simplistic way."
Societal impact
"How their apps can change the world for the best"
"the understanding that AI is to make lives better or improve it. That technology is a helper and a partner. That because human creates the program, it’s ok to have errors but make sure that you pick up the error and correct it."
Teacher Experience Question 7. What parts of this course would you bring to your classroom and what would you leave behind?
Leave Alexa device
"All of it, except for 1:1 Alexa device"
Keep everything
"I would bring everything."
"I would carry it out just as designed"
"this would be a great course over a longer period of time as is. Students would end the course gaining tremendous knowledge and experience"
Teacher Background Question 1. How comfortable are you teaching computer programming and technology concepts?
Somewhat comfortable
"Somewhat"
"Fairly comfortable"
Comfortable
"Very comfortable"
"Very comfortable with foundational and intermediate CS teaching"
"I am comfortable. I have attended Code.org trainings and PD with Edhesive"
Teacher Background Question 2. Would you describe yourself as tech-savvy?
Somewhat
"I would say I can adapt and learns the technology thru hands on practice and trial and error"
"Maybe"
Yes
"Yes"
"Yes, I am highly adaptable and competent using technology."
Teacher Background Question 3. Have you ever taught your students about AI or Computer Science?
No
"No"
Computer Science only
"Computer Science, but not AI."
"Computer Science."
"Yes to Computer Science."
Yes
"Yes"
"I taught AP CSP last year and we talked a lot about AI"
Great!