This page outlines my time at Google Summer of Code on the Sugarlabs organization, working on Make Your Own Lesson plans from May to August.
Contributor: khadar
Project GitHub: Github
GSoC project page: GSoC
Detailed explanation of code and how to setup locally: Code
There are some lesson plans in Music Blocks which are made by the teachers but for creating a lesson plan it takes time and Teaching the intersection of music and programming is new and challenging. Generative AI offers an opportunity to assist teachers in creating new and fun lessons that suits their curriculum. So, we will be using Generative AI concept and creating a chatBot which can generate lesson plans for teachers.
For teachers it takes nearly 8 hours to create a lesson plan we will be reducing it to 2 to 3 hours our chatbot will be providing a base for the lesson plan so the teachers can proceed from their further.
- Chainlit for frontEnd.
- llama3 model.
- chromadb database to store our data.
I learned a lot this summer, from coding and project to collaborating with an amazing mentors walter and devin.
- Training techniques - For training our model there were 2 to 3 methods like RAG, fine-tuning. I first tried with fine-tuning but the results were not upto mark and also we need to have more resources and a big dataset for that also it was a complex process. So, then we shifted to RAG which gave us the best results.
- Open Source Models - There are so many open source models in hugging face and ollama platforms. I learnt how to use these models and how to get the quantized models for these models.
- Conversational ChatBot - I have learned how to build a conversational ChatBot so the user can have engaging and dynamic interactions, receiving personalized responses and assistance based on their queries.
- Music Blocks - I learned some of the music concepts and built some projects in Music Blocks.
- Chunking PDF - The dataset for this project were pdf's so I trained my model using these pdf's by dividing them into semantic chunks so our model can understand it better.
- Chat Interface - Teachers can ask for lesson plans using natural language.
- Narrow Scope - Topics outside of music and programming are politely dismissed.
- Song - Every lesson is accompanied by a list of suggestions for songs that should work well with that lesson.
- Midi Files of that song - Users are provided a link to a MIDI of the song that they select. We are currently working on MIDI βMB conversion, so that the MIDI may be imported if desired.
- Conversational Bot - Ask follow up questions for elaboration.
- Suggestion for code structure β We are working on a feature to show the structure of the Music Blocks code. Since our code is visual this is not a trivial task.
- Resources - For the lesson plan we will be getting the links of the topic from wiki, youtube and the google search so the teachers can refer them as well.
- Doc File - The user can download the lesson plan as a docx file and can edit it according to their curriculum.
My mentor Walter and Devin were always available for questions, willing to explain the concepts, reviewing my work and giving me new ideas. Also, helping me set a new direction whenever I was blocked. I'm very grateful for his continued assistance, understanding, and knowledgeability!
Lastly, thanks to the GSoC and Sugarlabs for setting up this opportunity π
- Setting up the models locally llama2 and llama3 models.
- Building an conversational chatBot.
- Adding some good features into chatBot like history, song etc.
- In the intitial phase the answers given by our chatbot was a bit complicated for 8 to 10 years children to understand. So, I have trained my model and given prompts so that the answers which was generated by our chatbot be simple in a way that 8 to 10 years child can understand.
- As our chatbot should have a feature of conversational memory when we change the topic our chatbot must identify that it is a new topic and it should react accordingly initially it used to give all the responses from first question to last question when we ask a new question from another topic so I have removed this bug now our model can process our history.
- Also, our chatbot should be capable of giving a docx file for the users so that teachers can download it and edit it according to their curriculum.
- Also, for the songs initially we are asking the user for the index but as mentor suggested to put buttons for songs I have approached the chainlit organization and I have implemeted it now we will be giving buttons for the users so when they click on any particular song button based on that song the lesson plan will be generated.
- All this testing is done by mentors while I am doing the project.
In the community bonding period I have attended all the meetings for our projects and clarified my doubts, planned how to complete the project and tools that are open source that we can use for this project and the AI model which we have to use for this project.
Created multiple pdf's(data to train the model) from the guide and some lessonplans of Music Blocks I have separated these files into many small multiple pdf's so for our model is will be easy to understand as we will be training the model with each pdf as one chunk of pdf. I spent a lot of time in this period familiarizing myself with the codebase and solved 2 to 3 issues.
π Detailed Blog - CommunityBondingPeriod
I have trained the model following the provided guide and tested it with various prompts to verify its accuracy in generating responses. The model performed well, delivering accurate answers based on the training data.
To enhance user interaction, I developed a conversational ChatBot capable of retaining the context of all previous conversations. This memory feature allows the ChatBot to provide more coherent and contextually relevant responses, improving the overall user experience.
π Detailed Blog - week1
I have narrow down the scope of our chatBot like it can only answer Music Blocks questions if we ask any other question it will politely say this is Music Blocks chatBot please answer music Blocks questions and trained my model with some lesson plans for training I have used nomic-text-embedded model and for storing the data I have used vectorestore/chromadb.
Me, devin and walter tried to ask an Interesting question to confuse the chatBot like can you create a project on weather of india using Music Blocks but it responded correctly and I changed my model to llama3 which was recently released and responses from that model were interesting and accurate.
π Detailed Blog - week2
There were some lesson plans were missing which were created by devin which were in the website of music blocks and I have added those into my dataset and trained the model with these lesson plans as well.
I have posted my code on GitHub - MusicBlocks AI.
I have started working on the song feature for generating the lesson plan and adding multiple features to our chatBot.
Detailed Blog - week3.
I have added 2 features in our chatBot.
β¨ Song Feature - When the user asks to generate a lesson plan on any particular topic then our model has to provide the user a table of colums containg the index, song, and why that song is best for the lesson plan and ask the user to select the index of the user to select the index of the song. When the user selects the index of the song then our model should create a lesson plan on the topic user asked based on the song provided.
β¨ Midi Files - MIDI files are digital music files that store information about notes, timing, and instruments, allowing electronic devices to play music. They don't contain actual sound but instructions for generating sound. In the lesson plan I have added these midi files of the song that user selected. User can directly upload this into Music Blocks and it works like a magic and it gets all the blocks which are used in Music Blocks and arrange them to get the melody.
Detailed Blog - week4&5.
I have corrected a error that was found by devin regarding the history part and solved it and I have added some features that will enhance the user experience.
β¨ Login page - I have used the frontEnd of chainlit and I have added the login page for our chatBot.
β¨ User Review - I have added an option of review for the answer our chatBot generated so the developers will get an idea how our model is working and what are the code changes we have to do.
β¨ Storing the history - The user will generate the lesson plan but he store it some where right so I have added in the website a place so that user store all the previous conversations so when ever he wants to access them he can just go and refer them.
Detailed Blog - week6&7.
If we are able to provide a feature like the user can download the lesson plan and he can edit it like a doc file and the links of the midi files and the wiki pages should be as links in the doc so that the teacher can directly access them.
Also added a midi file gif and the user will know how to use the midi files in the music blocks.
Detailed Blog - week8&9
I have approached the chainlit open source program regarding the buttons for the webpage and they have implemented it and found one issue there also raised the issue there also included the buttons for the songs in our chatbot. I have also written the final Blog and also for the hoisting I have started research.
Detailed Blog - week10&11