Hello there! I’m Yuriko Kikuchi. This page is my final report submission for Google Summer of Code 2024 (GSoC 2024).
Here, I explain the development of new AI-powered features for Rocket.Chat that I worked on during the summer of 2024.
Addendum: I passed the Google Summer of Code final evaluation with this final report! You can get an overview of this project and my key takeaways from the slides summarizing my summer of OSS.
Please check them out!: Slides
Rocket.Chat is an open-source business chat tool. It's a software that allows companies and organizations to communicate on their own platform.
In this project, I worked on developing new features that can be utilized on the Rocket.Chat platform.
My project, AI Newsletter Generator and Publisher, aims to solve the challenge of creating engaging and informative newsletters for special interest groups; marketing and communications by the Rocket.Chat users communities.
By developing this AI-powered app, users will be able to effortlessly produce well-crafted newsletters from raw content using advanced language models. The app seamlessly integrates with Rocket.Chat, allowing users to deliver newsletters.
The main objective of my project was to create an AI newsletter generation and publishing app that seamlessly integrates with Rocket.Chat. Specifically, I aimed to:
-
Implement a prompt engineering system capable of generating different types of newsletters.
-
Develop a slash command interface allowing users to easily generate newsletters directly within Rocket.Chat.
-
Integrate open-source LLMs (such as Mistral, Llama2) to power the newsletter generation process.
-
Ensure the generated newsletters are appropriately formatted and ready for immediate use or further customization.
-
Create a foundation for future developments, including a user-friendly input interface and a newsletter delivery system.
These goals were designed to address the challenge of creating engaging and informative newsletters for Rocket.Chat user communities.
The system works as follows. Steps 5 and 6 are planned for future development:
- The user sends the content they want to include in the newsletter on the Rocket.Chat server.
- The user's input and the fixed prompt embedded in the code are sent to the LLM on the Rocket.Chat server.
- The LLM generates the newsletter.
- The user receives the output from the LLM on the Rocket.Chat server.
- If the user is not satisfied with the content received from the LLM, they can change the input and request regeneration.
- The user chooses which channel to share the received newsletter content in, adds text if necessary, and shares it in the channel.
Before becoming an engineer, I worked as a professional content writer for over four years. As part of that job, I spent two years writing and distributing weekly newsletters to more than 2,000 customers using a membership service.
I saw Rocket.Chat's project as the perfect opportunity to collaborate my experience as a writer with my current work as an engineer. And indeed, I was able to leverage my previous writing experience to bring good output to this project!
The prompt structure is divided into two main parts: User Input and Fixed Prompt. This design allows users to input content specific to their newsletter while ensuring a consistent output format.
Key features include a structured completion approach, style instructions, and few-shot learning examples. For detailed information on the prompt structure and examples, please refer to the document where you can view the full prompt.
In crafting this prompt, I opted for a structured completion approach. The entire prompt is written in a JSON structure, with clearly defined instructions for both the overall newsletter and each individual section.
I chose the structured completion approach for several reasons:
-
Precise Content Control: This method allows for more accurate control over the generated content. By providing a clear structure and specific instructions for each section, we can ensure that the AI model produces consistent, high-quality newsletters that meet exact requirements.
-
Consistency: The structured approach helps maintain consistency across different instances of newsletter creation, ensuring that each output follows a similar format and includes all necessary elements.
-
Ease of Maintenance: This approach makes it easier to maintain and update the prompt as needed. We can easily modify specific sections or add new instructions without disrupting the overall structure.
The structured completion approach has proven effective in my experiments, resulting in well-organized, comprehensive newsletters that consistently meet our quality standards.
The prompt includes specific instructions for each style option:
"style_instructions": {
"maintaining structure": "Use the provided subtitles and itemization, starting each item with a number (1, 2, 3).",
"free-form paragraphs": "Merge all the information into a flowing, conversational style spanning multiple paragraphs. Generate creative, engaging, and eye-catching subtitles that will draw the reader in. Present the content in a relatable manner, using appropriate transitions and connective phrases to create a natural flow. Avoid bullet points and numbered lists."
},
This ensures that the generated content adheres to the user's preferred style consistently throughout the newsletter.
This style selection creates the following differences. These two patterns have the same input for other parts, with only the style changed:
- For "maintaining structure":
**Your Questions Answered**
**Q: How do I set up Voice Control?**
A: Setting up Voice Control is a breeze! Simply follow the step-by-step guide provided in the user manual, and you'll be commanding your device with your voice in no time.
**Q: Can I collaborate on files in real-time?**
A: Absolutely! With our Online Collaboration feature, you and your team can work on files simultaneously, making collaboration more efficient than ever.
**Q: How often does the Automatic Backup occur?**
A: The Automatic Backup feature is designed to run regularly to ensure your data is always protected. The frequency can be customized to suit your needs.
- For "free-form paragraphs":
**Your Questions Answered**
Wondering how to set up Voice Control? It's a breeze! Just follow the simple instructions in our user manual or visit our Online Help Center at https://www.example.com/help.
Can you collaborate on files in real-time? Absolutely! With our Online Collaboration feature, you and your team can work on the same document simultaneously, making teamwork more efficient than ever.
How often does the Automatic Backup occur? The frequency of backups can be customized to your needs, ensuring that your data is always safe and secure.
These two patterns clearly demonstrate how differences in style in the output arise depending on the user's choice.
To improve the quality and consistency of the output, I incorporated few-shot learning into the prompt. These examples help guide the AI in generating newsletters that match the desired tone, structure, and content quality.
I used two excellent, publicly available newsletter examples. The newsletter examples were sourced from Audienceful, using only the text.
In my experiments, I found that the Mistral Large 2 model (with 123 billion parameters) performs exceptionally well with our structured completion approach. I also tested with Mistral NeMo, which offers a maximum context length of 128k tokens.
Key observations:
-
Effectiveness Across Model Sizes: The structured completion approach maintained consistent performance across different model sizes, from the larger Mistral Large 2 to the more versatile Mistral NeMo.
-
Output Quality Consistency: Despite variations in model size and capabilities, our approach consistently produced well-formatted and coherent newsletters.
These findings highlight the robustness and versatility of our structured completion approach, crucial for the long-term viability and scalability of our AI Newsletter Generator and Publisher project.
I developed slash commands to integrate the newsletter generator with Rocket.Chat:
/newsletter generate [user's input]: Command to generate a newsletter/newsletter help: Returns a list of commands related to the newsletter app
For the generate command, I create the command by separating the item name and user input with the | symbol. Here's an example of the generate command:
/newsletter generate product_name: "Acme SuperTech Pro" | new_features: "1. Voice Control, 2. Online Collaboration, 3. Automatic Backup" | benefits: "1. Hands-free convenience, 2. Seamless teamwork, 3. Peace of mind" | faq: "1. How do I set up Voice Control?, 2. Can I collaborate in real-time?, 3. How often does Automatic Backup occur?" | additional_info: "Help Center: https://example.com/help, Email: support@example.com" | team_name: "Acme Team" | style: "free-form paragraphs"
Below is a demonstration video of this process:
rocket.chat_final.submission.mp4
Please check the high-quality video here: YouTube
-
Initialize App Configuration: Set up the application to run on Rocket.Chat servers.
-
Implement Basic Slash Command Functionality: Enable sending user input to LLM via slash commands and receive newsletter output.
-
Develop User-Friendly Input Interface: Create a UI for users to input newsletter content easily.
-
Implement Newsletter Delivery Feature: Enable distribution of LLM-generated newsletters to specific Rocket.Chat channels or users.
-
Expand Scenario Selection: Extend the current "Product Release Article" scenario to include the "Community Engagement Newsletter" scenario for newsletter creation.
- A feature to regenerate LLM output: If the user is not satisfied with the LLM output, they can modify part of the input and regenerate the LLM output.
- Expansion of Prompt Types: Further advance prompt engineering to increase the types of scenarios that can be handled.
Prompt Engineering was the area where I invested most of my effort during the GSoC period.
With my mentors’ advice, over two months, I conducted more than 40 experiments to develop the current prompt structure. You can check the process here: Prompt Engineering Experiments.
Prompt Engineering is a continually evolving field. I plan to keep up with information on prompt engineering and continue experimenting to obtain better output from LLMs.
GSoC 2024 provided me with an exciting opportunity to lead a project entirely in English. Initially, conducting meetings in English was a new challenge, but I quickly adapted and thrived in this global environment.
I developed effective strategies, such as preparing detailed meeting notes, which not only structured our discussions but also rapidly improved my ability to communicate complex technical ideas in English. This approach led to clear and productive conversations with my mentors.
My mentors' supportive attitude was instrumental in my growth. Their encouragement allowed me to fully engage in discussions, ask questions, and express my ideas confidently. As a result, our meetings became enjoyable and highly productive collaborative sessions!
Thanks to the GSoC 2024 opportunity, I discovered the joy of participating in OSS in English and communicating with contributors from around the world! I've gained valuable skills in cross-cultural communication and global teamwork, which I'm eager to apply in future projects. I will continue to actively participate in English-language projects from now on! 🛫
I was fortunate to have three mentors:
They provided me with advice on everything from technical aspects to project management, both during meetings and asynchronously. Their constant reassurance that "We are here to help you" allowed me to enjoy GSoC 2024 with peace of mind. Thank you for a wonderful summer! 🌞
And to my fellow GSoC 2024 participants as Rocket.Chat contributors! Rocket.Chat provided opportunities to foster connections between contributors, calling us the "AI Club" and offering weekly meeting spaces. Connecting with AI Club members allowed me to ask questions, get ideas for solutions, and engage in AI discussions. The AI Club members were a great source of support. From the bottom of my heart, thank you 🤝
Rocket.Chat is a welcoming team where everyone is friendly and ready to lend a hand when you're in need. I'm so glad that my first OSS commit was with Rocket.Chat! I encourage you to dive into Rocket.Chat too 🚀
As I reflect on my GSoC 2024 journey with Rocket.Chat, I'm filled with a sense of accomplishment and gratitude. This project has been a significant milestone in my development as both a software engineer and an open-source contributor.
Key Achievements:
- Implemented a prompt engineering system for generating "Product Release Article" newsletters using advanced LLMs.
- Developed a slash command interface integrating seamlessly with Rocket.Chat.
- Optimized our system for use with Mistral Large 2 through extensive LLM experiments.
- Laid the groundwork for future enhancements of the AI Newsletter Generator.
Looking ahead, I'm excited about the potential impact of this work on the Rocket.Chat community and eager to continue contributing to its development.
I'm deeply grateful to my mentors, the Rocket.Chat team, and the GSoC program for this transformative opportunity. This experience has reinforced my passion for open-source development and AI applications, and I look forward to applying these learnings in my future endeavors.
If you're interested in this project or in me, please feel free to contact me 📨: