Skip to content

Instantly share code, notes, and snippets.

@bishoy-magdy

bishoy-magdy/gsoc-2022-work-product.md

Created September 10, 2022 16:48
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bishoy-magdy/c6fb86eebf9adf8eba94f7b7e3071e43 to your computer and use it in GitHub Desktop.
Save bishoy-magdy/c6fb86eebf9adf8eba94f7b7e3071e43 to your computer and use it in GitHub Desktop.
Download ZIP
GSoC'22 - Bishoy Magdy Adeeb - Final Report
Raw
gsoc-2022-work-product.md

1

GSoC'22 Joplin - Email Plugin - Final Report

Most Joplin users are researchers or workers in different fields. It's usually the university or company dealing with them via email, whether it's assignments, essays, or reports, so I built a plugin to add the ability to fetch email messages (including attachments) and convert them to Joplin notes in various formats, either by monitoring any new or unread messages from a specific email address or a specific mailbox by using IMAP or by uploading downloaded email messages to the plugin without having to be logged in.

Demo of a feature that locates converted messages to specific notebooks or adds tags to the note by adding @ or # in the email subject or first line of email content and forwarding the email to yourself.

demo.mp4

Demo of a feature that converts emails (including attachments) to notes without having to be logged into the plugin.

demo.mp4

Achieved

All major features that were suggested by mentors and community members during community bonding, as well as all their suggested modifications to my proposal, were implemented. In addition, as the project progressed, I suggested another feature on the forum, which was to add the ability to convert emails to notes without logging in. I implemented this idea after the community liked it.

These are the main features that have been added:

  • Login using the predefined IMAP config or log in manually.

  • Monitoring and fetching any new or unread messages from a specific email address.

  • Monitoring and fetching any new or unread messages from a specific mailbox.

  • Send the converted message to specific notebooks or add tags to the note by using @ or # in the email subject or first line of email content and forwarding the email to yourself.

  • Convert emails (including attachments) to notes without having to be logged into the plugin.

  • Convert email messages to notes in different formats (HTML, Markdown, Text).

  • Show attachments in different styles, whether they are in the form of a Table or Links.

Detailed Timeline

Week 1 (weekly Report)

  • Committed initial push in a repository.
  • Added a new menu item in Tools.
  • Added a new toolbar button in Note Toolbar.
  • Added the exportation types in the settings.

Week 2 (weekly Report)

  • Added a login screen (view Panel).
  • Added close behavior to the panel.

Week 3 (weekly Report)

  • Added the types of messages in webview using type predicates.
  • Added predefined imap default configurations for some email services.
  • Parsing email and returning the default configuration object appropriate for the type of email provider to make an IMAP connection.
  • Handling emails that start or end with a space.
  • Handling emails that are written in uppercase.
  • Added unit test env (JEST) and added unit tests for the emailParser function.

Week 4 (weekly Report)

  • Installed IMAP and its requirements and made the first IMAP connection in the plugin.
  • Added the manual connection screen.
  • I read about RFC 822, the message email format, and the message parser behavior.
  • I did a stress test for more than one email provider, which is to monitor the mailbox every 5 seconds if there is a new message from a specific email to check that there is no security problem that could have arisen.

Week 5 (weekly Report)

  • Created the Main Screen.
  • Open/Close Monitoring based on from field.
  • Added a Logout button
  • Created an error alarm when an error is caused while logging or monitoring.
  • I posted an additional feature in the forum to convert email to notes without logging into the plugin, and I added this feature to the timeline.

Week 6 (weekly Report)

  • Added the feature of fetching the message data from IMAP.
  • Installed the email parser.
  • Added the feature of parsing the HTML message and extracting the body using DOMParser.
  • Posted the converted emails to Joplin (for testing).
  • I made some changes to my PR based on a code review by mentors.

Week 7 (weekly Report)

  • Added the feature to save the last change on the screen (main screen) when the plugin is closed.
  • Created email parser class and added the unit test for the email parser class.
  • Added extracting plain text with links from the HTML message.
  • Created the class responsible for posting notes to Joplin.

Week 8 (weekly Report)

  • Added inline attachments to note.
  • Added a feature of converting emails (including attachments) into notes without having to log into the plugin.

Week 9 (weekly Report)

  • Added attachments link to note.
  • Added a feature to locate the note in the notebook and add tags through the email subject.
  • Added a few unit tests for the above feature.

Week 10 (weekly Report)

  • Wrap subject direction in RTL languages.
  • Added the unit tests for the noteLocationBySubject function.
  • Added export type, attachments style, include attachments features in the upload email screen (full control).
  • Added attachments style in the plugin setting
  • Made changes according to the feedback on my PR.
    • Fixed nested folder name.
    • Removed the feature of local marking messages.
    • Added the feature of marking messages using the email providers.
    • Fixed If 'has_more' is true in Joplin tags or folders.
    • Refacted TempFolder mechanism.

Week 11 (weekly Report)

  • Added the feature of monitoring a specific mailbox or folder.
  • Added the feature of multi-user accounts.
  • Added the feature of determining the folders and tags of the note based on the subject and the first line of email content.
  • Added the feature of remembering the last state of the plugin even if the user closes the Joplin and reconnects when the plugin runs again.
  • I've made some improvements in parallel
    • Added a loading spinner to all async buttons.
      • replaced the embedded email text function in the email parser with html-to-text ) (higher in efficiency and extraction).
    • I replaced the 'switch...case' in the panel bridge with 'if else' to make the code more readable.

Week 12 (weekly Report)

  • I created a PR that contained the last part of the project, and the PR was successfully merged.

  • This is what was modified and added during the code review on the PR

    • I fixed that if there are no folders mentioned, will automatically locate the notes in the 'email messages' folder.
    • Annotated 'attachmentsStyle' parameter to ''Table | Link"
    • Added some comments in the 'noteLocationBySubject.ts' file.
    • In the 'webviewApi' attributes, I changed the snake case to camel case.
    • Fixed some style labels on the main screen.
    • Renamed the addInlineAttachments function to addInlineImages to make it more clear.
    • Added tooltips beside the 'from' field
    • Added a refresh button beside the mailbox and notebook selectors to update them if there is a new mailbox or notebook.
    • Changed monitoring time from every 5s to every 1 minute.
    • Fixed plugin name in the package.json

  • The readme file has been added to the repository containing the features of the plugin, how to use it and some important notes

  • The project was published to npm and released on github.

Code Contributions

This link contains all successfully merged pull requests link.

What's left to do

There's already a long list of suggested features from the community. Some examples:

  • Marking converted email messages as to-do.

  • Removing Re, FW, Fwd, etc. from the forward email subject.

  • Removing mention folders or tags from the subject of the forward email.

  • Convert email messages to pdfs and putting them at the end of the note.

I plan to continue contributing to the plugin. Most of these ideas will be implemented sooner or later. Additionally, I will create a post in the forum for community feedback and any suggestions on improving the final UI.

Challenges and what I've learned

This project is the first large-scale project I've worked on so far. I faced many challenges, such as connecting to IMAP, fetching and parsing email messages, reading complex information about RFC 822, and organizing time between university final exams and GSoC, and I learned a lot of things, most notably:

  • Simplicity is the ultimate sophistication (Although there are other ways to implement this project, the simplest is definitely the best).

  • Dive deeper into complex info such as IMAP and messages of type RFC 822.

  • The importance of having unit tests in the project.

  • Break the ice. When dividing a large project into small parts and regularly working on those small parts, the end result will be a large and powerful project.

  • Consulting with the community and exchanging opinions is a very important thing, and it helps produce a better product.

  • Enhancing communication skills.

  • Time management.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment