Skip to content

Instantly share code, notes, and snippets.

@ajayns
Last active Aug 12, 2018
Embed
What would you like to do?
Final report by Ajay NS for Gatsby Source Plone, a GSoC 2018 project with Plone

Final Report for Google Summer of Code

Plone GatsbyJS Integration

Gatsby is a blazing fast static site generator for React. gatsby-source plugin is a Gatsby plugin that pulls the whole content tree from the plone.restapi and makes it available for querying via GraphQL in a hierarchical data structure, handling all native content types including images and files. gatsby-starter-plone is a Gatsby site that uses the plugin to source the data from a Plone site, showcasing all use cases with clear documentation for each feature. This is designed to work as a starter for kickstarting development with Gatsby and Plone. Relevant documentation for each feature as well as a narrative tutorial for the whole building process has also been written.

Using the gatsby-starter-plone, users can directly get into building Gatsby sites with Plone and use the narrative documentation as reference for each feature implemented. gatsby-source-plone includes feature-wise documentation for information on how it works and has CI and tests for long term reusability and open source contributions.

Get the code

The plugin to pull data from a Plone site to a Gatsby project, gatsby-source-plone: https://github.com/collective/gatsby-source-plone

Gatsby starter to kickstart development with gatsby-source-plone, gatsby-starter-plone: https://github.com/collective/gatsby-starter-plone

Live Demos

Documentation for gatsby-source-plone (Gatsby site generated from mock Plone site): https://collective.github.io/gatsby-source-plone/

Live Demo for gatsby-starter-plone: https://collective.github.io/gatsby-starter-plone/

Narrative Documentation for implementing each feature of plugin: https://collective.github.io/gatsby-source-plone/tutorial/

Features

  • gatsby-source-plone uses a search traversal method to get full content tree from a Plone site, Gatsby cache is used to optimize this process.
  • All the data from plone.restapi is passed on to nodes created by Gatsby for each content object, which can be later queried via GraphQL in a Gatsby project.
  • Files and images nodes are created separately, backlinked to node which contain them. This backlinking along with HTML serialization is handled by the plugin.
  • Documentation and samples have been written for RichText component which processes these backlinks (images and files) and serialized HTML to be handled and displayed by Gatsby site.
  • gatsby-starter-plone implements all the features of the plugin in a Gatsby site, allowing projects to be kickstarted super-fast.
  • Narrative documentation which guides you through implementing each feature of the plugin in a Gatsby project was written with example code from the starter.

Pending

Future plans

Pull Requests

Week 1

What's achieved so far:

  • As recommended by Timo and Asko, I've been following a documentation-driven approach to development, hence I've started out with a basic intro and detailed documentation about implement of the plugin using the @search traversal method.
  • Been exploring and playing around with Gatsby itself and it's plugins, also started implementation of @search traversal method. (collective/gatsby-source-plone#31 )
  • I've also refactored the existing code in gatsby-node.js , adding helpers to fetch data and create content digest, also cleaned it up.
  • Since it's planned to be well documented, quality npm package, I did a bit of research on them as well, and came up with updating the .gitignore and .npmignore files so as to follow best practices. (collective/gatsby-source-plone#28 )
  • To follow code quality best practices, _Prettier _has been configured and documented. Asko helped greatly in setting up tests, make build and continuous integration with Travis, while moving to docker for running the Plone site. (collective/gatsby-source-plone#26 )

What's pending, and planned for next week:

  • Right now, first priority would be setting up a md which lists the changes made, features added, bugs fixed, breaking changes etc for each version rolled out.
  • Work on @search traversal method is in progress and is expected to be done real soon, will test and make PR. This already has been documented in detail, only code implementation remains.
  • I'll also be tackling the issue: that **gatsby-source-filesystem ** name clash with node of type File, this would be solved, documentation would be updated. ref:collective/gatsby-source-plone#21
  • Further documentation is planned to be written on tree hierarchy.
  • Asko has been working on getting the site auto-published to gh-pages, and also commands to import and export content from a Plone site using jq. I'd be working on adding docs wherever required for these as well, so that a user has full control and understanding of what all tools are used/available.

Week 2

What's achieved so far:

  • While playing with Gatsby and it's plugins I noticed this issue that File type node clashes with nodes created by gatsby-source-filesystem and this could prove an issue as that type is reserved. With discussion, Plone was prepended to each content object type so as to prevent this. Docs were also updating taking this in consideration. (ref:collective/gatsby-source-plone#45)
  • Full content tree traversal using the @search endpoint approach was implemented. Code was refactored to improve code readability and also to optimize modularity. Furthermore, logging option was added so that in development, progress of the plugin in action can be tracked. (ref:collective/gatsby-source-plone#49)
  • Documentation for plugin options were added in a format similar to what's followed in the Gatsby official docs/site.
  • First feature (tree hierarchy) was merged with tests. Tree hierarchy using children and parent property of content objects, also added content to test this with in fixtures and also worked on an example use case scenario in the gatsby-starter-default. Worked on and understood a bit on Robot testing framework and wrote a test to check if the data exists in the result. (ref:collective/gatsby-source-plone#51)

What's pending, and planned for next week:

  • PloneSite node for @search traversed data is in progress. Most of the work is done except a example use case in the test project. (ref:collective/gatsby-source-plone#64)
  • Documentation for tree hierarchy illustrating it's usage in the test project is suggested and planned, could finish up taking tips from the mentors. (ref:collective/gatsby-source-plone#65). Planning to wrap this up as soon as possible and get on with the planned timeline.
  • Best way to handle authorization options for the plugin was indiscussion. Research was done on multiple approaches and we've discussed and settled on the best method. Work on this has already been started. The idea is to use bash, curl to get JWT which used with .env variables would automatically add it to gatsby-config.js and make request with auth headers. This would be the main goal for this week.
  • Fetching and using navigation and breadcrumb data from the Plone site has already been discussed. Work on this with an example and docs is also planned to start for this week.

What's in discussion and research:

  • File handling by the plugin is underdiscussion and an issue has been opened in the Gatsby community as well. The current idea that looks good is to pull files to Gatsby cache and link them from there to the project, this is done with help of createRemoteFileNode helper.
  • Testing is planned to be improved with better acceptance tests, so looking into Behavior Driven Development and its features.

Week 3

What I've achieved so far:

  • Fetches the Plone Site node separately and processes it. This is used to dynamically create an index page, and it was modified to work like the previously existing index page. (collective/gatsby-source-plone#64 )
  • @search endpoint returns items in batches of 25, which means a single API call wouldn't get us all of the items. A recursive approach was used to fetch data from API until the all the items were traversed (collective/gatsby-source-plone#68)
  • Authentication options were discussed in depth and the best approach was found. Using JWT, dotenv, a safe, efficient method was implemented. This was followed by good documentation to help out users handle authentication in their own Gatsby projects without much setting up work, while also being safe. (collective/gatsby-source-plone#69 andcollective/gatsby-source-plone#73)
  • Documentation that was pending for tree hierarchy was written, with sample generalized gist explaining how it works and also a complex use case from the gatsby-site. (collective/gatsby-source-plone#74 )
  • Overall code readability and quality was improved by reviewing and refactoring the code with best practices and referring guides. This makes sure standards are maintained and long term use of the plugin. (collective/gatsby-source-plone#70 )

What's pending, and planned for next week:

  • Expansions including breadcrumbs, navigation and so on are yet to be handled and data passed on to the nodes. Currently in progress, encountering a few bugs. (collective/gatsby-source-plone#77 )
  • Next step after getting it working would be to add documentation, use case in the gatsby-site
  • Issues noticed while experimenting with the plugin are to be noted and fixed. Currently there exists an issue with subsites noted down here: (collective/gatsby-source-plone#75 )
  • Further testing will be done, bugs to be noted and resolved. Feedback will be taken and all necessary changes, fixes will be worked on before the First Evaluation period so that a working plugin which traverses the whole Plone site using @search traversal developed.

Week 4

What's achieved in the previous week:

  • Fixed the issue while using subsites in Plone, both the baseUrl and the folder at base returns as content object. rootNode is separately processed now and is filtered out from content object list. (ref:collective/gatsby-source-plone#75)
  • Expansions support added to the plugin. It can now get data of expansions along with the usual data retrieved from plone.restapi. This can be configured in gatsby-config.js. (ref:collective/gatsby-source-plone#77)
  • GraphQL prevents special characters in it's object properties which caused @id and @components and such properties being unavailable for querying. A fix was made by replacing @ with _, in all places, including expansions. (ref:collective/gatsby-source-plone#85)
  • Example use case was built for breadcrumbs expansions with updated tests. In this process the slug method of page creation was replaced by using _path which are processed slugs obtained from the plugin itself. This greatly simplifies page creation and linking. (ref:collective/gatsby-source-plone#85)
  • Documentation was added for expansions, including explanation and configuration in gatsby-config.js (ref:collective/gatsby-source-plone#84)

First Evaluation:

  • I'm glad to say, even thought we did face quite a bit of issues on the way unexpectedly and came up with features while buildling, we're still
  • Post first-evaluation, the main things are planned are a recursive approach for traversal and file handling in the plugin. These are crucial and also tricky so will be researching quite a bit before moving forward.

Week 5

What's achieved in the previous week:

  • We've made significant progress in planning and implementing the recursive traversal algorithm. Initially, the code was refactored and modularized to improve readability and to add new features easily. (ref:collective/gatsby-source-plone#87)
  • After planning and a lot of testing, the recursive algorithm was implemented using a Breadth-First Search approach. (ref:collective/gatsby-source-plone#88)
  • Documentation was added for recursive traversal. (ref:collective/gatsby-source-plone#91)
  • Minor issues and problems faced on the way were fixed in the process itself.
  • Search parameters feature was added as a plugin option which allows you to query for certain content objects that are returned depending on the query. Documentation was also added for this. There was a bit of an issue converting query to strings/encoding to URI components, but then later with advice from @cekk, axios object parameters were used. This took some effort keeping on refactoring the code, but we have something that works smooth now. (ref:collective/gatsby-source-plone#90)

What's planned for the coming week:

  • I've already explored Gatsby v2 and also made contributions to the GatsbyJS project itself in helping their projects shift to v2. So the next step would be to port the plugin and tests to v2.
  • Further more, file handling is a major feature that'll require quite a bit of efforts. So research on that will be started along with examples to play around with. I'll be interacting with the Gatsby community itself to get guidance and tips on moving forward with this.
  • Unit testing is also under discussion and will be looking to implement basic testing from scratch, after porting to v2.

Week 6

What's achieved in the previous week:

  • Documentation was added for search parameters, for the users to get a good idea of how it works, and how to implement using gatsby-config.js plugin options. (ref: collective/gatsby-source-plone#93)

  • Our project plugin and the tester have been migrated to Gatsby v2, following the guide and fixing any breaking changes. Also encountered a Babel setup issue which wasn't occuring locally but was occuring for travis, and this was fixed doing proper research about the issue. (ref: collective/gatsby-source-plone#94)

  • Being not very familiar with testing, did research and played around with Jest for unit testing. A basic implementation was added for the plugin, to run tests for the helper functions. (ref: collective/gatsby-source-plone#97)

  • With @datakurre, a few improvements were made and code was cleaned up. Also, added subfolders and testing for deep folder structure.

What's planned for the coming week:

  • File handling handling is an important aspect that's yet to be implement, this week will mostly focus on researching about it's implementation and finding the best way to handle images, and other files.
  • It is planned for images to use gatsby-plugin-sharp, and for other files to be added as file nodes which are linked to files pulled locally by Gatsby.

Week 7

What's achieved in the previous week:

  • Best approaches to handling remote files and images were experimented and researched. Initially schema from @type endpoint of rest.api was planned to be used for checking for images and files. This method was unusable because of a pending issue in rest.api. (ref: collective/gatsby-source-plone#104)
  • Using image attribute from content, images were detected and configured to be handled by gatsby-source-filesystem. Gatsby-transformer-sharp was used to get responsive, lazy loaded, Sharp optimized versions of these. (ref: collective/gatsby-source-plone#102)
  • Using a similar method as used for handling images, files are handled checking for a file attribute. It's cached using source-filesystem, and it's made available to download using public URL, and configuration of gatsby-source-filesystem. (ref: collective/gatsby-source-plone#106)
  • Documentation was added to explain this workaround/configuration for source-filesystem and also a bit about file handling patterns. (ref: collective/gatsby-source-plone#108) gatsby-source-filesystem was updated to v2.

What's planned for the coming week:

  • Transforming HTML text fields and query types (collections) are to be processed by the plugin before making it available for GraphQL
  • Research on Plone tiles and it's support, along with configuring a client based search engine for the gatsby-site.

Week 8

What's achieved in the previous week:

  • Before getting into implementing new features, previous features were refined. Gatsby-source-filesystem was updated to v2 (ref: collective/gatsby-source-plone#109)
  • Documentation for file handling was added, describing gatsby-source-filesystem, it's usage and helper function createRemoteFileNode, and how it can be effectively used to process files and images. It also explains the workaround used to expose the publicURL field in GraphQL by using gatsby-source-filesystem on a static folder. (ref: collective/gatsby-source-plone#108)
  • The HTML parsing feature was implemented with examples and tests. React-html-parser was used to parse HTML content from strings to nodes, and react-serialize to allow it to be passed through GraphQL fields. Relative local links were replaced with gatsby-links during processing. (ref: collective/gatsby-source-plone#113)

What's planned for the coming week:

  • We're currently planning to have a briefing tomorrow on the goals to complete for the last month of coding, shipping a final project by the final evaluation period. Once we have that streamlined, I'll focusing on achieving these tasks and getting a fully functional plugin ready as soon as possible!

Week 9

What's achieved in the previous week:

  • The Babel transpilation setup of the plugin was updated to use Babel 7 (which is used by Gatsby v2) (ref: collective/gatsby-source-plone#120), along with this npm package setup and configs were updated to as to make the package cleaner and use the latest scripts. (ref: collective/gatsby-source-plone#122)
  • @datakurre did a great job helping out and implementing the images handling inside HTML content. Backlinks were used to efficiently handle this feature. Along with this, a RichText component was initialized, which would process HTML content inside any nodes and handle files, images, relative links accordingly. This a crucial step in working of the plugin with Plone sites. (ref: collective/gatsby-source-plone#121 )
  • It was noticed that when content objects have a large number of children, batching was done with plone.restapi. So when handling collections, this was also fixed by using a iterative approach to fetch all children. For collections, seen[path] approach was used, to save the traversed paths, preventing them from being traversed again in the recursive algorithm. (ref: collective/gatsby-source-plone#125, collective/gatsby-source-plone#126)
  • Furthermore, documentation was added for HTML parsing and handling collections features. (ref: collective/gatsby-source-plone#128)
  • The tests/gatsby-starter-default was updated with fixtures for events and collections.

What's planned for the coming week:

  • The GSoC project board is being followed to pickup and complete tasks before the end of the coding period.
  • Currently, the main goals pending would include narrative documentation for usage of the plugin and setting up a gatsby-starter-plone which would serve as easy to use starting point for Gatsby with Plone.

Week 10

What's achieved in the previous week:

What's planned for the coming week:

  • The major goals would be to work on implementing each feature of the plugin in gatsby-starter-plone and write documentation side by side.
  • Since it's the last few weeks of coding, planning to speed up the process and work on shipping the starter as soon as possible along with the narrative documentation.

Week 11-12

What's achieved in the previous week:

collective/gatsby-source-plone#140, collective/gatsby-source-plone#139 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment