Skip to content

Instantly share code, notes, and snippets.

@ajayns
Last active Aug 12, 2018
Embed
What would you like to do?
Final report by Ajay NS for Gatsby Source Plone, a GSoC 2018 project with Plone

Final Report for Google Summer of Code

Plone GatsbyJS Integration

Gatsby is a blazing fast static site generator for React. gatsby-source plugin is a Gatsby plugin that pulls the whole content tree from the plone.restapi and makes it available for querying via GraphQL in a hierarchical data structure, handling all native content types including images and files. gatsby-starter-plone is a Gatsby site that uses the plugin to source the data from a Plone site, showcasing all use cases with clear documentation for each feature. This is designed to work as a starter for kickstarting development with Gatsby and Plone. Relevant documentation for each feature as well as a narrative tutorial for the whole building process has also been written.

Using the gatsby-starter-plone, users can directly get into building Gatsby sites with Plone and use the narrative documentation as reference for each feature implemented. gatsby-source-plone includes feature-wise documentation for information on how it works and has CI and tests for long term reusability and open source contributions.

Get the code

The plugin to pull data from a Plone site to a Gatsby project, gatsby-source-plone: https://github.com/collective/gatsby-source-plone

Gatsby starter to kickstart development with gatsby-source-plone, gatsby-starter-plone: https://github.com/collective/gatsby-starter-plone

Live Demos

Documentation for gatsby-source-plone (Gatsby site generated from mock Plone site): https://collective.github.io/gatsby-source-plone/

Live Demo for gatsby-starter-plone: https://collective.github.io/gatsby-starter-plone/

Narrative Documentation for implementing each feature of plugin: https://collective.github.io/gatsby-source-plone/tutorial/

Features

  • gatsby-source-plone uses a search traversal method to get full content tree from a Plone site, Gatsby cache is used to optimize this process.
  • All the data from plone.restapi is passed on to nodes created by Gatsby for each content object, which can be later queried via GraphQL in a Gatsby project.
  • Files and images nodes are created separately, backlinked to node which contain them. This backlinking along with HTML serialization is handled by the plugin.
  • Documentation and samples have been written for RichText component which processes these backlinks (images and files) and serialized HTML to be handled and displayed by Gatsby site.
  • gatsby-starter-plone implements all the features of the plugin in a Gatsby site, allowing projects to be kickstarted super-fast.
  • Narrative documentation which guides you through implementing each feature of the plugin in a Gatsby project was written with example code from the starter.

Pending

Future plans

Pull Requests

Week 1

What's achieved so far:

  • As recommended by Timo and Asko, I've been following a documentation-driven approach to development, hence I've started out with a basic intro and detailed documentation about implement of the plugin using the @search traversal method.
  • Been exploring and playing around with Gatsby itself and it's plugins, also started implementation of @search traversal method. (https://github.com/collective/gatsby-source-plone/pull/31 )
  • I've also refactored the existing code in gatsby-node.js , adding helpers to fetch data and create content digest, also cleaned it up.
  • Since it's planned to be well documented, quality npm package, I did a bit of research on them as well, and came up with updating the .gitignore and .npmignore files so as to follow best practices. (https://github.com/collective/gatsby-source-plone/pull/28 )
  • To follow code quality best practices, _Prettier _has been configured and documented. Asko helped greatly in setting up tests, make build and continuous integration with Travis, while moving to docker for running the Plone site. (https://github.com/collective/gatsby-source-plone/pull/26 )

What's pending, and planned for next week:

  • Right now, first priority would be setting up a md which lists the changes made, features added, bugs fixed, breaking changes etc for each version rolled out.
  • Work on @search traversal method is in progress and is expected to be done real soon, will test and make PR. This already has been documented in detail, only code implementation remains.
  • I'll also be tackling the issue: that **gatsby-source-filesystem ** name clash with node of type File, this would be solved, documentation would be updated. ref:https://github.com/collective/gatsby-source-plone/issues/21
  • Further documentation is planned to be written on tree hierarchy.
  • Asko has been working on getting the site auto-published to gh-pages, and also commands to import and export content from a Plone site using jq. I'd be working on adding docs wherever required for these as well, so that a user has full control and understanding of what all tools are used/available.

Week 2

What's achieved so far:

  • While playing with Gatsby and it's plugins I noticed this issue that File type node clashes with nodes created by gatsby-source-filesystem and this could prove an issue as that type is reserved. With discussion, Plone was prepended to each content object type so as to prevent this. Docs were also updating taking this in consideration. (ref:https://github.com/collective/gatsby-source-plone/pull/45)
  • Full content tree traversal using the @search endpoint approach was implemented. Code was refactored to improve code readability and also to optimize modularity. Furthermore, logging option was added so that in development, progress of the plugin in action can be tracked. (ref:https://github.com/collective/gatsby-source-plone/pull/49)
  • Documentation for plugin options were added in a format similar to what's followed in the Gatsby official docs/site.
  • First feature (tree hierarchy) was merged with tests. Tree hierarchy using children and parent property of content objects, also added content to test this with in fixtures and also worked on an example use case scenario in the gatsby-starter-default. Worked on and understood a bit on Robot testing framework and wrote a test to check if the data exists in the result. (ref:https://github.com/collective/gatsby-source-plone/pull/51)

What's pending, and planned for next week:

  • PloneSite node for @search traversed data is in progress. Most of the work is done except a example use case in the test project. (ref:https://github.com/collective/gatsby-source-plone/pull/64)
  • Documentation for tree hierarchy illustrating it's usage in the test project is suggested and planned, could finish up taking tips from the mentors. (ref:https://github.com/collective/gatsby-source-plone/issues/65). Planning to wrap this up as soon as possible and get on with the planned timeline.
  • Best way to handle authorization options for the plugin was indiscussion. Research was done on multiple approaches and we've discussed and settled on the best method. Work on this has already been started. The idea is to use bash, curl to get JWT which used with .env variables would automatically add it to gatsby-config.js and make request with auth headers. This would be the main goal for this week.
  • Fetching and using navigation and breadcrumb data from the Plone site has already been discussed. Work on this with an example and docs is also planned to start for this week.

What's in discussion and research:

  • File handling by the plugin is underdiscussion and an issue has been opened in the Gatsby community as well. The current idea that looks good is to pull files to Gatsby cache and link them from there to the project, this is done with help of createRemoteFileNode helper.
  • Testing is planned to be improved with better acceptance tests, so looking into Behavior Driven Development and its features.

Week 3

What I've achieved so far:

What's pending, and planned for next week:

  • Expansions including breadcrumbs, navigation and so on are yet to be handled and data passed on to the nodes. Currently in progress, encountering a few bugs. (https://github.com/collective/gatsby-source-plone/pull/77 )
  • Next step after getting it working would be to add documentation, use case in the gatsby-site
  • Issues noticed while experimenting with the plugin are to be noted and fixed. Currently there exists an issue with subsites noted down here: (https://github.com/collective/gatsby-source-plone/issues/75 )
  • Further testing will be done, bugs to be noted and resolved. Feedback will be taken and all necessary changes, fixes will be worked on before the First Evaluation period so that a working plugin which traverses the whole Plone site using @search traversal developed.

Week 4

What's achieved in the previous week:

First Evaluation:

  • I'm glad to say, even thought we did face quite a bit of issues on the way unexpectedly and came up with features while buildling, we're still
  • Post first-evaluation, the main things are planned are a recursive approach for traversal and file handling in the plugin. These are crucial and also tricky so will be researching quite a bit before moving forward.

Week 5

What's achieved in the previous week:

  • We've made significant progress in planning and implementing the recursive traversal algorithm. Initially, the code was refactored and modularized to improve readability and to add new features easily. (ref:https://github.com/collective/gatsby-source-plone/pull/87)
  • After planning and a lot of testing, the recursive algorithm was implemented using a Breadth-First Search approach. (ref:https://github.com/collective/gatsby-source-plone/pull/88)
  • Documentation was added for recursive traversal. (ref:https://github.com/collective/gatsby-source-plone/pull/91)
  • Minor issues and problems faced on the way were fixed in the process itself.
  • Search parameters feature was added as a plugin option which allows you to query for certain content objects that are returned depending on the query. Documentation was also added for this. There was a bit of an issue converting query to strings/encoding to URI components, but then later with advice from @cekk, axios object parameters were used. This took some effort keeping on refactoring the code, but we have something that works smooth now. (ref:https://github.com/collective/gatsby-source-plone/pull/90)

What's planned for the coming week:

  • I've already explored Gatsby v2 and also made contributions to the GatsbyJS project itself in helping their projects shift to v2. So the next step would be to port the plugin and tests to v2.
  • Further more, file handling is a major feature that'll require quite a bit of efforts. So research on that will be started along with examples to play around with. I'll be interacting with the Gatsby community itself to get guidance and tips on moving forward with this.
  • Unit testing is also under discussion and will be looking to implement basic testing from scratch, after porting to v2.

Week 6

What's achieved in the previous week:

  • Documentation was added for search parameters, for the users to get a good idea of how it works, and how to implement using gatsby-config.js plugin options. (ref: https://github.com/collective/gatsby-source-plone/pull/93)

  • Our project plugin and the tester have been migrated to Gatsby v2, following the guide and fixing any breaking changes. Also encountered a Babel setup issue which wasn't occuring locally but was occuring for travis, and this was fixed doing proper research about the issue. (ref: https://github.com/collective/gatsby-source-plone/pull/94)

  • Being not very familiar with testing, did research and played around with Jest for unit testing. A basic implementation was added for the plugin, to run tests for the helper functions. (ref: https://github.com/collective/gatsby-source-plone/pull/97)

  • With @datakurre, a few improvements were made and code was cleaned up. Also, added subfolders and testing for deep folder structure.

What's planned for the coming week:

  • File handling handling is an important aspect that's yet to be implement, this week will mostly focus on researching about it's implementation and finding the best way to handle images, and other files.
  • It is planned for images to use gatsby-plugin-sharp, and for other files to be added as file nodes which are linked to files pulled locally by Gatsby.

Week 7

What's achieved in the previous week:

What's planned for the coming week:

  • Transforming HTML text fields and query types (collections) are to be processed by the plugin before making it available for GraphQL
  • Research on Plone tiles and it's support, along with configuring a client based search engine for the gatsby-site.

Week 8

What's achieved in the previous week:

  • Before getting into implementing new features, previous features were refined. Gatsby-source-filesystem was updated to v2 (ref: https://github.com/collective/gatsby-source-plone/pull/109)
  • Documentation for file handling was added, describing gatsby-source-filesystem, it's usage and helper function createRemoteFileNode, and how it can be effectively used to process files and images. It also explains the workaround used to expose the publicURL field in GraphQL by using gatsby-source-filesystem on a static folder. (ref: https://github.com/collective/gatsby-source-plone/pull/108)
  • The HTML parsing feature was implemented with examples and tests. React-html-parser was used to parse HTML content from strings to nodes, and react-serialize to allow it to be passed through GraphQL fields. Relative local links were replaced with gatsby-links during processing. (ref: https://github.com/collective/gatsby-source-plone/pull/113)

What's planned for the coming week:

  • We're currently planning to have a briefing tomorrow on the goals to complete for the last month of coding, shipping a final project by the final evaluation period. Once we have that streamlined, I'll focusing on achieving these tasks and getting a fully functional plugin ready as soon as possible!

Week 9

What's achieved in the previous week:

What's planned for the coming week:

  • The GSoC project board is being followed to pickup and complete tasks before the end of the coding period.
  • Currently, the main goals pending would include narrative documentation for usage of the plugin and setting up a gatsby-starter-plone which would serve as easy to use starting point for Gatsby with Plone.

Week 10

What's achieved in the previous week:

What's planned for the coming week:

  • The major goals would be to work on implementing each feature of the plugin in gatsby-starter-plone and write documentation side by side.
  • Since it's the last few weeks of coding, planning to speed up the process and work on shipping the starter as soon as possible along with the narrative documentation.

Week 11-12

What's achieved in the previous week:

https://github.com/collective/gatsby-source-plone/pull/140, https://github.com/collective/gatsby-source-plone/pull/139 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment