Skip to content

Instantly share code, notes, and snippets.

@AshKelly

AshKelly/gsoc_summary.md

Last active August 13, 2018 10:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AshKelly/9f45a6271a5f76d0b5441fb94005b2da to your computer and use it in GitHub Desktop.
Save AshKelly/9f45a6271a5f76d0b5441fb94005b2da to your computer and use it in GitHub Desktop.
Download ZIP
This gist contains a summary of my GSoC project with yt (NumFocus)
Raw
gsoc_summary.md

GSOC 2018 Work product submission.

Summary of GSoC project with yt-project (NumFocus)

yt is a package for analyzing and visualizing volumetric data. The current infrastructure of yt is designed to deal with adaptive mesh refinement (AMR) data. However, a wide range of modern N-body hydrodynamics simulations now use smoothed particle hydrodynamics (SPH) which produces data consisting of volume filling fields (see Springel 2015 and Schaller 2013).

The project aimed to continue the development and introduction of a yt API for SPH data that ensures all SPH fields return data defined at the locations of the particles - the initial development is discussed here. The summer project focussed on adding "scatter" and "gather" pixelization routines to a variety of yt data containers.

Below I will list the main features implemented along with links to the code, more detials can be found below about each of these features such as useage, examples and technical details about the implementation.

Minimum

  • Task 1: The first task was to add the "scatter" approach to the arbitrary grid container - this could then be generalized to different containers - PR

    • The scatter approach is the "defacto" method of interpolating onto grids. This is because it is fairly intuitive and can be incredibly efficient in the case of uniform grids.

  • Task 2: After task 1 I was much more familiar with the yt codebase, particularly the yt-4.0 branch. At this point I was able to add the "gather" approach to the arbitrary grid and slice plots. - PR.

    • The gather approach is generally much slower than the scatter approach, but is has the advantages that it is "truer" to the data.
    • In smoothed particle hydrodynamics the concept of ever having a zero density is a little strange. This can occur with the scatter approach - the gather approach avoids this.
    • This task involved interfacing with the Cykdtree and producing new utility and pixelization routines.

  • Task 3: After sucessfully adding "scatter" and "gather" to the arbitrary grid I was able to generalise this to also work with the covering grid - PR.

    • The covering grid is a data container which is often more familiar for users with an AMR background. As such it was important to ensure that this feature was still available.

  • Task 4: With the succesful addition of low level cython routines for both "scatter" and "gather" it was time to re-introduce an octree into yt to allow as much backwards compatibility as possible - PR this is now in the testing phase.

    • The current version of yt, interpolates SPH onto a global octree which then allows it to be compatible with much of the yt internals which are designed to work with grid data. Whilst this approach is generally not desirable for yt users, it is important to attempt to maintain backwards compatibility such that users can reproduce previous science results.
    • This feature may also allow users to load up particle data and smooth onto an octree and then export for use with other packages.
    • All of the desired functionality of this feature has been added and I am currently in the process of generating tests and documentation.

Optional

  • Task 5: I started to work on adding a method to calcualte the value of a non-local field. These fields require a different kernel - PR

    • This requires interfacing with the yt field system at the initial level when a dataset is loaded.
    • This PR is still under heavy development, so far I have added a stub where the machinery for these calculations will occur.
    • The next step is to introduce the basic non-local field calculations in Cython using a hardcoded cubic spline kernel for testing purposes.

  • Task 6: I sthen started to work in improving swift integration. Swift is a particle based fluid code being developed at the ICC in Durham and would benefit greatly from easy use with yt visualization. - PR this is now in the testing phase

    • This feature required the development of a dedicated front end. The current branch of yt interprates the data as a Gadget dataset. Whilst this is useful as it allows the data to be loaded, it incorrectly calculates the units of the data and also adds a lot of unneccesary overhead relating to chemical abundances.
    • The open PR is awaiting some documentation and unit tests before it can be successfully merged.

During my summer project I also fixed two bugs which were reported:

  • Bug 1: Repeated normalization - PR

    • This was a bug which was spotted during the development of the scatter and gather routines.

  • Bug 2: Projection artefacts - PR

    • This was a bug reported by a user already using the new features in yt-4.0 on the development branch. This bug related to an unfilled array which was leading to undefined behaviour.

Before the summer I also added a feature:

  • Feature 1: Custom labels - PR
    • This PR allows users to overwrite default labels on yt projection plots, this was requested by a user.

Some metrics

1 code review completed

8 merged pull requests

2000 lines of code added

450 lines of code removed

Future plans

During this Google Summer of Code project I have absolutely loved my time working on the yt code base. The community has been incredibly supportive and it is so rewarding to see new features get used immediately by an incredibly verbose user base. This provides a great motivation to develop new features and provides a really robust test drive to get feedback and bug reporting.

A couple of highlights for me:

  • Merging my second major PR - add the "gather" approach to the arbitrary grid and slice plots. This was a lot of hard work and required a range of skills from cython routines and python interfacing to documentation and tests. I think this was the PR where I really felt like I was capable of producing good work for yt and I felt like I had a handle on how the internals worked.

  • I fixed a bug (Projection artefacts - PR)! This was a bug reported by a keen user of the new development branch. This was not a task I was assigned to, but I spotted it on the issues list and felt like it was something I could help with. It was incredibly satisifying to take on an extra responsibility and then be successful!

  • I became a project memeber!

  • A code review! Code reviews are such an important aspect of open source code development. In the absence of code reviews, the quality of the code and the maintainability would drop so rapidly. It was a great oppurtunity to do my first code review and recieve good feedback - "Thanks for the solid review.".

I have enjoyed my time with open source development and yt so much that I plan to continue being an active contributor to the yt-4.0 branch! This also lines up with the topic of my PhD which I start in October at the ICC, Durham University. In this PhD I will work heavily on AREPO and swift datasets which will provide ample motivation to continue improving particle compatibility. A few of the next tasks I am excited to work on are:

  • Spatial chunking - this would allow us to improve efficieny for a bunch of our particle tasks
  • Adding a basic "gather" approach to the current projection plots
  • Ray-tracing on a voronoi mesh
  • Start work on a yt-4.0 paper
  • SPH volume rendering
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment