pa-17168-investigation-sept-21-2018.md

Jira ticket link

https://parsable.atlassian.net/browse/PA-17168

Problem

It appears as if at one point (Aug 21 2018) a step had a description with four elements. But later on (Aug 22 2018) three of the elements had disappeared without a trace.

Details

• Environment: EU
• JobTemplate: f1ad2546-7822-4927-b381-6a62134301e0
• Step: 13bbcb02-ad7d-4b6b-a520-f8db2a531992
• Step Client Id: 75a854a7-c56f-497b-a603-ee13a4e101e4

Why do we think the description was larger than it now appears?

SLB supplied us with a JSON document they say was retrieved via JobTemplate::Get. Based on logs in Splunk, that likely occurred at 16:00:59 UTC. SLB supplied us with a second JSON document they say they obtained by the same means. They say it comes from the next day. It's missing the content as described above.

Even if we doubted the validity of the first JSON document (which conflicts with our records), there remains a PDF of the JobTemplate that clearly shows a now-missing description.

Where did we look for history of the description?

I checked in the steps table, using the step id and client step id. None of the versions in the database show any hint of the three missing step description elements.

Here's what I would have expected to see:

• a step row with version n and a description matching the first JSON document
• another step row with version n+1, and a description matching the second JSON document

Instead what I found was:

• a single step row with a description matching the second JSON document (version=1646)

Interesting things

Status: "InReview" --> "Unpublished"

The earlier version has its status field set to InReview, whereas the later version has that same field as Unpublished. Is this strange? I don't even know.

No record of any update

I'm unable to find any record in Splunk of any request that would have made changes to the step in question between the two JSON document retrievals. I searched Aug 21 to Aug 23 inclusive. I did find a couple of requests that modified the metadata of the JobTemplate, but:

1. these didn't have any step description impact
2. they occurred before the first JSON document retrieval, so their affect would have been identical for both versions

What now?

Well, I don't know, really. If there are any more places that we could find either:

1. an audit log of actions against this JobTemplate, or
2. a better historical record of all the revisions to this JobTemplate, then that would be really handy.

But somehow I suspect neither of these is the case. So I'm looking for ideas on where to look next.

splunk-records.txt

This is a subset of the requests  against this JobTemplate.
I selected only the ones that I thought could potentially alter the JobTemplate.

    15:50:26 - FieldDataService          :: GetNumberUnits
    15:50:26 - GroupHeaderContentService :: IndexCategories
    15:50:26 - GroupHeaderContentService :: IndexEquipment
    15:50:26 - GroupHeaderContentService :: IndexNoticeIcons
    15:50:27 - FieldDataService          :: GetNumberUnits
    15:50:27 - GroupHeaderContentService :: IndexCategories
    15:50:27 - GroupHeaderContentService :: IndexEquipment
    15:50:27 - GroupHeaderContentService :: IndexNoticeIcons
    15:50:27 - JobTemplateService        :: Fetch
    15:50:28 - JobTemplateService        :: AcquireLock
    15:50:28 - JobTemplateService        :: Fetch

    16:00:44 - JobTemplateService        :: Update
    16:00:46 - JobTemplateService        :: ForceAcquireLock
    16:00:47 - JobTemplateService        :: UpdateRootHeaders
    16:00:49 - JobTemplateService        :: Fetch
    16:00:58 - CallReviewWebhook         :: POST returned successful response
    16:00:57 - JobTemplateService        :: AcquireLock
    16:00:57 - JobTemplateService        :: BeginReview
    16:00:59 - JobTemplateService        :: Get
    16:01:02 - JobTemplateService        :: Fetch
    16:01:02 - JobTemplateService        :: ReleaseLock

    02:34:32 - JobTemplateService        :: GetPublicVersion (Aug 22) (version == 4)

For a more thorough list, start with:

https://parsable.splunkcloud.com/en-US/app/search/search?q=search%20f1ad2546-7822-4927-b381-6a62134301e0%20%7C%20stats%20values(msg)&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=1534809600&latest=1535065200&display.events.type=list&display.page.search.tab=statistics&display.general.type=statistics&sid=1537581712.16353