jsquire/auto-triage-classify.md

## auto-triage-classify.md

      
    Raw
  

              auto-triage-classify.md
            
          
    Overview

Issues in the Azure SDK repositories use a two label system in order to classify the issue and identify the responsible party.  Each issue must have a "service label" (pink) and a "category label" (yellow) to be valid.  It is this combination of labels that identifies which specific library an issue is for and, by extension, who should be responsible for the issue.  For example, an issue tagged with Event Hubs and Client is owned by Jesse's team.  An issue tagged with Event Hubs and Mgmt is owned by Arthur's team.
This constraint is built into the AI label service; if it cannot successfully predict both labels, it will return none.  When performing automated triage, the rule will always have access to both tags and needs to take both into account to find the correct owner to triage the issue to.  Currently, this is not how triage handles the scenario.  Instead, the rule will collect the owners of both tags and consider them both owners.  For example, an issue tagged with Event Hubs and Mgmt would be routed to both Jesse and Arthur.
Problem statements


The automated triage rule needs to consider both tags returned by the AI label service as a pair to find the correct owner.


The CODEOWNERS metadata must allow for ownership to be differentiated between client libraries and management libraries.


Recommendation

I would like to see us implement both of the options outlined below.  I do not see them as mutually exclusive, and I think that they would work together to cover all scenarios.
Proposed options

Option: Use "read from the bottom and take the first match" logic

When GitHub processed CODEOWNERS to assign reviewers for pull requests, it does so by processing the file, starting at the bottom and working its way up until a match is found.  If the CODEOWNERS contains multiple path segments would  that match a file path, GitHub will use the first match that it finds, closest to the bottom of the file.  This makes the ordering of paths in CODEONWERS meaningful; more general paths should appear earlier in the file and more specific paths which appear later may override them.
For this option, let's assume that we apply the same concept to label matches.  Consider the following CODEOWNERS:
# ServiceLabel: %Mgmt
# AzureSdkOwners: @arthurma

... <SNIP> ...

# ServiceLabel: %Event Hubs
# AzureSdkOwners: @jsquire
# ServiceOwners:  @service1

When an issue comes in and the AI label service returns Event Hubs and Mgmt, the triage rule would:

Start processing at the bottom of the file.
Match on Event Hubs and assume @jsquire should be assigned.
Stop processing the file and return the match.

Benefits


This flow is desirable because for our language repositories, ambiguous matches would follow a known set of rules consistent with how PR reviewers are matched today.


When combined with the proposed option for matching multiple tags, this allows for great flexibility to define general matches when one team owns all management issues and to override any exceptions to that rule with a more specific block.


This does not require that we generate a second set of labels for "service - Management", doubling the number of labels in our repository and making reporting more difficult.


Challenges


As we continue to shift left and more service teams generate their libraries, we may see management ownership also follow this pattern and no longer have an easy central owner.

Option: Multiple service labels have an AND relationship for match criteria

Currently the ServiceLabel marker supports multiple tags, but treats them as individual items.  Either tag could be matched against those returned by the AI label service.  This would mean that if we started using Client and Mgmt in those blocks, they would all be a match for AI label service results, which is not the desired behavior.
For this option, let's assume that we treat the labels as an AND relationship and any match must have all labels.  Consider the following CODEOWNERS:
# ServiceLabel: %Event Hubs %Mgmt
# AzureSdkOwners: @arthurma

# ServiceLabel: %Event Hubs %Client
# AzureSdkOwners: @jsquire
# ServiceOwners:  @service1

When an issue comes in and the AI label service returns Event Hubs and Mgmt, the triage rule would:

Start processing at the bottom of the file.
Match on the first Event Hubs but fail to match %Client.
Continue processing the file, moving upwards to the next block.
Match on Event Hubs and Mgmt in the second block; @arthurma will be assigned.

Benefits


This flow is desirable because it allows for explicit distinction between client and management libraries.


When combined with the option for first match semantics, this allows for great flexibility to define general matches when one team owns all management issues and to override any exceptions to that rule with a more specific block.


This does not require that we generate a second set of labels for "service - Management", doubling the number of labels in our repository and making reporting more difficult.


Challenges


This would require that we have at least two blocks for each service label, one client and one management.  Likely, there would also be a PR block.  That's a lot of metadata for a single service.
Parsing the ServiceLabel block would need to be done in a specific way to avoid the need to match all permuatations.  (exmaple below)

Rejected options

Option: Add new labels for each service, specific to management

In this option, each service label in our common set (the pink ones) would have a second copy that uses a suffix of " - Management" or similar.
Benefits


No rule changes needed.

Challenges


The number of labels would explode, adding a ton of noise and negatively impacting reporting.
This would require that we have at least two blocks for each logical service, one client and one management.  Likely, there would also be a PR block.  That's a lot of metadata for a single service.

Examples

Match multiple tags

This approach uses the set of labels returned from the AI label service as the target for matches and tests each ServiceLabel against them.  Because we have a known small set (exactly 2 or 0) coming from the service, the time to build a hash table for efficient matching.  This appraoch allows us to avoid trying to match every permutation of ServiceLabel items.
Note: prototype-level code; not intended to be production worthy.
public async Task Main()
{
    var issueLabels = new HashSet<string>(await QueryLabelService());

    if (issueLabels.Count == 0)
    {
        // Mark for manual triage.
    }

    // At this point, we know that we have exactly two labels.

    var blockNumber = 0;
    var useBlockNumber = -1;

    foreach (var line in ReadCodeOwnersLinesBottomToTop())
    {
        // For brevity, we'll only deal with labels and skip case-insensitivity.
        
        if (!line.Trim().StartsWith("# ServiceLabel:"))
        {
            continue;
        }

        ++blockNumber;
        var match = true;

        foreach (var serviceLabel in ParseServiceLabels(line))
        {
            match = match && issueLabels.Contains(serviceLabel);
            
            if (!match)
            {
                break;
            }
        }

        if (match)
        {
        
            // Yay, we know the owner(s) are part of this block.
            
            useBlockNumber = blockNumber;
            $"Match found for block #{blockNumber}".Dump();
            
            break;
        }
    }

    // For brevity, we're only tracking the block number, the actual 
    // implementation would use the owners from that block.
    
    (useBlockNumber switch
    {
        _ when useBlockNumber > 0 => $"Final match is block #{useBlockNumber}",
        _ => "No match"
    })
    .Dump();        
}