invalid_characters_override_list_v4_reproduction.md

This gist is related to this blog post

Here are some steps you can follow to reproduce the issue and the misclassifications due to the incorrectly formatted entries in the override list of the Topics API feature.

1. Get the override_list.pb.gz shipped with model v4 (current version of the model.tflite for the Topics API shipped in Chrome).

   • Visit chrome://topics-internals, under the Classifier tab, get the path where the model is stored, in that same folder you will find the override_list.pb.gz file
   • For archival purposes, the file can also be found here

2. Then, check that the override list contains domains with the invalid "/" character and that these domains are flipped around that character, you will need to decompress the protobuf file, get the corresponding .proto, and decode it. Here is how I do it, you will need these 2 scripts:

   convert_pb_override.sh bash script:

   #!/bin/bash
   
   override_pb_gz=$1
   override_tsv=$2
   
   override_pb=override.pb
   proto_path=page_topics_override_list.proto
   python_proto_path=page_topics_override_list_pb2.py
   if [ ! -f $override_tsv ]
   then
       # Fetch page_topics_override_list.proto
       wget -q -O $proto_path https://raw.githubusercontent.com/chromium/chromium/main/components/optimization_guide/proto/page_topics_override_list.proto
       protoc $proto_path --python_out=.
       # Decompress override.pb.gz
       gzip -cdk $override_pb_gz > $override_pb
       python3 convert_pb_override.py $override_pb > $override_tsv
       rm $proto_path $override_pb $python_proto_path
   fi

   convert_pb_override.py python script:

   import argparse
   import pandas as pd
   import page_topics_override_list_pb2
   
   # Create Argument Parser
   parser = argparse.ArgumentParser(
       prog="python3 convert_pb_override.py",
       description="Convert .pb override list to .tsv",
   )
   parser.add_argument("input_file", help="input file")
   args = parser.parse_args()
   
   # Load override list
   override_list = page_topics_override_list_pb2.PageTopicsOverrideList()
   with open(args.input_file, "rb") as f:
       override_list.ParseFromString(f.read())
   print("domain\ttopics")
   for entry in override_list.entries:
       line = "{}".format(entry.domain)
       first_topic = True
       for id in entry.topics.topic_ids:
           if first_topic:
               line += "\t{}".format(id)
               first_topic = False
           else:
               line += ",{}".format(id)
       print(line)

   • And then run:

   # Decode override list to .tsv format
   ./convert_pv_override.sh override_list.pb.gz override_list.tsv
   # Extract domains (and corresponding topics) with invalid character:
   grep ".*[^[:alpha:][:space:][:digit:]^,].*" override_list.tsv

• I obtain the following incorrectly formatted domains.

The domain entries in that override list are supposed to be pre-processed the same way as the input that would be passed to the model.tflite of the Topics API. This means: take the FQDN, remove any "www." prefix if present in the domain to classify, and then replace the following characters "-", "_", ".", "+" by a whitespace (https://source.chromium.org/chromium/chromium/src/+/main:components/browsing_topics/annotator_impl.cc;l=269).

Some examples:

• candy-crush-soda-saga.web.app -> candy crush soda saga web app and not web app/candy crush soda saga
• subscribe.free.fr -> subscribe free fr and not: free fr/subscribe
• uk.instructure.com -> uk instructure com and not: instructure com/uk

As a result, when these domains are classified by the Topics API in Chrome, no match is found in the override list for the domain correctly pre-processed. Thus, they are classified by the ML model which does not output the intended classification (Chrome classification can be obtained from chrome://topics-internals):

• candy-crush-soda-saga.web.app -> 183. Computer & video games - 186. Casual games - 1. Arts & entertainment by ML model -> 186. Casual games - 215. Internet & telecom from override list for web app/candy crush soda saga
• subscribe.free.fr -> 217. Internet service providers (ISPs) by ML model -> 217. Internet service providers (ISPs) - 365. Movie & TV streaming - 129. Consumer electronics - 218.Phone service providers from override list for free fr/subscribe
• uk.instructure.com -> 229. Colleges & universities - 227. Education by ML model -> 230. Distance learning - 234. Standardized & admissions tests - 140. Software - 227. Education - 229. Colleges & universities from override list for instructure com/uk