Simple example of extracting tokens from a file in Sublime

simple_token_scanner.py

import sublime
import sublime_plugin


class VariableHoverEventListener(sublime_plugin.EventListener):
    def get_hover_token(self, tokens, point):
        """
        Given a list of tokens, find the one that's at the point given.
        """
        for index, item in enumerate(tokens):
            if item[0].contains(point):
                return index

        return -1


    def on_hover(self, view, point, hover_zone):
        # If the mouse is not hovering over the text in the buffer, we don't
        # care.
        if hover_zone != sublime.HOVER_TEXT:
            return

        # Get all of the tokens on the line that was hovered, then find the
        # index of the one the user hovered on; if someone that's not found,
        # leave. That should never happen.
        tokens = view.extract_tokens_with_scopes(view.line(point))
        idx = self.get_hover_token(tokens, point)
        if idx == -1:
            return

        # Start with an empty result region
        result = None

        # Starting at the region the user hovered over and going leftwards,
        # combine regions until we find something that's not a variable or not
        # an punctuation accessor.
        #
        # For each found region, combine it with the existing result region.
        while True:
            # If this isn't what we want, we're done scanning.
            if sublime.score_selector(tokens[idx][1], "variable, punctuation.accessor") == 0:
                break

            # Combine this region with the previous region, if any
            result = tokens[idx][0] if result is None else result.cover(tokens[idx][0])

            # Move back one token
            idx = idx - 1
            if idx == -1:
                break;

        if result is not None:
            print(view.substr(result))
        else:
            print('nothing exciting, sorry')

This is a simple plugin example for Sublime Text make in Sublime Text Live #119; it demonstrates using the extract_tokens_with_scopes() functionality in the view class to find hovered variable references, including their accessor characters.

The user that asked the question was interested in being able to look up variable references including their accessor scopes, going leftwards; for example, given:

    some_function(this->object.field);

Hovering over field should return this->object.field while hovering over object should return this-> object.

In this simple example, the result (if an) is displayed to the console. This does not handle the case where the accessor may span multiple lines; everything must be on the same line.

A more robust implementation of this would continue looking back through the code, skipping over whitespace tokens and moving up through lines in the file if needed.