Making a custom scorer in sklearn that only looks at certain labels when calculating model metrics. For example: This creates a f1_macro scorer object that only looks at the '-1' and '1' labels of a target variable.

sklearn_custom_scorer_labels.py

from sklearn.metrics import f1_score, make_scorer
scorer = make_scorer(f1_score, labels=[-1, 1], average='macro')

# use like:
cv = cross_val_score(model, X, Y, scoring=scorer)