Problem with graphite aggregates and NULL values coming from some hosts

graphite-null-value-aggregates-problem

We're using graphite 0.9.12.

We have per-datacenter aggregates and total-across-datacenters
aggregates, constructed as follows (flush interval is 10s):

  # Total
  <project>.<env>.aggregates.total.recommender.api.suggestions.rate (10) = sum <project>.<env>.*.*.counters.recommender.api.suggestions.rate

  # Per-datacenter
  <project>.<env>.aggregates.dc.<datacenter>.recommender.api.suggestions.rate (10) = sum <project>.<env>.<datacenter>.*.counters.recommender.api.suggestions.rate

Since we migrated our Graphite to a new machine, our "aggregates.total" metric graph was screwed up.

The values are correct for the past hour or so only, but if you go back in time, the values are either zero, or some strange very low noisy line is displayed. (I understand this is explained like shit... sorry)

We figured out one of our datacenter aggregates (which is then aggregated to form the aggregates.total above) was spitting out
"null" values, that is, the json that would be returned for that metric was:

  {
    "target": "recsys.production.aggregates.dc.ashburn.recommender.api.suggestions.rate",
    "datapoints": [[null, 1416220080], [null, 1416220140], ...]
  }

Note that we have had deleteIdleStats=true for a long time now, so that is totally expected, b/c that datacenter is idle/standby.

What's not expected is that carbon-aggregator was working fine and producing the correct aggregates until we migrated graphite to a new system, and now it somehow can't "ignore" the null value when aggregating, so that the dc-total graph is broken.

Does this ring any bells?

Thanks for even reading all this,
and have a splendid day.

-- 
Cosimo