Query distributed mode does not fetch from storage gateways

[hanem100k]

Query, store gateway

Thanos, Prometheus and Golang version used:
Thanos version 0.26.1

Object Storage Provider:
Azure blob store

What happened:
Having a set of receivers and sharded store gateways. Whenever query distributed mode is enabled, data will only be returned from receivers but not stores.

Removing query.mode=distributed from arguments solves the issue.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
It is reproducible 100% when you query for a time range that exceeds your receivers.

Metrics with distribution enabled:

Metrics without distribution enabled:

[MichaHoffmann]

Note:

thanos/pkg/query/remote_engine.go

Line 116 in 883fade

func (r *remoteEngine) MinT() int64 {

likely the MinT() computation is off for usecases where we have [sidecar,receiver,..]+storagegw endpoints on same remote engine.

[fpetkovski]

@hanem100k can you try attaching a separate querier to store gateways?

[hanem100k]

I do have a Top level [QFE] -> [Q] -> Leaf [Q] -> [receivers, stores]

There is a single Query in the top layer and there are N leafs.

If I go into the leaf Querier directly, it returns ok from stores as well as receivers.

@fpetkovski

can you try attaching a separate querier to store gateways?

Do you mean that I should retry the distributed mode by having a Querier for stores only and one for receivers only?

[fpetkovski]

Actually we are going to fix this configuration with @MichaHoffmann soon, so let's wait for the new patch release.