Error while loading events, show in logs of one of clickhouse shard

[stouch]

Summary

Got error when loading events, but ingest, rotor and console work :

shard1 error:

And rest of services look OK :

shard0

bulker:

ingest:

│ time="2024-08-19 16:02:36" level=info msg="[metrics_server] Starting metrics server on 0.0.0.0:9091"                                                                   │││ │ ││││││  │    │         │           │                 │                                                 
│ time="2024-08-19 16:02:36" level=info msg="Starting http server on 0.0.0.0:3049"                                                                                       │││ │ ││││││  │    │         │           │                 │───────────────────────┐───────────────┐────────┐
│ time="2024-08-19 16:04:01" level=error msg="[ch_events_log] Error preparing batch: code: 60, message: Table default.events_log does not exist"  

System configuration and versions

k8s install of Jitsu v2.8.0

(https://github.com/stafftastic/jitsu-chart)

[vklimontovich]

Seems like the table that for storing event logs is configured incorrectly. It should be created in newjitsu_metrics DB, see compose file for a reference

[stouch]

It might be because of https://github.com/stafftastic/jitsu-chart/blob/fb5d459c9c831d25c99a49981c92de43b174c1d8/templates/_helpers.tpl#L145
-> when we use the jitsu-chart, it uses by default the "default" db.

It changed recently ?

[stouch]

@hendrikheil After investigation, I found that this error of unknown table does not always occur (sometimes the data load well in my console). Then I went to my k8s clickhouse shards and find out that in one of the shard the db does not exist :

I got 1 replica and 2 shards :

This is the shard for which the table does not exist :

That shard does have it :

Any idea what I could do to fix this ?

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.