Am I breaking Argo?

[static-moonlight]

I encountered a phenomenon that doesn't make sense to me. Yesterday I added a new workflow. That workflow is very short-lived (~30s) and is being executed 4 times every minute. For about 12h everything seems to work fine. Then I looked at this:

This is the number of successful workflows. The lines on top (at ~60) are the new workflows I added. At some point this number is slowly going down. Implying that at least some of the workflow runs didn't happen. However, there are no errors. This can only happen in 1 of 2 scenarios:

1. The workflow is not executed at all (because it's not triggered)
2. The metrics are just not reported

If I look at the data, everything is there, and complete. Which means the workflows were running. So it can only be a problem with the metrics, giving me false readings. But why out of the sudden? A problem with the monitoring server? Unlikely. Because, that would show for all workflows, not just the new ones.

Also, I am seeing this in the logs of the workflows controller, a lot:

was unable to obtain the node for [...], taskName [...]

Some of those warning are related to the new workflow, some for existing ones (not effected by this phenomenon). So I'm not even sure if this is related or not.

There doesn't seem to be any errors in the logs.

• I am using the Argo cluster-install (so one Argo Instance for all namespaces)
• Currently I'm running this:
  • Namespace 1: 16 workflows every 5 minutes + 10 every 3h (distributed evenly over the 3h)
  • Namespace 2: 2 workflows every 5 minutes
  • Namespace 3: 5 workflows every minute

These are still fairly small numbers. Plus I'm not even done yet, roughly half way through. So the number of workflows is going to double in the end.

So I was asking myself:

1. What would be necessary for Argo to execute the workflow successfully, but then forget to report the (workflow) metrics?
2. Did I just breach though some threshold, and Argo can't handle that many small workflows?
3. Are there some key metrics which can tell me if the Argo workflow controller is still operating in the green?
4. Should I try to use the namespace-install instead?
5. Should I pick something else, other than Argo, for those every-1-minute workflows?

I'm really not sure what to make of this. Until I can figure that out I don't really want to put more workload into that system.

[Joibel]

You don't tell us what metric you are collecting.

I assume you're collecting argo_workflows_count for phase=Successful. This is only a gauge (not a counter in metrics terminolgy) of the number of workflows still in your cluster. I suspect you have some workflow tidying happening deleting your workflows and making the gauge go downwards.

This changes in 3.6 with argo_workflows_total_count which is a true counter and should do what you want.

[static-moonlight]

Sorry. I am collection custom metrics:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
[...]
  artifactGC:
    serviceAccountName: [...]
    strategy: OnWorkflowDeletion
    forceFinalizerRemoval: true
  podGC:
    strategy: OnPodCompletion
  archiveLogs: true
  activeDeadlineSeconds: 300
  ttlStrategy:
    secondsAfterSuccess: 0 # Don't keep successful workflows
    secondsAfterFailure: 604800 # Keep failed workflows for 1 week
[...]
  metrics:
    prometheus:
      - name: duration
        help: workflow duration
        labels:
          - key: product_key
            value: "{{workflow.parameters.PRODUCT_KEY}}"
          - key: product_group
            value: [...]
          - key: action
            value: distribute
        gauge:
          realtime: false
          value: "{{workflow.duration}}"
      - name: result
        help: workflow result
        labels:
          - key: product_key
            value: "{{workflow.parameters.PRODUCT_KEY}}"
          - key: product_group
            value: [...]
          - key: action
            value: distribute
          - key: status
            value: "{{=lower(workflow.status)}}"
        counter:
          value: "1"
[...]

The graph, I was showing above, came from this query:

changes(argo_workflows_result{status="succeeded"}[1h])

[static-moonlight]

BTW: I also forgot to mention that CPU, memory and storage of the Kubernetes nodes are comfortably in the green. The machines are practically bored.

[static-moonlight]

I might be ready for a vacation ...

Sorry that I bothered you with this ticket. After some digging (looking at the data, the messages, the logs, the raw metrics data, ... everything) ... in the end it came down to a badly picked range function for metrics visualization in Grafana. When we use increase(...) instead of changes(...) everything looks exactly as it is supposed to and the metrics visualization now fits to what's really going on.

[Joibel]

Thanks for letting us know what wasn't working for you. I hope you can get a holiday soon too 👍