[Bug] OOM kill because sudden memory increase and requests

[alzger]

🐛 Bug Report

snarkOS has presented several OOM Kill incidents with the following characteristics:

Incident Date: June 30, 2024
Affected Service: snarkos Service
Nodes Affected: Node A, Node B, Node C
Summary
On June 30, 2024, the snarkos service on testnet beta client nodes Node A, Node B, and Node C experienced an outage due to the OOM (Out of Memory) killer being invoked around the same time on all three nodes.
Incident Details
Node A:

• Node type: Core Client
• Timestamp: 9:19:42 PM EDT
• Memory Requested by snarkos: 81.6 GB
• Memory Used by snarkos before kill: 31.6 GB
• RAM Memory on the server: 32GB
• Last Block Advanced before the incident: 55839

Node B:

• Node type: Core Client
• Timestamp: 9:19:44 PM EDT
• Memory Requested by snarkos: 90.8 GB
• Memory Used by snarkos before kill: 31.6 GB
• RAM Memory on the server: 32GB
• Last Block Advanced before the incident: 55839

Node C:

• Node type: Core Client
• Timestamp: 9:53:52 PM EDT
• Memory Requested by snarkos: 83.5 GB
• Memory Used by snarkos before kill: 31.6 GB
• RAM Memory on the server: 32GB
• Last Block Advanced before the incident: 56105

Effect on Validator(s)
Validator nodes were minimally affected by the client nodes going down, only experiencing a brief spike of 15% more memory usage than normal.

Incident Date: July 2, 2024
Affected Service: snarkos Service
Nodes Affected: Node D, Node A
Summary
On June 30, 2024, the snarkos service on testnet beta client nodes Node D, and Node A experienced an outage due to the OOM (Out of Memory) killer being invoked at different times.
Incident Details
Node D:

• Node type: Core Client
• Timestamp: 2:53:44 AM EDT
• Memory Requested by snarkos: 95.8 GB
• Memory Used by snarkos before kill: 30.2 GB
• RAM Memory on the server: 32GB
• Last Block Advanced before the incident: 68199

Node A:

• Node type: Core Client
• Timestamp: 11:03:21 PM EDT
• Memory Requested by snarkos: 91.5 GB
• Memory Used by snarkos before kill: 30.2 GB
• RAM Memory on the server: 32GB
• Last Block Advanced before the incident: 70926

Effect on Validator(s)
Validator nodes were minimally affected by the client nodes going down, only experiencing a brief spike of 7% more memory usage than normal.

Steps to Reproduce

Not known, but the blocks close to the last advanced by the clients should indicate what triggered the issue and lead to way to reproduce.

It's been mentioned that blocks 55840, 56108, 68201, 70928 have complex program deployments.

Expected Behavior

snarkOS controlling the amount that it requests and not letting the amount of complexity of deployments to dictate memory, so that also it does not becomes a security issue (like DDoS).

Your Environment

• snarkOS Version testnet-beta-v0.3.0
• Rust Version 1.79.0
• Ubuntu 22.04LTS

[vicsn]

Haruka:

block 55840, 56108, 68201, 70928 all had fairly complex program deployments (very high fee)

[vicsn]

Might be: #2970

[jhaineault-aleo]

More snarkos OOM kill incidents:

Incident Date: July 4, 2024
Affected Service: snarkos Service
Nodes Affected: Node E, Node A, Node B
Summary
On July 4, 2024, the snarkos service on client nodes Node E, Node A and Node B experienced an outage due to the OOM (Out of Memory) killer being invoked at around the same time on all three nodes.

Incident Details

Node IP: Node E

• Node type: Core Client
• Timestamp: 7:16:52 AM EDT
• Memory Requested by snarkos: 105.8 GB
• Memory Used: 30.1 GB
• RAM Memory on the server: 32GB
• Last Block Advanced before the incident: 90153

Node: Node A

• Node type: Core Client
• Timestamp: 7:16:42 AM EDT
• Memory Requested by snarkos: 107.3 GB
• Memory Used: 30.0 GB
• RAM Memory on the server: 32GB
• Last Block Advanced before the incident: 90153

Node: Node B

• Node type: Core Client
• Timestamp: 7:16:42 AM EDT
• Memory Requested by snarkos: 109.6 GB
• Memory Used: 30.0 GB
• RAM Memory on the server: 32GB
• Last Block Advanced before the incident: 90153

Effect on Validator(s)
Validator nodes were minimally affected by the client nodes going down, only experiencing a brief spike of 12% more memory usage than normal.

Incident Date: July 5, 2024
Affected Service: snarkos Service
Nodes Affected: Node C
Summary
On July 5, 2024, the snarkos service on client node 34.100.171.155 experienced an outage due to the OOM (Out of Memory) killer being invoked.

Incident Details

Node IP: Node C

• Node type: Core Client
• Timestamp: 6:13:25 AM EDT
• Memory Requested by snarkos: 104.7 GB
• Memory Used: 30.1 GB
• RAM Memory on the server: 32GB
• Last Block Advanced before the incident: 99991

Effect on Validator(s)
Validator nodes were minimally affected by the client nodes going down, only experiencing a brief spike of 4% more memory usage than normal.

[vicsn]

I'll be working on reimplementing #2970 tomorrow