Skip to content

Commit

Permalink
Fix stress test op count (microsoft#14499)
Browse files Browse the repository at this point in the history
## Description

ADO:3647

When the stress test clients receive a nack, the client will reconnect
and the number of ops it has already sent will reset. This means that
for every nack and depending on when it actually happens (within the op
send window), the client will send more ops than configured, as it
thinks it starts fresh.

In the worst case scenario, the test is configured to send 10k ops with
120 clients => each client sends ~80 ops and each client receives a nack
at around sending op number 79. The test will end up sending ~20k ops,
which will double for every time this happens to all clients.

Additionally, if one of the clients keeps getting nacked, it will send
ops until it reaches 80 without getting nacked. This means it's possible
(but improbable) for a client to run forever.
  • Loading branch information
andre4i committed Mar 9, 2023
1 parent 7f63f59 commit d275d67
Showing 1 changed file with 27 additions and 3 deletions.
30 changes: 27 additions & 3 deletions packages/test/test-service-load/src/loadTestDataStore.ts
Original file line number Diff line number Diff line change
Expand Up @@ -584,6 +584,24 @@ class LoadTestDataStore extends DataObject implements ILoadTest {
// To avoid growing the file size unnecessarily, not all clients should be sending large ops
const maxClientsSendingLargeOps = config.testConfig.content?.numClients ?? 1;
let opsSent = 0;
let largeOpsSent = 0;

const reportOpCount = (reason: string, error?: Error) => {
config.logger.sendTelemetryEvent(
{
eventName: "OpCount",
reason,
runId: config.runId,
documentOpCount: dataModel.counter.value,
localOpCount: opsSent,
localLargeOpCount: largeOpsSent,
},
error,
);
};

this.runtime.once("dispose", () => reportOpCount("Disposed"));
this.runtime.once("disconnected", () => reportOpCount("Disconnected"));

const sendSingleOp = () => {
if (
Expand All @@ -609,10 +627,11 @@ class LoadTestDataStore extends DataObject implements ILoadTest {
opsSent,
largeOpRate,
});
} else {
dataModel.counter.increment(1);

largeOpsSent++;
}

dataModel.counter.increment(1);
opsSent++;
};

Expand Down Expand Up @@ -645,7 +664,7 @@ class LoadTestDataStore extends DataObject implements ILoadTest {
};

try {
while (opsSent < clientSendCount && !this.disposed) {
while (dataModel.counter.value < clientSendCount && !this.disposed) {
// this enables a quick ramp down. due to restart, some clients can lag
// leading to a slow ramp down. so if there are less than half the clients
// and it's partner is done, return true to complete the runner.
Expand All @@ -660,7 +679,12 @@ class LoadTestDataStore extends DataObject implements ILoadTest {
}
await sendSingleOpAndThenWait();
}

reportOpCount("Completed");
return !this.runtime.disposed;
} catch (error: any) {
reportOpCount("Exception", error);
throw error;
} finally {
dataModel.printStatus();
}
Expand Down

0 comments on commit d275d67

Please sign in to comment.