Error handling in custom benchmark

[jikim-msft]

Description

#AB12925

Found a bug in benchmarkCustom() function not throwing an error.

This PR makes following changes:

• Adds try-catch logic to the benchmarkCustom() function.
• Adds error event from the benchmarkCustom() function which gets emitted to the MochaReporter.
• Enables --verbose on the test:customBenchmarks script for better logging.

Result

With the changes above, the following code throws an error on console:

			benchmarkCustom({
				only: true,
				type: BenchmarkType.Measurement,
				title: `SharedTree`,
				run: async (reporter) => {
					throw new Error("This error is INTENTIONAL.");
				},
			});

[msfluid-bot]

⯅ @fluid-example/bundle-size-tests: +245 Bytes

Metric Name	Baseline Size	Compare Size	Size Diff
aqueduct.js	461.15 KB	461.18 KB	⯅ +35 Bytes
azureClient.js	559.19 KB	559.24 KB	⯅ +49 Bytes
connectionState.js	680 Bytes	680 Bytes	■ No change
containerRuntime.js	261.99 KB	262 KB	⯅ +14 Bytes
fluidFramework.js	399.76 KB	399.77 KB	⯅ +14 Bytes
loader.js	134.26 KB	134.28 KB	⯅ +14 Bytes
map.js	42.39 KB	42.39 KB	⯅ +7 Bytes
matrix.js	146.56 KB	146.56 KB	⯅ +7 Bytes
odspClient.js	526.47 KB	526.52 KB	⯅ +49 Bytes
odspDriver.js	97.72 KB	97.74 KB	⯅ +21 Bytes
odspPrefetchSnapshot.js	42.78 KB	42.79 KB	⯅ +14 Bytes
sharedString.js	163.26 KB	163.26 KB	⯅ +7 Bytes
sharedTree.js	390.27 KB	390.28 KB	⯅ +7 Bytes
Total Size	3.3 MB	3.3 MB	⯅ +245 Bytes

---

Baseline commit: b7e4e1e

Generated by 🚫 dangerJS against 3f2a5c1

[alexvy86]

A couple of things but it looks pretty good to me. We should make sure that with these changes, using benchmarkCustom() behaves as expected in tree or tablebench, both in success and failure cases, and both with --perfMode and without. The usual pnpm-linking / override method to test that should still work.

[jikim-msft]

/*!
 * Copyright (c) Microsoft Corporation and contributors. All rights reserved.
 * Licensed under the MIT License.
 */

"use strict";

const getFluidTestMochaConfig = require("@fluid-internal/mocha-test-setup/mocharc-common");

const packageDir = __dirname;
const config = getFluidTestMochaConfig(packageDir);

// TODO: Revert this
const spec =
	config["spec"] !== undefined
		? Array.isArray(config["spec"])
			? config["spec"]
			: [config["spec"]] // If string, wrap as array to use spread operator
		: []; // If undefined, use an empty array

spec.push("--perfMode");

module.exports = {
	...config,
	spec: spec,
};

1. Add --perfMode to .mocharc.cjs's spec and run pnpm run test:mocha

2. Add --perfMode flag to the script and run pnpm run test:mocha -- --perfMode

I added logging to ensure that the script & .mocharc BOTH work in --perfMode

if (isInPerformanceTestingMode) {
    console.log(`PERFMODE Benchmark error: ${benchmarkError.error}`);
    return;
}

1. pnpm run test:mocha:verbose -- --perfMode

2. "test:mocha": "cross-env FLUID_TEST_VERBOSE=1 mocha \"dist/test/**/*.js\""

[jikim-msft]

With the change in the Reporter.ts, errors are correctly logged using the table.

[alexvy86]

With the change in the Reporter.ts, errors are correctly logged using the table.

Awesome! Can you confirm that things look right if there's a mix of successful and failing tests in a given suite? The successful test should still get the custom data printed, and I'd like to see that the failed tests that won't report those columns still look correct and don't affect the rendering of the successful tests.

[jikim-msft]

benchmarkCustom() is now able to report all the test results without crashing.

#22245 (comment)

Also from the comment above, I wanted to make sure that the error messages from --perfMode and non-perfMode should be different as the events get emitted at different time.

--perfMode: return ;
non perfMode: throw error

Both emit benchmark end before running isInPerformanceTestingMode.

[jikim-msft]

pnpm run test

pnpm run test:customBenchmarks

pnpm run test:customBenchmarks -- --perfMode

[github-actions]

🔗 No broken links found! ✅

Your attention to detail is admirable.

linkcheck output

> fluidframework-docs@0.25.0 ci:linkcheck /home/runner/work/FluidFramework/FluidFramework/docs
> start-server-and-test ci:start 1313 linkcheck:full

1: starting server using command "npm run ci:start"
and when url "[ 'http://127.0.0.1:1313' ]" is responding with HTTP status code 200
running tests using command "npm run linkcheck:full"


> fluidframework-docs@0.25.0 ci:start
> http-server ./public --port 1313 --silent


> fluidframework-docs@0.25.0 linkcheck:full
> npm run linkcheck:fast -- --external


> fluidframework-docs@0.25.0 linkcheck:fast
> linkcheck http://localhost:1313 --skip-file skipped-urls.txt --external

Crawling...

Stats:
  378023 links
    2980 destination URLs
       2 URLs ignored
       0 warnings
       0 errors

[jikim-msft]

Verified that the errors are logging in both non-perfMode and perfMode.