Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices #10111

sohami · 2023-09-18T21:20:37Z

Description

There can be cases where one or more slice may not have timing related information for its leaves in contexts map. During creation of slice and query level breakdown map it needs to handle such cases by using default values correctly. Also updating the min/max/avg sliceNodeTime to not include time to create weight and wait times by slice threads. It will reflect the min/max/avg execution time of each slice whereas totalNodeTime will reflect the total query time.

Related Issues

Resolves #9815

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-09-18T21:52:16Z

Compatibility status:

Checks if related components are compatible with change 9fb011c

Incompatible components

Incompatible components: [https://github.com/opensearch-project/k-nn.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

sohami · 2023-09-18T21:59:10Z

@ticheng-aws @reta Can you please help with review.

github-actions · 2023-09-18T22:03:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25816/
CommitID: d17a81d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sohami · 2023-09-19T17:46:53Z

We need to update CHANGELOG.md as well.

Don't see a need for Changelog as it is a bug fix for unreleased changes.

.../src/test/java/org/opensearch/search/profile/query/ConcurrentQueryProfileBreakdownTests.java

github-actions · 2023-09-20T03:07:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}

URL: https://build.ci.opensearch.org/job/gradle-check/25889/
CommitID: 1d04023
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

server/src/main/java/org/opensearch/search/profile/query/ConcurrentQueryProfileBreakdown.java

…or slices. There can be cases where one or more slice may not have timing related information for its leaves in contexts map. During creation of slice and query level breakdown map it needs to handle such cases by using default values correctly. Also updating the min/max/avg sliceNodeTime to not include time to create weight and wait times by slice threads. It will reflect the min/max/avg execution time of each slice whereas totalNodeTime will reflect the total query time. Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

github-actions · 2023-09-20T19:44:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25943/
CommitID: 9fb011c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-20T21:24:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25965/
CommitID: 9fb011c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sohami · 2023-09-21T03:31:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌

URL: https://build.ci.opensearch.org/job/gradle-check/25965/

CommitID: 9fb011c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

All known issues:
#5030
#10133

reta · 2023-09-21T11:55:52Z

@sohami I understand the frustration regarding failing checks but we should not merge pull requests that have them failed

opensearch-trigger-bot · 2023-09-21T11:59:16Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-10111-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9f0e01743de1bf23264945d54d0d5ab20593020b
# Push it to GitHub
git push --set-upstream origin backport/backport-10111-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-10111-to-2.x.

…or slices (opensearch-project#10111) * Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices. There can be cases where one or more slice may not have timing related information for its leaves in contexts map. During creation of slice and query level breakdown map it needs to handle such cases by using default values correctly. Also updating the min/max/avg sliceNodeTime to not include time to create weight and wait times by slice threads. It will reflect the min/max/avg execution time of each slice whereas totalNodeTime will reflect the total query time. Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Address review comments Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> --------- Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

…or slices (opensearch-project#10111) * Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices. There can be cases where one or more slice may not have timing related information for its leaves in contexts map. During creation of slice and query level breakdown map it needs to handle such cases by using default values correctly. Also updating the min/max/avg sliceNodeTime to not include time to create weight and wait times by slice threads. It will reflect the min/max/avg execution time of each slice whereas totalNodeTime will reflect the total query time. Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Address review comments Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> --------- Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…or slices (opensearch-project#10111) * Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices. There can be cases where one or more slice may not have timing related information for its leaves in contexts map. During creation of slice and query level breakdown map it needs to handle such cases by using default values correctly. Also updating the min/max/avg sliceNodeTime to not include time to create weight and wait times by slice threads. It will reflect the min/max/avg execution time of each slice whereas totalNodeTime will reflect the total query time. Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Address review comments Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> --------- Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

…#10898) * Add support for query profiler with concurrent aggregation (#9248) * Add support for query profiler with concurrent aggregation (#9248) Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Refactor and work on the PR comments Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Update collectorToLeaves mapping for children breakdowns post profile metric collection and before creating the results Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Refactor logic to compute the slice level breakdown stats and query level breakdown stats for concurrent search case Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Fix QueryProfilePhaseTests and QueryProfileTests, and parameterize QueryProfilerIT with concurrent search enabled Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Handle the case when there are no leaf context to compute the profile stats to return default stats for all breakdown type along with min/max/avg values. Replace queryStart and queryEnd time with queryNodeTime Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Add UTs for ConcurrentQueryProfileBreakdown Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Add concurrent search stats test into the QueryProfilerIT Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Address review comments Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> --------- Signed-off-by: Ticheng Lin <ticheng@amazon.com> Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> Co-authored-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices (#10111) * Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices. There can be cases where one or more slice may not have timing related information for its leaves in contexts map. During creation of slice and query level breakdown map it needs to handle such cases by using default values correctly. Also updating the min/max/avg sliceNodeTime to not include time to create weight and wait times by slice threads. It will reflect the min/max/avg execution time of each slice whereas totalNodeTime will reflect the total query time. Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Address review comments Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> --------- Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Fix flaky query profile phase tests with concurrent search enabled (#10547) (#10547) Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Introduce ConcurrentQueryProfiler to profile query using concurrent segment search path and support concurrency during rewrite and create weight (#10352) * Fix timer race condition in profile rewrite and create weight for concurrent segment search (#10352) Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Refactor and work on the PR comments (#10352) Signed-off-by: Ticheng Lin <ticheng@amazon.com> --------- Signed-off-by: Ticheng Lin <ticheng@amazon.com> --------- Signed-off-by: Ticheng Lin <ticheng@amazon.com> Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> Co-authored-by: Ticheng Lin <51488860+ticheng-aws@users.noreply.github.com>

ticheng-aws · 2023-11-07T23:36:49Z

Resolves #9869

…or slices (opensearch-project#10111) * Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices. There can be cases where one or more slice may not have timing related information for its leaves in contexts map. During creation of slice and query level breakdown map it needs to handle such cases by using default values correctly. Also updating the min/max/avg sliceNodeTime to not include time to create weight and wait times by slice threads. It will reflect the min/max/avg execution time of each slice whereas totalNodeTime will reflect the total query time. Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Address review comments Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> --------- Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

github-actions bot added bug Something isn't working flaky-test Random test failure that succeeds on second run labels Sep 18, 2023

sohami added the skip-changelog label Sep 18, 2023

sohami force-pushed the pr-9815 branch from d17a81d to 6a316e2 Compare September 18, 2023 21:25

ticheng-aws approved these changes Sep 19, 2023

View reviewed changes

ticheng-aws reviewed Sep 19, 2023

View reviewed changes

.../src/test/java/org/opensearch/search/profile/query/ConcurrentQueryProfileBreakdownTests.java Show resolved Hide resolved

sohami force-pushed the pr-9815 branch from 6a316e2 to 1d04023 Compare September 20, 2023 02:20

reta reviewed Sep 20, 2023

View reviewed changes

server/src/main/java/org/opensearch/search/profile/query/ConcurrentQueryProfileBreakdown.java Outdated Show resolved Hide resolved

reta approved these changes Sep 20, 2023

View reviewed changes

sohami added 2 commits September 20, 2023 11:55

Address review comments

9fb011c

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

sohami force-pushed the pr-9815 branch from 1d04023 to 9fb011c Compare September 20, 2023 18:55

sohami merged commit 9f0e017 into opensearch-project:main Sep 21, 2023
10 of 11 checks passed

reta added the backport 2.x Backport to 2.x branch label Sep 21, 2023

opensearch-trigger-bot bot added the backport-failed label Sep 21, 2023

ticheng-aws mentioned this pull request Oct 23, 2023

Introduce ConcurrentQueryProfiler to profile query using concurrent segment search path and support concurrency during rewrite and create weight #10352

Merged

7 tasks

sohami mentioned this pull request Oct 24, 2023

Backport of PRs for profile related changes in concurrent search path #10898

Merged

8 tasks

ticheng-aws mentioned this pull request Nov 7, 2023

[Concurrent Segment Search] [BUG] org.opensearch.search.profile.query.QueryProfilerIT.testProfileMatchesRegular {p0={"search.concurrent_segment_search.enabled":"true"} p1=true} is flaky #9869

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices #10111

Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices #10111

sohami commented Sep 18, 2023

github-actions bot commented Sep 18, 2023 •

edited

Loading

sohami commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

sohami commented Sep 19, 2023

github-actions bot commented Sep 20, 2023

github-actions bot commented Sep 20, 2023

github-actions bot commented Sep 20, 2023

sohami commented Sep 21, 2023

Gradle Check (Jenkins) Run Completed with:

reta commented Sep 21, 2023

opensearch-trigger-bot bot commented Sep 21, 2023

ticheng-aws commented Nov 7, 2023

Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices #10111

Fix NPE in ConcurrentQueryProfile while computing the breakdown map for slices #10111

Conversation

sohami commented Sep 18, 2023

Description

Related Issues

Check List

github-actions bot commented Sep 18, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

sohami commented Sep 18, 2023

github-actions bot commented Sep 18, 2023

Gradle Check (Jenkins) Run Completed with:

sohami commented Sep 19, 2023

github-actions bot commented Sep 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 20, 2023

Gradle Check (Jenkins) Run Completed with:

sohami commented Sep 21, 2023

Gradle Check (Jenkins) Run Completed with:

reta commented Sep 21, 2023

opensearch-trigger-bot bot commented Sep 21, 2023

ticheng-aws commented Nov 7, 2023

github-actions bot commented Sep 18, 2023 •

edited

Loading