Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Pending Request causing timeout #636

Merged
merged 3 commits into from
Jul 9, 2024
Merged

Conversation

ikreymer
Copy link
Member

@ikreymer ikreymer commented Jul 8, 2024

Occasionally, it seems some pending requests that only go through Network.requestWillBeSent but never receive a finished/failed/blocked message persist in the pending request list.

Instead of waiting for them to finish until timeout, just drop these requests. Requests that have never been intercepted (fetch is set) and are not being loaded asynchronously (asyncLoading is set) should be discarded when the page is done. Large requests that are streaming the response body, etc.. will have one of these flags set.

(Chrome devtools also handles these by marking their state to 'unknown')..

Test:

  • Run a crawl of a few pages of: https://www.cbc.ca/news/canada with debug logging
  • Observe a few (possibly blocked) tracking requests are stuck in "Finishing pending requests for page" without this fix.
  • With this fix, they should be dropped and Removing pending request that was never fetched is logged.

…ere never intercepted

by requestPaused() and not being async loaded (only intercepted by requestWillBeSent, so may never have been sent)
reduces page waiting for spurrious requests
@ikreymer ikreymer requested a review from tw4l July 8, 2024 20:16
Copy link
Contributor

@tw4l tw4l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great improvement, and thanks for the URL to test against :)

const { redirectResponse, requestId, request, type } = params;

const { headers, method, url } = request;

logNetwork("Network.requestWillBeSent", {
requestId,
url,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice addition

@ikreymer ikreymer merged commit 0038e3a into main Jul 9, 2024
4 checks passed
ikreymer added a commit that referenced this pull request Jul 9, 2024
Don't wait for requests that have been not intercepted (`intercepting` is not set) and are not loaded asynchronously (`asyncLoading` is not set) in awaitPageResources() when page is done. Occasionally, it seems some pending requests that only get added via `Network.requestWillBeSent` but never receive a finished/failed message may persist in the pending request list, and will now be discarded.
(Large requests that have a streaming response body will have either `intercepting` or `asyncLoading` set and will not be affected)
@ikreymer ikreymer deleted the fix-pending-request-not-fetched branch July 9, 2024 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants