Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ytInitialPlayerResponse checker to let user easily know problem #3027

Merged
merged 1 commit into from
Jul 25, 2024

Conversation

DGideas
Copy link
Contributor

@DGideas DGideas commented Jul 25, 2024

Today I have tried use you-get to download video from YouTube. I noticed one of my hosts from OVH Cloud FR failed to download the example video:

(venv) ubuntu@fr1:/tmp$ you-get "https://www.youtube.com/watch?v=jNQXAC9IVRw" --debug
[DEBUG] Extracting from the video page...
[DEBUG] get_content: https://www.youtube.com/watch?v=jNQXAC9IVRw
[DEBUG] Retrieving the player code...
[DEBUG] get_content: https://www.youtube.com/s/player/1f8742dc/player_ias.vflset/fr_FR/base.js
[DEBUG] Loading ytInitialPlayerResponse...
you-get: version 0.4.1718, a tiny downloader that scrapes the web.
you-get: Namespace(version=False, help=False, info=False, url=False, json=False, no_merge=False, no_caption=False, postfix=False, prefix=None, force=False, skip_existing_file_size_check=False, format=None, output_filename=None, output_dir='.', player=None, cookies=None, timeout=600, debug=True, input_file=None, password=None, playlist=False, first=None, last=None, size=None, auto_rename=False, insecure=False, http_proxy=None, extractor_proxy=None, no_proxy=False, socks_proxy=None, stream=None, itag=None, m3u8=False, URL=['https://www.youtube.com/watch?v=jNQXAC9IVRw'])
Traceback (most recent call last):
  File "/home/ubuntu/venv/bin/you-get", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/__main__.py", line 92, in main
    main(**kwargs)
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/common.py", line 1883, in main
    script_main(any_download, any_download_playlist, **kwargs)
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/common.py", line 1772, in script_main
    download_main(
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/common.py", line 1386, in download_main
    download(url, **kwargs)
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/common.py", line 1874, in any_download
    m.download(url, **kwargs)
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/extractor.py", line 48, in download_by_url
    self.prepare(**kwargs)
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/extractors/youtube.py", line 207, in prepare
    self.title = ytInitialPlayerResponse["videoDetails"]["title"]
                 ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'videoDetails'

In advanced investigation, I noticed that the returned ytInitialPlayerResponse which raised exception have the following format:

{
    "responseContext": {...
    },
    "playabilityStatus": {...
    },
    "trackingParams": "CAAQu2kiEwiylZzn6sGHAxXVxkkHHVxgAe4=",
    "adBreakHeartbeatParams": "Q0FBJTNE",
    "frameworkUpdates": {...
    },
}

Which a normal ytInitialPlayerResponse from my another host should look like this:

{
    "responseContext": {...
    },
    "playabilityStatus": {...
    },
    "streamingData": {...
    },
    "playbackTracking": {...
    },
    "captions": {...
    },
    "videoDetails": {...
    },
    "annotations": {...
    },
    "playerConfig": {...
    },
    "storyboards": {...
    },
    "microformat": {...
    },
    "cards": {...
    },
    "trackingParams": "CAAQu2kiEwiylZzn6sGHAxXVxkkHHVxgAe4=",
    "attestation": {...
    },
    "videoQualityPromoSupportedRenderers": {...
    },
    "messages": {...
    },
    "adBreakHeartbeatParams": "Q0FBJTNE",
    "cpnInfo": {...
    },
    "frameworkUpdates": {...
    },
}

The content inner playabilityStatus has the answer why ytInitialPlayerResponse can not return video details:

    "playabilityStatus": {
        "status": "LOGIN_REQUIRED",
        "reason": "Connectez-vous pour confirmer que vous n'\u00eates pas un robot",
        ...
    }

I submit this commit to raise AssertionError to users when returned status is not okay, this error message helps user know about what happened.

After this commit, user could get a better error message like this:

(env) ubuntu@fr1:/tmp/you-get$ ./you-get "https://www.youtube.com/watch?v=jNQXAC9IVRw" --debug
/tmp/you-get/src/you_get/extractors/youtube.py:198: SyntaxWarning: invalid escape sequence '\W'
  if re.search('\Wlist=', self.url) and not kwargs.get('playlist'):
/tmp/you-get/src/you_get/extractors/youtube.py:206: SyntaxWarning: invalid escape sequence '\.'
  jsUrl = re.search('([^"]*/base\.js)"', video_page).group(1)
/tmp/you-get/src/you_get/extractors/youtube.py:214: SyntaxWarning: invalid escape sequence '\s'
  ytInitialPlayerResponse = json.loads(re.search('ytInitialPlayerResponse\s*=\s*([^\n]+?});(\n|</script>|var )', video_page).group(1))
[DEBUG] Extracting from the video page...
[DEBUG] get_content: https://www.youtube.com/watch?v=jNQXAC9IVRw
[DEBUG] Retrieving the player code...
[DEBUG] get_content: https://www.youtube.com/s/player/3400486c/player_ias.vflset/fr_FR/base.js
[DEBUG] Loading ytInitialPlayerResponse...
you-get: version 0.4.1718, a tiny downloader that scrapes the web.
you-get: Namespace(version=False, help=False, info=False, url=False, json=False, no_merge=False, no_caption=False, postfix=False, prefix=None, force=False, skip_existing_file_size_check=False, format=None, output_filename=None, output_dir='.', player=None, cookies=None, timeout=600, debug=True, input_file=None, password=None, playlist=False, first=None, last=None, size=None, auto_rename=False, insecure=False, http_proxy=None, extractor_proxy=None, no_proxy=False, socks_proxy=None, stream=None, itag=None, m3u8=False, URL=['https://www.youtube.com/watch?v=jNQXAC9IVRw'])
Traceback (most recent call last):
  File "/tmp/you-get/./you-get", line 11, in <module>
    you_get.main(repo_path=_filepath)
  File "/tmp/you-get/src/you_get/__main__.py", line 92, in main
    main(**kwargs)
  File "/tmp/you-get/src/you_get/common.py", line 1883, in main
    script_main(any_download, any_download_playlist, **kwargs)
  File "/tmp/you-get/src/you_get/common.py", line 1772, in script_main
    download_main(
  File "/tmp/you-get/src/you_get/common.py", line 1386, in download_main
    download(url, **kwargs)
  File "/tmp/you-get/src/you_get/common.py", line 1874, in any_download
    m.download(url, **kwargs)
  File "/tmp/you-get/src/you_get/extractor.py", line 48, in download_by_url
    self.prepare(**kwargs)
  File "/tmp/you-get/src/you_get/extractors/youtube.py", line 215, in prepare
    self.check_playability_response(ytInitialPlayerResponse)
  File "/tmp/you-get/src/you_get/extractors/youtube.py", line 184, in check_playability_response
    raise AssertionError(
AssertionError: Server refused to provide video details. Returned status: LOGIN_REQUIRED, reason: Connectez-vous pour confirmer que vous n'êtes pas un robot.

@soimort soimort merged commit 4ab0221 into soimort:develop Jul 25, 2024
9 checks passed
@soimort
Copy link
Owner

soimort commented Jul 25, 2024

Nice. Thanks

@DGideas DGideas deleted the patch-1 branch July 26, 2024 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants