1
0
Fork 0
mirror of https://github.com/yt-dlp/yt-dlp synced 2025-01-19 10:26:56 +01:00
Commit graph

1306 commits

Author SHA1 Message Date
bashonly
52414d64ca
[utils] js_to_json: Handle Array objects
Authored by: Grub4K, std-move

Co-authored-by: std-move <26625259+std-move@users.noreply.github.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
2023-09-21 16:51:57 -05:00
coletdjnz
20fbbd9249
[networking] Fix various socks proxy bugs ()
- Fixed support for IPv6 socks proxies
- Fixed support for IPv6 over socks5
- Fixed --source-address not being obeyed for socks4 and socks5
- Fixed socks4a when the destination address is an IPv4 address

Closes https://github.com/yt-dlp/yt-dlp/issues/7959
Fixes https://github.com/ytdl-org/youtube-dl/issues/15368

Authored by: coletdjnz
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
Co-authored-by: bashonly <bashonly@bashonly.com>
2023-09-18 07:33:26 +00:00
coletdjnz
fcd6a76adc
[tests] Add tests for socks proxies ()
Authored by: coletdjnz
2023-08-25 07:10:44 +00:00
coletdjnz
db7b054a61
[networking] Add request handler preference framework ()
Preference functions that take a request and a request handler instance can be registered to prioritize different request handlers per request.

Authored by: coletdjnz
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
2023-08-04 22:17:48 +00:00
pukkandan
6148833f5c
[cleanup] Misc 2023-07-30 04:06:18 +05:30
pukkandan
3f7965105d
[utils] HTTPHeaderDict: Handle byte values 2023-07-30 03:18:10 +05:30
pukkandan
de20687ee6
[test] Fix test_load_certifi
Closes , 
2023-07-29 21:53:00 +05:30
coletdjnz
4bf912282a
[networking] Remove dot segments during URL normalization ()
This implements RFC3986 5.2.4 remove_dot_segments during the URL normalization process.

Closes , 

Authored by: coletdjnz
2023-07-28 22:40:20 +00:00
coletdjnz
bbeacff7fc
[networking] Ignore invalid proxies in env ()
Authored by: coletdjnz
2023-07-27 20:26:02 +05:30
coletdjnz
95abea9a03
[test] Fix httplib_validation_errors test for old Python versions ()
Fixes https://github.com/yt-dlp/yt-dlp/issues/7674

Authored by: coletdjnz
2023-07-24 19:18:52 +00:00
coletdjnz
86aea0d3a2
[networking] Add strict Request extension checking ()
Authored by: coletdjnz
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
2023-07-23 05:17:15 +00:00
pukkandan
a250b24733
[compat] Ensure submodules are imported correctly
Closes 
2023-07-22 18:10:35 +05:30
pukkandan
62b5c94cad
[cleanup] Misc fixes
Closes 
2023-07-22 09:09:52 +05:30
pukkandan
a264433c9f
[outtmpl] Fix replacement for playlist_index 2023-07-22 08:57:43 +05:30
bashonly
e57eb98222
[fd/external] Fix ffmpeg input from stdin ()
Bugfix for 1ceb657bdd

Authored by: bashonly
2023-07-22 02:32:49 +00:00
bashonly
af86873218
[utils] Improve parse_duration
Authored by: bashonly
2023-07-20 08:40:31 -05:00
bashonly
75dc8e673b
[networking] Fix --legacy-server-connect ()
Bugfix for 227bf1a33b

Authored by: bashonly
2023-07-20 13:31:17 +00:00
bashonly
71baa490eb
[networking] Fix POST requests with zero-length payloads ()
Bugfix for 227bf1a33b

Authored by: bashonly
2023-07-20 13:23:30 +00:00
bashonly
42ded0a429
[fd/external] Fixes to cookie handling
- Fix bug in `axel` Cookie header arg
- Pass cookies to `curl` as strings
- Write session cookies for `aria2c` and `wget`

Closes 
Authored by: bashonly
2023-07-15 15:25:51 -05:00
bashonly
6c5211cebe
[core] Fix HTTP headers and cookie handling
- Remove `Cookie` header from `http_headers` immediately after loading into cookiejar
- Restore compat for `--load-info-json` cookies
- Add more tests
- Fix improper passing of Cookie header by `MailRu` extractor

Closes 
Authored by: bashonly, pukkandan
2023-07-15 15:25:45 -05:00
coletdjnz
3d2623a898
[compat, networking] Deprecate old functions ()
Authored by: coletdjnz, pukkandan
2023-07-15 16:18:35 +05:30
coletdjnz
227bf1a33b
[networking] Rewrite architecture ()
New networking interface consists of a `RequestDirector` that directs
each `Request` to appropriate `RequestHandler` and returns the
`Response` or raises `RequestError`. The handlers define adapters to
transform its internal Request/Response/Errors to our interfaces.

User-facing changes:
- Fix issues with per request proxies on redirects for urllib
- Support for `ALL_PROXY` environment variable for proxy setting
- Support for `socks5h` proxy
   - Closes https://github.com/yt-dlp/yt-dlp/issues/6325, https://github.com/ytdl-org/youtube-dl/issues/22618, https://github.com/ytdl-org/youtube-dl/pull/28093
- Raise error when using `https` proxy instead of silently converting it to `http`

Authored by: coletdjnz
2023-07-15 16:18:35 +05:30
pukkandan
c365dba843
[networking] Add module ()
No actual changes - code is only moved around
2023-07-15 16:18:34 +05:30
pukkandan
1b392f905d
[utils] Add temporary shim for logging
Related: , 
2023-07-15 16:18:34 +05:30
Mahmoud Abdel-Fattah
2af4eeb772
[utils] clean_podcast_url: Handle more trackers ()
Authored by: mabdelfattah, bashonly
Closes 
2023-07-11 06:30:38 +05:30
Simon Sawicki
3121512228
[core] Change how Cookie headers are handled
Cookies are now saved and loaded under `cookies` key in the info dict
instead of `http_headers.Cookie`. Cookies passed in headers are
auto-scoped to the input URLs with a warning.

Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj

Authored by: Grub4K
2023-07-06 23:14:39 +05:30
coletdjnz
f8b4bcc0a7
[core] Prevent Cookie leaks on HTTP redirect
Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj

Authored by: coletdjnz
2023-07-06 23:14:39 +05:30
bashonly
1ceb657bdd
[fd/external] Scope cookies
- ffmpeg: Calculate cookies from cookiejar and pass with `-cookies` arg instead of `-headers`
- aria2c, curl, wget: Write cookiejar to file and use external FD built-in cookiejar support
- httpie: Calculate cookies from cookiejar instead of `http_headers`
- axel: Calculate cookies from cookiejar and disable http redirection if cookies are passed
    - May break redirects, but axel simply don't have proper cookie support

Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj

Authored by: bashonly, coletdjnz
2023-07-06 23:14:38 +05:30
pukkandan
47bcd43724
[outtmpl] Pad playlist_index etc even when with internal formatting
Closes 
2023-07-06 20:22:03 +05:30
pukkandan
cd810afe2a
[extractor/youtube] Improve nsig function name extraction 2023-06-22 13:27:18 +05:30
pukkandan
42f2d40b47
Update to ytdl-commit-07af47
[YouTube] Improve fix for ae8ba2c
07af47960f
2023-06-21 09:21:23 +05:30
pukkandan
1619ab3e67
Bugfix for ebe1b4e34f 2023-06-21 09:21:22 +05:30
pukkandan
ad54c9130e
[cleanup] Misc
Closes , Closes , Closes , Closes , Closes 
Authored by: mikf, freezboltz, pukkandan
2023-06-21 09:21:20 +05:30
pukkandan
ebe1b4e34f
[outtmpl] Fix some minor bugs
Closes 
2023-06-21 06:10:39 +05:30
pukkandan
eedda5252c
[utils] FormatSorter: Improve size and br
Closes 

Previously, when some formats have accurate size and some approximate,
the ones with accurate size was always prioritized

For formats with known tbr and unknown vbr/abr, we were setting
(vbr=tbr, abr=0) for sorting to work. This is no longer needed.

Authored by pukkandan, u-spec-png
2023-06-21 06:10:38 +05:30
pukkandan
01aba2519a
[jsinterp] Fix global object extraction
Closes 
2023-06-18 04:11:15 +05:30
bashonly
b4a252fba8
[jsinterp] Fix division ()
* Fixes nsig decryption for Youtube JS player `8c7583ff`

Authored by: bashonly
2023-06-10 22:49:12 +00:00
Matt Broadway
b38d4c941d
[cookies] Update for chromium changes ()
Authored by: mbway
2023-05-29 19:21:35 +05:30
coletdjnz
daafbf49b3
[core] Support decoding multiple content encodings ()
Authored by: coletdjnz
2023-05-27 10:40:05 +00:00
coletdjnz
b87e01c123
[cookies] Move YoutubeDLCookieJar to cookies module ()
Authored by: coletdjnz
2023-05-27 07:08:19 +00:00
coletdjnz
08916a49c7
[core] Improve HTTP redirect handling ()
Aligns HTTP redirect handling with what browsers commonly do and RFC standards. 

Fixes issues afac4caa7d missed.

Authored by: coletdjnz
2023-05-27 07:06:13 +00:00
pukkandan
7aeda6cc9e
[jsinterp] Do not compile regex 2023-05-24 23:30:45 +05:30
pukkandan
4823ec9f46
Update to ytdl-commit-d1c6c5
[YouTube] [core] Improve platform debug log, based on yt-dlp
d1c6c5c4d6

Except:
    * 6ed34338285f722d0da312ce0af3a15a077a3e2a [jsinterp] Add short-cut evaluation for common expression
        * There was no performance improvement when tested with https://github.com/ytdl-org/youtube-dl/issues/30641
    * e8de54bce50f6f77a4d7e8e80675f7003d5bf630 [core] Handle `/../` sequences in HTTP URLs
        * We plan to implement this differently
2023-05-24 23:30:43 +05:30
pukkandan
6f2287cb18
[cleanup] Misc
Closes , closes 
2023-05-20 04:23:41 +05:30
pukkandan
1d7656184c
[jsinterp] Handle NaN in bitwise operators
Closes 
2023-05-20 04:07:17 +05:30
Simon Sawicki
b079c26f0a
[utils] traverse_obj: More fixes ()
- Fix result when branching with `traverse_string`
- Fix `slice` path on `dict`s
- Fix tests and docstrings from 21b5ec86c2
- Add `is_iterable_like` helper function

Authored by: Grub4K
2023-04-30 19:50:22 +02:00
pukkandan
7cf51f2191
[jsinterp] Handle negative numbers better
Closes 
2023-04-27 07:52:09 +05:30
Simon Sawicki
21b5ec86c2
[utils] traverse_obj: Allow iterables in traversal ()
Authored by: Grub4K
2023-04-24 19:56:35 +02:00
pukkandan
ec9311c41b
[outtmpl] Support str.format syntax inside replacements
Closes 
2023-04-24 18:43:54 +05:30
pukkandan
78fde6e339
[outtmpl] Allow \n in replacements and default.
Fixes: https://github.com/yt-dlp/yt-dlp/issues/6808#issuecomment-1510055357
Fixes: https://github.com/yt-dlp/yt-dlp/issues/6808#issuecomment-1510363645
2023-04-24 18:28:30 +05:30
Simon Sawicki
0898c5c8cc
[utils] js_to_json: Implement template strings ()
Authored by: Grub4K
2023-03-25 19:41:28 +01:00
pukkandan
f68434cc74
[extractor] Extract more metadata from ISM
Fixes 81b6102d20 (r105892531)
2023-03-25 13:18:21 +05:30
pukkandan
9acf1ee25f
[jsinterp] Handle Date at epoch 0
Closes 
2023-03-03 16:55:06 +05:30
pukkandan
65f6e80780
[dependencies] Simplify Cryptodome
Closes , closes , closes 
2023-02-28 23:15:13 +05:30
pukkandan
45b2ee6f4f
Update to ytdl-commit-2dd6c6e
[YouTube] Avoid crash if uploader_id extraction fails
2dd6c6edd8

Except:
    * 295736c9cba714fb5de7d1c3dd31d86e50091cf8 [jsinterp] Improve parsing
    * 384f632e8a9b61e864a26678d85b2b39933b9bae [ITV] Overhaul ITV extractor
    * 33db85c571304bbd6863e3407ad8d08764c9e53b [feat]: Add support to external downloader aria2p
2023-02-17 17:52:23 +05:30
Simon Sawicki
6839ae1f6d
[utils] traverse_obj: Fix more bugs
and cleanup uses of `default=[]`

Continued from b1bde57bef
2023-02-10 19:36:55 +05:30
Simon Sawicki
b1bde57bef
[utils] traverse_obj: Fix several behavioral problems
See  for further info

Authored by: Grub4K
2023-02-08 04:11:08 +01:00
pukkandan
f6a765ceb5
[dependencies] Standardize Cryptodome imports 2023-02-08 07:28:46 +05:30
Simon Sawicki
776995bc10
[utils] traverse_obj: Various improvements
- Add `set` key for transformations/filters
- Add `re.Match` group names
- Fix behavior for `expected_type` with `dict` key
- Raise for filter function signature mismatch in debug

Authored by: Grub4K
2023-02-02 06:40:19 +01:00
pukkandan
8b008d6254
[jsinterp] Support if statements
Closes 
2023-02-01 09:40:16 +05:30
Simon Sawicki
773c272d66
Fix config locations ()
Bug in 8e40b9d1ec
Closes 

Authored by: Grub4k, coletdjnz, pukkandan
2023-01-07 00:31:00 +05:30
Matthew
8e40b9d1ec
Improve plugin architecture ()
to make plugins easier to develop and use:
* Plugins are now loaded as namespace packages.
* Plugins can be loaded in any distribution of yt-dlp (binary, pip, source, etc.).
* Plugin packages can be installed and managed via pip, or dropped into any of the documented locations.
* Users do not need to edit any code files to install plugins.
* Backwards-compatible with previous plugin architecture.

As a side-effect, yt-dlp will now search in a few more locations for config files.

Closes https://github.com/yt-dlp/yt-dlp/issues/1389

Authored by: flashdagger, coletdjnz, pukkandan, Grub4K
Co-authored-by: Marcel <flashdagger@googlemail.com>
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
2023-01-01 04:29:22 +00:00
pukkandan
fbb7383306
Add weba to known extensions 2022-12-30 15:32:47 +05:30
ChillingPepper
d5f043d127
[utils] js_to_json: Fix bug in f55523c ()
Authored by: ChillingPepper, pukkandan
2022-12-30 12:08:38 +05:30
pukkandan
9f14daf22b
[extractor] Deprecate _sort_formats 2022-11-17 11:40:17 +05:30
pukkandan
6368e2e639
[cleanup] Misc
Closes 
2022-11-16 06:57:07 +05:30
Robert Geislinger
7aaf4cd2a8
[cleanup] Misc
Closes , Closes 

Authored by: pukkandan, Alienmaster
2022-11-11 15:48:29 +05:30
pukkandan
495322b95b
[test] Allow extract_flat in download tests
Authored by: coletdjnz, pukkandan
2022-11-10 07:32:35 +05:30
Matthew
86973308cd
[extractor/youtube:tab] Update tab handling for redesign ()
Closes , , 
Authored by: coletdjnz, pukkandan
2022-11-09 14:28:44 +05:30
pukkandan
46d09f8707
[cleanup] Lint and misc cleanup 2022-11-07 02:32:36 +05:30
pukkandan
db4678e448
Update to ytdl-commit-de39d128
[extractor/ceskatelevize] Back-port extractor from yt-dlp
de39d1281c

Closes , Closes , Closes 
2022-11-07 02:18:30 +05:30
Alex Karabanov
c9bd65185c
[extractor/zenyandex] Fix extractors (, )
Closes 
Authored by:  lksj, puc9, pukkandan

Co-authored-by: puc9 <51006296+puc9@users.noreply.github.com>
2022-10-25 15:50:48 +05:30
Ajay Ramachandran
63c547d71c
[SponsorBlock] Support chapter category ()
Authored by: ajayyy, pukkandan
2022-10-18 22:21:57 +05:30
Simon Sawicki
a71b812f53
[utils] js_to_json: Improve escape handling ()
Authored by: Grub4K
2022-10-13 01:52:17 +05:30
Simon Sawicki
36069409ec
[cookies] Improve LenientSimpleCookie ()
Closes  
Authored by: Grub4K
2022-10-11 09:09:12 +05:30
pukkandan
0468a3b325
[jsinterp] Improve separating regex
Fixes https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1273974909
2022-10-11 08:02:26 +05:30
Matthew
4c9a1a3ba5
[extractor/wordpress:mb.miniAudioPlayer] Add embed extractor ()
Closes https://github.com/yt-dlp/yt-dlp/issues/4994

Authored by: coletdjnz
2022-10-09 05:55:26 +00:00
Simon Sawicki
7b0127e1e1
[utils] traverse_obj: Allow re.Match objects ()
Authored by: Grub4K
2022-10-09 07:01:37 +05:30
Simon Sawicki
f99bbfc983
[utils] traverse_obj: Always return list when branching ()
Fixes 
Authored by: Grub4K
2022-10-09 06:57:32 +05:30
nixxo
81b6102d20
[downloader/ism] Support ec-3 codec ()
Closes 
Authored by: nixxo
2022-09-30 23:03:29 +05:30
Simon Sawicki
ab029d7e92
[utils] traverse_obj: Rewrite, document and add tests ()
Authored by: Grub4K
2022-09-26 02:33:19 +05:30
Simon Sawicki
8817a80d3a
[cookies] Parse cookies leniently ()
Closes , 
Authored by: Grub4K
2022-09-16 22:32:00 +05:30
Elyse
7657ec7ed6
[utils] base_url: URL paths can contain & ()
Authored by: elyse0
Closes 
2022-09-04 08:39:45 +05:30
pukkandan
07a1250e0e
[outtmpl] Curly braces to filter keys 2022-09-03 18:28:24 +05:30
pukkandan
1a7c9fad9f
[jsinterp] Workaround operator associativity issue
https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1235384480
2022-09-02 20:45:55 +05:30
pukkandan
d2c8aadf79
[cleanup] Misc
Closes , Closes , Closes 
Authored by: pukkandan, MrRawes, DavidH-2022
2022-09-01 16:49:03 +05:30
pukkandan
1ac7f46184
Update to ytdl-commit-ed5c44e7
[compat] Replace deficient ChainMap class in Py3.3 and earlier
ed5c44e7b7
2022-09-01 16:46:32 +05:30
pukkandan
05deb747bb
[jsinterp] Fix escape in regex 2022-09-01 16:46:32 +05:30
Elyse
f26af78a8a
[jsinterp] Add charcodeAt and bitwise overflow ()
Authored by: elyse0
2022-08-31 21:01:22 +05:30
pukkandan
d81ba7d491
[jsinterp, extractor/youtube] Minor fixes 2022-08-30 18:13:37 +05:30
pukkandan
c4b2df872d
[jsinterp] Fix _separate
Ref: https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1231126941
2022-08-30 16:06:40 +05:30
pukkandan
164b03c486
[jsinterp] Fix bug in operator precedence
Fixes https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1226659543
2022-08-25 09:40:46 +05:30
pukkandan
e5458d1d88
Fix lazy extractor bug in fe7866d0ed
and add test

Fixes https://github.com/yt-dlp/yt-dlp/pull/3234#issuecomment-1225347071
2022-08-24 15:19:58 +05:30
pukkandan
90a1df305b
[test] Fix test_youtube_signature 2022-08-21 00:51:03 +05:30
pukkandan
be13a6e525
[jsinterp] Bring on-par with youtube-dl
Code from: https://github.com/ytdl-org/youtube-dl/pull/31175, https://github.com/ytdl-org/youtube-dl/pull/31182

Authored by pukkandan, dirkf
2022-08-19 05:08:21 +05:30
pukkandan
f6ca640b12
[jsinterp] Fix for youtube player 1f7d5369
Closes  again
2022-08-18 16:38:35 +05:30
pukkandan
6d3e7424bf
[jsinterp] Fix for youtube player c81bbb4a 2022-08-16 06:53:45 +05:30
pukkandan
49b4ceaedf
[jsinterp] Bring or-par with youtube-dl
Partially cherry-picked from: d231b56717

Authored by pukkandan, dirkf
2022-08-15 03:31:49 +05:30
pukkandan
d711839760
Update to ytdl-commit-e6a836d
[core] Make `--max-downloads ...` stop immediately on reaching the limit
e6a836d54c
2022-08-15 03:31:48 +05:30
pukkandan
8f53dc44a0
[jsinterp] Handle new youtube signature functions
Closes 
2022-08-14 05:12:32 +05:30