Jody Bruchon
a9ac178eb1
Use 64 KiB buffered writes for performance and less fragmentation
...
I use yt-dlp on Windows writing to a Linux system via SMB over a
10GbE connection and downloading via 400 Mbps cable internet. I
have observed that downloads often seem to start very fast (40+
MiB/sec) but then throttle down to 8-20 MiB/sec. I also observed
a large amount of disk thrashing for such a large array and small
amount of data that's supposedly being written sequentially.
The problem is two-fold. Downloaded fragments are stored using a
very short-lived *-FragX file, then immediately appended to the
stream upon fragment completion, and deleted. Both operations use
small write buffers. When the OS write buffers start to flush, the
two sets of writes plus the large number of writes start to force
competition to complete the queued writes in different areas of
the volume.
Python defaults to sending writes at the underlying device's
"block size" or a fallback to io.DEFAULT_BUFFER_SIZE. In practical
terms, this means a write buffer of 4096 or 8192 bytes. This
commit increases most write buffers to 65536 (64 KiB) using the
open() buffering=X option, significantly speeding up writes of
larger chunks of data and reducing potential fragmentation in low
disk space conditions. With these changes, I consistently see fast
downloads and the array thrashing is noticeably lessened.
2024-12-19 11:36:00 -05:00
bashonly
d298693b1b
[ie/soundcloud] Various fixes ( #11820 )
...
CodeQL / Analyze (push) Has been cancelled
Core Tests / Core Tests (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
- Fix original/download formats so that they are considered bestaudio
- Raise appropriate error if track is DRM-protected
Authored by: bashonly
2024-12-15 20:16:04 +00:00
bashonly
09a6c68712
[ie/youtube] Add age-gate workaround for some embeddable videos ( #11821 )
...
Closes #11296
Authored by: bashonly
2024-12-15 20:09:48 +00:00
bashonly
1a8851b689
[ie/youtube] Fix uploader_id extraction ( #11818 )
...
Closes #11816
Authored by: bashonly
2024-12-15 20:07:18 +00:00
bashonly
b91c3925c2
[update] Check 64-bitness when upgrading ARM builds ( #11819 )
...
Closes #11813
Authored by: bashonly
2024-12-15 19:55:30 +00:00
bashonly
3d3ee458c1
[update] Fix endless update loop for linux_exe builds ( #11827 )
...
Closes #11808
Authored by: bashonly
2024-12-15 19:47:50 +00:00
github-actions[bot]
2037a6414f
Release 2024.12.13
...
CodeQL / Analyze (push) Has been cancelled
Core Tests / Core Tests (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
Created by: bashonly
:ci skip all
2024-12-13 10:35:40 +00:00
sepro
5421669626
[cleanup] Make more playlist entries lazy ( #11763 )
...
Authored by: seproDev
2024-12-13 10:25:29 +00:00
bashonly
dc3c4fddcc
[ie/youtube] Prioritize original language over auto-dubbed audio ( #11803 )
...
Closes #11753
Authored by: bashonly
2024-12-13 10:21:48 +00:00
bashonly
5460cd9189
[ie/youtube] Fix signature function extraction for 2f1832d2 ( #11801 )
...
Closes #11798
Authored by: bashonly
2024-12-13 09:43:08 +00:00
Crypto90
f6c73aad5f
[ie/youtube:search_url] Fix playlist searches ( #11782 )
...
CodeQL / Analyze (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
Closes #11666
Authored by: Crypto90
2024-12-12 13:54:11 +00:00
Pew
d5e2a379f2
[ie/youtube] Fix release_date extraction ( #11759 )
...
Authored by: MutantPiggieGolem1
2024-12-12 13:46:52 +00:00
bashonly
bc262bcad4
[ie/patreon:campaign] Support /c/ URLs ( #11756 )
...
Closes #11755
Authored by: bashonly
2024-12-12 13:44:19 +00:00
bashonly
f4d3e9e6dc
[ie/soundcloud] Fix extraction ( #11777 )
...
Authored by: bashonly
2024-12-12 13:39:38 +00:00
github-actions[bot]
6fef824025
Release 2024.12.06
...
CodeQL / Analyze (push) Has been cancelled
Core Tests / Core Tests (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
Created by: bashonly
:ci skip all
2024-12-06 16:07:07 +00:00
bashonly
4bd2655398
[ie/youtube] Raise if n function returns input value ( #11752 )
...
Improve a95ee6d880
2024-12-06 15:58:44 +00:00
bashonly
a95ee6d880
[ie/youtube] Fix n sig extraction for player 3bb1f723 ( #11750 )
...
Closes #11744
Authored by: bashonly
2024-12-06 15:35:18 +00:00
bashonly
4c85ccd136
[ie/youtube] Fix signature function extraction ( #11751 )
...
Closes #11748
Authored by: bashonly
2024-12-06 15:34:13 +00:00
bashonly
2feb28028e
[ie/soundcloud] Fix formats extraction ( #11742 )
...
Authored by: bashonly
2024-12-06 15:02:30 +00:00
N/Ame
fca3eb5f8b
[ie/bilibili] Fix HD formats extraction ( #11734 )
...
CodeQL / Analyze (push) Has been cancelled
Core Tests / Core Tests (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
Fixes dc16876480Closes #10554
Authored by: grqz
2024-12-04 23:11:55 +00:00
bashonly
2e49c789d3
[ie/youtube] Player client maintenance ( #11724 )
...
Closes #11686
Authored by: bashonly
2024-12-04 22:33:14 +00:00
wesson09
354cb4026c
[cookies] Add --cookies-from-browser support for MS Store Firefox ( #11731 )
...
Authored by: wesson09
2024-12-04 18:41:58 +01:00
github-actions[bot]
cfa76f35d2
Release 2024.12.03
...
CodeQL / Analyze (push) Has been cancelled
Core Tests / Core Tests (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
Created by: bashonly
:ci skip all
2024-12-03 20:30:33 +00:00
bashonly
2b67ac300a
[cleanup] Misc ( #11716 )
...
Authored by: bashonly, seproDev
Co-authored-by: sepro <sepro@sepr0.com>
2024-12-03 20:22:21 +00:00
bashonly
c038a7b187
[ie/vk] Fix extractors ( #11715 )
...
Closes #5832 , Closes #11471 , Closes #11646 , Closes #11670
Authored by: bashonly
2024-12-03 14:28:43 +00:00
Link
a13a336aa6
[ie/bilibili] Fix subtitles and chapters extraction ( #11708 )
...
Authored by: xiaomac
2024-12-03 04:08:46 +00:00
N/Ame
dc16876480
[ie/bilibili] Always try to extract HD formats ( #10559 )
...
Closes #10554
Authored by: grqz
2024-12-03 03:44:03 +00:00
N/Ame
f05a1cd149
[ie/bilibili] Fix supporter-only video extraction ( #11711 )
...
Fix bug in 239f5f36feCloses #11702
Authored by: grqz, bashonly
Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>
2024-12-03 01:19:22 +00:00
sepro
d8fb349086
[cleanup] Bump ruff to 0.8.x ( #11608 )
...
CodeQL / Analyze (push) Has been cancelled
Core Tests / Core Tests (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
Authored by: seproDev
2024-12-02 16:29:30 +01:00
sepro
2bea793632
[ie/MicrosoftEmbed] Make format extraction non fatal ( #11654 )
...
Authored by: seproDev
2024-12-02 16:22:16 +01:00
Elan Ruusamäe
62cba8a1be
[ie/duoplay] Fix extractor ( #11588 )
...
CodeQL / Analyze (push) Waiting to run
Download Tests / Quick Download Tests (push) Waiting to run
Download Tests / Full Download Tests (push) Waiting to run
Quick Test / Core Test (push) Waiting to run
Quick Test / Code check (push) Waiting to run
Release (master) / release (push) Waiting to run
Release (master) / publish_pypi (push) Blocked by required conditions
Authored by: glensc, bashonly
Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>
2024-12-01 22:33:11 +00:00
N/Ame
239f5f36fe
[ie/bilibili] Fix extractor ( #11667 )
...
Closes #11665
Authored by: grqz
2024-12-01 21:55:18 +00:00
bashonly
0d146c1e36
[ie/youtube] Adjust player clients for site changes ( #11663 )
...
Closes #11640
Authored by: bashonly
2024-12-01 15:25:09 +00:00
DarkZeros
cd0f934604
[ie/mitele] Fix extractor ( #11683 )
...
Closes #11690
Authored by: DarkZeros
2024-12-01 14:21:57 +00:00
N/Ame
360aed810a
[ie/instagram] Support share URLs ( #11677 )
...
Closes #11630
Authored by: grqz
2024-12-01 14:16:50 +00:00
bashonly
00dcde7286
[ie/dropbox] Fix password-protected video extraction ( #11636 )
...
CodeQL / Analyze (push) Has been cancelled
Core Tests / Core Tests (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
Closes #11634
Authored by: bashonly
2024-11-27 01:47:28 +00:00
bashonly
910ecc4229
[ie/tiktok] Deprioritize animated thumbnails ( #11645 )
...
Closes #11641
Authored by: bashonly
2024-11-27 00:45:01 +00:00
bashonly
0a0d80800b
[ie/dacast] Fix HLS AES formats extraction ( #11644 )
...
Closes #11643
Authored by: bashonly
2024-11-26 23:18:48 +00:00
Simon Sawicki
e0500cbf79
[ie] Handle fragmented formats in _remove_duplicate_formats ( #11637 )
...
Authored by: Grub4K
2024-11-27 00:05:07 +01:00
Jakob Kruse
4b5eec0aaa
[ie/chaturbate] Fix support for non-public streams ( #11624 )
...
CodeQL / Analyze (push) Has been cancelled
Core Tests / Core Tests (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
Fix bug in 720b3dc453Closes #11623
Authored by: jkruse
2024-11-24 22:20:30 +00:00
sepro
fe70f20aed
[ie/youtube:tab] Fix playlists tab extraction ( #11615 )
...
Closes #11524
Authored by: seproDev
2024-11-23 22:46:50 +01:00
N/Ame
e0f1ae813b
[ie/facebook] Support more groups URLs ( #11576 )
...
CodeQL / Analyze (push) Has been cancelled
Core Tests / Core Tests (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
Authored by: grqz
2024-11-23 19:47:37 +00:00
sepro
7d6c259a03
Add playlist_webpage_url field ( #11613 )
...
Closes #10827
Authored by: seproDev
2024-11-23 20:42:35 +01:00
gitninja1234
16336c51d0
[ie/stripchat] Fix extractor ( #11596 )
...
Closes #11587
Authored by: gitninja1234
2024-11-23 19:40:45 +00:00
bashonly
ccf0a6b86b
[cleanup] Misc ( #11574 )
...
Authored by: bashonly, pzhlkj6612
Co-authored-by: Mozi <29089388+pzhlkj6612@users.noreply.github.com>
2024-11-23 18:51:51 +00:00
github-actions[bot]
f919729538
Release 2024.11.18
...
CodeQL / Analyze (push) Has been cancelled
Core Tests / Core Tests (push) Has been cancelled
Download Tests / Quick Download Tests (push) Has been cancelled
Download Tests / Full Download Tests (push) Has been cancelled
Quick Test / Core Test (push) Has been cancelled
Quick Test / Code check (push) Has been cancelled
Release (master) / release (push) Has been cancelled
Release (master) / publish_pypi (push) Has been cancelled
Created by: bashonly
:ci skip all
2024-11-18 05:45:05 +00:00
bashonly
7ea2787920
[ie/reddit] Improve error handling ( #11573 )
...
Authored by: bashonly
2024-11-18 05:36:38 +00:00
bashonly
f7257588bd
[ie/digitalconcerthall] Support login with access/refresh tokens ( #11571 )
...
Removes broken support for login with email and password
Removes obsolete `prefer_combined_hls` extractor-arg
Closes #11404 , Closes #11436
Authored by: bashonly
2024-11-18 05:16:17 +00:00
bashonly
da252d9d32
[cleanup] Misc ( #11554 )
...
Closes #6884
Authored by: bashonly, Grub4K, seproDev
Co-authored-by: Simon Sawicki <contact@grub4k.xyz>
Co-authored-by: sepro <sepro@sepr0.com>
2024-11-17 23:25:05 +00:00
gillux
e079ffbda6
[ie/litv] Fix extractor ( #11071 )
...
Authored by: jiru
2024-11-17 21:37:15 +00:00
