Currently this only applies to grain synth generation,
but the prior implementation of HDR detection only looked at the props
on the input video. Those may not necessarily be set correctly even if
the user is encoding to HDR, so we want to prioritize looking at the
encoder params, then fallback to checking the input if the user has not
set a transfer characteristic on the encoder.
* Add zones support
Currently implemented:
- Overriding the selected encoder & number of passes
- Overriding or adding video params
- Overriding photon noise setting
- Overriding min/max scene length
Closes#267
* Error if zoned encoder does not support output pixel format
* Fix crash if zones change number of passes
* Set passes to 1 for zones with rt mode
People have been requesting this because the default chunk order tends
to skew toward underestimating the final filesize. Longest first is kept
as the default because it provides the best performance.
This commit mainly changes the behavior in 2 ways:
1. We now get the frame count and do scene detection in parallel. Initially, only the number of processed frames are shown, and the percentage stays at 100%, and the total frame count is 0. When the actual frame count is available, the progress bar is updated to reflect that. This can help I/O bound situations where the source is very large and the frame count takes a long time to calculate.
2. We also generate the vapoursynth cache in parallel with the scene detection and calculating the frame count. This eliminates the pause between scene detection and actual encoding (if vspipe finishes generating the cache in time, which will be the case 99% of the time, and it will still reduce the overhead in the very unlikely case that it hasn't finished yet).
Also fixes bug where vapoursynth input wouldn't work if you specified anything other than lsmash or ffms2 for the chunk method.
* Add kbps and final file size estimates to progress bar
This looks at the size of each chunk when it completes and saves that
size in kilobytes into the done.json file. For backwards compatibility
for encodes resumed from an earlier version of av1an, if the done.json
does not contain the chunk size, then on resume, av1an will lookup the
size of that chunk. The estimates on the progress bar also handle
resumes and chunk crashes gracefully.
* Include audio size in estimates
I kind of wish that we wouldn't use clippy::pedantic and
clippy::nursery, as the number of `allow` lints at the top of src/lib.rs
is becoming quite absurd. However, I at least wanted to clear out the
large set of warnings being issued, regardless of that discussion.
The SSE4.1/SSSE3 implementation of the parsing routines for
aomenc/vpxenc outperform the old regex by a factor of about 5000x
(3.5 μs vs. 0.7 ns to parse a single line on a Zen 1 1950x).
The idea is to eventually remove regex as a dependency entirely,
especially since there is an open PR to indicatif that speeds up the
template expansion (which replaces regexes with hand-written parsing).
* Improve logging (especially with `--verbose`)
Much more information is now logged when using `--verbose` (all INFO
messages), and the output is also much cleaner.
The issue of logs being hidden while the progress bar is active
because of conflicting ANSI escape codes has been completely fixed, and
no more undesirable hacks are required to correctly display the
set_thread_affinity warning message correctly, for example. This has
been achieved by using indicatif's `println` function instead of just
printing to stderr directly.
flexi_logger has been moved to av1an-cli as it should be, since a
logging framework that actually prints things and handles log level
filtering is not the responsibility of the core library. However, the
actual progress bar code with indicatif still lives in av1an-core for
now.
* Make `--quiet` conflict with `--verbose`
* Fix resuming and improve error handling slightly
The main change that this commit introduces is an `EncoderCrash` type
for errors, which also reports the stderr of the process piping to
the encoder so that it is easier to tell what the actual error is.
Target quality probe crashes are now properly logged with the
`EncoderCrash` type, replacing the old panicking code and making the
error significantly easier to read.
An "optimization" has been implemented which does not wait for the child
process piping to the encoder to exit, and instead collects its stderr
output asynchronously and immediately exits upon an encoder crash. This
does not yet apply to target quality probes, as there is not yet a
single interface for running an encoder (it is done manually for now).
This will change in the future, but this patch is getting too big for
everything so for now we just capture stderr synchronously for target
quality probes.
- Also update `get_percentile` to use an O(N) algorithm from the
standard library instead of O(N*log N).
* Refactor into `Input` type for handling vapoursynth input
Fixes several issues, namely `--vmaf` not working on vapoursynth input
as well as unifying the VMAF interface, which also results in
`--vmaf-threads` being passed correctly to `--vmaf`, which is massively
faster now as a result. If `--vmaf-threads` is not specified, then
the number of detected CPU threads is used for `--vmaf`.
Additionally, VMAF plots are rendered as SVGs now, which also removes
some dependencies since we don't use the bitmap backend at all anymore.
* Small cleanup: more consistent usage of `as_path`
This leaves the default at medium, which is an improved version
of the current scenecut detection method.
The fast method has also been improved from its previous version.
The slow method is a completely new implementation, which has
been shown to be more accurate, but is also about 15% slower.
This also removes the av-scenechange-fast split method,
in favor of making a separate CLI option for --sc-method,
which can be set to "fast", "medium", or "slow".
This also adds a CLI option for scenecut downscaling,
which can be specified by providing a maximum height e.g.
--sc-downscale-height 720 to downscale any input which is
above 720p to 720p for scenecut detection.
This will avoid upscaling anything below 720p, because that
would just slow down scenecut detection for no benefit.
- This removes the `parking_lot::Mutex` wrapping the multi progress bar, as it was not actually necessary. This also removes `parking_lot` as a dependency, as it is unused now.
- Rework the implementation of `mkvmerge` concatenation to not perform any unnecessary UTF-8 validation or allocations. This requires an algorithm to specify chunks with a '+' in between, but without a trailing '+', without later removing arguments (as the API of `std::process:Command` does not support this). This does not change the behavior of `-c mkvmerge` in any way, however.
- Also fix some miscellaneous clippy warnings.
In some cases concatenation silently fails. To prevent loss of the
encoded data we now check if the output file exists. If it does not
exist we do not delete the temp folder.
Switch back to an old version of sysinfo, as 0.20 requires 1.54 to compile it, which is too new for MSYS2. Many minor simplifications.
We also switch to structopt for now instead of the beta release of clap, as clap officially recommends to not use the beta yet, and it caused some problems when compiling on Windows or with an older compiler.
Some minor changes:
- The `decow_strings` function has been removed, and we deal with `Cow<str>` properly now.
- The `remove_patterns` function now takes a mutable reference to a `Vec<String>` instead of copying one and modifing it to reduce complexity and overhead.
- The structs used to serialize the VMAF json result have been renamed to `VmafResult`, `Metrics`, and `VmafScore` (they were previously `Foo`, `Bar`, and `Baz`)
- Using an enum instead of a string as the argument to `log_probes` for better type safety and less overhead
- `frame_check_output` now warns if there is a mismatch, and the message has been updated to be more clear that a frame mismatch has occurred
- Several other small changes
Selectable with `--split-method av-scenechange-fast`.
The fast method enables downscaling the video before scene detection,
and uses av-scenechange's fast mode, which is similar to pyscenedetect.
This also removes the downscaling from the normal scenecut mode,
since downscaling does affect accuracy of the scene detection.
FFmpeg requires a very specific syntax to escape the file path in
Windows when using a filter, so this has been implemented with
conditional compilation. The behavior is the same on Windows, except
that the path is converted to an absolute path. Also, the only usage of
`std::env::current_dir` has been removed, since it was being used as an
ad-hoc way to convert a relative path to an absolute one, which doesn't
work if the temporary directory is in a different directory than the
current one.
We no longer kill the child processes from `process_pipe`,
since calling `wait` or `wait_with_output` closes the stdin handle
anyway.
Also, all usages of `cfg!(windows)` have been changed to `cfg!(target_os
= "windows")` to keep a consistent style.
`DoneJson` is now a concurrent data structure, via `DashMap` and atomics
in the standard library. This should help avoid any potential race
conditions involved with one thread reading `done.json` while another
thread writes to it. Now, we only write to `done.json`, so any failure
from parsing the json while encoding is impossible.
The audio encoding is now done on a separate thread, and it is not
assumed that the audio encoding was finished if the `--resume` flag was
specified, which is now kept track of within `done.json`.
The initialization of the progress bar(s) now does not occur if the
corresponding initialization functions were not called, and the
functions to update the progress bar(s) are a NO-OP if it was not
initialized, rather than implicitly initializing it through the use of
the `Lazy` type from `once_cell`.
Additionally, all 3 concatenation functions now reside in the `concat` module.
Implements more robust logging via flexi_logger. This replaces the old
logging implementation, and is also more ergonomic, since macros from the
`log` crate are now used instead of `format!` directly. Now, warnings
and errors are also logged to stderr, in addition to the log file.
This also removes av1an-pyo3, and integrates the code into av1an-core.
Concatenation with FFmpeg is now handled on Windows differently than
other platforms through conditional compilation. FFmpeg seems to need
double backslashes in the concat file, so this is explicitly handled
in the Windows build now. Also, the spinner is now disabled on Windows
builds since the default command prompt cannot display the characters
correctly.
Previously, we represented quantizer values as an owned `String` for
constructing various encoder commands, which is much slower and less
robust than representing the quantizer as an integral value. This has
been updated to represent the `q` and `n_threads` both as a `usize`, and
only convert to a `String` for formatting as needed.
Furthermore, several struct fields containing paths have been updated to
use a `PathBuf` or `&Path` instead of a `String`. These are the first
steps required to supporting non-UTF8 filenames on Windows and
Unix-based operating systems.
`run_vmaf_on_chunk`'s function signature has been update to take generic
arguments that satisfy `AsRef<Path>`, which allows the caller to save
themselves from creating a new `Path` if the original argument was a
`String`, for example.
* Do not represent the concatenation method as a string
* Fix pedantic clippy warnings
* Remove Cargo.lock from .gitignore
* Fix startup_check for ivf concatenation
* Add doc comment to `Args` for better help generation in Clap
* Use Display impl instead of directly using str::From for `Encoder` and `ConcatMethod`
* Do not unwrap the result of killing child processes
* Assert that VMAF calculation was successful
* Use more idiomatic Rust
This is a hard error in clippy,
so currently if compiled with clippy enabled,
compilation will fail. This commit allows compilation
with clippy enabled.
DataHoarder
