f3c8857421
* Add zstd dep * Implement backend backup with zstd * Implement backup thinning * Write backup meta * Use new file ending anki21b * Asynchronously backup on collection close in Rust * Revert "Add zstd dep" This reverts commit 3fcb2141d2be15f907269d13275c41971431385c. * Add zstd again * Take backup col path from col struct * Fix formatting * Implement backup restoring on backend * Normalize restored media file names * Refactor `extract_legacy_data()` A bit cumbersome due to borrowing rules. * Refactor * Make thinning calendar-based and gradual * Consider last kept backups of previous stages * Import full apkgs and colpkgs with backend * Expose new backup settings * Test `BackupThinner` and make it deterministic * Mark backup_path when closing optional * Delete leaky timer * Add progress updates for restoring media * Write restored collection to tempfile first * Do collection compression in the background thread This has us currently storing an uncompressed and compressed copy of the collection in memory (not ideal), but means the collection can be closed without waiting for compression to complete. On a large collection, this takes a close and reopen from about 0.55s to about 0.07s. The old backup code for comparison: about 0.35s for compression off, about 8.5s for zip compression. * Use multithreading in zstd compression On my system, this reduces the compression time of a large collection from about 0.55s to 0.08s. * Stream compressed collection data into zip file * Tweak backup explanation + Fix incorrect tab order for ignore accents option * Decouple restoring backup and full import In the first case, no profile is opened, unless the new collection succeeds to load. In the second case, either the old collection is reloaded or the new one is loaded. * Fix number gap in Progress message * Don't revert backup when media fails but report it * Tweak error flow * Remove native BackupLimits enum * Fix type annotation * Add thinning test for whole year * Satisfy linter * Await async backup to finish * Move restart disclaimer out of backup tab Should be visible regardless of the current tab. * Write restored collection in chunks * Refactor * Write media in chunks and refactor * Log error if removing file fails * join_backup_task -> await_backup_completion * Refactor backup.rs * Refactor backup meta and collection extraction * Fix wrong error being returned * Call sync_all() on new collection * Add ImportError * Store logger in Backend, instead of creating one on demand init_backend() accepts a Logger rather than a log file, to allow other callers to customize the logger if they wish. In the future we may want to explore using the tracing crate as an alternative; it's a bit more ergonomic, as a logger doesn't need to be passed around, and it plays more nicely with async code. * Sync file contents prior to rename; sync folder after rename. * Limit backup creation to once per 30 min * Use zstd::stream::copy_decode * Make importing abortable * Don't revert if backup media is aborted * Set throttle implicitly * Change force flag to minimum_backup_interval * Don't attempt to open folders on Windows * Join last backup thread before starting new one Also refactor. * Disable auto sync and backup when restoring again * Force backup on full download * Include the reason why a media file import failed, and the file path - Introduce a FileIoError that contains a string representation of the underlying I/O error, and an associated path. There are a few places in the code where we're currently manually including the filename in a custom error message, and this is a step towards a more consistent approach (but we may be better served with a more general approach in the future similar to Anyhow's .context()) - Move the error message into importing.ftl, as it's a bit neater when error messages live in the same file as the rest of the messages associated with some functionality. * Fix importing of media files * Minor wording tweaks * Save an allocation I18n strings with replacements are already strings, so we can skip the extra allocation. Not that it matters here at all. * Terminate import if file missing from archive If a third-party tool is creating invalid archives, the user should know about it. This should be rare, so I did not attempt to make it translatable. * Skip multithreaded compression on small collections Co-authored-by: Damien Elmes <gpg@ankiweb.net> |
||
|---|---|---|
| .. | ||
| remote | ||
| BUILD.bazel | ||
| BUILD.reqwest.native.bazel | ||
| BUILD.reqwest.rustls.bazel | ||
| BUILD.term-0.7.0.bazel | ||
| crates.bzl | ||
| licenses.json | ||
| README.md | ||
| update.py | ||
This folder integrates Rust crates.io fetching into Bazel.
To add or update dependencies, ensure a local Rust environment is available
(eg source tools/cargo-env), then install cargo-raze:
cargo install cargo-raze --version 0.14.1
cargo install cargo-license
After adding/updating dependencies in ../rslib/Cargo.toml, change to this folder and run:
$ python update.py
or
$ REPIN=1 python update.py
The former will apply added crates and adjusted version numbers, while leaving most crate versions alone. The latter will also update pinned dependencies to their latest compatible versions.
Note: cargo-raze does not currently work when run from Windows, and nobody has investigated why yet. For now, you'll need a Mac or Linux machine, or will need to run update.py from within WSL.
A couple of crates need extra work to build with Bazel, and are listed in ../Cargo.toml. For example:
[package.metadata.raze.crates.pyo3.'*']
compile_data_attr = "glob([\"**\"])"
With minor version updates, you should not normally need to modify the entries in that file.
Because update.py modifies a lot of files in remote/, it makes it difficult to review in a PR, and the changes can sometimes break platforms like Windows. For this reason, please don't submit PRs that do minor version bumps - those will typically be done after stable releases. If you need a new crate for a feature you're working on, please raise it in an issue first.
Reqwest
Things are complicated with reqwest at the moment, because:
- we're using a fork to implement better timeouts for syncing
- we want to build it with different features on Linux (where we can't build a wheel that links to OpenSSL), and on other platforms.
For minor version bumps, update.py should take care of updating the versions of reqwest dependencies.
After making a big update to reqwest via an updated fork, the vendored BUILD.reqwest.* files may need updating. To do that, comment native-tls from the features in rslib/Cargo.toml and run update.py, and copy the file in remote/ over the old vendored file. Then comment the other two deps out, add native-tls back, and repeat the process.