Commit Graph

138 Commits

Author SHA1 Message Date
RumovZ
f3c8857421
Backups (#1685)
* Add zstd dep

* Implement backend backup with zstd

* Implement backup thinning

* Write backup meta

* Use new file ending anki21b

* Asynchronously backup on collection close in Rust

* Revert "Add zstd dep"

This reverts commit 3fcb2141d2be15f907269d13275c41971431385c.

* Add zstd again

* Take backup col path from col struct

* Fix formatting

* Implement backup restoring on backend

* Normalize restored media file names

* Refactor `extract_legacy_data()`

A bit cumbersome due to borrowing rules.

* Refactor

* Make thinning calendar-based and gradual

* Consider last kept backups of previous stages

* Import full apkgs and colpkgs with backend

* Expose new backup settings

* Test `BackupThinner` and make it deterministic

* Mark backup_path when closing optional

* Delete leaky timer

* Add progress updates for restoring media

* Write restored collection to tempfile first

* Do collection compression in the background thread

This has us currently storing an uncompressed and compressed copy of
the collection in memory (not ideal), but means the collection can be
closed without waiting for compression to complete. On a large collection,
this takes a close and reopen from about 0.55s to about 0.07s. The old
backup code for comparison: about 0.35s for compression off, about
8.5s for zip compression.

* Use multithreading in zstd compression

On my system, this reduces the compression time of a large collection
from about 0.55s to 0.08s.

* Stream compressed collection data into zip file

* Tweak backup explanation

+ Fix incorrect tab order for ignore accents option

* Decouple restoring backup and full import

In the first case, no profile is opened, unless the new collection
succeeds to load.
In the second case, either the old collection is reloaded or the new one
is loaded.

* Fix number gap in Progress message

* Don't revert backup when media fails but report it

* Tweak error flow

* Remove native BackupLimits enum

* Fix type annotation

* Add thinning test for whole year

* Satisfy linter

* Await async backup to finish

* Move restart disclaimer out of backup tab

Should be visible regardless of the current tab.

* Write restored collection in chunks

* Refactor

* Write media in chunks and refactor

* Log error if removing file fails

* join_backup_task -> await_backup_completion

* Refactor backup.rs

* Refactor backup meta and collection extraction

* Fix wrong error being returned

* Call sync_all() on new collection

* Add ImportError

* Store logger in Backend, instead of creating one on demand

init_backend() accepts a Logger rather than a log file, to allow other
callers to customize the logger if they wish.

In the future we may want to explore using the tracing crate as an
alternative; it's a bit more ergonomic, as a logger doesn't need to be
passed around, and it plays more nicely with async code.

* Sync file contents prior to rename; sync folder after rename.

* Limit backup creation to once per 30 min

* Use zstd::stream::copy_decode

* Make importing abortable

* Don't revert if backup media is aborted

* Set throttle implicitly

* Change force flag to minimum_backup_interval

* Don't attempt to open folders on Windows

* Join last backup thread before starting new one

Also refactor.

* Disable auto sync and backup when restoring again

* Force backup on full download

* Include the reason why a media file import failed, and the file path

- Introduce a FileIoError that contains a string representation of
the underlying I/O error, and an associated path. There are a few
places in the code where we're currently manually including the filename
in a custom error message, and this is a step towards a more consistent
approach (but we may be better served with a more general approach in
the future similar to Anyhow's .context())
- Move the error message into importing.ftl, as it's a bit neater
when error messages live in the same file as the rest of the messages
associated with some functionality.

* Fix importing of media files

* Minor wording tweaks

* Save an allocation

I18n strings with replacements are already strings, so we can skip the
extra allocation. Not that it matters here at all.

* Terminate import if file missing from archive

If a third-party tool is creating invalid archives, the user should know
about it. This should be rare, so I did not attempt to make it
translatable.

* Skip multithreaded compression on small collections

Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-03-07 15:11:31 +10:00
Damien Elmes
2c80bcb571 Update to latest rules_rust, which unblocks the tokio upgrade
Rust version has been pinned, as 1.59 gets stuck compiling pyo3_macros,
and appears to have other issues:
https://github.com/rust-lang/rust/issues/94390
2022-03-03 19:48:17 +10:00
Damien Elmes
89bce6d829 Update Rust deps; pin Tokio
Tokio has had to be pinned, because the 1.17 release introduces
a dependency on windows_sys, which fails to build on Windows on
Bazel.

The issue appears to be the build script of a subcrate - it is using
CARGO_MANIFEST_DIR to update the linking path so windows.lib can be
found (it's contained in that crate), but the path is set incorrectly.

dfc25285a2/crates/targets/x86_64_msvc/build.rs

One way we might be able to work around it is to add to the link path
in our own build script.
2022-02-25 17:10:47 +10:00
Damien Elmes
95dbf30fb9 updates to the build process and binary bundles
All platforms:

- rename scripts/ to tools/: Bazelisk expects to find its wrapper script
(used by the Mac changes below) in tools/. Rather than have a separate
scripts/ and tools/, it's simpler to just move everything into tools/.
- wheel outputs and binary bundles now go into .bazel/out/dist. While
not technically Bazel build products, doing it this way ensures they get
cleaned up when 'bazel clean' is run, and it keeps them out of the source
folder.
- update to the latest Bazel

Windows changes:

- bazel.bat has been removed, and tools\setup-env.bat has been added.
Other scripts like .\run.bat will automatically call it to set up the
environment.
- because Bazel is now on the path, you can 'bazel test ...' from any
folder, instead of having to do \anki\bazel.
- the bat files can handle being called from any working directory,
so things like running "\anki\tools\python" from c:\ will work.
- build installer as part of bundling process

Mac changes:

- `arch -arch x86_64 bazel ...` will now automatically use a different
build root, so that it is cheap to switch back and forth between archs
on a new Mac.
- tools/run-qt* will now automatically use Rosetta
- disable jemalloc in Mac x86 build for now, as it won't build under
Rosetta (perhaps due to its build scripts using $host_cpu instead of
$target_cpu)
- create app bundle as part of bundling process

Linux changes:

- remove arm64 orjson workaround in Linux bundle, as without a
readily-available, relatively distro-agonstic PyQt/Qt build
we can use, the arm64 Linux bundle is of very limited usefulness.
- update Docker files for release build
- include fcitx5 in both the qt5 and qt6 bundles
- create tarballs as part of the bundling process
2022-02-10 19:23:07 +10:00
RumovZ
d55f080733
V3 parent limits (#1638)
* avoid repinning Rust deps by default

* add id_tree dependency

* Respect intermediate child limits in v3

* Test new behaviour of v3 counts

* Rework v3 queue building to respect parent limits

* Add missing did field to SQL query

* Fix `LimitTreeMap::is_exhausted()`

* Rework tree building logic

https://github.com/ankitects/anki/pull/1638#discussion_r798328734

* Add timer for build_queues()

* `is_exhausted()` -> `limit_reached()`

* Move context and limits into `QueueBuilder`

This allows for moving more logic into QueueBuilder, so less passing
around of arguments. Unfortunately, some tests will require additional
work to set up.

* Fix stop condition in new_cards_by_position

* Fix order gather order of new cards by deck

* Add scheduler/queue/builder/burying.rs

* Fix bad tree due to unsorted child decks

* Fix comment

* Fix `cap_new_to_review_rec()`

* Add test for new card gathering

* Always sort `child_decks()`

* Fix deck removal in `cap_new_to_review_rec()`

* Fix sibling ordering in new card gathering

* Remove limits for deck total count with children

* Add random gather order

* Remove bad sibling order handling

All routines ensure ascending order now.
Also do some other minor refactoring.

* Remove queue truncating

All routines stop now as soon as the root limit is reached.

* Move deck fetching into `QueueBuilder::new()`

* Rework new card gather and sort options

https://github.com/ankitects/anki/pull/1638#issuecomment-1032173013

* Disable new sort order choices ...

depending on set gather order.

* Use enum instead of numbers

* Ensure valid sort order setting

* Update new gather and sort order tooltips

* Warn about random insertion order with v3

* Revert "Add timer for build_queues()"

This reverts commit c9f5fc6ebe87953c17a0c842990b009b5596c69c.

* Update rslib/src/storage/card/mod.rs (dae)

* minor wording tweaks to the tooltips (dae)

+ move legacy strings to bottom
+ consistent capitalization (our leech action still needs fixing,
but that will require introducing a new 'suspend card' string as the
existing one is used elsewhere as well)
2022-02-10 09:55:43 +10:00
Damien Elmes
bbc191711d update Rust deps 2022-01-15 14:59:43 +10:00
Damien Elmes
ec0ee4d162 rebuild Rust build files with cargo-raze 0.14.1
Tags are now sorted; updating this before a crate update.
2022-01-15 14:33:53 +10:00
Damien Elmes
4d431fb7af move linkchecker into separate crate
The feature-based approach didn't work with cargo-raze, leading
to ./update.py in cargo/ breaking.
2021-12-20 17:27:43 +10:00
Damien Elmes
c2ab0de34f update Rust deps 2021-12-03 20:35:53 +10:00
Damien Elmes
67ee6f9c0e update to Rust 1.57 + latest rules_rust 2021-12-03 20:35:52 +10:00
Damien Elmes
11042a16ce Fix latest Rusqlite on Windows
https://github.com/bazelbuild/rules_rust/pull/1025
2021-11-19 10:44:02 +10:00
Damien Elmes
8b3d964866 roll rusqlite back due to Windows breakage 2021-11-18 21:39:23 +10:00
Damien Elmes
d31dccc388 update Rust deps 2021-11-18 20:58:41 +10:00
RumovZ
0efa3f944f
Garbage collect unused Fluent strings (#1482)
* Canonify import of i18n module

Should always be imported as `tr`, or `tr2` if there is a name collision
(Svelte).

* Add helper for garbage collecting ftl strings

Also add a serializer for ftl asts.

* Add helper for filter-mapping `DirEntry`s

* Fix `i18n_helpers/BUILD.bazel`

* run cargo-raze

* Refactor `garbage_collection.rs`

- Improve helper for file iterating
- Remove unused terms as well
- Fix issue with checking for nested messages by switching to a regex-
based approach (which runs before deleting)
- Some more refactorings and lint fixes

* Fix lints in `serialize.rs`

* Write json pretty and sorted

* Update `serialize.rs` and fix header

* Fix doc and remove `dbg!`

* Add binaries for ftl garbage collection

Also relax type constraints and strip debug tests.

* add rust_binary targets for i18n helpers (dae)

* add scripts to update desktop usage/garbage collect (dae)

Since we've already diverged from 2.1.49, we won't gain anything
from generating a stable json just yet. But once 2.1.50 is released,
we should run 'ftl/update-desktop-usage.sh stable'.

* add keys from AnkiMobile (dae)

* Mention caveats in `remove-unused.sh`
2021-11-12 18:19:01 +10:00
Abdo
896d11546b
Fix invalid path in cargo-env.bat (#1477)
* Fix invalid path in cargo-env.bat

* Assume cargo-env.bat is run from repo root; fix package script (dae)

* mention cargo-raze doesn't work on Windows (dae)
2021-11-05 11:08:23 +10:00
Damien Elmes
e30411e0b2 update Rust deps with cargo-raze 0.13.0 2021-10-26 08:16:40 +10:00
Damien Elmes
7797f88553 add aarch64-apple to Rust targets 2021-10-16 18:07:39 +10:00
RumovZ
3672b0fe73
Switch CardInfoDialog to ts page (#1414)
* Only collect card stats on the backend ...

... instead of rendering an HTML string using askama.

* Add ts page Card Info

* Update test for new `col.card_stats()`

* Remove obsolete CardStats code

* Use new ts page in `CardInfoDialog`

* Align start and end instead of left and right

Curiously, `text-align: start` does not work for `th` tags if assigned
via classes.

* Adopt ts refactorings after rebase

#1405 and #1409

* Clean up `ts/card-info/BUILD.bazel`

* Port card info logic from Rust to TS

* Move repeated field to the top

https://github.com/ankitects/anki/pull/1414#discussion_r725402730

* Convert pseudo classes to interfaces

* CardInfoPage -> CardInfo

* Make revlog in card info optional

* Add legacy support for old card stats

* Check for undefined instead of falsy

* Make Revlog separate component

* drop askama dependency (dae)

* Fix nightmode for legacy card stats
2021-10-14 19:22:47 +10:00
Damien Elmes
944b064e54 update Rust deps 2021-10-02 20:42:03 +10:00
Damien Elmes
faec7f8e3d update cargo-raze instructions 2021-10-02 16:24:44 +10:00
Damien Elmes
6b5ee469bb attempt to fix Linux build 2021-07-23 21:00:51 +10:00
Damien Elmes
379694915e add linkcheck to Bazel 2021-07-23 20:22:32 +10:00
Damien Elmes
c944dd048e strip invalid Unicode chars in media check 2021-07-17 18:30:19 +10:00
Damien Elmes
bf507cca98 move from Python's URI escaping to IRI escaping in Rust
Should make non-Latin text readable in the HTML editor, without the
breakages reverted in the previous change.
2021-07-16 10:38:00 +10:00
Damien Elmes
576b141e2b add back in missing check for upload size
Compression now happens up-front, so we can tell in advance if the
upload size has been exceeded.
2021-06-29 10:50:03 +10:00
Damien Elmes
ca0374782e update to latest rusqlite 2021-06-25 16:22:21 +10:00
Damien Elmes
59e17950ad update most rust deps; skip rusqlite 2021-06-25 15:35:25 +10:00
Damien Elmes
a58826941f add rustls-native-certs
closes #1241
2021-06-18 18:15:41 +10:00
Damien Elmes
e6a7b8fba1 update Rust deps 2021-06-18 18:12:06 +10:00
Damien Elmes
ecdd668f9a update Rust deps 2021-06-16 16:10:57 +10:00
Damien Elmes
dfba66faa9 update Rust deps 2021-05-07 18:22:27 +10:00
Damien Elmes
92e4de16cf report pinned package issues upstream 2021-04-27 23:11:22 +10:00
Damien Elmes
cf2613e869 update Rust deps
- tokio 1.0
- updated reqwest, thanks to Rumo
- other minor dep updates

the reqwest build file has been split into two, as it was awkward
to manually update the combined file, and the platform gate is now
on the target in rslib/
2021-04-27 22:18:12 +10:00
Damien Elmes
c49d6ce49f run black/isort on Python scripts 2021-04-14 18:22:02 +10:00
Damien Elmes
948fc5f777 add missing copyright headers to *.py 2021-04-13 18:45:35 +10:00
Damien Elmes
1704f7fe80 drop dependency on failure crate 2021-04-01 17:21:13 +10:00
Damien Elmes
89d249b3b6 update to the latest rules_rust + security framework update 2021-03-27 19:28:19 +10:00
Damien Elmes
c9c5ecdd31 update cargo/README.md
closes #1096
2021-03-27 14:56:51 +10:00
Damien Elmes
a96ac98605 update Rust deps 2021-03-27 14:47:16 +10:00
Damien Elmes
e4002d7a5e update to latest Fluent 2021-03-27 13:24:11 +10:00
Damien Elmes
9aece2a7b8 rework translation handling
Instead of generating a fluent.proto file with a giant enum, create
a .json file representing the translations that downstream consumers
can use for code generation.

This enables the generation of a separate method for each translation,
with a docstring that shows the actual text, and any required arguments
listed in the function signature.

The codebase is still using the old enum for now; updating it will need
to come in future commits, and the old enum will need to be kept
around, as add-ons are referencing it.

Other changes:

- move translation code into a separate crate
- store the translations on a per-file/module basis, which will allow
us to avoid sending 1000+ strings on each JS page load in the future
- drop the undocumented support for external .ftl files, that we weren't
using
- duplicate strings in translation files are now checked for at build
time
- fix i18n test failing when run outside Bazel
- drop slog dependency in i18n module
2021-03-26 09:41:32 +10:00
Damien Elmes
19c217d8a4 add note about bumping Rust deps 2021-03-15 13:29:36 +10:00
Arthur Milchior
80db5dc7d5 update is executable
Without this change, I can't just do `./update.py` and need to do `python3 update.py`
2021-03-13 07:35:36 +01:00
Damien Elmes
77accb458f roll back slog-term due to Windows breakage 2021-03-10 12:22:11 +10:00
Damien Elmes
265e91c13f add strum crate 2021-03-10 11:53:27 +10:00
Damien Elmes
0207f6c0ab update Rust deps 2021-03-10 11:53:27 +10:00
Damien Elmes
ded626f0b9 render deck description with markdown; strip images
To support images on that screen, we'll first need to adjust the base url
for each platform, or rewrite the local image URLs, as otherwise they
are resolved to _anki/pages/...
2021-02-06 15:02:40 +10:00
Damien Elmes
6ba321f818 sanitize deck description in decks screen on backend
Looks like ammonia only adds about 800k to the release binary
2021-02-06 13:42:38 +10:00
Damien Elmes
da407a5000 pin proc_macro_nested due to Windows breakage 2021-02-03 21:17:11 +10:00
Damien Elmes
790562cd08 update Rust deps 2021-02-03 20:29:48 +10:00
Damien Elmes
9225c804db update Rust deps; add async_trait 2021-01-11 19:19:45 +10:00
Damien Elmes
eca27b3710 add aarch64-linux to BUILD.request.bazel 2020-12-30 20:39:27 +10:00
Damien Elmes
3f3f4b5c36 add aarch64 Linux to cargo; update deps 2020-12-30 13:33:16 +10:00
Damien Elmes
40c5f8a907 update prost to new 0.7 release+auto stage changes
Allows us to drop slow git repo.
2020-12-28 10:12:35 +10:00
Damien Elmes
c4382e38a1 update to stable pyo3 0.13 release 2020-12-23 10:08:38 +10:00
Damien Elmes
72baccd423 update Rust deps 2020-12-23 10:06:35 +10:00
Damien Elmes
91a9307c39 update to cargo-raze 0.8.0 release 2020-12-18 11:56:56 +10:00
Damien Elmes
71f742aa56 update Rust deps 2020-12-18 11:56:28 +10:00
Damien Elmes
d49416649c update to latest cargo-raze 2020-12-15 20:28:10 +10:00
Damien Elmes
75cbbea048
Merge pull request #848 from cecini/rslibdep
Update Rust deps, and nom to v6
2020-12-15 20:04:02 +10:00
cecini
d37ea0f5f7 Update rslib dep nom 2020-12-15 08:19:52 +00:00
cecini
aa715eee93 fix update.py for compatible 2020-12-15 08:09:11 +00:00
Damien Elmes
85b7e1c623 drop unused i686 references
https://forums.ankiweb.net/t/changing-ankis-build-system-to-bazel/4737/9
2020-12-09 15:45:01 +10:00
Damien Elmes
93c1289e69 bump rust deps 2020-12-09 15:34:43 +10:00
Damien Elmes
78125aa974 use compile_data with cargo raze
requires https://github.com/ankitects/cargo-raze/releases/tag/anki-2020-12-01
2020-12-01 16:48:45 +10:00
Damien Elmes
9a1ddec553 make pyo/ring data arguments more specific 2020-12-01 16:48:45 +10:00
Damien Elmes
4c3577b12e update Rust deps incl. Pyo3
This includes the refcount bug fix from pyo3 0.12.4
2020-12-01 16:48:45 +10:00
Damien Elmes
c9b3ed1eae switch to workspace for Rust code 2020-11-24 18:41:03 +10:00
Damien Elmes
4abf244101 update rust deps 2020-11-24 16:57:37 +10:00
Damien Elmes
e9fdb5600c use ring on Linux; native-tls on other platforms
Python wheels on Linux require statically linked SSL libraries.
We were previously relying on the native-tls-vendored feature in
reqwest, but that does not work with Bazel, as openssl-src makes
assumptions that break when sandboxed. The static libs distributed
by distros like Ubuntu fail to link, and while we could potentially
build OpenSSL ourselves, we'd then need to keep it up to
date.

On Windows and Mac however, native-tls is preferable to ring, as it
allows us to get free updates from the OS, and results in
a smaller library.

Rust currently only supports platform-specific features in nightly,
and cargo-raze does not have support for them, so we currently need
to override the generated build file with a hand-crafted one that
specifies the relative features/deps for each platform.

update.py has been updated to automatically keep the version numbers
in this file up to date, so it should hopefully not prove too hard to
maintain going forward.
2020-11-15 15:25:28 +10:00
Damien Elmes
4bf93a9af0 add native-tls in preparation for Linux-only ring 2020-11-15 14:02:53 +10:00
Damien Elmes
a0c658621b update Rust deps 2020-11-15 13:31:24 +10:00
Damien Elmes
0489c92908 drop El Capitan support 2020-11-12 20:03:41 +10:00
Damien Elmes
835c29f8ca dump crate licenses as part of update run 2020-11-12 20:03:41 +10:00
Damien Elmes
1b3922d6a5 rs dep updates 2020-11-12 20:03:41 +10:00
Damien Elmes
b74166bc78 update to cargo-raze 0.7.0 + latest deps
Gives us wildcard entries in raze.toml, and solves warnings about
unrecognized "raze" entries.
2020-11-10 09:48:42 +10:00
Damien Elmes
9fe53ff549 switch to the stable Python ABI
Should allow the library to be imported in both Python 3.8+3.9.
2020-11-03 20:29:07 +10:00
Damien Elmes
3c12cb1600 update to latest fluent libs, and integrate maximum digit handling
We now limit number of digits in our formatter, instead of relying
on an upstream patch.
2020-11-03 14:10:45 +10:00
Damien Elmes
ef79596c5b add shallow_since to a bunch of git deps 2020-11-03 11:42:11 +10:00
Damien Elmes
8f5474bef0 ues native roots with rustls 2020-11-03 10:42:20 +10:00
Damien Elmes
e54bb9d1bf update env_logger, and minor dep updates 2020-11-03 09:04:57 +10:00
Damien Elmes
27e720c6b8 README updates 2020-11-03 09:03:00 +10:00
Damien Elmes
35f5358bf7 reqwest was missing default-features = false
Still needed until we fix OpenSSL compilation on Linux
2020-11-02 21:53:24 +10:00
Alan Du
562ea403b3 Update to PyO3 0.12
+ cargo raze by Damien
2020-11-02 18:38:34 +10:00
Damien Elmes
a079bb1820 update rusqlite and pin-project
Thanks to Alan for the heads up
2020-11-02 18:26:19 +10:00
Damien Elmes
249916af38 remote/ needs to be removed before updating 2020-11-02 18:25:50 +10:00
Damien Elmes
dc3bd07919 update cargo deps; add script to copy them from rslib/ 2020-11-02 18:17:26 +10:00
Damien Elmes
aea0a6fcc6 initial Bazel conversion
Running and testing should be working on the three platforms, but
there's still a fair bit that needs to be done:

- Wheel building + testing in a venv still needs to be implemented.
- Python requirements still need to be compiled with piptool and pinned;
need to compile on all platforms then merge
- Cargo deps in cargo/ and rslib/ need to be cleaned up, and ideally
unified into one place
- Currently using rustls to work around openssl compilation issues
on Linux, but this will break corporate proxies with custom SSL
authorities; need to conditionally use openssl or use
https://github.com/seanmonstar/reqwest/pull/1058
- Makefiles and docs still need cleaning up
- It may make sense to reparent ts/* to the top level, as we don't
nest the other modules under a specific language.
- rspy and pylib must always be updated in lock-step, so merging
rspy into pylib as a private module would simplify things.
- Merging desktop-ftl and mobile-ftl into the core ftl would make
managing and updating translations easier.
- Obsolete scripts need removing.
- And probably more.
2020-11-01 14:26:58 +10:00