Commit Graph

167 Commits

Author SHA1 Message Date
RumovZ
d55f080733
V3 parent limits (#1638)
* avoid repinning Rust deps by default

* add id_tree dependency

* Respect intermediate child limits in v3

* Test new behaviour of v3 counts

* Rework v3 queue building to respect parent limits

* Add missing did field to SQL query

* Fix `LimitTreeMap::is_exhausted()`

* Rework tree building logic

https://github.com/ankitects/anki/pull/1638#discussion_r798328734

* Add timer for build_queues()

* `is_exhausted()` -> `limit_reached()`

* Move context and limits into `QueueBuilder`

This allows for moving more logic into QueueBuilder, so less passing
around of arguments. Unfortunately, some tests will require additional
work to set up.

* Fix stop condition in new_cards_by_position

* Fix order gather order of new cards by deck

* Add scheduler/queue/builder/burying.rs

* Fix bad tree due to unsorted child decks

* Fix comment

* Fix `cap_new_to_review_rec()`

* Add test for new card gathering

* Always sort `child_decks()`

* Fix deck removal in `cap_new_to_review_rec()`

* Fix sibling ordering in new card gathering

* Remove limits for deck total count with children

* Add random gather order

* Remove bad sibling order handling

All routines ensure ascending order now.
Also do some other minor refactoring.

* Remove queue truncating

All routines stop now as soon as the root limit is reached.

* Move deck fetching into `QueueBuilder::new()`

* Rework new card gather and sort options

https://github.com/ankitects/anki/pull/1638#issuecomment-1032173013

* Disable new sort order choices ...

depending on set gather order.

* Use enum instead of numbers

* Ensure valid sort order setting

* Update new gather and sort order tooltips

* Warn about random insertion order with v3

* Revert "Add timer for build_queues()"

This reverts commit c9f5fc6ebe87953c17a0c842990b009b5596c69c.

* Update rslib/src/storage/card/mod.rs (dae)

* minor wording tweaks to the tooltips (dae)

+ move legacy strings to bottom
+ consistent capitalization (our leech action still needs fixing,
but that will require introducing a new 'suspend card' string as the
existing one is used elsewhere as well)
2022-02-10 09:55:43 +10:00
Damien Elmes
4d431fb7af move linkchecker into separate crate
The feature-based approach didn't work with cargo-raze, leading
to ./update.py in cargo/ breaking.
2021-12-20 17:27:43 +10:00
RumovZ
3e0c9dc866
New TTS/AV tag handling (#1559)
* Add new `card_rendering` mod

Parses a text with av/tts tags and strips or extracts tags.

* Replace old `extract_av_tags` and `strip_av_tags`

... with new `card_rendering` mod

* ressource -> resource

* Add AV prettifier for use in browser table

* Accept String in av tag routines

... and avoid redundant writes if no changes need to be made.

* add benchmarking with criterion; make links test optional (dae)

cargo install cargo-criterion, then run ./bench.sh

* performance comparison: creating HashMap up front (dae)

the previous solution:

anki_tag_parse          time:   [1.8401 us 1.8437 us 1.8476 us]

this solution:

anki_tag_parse          time:   [2.2420 us 2.2447 us 2.2477 us]
                        change: [+21.477% +21.770% +22.066%] (p = 0.00 < 0.05)
                        Performance has regressed.

* Revert "performance comparison: creating HashMap up front" (dae)

This reverts commit f19126a2f15b729b825825a49283f63ab13474d0.

* add missing header

* Write error message if tts lang is missing

* `Tag` -> `Directive`
2021-12-17 19:04:42 +10:00
Damien Elmes
11042a16ce Fix latest Rusqlite on Windows
https://github.com/bazelbuild/rules_rust/pull/1025
2021-11-19 10:44:02 +10:00
Damien Elmes
8b3d964866 roll rusqlite back due to Windows breakage 2021-11-18 21:39:23 +10:00
Damien Elmes
d31dccc388 update Rust deps 2021-11-18 20:58:41 +10:00
Damien Elmes
26123ce229 update to edition 2021 2021-11-18 20:51:10 +10:00
RumovZ
3672b0fe73
Switch CardInfoDialog to ts page (#1414)
* Only collect card stats on the backend ...

... instead of rendering an HTML string using askama.

* Add ts page Card Info

* Update test for new `col.card_stats()`

* Remove obsolete CardStats code

* Use new ts page in `CardInfoDialog`

* Align start and end instead of left and right

Curiously, `text-align: start` does not work for `th` tags if assigned
via classes.

* Adopt ts refactorings after rebase

#1405 and #1409

* Clean up `ts/card-info/BUILD.bazel`

* Port card info logic from Rust to TS

* Move repeated field to the top

https://github.com/ankitects/anki/pull/1414#discussion_r725402730

* Convert pseudo classes to interfaces

* CardInfoPage -> CardInfo

* Make revlog in card info optional

* Add legacy support for old card stats

* Check for undefined instead of falsy

* Make Revlog separate component

* drop askama dependency (dae)

* Fix nightmode for legacy card stats
2021-10-14 19:22:47 +10:00
Damien Elmes
944b064e54 update Rust deps 2021-10-02 20:42:03 +10:00
RumovZ
80264c3b56 Add links.rs for checking and building links 2021-07-22 10:03:03 +02:00
Damien Elmes
c944dd048e strip invalid Unicode chars in media check 2021-07-17 18:30:19 +10:00
Damien Elmes
bf507cca98 move from Python's URI escaping to IRI escaping in Rust
Should make non-Latin text readable in the HTML editor, without the
breakages reverted in the previous change.
2021-07-16 10:38:00 +10:00
Damien Elmes
576b141e2b add back in missing check for upload size
Compression now happens up-front, so we can tell in advance if the
upload size has been exceeded.
2021-06-29 10:50:03 +10:00
Damien Elmes
ca0374782e update to latest rusqlite 2021-06-25 16:22:21 +10:00
Damien Elmes
59e17950ad update most rust deps; skip rusqlite 2021-06-25 15:35:25 +10:00
Damien Elmes
a58826941f add rustls-native-certs
closes #1241
2021-06-18 18:15:41 +10:00
Damien Elmes
92e4de16cf report pinned package issues upstream 2021-04-27 23:11:22 +10:00
Damien Elmes
cf2613e869 update Rust deps
- tokio 1.0
- updated reqwest, thanks to Rumo
- other minor dep updates

the reqwest build file has been split into two, as it was awkward
to manually update the combined file, and the platform gate is now
on the target in rslib/
2021-04-27 22:18:12 +10:00
Damien Elmes
1704f7fe80 drop dependency on failure crate 2021-04-01 17:21:13 +10:00
Damien Elmes
e4002d7a5e update to latest Fluent 2021-03-27 13:24:11 +10:00
Damien Elmes
9aece2a7b8 rework translation handling
Instead of generating a fluent.proto file with a giant enum, create
a .json file representing the translations that downstream consumers
can use for code generation.

This enables the generation of a separate method for each translation,
with a docstring that shows the actual text, and any required arguments
listed in the function signature.

The codebase is still using the old enum for now; updating it will need
to come in future commits, and the old enum will need to be kept
around, as add-ons are referencing it.

Other changes:

- move translation code into a separate crate
- store the translations on a per-file/module basis, which will allow
us to avoid sending 1000+ strings on each JS page load in the future
- drop the undocumented support for external .ftl files, that we weren't
using
- duplicate strings in translation files are now checked for at build
time
- fix i18n test failing when run outside Bazel
- drop slog dependency in i18n module
2021-03-26 09:41:32 +10:00
Damien Elmes
4bcd2b68ab roll back slog-term due to Windows break (2/2)
The cargo.toml update was missing from the previous commit, and term
was included directly when it didn't need to be.
2021-03-10 13:28:32 +10:00
Damien Elmes
265e91c13f add strum crate 2021-03-10 11:53:27 +10:00
Damien Elmes
2c6b6734b5 experimental queue building
Still a work in progress, and hidden behind a feature flag.
2021-03-01 12:18:21 +10:00
Damien Elmes
ded626f0b9 render deck description with markdown; strip images
To support images on that screen, we'll first need to adjust the base url
for each platform, or rewrite the local image URLs, as otherwise they
are resolved to _anki/pages/...
2021-02-06 15:02:40 +10:00
Damien Elmes
6ba321f818 sanitize deck description in decks screen on backend
Looks like ammonia only adds about 800k to the release binary
2021-02-06 13:42:38 +10:00
Damien Elmes
da407a5000 pin proc_macro_nested due to Windows breakage 2021-02-03 21:17:11 +10:00
Damien Elmes
9225c804db update Rust deps; add async_trait 2021-01-11 19:19:45 +10:00
Damien Elmes
40c5f8a907 update prost to new 0.7 release+auto stage changes
Allows us to drop slow git repo.
2020-12-28 10:12:35 +10:00
cecini
d37ea0f5f7 Update rslib dep nom 2020-12-15 08:19:52 +00:00
Damien Elmes
c9b3ed1eae switch to workspace for Rust code 2020-11-24 18:41:03 +10:00
Damien Elmes
4bf93a9af0 add native-tls in preparation for Linux-only ring 2020-11-15 14:02:53 +10:00
Damien Elmes
0489c92908 drop El Capitan support 2020-11-12 20:03:41 +10:00
Damien Elmes
3c12cb1600 update to latest fluent libs, and integrate maximum digit handling
We now limit number of digits in our formatter, instead of relying
on an upstream patch.
2020-11-03 14:10:45 +10:00
Damien Elmes
ef79596c5b add shallow_since to a bunch of git deps 2020-11-03 11:42:11 +10:00
Damien Elmes
8f5474bef0 ues native roots with rustls 2020-11-03 10:42:20 +10:00
Damien Elmes
e54bb9d1bf update env_logger, and minor dep updates 2020-11-03 09:04:57 +10:00
Damien Elmes
35f5358bf7 reqwest was missing default-features = false
Still needed until we fix OpenSSL compilation on Linux
2020-11-02 21:53:24 +10:00
Damien Elmes
a079bb1820 update rusqlite and pin-project
Thanks to Alan for the heads up
2020-11-02 18:26:19 +10:00
Damien Elmes
dc3bd07919 update cargo deps; add script to copy them from rslib/ 2020-11-02 18:17:26 +10:00
Damien Elmes
aea0a6fcc6 initial Bazel conversion
Running and testing should be working on the three platforms, but
there's still a fair bit that needs to be done:

- Wheel building + testing in a venv still needs to be implemented.
- Python requirements still need to be compiled with piptool and pinned;
need to compile on all platforms then merge
- Cargo deps in cargo/ and rslib/ need to be cleaned up, and ideally
unified into one place
- Currently using rustls to work around openssl compilation issues
on Linux, but this will break corporate proxies with custom SSL
authorities; need to conditionally use openssl or use
https://github.com/seanmonstar/reqwest/pull/1058
- Makefiles and docs still need cleaning up
- It may make sense to reparent ts/* to the top level, as we don't
nest the other modules under a specific language.
- rspy and pylib must always be updated in lock-step, so merging
rspy into pylib as a private module would simplify things.
- Merging desktop-ftl and mobile-ftl into the core ftl would make
managing and updating translations easier.
- Obsolete scripts need removing.
- And probably more.
2020-11-01 14:26:58 +10:00
Damien Elmes
53a984ba15 bump version 2020-10-02 10:22:21 +10:00
Damien Elmes
c125893eba bump version 2020-09-24 10:52:24 +10:00
Aleksa Sarai
d2ada5a1ee
rslib: disable bzip2 feature
We can't link against libzip2.so.1 when building manylinux wheels, but
the only dependency which required this was "zip" and we don't appear to
make use of bzip2 in Anki.

This fixes the following "make build" error on Linux:

  x maturin failed
    Caused by: Failed to ensure manylinux compliance
    Caused by: Your library is not manylinux compliant because it links the following forbidden libraries: ["libbz2.so.1"]

Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
2020-09-05 19:39:11 +10:00
Aleksa Sarai
312393e825
rs: remove most &mut references in BackendService
The previous implementation had some slightly questionable memory safety
properties (older versions of PyO3 didn't uphold the Rust aliasing rules
and would thus create multiple &mut references to #[pyclass] objects).
This explains why Backend has internal Mutex<T>s even though all of its
methods took &mut self.

The solution is to simply make all methods take &self, which luckily
doesn't pose too make issues -- most of the code inside Backend already
has sufficient locking. The only two things which needed to be
explicitly handled where:

1. "self.runtime" which was fairly easy to handle. All usages of
   the Runtime only require an immutable reference to create a new
   Handle, so we could switch to OnceCell which provides
   lazy-initialisation semantics without needing a more heavy-handed
   Mutex<tokio::runtime::Handle>.

2. "self.sync_abort" was simply wrapped in a Mutex<>, though some of the
   odd semantics of sync_abort (not being able to handle multiple
   processes synchronising at the same time) become pretty obvious with
   this change (for now we just log a warning in that case). In
   addition, switch to an RAII-style guard to make sure we don't forget
   to clear the abort_handle.

As a result, we now no longer break Rust's aliasing rules and we can
build with newer versions of PyO3 which have runtime checks for these
things (and build on stable Rust).

Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
2020-09-05 19:37:59 +10:00
Damien Elmes
819dce3ab6 bump version 2020-08-31 11:07:54 +10:00
Damien Elmes
fe8e3c7c7e use commit hash in reqwests dependency 2020-08-26 18:50:25 +10:00
Damien Elmes
7f15d06c58 vendor reqwest for now, to support read/write timeouts
https://github.com/seanmonstar/reqwest/issues/870

adding back now that fix has been applied to a fork of tokio-io-timeout
2020-08-26 13:24:51 +10:00
Damien Elmes
7e6a65ed3d bump version 2020-08-26 08:05:45 +10:00
Damien Elmes
ef7f61ab39 Revert "vendor reqwest for now, to support read/write timeouts"
This reverts commit cf247baf77.

The separate read and write timeouts were causing syncs to fail - a
long full upload would time out due to no read activity for example.
Reverting until a better fix can be put in place.
2020-08-25 13:28:54 +10:00
Damien Elmes
7d7c791504 bump version 2020-08-24 11:28:22 +10:00
Damien Elmes
cf247baf77 vendor reqwest for now, to support read/write timeouts
https://github.com/seanmonstar/reqwest/issues/870
2020-08-21 18:50:00 +10:00
Damien Elmes
b380f304bf bump version 2020-08-09 13:44:59 +10:00
Damien Elmes
227703f9fa bump version 2020-07-28 12:59:56 +10:00
Damien Elmes
412f1777bd bump version 2020-07-21 15:54:14 +10:00
Damien Elmes
1205db6484 panic fix was merged into hyper 0.13.7 2020-07-16 09:09:13 +10:00
Damien Elmes
df5487875b vendor coarsetime so we can get the use after free fix 2020-07-13 13:34:37 +10:00
Damien Elmes
75443b5171 update rslib/rspy deps 2020-07-12 11:38:25 +10:00
Damien Elmes
b51f03085e migrate card stats to backend
Currently this renders the HTML directly like the previous Python
implementation - doing it in JS would probably make more sense in the
future.
2020-06-15 17:22:16 +10:00
Damien Elmes
10a1d06dce use vendored hyper for panic fix 2020-06-11 13:30:35 +10:00
Damien Elmes
9e56720c2b Revert "use trust-dns in reqwest"
This reverts commit abe90d0772.

trust-dns has introduced its own issues, such as when resolv.conf is
inaccessible, or contains hostnames with certain characters
2020-06-08 22:03:24 +10:00
Damien Elmes
abe90d0772 use trust-dns in reqwest
The default DNS resolution, if interrupted by an Abortable, will
cause a panic:

57ef271500/src/client/connect/dns.rs (L144)
2020-06-03 15:47:54 +10:00
Damien Elmes
ac219ae728 use a shared async runtime instead of recreating each time 2020-06-02 09:35:27 +10:00
Damien Elmes
529e89f48e full upload/download HTTP code 2020-06-01 13:57:10 +10:00
Damien Elmes
4fcb10bfa9 start on HTTP client for syncing 2020-06-01 13:57:10 +10:00
Damien Elmes
8bd5d756e2 bump version 2020-05-18 12:50:20 +10:00
Damien Elmes
da43d22aa5 update rusqlite 2020-05-16 19:49:08 +10:00
Damien Elmes
83bcb084fe template changes and card generation
Cloze cards are not yet supported, missing decks are not handled,
and more testing is still required.
2020-05-12 21:13:33 +10:00
Damien Elmes
f24dc05c8d the start of note adding, and note type changes 2020-05-12 21:13:33 +10:00
Damien Elmes
08ae4e7900 update crate version
make check was missing this
2020-05-11 20:41:16 +10:00
Damien Elmes
36ea980725 update rslib version 2020-05-02 10:46:58 +10:00
Damien Elmes
00cf4fc585 bump version 2020-04-28 08:57:56 +10:00
Damien Elmes
11fdd96a44 add socks support 2020-04-09 14:28:36 +10:00
Damien Elmes
92fbe3bfba update rusqlite 2020-04-09 12:45:51 +10:00
Damien Elmes
c950a00610 Cow<str> workaround no longer required 2020-04-07 10:08:03 +10:00
Damien Elmes
caa513899d update rusqlite for param count assert fix 2020-04-07 08:54:57 +10:00
Damien Elmes
ac4284b2de update tag handling
- tag list stored in a separate DB table
- non-wildcard searches now do full unicode case folding
(eg tag:masse matches 'Maße')
- wildcard matches do simple unicode case folding
- some functions haven't been updated yet, so ascii folding will
continue to be used in some operations
2020-04-03 19:34:46 +10:00
Damien Elmes
ee71e5dd59 use support URL and point to source repo 2020-04-03 08:44:59 +10:00
evandrocoan
069cc11a86 Created the GitHub Actions step Upload to PyPi to run only on
tagged releases of anki:
https://anki.tenderapp.com/discussions/ankidesktop/40025-anki-in-python-package-index
2020-04-02 02:57:34 -03:00
Damien Elmes
c9da4bc1a6 allow aborting a media sync while http req in flight
The progress handling code needs a rethink, as we now have two separate
ways to flag that the media sync should abort. In the future, it may
make sense to switch to polling the backend for progress, instead of
passing a callback in.
2020-04-01 09:49:25 +10:00
Damien Elmes
2810d3883b roll back the vec cache changes
prepare_cached() is sufficiently fast, and allowing the vec cache
to persist across calls is complicated due to lifetime restrictions
2020-03-29 09:26:09 +10:00
Damien Elmes
1a1a00d50f Revert "add benchmark for vec cache"
This reverts commit 82ed288dc5.
2020-03-29 09:16:11 +10:00
Damien Elmes
82ed288dc5 add benchmark for vec cache
test storage::sqlite::bench::bench_hash_cache ... bench:         399 ns/iter (+/- 27)
test storage::sqlite::bench::bench_no_cache   ... bench:       4,854 ns/iter (+/- 499)
test storage::sqlite::bench::bench_vec_cache  ... bench:           0 ns/iter (+/- 0)
2020-03-27 09:59:48 +10:00
Damien Elmes
2f4e35d566 store cached statements in a vec instead of separate optionals 2020-03-27 09:51:19 +10:00
Damien Elmes
289bdde20c handle fields with embedded nuls 2020-03-24 14:53:33 +10:00
Damien Elmes
4ff17d31b3 add unicase collation
sqlite's like is hard-coded to use ASCII comparisons, so we can't
take advantage of this yet
2020-03-21 12:40:20 +10:00
Damien Elmes
7eab504126 add field_at_index() sql func 2020-03-20 21:15:23 +10:00
Damien Elmes
761d1d1812 add card queue/type enums 2020-03-20 21:15:23 +10:00
Damien Elmes
f617760d04 bump version 2020-03-19 10:46:01 +10:00
Damien Elmes
071722c057 rspy version 2020-03-06 19:15:02 +10:00
Damien Elmes
f7ec2faba6 update to failure 0.1.7 2020-03-06 15:08:58 +10:00
Damien Elmes
bb0e5dfa93 work around bug in failure crate 2020-03-05 11:06:44 +10:00
Damien Elmes
ba4fe71eba make logging explicit, and support logging to a file 2020-02-29 20:44:01 +10:00
Alan Du
3ebdaa14de Update blake3 to 0.2 2020-02-26 22:50:48 -05:00
Alan Du
3867c20cfd Update prost-build version to 0.6
This is the same version as Prost, and lets us reduce the number of deps
we have to compile
2020-02-26 22:48:16 -05:00
Damien Elmes
b412747a16 add workaround for panic on 32 bit systems
https://anki.tenderapp.com/discussions/beta-testing/1817-failed-to-generate-operands-out-of-fluentnumber-could-not-convert-string-to-integer
2020-02-25 13:22:41 +10:00
Damien Elmes
4430c67069 rework Fluent handling
- all .ftl files for a language are concatenated into a single file
at build time
- all languages are included in the binary
- external ftl files placed in the ftl folder can override the
built-in definitions
- constants are automatically generated for each string key
- dropped the separate StringsGroup enum
2020-02-23 13:22:50 +10:00
Damien Elmes
e31d576740 fluent-rs git no longer required 2020-02-21 21:30:11 +10:00
Damien Elmes
ffffdf2c85 fix revision pin 2020-02-21 15:29:38 +10:00
Damien Elmes
370bb38b8b update to latest fluent-rs and add basic locale-aware decimals
- git version pinned at the moment until the concurrency fix
lands in 0.10.2
- currently float values are hard-coded at 2 decimal places;
we should switch to using NUMBER() in the future
2020-02-21 15:29:38 +10:00