Commit Graph

1884 Commits

Author SHA1 Message Date
Damien Elmes
ded805b504
Switch Rust import style (#2330)
* Prepare to switch Rust import style

* Run nightly format

Closes #2320

* Clean up a few imports

* Enable comment wrapping

* Wrap comments
2023-01-18 21:39:55 +10:00
Damien Elmes
9d84f357b6 Probable fix for flaky test 2023-01-18 21:32:08 +10:00
Damien Elmes
c923553a53 Replace newlines with linebreaks when importing without HTML
https://forums.ankiweb.net/t/line-breaks-are-removed-when-importing-csv-on-linux/26451
2023-01-18 21:00:33 +10:00
Damien Elmes
cf45cbf429
Rework syncing code, and replace local sync server (#2329)
This PR replaces the existing Python-driven sync server with a new one in Rust.
The new server supports both collection and media syncing, and is compatible
with both the new protocol mentioned below, and older clients. A setting has
been added to the preferences screen to point Anki to a local server, and a
similar setting is likely to come to AnkiMobile soon.

Documentation is available here: <https://docs.ankiweb.net/sync-server.html>

In addition to the new server and refactoring, this PR also makes changes to the
sync protocol. The existing sync protocol places payloads and metadata inside a
multipart POST body, which causes a few headaches:

- Legacy clients build the request in a non-deterministic order, meaning the
entire request needs to be scanned to extract the metadata.
- Reqwest's multipart API directly writes the multipart body, without exposing
the resulting stream to us, making it harder to track the progress of the
transfer. We've been relying on a patched version of reqwest for timeouts,
which is a pain to keep up to date.

To address these issues, the metadata is now sent in a HTTP header, with the
data payload sent directly in the body. Instead of the slower gzip, we now
use zstd. The old timeout handling code has been replaced with a new implementation
that wraps the request and response body streams to track progress, allowing us
to drop the git dependencies for reqwest, hyper-timeout and tokio-io-timeout.

The main other change to the protocol is that one-way syncs no longer need to
downgrade the collection to schema 11 prior to sending.
2023-01-18 12:43:46 +10:00
RumovZ
1be30573e1
Replace dissimilar crate with difflib (#2322)
This also inserts the expected text if it's missing at the very
beginning of the provided text.
2023-01-16 09:49:34 +10:00
RumovZ
6df7be90c5
Fix negated introduced:x search (#2306)
1. Add outer brackets.
2. Coalesce aggregate, because `null and true` is `null` in SQL land,
so cards that were not introduced, but manually rescheduled in the
period of interest, would not show up in a negated search.
2023-01-10 08:49:35 +10:00
Damien Elmes
07fd88ddea Allow timestamps to be a day ahead
https://github.com/ankitects/anki/issues/1895#issuecomment-1374574230
2023-01-09 10:04:48 +10:00
dependabot[bot]
4e92c29815
Bump tokio from 1.23.0 to 1.23.1 (#2303)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.23.0 to 1.23.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.23.0...tokio-1.23.1)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-07 10:01:56 +10:00
Damien Elmes
8206f39bdc Fix sparse today stats
Not safe to assume revlog is ordered by query
2023-01-05 10:14:37 +10:00
Damien Elmes
0eddb25287
Integrate AnkiDroid's backend patches into the repo (#2290)
* Relax chrono specification for AnkiDroid

https://github.com/ankidroid/Anki-Android-Backend/pull/251

* Add AnkiDroid service and AnkiDroid customizations

Most of the work here was done by David in the Backend repo; integrating
it into this repo for ease of future maintenance.

Based on 5d9f262f4c
with some tweaks:

- Protobuf imports have been fixed to match the recent refactor
- FatalError has been renamed to AnkidroidPanicError
- Tweaks to the desktop code to deal with the extra arg to open_collection,
and exclude AnkiDroid service methods from our Python code.

* Refactor AnkiDroid's DB code to avoid uses of unsafe
2023-01-03 13:11:23 +10:00
Damien Elmes
4c5a2461d0 Handle broken utimes() on Android
Closes https://github.com/ankidroid/Anki-Android/issues/12827
Closes https://github.com/ankidroid/Anki-Android/issues/12974
2022-12-28 16:33:06 +10:00
Damien Elmes
a38121a713 Add Debug impl for Collection 2022-12-24 10:44:41 +10:00
Damien Elmes
a82f8d6872 Remove unused before_upload method 2022-12-24 10:44:41 +10:00
Damien Elmes
0570cfdf48 Migrate from slog to tracing
The Rust community appear to have converged on tracing - it's used by
the Rust compiler, and receives close to 10x the number of downloads
that slog does. Its API is more ergonomic, and it does a much nicer
job with async rust.

To make this change, we no longer pass around explicit loggers, and rely
on a globally-registered one. The log file location has been changed
from one in each profile folder to a single one in the base folder. This
will remain empty for most users, since only errors are logged by default,
but may be useful for debugging future changes.
2022-12-24 10:44:40 +10:00
Damien Elmes
d294fff882 Simplify solution for #2279 2022-12-19 18:13:34 +10:00
Damien Elmes
67acaa17e4 Fix panic when adding non-Latin text to fields
Closes #2279
2022-12-19 18:02:43 +10:00
TRIAEIOU
9901ae428a
Nested clozes and increased cloze meta data (#2141)
* Nested clozes and increased cloze meta data

* Update contributors

* This reverts commit 3423df73f89f04a606b1bff3542a68a49ca52e9f.

* Update CONTRIBUTORS

* Formating

* Formating

* Formating

* Formating

* Formating

* Formating

* Formating

* Formating

* Code refactor

* Formating

* Formating

* Formating

* Formating and dead code

* Correct test case

* Remove Hint and Close storage of token string

* Update

* Formating

* Formating

* Formating

* Use write! instead of .push_str(&format).

* Formating
2022-12-19 12:03:15 +10:00
Damien Elmes
0555f4898c Update crate name in docstrings to fix 'cargo test' 2022-12-16 22:30:54 +10:00
Damien Elmes
37151213cd Move more of the graph processing into the backend
The existing architecture serializes all cards and revlog entries in
the search range into a protobuf message, which the web frontend needs
to decode and then process. The thinking at the time was that this would
make it easier for add-ons to add extra graphs, but in the ~2.5 years
since the new graphs were introduced, no add-ons appear to have taken
advantage of it.

The cards and revlog entries can grow quite large on large collections -
on a collection I tested with approximately 2.5M reviews, the serialized
data is about 110MB, which is a lot to have to deserialize in JavaScript.

This commit shifts the preliminary processing of the data to the Rust end,
which means the data is able to be processed faster, and less needs to
be sent to the frontend. On the test collection above, this reduces the
serialized data from about 110MB to about 160KB, resulting in a more
than 2x performance improvement, and reducing frontend memory usage from
about 400MB to about 40MB.

This also makes #2043 more feasible - while it is still about 50-100%
slower than protobufjs, with the much smaller message size, the difference
is only about 10ms.
2022-12-16 21:42:17 +10:00
Damien Elmes
fa625d7ad8
Minor Rust cleanups (#2272)
* Run cargo +nightly fmt

* Latest prost-build includes clippy workaround

* Tweak Rust protobuf imports

- Avoid use of stringify!(), as JetBrains editors get confused by it
- Stop merging all protobuf symbols into a single namespace

* Remove some unnecessary qualifications

Found via IntelliJ lint

* Migrate some asserts to assert_eq/ne

* Remove mention of node_modules exclusion

This no longer seems to be necessary after migrating away from Bazel,
and excluding it means TS/Svelte files can't be edited properly.
2022-12-16 21:40:27 +10:00
Damien Elmes
22ecef6fb2
Merge pull request #2255 from RumovZ/replace-pct-str-2
Replace pct-str with local percent_encoding crate
2022-12-16 11:21:04 +00:00
Damien Elmes
35cbde65d7 Revert "Always scan for media changes"
This reverts commit 09cb8b3cf6.

Overhead on larger folders/slower devices is more than I originally
anticipated, and can run into multiple seconds. This seems to be
particularly egregious on mobile, which I presume is due to sandboxing
overhead.
2022-12-15 20:40:20 +10:00
Damien Elmes
86b52f7626 Add a small unit test
pct-str encoded the / character as well, but the difference shouldn't
matter in our case.
2022-12-12 14:54:11 +10:00
RumovZ
c888ccc285 Replace pct-str with local ascii_percent_encoding 2022-12-09 11:49:39 +01:00
RumovZ
0b206b8a81 Ignore non-ASCII chars in ascii_percent_encoding 2022-12-09 11:47:59 +01:00
RumovZ
55f27779cf Add local copy of percent_encoding crate 2022-12-09 11:46:00 +01:00
RumovZ
0e7f02bfb7
Update Chrono Crate (#2242)
* Remove deprecated `and_hms()`

* Update chrono

* Update licenses and fix script

* Remove deprecated Date struct

* Remove chrono pin

* Skip format check on .vscode

Was failing for no reason.

* Replace deprecated chrono functions

* Add cargo-deny to update-licenses & pin versions (dae)

* Remove time 0.1 dependency  (dae)

We don't need to wait for chrono 0.5; it was provided behind a legacy
feature flag.
2022-12-07 17:00:14 +10:00
Damien Elmes
7e3ac53869 Allow unit tests to pass even when Qt translations overridden 2022-12-04 14:25:54 +10:00
RumovZ
90a3bbfdb8
Cap calculated hard delay secs at again delay + 1d (#2229) 2022-12-03 18:54:40 +10:00
Damien Elmes
e3167c4e3c Update incompatible crates 2022-11-30 12:38:10 +10:00
Damien Elmes
b4f4df0030 Pin chrono; update compatible Rust crates
Also pin reqwest in our other crates
2022-11-30 12:38:10 +10:00
Damien Elmes
0ac7969e2a Use workspace package info in more crates; mark private for cargo-deny 2022-11-30 12:19:56 +10:00
Damien Elmes
e497a56f54 Re-enable formatting for .toml files 2022-11-28 09:16:28 +10:00
Damien Elmes
7972708029 Remove rslib cargo config
Was added during initial Bazel experimentation back in 2020
2022-11-27 17:50:36 +10:00
Damien Elmes
5e0a761b87
Move away from Bazel (#2202)
(for upgrading users, please see the notes at the bottom)

Bazel brought a lot of nice things to the table, such as rebuilds based on
content changes instead of modification times, caching of build products,
detection of incorrect build rules via a sandbox, and so on. Rewriting the build
in Bazel was also an opportunity to improve on the Makefile-based build we had
prior, which was pretty poor: most dependencies were external or not pinned, and
the build graph was poorly defined and mostly serialized. It was not uncommon
for fresh checkouts to fail due to floating dependencies, or for things to break
when trying to switch to an older commit.

For day-to-day development, I think Bazel served us reasonably well - we could
generally switch between branches while being confident that builds would be
correct and reasonably fast, and not require full rebuilds (except on Windows,
where the lack of a sandbox and the TS rules would cause build breakages when TS
files were renamed/removed).

Bazel achieves that reliability by defining rules for each programming language
that define how source files should be turned into outputs. For the rules to
work with Bazel's sandboxing approach, they often have to reimplement or
partially bypass the standard tools that each programming language provides. The
Rust rules call Rust's compiler directly for example, instead of using Cargo,
and the Python rules extract each PyPi package into a separate folder that gets
added to sys.path.

These separate language rules allow proper declaration of inputs and outputs,
and offer some advantages such as caching of build products and fine-grained
dependency installation. But they also bring some downsides:

- The rules don't always support use-cases/platforms that the standard language
tools do, meaning they need to be patched to be used. I've had to contribute a
number of patches to the Rust, Python and JS rules to unblock various issues.
- The dependencies we use with each language sometimes make assumptions that do
not hold in Bazel, meaning they either need to be pinned or patched, or the
language rules need to be adjusted to accommodate them.

I was hopeful that after the initial setup work, things would be relatively
smooth-sailing. Unfortunately, that has not proved to be the case. Things
frequently broke when dependencies or the language rules were updated, and I
began to get frustrated at the amount of Anki development time I was instead
spending on build system upkeep. It's now about 2 years since switching to
Bazel, and I think it's time to cut losses, and switch to something else that's
a better fit.

The new build system is based on a small build tool called Ninja, and some
custom Rust code in build/. This means that to build Anki, Bazel is no longer
required, but Ninja and Rust need to be installed on your system. Python and
Node toolchains are automatically downloaded like in Bazel.

This new build system should result in faster builds in some cases:

- Because we're using cargo to build now, Rust builds are able to take advantage
of pipelining and incremental debug builds, which we didn't have with Bazel.
It's also easier to override the default linker on Linux/macOS, which can
further improve speeds.
- External Rust crates are now built with opt=1, which improves performance
of debug builds.
- Esbuild is now used to transpile TypeScript, instead of invoking the TypeScript
compiler. This results in faster builds, by deferring typechecking to test/check
time, and by allowing more work to happen in parallel.

As an example of the differences, when testing with the mold linker on Linux,
adding a new message to tags.proto (which triggers a recompile of the bulk of
the Rust and TypeScript code) results in a compile that goes from about 22s on
Bazel to about 7s in the new system. With the standard linker, it's about 9s.

Some other changes of note:

- Our Rust workspace now uses cargo-hakari to ensure all packages agree on
available features, preventing unnecessary rebuilds.
- pylib/anki is now a PEP420 implicit namespace, avoiding the need to merge
source files and generated files into a single folder for running. By telling
VSCode about the extra search path, code completion now works with generated
files without needing to symlink them into the source folder.
- qt/aqt can't use PEP420 as it's difficult to get rid of aqt/__init__.py.
Instead, the generated files are now placed in a separate _aqt package that's
added to the path.
- ts/lib is now exposed as @tslib, so the source code and generated code can be
provided under the same namespace without a merging step.
- MyPy and PyLint are now invoked once for the entire codebase.
- dprint will be used to format TypeScript/json files in the future instead of
the slower prettier (currently turned off to avoid causing conflicts). It can
automatically defer to prettier when formatting Svelte files.
- svelte-check is now used for typechecking our Svelte code, which revealed a
few typing issues that went undetected with the old system.
- The Jest unit tests now work on Windows as well.

If you're upgrading from Bazel, updated usage instructions are in docs/development.md and docs/build.md. A summary of the changes:

- please remove node_modules and .bazel
- install rustup (https://rustup.rs/)
- install rsync if not already installed  (on windows, use pacman - see docs/windows.md)
- install Ninja (unzip from https://github.com/ninja-build/ninja/releases/tag/v1.11.1 and
  place on your path, or from your distro/homebrew if it's 1.10+)
- update .vscode/settings.json from .vscode.dist
2022-11-27 15:24:20 +10:00
Stefan Kangas
5551a37f03
Fix typos (#2210) 2022-11-24 20:18:57 +10:00
Damien Elmes
ade2738458 Fix invalid utf8 not being detected
- Rusqlite was returning a different error
- The sort field may also contain invalid utf8
2022-11-22 13:27:20 +10:00
Damien Elmes
063623af3c Format .toml files with dprint 2022-11-09 20:03:49 +10:00
dobefore
0c9cecd7a3
Fix error while compiling rslib (#2187)
* protobuf generate error

pass --experimental_allow_proto3_optional to protoc
add missing dereive trait to struct Daylimit

* delete invalid derive trait Eq

* remove argument from protoc

--experimental_allow_proto3_optional
2022-11-09 12:36:23 +10:00
Damien Elmes
c9a9f38ea9 Remove untranslated 'see old deck options' notice
It was intended to be a temporary message, and it's been about 15 months.

https://forums.ankiweb.net/t/anki-2-1-55-beta-3/24295/42
2022-11-03 12:05:19 +10:00
Abdo
95d0c78b78
Fix issue when dragged deck is a prefix of target (#2149) 2022-10-29 09:42:34 +10:00
Abdo
abbfdbf420
Fix grandchild tag not being reparentable under the same root (#2148)
Introduced in #2146
2022-10-29 09:39:44 +10:00
Abdo
951c80a4e9
Fix some issues with tag reparenting (#2146)
* Fix reparented_name not correctly handling tags that are prefixes of the
new parent

To reproduce the issue:
1. Add two tags: `a` and `ab`.
2. From the browser's sidebar, drag & drop `a` into `ab`.

Result: panic

* Fix reparent_tags panicking if new parent is a child of source tag

This is the "foo, foo::bar" case that should be a no-op.

* Add more tests for tag reparenting
2022-10-28 21:42:10 +10:00
RumovZ
c521753057
Refactor error handling (#2136)
* Add crate snafu

* Replace all inline structs in AnkiError

* Derive Snafu on AnkiError

* Use snafu for card type errors

* Use snafu whatever error for InvalidInput

* Use snafu for NotFoundError and improve message

* Use snafu for FileIoError to attach context

Remove IoError.
Add some context-attaching helpers to replace code returning bare
io::Errors.

* Add more context-attaching io helpers

* Add message, context and backtrace to new snafus

* Utilize error context and backtrace on frontend

* Rename LocalizedError -> BackendError.
* Remove DocumentedError.
* Have all backend exceptions inherit BackendError.

* Rename localized(_description) -> message

* Remove accidentally committed experimental trait

* invalid_input_context -> ok_or_invalid

* ensure_valid_input! -> require!

* Always return `Err` from `invalid_input!`

Instead of a Result to unwrap, the macro accepts a source error now.

* new_tempfile_in_parent -> new_tempfile_in_parent_of

* ok_or_not_found -> or_not_found

* ok_or_invalid -> or_invalid

* Add crate convert_case

* Use unqualified lowercase type name

* Remove uses of snafu::ensure

* Allow public construction of InvalidInputErrors (dae)

Needed to port the AnkiDroid changes.

* Make into_protobuf() public (dae)

Also required for AnkiDroid. Not sure why it worked previously - possible
bug in older Rust version?
2022-10-21 18:02:12 +10:00
RumovZ
b8c294bf4d
Explicitly evaluate symlink on Windows (#2135) 2022-10-19 20:08:58 +10:00
Damien Elmes
fb9c934ef2 Use protoc from Bazel if missing from path
Closes #2134
2022-10-17 09:58:51 +10:00
Damien Elmes
cfb309e6b3 Update Rust deps 2022-09-24 13:22:46 +10:00
Damien Elmes
a39a3b4d34 Update to latest rules_rust and Rust 1.64 2022-09-24 11:12:58 +10:00
Damien Elmes
b97a8bfa26 Fix daily counts being included in apkg import
Fixes https://github.com/ankidroid/Anki-Android/issues/12477
2022-09-24 09:05:29 +10:00
Damien Elmes
09cb8b3cf6 Always scan for media changes
The check runs in the background, and the dentry cache and SQLite
page cache should make this more efficient after the first run.

Closes #1736
2022-09-20 16:13:10 +10:00
RumovZ
e7af0febb1
More template checks (#2032)
* Show warning if multiple type boxes are used

* Report templates referencing media in Media Check

* Apply suggestions from code review

* Fix media-check.ftl

* Only report media references with fields

Like `<img src={{Front}}>`.
Also report Anki sound tags and latex.

* Loop existing media regexes
2022-09-05 16:52:25 +10:00
RumovZ
e39fb74e82
Enable state-dependent custom scheduling data (#2049)
* Enable state-dependent custom scheduling data

* Next(Card)States -> SchedulingStates

The fact that `current` was included in `next` always bothered me,
and custom data is part of the card state, so that was a bit confusing
too.

* Store custom_data in SchedulingState

* Make custom_data optional when answering

Avoids having to send it 4 extra times to the frontend, and avoids the
legacy answerCard() API clobbering the stored data.

Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-09-05 16:48:01 +10:00
Damien Elmes
0bcb3a3564 Add non-legacy backend interface for adding new decks 2022-09-04 14:12:29 +10:00
RumovZ
1f8189fe91
Some import/export features and fixes (#2038)
* Keep filtered decks when importing apkg

If all original decks exist and scheduling is included.

* Create missing decks from csv

* Export original decks if with_scheduling

* Also remap original deck ids on import

* Update imported filtered decks

* Fix meta column being mapped to tags

* Fix ids in csv deck and notetype columns

Note: This implies names which parse to an i64 will be seen as ids,
likely resulting in the intended deck/notetype not being found.

* Check for scheduling with revlog and deck configs

Might help with cases in which scheduling was included, but all cards
are new. In such a case, filtered deck should not be converted.

* Fix duplicate with same GUID being created

* Remove redundant `distinct`s from sql query

* Match notes by _either_ guid _or_ first field

* Refactor to emphasise GUID/first field distinction

* Export default deck and config if with scheduling

* Fix default deck being exported if it's a parent
2022-09-03 11:29:06 +10:00
RumovZ
31b7464c67
Add card meta for persisting custom scheduling state (#2040)
* Add card meta for persisting custom scheduling state

* Rename meta -> custom_data

* Enforce limits on size of custom data

Large values will slow down table scans of the cards table, and it's
easier to be strict now and possibly relax things in the future than
the opposite.

* Pack card states and customData into a single message

+ default customData to empty if it can't be parsed

Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-09-02 11:22:49 +10:00
Matthias Metelka
d110c4916c
Introduce setting to collapse field by default (#1990)
* Introduce setting to collapse field by default

* Fix schema order

* Change wording from adjective to imperative

sounds a bit less clunky

* Update rslib/src/notetype/schema11.rs (dae)

* Keep settings in single column

* Add back Toggle Visual Editor string

* Add RichTextBadge component and show it conditionally

* Reverse input order depending on default setting

* Make PlainTextInput border-radius responsive to toggle states

* Prevent first Collapsible transition differently

* Focus inputs after Collapsible transition

The double tick calls are just a temporary solution until I find the exact moment an input is focusable again.

* Use requestAnimationFrame to await focusable state

Note: Svelte tick doesn't seem to work in this scenario.
2022-08-31 23:34:39 +10:00
RumovZ
79fbb6c8d8
Keep content of unmapped fields when importing (#2023)
* Keep content of unmapped fields when importing

* Test new behaviour

* Fix typo in `canonify_tags_without_resgistering`

* Log updated note instead of original one

* Revert merging imported tags

But keep old note tags if no new ones are provided.
2022-08-24 16:04:32 +10:00
Matthias Metelka
d1cbb86178
Default input setting in fields dialog (#1987)
* Introduce field setting to use plain text editor by default

* Remove leftover function from #1476

* Use boolean instead of string

* Simplify clear_other_field_duplicates

* Convert plain text key to camelCase

* Move HTML item below the existing checkbox, instead of to the right (dae)

Showing it on the right is more space efficient, but feels a bit
cluttered IMHO.
2022-08-18 12:30:18 +10:00
Damien Elmes
bd2f866c68 Fix {{CardFlags}} not working with flags 5-7
https://forums.ankiweb.net/t/please-activate-flags-5-7-on-the-cardflag/22143
2022-08-08 10:56:07 +10:00
Damien Elmes
f017525ea8 Handle deeply nested decks 2022-08-03 14:37:56 +10:00
Damien Elmes
340da23104 Wrap type answer output in code tag; remove extra div
This would make more sense as a class on the outer div, but existing
templates reference code#typeans, so avoid breaking things for now.
2022-07-22 20:37:56 +10:00
Damien Elmes
ad8c23fc3d Remove resolved comment 2022-07-22 20:06:45 +10:00
Damien Elmes
09eef8e746 Spelling fix 2022-07-22 19:32:31 +10:00
Damien Elmes
1e0be26b7e Partially migrate type answer to backend
Partially completes #1068, and will allow mobile clients to drop
their separate diff-match-patch imports. Does not yet try to handle
case folding or combining-char stripping, and leaves some of the outer
HTML wrapping up to the frontend for now.

The logic for rendering the provided string has changed: missing chars
are now only inserted if they follow a correct section, and the original
text is shown instead of hyphens. This is an experiment, and can be
changed if it's not well received.
2022-07-22 19:28:34 +10:00
RumovZ
173a5bfed5
Improve temporary table handling (#1976)
* Use all_cards_for_search() helper

* Add CardTableGuard

* Add for_each_card_in_search() helper

* Add all_cards_for_search_in_order() helper

* Add all_cards_for_ids() helper

* Return siblings for bury instead of only searching

* Remove redundant clear_searched_cards_table() calls

* Add with_searched_cards_table()

* Remove false comment

* Add NoteTableGuard

* Add with_ids_in_searched_notes_table() helper

* Make some last routines use table helpers
2022-07-22 17:51:26 +10:00
RumovZ
09841c7c0f
Populate media DB on import (#1977)
* Refactor MediaManager transactions

* Add media entries when importing

* Fix legacy apkg import not setting csum

Also test for this in the roundtrip test.

* Avoid reallocating MediaCopier's buffer

* Make sha1 optional (dae)
2022-07-22 17:50:15 +10:00
RumovZ
b9fd6688d2
Fix field check for cloze conditional (#1975) 2022-07-20 20:26:26 +10:00
RumovZ
5f9fa51026
Fix missing limits field in tests (#1973) 2022-07-19 18:51:32 +10:00
RumovZ
cc929687ae
Deck-specific Limits (#1955)
* Add deck-specific limits to DeckNormal

* Add deck-specific limits to schema11

* Add DeckLimitsDialog

* deck_limits_qt6.py needs to be a symlink

* Clear duplicate deck setting keys on downgrade

* Export deck limits when exporting with scheduling

* Revert "deck_limits_qt6.py needs to be a symlink"

This reverts commit 4ee7be1e10c4e8c49bb20de3bf45ac18b5e2d4f6.

* Revert "Add DeckLimitsDialog"

This reverts commit eb0e2a62d33df0b518d9204a27b09e97966ce82a.

* Add day limits to DeckNormal

* Add deck and day limits mock to deck options

* Revert "Add deck and day limits mock to deck options"

This reverts commit 0775814989e8cb486483d06727b1af266bb4513a.

* Add Tabs component for daily limits

* Add borders to tabs component

* Revert "Add borders to tabs component"

This reverts commit aaaf5538932540f944d92725c63bb04cfe97ea14.

* Implement tabbed limits properly

* Add comment to translations

* Update rslib/src/decks/limits.rs

Co-authored-by: Damien Elmes <dae@users.noreply.github.com>

* Fix camel case in clear_other_duplicates()

* day_limit → current_limit

* Also import day limits

* Remember last used day limits

* Add day limits to schema 11

* Tweak comment (dae)

* Exclude day limit in export (dae)

* Tweak tab wording (dae)

* Update preset limits on preset change

* Explain tabs in tooltip (dae)

* Omit deck and today limits if v2 is enabled

* Preserve deck limit when switching to today limit
2022-07-19 18:27:25 +10:00
Matthias Metelka
d1ba48bc48
Expose cloze text as HTML attribute on question side (#1968)
* Expose cloze text as attribute on front side

* Update test_models.py

* Update template_filters.rs

* Escape HTML for data-attribute

* Use minimal HTML encoding in Rust

to match Python's html.escape and pass tests.

* Rename attribute to data-cloze

to make it more generic.

* Run formatter

* Revert to using Rust encode_attribute and add helper function for tests
2022-07-19 04:22:57 +10:00
RumovZ
f6390e9455
Adjust remaining steps after config update (#1956)
* Adjust remaining steps after config update

* Handle relearning steps separately

Also refactor a lot.

* Also adjust remaining steps after deck change

* Test step adjustment after config update

* Fix `SearchBuilder::(re)learning_cards()`

* Fix step adjustment after deck change

* Test step adjustment after deck change

* Fix test name

* Readjust remaining steps according to last delay

Also atomize tests and add some tooling.
2022-07-14 11:24:34 +10:00
RumovZ
8c515e316e
Template err improvements (#1953)
* Throw error for unknown condition fields as well

So if 'foo' is not a field, refuse to save a template containing
`{{#foo}}bar{{/foo}}`. Previously, only `{{foo}}` would be checked.
As a side effect, templates which *only* contain fields as conditions
may be saved. Meh.

* Display template errors in q/a columns only

So the affected browser row remains active and the user can fix the
template more easily.

* Specify if error occured in a browser template

* Minor wording tweak (dae)

There's an argument for using the exact wording as well, but this just
reads a little more naturally to me.
2022-07-09 13:00:03 +10:00
Sam Penny
9dcceff4af
Remember previous choices in reposition dialog (#1950)
* remember previous choices in reposition dialog

* remember previous choice for randomize option as well

* fix failing test
2022-07-08 11:28:38 +10:00
RumovZ
2bbd1fca2a
Skip card generation if fronts remain unchanged (#1949)
Closes #1945
2022-07-06 19:26:14 +10:00
RumovZ
bebc6fd23a
Disregard manual reschedulings in introduced:x (#1932) 2022-06-27 17:20:36 +10:00
Damien Elmes
5ea78e1c8e Since DupeResolution is in CsvMetadata, we don't need to pass it separately
Follow-up to #1930
2022-06-27 17:15:54 +10:00
Damien Elmes
67e4edcd8b Expose backend_proto publicly for AnkiDroid, and rename to pb
We were aliasing it on import half the time anyway
2022-06-27 15:27:53 +10:00
Damien Elmes
6ff159304d Fix backups failing on Android 2022-06-27 15:27:51 +10:00
RumovZ
b6a7760cb4
Restore and save last dupe resolution setting (#1930)
* Restore dupe resolution setting

* Save dupe resolution setting

* Push config logic into backend (dae)
2022-06-24 15:10:06 +10:00
RumovZ
8478492190
Check ids when gathering data (#1928)
This will throw an error if a card, note or revlog id from the future
is found during apkg import or export.
2022-06-24 13:56:52 +10:00
RumovZ
1b8bab73d1
Csv import tweaks (#1920)
* Don't use special label for tags column

This was setting the label of whichever column the user chose for tags
to "Tags". Showing field content is more helpful.

* Map tags column automatically like fields
2022-06-22 09:44:12 +10:00
Damien Elmes
4d51ee8a64 Fix number of days in a (standard) year
https://forums.ankiweb.net/t/prop-ivl-faulty/20665
2022-06-13 09:09:27 +10:00
Damien Elmes
f15e294981 Fix collection load failure on Android
Loading a larger collection with V16 enabled was failing in openCollection
with error 6410, similar to https://github.com/simolus3/drift/issues/876
2022-06-10 23:21:35 +10:00
RumovZ
6da5e5b042
CSV import/export fixes and features (#1898)
* Fix footer moving upwards

* Fix column detection

Was broken because escaped line breaks were not considered.
Also removes delimiter detection on `#columns:` line. User must use tabs
or set delimiter beforehand.

* Add CSV preview

* Parse `#tags column:`

* Optionally export deck and notetype with CSV

* Avoid clones in CSV export

* Prevent bottom of page appearing under footer (dae)

* Increase padding to 1em (dae)

With 0.5em, when a vertical scrollbar is shown, it sits right next to
the right edge of the content, making it look like there's no right
margin.

* Experimental changes to make table fit+scroll (dae)

- limit individual cells to 15em, and show ellipses when truncated
- limit total table width to body width, so that inner table is shown
with scrollbar
- use class rather than id - ids are bad practice in Svelte components,
as more than one may be displayed on a single page

* Skip importing foreign notes with filtered decks

Were implicitly imported into the default deck before.
Also some refactoring to fetch deck ids and names beforehand.

* Hide spacer below hidden field mapping

* Fix guid being replaced when updating note

* Fix dupe identity check

Canonify tags before checking if dupe is identical, but only add update
tags later if appropriate.

* Fix deck export for notes with missing card 1

* Fix note lines starting with `#`

csv crate doesn't support escaping a leading comment char. :(

* Support import/export of guids

* Strip HTML from preview rows

* Fix initially set deck if current is filtered

* Make isHtml toggle reactive

* Fix `html_to_text_line()` stripping sound names

* Tweak export option labels

* Switch to patched rust-csv fork

Fixes writing lines starting with `#`, so revert 5ece10ad05f331.

* List column options with first column field

* Fix flag for exports with HTML stripped
2022-06-09 10:28:01 +10:00
Damien Elmes
59ee399c5f Update regex crate for CVE 2022-06-07 08:39:07 +10:00
RumovZ
42cbe42f06
Plaintext import/export (#1850)
* Add crate csv

* Add start of csv importing on backend

* Add Menomosyne serializer

* Add csv and json importing on backend

* Add plaintext importing on frontend

* Add csv metadata extraction on backend

* Add csv importing with GUI

* Fix missing dfa file in build

Added compile_data_attr, then re-ran cargo/update.py.

* Don't use doubly buffered reader in csv

* Escape HTML entities if CSV is not HTML

Also use name 'is_html' consistently.

* Use decimal number as foreign ease (like '2.5')

* ForeignCard.ivl → ForeignCard.interval

* Only allow fixed set of CSV delimiters

* Map timestamp of ForeignCard to native due time

* Don't trim CSV records

* Document use of empty strings for defaults

* Avoid creating CardGenContexts for every note

This requires CardGenContext to be generic, so it works both with an
owned and borrowed notetype.

* Show all accepted file types  in import file picker

* Add import_json_file()

* factor → ease_factor

* delimter_from_value → delimiter_from_value

* Map columns to fields, not the other way around

* Fallback to current config for csv metadata

* Add start of new import csv screen

* Temporary fix for compilation issue on Linux/Mac

* Disable jest bazel action for import-csv

Jest fails with an error code if no tests are available, but this would
not be noticable on Windows as Jest is not run there.

* Fix field mapping issue

* Revert "Temporary fix for compilation issue on Linux/Mac"

This reverts commit 21f8a261408cdae49ec031aa21a1b659c4f66d82.

* Add HtmlSwitch and move Switch to components

* Fix spacing and make selectors consistent

* Fix shortcut tooltip

* Place import button at the top with path

* Fix meta column indices

* Remove NotetypeForString

* Fix queue and type of foreign cards

* Support different dupe resolution strategies

* Allow dupe resolution selection when importing CSV

* Test import of unnormalized text

Close  #1863.

* Fix logging of foreign notes

* Implement CSV exports

* Use db_scalar() in notes_table_len()

* Rework CSV metadata

- Notetypes and decks are either defined by a global id or by a column.
- If a notetype id is provided, its field map must also be specified.
- If a notetype column is provided, fields are now mapped by index
instead of name at import time. So the first non-meta column is used for
the first field of every note, regardless of notetype. This makes
importing easier and should improve compatiblity with files without a
notetype column.
- Ensure first field can be mapped to a column.
- Meta columns must be defined as `#[meta name]:[column index]` instead
of in the `#columns` tag.
- Column labels contain the raw names defined by the file and must be
prettified by the frontend.

* Adjust frontend to new backend column mapping

* Add force flags for is_html and delimiter

* Detect if CSV is HTML by field content

* Update dupe resolution labels

* Simplify selectors

* Fix coalescence of oneofs in TS

* Disable meta columns from selection

Plus a lot of refactoring.

* Make import button stick to the bottom

* Write delimiter and html flag into csv

* Refetch field map after notetype change

* Fix log labels for csv import

* Log notes whose deck/notetype was missing

* Fix hiding of empty log queues

* Implement adding tags to all notes of a csv

* Fix dupe resolution not being set in log

* Implement adding tags to updated notes of a csv

* Check first note field is not empty

* Temporary fix for build on Linux/Mac

* Fix inverted html check (dae)

* Remove unused ftl string

* Delimiter → Separator

* Remove commented-out line

* Don't accept .json files

* Tweak tag ftl strings

* Remove redundant blur call

* Strip sound and add spaces in csv export

* Export HTML by default

* Fix unset deck in Mnemosyne import

Also accept both numbers and strings for notetypes and decks in JSON.

* Make DupeResolution::Update the default

* Fix missing dot in extension

* Make column indices 1-based

* Remove StickContainer from TagEditor

Fixes line breaking, border and z index on ImportCsvPage.

* Assign different key combos to tag editors

* Log all updated duplicates

Add a log field for the true number of found notes.

* Show identical notes as skipped

* Split tag-editor into separate ts module (dae)

* Add progress for CSV export

* Add progress for text import

* Tidy-ups after tag-editor split (dae)

- import-csv no longer depends on editor
- remove some commented lines
2022-06-01 20:26:16 +10:00
dobefore
b4cffe2339
rust impl stdError trait (#1890)
* impl stdError trait

* Update CONTRIBUTORS
2022-05-31 17:52:12 +10:00
Damien Elmes
5d835ae813 Always generate missing cards when saving card templates
https://forums.ankiweb.net/t/anki-doesnt-generate-cards-after-un-doing-selective-card-generation/20283
2022-05-26 12:44:57 +10:00
Damien Elmes
82196753ec Rework display of available cards in custom study
In v3, it's more informative to show the count of child decks separately,
since increasing the limit of the current deck does not increase the limits
of child decks. When we rework the decks list in the future, a tooltip
will hopefully provide an easier way for users to see where cards are
available, and where limits are being applied.

Closes #1868
2022-05-20 17:52:02 +10:00
RumovZ
6c807c1b2e
Fix review queue if limit is reached (#1855) 2022-05-10 12:11:35 +10:00
Damien Elmes
d946e5ddd5
Fix for crash with invalid dates on Windows (#1837)
https://forums.ankiweb.net/t/bug-report-crashing-when-opening-deck-browser/19768

Caused by a note mtime that was 1000x larger than it should have been.
Check DB will now fix this case (but there are others it still does not
cover, such as invalid card/note IDs).

https://docs.rs/chrono/0.4.19/x86_64-pc-windows-msvc/src/chrono/sys/windows.rs.html#128
2022-05-07 10:30:23 +10:00
Damien Elmes
c8a4e5ea22 Add more progress + abort points to export code
The old `media_files_did_export` hook has been kept around for use with
the legacy apkg exporter (an add-on uses it), and a new 
`legacy_export_progress` hook has been added so we can get progress
from the new colpkg exporter until we move over fully to the new code.
2022-05-06 15:32:23 +10:00
RumovZ
99c4ec93d2
Fix some apkg import issues (#1836)
* Allow gathering of orphaned decks

* Skip redundant media checking in apkg import
2022-05-06 09:22:15 +10:00
Damien Elmes
ac1cbaf719 Fix implicit transaction not being cleared on transact() failure
"release foo" clears the implicit outer transaction, but "rollback to foo"
does not.
2022-05-05 13:39:41 +10:00
Damien Elmes
1297beff63 Minor progress tweaks to apkg importing
- Set progress as soon as possible; previously the extracting and
gathering steps happened prior to the first progress.
- Add "extracting" and "gathering" progress steps, so that large imports
show more feedback in the early stage, and can be more quickly
interrupted.
2022-05-05 12:23:05 +10:00
RumovZ
5f9451f547
Add apkg import/export on backend (#1743)
* Add apkg export on backend

* Filter out missing media-paths at write time

* Make TagMatcher::new() infallible

* Gather export data instead of copying directly

* Revert changes to rslib/src/tags/

* Reuse filename_is_safe/check_filename_safe()

* Accept func to produce MediaIter in export_apkg()

* Only store file folder once in MediaIter

* Use temporary tables for gathering

export_apkg() now accepts a search instead of a deck id. Decks are
gathered according to the matched notes' cards.

* Use schedule_as_new() to reset cards

* ExportData → ExchangeData

* Ignore ascii case when filtering system tags

* search_notes_cards_into_table →

search_cards_of_notes_into_table

* Start on apkg importing on backend

* Fix due dates in days for apkg export

* Refactor import-export/package

- Move media and meta code into appropriate modules.
- Normalize/check for normalization when deserializing media entries.

* Add SafeMediaEntry for deserialized MediaEntries

* Prepare media based on checksums

- Ensure all existing media files are hashed.
- Hash incoming files during preparation to detect conflicts.
- Uniquify names of conflicting files with hash (not notetype id).
- Mark media files as used while importing notes.
- Finally copy used media.

* Handle encoding in `replace_media_refs()`

* Add trait to keep down cow boilerplate

* Add notetypes immediately instaed of preparing

* Move target_col into Context

* Add notes immediately instaed of preparing

* Note id, not guid of conflicting notes

* Add import_decks()

* decks_configs → deck_configs

* Add import_deck_configs()

* Add import_cards(), import_revlog()

* Use dyn instead of generic for media_fn

Otherwise, would have to pass None with type annotation in the default
case.

* Fix signature of import_apkg()

* Fix search_cards_of_notes_into_table()

* Test new functions in text.rs

* Add roundtrip test for apkg (stub)

* Keep source id of imported cards (or skip)

* Keep source ids of imported revlog (or skip)

* Try to keep source ids of imported notes

* Make adding notetype with id undoable

* Wrap apkg import in transaction

* Keep source ids of imported deck configs (or skip)

* Handle card due dates and original due/did

* Fix importing cards/revlog

Card ids are manually uniquified.

* Factor out card importing

* Refactor card and revlog importing

* Factor out card importing

Also handle missing parents .

* Factor out note importing

* Factor out media importing

* Maybe upgrade scheduler of apkg

* Fix parent deck gathering

* Unconditionally import static media

* Fix deck importing edge cases

Test those edge cases, and add some global test helpers.

* Test note importing

* Let import_apkg() take a progress func

* Expand roundtrip apkg test

* Use fat pointer to avoid propogating generics

* Fix progress_fn type

* Expose apkg export/import on backend

* Return note log when importing apkg

* Fix archived collection name on apkg import

* Add CollectionOpWithBackendProgress

* Fix wrong Interrupted Exception being checked

* Add ClosedCollectionOp

* Add note ids to log and strip HTML

* Update progress when checking incoming media too

* Conditionally enable new importing in GUI

* Fix all_checksums() for media import

Entries of deleted files are nulled, not removed.

* Make apkg exporting on backend abortable

* Return number of notes imported from apkg

* Fix exception printing for QueryOp as well

* Add QueryOpWithBackendProgress

Also support backend exporting progress.

* Expose new apkg and colpkg exporting

* Open transaction in insert_data()

Was slowing down exporting by several orders of magnitude.

* Handle zstd-compressed apkg

* Add legacy arg to ExportAnkiPackage

Currently not exposed on the frontend

* Remove unused import in proto file

* Add symlink for typechecking of import_export_pb2

* Avoid kwargs in pb message creation, so typechecking is not lost

Protobuf's behaviour is rather subtle and I had to dig through the docs
to figure it out: set a field on a submessage to automatically assign 
the submessage to the parent, or call SetInParent() to persist a default
version of the field you specified.

* Avoid re-exporting protobuf msgs we only use internally

* Stop after one test failure

mypy often fails much faster than pylint

* Avoid an extra allocation when extracting media checksums

* Update progress after prepare_media() finishes

Otherwise the bulk of the import ends up being shown as "Checked: 0"
in the progress window.

* Show progress of note imports

Note import is the slowest part, so showing progress here makes the UI
feel more responsive.

* Reset filtered decks at import time

Before this change, filtered decks exported with scheduling remained
filtered on import, and maybe_remove_from_filtered_deck() moved cards
into them as their home deck, leading to errors during review.

We may still want to provide a way to preserve filtered decks on import,
but to do that we'll need to ensure we don't rewrite the home decks of
cards, and we'll need to ensure the home decks are included as part of
the import (or give an error if they're not).

https://github.com/ankitects/anki/pull/1743/files#r839346423

* Fix a corner-case where due dates were shifted by a day

This issue existed in the old Python code as well. We need to include
the user's UTC offset in the exported file, or days_elapsed falls back
on the v1 cutoff calculation, which may be a day earlier or later than
the v2 calculation.

* Log conflicting note in remapped nt case

* take_fields() → into_fields()

* Alias `[u8; 20]` with `Sha1Hash`

* Truncate logged fields

* Rework apkg note import tests

- Use macros for more helpful errors.
- Split monolith into unit tests.
- Fix some unknown error with the previous test along the way.
(Was failing after 969484de4388d225c9f17d94534b3ba0094c3568.)

* Fix sorting of imported decks

Also adjust the test, so it fails without the patch. It was only passing
before, because the parent deck happened to come before the
inconsistently capitalised child alphabetically. But we want all parent
decks to be imported before their child decks, so their children can
adopt their capitalisation.

* target[_id]s → existing_card[_id]s

* export_collection_extracting_media() → ...

export_into_collection_file()

* target_already_exists→card_ordinal_already_exists

* Add search_cards_of_notes_into_table.sql

* Imrove type of apkg export selector/limit

* Remove redundant call to mod_schema()

* Parent tooltips to mw

* Fix a crash when truncating note text

String::truncate() is a bit of a footgun, and I've hit this before
too :-)

* Remove ExportLimit in favour of separate classes

* Remove OpWithBackendProgress and ClosedCollectionOp

Backend progress logic is now in ProgressManager. QueryOp can be used
for running on closed collection.

Also fix aborting of colpkg exports, which slipped through in #1817.

* Tidy up import log

* Avoid QDialog.exec()

* Default to excluding scheuling for deck list deck

* Use IncrementalProgress in whole import_export code

* Compare checksums when importing colpkgs

* Avoid registering changes if hashes are not needed

* ImportProgress::Collection → ImportProgress::File

* Make downgrading apkgs depend on meta version

* Generalise IncrementableProgress

And use it in entire import_export code instead.

* Fix type complexity lint

* Take count_map for IncrementableProgress::get_inner

* Replace import/export env with Shift click

* Accept all args from update() for backend progress

* Pass fields of ProgressUpdate explicitly

* Move update_interval into IncrementableProgress

* Outsource incrementing into Incrementor

* Mutate ProgressUpdate in progress_update callback

* Switch import/export legacy toggle to profile setting

Shift would have been nice, but the existing shortcuts complicate things.
If the user triggers an import with ctrl+shift+i, shift is unlikely to
have been released by the time our code runs, meaning the user accidentally
triggers the new code. We could potentially wait a while before bringing
up the dialog, but then we're forced to guess at how long it will take the
user to release the key.

One alternative would be to use alt instead of shift, but then we need to
trigger our shortcut when that key is pressed as well, and it could
potentially cause a conflict with an add-on that already uses that
combination.

* Show extension in export dialog

* Continue to provide separate options for schema 11+18 colpkg export

* Default to colpkg export when using File>Export

* Improve appearance of combo boxes when switching between apkg/colpkg

+ Deal with long deck names

* Convert newlines to spaces when showing fields from import

Ensures each imported note appears on a separate line

* Don't separate total note count from the other summary lines

This may come down to personal preference, but I feel the other counts
are equally as important, and separating them feels like it makes it
a bit easier to ignore them.

* Fix 'deck not normal' error when importing a filtered deck for the 2nd time

* Fix [Identical] being shown on first import

* Revert "Continue to provide separate options for schema 11+18 colpkg export"

This reverts commit 8f0b2c175f4794d642823b60414d142a12768441.

Will use a different approach

* Move legacy support into a separate exporter option; add to apkg export

* Adjust 'too new' message to also apply to .apkg import case

* Show a better message when attempting to import new apkg into old code

Previously the user could end seeing a message like:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte

Unfortunately we can't retroactively fix this for older clients.

* Hide legacy support option in older exporting screen

* Reflect change from paths to fnames in type & name

* Make imported decks normal at once

Then skip special casing in update_deck(). Also skip updating
description if new one is empty.

Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-05-02 21:12:46 +10:00
Damien Elmes
d2ec004ac3 Fix custom study applying previously-selected limit
extend_limits() mutates deck, so we need to re-fetch it.

Closes #1829
2022-05-02 20:57:29 +10:00
Abdo
a3d9a25d18
Extends the SearchNode interface to support field and unqualified searches (#1821)
* Add field search option to SearchNode

* Add unqualified/literal search option to SearchNode

* Rename text to literal_text

* Interpret field name parameter literally
2022-04-28 14:09:51 +10:00
RumovZ
8ccadd96e6
Store last position when scheduling as review (#1800) 2022-04-14 18:53:58 +10:00
Damien Elmes
51a0641bba Sort field change without field change should mark schema modified
Closes #1789
2022-04-10 08:52:55 +10:00
RumovZ
4a6767b1fc
Implicitly group when joining searches (#1759)
* Implicitly group when joining searches

* Allow joining search types directly

* Test search joining

* Add comment for future selves (dae)

* Add one more assert that shows nested grouping (dae)

* Join user searches without grouping again

* Flatten a few clauses in custom study (dae)
2022-04-09 13:22:27 +10:00
Abdo
964f0a5763
Add relative overdueness to review order (#1757)
* Add relative overdueness to review order

* Add test for relative overdue
2022-04-09 13:20:09 +10:00
Damien Elmes
4673fb5fe4 Colpkg import now marks collection as modified
Addresses a corner case where a backup is imported into an
unchanged collection, resulting in the sync indicator showing a full
sync is required, which doesn't go away as syncing thinks no changes
are required.

An alternative way to solve this would be to reverse the order of checks
in the syncing code. It would have the advantage of retaining the
modification time of the backup, but any action like clicking on a deck
would cause it to be updated anyway, so I'm not sure that buys us much.
2022-04-03 16:07:39 +10:00
Damien Elmes
d90608198f Don't rely on frontend to cap time taken in v3 2022-04-02 20:06:23 +10:00
RumovZ
dc36a08f3a
Also restore/keep position of new cards (#1760)
* Also restore/keep position of new cards

* Refactor Card::last_position()
2022-03-31 14:31:13 +10:00
Damien Elmes
9d64afc7bc Handle gaps in media in colpkg imports
Our old Python code was also skipping numbers when it encountered a
directory, leading to a colpkg that couldn't be imported with our new
code.
2022-03-31 13:32:38 +10:00
RumovZ
ab57fa484b
Fix SearchBuilder (#1754)
* Fix missing search grouping

* Fix SearchBuilder::or_join

* Unify search concatenations
2022-03-29 10:53:02 +10:00
RumovZ
f1488b5983
Card type error (#1749)
* TemplateSaveError -> CardTypeError

* Don't show success tooltip if export fails

* Attach help page to error

Show help link if export fails due to card type error.

* Add type (dae)

* Add shared show_exception() (dae)

- Use a shared routine for printing standard backend errors, so that
we can take advantage of the help links in eg. the card layout screen
as well.
- The truthiness check on help in showInfo() would have ignored the
enum 0 value.
- Close the exporting dialog on a documented failure as well

* Fix local variable help_page
2022-03-28 22:17:50 +10:00
RumovZ
dd16890c11
Add Deleted error and disable all bad browser rows (#1742)
* Add Deleted error and disable all bad browser rows

* Avoid error when opening the browse screen to a card with a missing note (dae)

* In cards mode, a missing note is NotFound, not Deleted (dae)

So we distinguish between referential integrity error, and explicit
deletion.

* Remove redundant try block
2022-03-28 19:06:19 +10:00
Damien Elmes
bbcab21f13 Start random card position from 1
Anki's DB schema unfortunately uses odid=0 instead of null to indicate
a lack of an original due date, so having a due position of 0 leads to
the temporary due date not being reset when the card is removed from
a filtered deck.

https://forums.ankiweb.net/t/anki-2-1-50-beta-6-9-stable-release/18181/52
2022-03-23 22:00:25 +10:00
Damien Elmes
6735b23e5b Force a schema change on colpkg import
If we don't force a full sync when restoring, any items that were added
since the backup may have already been sent to AnkiWeb, and they
won't have deletion records. After the user restores from a backup,
they'll end up in a state where their local and AnkiWeb collections
differ, and the changes will not sync. The old backup code forced a schema
change, but we weren't previously doing it via File>Import.
2022-03-22 13:10:32 +10:00
Damien Elmes
4515c41d2c
Backup improvements (#1728)
* Collection needs to be closed prior to backup even when not downgrading

* Backups -> BackupLimits

* Some improvements to backup_task

- backup_inner now returns the error instead of logging it, so that
the frontend can discover the issue when they await a backup (or create
another one)
- start_backup() was acquiring backup_task twice, and if another thread
started a backup between the two locks, the task could have been accidentally
overwritten without awaiting it

* Backups no longer require a collection close

- Instead of closing the collection, we ensure there is no active
transaction, and flush the WAL to disk. This means the undo history
is no longer lost on backup, which will be particularly useful if we
add a periodic backup in the future.
- Because a close is no longer required, backups are now achieved with
a separate command, instead of being included in CloseCollection().
- Full sync no longer requires an extra close+reopen step, and we now
wait for the backup to complete before proceeding.
- Create a backup before 'check db'

* Add File>Create Backup

https://forums.ankiweb.net/t/anki-mac-os-no-backup-on-sync/6157

* Defer checkpoint until we know we need it

When running periodic backups on a timer, we don't want to be fsync()ing
unnecessarily.

* Skip backup if modification time has not changed

We don't want the user leaving Anki open overnight, and coming back
to lots of identical backups.

* Periodic backups

Creates an automatic backup every 30 minutes if the collection has been
modified.

If there's a legacy checkpoint active, tries again 5 minutes later.

* Switch to a user-configurable backup duration

CreateBackup() now uses a simple force argument to determine whether
the user's limits should be respected or not, and only potentially
destructive ops (full download, check DB) override the user's configured
limit.

I considered having a separate limit for collection close and automatic
backups (eg keeping the previous 5 minute limit for collection close),
but that had two downsides:

- When the user closes their collection at the end of the day, they'd
get a recent backup. When they open the collection the next day, it
would get backed up again within 5 minutes, even though not much had
changed.
- Multiple limits are harder to communicate to users in the UI

Some remaining decisions I wasn't 100% sure about:

- If force is true but the collection has not been modified, the backup
will be skipped. If the user manually deleted their backups without
closing Anki, they wouldn't get a new one if the mtime hadn't changed.
- Force takes preference over the configured backup interval - should
we be ignored the user here, or take no backups at all?

Did a sneaky edit of the existing ftl string, as it hasn't been live
long.

* Move maybe_backup() into Collection

* Use a single method for manual and periodic backups

When manually creating a backup via the File menu, we no longer make
the user wait until the backup completes. As we continue waiting for
the backup in the background, if any errors occur, the user will get
notified about it fairly quickly.

* Show message to user if backup was skipped due to no changes

+ Don't incorrectly assert a backup will be created on force

* Add "automatic" to description

* Ensure we backup prior to importing colpkg if collection open

The backup doesn't happen when invoked from 'open backup' in the profile
screen, which matches Anki's previous behaviour. The user could
potentially clobber up to 30 minutes of their work if they exited to
the profile screen and restored a backup, but the alternative is we
create backups every time a backup is restored, which may happen a number
of times if the user is trying various ones. Or we could go back to a
separate throttle amount for this case, at the cost of more complexity.

* Remove the 0 special case on backup interval; minimum of 5 minutes

https://github.com/ankitects/anki/pull/1728#discussion_r830876833
2022-03-21 19:40:42 +10:00
Damien Elmes
e2eff7deae Fix change notetype affecting cards outside selection
Accidentally introduced by the search refactoring in #1600, this lead
to a query that matched items outside of the selected notes, eg

(n.id in (1506029488152) and c.ord = 43 or c.ord = 4)

Closes #1727
2022-03-20 15:36:54 +10:00
Damien Elmes
d100f7a2c8 Don't fsync media files on import
I was seeing import speeds of only 10-20 files a second before this
change.
2022-03-19 23:31:11 +10:00
RumovZ
16fe18d033
Refactor export-import code and resolve fixmes (#1723)
* Write media files in chunks

* Test media file writing

* Add iter `ReadDirFiles`

* Remove ImportMediaError, fail fatally instead

Partially reverts commit f8ed4d89ba.

* Compare hashes of media files to be restored

* Improve `MediaCopier::copy()`

* Restore media files atomically with tempfile

* Make downgrade flag an enum

* Remove SchemaVersion::Latest in favour of Option

* Remove sha1 comparison again

* Remove unnecessary repr(u8) (dae)
2022-03-18 19:31:55 +10:00
RumovZ
5781e86995
Don't propogate limit extensions in v3 scheduler (#1724) 2022-03-18 19:16:31 +10:00
Damien Elmes
5c1abb7f09 Add missing file header 2022-03-17 21:09:20 +10:00
Damien Elmes
dadb69a14b Fix some clippy lints + imports 2022-03-17 21:03:39 +10:00
Damien Elmes
bf8e70c70f Ensure partial colpkg file removed if export fails 2022-03-17 20:58:36 +10:00
Damien Elmes
f8ed4d89ba Add new error variant to frontend; ensure errors are mapped appropriately 2022-03-17 20:58:35 +10:00
Damien Elmes
30e53c51ab Tiny cleanup missing from previous commit
Had intended to apply it prior to pushing.
2022-03-17 17:49:18 +10:00
Damien Elmes
4687620f5e Expand normalization checks on import/export
The old Python code was only checking for NFC encoding, but we should
check for other issues like special filenames on windows (eg con.mp3)

- On export, the user is told to use Check Media if their media has
invalid filenames.
- On import, legacy packages will be transparently normalized. Since we're
doing the checks on export as well, any invalid names in a v3 package
are an error.
2022-03-17 17:31:19 +10:00
Damien Elmes
57a4495d92 Check for attempted path traversal on import 2022-03-17 16:40:52 +10:00
Damien Elmes
c2e8d89fc6
Colpkg fixes (#1722)
* Fix legacy colpkg import; disable v3 import/export; add roundtrip test

The test has revealed we weren't decompressing the media files on v3
import. That's easy to fix, but means all files need decompressing
even when they already exist, which is not ideal - it would be better
to store size/checksum in the metadata instead.

* Switch media and meta to protobuf; re-enable v3 import/export

- Fixed media not being decompressed on import
- The uncompressed size and checksum is now included for each media
entry, so that we can quickly check if a given file needs to be extracted.
We're still just doing a naive size comparison on colpkg import at the
moment, but we may want to use a checksum in the future, and will need
a checksum for apkg imports.
- Checksums can't be efficiently encoded in JSON, so the media list
has been switched to protobuf to reduce the the space requirements.
- The meta file has been switched to protobuf as well, for consistency.
This will mean any colpkg files exported with beta7 will be
unreadable.

* Avoid integer version comparisons

* Re-enable v3 test

* Apply suggestions from code review

Co-authored-by: RumovZ <gp5glkw78@relay.firefox.com>

* Add export_colpkg() method to Collection

More discoverable, and easier to call from unit tests

* Split import/export code out into separate folders

Currently colpkg/*.rs contain some routines that will be useful for
apkg import/export as well; in the future we can refactor them into a
separate file in the parent module.

* Return a proper error when media import fails

This tripped me up when writing the earlier unit test - I had called
the equivalent of import_colpkg()?, and it was returning a string error
that I didn't notice. In practice this should result in the same text
being shown in the UI, but just skips the tooltip.

* Automatically create media folder on import

* Move roundtrip test into separate file; check collection too

* Remove zstd version suffix

Prevents a warning shown each time Rust Analyzer is used to check the
code.

Co-authored-by: RumovZ <gp5glkw78@relay.firefox.com>
2022-03-17 15:11:23 +10:00
Damien Elmes
ed9476c856 Update Rust deps
Primarily for https://blog.rust-lang.org/2022/03/08/cve-2022-24713.html
2022-03-15 16:51:52 +10:00
RumovZ
e759885734
Backend colpkg exporting (#1719)
* Implement colpkg exporting on backend

* Use exporting logic in backup.rs

* Refactor exporting.rs

* Add backend function to export collection

* Refactor backend/collection.rs

* Use backend for colpkg exporting

* Don't use default zip compression for media

* Add exporting progress

* Refactor media file writing

* Write dummy collections

* Localize dummy collection note

* Minimize dummy db size

* Use `NamedTempFile::new()` instead of `new_in`

* Drop redundant v2 dummy collection

* COLLECTION_VERSION -> PACKAGE_VERSION

* Split `lock_collection()` into two to drop flag

* Expose new colpkg in GUI

* Improve dummy collection message

* Please type checker

* importing-colpkg-too-new -> exporting-...

* Compress the media map in the v3 package (dae)

On collections with lots of media, it can grow into megabytes.

Also return an error in extract_media_file_names(), instead of masking
it as an optional.

* Store media map as a vector in the v3 package (dae)

This compresses better (eg 280kb original, 100kb hashmap, 42kb vec)

In the colpkg import case we don't need random access. When importing
an apkg, we will need to be able to fetch file data for a given media
filename, but the existing map doesn't help us there, as we need
filename->index, not index->filename.

* Ensure folders in the media dir don't break the file mapping (dae)
2022-03-15 16:48:02 +10:00
Damien Elmes
2d6dd0630f Switch card.original_position to a proto3 optional 2022-03-10 20:48:26 +10:00
Damien Elmes
f3e81c8a95 Move custom study tag and limit gathering+saving into the backend
Ideally this would have been in beta 6 :-) No add-ons appear to be
using customstudy.py/taglimit.py though, so it should hopefully not be
disruptive.

In the earlier custom study changes, we didn't get around to addressing
issue #1136. Now instead of trying to determine the maximum increase
to allow (which doesn't work correctly with nested decks), we just
present the total available to the user again, and let them decide. There's
plenty of room for improvement here still, but further work here might
be better done once we look into decoupling deck limits from deck presets.

Tags and available cards are fetched prior to showing the dialog now,
and will show a progress dialog if things take a while.

Tags are stored in an aux var now, so they don't inflate the deck
object size.
2022-03-10 16:23:03 +10:00
RumovZ
b9c3b12f71
Optionally restore original position and reset counts when forgetting (#1714)
* Add forget prompt with options

- Restore original position
- Reset reps and lapses

* Restore position when resetting for export

* Add config context to avoid passing keys

* Add routine to fetch defaults; use method-specific enum (dae)

* Keep original position by default (dae)

* Fix code completion for forget dialog (dae)

Needs to be a symbolic link to the generated file
2022-03-09 16:51:41 +10:00
RumovZ
c21e6e2b97
Disable full screen on Windows with OpenGL (#1715) 2022-03-09 14:21:54 +10:00
Damien Elmes
e6028cd9a4 Preserve old translation
7f076ee92a..9f37a0245c (r820447434)
2022-03-07 18:14:12 +10:00
RumovZ
f3c8857421
Backups (#1685)
* Add zstd dep

* Implement backend backup with zstd

* Implement backup thinning

* Write backup meta

* Use new file ending anki21b

* Asynchronously backup on collection close in Rust

* Revert "Add zstd dep"

This reverts commit 3fcb2141d2be15f907269d13275c41971431385c.

* Add zstd again

* Take backup col path from col struct

* Fix formatting

* Implement backup restoring on backend

* Normalize restored media file names

* Refactor `extract_legacy_data()`

A bit cumbersome due to borrowing rules.

* Refactor

* Make thinning calendar-based and gradual

* Consider last kept backups of previous stages

* Import full apkgs and colpkgs with backend

* Expose new backup settings

* Test `BackupThinner` and make it deterministic

* Mark backup_path when closing optional

* Delete leaky timer

* Add progress updates for restoring media

* Write restored collection to tempfile first

* Do collection compression in the background thread

This has us currently storing an uncompressed and compressed copy of
the collection in memory (not ideal), but means the collection can be
closed without waiting for compression to complete. On a large collection,
this takes a close and reopen from about 0.55s to about 0.07s. The old
backup code for comparison: about 0.35s for compression off, about
8.5s for zip compression.

* Use multithreading in zstd compression

On my system, this reduces the compression time of a large collection
from about 0.55s to 0.08s.

* Stream compressed collection data into zip file

* Tweak backup explanation

+ Fix incorrect tab order for ignore accents option

* Decouple restoring backup and full import

In the first case, no profile is opened, unless the new collection
succeeds to load.
In the second case, either the old collection is reloaded or the new one
is loaded.

* Fix number gap in Progress message

* Don't revert backup when media fails but report it

* Tweak error flow

* Remove native BackupLimits enum

* Fix type annotation

* Add thinning test for whole year

* Satisfy linter

* Await async backup to finish

* Move restart disclaimer out of backup tab

Should be visible regardless of the current tab.

* Write restored collection in chunks

* Refactor

* Write media in chunks and refactor

* Log error if removing file fails

* join_backup_task -> await_backup_completion

* Refactor backup.rs

* Refactor backup meta and collection extraction

* Fix wrong error being returned

* Call sync_all() on new collection

* Add ImportError

* Store logger in Backend, instead of creating one on demand

init_backend() accepts a Logger rather than a log file, to allow other
callers to customize the logger if they wish.

In the future we may want to explore using the tracing crate as an
alternative; it's a bit more ergonomic, as a logger doesn't need to be
passed around, and it plays more nicely with async code.

* Sync file contents prior to rename; sync folder after rename.

* Limit backup creation to once per 30 min

* Use zstd::stream::copy_decode

* Make importing abortable

* Don't revert if backup media is aborted

* Set throttle implicitly

* Change force flag to minimum_backup_interval

* Don't attempt to open folders on Windows

* Join last backup thread before starting new one

Also refactor.

* Disable auto sync and backup when restoring again

* Force backup on full download

* Include the reason why a media file import failed, and the file path

- Introduce a FileIoError that contains a string representation of
the underlying I/O error, and an associated path. There are a few
places in the code where we're currently manually including the filename
in a custom error message, and this is a step towards a more consistent
approach (but we may be better served with a more general approach in
the future similar to Anyhow's .context())
- Move the error message into importing.ftl, as it's a bit neater
when error messages live in the same file as the rest of the messages
associated with some functionality.

* Fix importing of media files

* Minor wording tweaks

* Save an allocation

I18n strings with replacements are already strings, so we can skip the
extra allocation. Not that it matters here at all.

* Terminate import if file missing from archive

If a third-party tool is creating invalid archives, the user should know
about it. This should be rare, so I did not attempt to make it
translatable.

* Skip multithreaded compression on small collections

Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-03-07 15:11:31 +10:00
Damien Elmes
864904729b Fix an intermittent failure in one of the Rust unit tests 2022-03-07 14:18:17 +10:00
RumovZ
5426164ebf
Add regex tag search (#1707) 2022-03-04 18:43:27 +10:00
Damien Elmes
2c80bcb571 Update to latest rules_rust, which unblocks the tokio upgrade
Rust version has been pinned, as 1.59 gets stuck compiling pyo3_macros,
and appears to have other issues:
https://github.com/rust-lang/rust/issues/94390
2022-03-03 19:48:17 +10:00
RumovZ
508b5ab947
Remove top_deck_id arg in deck_tree() (#1702)
Counts don't propogate correctly anymore (#1678).
2022-03-02 15:30:32 +10:00
Damien Elmes
6941bccde4 Add support for proto3 optional scalars
Protobuf 3.15 introduced support for marking scalar fields like
uint32 as optional, and all of our tooling appears to support it
now. This allows us to use simple optional/null checks in our Rust/
TypeScript code, without having to resort to an inner message.

I had to apply a minor patch to protobufjs to get this working with
the json-module output; this has also been submitted upstream:
https://github.com/protobufjs/protobuf.js/pull/1693

I've modified CardStatsResponse as an example of the new syntax.

One thing to note: while the Rust and TypeScript bindings use optional/
null fields, as that is the norm in those languages, Google's Python
bindings are not very Pythonic. Referencing an optional field that is
missing will yield the default value, and a separate HasField() call
is required, eg:

```
>>> from anki.stats_pb2 import CardStatsResponse as R
... msg = R.FromString(b"")
... print(msg.first_review)
... print(msg.HasField("first_review"))
0
False
```
2022-02-27 19:42:06 +10:00
Damien Elmes
89bce6d829 Update Rust deps; pin Tokio
Tokio has had to be pinned, because the 1.17 release introduces
a dependency on windows_sys, which fails to build on Windows on
Bazel.

The issue appears to be the build script of a subcrate - it is using
CARGO_MANIFEST_DIR to update the linking path so windows.lib can be
found (it's contained in that crate), but the path is set incorrectly.

dfc25285a2/crates/targets/x86_64_msvc/build.rs

One way we might be able to work around it is to add to the link path
in our own build script.
2022-02-25 17:10:47 +10:00
Damien Elmes
d52c36e920 Make flag changes undoable again
The previous change in 1871b57663 failed
to consider the browser refreshing case, as reported here:
https://forums.ankiweb.net/t/anki-2-1-50-beta-3-4/17501/30

I previously attempted to solve this by having SetFlag skip the queue
rebuild, then mutating the captured mtimes in the queues. That didn't
work correctly when undoing, as the queue mutations weren't recorded.
This approach combines that attempt and the previous change: flag
setting is an undoable operation again, but does not change the card's
modification time, so it can be applied/undone without a queue build
being required. Instead of special-casing flag changes in the review
screen, we now just redraw the flag on changes.card, as any other card
op will have triggered a queue rebuild.
2022-02-25 15:06:07 +10:00
RumovZ
a4d61fe7d9
Original position (#1677)
* Replace Card.data with .original_position

* Use and update original position in v3

* Show original position in card info

* Revert restoring original position for now

* Fix pb card to/from pylib card

* Try original_position as the last pb field

* minor wording tweaks (dae)
2022-02-22 22:48:21 +10:00
RumovZ
131a94dd85
Config for burying interday learning cards (#1680)
* Add config for burying interday learning cards

* Expose bury interday learning config in GUI
2022-02-22 21:37:59 +10:00
Abdo
fd6a78e7cf
Add option to ignore accents in search by default (#1667)
* Add option to ignore accents in unqualified search

* Support ignoring accents in word boundary search

* Update ftl/core/preferences.ftl
2022-02-17 16:30:52 +10:00
Damien Elmes
1871b57663 don't put flag changes in the undo queue, and don't bump mtime
This is not ideal, but I struggled to come up with a better solution.

Background:

- The scheduler records the mtime of cards as it's building the queues,
and will throw an error in get_queued_cards() if the card on the DB
has a different mtime. This is to catch bugs - any operation that modifies
cards should be triggering a queue rebuild, or should adjust the queues
appropriately.
- The review screen skips the usual queue rebuild redraw, and directly
updates the flag icon. This is because a rebuild could cause a different
card to appear, or the answer side to switch back to the question side,
neither of which the user expects when they flag a card.

The current behaviour was broken: the queue rebuilding was still happening
on the backend, and the frontend was just failing to reflect it.

I initially tried to special-case Op::SetFlag, having it skip the queue
rebuild, and having set_card_flag() update the mtimes in the active
queue. But those mutations weren't captured by the undo log, so they
didn't get undone when undoing the set flag operation. We could perhaps
work around it by adding a separate undo entry to capture the mutation,
but it started to feel like it would be a pain to maintain moving forward.

By skipping the undo queue and retaining the same mtime, no queue
rebuild is required. Because we're setting usn, the cards will still
sync, but as mtime is not bumped, in the case of a conflict, an older
unsynced change from another client may revert the flag change.

Fixes https://forums.ankiweb.net/t/anki-2-1-50-beta-1-2/15608/145
2022-02-13 16:27:33 +10:00
RumovZ
6a21261c84
Round calculated Hard days (#1661)
* Round calculated Hard days

* clarify docstring (dae)
2022-02-13 13:37:52 +10:00
RumovZ
d55f080733
V3 parent limits (#1638)
* avoid repinning Rust deps by default

* add id_tree dependency

* Respect intermediate child limits in v3

* Test new behaviour of v3 counts

* Rework v3 queue building to respect parent limits

* Add missing did field to SQL query

* Fix `LimitTreeMap::is_exhausted()`

* Rework tree building logic

https://github.com/ankitects/anki/pull/1638#discussion_r798328734

* Add timer for build_queues()

* `is_exhausted()` -> `limit_reached()`

* Move context and limits into `QueueBuilder`

This allows for moving more logic into QueueBuilder, so less passing
around of arguments. Unfortunately, some tests will require additional
work to set up.

* Fix stop condition in new_cards_by_position

* Fix order gather order of new cards by deck

* Add scheduler/queue/builder/burying.rs

* Fix bad tree due to unsorted child decks

* Fix comment

* Fix `cap_new_to_review_rec()`

* Add test for new card gathering

* Always sort `child_decks()`

* Fix deck removal in `cap_new_to_review_rec()`

* Fix sibling ordering in new card gathering

* Remove limits for deck total count with children

* Add random gather order

* Remove bad sibling order handling

All routines ensure ascending order now.
Also do some other minor refactoring.

* Remove queue truncating

All routines stop now as soon as the root limit is reached.

* Move deck fetching into `QueueBuilder::new()`

* Rework new card gather and sort options

https://github.com/ankitects/anki/pull/1638#issuecomment-1032173013

* Disable new sort order choices ...

depending on set gather order.

* Use enum instead of numbers

* Ensure valid sort order setting

* Update new gather and sort order tooltips

* Warn about random insertion order with v3

* Revert "Add timer for build_queues()"

This reverts commit c9f5fc6ebe87953c17a0c842990b009b5596c69c.

* Update rslib/src/storage/card/mod.rs (dae)

* minor wording tweaks to the tooltips (dae)

+ move legacy strings to bottom
+ consistent capitalization (our leech action still needs fixing,
but that will require introducing a new 'suspend card' string as the
existing one is used elsewhere as well)
2022-02-10 09:55:43 +10:00
RumovZ
4909302147
Intersperse more evenly, mainly at start and end (#1651)
Closes #1645.
2022-02-09 12:45:37 +10:00
RumovZ
872b6df22a
Optimise searching in (all) fields (#1622)
* Avoid rebuilding regex in field search

* Special case search in all fields

* Don't repeat mid nodes in field search sql

Small speed gain for searches like `*:re:foo` and reduces the sql tree
depth if a lot of field names of the same notetype match.

* Add sql function to match fields with regex

* Optimise used field search algorithm

- Searching in all fields is a special case.
- Using native SQL comparison is preferred.
- For Regex, use newly added SQL function.

* Please clippy

* Avoid pyramid of doom

* nt_fields -> matched_fields

* Add tests for regex and all field searches

* minor tweaks for readability (dae)
2022-01-24 20:30:08 +10:00
RumovZ
d4e3cc7a0a
Remove into_node_list() (#1609) 2022-01-21 21:23:52 +10:00
RumovZ
9aca778a93
Backend Custom Study (#1600)
* Implement custom study on backend

* Switch frontend to backend custom study

* Skip typecheck for new pb classes

* Build tag search string on backend

Also fixes escaping of special characters in tag names.

* `cram.cards` -> `cram.card_limit`

* Assign more meaningful names in `TagLimit`

* Broaden rustfmt glob

* Use `invalid_input()` helper

* Assign `FilteredDeckForUpdate` to temp var

* Implement `SearchBuilder`

* Rewrite `custom_study()` with `SearchBuilder`

* Replace match macros with `SearchBuilder`

* Remove `into_nodes_list` & `concatenate_searches`
2022-01-20 14:25:22 +10:00
RumovZ
1dae2f42ff
Make note: and card: search require full match (#1606) 2022-01-20 11:57:16 +10:00
RumovZ
77b1546583
Remove redundant error mapping (#1593)
Helpful regex error in Find&Replace was obscured by generic
"Invalid input" error.
2022-01-16 13:46:27 +10:00
Damien Elmes
bbc191711d update Rust deps 2022-01-15 14:59:43 +10:00
Abdo
aaaea03c72
Fix new preview card's position being interpreted as a date (#1577)
* Fix new preview card's position being interpreted as a date

Can be reproduced by opening the Card Info screen of a new preview card
not answered yet.

* Update rslib/src/stats/card.rs
2022-01-06 19:52:54 +10:00
Damien Elmes
693672d7a3 remove extraneous to_vec() call 2021-12-24 13:09:19 +10:00
Damien Elmes
b7907741ec fix SQL formatting issue 2021-12-24 12:42:42 +10:00
Damien Elmes
8ed1fa7e99 fix interday learning cards not being buried
https://forums.ankiweb.net/t/2-1-49-mac-automatic-unbury-still-not-working/14799/43
2021-12-24 12:25:16 +10:00
Damien Elmes
4d431fb7af move linkchecker into separate crate
The feature-based approach didn't work with cargo-raze, leading
to ./update.py in cargo/ breaking.
2021-12-20 17:27:43 +10:00
RumovZ
3e0c9dc866
New TTS/AV tag handling (#1559)
* Add new `card_rendering` mod

Parses a text with av/tts tags and strips or extracts tags.

* Replace old `extract_av_tags` and `strip_av_tags`

... with new `card_rendering` mod

* ressource -> resource

* Add AV prettifier for use in browser table

* Accept String in av tag routines

... and avoid redundant writes if no changes need to be made.

* add benchmarking with criterion; make links test optional (dae)

cargo install cargo-criterion, then run ./bench.sh

* performance comparison: creating HashMap up front (dae)

the previous solution:

anki_tag_parse          time:   [1.8401 us 1.8437 us 1.8476 us]

this solution:

anki_tag_parse          time:   [2.2420 us 2.2447 us 2.2477 us]
                        change: [+21.477% +21.770% +22.066%] (p = 0.00 < 0.05)
                        Performance has regressed.

* Revert "performance comparison: creating HashMap up front" (dae)

This reverts commit f19126a2f15b729b825825a49283f63ab13474d0.

* add missing header

* Write error message if tts lang is missing

* `Tag` -> `Directive`
2021-12-17 19:04:42 +10:00
RumovZ
2221d0a520
Make Hard use current step's interval if it's not the first one (#1561)
* Make hard repeat the current step's interval in v3

Unless for the first step to avoid identical interval with Again.

* Make Hard repeat the current step's interval in v2

* Adjust test to new Hard behaviour
2021-12-16 22:02:13 +10:00
RumovZ
80ed94ed08
Cap steps (#1548)
* Fix steps being mistaken for seconds

* Cap steps at `u32::max` seconds

* Fix overflow of steps in Rust

* Prevent overflow of `IntervalKind`

* Prevent overflow in `revlod/mod.rs`

Also replace some `as` with `from` and `try_from` as is recommended to
highlight potential issues.

* Ensure v2 doesn't store overflowing revlog ivls

* Lower steps cap in deck options

Whereas large card intervals are converted to days, revlog intervals use
i32s to store large numbers of seconds.

* Format
2021-12-15 18:46:26 +10:00
Damien Elmes
8d90b6b061 run buildifier/buildozer to tidy up BUILD files 2021-12-14 09:18:24 +10:00
Damien Elmes
0c821859c5 check for missed dbg!() statements
This works in our main code, but doesn't catch dbg!() statements in
our build scripts.
2021-12-14 09:05:14 +10:00
Damien Elmes
afff4fc437 order by template after note id in filtered decks
This brings the behaviour a bit closer to the default ordering of new
cards when they are reset, and is better than an undefined template
order. But it's a stopgap solution, and in the long run, filtered decks
need a bit of a rethink with the improved ordering than v3 has brought.
2021-12-10 18:53:50 +10:00
Damien Elmes
d515939340 fix error when gathering new cards in reverse position
Also simplify order clause - with did and queue limited to a constant,
SQLite is smart enough to use the covering index for sorting by due.
2021-12-06 17:08:00 +10:00
Damien Elmes
c2ab0de34f update Rust deps 2021-12-03 20:35:53 +10:00
Damien Elmes
67ee6f9c0e update to Rust 1.57 + latest rules_rust 2021-12-03 20:35:52 +10:00
Damien Elmes
8de3eaea65 fix Clippy lints in Rust 1.57 2021-12-03 19:53:37 +10:00
Damien Elmes
d7f5555642 deck_tree() could be made to unbury the current day's cards
When provided with a future timestamp to estimate the next day's cards,
it lead to the current day's buried cards being unburied.

Fixes
https://forums.ankiweb.net/t/inconsistent-card-counts-when-syncing/15496
2021-12-02 21:07:21 +10:00
Damien Elmes
9daa0c9acb fix new cards not appearing in correct order in v3
This was broken by an SQLite upgrade - previously we received the rows
in ix_cards_sched order, but recent versions use a table scan for that
query when the order is unspecified. Solved by being explicit about the
order we expect results to arrive.

https://forums.ankiweb.net/t/skipping-new-cards/15410
2021-12-02 17:20:48 +10:00
Matthias Metelka
68092082f2
Change Notetype UI Rework (#1499)
* Enable access to old notetype name

* Set minimum height for ChangeNotetypeDialog

* Add bootstrap icons to change-notetype

* Move alert up and make it collapsible

* Tweak some CSS

- Add variables --sticky-bg and --sticky-border to StickyContainer
- Tweak base.css

* Add translatable string "(Nothing)"

* Rework ChangeNotetype screen

* Initially load option at newIndex and remaining options on focus

Optimization for big notetypes:
Should increase efficiency from O(n²) to O(n). Test on notetype with 500 templates shows significant improvement in load time (~10s down to ~1s).

* Try to satisfy rust test

* Change arrow direction depending on reading direction

+ add 0.5em top padding to main

* Create Alert.svelte

* Introduce CSS variable --pane-bg

* Revert "Initially load option at newIndex and remaining options on focus"

This reverts commit f42beee45c27dba9433d76217fb583b117fb5231.

* Final cleanup

* Refine padding/gutter
2021-11-24 12:09:55 +10:00
Henrik Giesel
2778b9220c
Mathjax editor improvements (#1502)
* Remove unnecessary stopPropagation of mathjax-overlay events

* Use CodeMirror component for MathjaxHandle

* Refactor ResizeObserver code in MathjaxHandle

* Wrap setRange in CodeMirror in try/catch

* Add Mathjax Editor bottom margin

* Add custom Enter and Shift+Enter shortcuts for the MathjaxHandle

* Format

* Move placeCaretAfter to domlib

* Move focus back to field after editing Mathjax

* Put Cursor after Mathjax after accepting

* Add delete button for Mathjax

* Change border color of mathjax menu

* Refactor into MathjaxMenu

* Put caretKeyword in variable

* Use one ResizeObserver for all Mathjax images

* Add minmimum width for Mathjax editor

* is still smaller than minimal window width

* Add bazel directories to .prettierignore and format from root

* exclude ftl/usage (dae)

the json files that live there are output from our tooling, and
formatting them means an extra step each time we want to update them

also exclude .mypy_cache, which is output by scripts/mypy*

* minor ftl tweak: newline -> new line  (dae)
2021-11-23 10:27:32 +10:00
Damien Elmes
11042a16ce Fix latest Rusqlite on Windows
https://github.com/bazelbuild/rules_rust/pull/1025
2021-11-19 10:44:02 +10:00
Damien Elmes
8b3d964866 roll rusqlite back due to Windows breakage 2021-11-18 21:39:23 +10:00
Damien Elmes
d31dccc388 update Rust deps 2021-11-18 20:58:41 +10:00
Damien Elmes
26123ce229 update to edition 2021 2021-11-18 20:51:10 +10:00
RumovZ
72b08cb7d5
Smooth fuzz (#1493)
* Add separate `fuzz.rs`

* Switch to a smoother fuzz calculation
2021-11-17 07:23:19 +10:00
Damien Elmes
13382f7f87 avoid fuzzing until interval reaches 3 days
e5e47a31fe (r748827327)

+ switched assert_lower_middle_upper to a macro, so that when it fails,
the reported line number is the original call site, instead of one inside
the helper function
2021-11-15 15:48:58 +10:00
RumovZ
b1142c12d9
Fix constrained_fuzz_bounds() (#1490)
... for cases where the entire fuzz range is above `maximum`.
Also improve hanling if the entire range is below `minimum` and
readability.
2021-11-15 15:41:43 +10:00
Damien Elmes
311e70822d fix a link anchor 2021-11-14 11:44:54 +10:00
Damien Elmes
5f05eeb922 unbury when refreshing queues
While we already unbury when refreshing the deck list, if the user
resumes study on a new day without refreshing the deck list, burying
could end up being delayed.

Possible fix for https://forums.ankiweb.net/t/buried-cards-in-ankimobile-beta-20081-3/14753/3
2021-11-14 10:06:47 +10:00
Damien Elmes
044d253306 fix underflow in fuzz code, leading to large intervals
https://forums.ankiweb.net/t/buried-cards-in-ankimobile-beta-20081-3/14753
2021-11-14 09:17:37 +10:00
RumovZ
0efa3f944f
Garbage collect unused Fluent strings (#1482)
* Canonify import of i18n module

Should always be imported as `tr`, or `tr2` if there is a name collision
(Svelte).

* Add helper for garbage collecting ftl strings

Also add a serializer for ftl asts.

* Add helper for filter-mapping `DirEntry`s

* Fix `i18n_helpers/BUILD.bazel`

* run cargo-raze

* Refactor `garbage_collection.rs`

- Improve helper for file iterating
- Remove unused terms as well
- Fix issue with checking for nested messages by switching to a regex-
based approach (which runs before deleting)
- Some more refactorings and lint fixes

* Fix lints in `serialize.rs`

* Write json pretty and sorted

* Update `serialize.rs` and fix header

* Fix doc and remove `dbg!`

* Add binaries for ftl garbage collection

Also relax type constraints and strip debug tests.

* add rust_binary targets for i18n helpers (dae)

* add scripts to update desktop usage/garbage collect (dae)

Since we've already diverged from 2.1.49, we won't gain anything
from generating a stable json just yet. But once 2.1.50 is released,
we should run 'ftl/update-desktop-usage.sh stable'.

* add keys from AnkiMobile (dae)

* Mention caveats in `remove-unused.sh`
2021-11-12 18:19:01 +10:00
Damien Elmes
cb07b232d8 add a builder for Collection 2021-11-06 14:43:41 +10:00
RumovZ
283776d8e7
Rework v3 fuzzing (#1474)
* Remove flooring in v3 scheduler code

It is no longer supposed to be an exact port of the old Python code.

* Rework v3 fuzzing

https://github.com/ankitects/anki/issues/1416#issuecomment-958208149

* Ensure length of fuzz range is larger than 1

Only for new intervals larger than 1 and respecting max review interval.

* add the beginnings of a unit test

* Clarify `fuzz_factor` doc string

* Fix Python tests for 2021 scheduler

* Fix fuzz test

1.0 is not a valid fuzz factor.

* Add tests for fuzzing in Rust

* Use range notation in fuzz factor doc

* Strip redundant tests
2021-11-06 10:39:24 +10:00
Matthias Metelka
371f731e30
Editor Field Descriptions (#1476)
* Add description input to fields dialog

QLineEdit seems like the best option, as it saves space and motivates users to keep their descriptions concise.

* Add setDescriptions to note initialization script

Went for the extra function instead of including it in setFields to prevent potential add-on breakages.

* Add tooltip next to field name if description is set

* Refactor code according to suggestions

Set default tooltip placement to right instead of bottom

Use .get() for fld["description"]

Fix tab order in fields dialog

Swap out abbreviation "desc" for full length name to keep consistency

* Update Protobuf and Rust for description

Add description to notetypes.proto and schema11

Co-authored-by: RumovZ <RumovZ@users.noreply.github.com>

* Fix tooltips not updating with description

Remove redundant variable tooltipOptions

Update previousTooltip within reactive function

* Move LabelDescription out of LabelName

Co-authored-by: Henrik Giesel <hgiesel@users.noreply.github.com>

* Decrease icon size and fix alignment

Co-Authored-By: Henrik Giesel <hengiesel@gmail.com>

* the new key needs to be cleared from fields, not the notetype itself

Co-authored-by: RumovZ <RumovZ@users.noreply.github.com>
Co-authored-by: Henrik Giesel <hengiesel@gmail.com>
Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2021-11-06 09:42:48 +10:00
Damien Elmes
3843a4509e fix sort order toggling
We're getting an enum instead of an int in Qt6

normal/reversed have been renamed to ascending/descending; no add-ons
appear to be using the old versions.
2021-10-30 09:14:43 +10:00
RumovZ
16ad0137f7
Use backend for getting deck children (#1465)
* Add backend routine for child deck names and ids

* Use backend for getting deck children

Instead of flawed case-sensitive Python logic.
2021-10-29 19:43:17 +10:00
Damien Elmes
0e00c4a461 fix new cards not being correctly limited
https://forums.ankiweb.net/t/ios-beta-20080-2-more-new-cards-after-review-limit-is-met/13728/10
2021-10-29 12:12:34 +10:00
RumovZ
7e23c9fe4f
Fix quotation of "and" and "or" in search (#1463) 2021-10-28 19:23:56 +10:00
RumovZ
3ab9712c18
Stop trimming filename references before encoding (#1462)
Closes  #1430
2021-10-28 19:22:51 +10:00
Damien Elmes
e30411e0b2 update Rust deps with cargo-raze 0.13.0 2021-10-26 08:16:40 +10:00
RumovZ
3cdb3d72c1
Do not bury suspended cards (#1447)
* Skip burying for suspended cards

* Inform about number of buried cards
2021-10-23 11:04:26 +10:00
RumovZ
1c9b5a2e83
Card info cleanup (#1446)
* Cast proto interface to type ...

... instead of using non-null assertions in Revlog.svelte.

* Remove OptionalInt32 and OptionalUInt32
2021-10-23 11:00:43 +10:00
RumovZ
56d8402f89
Fix underflow of learning count (#1444)
`counts.learning` includes interday learning cards, so it is not
suitable to determine how many cards from the (intraday!) learning queue
are already included in the learning count when updating it.
2021-10-22 20:58:06 +10:00
Damien Elmes
61f3b71664 fix a clippy lint in 1.56 2021-10-22 12:03:54 +10:00
Damien Elmes
3d416b4a68 convert sql_format to ts_project; move into separate folder 2021-10-18 19:39:45 +10:00
Damien Elmes
a29bd7c9f0 unbury when rendering deck list
We were previously relying on the sched_timing_today() call in the
backend, but v3 doesn't call it, leading to cards remaining buried.
2021-10-16 21:11:09 +10:00
RumovZ
3672b0fe73
Switch CardInfoDialog to ts page (#1414)
* Only collect card stats on the backend ...

... instead of rendering an HTML string using askama.

* Add ts page Card Info

* Update test for new `col.card_stats()`

* Remove obsolete CardStats code

* Use new ts page in `CardInfoDialog`

* Align start and end instead of left and right

Curiously, `text-align: start` does not work for `th` tags if assigned
via classes.

* Adopt ts refactorings after rebase

#1405 and #1409

* Clean up `ts/card-info/BUILD.bazel`

* Port card info logic from Rust to TS

* Move repeated field to the top

https://github.com/ankitects/anki/pull/1414#discussion_r725402730

* Convert pseudo classes to interfaces

* CardInfoPage -> CardInfo

* Make revlog in card info optional

* Add legacy support for old card stats

* Check for undefined instead of falsy

* Make Revlog separate component

* drop askama dependency (dae)

* Fix nightmode for legacy card stats
2021-10-14 19:22:47 +10:00
Damien Elmes
ca32a46594 handle AnkiDroid's two element new intervals
https://github.com/ankidroid/Anki-Android/issues/8889
2021-10-07 23:15:51 +10:00
Damien Elmes
944b064e54 update Rust deps 2021-10-02 20:42:03 +10:00
Damien Elmes
8e56853c50 fix a clippy lint in latest Rust 2021-10-02 14:13:49 +10:00
Damien Elmes
0bb273a0ed replace the old stripHTML() methods with the backend implementation
Python's regex engine performs pathologically on regexes like
'<!--.*?-->' when fed a large string of repeating '<!--' clauses.
Thanks to JaimeSlome / security@huntr.dev for the report; closes #1380.

Solved by switching to the Rust implementation, which does not suffer
from this issue.

entsToText(), minimizeHTML(), and the old regex constants have been
removed; they do not appear to be used by any add-ons.
2021-10-01 23:15:45 +10:00
Damien Elmes
a174c41801
Merge pull request #1397 from RumovZ/column-tooltips
Add tooltips for some browser columns
2021-10-01 19:12:59 +10:00
Damien Elmes
3940b0587d avoid extra allocation in replacement 2021-10-01 14:43:38 +10:00
RumovZ
ee2ecd0700 Add tooltips for some browser columns 2021-09-30 13:15:09 +02:00
abdo
018594c5fb Strip isolation chars from filenames in Check Media screen 2021-09-27 22:15:37 +03:00
Damien Elmes
9daf037c0b
Merge pull request #1264 from hgiesel/tageditor2
Svelte Tag Editor v2
2021-09-15 20:21:16 +10:00
Damien Elmes
70a367137d handle filtered case when repositioning 2021-09-13 14:57:41 +10:00
Damien Elmes
6296e93c21 check for invalid collections in full_upload() 2021-09-13 11:50:13 +10:00
Damien Elmes
6fb6ffa1d7 tentative fix for learning count underflow
The 'avoid showing learning card twice' logic is now only applied
when the next learning card was already due to be shown. This'll mean
there will be cases where a learning card does get shown twice near
the end, but it makes the behaviour easier to reason about, for both
us and end users.
2021-09-13 11:08:55 +10:00
Damien Elmes
c9d3e5462e bikeshedding: amount -> match_limit
+ convert from u32 in backend method
2021-09-12 11:57:49 +10:00
Henrik Giesel
e85c93f3e7 Cap tag suggestions at 500 2021-09-10 01:13:50 +02:00
Damien Elmes
4da1c77220 add basic tag completion to backend
Matches should arrive in alphabetical order. Currently results are not
capped (JS should be able to handle ~1k tags without too much hassle),
and no reordering based on match location is done. Matches are substring
based, and multiple can be provided, eg "foo::bar" will match
"foof::baz::abbar".

This is not hooked up properly on the frontend at the moment -
updateSuggestions() seems to be missing the most recently typed character,
and is not updating the list of completions half the time.
2021-09-09 15:38:08 +02:00
Damien Elmes
a961013961 fix case of fields table 2021-09-07 10:22:22 +10:00
Damien Elmes
9a27e44ff8 don't panic when invalid note id provided to change notetype
https://forums.ankiweb.net/t/fatal-error-while-updating-deck/12743/2
2021-08-29 14:00:55 +10:00
Damien Elmes
dffa38c479 expose uncapped counts and total in tree node
In the future this information could be revealed on hover, to make it
clearer how limits are being applied.
2021-08-22 15:32:46 +10:00
Damien Elmes
05124eecf7 adding options to sort by ascending/descending ease 2021-08-22 15:32:46 +10:00
Damien Elmes
62223499c4 fix errors when undoing/redoing after a queue-invalidating operation
There were a few issues going on here:

- If some operation had invalidated the queues, they were subsequently
recreated with a call to .get_queues() in the undo handling code. This
could happen after the changes to the card had already been reverted,
leading to a queue state that didn't match our expectations.
- More generally, it's not safe to assume our mutations will apply
cleanly after the queue has been rebuilt. The next card will vary
depending on the number of remaining cards when interspersing cards of
different types, and a queue-invalidating operation will have changed
the learning cutoff.

So rather than rebuilding the queues on demand, we now check that they
already exist, and were created at the time we expect. If not, we
invalidate them and skip applying the mutations, and a subsequent
refresh of the UI should rebuild the queues correctly.

As part of this change, the cutoff snapshot has been moved into the
normal answer update object.

One possible downside here is that adding a note during review may cause
a newly due learning card to appear when undoing a different review.
If this proves to be a problem, we could potentially note down the
learning cutoff and apply it when queues are rebuilt later.
2021-08-22 15:32:46 +10:00
Damien Elmes
b9402b5c47 support limiting interday learning cards by review limit again
Context: https://forums.ankiweb.net/t/more-cards-today-question-about-v3/12400/10

Previously, interday learning cards and reviews were gathered at the
same time in v3, with the review limit being applied to both of them. The
order cards were gathered in would change the ratio of gathered learning
cards and reviews, but as they were displayed together in a single count,
a changing ratio was not apparent, and no special handling was required
by the deck tree code.

Showing interday learning cards in the learning count, while still
applying a review limit to them, makes things more complicated, as
a changing ratio will result in different counts. The deck tree code
is not able to know which order cards will appear in, so without changes,
we would have had a situation where the deck list may show different counts
to those seen when clicking on a deck.

One way to solve this would have been to introduce a separate limit for
interday learning cards. But this would have meant users needed to
juggle two different limits, instead of having a single one that controls
total number of (non-intraday) cards shown.

Instead, the scheduler now fetches interday cards prior to reviews -
the rationale for that order is that learning cards tend to be more
fragile/urgent than reviews. The option to show learning cards
before/after/mixed with reviews still exists, but it applies only after
cards have been capped to the daily limit.

To ensure the deck tree code matches the counts the scheduler gives,
it too applies limits to interday learning cards first, and reviews
afterwards.
2021-08-22 15:32:46 +10:00
Damien Elmes
6977b6ab6f fix new clippy warning 2021-08-22 15:32:46 +10:00
Damien Elmes
131c8b72f8 add options to v3 to preserve new card gather order
Allows cards to be presented in deck order when gather priority is set
to 'deck'.
2021-08-20 12:03:32 +10:00
Damien Elmes
53fe7e574e handle ampersand entities in image filenames
In the old HTML editor, filenames were % escaped before feeding them to
beautifulsoup, causing bare ampersands to be left alone. The new HTML
editor reads content from the DOM, where a bare ampersand has been
transformed into an &amp;, and that gets saved back into the field,
so the media check now needs to deal with it for images as well.

https://forums.ankiweb.net/t/causing-problems-with-image-names/12171
2021-08-19 23:43:40 +10:00
Damien Elmes
48c121e4f3 filtered decks w/ scheduling disabled in v3 now log reviews 2021-08-19 20:25:29 +10:00
Damien Elmes
8830d33826 revert some interday learning changes in v3
Interday learning cards are now counted in the learning count again,
and are no longer subject to the daily review limit.

The thinking behind the original change was that interday learning cards
are scheduled more like reviews, and counting them in the review count
would allow the learning count to focus on intraday learning - the red
number reflecting the fact that they are the most fragile memories. And
counting them together made it practical to apply the review limit
to both at once.

Since the release, there have been a number of users expecting to see
interday learning cards included in the learning count (the latest being
https://forums.ankiweb.net/t/feedback-and-a-feature-adjustment-request-for-2-1-45/12308),
and a good argument can be made for that too - they are, after all, listed
in the learning steps, and do tend to be harder than reviews. Short of
introducing another count to keep track of interday and intraday learning
separately, moving back to the old behaviour seems like the best move.

This also means it is not really practical to apply the review limit to
interday learning cards anymore, as the limit would be split between two
different numbers, and how much each number is capped would depend on
the order cards are introduced. The scheduler could figure this out, but
the deck list code does not know card order, and would need significant
changes to be able to produce numbers that matched the scheduler. And
even if we ignore implementation complexities, I think it would be more
difficult for users to reason about - the influence of the review limit
on new cards is confusing enough as it is.
2021-08-19 16:40:12 +10:00
Damien Elmes
272a610832 ensure interday learning cards update deck limits
https://forums.ankiweb.net/t/more-cards-today-question-about-v3/12400
2021-08-19 12:50:11 +10:00
Damien Elmes
d333d0da74 allow adjusting tag case in rename
https://forums.ankiweb.net/t/2-1-46-renaming-tags-to-tags-not-working/12426
2021-08-19 11:46:01 +10:00
Damien Elmes
1f4d54efda allow repositioning of new cards while suspended
https://forums.ankiweb.net/t/no-longer-can-reposition-a-suspended-card/12241
2021-08-15 15:06:25 +10:00
Damien Elmes
187944615e check both queues when popping answered card
If multiple answers are applied in succession, a newly-due learning card
may be in front of the next main entry.
2021-08-09 18:29:04 +10:00
abdo
52fac0557a Fix double quotes being escaped twice in dupe search
maybe_quote() already escapes double quotes.
2021-08-09 05:05:20 +03:00
Damien Elmes
6548ebb516 fall back on default deck in v3 queue gather as well
Missed in the previous change.

https://forums.ankiweb.net/t/error-when-trying-to-choose-deck-in-new-anki/11829/11
2021-08-09 11:04:37 +10:00
Damien Elmes
94913ec23f fallback on default deck in congrats screen
https://forums.ankiweb.net/t/error-when-trying-to-choose-deck-in-new-anki/11829
2021-08-04 12:57:46 +10:00
Damien Elmes
746b19e38c fix undoing last learning card corner case 2021-08-04 10:29:01 +10:00
Damien Elmes
7ba35b7249 support updating multiple notes in one transaction/undo op 2021-08-02 17:07:26 +10:00
Damien Elmes
88a3fd8d7b support updating multiple cards in one transaction/undo op 2021-08-02 16:59:02 +10:00
Damien Elmes
cc65196b0a revert to defaults when schema 11 deck config can not be read
Allows decks with missing/null values like ivlFct to be opened.

https://forums.ankiweb.net/t/i-cant-access-ankiweb/11814
2021-08-02 16:27:48 +10:00
Damien Elmes
070f57fcc5 don't hide learning count on congrats screen when learning is overdue
The v3 scheduler will delay the final card from being shown twice in
a row, but the overdue case was being treated the same as the no-learning
case, leading to the message being hidden.
2021-08-02 15:57:09 +10:00
Damien Elmes
3cec1a35dc update learning cutoff when counts are zero 2021-08-02 15:03:14 +10:00
Damien Elmes
94c7c3282b fix learning cutoff not updating during review 2021-08-02 15:03:14 +10:00
Damien Elmes
cf731c6bad fix v3 not honoring initial ease factor
Closes #1317
2021-07-31 14:57:04 +10:00
RumovZ
d090bd7e21 Use existing 'no such field' tr string 2021-07-29 08:30:49 +02:00
RumovZ
d6e5f3c67c Allow duplicate templates if {{Card}} is used 2021-07-28 22:32:38 +02:00
RumovZ
c0dd769090 Skip new notetype checks when importing apkg 2021-07-28 21:46:51 +02:00
RumovZ
be1b524396 Find template errors hidden by conditionals 2021-07-28 12:13:14 +02:00
RumovZ
4d7fcf585a Check for invalid conditionals on templates 2021-07-28 11:53:31 +02:00
Damien Elmes
6fe6570da3 fix reqwest build on Linux again 2021-07-24 10:18:09 +10:00
Damien Elmes
d73852f272 use separate integration test for links
If we run into issues with unreliable network connections in the future,
we'll be able to mark the test as flaky so Bazel can retry it multiple
times.
2021-07-24 10:12:25 +10:00
Damien Elmes
56545db339
Merge pull request #1308 from RumovZ/check-help
Build and check help links on the backend
2021-07-24 09:37:23 +10:00
RumovZ
07752f43cd Fix linkcheck result message 2021-07-23 19:40:51 +02:00
Damien Elmes
b1dedb1b1f disable link check outside CI 2021-07-23 20:22:32 +10:00
Damien Elmes
0578729468 be explicit about .html suffix 2021-07-23 20:22:32 +10:00
Damien Elmes
1ecda8cf4d fix clippy lints 2021-07-23 20:22:32 +10:00
Damien Elmes
379694915e add linkcheck to Bazel 2021-07-23 20:22:32 +10:00
RumovZ
503bdb8c22 Add check to ensure parsed URL equals link 2021-07-23 11:44:19 +02:00
RumovZ
6ef52726da Improve error message for unknown fragment 2021-07-23 10:32:29 +02:00
RumovZ
f2f19e8b45 Remove native HelpPage enum
Also remove oneof from pb enum and handle strs in Python.
2021-07-22 16:32:49 +02:00
RumovZ
55e1f178ef Add links.proto and backend module 2021-07-22 10:05:38 +02:00
RumovZ
80264c3b56 Add links.rs for checking and building links 2021-07-22 10:03:03 +02:00
Damien Elmes
5e10087aae handle notes with missing cards in browser
https://forums.ankiweb.net/t/2-1-45-release-candidate/11362/30
2021-07-22 14:58:57 +10:00
Damien Elmes
c944dd048e strip invalid Unicode chars in media check 2021-07-17 18:30:19 +10:00
Damien Elmes
2f434dd74d fix comment + copy/paste error 2021-07-17 09:02:14 +10:00
Aleksa Sarai
d7caafe91c
scheduler: use deck config's initial ease in set_due_date
Previously we would just use 250% ease for any new card that had no
pre-configured ease, but this will result in decks that have
non-standard ease values to have "set due date" cards in them that don't
match. In order to make this somewhat more efficient, we cache
deckid->ease lookups during this operation.

Ref: <https://forums.ankiweb.net/t/set-due-date-doesnt-obey-default-ease-factor/9184>
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
2021-07-16 12:33:15 +10:00
Damien Elmes
96e6e216a7
Merge pull request #1293 from RumovZ/find-and-replace
Find & Replace
2021-07-16 10:47:16 +10:00
Damien Elmes
bf507cca98 move from Python's URI escaping to IRI escaping in Rust
Should make non-Latin text readable in the HTML editor, without the
breakages reverted in the previous change.
2021-07-16 10:38:00 +10:00
RumovZ
a5193339e7 Rework Find & Replace dialog:
- Add option to affect whole collection
- Allow to open without selection
- Add parameter for presetting field
2021-07-13 16:33:45 +02:00
RumovZ
b148b7b5e0 Fix find & replace for notes without 'field_name'
Distinguish between no 'field_name' passed and 'field_name' not on note.
2021-07-13 16:21:13 +02:00
Damien Elmes
f649f6c92a minor tidyup in protobuf build script 2021-07-12 16:15:38 +10:00
Damien Elmes
9c354f5e6b drop the cloze-in-basic check for now
Unfortunately a popular note taking tool has been misusing cloze
markers in its deck exports. We may want to add this back in the
future, but we'll probably want to start by warning users, to give
people time to adjust.
2021-07-11 20:31:02 +10:00
Damien Elmes
0e7411188b configs.proto plural workaround no longer necessary 2021-07-11 19:35:18 +10:00
Damien Elmes
e61a611af7 rename Config in protobuf to avoid conflict with module name
+ use the enum directly, instead of wrapping it in an object

Python code retains the old "Config" name.
2021-07-11 19:27:08 +10:00
Damien Elmes
185e9acd22 split out remaining tags, stats, media and rendering 2021-07-10 23:16:18 +10:00
Damien Elmes
35b059ecdb split out sync, search, scheduler & config 2021-07-10 21:33:12 +10:00
Damien Elmes
9e0a295ab9 split out decks, deckconfig, notes, notetypes 2021-07-10 20:44:22 +10:00
Damien Elmes
18851ace47 split out cards and collection 2021-07-10 19:52:31 +10:00
Damien Elmes
616db33c0e refactor protobuf handling for split/import
In order to split backend.proto into a more manageable size, the protobuf
handling needed to be updated. This took more time than I would have
liked, as each language handles protobuf differently:

- The Python Protobuf code ignores "package" directives, and relies
solely on how the files are laid out on disk. While it would have been
nice to keep the generated files in a private subpackage, Protobuf gets
confused if the files are located in a location that does not match
their original .proto layout, so the old approach of storing them in
_backend/ will not work. They now clutter up pylib/anki instead. I'm
rather annoyed by that, but alternatives seem to be having to add an extra
level to the Protobuf path, making the other languages suffer, or trying
to hack around the issue by munging sys.modules.
- Protobufjs fails to expose packages if they don't start with a capital
letter, despite the fact that lowercase packages are the norm in most
languages :-( This required a patch to fix.
- Rust was the easiest, as Prost is relatively straightforward compared
to Google's tools.

The Protobuf files are now stored in /proto/anki, with a separate package
for each file. I've split backend.proto into a few files as a test, but
the majority of that work is still to come.

The Python Protobuf building is a bit of a hack at the moment, hard-coding
"proto" as the top level folder, but it seems to get the job done for now.

Also changed the workspace name, as there seems to be a number of Bazel
repos moving away from the more awkward reverse DNS naming style.
2021-07-10 19:17:05 +10:00
Damien Elmes
80b98e0db8 move protobuf into separate folder in preparation for multiple files 2021-07-09 21:02:40 +10:00
RumovZ
5067622751 Add pylib/browser.py for literal config keys
Also, remove config bools for sort order.
2021-07-05 12:44:48 +02:00
Henrik Giesel
aacf20531e Add NOTETYPE_FIELD sidebar items 2021-06-30 21:02:23 +02:00
Damien Elmes
79ec9b14b1 zero out graves before comparison, instead of at creation
Will allow us to turn the check back on in the future without a client
update
2021-06-30 10:08:52 +10:00
Damien Elmes
3ee18fb854 ignore graves in sanity check 2021-06-30 09:31:02 +10:00
Damien Elmes
576b141e2b add back in missing check for upload size
Compression now happens up-front, so we can tell in advance if the
upload size has been exceeded.
2021-06-29 10:50:03 +10:00
Damien Elmes
e3c9808b79 catch invalid card ordinals in DB check
https://forums.ankiweb.net/t/error-corrupted-note/10976
2021-06-29 10:50:03 +10:00
Damien Elmes
2a93355824 PEP8 cards.py 2021-06-27 12:12:23 +10:00
Damien Elmes
ca0374782e update to latest rusqlite 2021-06-25 16:22:21 +10:00
Damien Elmes
59e17950ad update most rust deps; skip rusqlite 2021-06-25 15:35:25 +10:00
Damien Elmes
73d9391f64 update undo skipping; exclude deck/tag expand/collapse
Instead of calling a method inside the transaction body, routines
can now pass Op::SkipUndo if they wish the changes to be discarded
at the end of the transaction. The advantage of doing it this way is
that the list of changes can still be returned, allowing the sync
indicator to update immediately.

Closes #1252
2021-06-25 09:16:15 +10:00
Damien Elmes
b693bdbf28 cope with collections where latexsvg is not a bool 2021-06-24 15:29:21 +10:00
Damien Elmes
47284867f9 report object type where json decode fails 2021-06-24 15:28:24 +10:00
Damien Elmes
0b697f5161 expose scheduler js option in deck config 2021-06-24 14:38:10 +10:00
Damien Elmes
114eec6585 allow customization of default search text
https://forums.ankiweb.net/t/whats-the-default-search-when-browsing-now-also-how-do-i-choose-to-browse-current-deck/8922
2021-06-24 11:24:30 +10:00
Damien Elmes
b392020798 fix clippy lints for latest Rust 2021-06-21 13:09:36 +10:00
Damien Elmes
c79f8ba88f in/out -> request/response
The saved characters weren't worth the increased difficulty when
reading, and the fact that we were deviating from protobuf norms.
2021-06-20 15:49:20 +10:00
Damien Elmes
a58826941f add rustls-native-certs
closes #1241
2021-06-18 18:15:41 +10:00
Damien Elmes
2e53dc63c8
Merge pull request #1230 from RumovZ/fields-check
Check for misplaced or missing clozes when adding and in the editor
2021-06-17 21:26:16 +10:00
RumovZ
87a50f22e6 Assign dupe error the lowest precedence 2021-06-16 11:42:40 +02:00
Henrik Giesel
0f658de702 Add escape_anki_wildcards_for_search_node 2021-06-16 09:25:27 +02:00
Henrik Giesel
b8142d33c8 Revert "Remove special treatment of tag:none"
This reverts commit 95285ef66d02d9d3f0d560f70514b16e8a2f7bf5.
2021-06-16 17:19:21 +10:00
Henrik Giesel
b19e07c135 Remove special treatment of tag:none 2021-06-16 17:19:21 +10:00
Henrik Giesel
c5faf39d7c Make Browser root nodes use "_*" uniformly 2021-06-16 17:19:21 +10:00
Damien Elmes
ecdd668f9a update Rust deps 2021-06-16 16:10:57 +10:00
RumovZ
d7340d3f07
Fix missing hyphen escape when normalizing search (#1233)
* Fix missing hyphen escape when normalizing search

* Add some more tests for normalization

* Use char to satisfy clippy
2021-06-15 10:02:39 +10:00