Commit Graph

1647 Commits

Author SHA1 Message Date
Damien Elmes
321402a6d0 Revert to blanking out missed chars in provided text
Quite a few users seem surprised to have missed characters appear in
their input text
2023-02-02 19:13:11 +10:00
Damien Elmes
2c952cf3af Wrap expected text in code block when no answer provided 2023-02-02 18:01:23 +10:00
Kaben Nanlohy
77bba533ea
Allow burying cards in browser (#2351)
* Allow burying cards in browser

This code is based on existing "toggle suspend" command in browser.

- Adds "toggle bury" command to browser cards menu.
- Adds "browsing-toggle-bury" to core translation. Only english-language.
- Adds "buried" coloring to rows for buried cards in browser table.

Not yet done:

- Keyboard shortcut for "toggle bury" action.
- Non-english translations.

* Add contributor as requested in CONTRIBUTORS.

* Fix formatting in browser_table.rs.

* Add keyboard shortcut to "toggle bury" command.

This adds keyboard shortcut "ctrl-shift-j" to "toggle bury" command in
browser cards menu.

* Simplify logic for color of buried-card rows.
2023-01-30 19:21:06 +10:00
Damien Elmes
d20a7d291f Validate and clamp deck config when loading
Fixes #2353
2023-01-28 21:51:47 +10:00
Damien Elmes
17480a2c80 Use HTTP/1.1 for syncing
HTTP2 + a request body size not being declared up front was creating extra
work for AnkiWeb.
2023-01-28 21:51:47 +10:00
Damien Elmes
4142de57e2 Fix clean build failure due to protoc change
da7d4dd2fc changed the name of the env
var in .cargo/config.toml, causing the check in setup_protoc() to think
a custom path had been provided, which skipped the download and extract
step.
2023-01-26 09:33:39 +10:00
Damien Elmes
17b33f8298
Use Rust nightly for formatting (#2348)
* Support specifying a working dir to a build command

* Use nightly for formatting

* Pass valid TERM in from environment

Rustfmt depends on a valid setting, and not just the var to be non-empty.

* Wrap comment
2023-01-25 23:35:53 +10:00
Mani
da7d4dd2fc
Use a ninja variable for Protoc binary (#2345)
* Use a ninja variable for Protoc binary

* fix whitespace
2023-01-23 20:44:47 +10:00
Damien Elmes
dc0185720a Fix outer whitespace not being trimmed in type answer 2023-01-23 17:20:35 +10:00
Damien Elmes
c8275257ce Probable fix for future due graph
Cards due earlier today will have a negative offset like -78000(secs).
The old typescript code was using floating point division to yield -1;
with integer division we get 0 instead.

https://forums.ankiweb.net/t/wrong-info-when-hovering-over-future-due-graph/26522
2023-01-20 00:18:13 +10:00
Damien Elmes
19bbcb4cc0 Use backend for extracting cloze text to type
Closes #2311
2023-01-18 23:05:28 +10:00
Damien Elmes
03814fa1c4 Add some extra info to the 'modified without updating' message
For #2315
2023-01-18 22:38:28 +10:00
Damien Elmes
943dddf28f
Update Rust deps (#2332)
* Temporarily disable hakari

* Upgrade compatible deps except Chrono

* Update semver-incompatible crates

* Re-enable hakari

* Update licenses & cargo-deny

* Fix new clippy lints

* Update to latest Rust
2023-01-18 22:24:29 +10:00
Damien Elmes
ded805b504
Switch Rust import style (#2330)
* Prepare to switch Rust import style

* Run nightly format

Closes #2320

* Clean up a few imports

* Enable comment wrapping

* Wrap comments
2023-01-18 21:39:55 +10:00
Damien Elmes
9d84f357b6 Probable fix for flaky test 2023-01-18 21:32:08 +10:00
Damien Elmes
c923553a53 Replace newlines with linebreaks when importing without HTML
https://forums.ankiweb.net/t/line-breaks-are-removed-when-importing-csv-on-linux/26451
2023-01-18 21:00:33 +10:00
Damien Elmes
cf45cbf429
Rework syncing code, and replace local sync server (#2329)
This PR replaces the existing Python-driven sync server with a new one in Rust.
The new server supports both collection and media syncing, and is compatible
with both the new protocol mentioned below, and older clients. A setting has
been added to the preferences screen to point Anki to a local server, and a
similar setting is likely to come to AnkiMobile soon.

Documentation is available here: <https://docs.ankiweb.net/sync-server.html>

In addition to the new server and refactoring, this PR also makes changes to the
sync protocol. The existing sync protocol places payloads and metadata inside a
multipart POST body, which causes a few headaches:

- Legacy clients build the request in a non-deterministic order, meaning the
entire request needs to be scanned to extract the metadata.
- Reqwest's multipart API directly writes the multipart body, without exposing
the resulting stream to us, making it harder to track the progress of the
transfer. We've been relying on a patched version of reqwest for timeouts,
which is a pain to keep up to date.

To address these issues, the metadata is now sent in a HTTP header, with the
data payload sent directly in the body. Instead of the slower gzip, we now
use zstd. The old timeout handling code has been replaced with a new implementation
that wraps the request and response body streams to track progress, allowing us
to drop the git dependencies for reqwest, hyper-timeout and tokio-io-timeout.

The main other change to the protocol is that one-way syncs no longer need to
downgrade the collection to schema 11 prior to sending.
2023-01-18 12:43:46 +10:00
RumovZ
1be30573e1
Replace dissimilar crate with difflib (#2322)
This also inserts the expected text if it's missing at the very
beginning of the provided text.
2023-01-16 09:49:34 +10:00
RumovZ
6df7be90c5
Fix negated introduced:x search (#2306)
1. Add outer brackets.
2. Coalesce aggregate, because `null and true` is `null` in SQL land,
so cards that were not introduced, but manually rescheduled in the
period of interest, would not show up in a negated search.
2023-01-10 08:49:35 +10:00
Damien Elmes
07fd88ddea Allow timestamps to be a day ahead
https://github.com/ankitects/anki/issues/1895#issuecomment-1374574230
2023-01-09 10:04:48 +10:00
dependabot[bot]
4e92c29815
Bump tokio from 1.23.0 to 1.23.1 (#2303)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.23.0 to 1.23.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.23.0...tokio-1.23.1)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-07 10:01:56 +10:00
Damien Elmes
8206f39bdc Fix sparse today stats
Not safe to assume revlog is ordered by query
2023-01-05 10:14:37 +10:00
Damien Elmes
0eddb25287
Integrate AnkiDroid's backend patches into the repo (#2290)
* Relax chrono specification for AnkiDroid

https://github.com/ankidroid/Anki-Android-Backend/pull/251

* Add AnkiDroid service and AnkiDroid customizations

Most of the work here was done by David in the Backend repo; integrating
it into this repo for ease of future maintenance.

Based on 5d9f262f4c
with some tweaks:

- Protobuf imports have been fixed to match the recent refactor
- FatalError has been renamed to AnkidroidPanicError
- Tweaks to the desktop code to deal with the extra arg to open_collection,
and exclude AnkiDroid service methods from our Python code.

* Refactor AnkiDroid's DB code to avoid uses of unsafe
2023-01-03 13:11:23 +10:00
Damien Elmes
4c5a2461d0 Handle broken utimes() on Android
Closes https://github.com/ankidroid/Anki-Android/issues/12827
Closes https://github.com/ankidroid/Anki-Android/issues/12974
2022-12-28 16:33:06 +10:00
Damien Elmes
a38121a713 Add Debug impl for Collection 2022-12-24 10:44:41 +10:00
Damien Elmes
a82f8d6872 Remove unused before_upload method 2022-12-24 10:44:41 +10:00
Damien Elmes
0570cfdf48 Migrate from slog to tracing
The Rust community appear to have converged on tracing - it's used by
the Rust compiler, and receives close to 10x the number of downloads
that slog does. Its API is more ergonomic, and it does a much nicer
job with async rust.

To make this change, we no longer pass around explicit loggers, and rely
on a globally-registered one. The log file location has been changed
from one in each profile folder to a single one in the base folder. This
will remain empty for most users, since only errors are logged by default,
but may be useful for debugging future changes.
2022-12-24 10:44:40 +10:00
Damien Elmes
d294fff882 Simplify solution for #2279 2022-12-19 18:13:34 +10:00
Damien Elmes
67acaa17e4 Fix panic when adding non-Latin text to fields
Closes #2279
2022-12-19 18:02:43 +10:00
TRIAEIOU
9901ae428a
Nested clozes and increased cloze meta data (#2141)
* Nested clozes and increased cloze meta data

* Update contributors

* This reverts commit 3423df73f89f04a606b1bff3542a68a49ca52e9f.

* Update CONTRIBUTORS

* Formating

* Formating

* Formating

* Formating

* Formating

* Formating

* Formating

* Formating

* Code refactor

* Formating

* Formating

* Formating

* Formating and dead code

* Correct test case

* Remove Hint and Close storage of token string

* Update

* Formating

* Formating

* Formating

* Use write! instead of .push_str(&format).

* Formating
2022-12-19 12:03:15 +10:00
Damien Elmes
0555f4898c Update crate name in docstrings to fix 'cargo test' 2022-12-16 22:30:54 +10:00
Damien Elmes
37151213cd Move more of the graph processing into the backend
The existing architecture serializes all cards and revlog entries in
the search range into a protobuf message, which the web frontend needs
to decode and then process. The thinking at the time was that this would
make it easier for add-ons to add extra graphs, but in the ~2.5 years
since the new graphs were introduced, no add-ons appear to have taken
advantage of it.

The cards and revlog entries can grow quite large on large collections -
on a collection I tested with approximately 2.5M reviews, the serialized
data is about 110MB, which is a lot to have to deserialize in JavaScript.

This commit shifts the preliminary processing of the data to the Rust end,
which means the data is able to be processed faster, and less needs to
be sent to the frontend. On the test collection above, this reduces the
serialized data from about 110MB to about 160KB, resulting in a more
than 2x performance improvement, and reducing frontend memory usage from
about 400MB to about 40MB.

This also makes #2043 more feasible - while it is still about 50-100%
slower than protobufjs, with the much smaller message size, the difference
is only about 10ms.
2022-12-16 21:42:17 +10:00
Damien Elmes
fa625d7ad8
Minor Rust cleanups (#2272)
* Run cargo +nightly fmt

* Latest prost-build includes clippy workaround

* Tweak Rust protobuf imports

- Avoid use of stringify!(), as JetBrains editors get confused by it
- Stop merging all protobuf symbols into a single namespace

* Remove some unnecessary qualifications

Found via IntelliJ lint

* Migrate some asserts to assert_eq/ne

* Remove mention of node_modules exclusion

This no longer seems to be necessary after migrating away from Bazel,
and excluding it means TS/Svelte files can't be edited properly.
2022-12-16 21:40:27 +10:00
Damien Elmes
22ecef6fb2
Merge pull request #2255 from RumovZ/replace-pct-str-2
Replace pct-str with local percent_encoding crate
2022-12-16 11:21:04 +00:00
Damien Elmes
35cbde65d7 Revert "Always scan for media changes"
This reverts commit 09cb8b3cf6.

Overhead on larger folders/slower devices is more than I originally
anticipated, and can run into multiple seconds. This seems to be
particularly egregious on mobile, which I presume is due to sandboxing
overhead.
2022-12-15 20:40:20 +10:00
Damien Elmes
86b52f7626 Add a small unit test
pct-str encoded the / character as well, but the difference shouldn't
matter in our case.
2022-12-12 14:54:11 +10:00
RumovZ
c888ccc285 Replace pct-str with local ascii_percent_encoding 2022-12-09 11:49:39 +01:00
RumovZ
0b206b8a81 Ignore non-ASCII chars in ascii_percent_encoding 2022-12-09 11:47:59 +01:00
RumovZ
55f27779cf Add local copy of percent_encoding crate 2022-12-09 11:46:00 +01:00
RumovZ
0e7f02bfb7
Update Chrono Crate (#2242)
* Remove deprecated `and_hms()`

* Update chrono

* Update licenses and fix script

* Remove deprecated Date struct

* Remove chrono pin

* Skip format check on .vscode

Was failing for no reason.

* Replace deprecated chrono functions

* Add cargo-deny to update-licenses & pin versions (dae)

* Remove time 0.1 dependency  (dae)

We don't need to wait for chrono 0.5; it was provided behind a legacy
feature flag.
2022-12-07 17:00:14 +10:00
Damien Elmes
7e3ac53869 Allow unit tests to pass even when Qt translations overridden 2022-12-04 14:25:54 +10:00
RumovZ
90a3bbfdb8
Cap calculated hard delay secs at again delay + 1d (#2229) 2022-12-03 18:54:40 +10:00
Damien Elmes
e3167c4e3c Update incompatible crates 2022-11-30 12:38:10 +10:00
Damien Elmes
b4f4df0030 Pin chrono; update compatible Rust crates
Also pin reqwest in our other crates
2022-11-30 12:38:10 +10:00
Damien Elmes
0ac7969e2a Use workspace package info in more crates; mark private for cargo-deny 2022-11-30 12:19:56 +10:00
Damien Elmes
e497a56f54 Re-enable formatting for .toml files 2022-11-28 09:16:28 +10:00
Damien Elmes
7972708029 Remove rslib cargo config
Was added during initial Bazel experimentation back in 2020
2022-11-27 17:50:36 +10:00
Damien Elmes
5e0a761b87
Move away from Bazel (#2202)
(for upgrading users, please see the notes at the bottom)

Bazel brought a lot of nice things to the table, such as rebuilds based on
content changes instead of modification times, caching of build products,
detection of incorrect build rules via a sandbox, and so on. Rewriting the build
in Bazel was also an opportunity to improve on the Makefile-based build we had
prior, which was pretty poor: most dependencies were external or not pinned, and
the build graph was poorly defined and mostly serialized. It was not uncommon
for fresh checkouts to fail due to floating dependencies, or for things to break
when trying to switch to an older commit.

For day-to-day development, I think Bazel served us reasonably well - we could
generally switch between branches while being confident that builds would be
correct and reasonably fast, and not require full rebuilds (except on Windows,
where the lack of a sandbox and the TS rules would cause build breakages when TS
files were renamed/removed).

Bazel achieves that reliability by defining rules for each programming language
that define how source files should be turned into outputs. For the rules to
work with Bazel's sandboxing approach, they often have to reimplement or
partially bypass the standard tools that each programming language provides. The
Rust rules call Rust's compiler directly for example, instead of using Cargo,
and the Python rules extract each PyPi package into a separate folder that gets
added to sys.path.

These separate language rules allow proper declaration of inputs and outputs,
and offer some advantages such as caching of build products and fine-grained
dependency installation. But they also bring some downsides:

- The rules don't always support use-cases/platforms that the standard language
tools do, meaning they need to be patched to be used. I've had to contribute a
number of patches to the Rust, Python and JS rules to unblock various issues.
- The dependencies we use with each language sometimes make assumptions that do
not hold in Bazel, meaning they either need to be pinned or patched, or the
language rules need to be adjusted to accommodate them.

I was hopeful that after the initial setup work, things would be relatively
smooth-sailing. Unfortunately, that has not proved to be the case. Things
frequently broke when dependencies or the language rules were updated, and I
began to get frustrated at the amount of Anki development time I was instead
spending on build system upkeep. It's now about 2 years since switching to
Bazel, and I think it's time to cut losses, and switch to something else that's
a better fit.

The new build system is based on a small build tool called Ninja, and some
custom Rust code in build/. This means that to build Anki, Bazel is no longer
required, but Ninja and Rust need to be installed on your system. Python and
Node toolchains are automatically downloaded like in Bazel.

This new build system should result in faster builds in some cases:

- Because we're using cargo to build now, Rust builds are able to take advantage
of pipelining and incremental debug builds, which we didn't have with Bazel.
It's also easier to override the default linker on Linux/macOS, which can
further improve speeds.
- External Rust crates are now built with opt=1, which improves performance
of debug builds.
- Esbuild is now used to transpile TypeScript, instead of invoking the TypeScript
compiler. This results in faster builds, by deferring typechecking to test/check
time, and by allowing more work to happen in parallel.

As an example of the differences, when testing with the mold linker on Linux,
adding a new message to tags.proto (which triggers a recompile of the bulk of
the Rust and TypeScript code) results in a compile that goes from about 22s on
Bazel to about 7s in the new system. With the standard linker, it's about 9s.

Some other changes of note:

- Our Rust workspace now uses cargo-hakari to ensure all packages agree on
available features, preventing unnecessary rebuilds.
- pylib/anki is now a PEP420 implicit namespace, avoiding the need to merge
source files and generated files into a single folder for running. By telling
VSCode about the extra search path, code completion now works with generated
files without needing to symlink them into the source folder.
- qt/aqt can't use PEP420 as it's difficult to get rid of aqt/__init__.py.
Instead, the generated files are now placed in a separate _aqt package that's
added to the path.
- ts/lib is now exposed as @tslib, so the source code and generated code can be
provided under the same namespace without a merging step.
- MyPy and PyLint are now invoked once for the entire codebase.
- dprint will be used to format TypeScript/json files in the future instead of
the slower prettier (currently turned off to avoid causing conflicts). It can
automatically defer to prettier when formatting Svelte files.
- svelte-check is now used for typechecking our Svelte code, which revealed a
few typing issues that went undetected with the old system.
- The Jest unit tests now work on Windows as well.

If you're upgrading from Bazel, updated usage instructions are in docs/development.md and docs/build.md. A summary of the changes:

- please remove node_modules and .bazel
- install rustup (https://rustup.rs/)
- install rsync if not already installed  (on windows, use pacman - see docs/windows.md)
- install Ninja (unzip from https://github.com/ninja-build/ninja/releases/tag/v1.11.1 and
  place on your path, or from your distro/homebrew if it's 1.10+)
- update .vscode/settings.json from .vscode.dist
2022-11-27 15:24:20 +10:00
Stefan Kangas
5551a37f03
Fix typos (#2210) 2022-11-24 20:18:57 +10:00
Damien Elmes
ade2738458 Fix invalid utf8 not being detected
- Rusqlite was returning a different error
- The sort field may also contain invalid utf8
2022-11-22 13:27:20 +10:00