This PR replaces the existing Python-driven sync server with a new one in Rust.
The new server supports both collection and media syncing, and is compatible
with both the new protocol mentioned below, and older clients. A setting has
been added to the preferences screen to point Anki to a local server, and a
similar setting is likely to come to AnkiMobile soon.
Documentation is available here: <https://docs.ankiweb.net/sync-server.html>
In addition to the new server and refactoring, this PR also makes changes to the
sync protocol. The existing sync protocol places payloads and metadata inside a
multipart POST body, which causes a few headaches:
- Legacy clients build the request in a non-deterministic order, meaning the
entire request needs to be scanned to extract the metadata.
- Reqwest's multipart API directly writes the multipart body, without exposing
the resulting stream to us, making it harder to track the progress of the
transfer. We've been relying on a patched version of reqwest for timeouts,
which is a pain to keep up to date.
To address these issues, the metadata is now sent in a HTTP header, with the
data payload sent directly in the body. Instead of the slower gzip, we now
use zstd. The old timeout handling code has been replaced with a new implementation
that wraps the request and response body streams to track progress, allowing us
to drop the git dependencies for reqwest, hyper-timeout and tokio-io-timeout.
The main other change to the protocol is that one-way syncs no longer need to
downgrade the collection to schema 11 prior to sending.
* Layout toolbar using CSS grid, introducing left and right trays
The trays provide a space for add-ons to introduce their own widgets to the toolbar without interfering with each other.
* Align tray items to the top
* Move absolutely positioned add-on items to right toolbar tray
Workaround that fixes breakages in add-ons like AMBOSS, Study Timer, and potentially others that currently still inject absolutely positioned elements into the toolbar using `top_toolbar_did_init_links`.
* Account for add-ons that add manual padding (e.g. Study Timer)
* Add docstrings and slightly refactor
* Tweak item alignment
* Introduce hooks for extending left and right toolbar trays
* Assign CSS classes to all tray items
* Add disclaimer on transitional nature of new hooks
* Give webviews a slide-in animation
if reduced motion isn't set.
* Auto-hide toolbar in review mode
moving the mouse above the main webview expands the toolbar. When the mouse leaves the toolbar, it will collapse after a delay of 2s.
* Save some space on bottom toolbars
* Use props for all hard-coded transition durations
and decrease most commonly used duration (200ms) to 150ms.
* Move auto-hide logic into ToolbarWebView
and handle auto-hide specific events in the respective webview subclasses.
* Fix typing issues
* Fix flickering issue
* Add auto_hide_toolbar opt-in to preferences
* Rename hide_toolbar to collapse_toolbar
to better describe the dock-like behaviour.
* Rename setting to minimize_distractions
* Reduce calls to pm in eventFilter
* Run formatter
* Revert setting title to something more specific
* Increase default animation time to 180ms
* Inset toolbar in review mode
when auto-hide is not enabled.
* Use card background on toolbar and add glass effect
* Use flatten/elevate over inset/outset
* Use flatten/elevate over inset/outset
* Update toolbar.py
* Fix toolbar background delay
* Tweak styles
* Use "collapse" instead of "auto-hide"
* Fix background misalignment in collapse mode
* Do not collapse toolbar when pointer is outside MainWebView
* Reduce hide_timer interval to 1000ms
* Use CSS to hide toolbar instead of setting webview height
* Add guard to prevent backdrop-filter: blur on Qt 5.14
* Apply transition to body instead of toolbar
to not complicate things for #2301.
* Fix Qt 5.14 and apply guard globally
* Fix background image scaling difference
* Tweak preference wording (dae)
Qt seems to be treating min-width more like width, and truncates text.
The issue is not limited to macOS with force_custom_styles: with a
sufficiently large amount of text on a button, it truncates on other
platforms too.
https://forums.ankiweb.net/t/2-1-55-issue-with-auto-browser-and-upload-download-dialog/25636
There are a few other uses of min-width in the styling - it may be worth
checking whether they suffer from the same issue as well.
The Rust community appear to have converged on tracing - it's used by
the Rust compiler, and receives close to 10x the number of downloads
that slog does. Its API is more ergonomic, and it does a much nicer
job with async rust.
To make this change, we no longer pass around explicit loggers, and rely
on a globally-registered one. The log file location has been changed
from one in each profile folder to a single one in the base folder. This
will remain empty for most users, since only errors are logged by default,
but may be useful for debugging future changes.
* Use a webview to show add-on's config help
This allows add-ons to embed images for example.
* Improve initial size of splitter widgets
* Decrease font size and margin of webview
This reverts commit 5dc79e22cd.
I appear to have been confused in my earlier testing, as reverting
this change seems to make no difference the top bar on first startup,
and fixes a bunch of other regressions that the original change introduced.
https://github.com/ankitects/anki/issues/2269
* Swap flag and mark indicator position in RTL mode
* Make buttons of bottom toolbar align to edge of screen in RTL mode
* Use start instead of left and end instead of right
Ensures the background is the correct color by the time the webview
is shown. We keep the #night check for now, as it's useful when testing
in an external browser.
The starting size of a webview seems to be 640x480, but if it is hidden
without retainSizeWhenHidden being set, the dialog it contains can end
up with a height of 0, which prevents the dialog from being shown.
By being explicit about our desired starting size, we can use a more
useful default, and avoid the issue of missing dialogs.
* Facilitate updating of hooks
- Add instructions in contributing.md
- Change addon_config_editor_will_update_json hook to work with the new
hookslib code
* Fix typo in docs
* Always run replaced hook
* Use lowercase list for typing
* Forbid defining both a replaced and a legacy hook
* Persist collapsed- and field states with SessionOptions object
* Format types.ts
* Replace format function with f-string
* Give setters more descriptive parameter names
* Do not use default prefix for descriptions and fonts
since they are not meant to be changed via Svelte.
When opening the graphs screen in dark mode, we want to load the
page first and then reveal the webview, to prevent a flash of white
that can appear as the page loads. Previously we did this for any
call to load_ts_page(), but this results in flicker when refreshing
an existing webview, such as the move from deck list to congrats screen.
In those cases, at least on the machines I have to test with here, the
refresh is smoother without the hide and show step.
The new window case is still not ideal - while the hide+show prevents a
flash of white, there is a flash of black instead, presumably as the
webview draws the initially-blank framebuffer with the contents of the
webview.
* Fix QMenu item not having different color on hover
due to the color changes in #2220.
* Remove strong border on pressed Qt widgets
* Make button gradient more subtle
by changing gradient-start on hover instead of gradient-end.
* Apply QPushButton style to QSpinBox buttons
* Improve margin of QComboBox arrow
* Make button-bg same color as button-gradient-end
This makes the hover gradient more subtle.
This reverts commit fa4fc3e15a.
Issue turned out to be a packaging problem, and this should not be
required as the socket should be held open even if removed.
This reverts commit ee70006ec4.
There have been a number of people complaining that the current
behaviour is not intuitive, and they have a point - it's not the
way browsers behave when you navigate between pages.
I was able to reproduce the crashes fairly reliably by opening the
prefs screen on startup and shutting down the app after 600ms; after
this change the crashes no longer seem to occur.
* Give deck browser table an elevated container look
* Tweak colors of elevated containers (e.g. in deck options)
* Prepare editor fields for custom backgrounds
* Tweak field margin and add explanatory comment
(for upgrading users, please see the notes at the bottom)
Bazel brought a lot of nice things to the table, such as rebuilds based on
content changes instead of modification times, caching of build products,
detection of incorrect build rules via a sandbox, and so on. Rewriting the build
in Bazel was also an opportunity to improve on the Makefile-based build we had
prior, which was pretty poor: most dependencies were external or not pinned, and
the build graph was poorly defined and mostly serialized. It was not uncommon
for fresh checkouts to fail due to floating dependencies, or for things to break
when trying to switch to an older commit.
For day-to-day development, I think Bazel served us reasonably well - we could
generally switch between branches while being confident that builds would be
correct and reasonably fast, and not require full rebuilds (except on Windows,
where the lack of a sandbox and the TS rules would cause build breakages when TS
files were renamed/removed).
Bazel achieves that reliability by defining rules for each programming language
that define how source files should be turned into outputs. For the rules to
work with Bazel's sandboxing approach, they often have to reimplement or
partially bypass the standard tools that each programming language provides. The
Rust rules call Rust's compiler directly for example, instead of using Cargo,
and the Python rules extract each PyPi package into a separate folder that gets
added to sys.path.
These separate language rules allow proper declaration of inputs and outputs,
and offer some advantages such as caching of build products and fine-grained
dependency installation. But they also bring some downsides:
- The rules don't always support use-cases/platforms that the standard language
tools do, meaning they need to be patched to be used. I've had to contribute a
number of patches to the Rust, Python and JS rules to unblock various issues.
- The dependencies we use with each language sometimes make assumptions that do
not hold in Bazel, meaning they either need to be pinned or patched, or the
language rules need to be adjusted to accommodate them.
I was hopeful that after the initial setup work, things would be relatively
smooth-sailing. Unfortunately, that has not proved to be the case. Things
frequently broke when dependencies or the language rules were updated, and I
began to get frustrated at the amount of Anki development time I was instead
spending on build system upkeep. It's now about 2 years since switching to
Bazel, and I think it's time to cut losses, and switch to something else that's
a better fit.
The new build system is based on a small build tool called Ninja, and some
custom Rust code in build/. This means that to build Anki, Bazel is no longer
required, but Ninja and Rust need to be installed on your system. Python and
Node toolchains are automatically downloaded like in Bazel.
This new build system should result in faster builds in some cases:
- Because we're using cargo to build now, Rust builds are able to take advantage
of pipelining and incremental debug builds, which we didn't have with Bazel.
It's also easier to override the default linker on Linux/macOS, which can
further improve speeds.
- External Rust crates are now built with opt=1, which improves performance
of debug builds.
- Esbuild is now used to transpile TypeScript, instead of invoking the TypeScript
compiler. This results in faster builds, by deferring typechecking to test/check
time, and by allowing more work to happen in parallel.
As an example of the differences, when testing with the mold linker on Linux,
adding a new message to tags.proto (which triggers a recompile of the bulk of
the Rust and TypeScript code) results in a compile that goes from about 22s on
Bazel to about 7s in the new system. With the standard linker, it's about 9s.
Some other changes of note:
- Our Rust workspace now uses cargo-hakari to ensure all packages agree on
available features, preventing unnecessary rebuilds.
- pylib/anki is now a PEP420 implicit namespace, avoiding the need to merge
source files and generated files into a single folder for running. By telling
VSCode about the extra search path, code completion now works with generated
files without needing to symlink them into the source folder.
- qt/aqt can't use PEP420 as it's difficult to get rid of aqt/__init__.py.
Instead, the generated files are now placed in a separate _aqt package that's
added to the path.
- ts/lib is now exposed as @tslib, so the source code and generated code can be
provided under the same namespace without a merging step.
- MyPy and PyLint are now invoked once for the entire codebase.
- dprint will be used to format TypeScript/json files in the future instead of
the slower prettier (currently turned off to avoid causing conflicts). It can
automatically defer to prettier when formatting Svelte files.
- svelte-check is now used for typechecking our Svelte code, which revealed a
few typing issues that went undetected with the old system.
- The Jest unit tests now work on Windows as well.
If you're upgrading from Bazel, updated usage instructions are in docs/development.md and docs/build.md. A summary of the changes:
- please remove node_modules and .bazel
- install rustup (https://rustup.rs/)
- install rsync if not already installed (on windows, use pacman - see docs/windows.md)
- install Ninja (unzip from https://github.com/ninja-build/ninja/releases/tag/v1.11.1 and
place on your path, or from your distro/homebrew if it's 1.10+)
- update .vscode/settings.json from .vscode.dist
