anki/python
Damien Elmes cf45cbf429
Rework syncing code, and replace local sync server (#2329)
This PR replaces the existing Python-driven sync server with a new one in Rust.
The new server supports both collection and media syncing, and is compatible
with both the new protocol mentioned below, and older clients. A setting has
been added to the preferences screen to point Anki to a local server, and a
similar setting is likely to come to AnkiMobile soon.

Documentation is available here: <https://docs.ankiweb.net/sync-server.html>

In addition to the new server and refactoring, this PR also makes changes to the
sync protocol. The existing sync protocol places payloads and metadata inside a
multipart POST body, which causes a few headaches:

- Legacy clients build the request in a non-deterministic order, meaning the
entire request needs to be scanned to extract the metadata.
- Reqwest's multipart API directly writes the multipart body, without exposing
the resulting stream to us, making it harder to track the progress of the
transfer. We've been relying on a patched version of reqwest for timeouts,
which is a pain to keep up to date.

To address these issues, the metadata is now sent in a HTTP header, with the
data payload sent directly in the body. Instead of the slower gzip, we now
use zstd. The old timeout handling code has been replaced with a new implementation
that wraps the request and response body streams to track progress, allowing us
to drop the git dependencies for reqwest, hyper-timeout and tokio-io-timeout.

The main other change to the protocol is that one-way syncs no longer need to
downgrade the collection to schema 11 prior to sending.
2023-01-18 12:43:46 +10:00
..
licenses.json pip/ -> python/ 2021-10-18 19:50:40 +10:00
licenses.sh updates to the build process and binary bundles 2022-02-10 19:23:07 +10:00
mkempty.py Move away from Bazel (#2202) 2022-11-27 15:24:20 +10:00
README.md Move away from Bazel (#2202) 2022-11-27 15:24:20 +10:00
requirements.anki.in Move away from Bazel (#2202) 2022-11-27 15:24:20 +10:00
requirements.aqt.in Switch back to winrt to see if it fixes slow TTS 2022-12-06 20:30:54 +10:00
requirements.base.in Move away from Bazel (#2202) 2022-11-27 15:24:20 +10:00
requirements.base.txt Move away from Bazel (#2202) 2022-11-27 15:24:20 +10:00
requirements.bundle.in Move away from Bazel (#2202) 2022-11-27 15:24:20 +10:00
requirements.bundle.txt Update certifi to fix security alert 2022-12-11 11:42:08 +10:00
requirements.dev.in Add dev tools for live-reloading Anki's web views (#2151) 2023-01-03 11:55:58 +10:00
requirements.dev.txt Add dev tools for live-reloading Anki's web views (#2151) 2023-01-03 11:55:58 +10:00
requirements.qt5_14.in Move away from Bazel (#2202) 2022-11-27 15:24:20 +10:00
requirements.qt5_14.txt Move away from Bazel (#2202) 2022-11-27 15:24:20 +10:00
requirements.qt5_15.in Move away from Bazel (#2202) 2022-11-27 15:24:20 +10:00
requirements.qt5_15.txt Move away from Bazel (#2202) 2022-11-27 15:24:20 +10:00
requirements.qt6_3.in Revert to Qt 6.3.1 on macOS 2022-12-14 15:25:10 +10:00
requirements.qt6_3.txt Revert to Qt 6.3.1 on macOS 2022-12-14 15:25:10 +10:00
requirements.qt6_4.in Update Windows/Linux to Qt 6.4.2 2023-01-09 16:22:47 +10:00
requirements.qt6_4.txt Update Windows/Linux to Qt 6.4.2 2023-01-09 16:22:47 +10:00
requirements.win.in Switch back to winrt to see if it fixes slow TTS 2022-12-06 20:30:54 +10:00
requirements.win.txt Switch back to winrt to see if it fixes slow TTS 2022-12-06 20:30:54 +10:00
update_python_deps.sh Revert to Qt 6.3.1 on macOS 2022-12-14 15:25:10 +10:00
update_win_deps.bat Fix TTS handling on Windows 2022-11-29 13:04:51 +10:00
write_wheel.py Rework syncing code, and replace local sync server (#2329) 2023-01-18 12:43:46 +10:00

  • To achieve reproducible builds we use pip-tools to lock packages to a particular version - see update_python_deps.sh
  • write_wheel.py is used to generate our wheels.