Commit Graph

1245 Commits

Author SHA1 Message Date
Damien Elmes
f6486da233
Various tweaks to I/O code (#2478)
* Allow user to select I/O notetype instead of enforcing a specific name

* Display a clearer error when I/O note is missing an image

Opening the card layout screen from "manage notetypes" was showing an
error about the Anki version being too old.

Replacement error is not currently translatable.

* Preserve existing notetype when adding I/O notetype

* Add a 'from clipboard' string

The intention is to use this in the future to allow an image occlusion
to be created from an image on the clipboard.

* Tweak I/O init

- Use union type instead of multiple nullable values
- Pass the notetype id in to initialization

* Fix image insertion in I/O note

- The regex expected double quotes, and we were using single ones
- Image tags don't need to be closed

* Use more consistent naming in image_occlusion.proto

* Tweaks to default I/O notetype

- Show the header on the front side as well (I presume this is what
users expect; if not am happy to revert)
- Don't show comments on card (again, I presume users expect to use
this field to add notes that aren't displayed during review, as they
can use back extra for that)

* Fix sticky footer missing background

Caused by earlier CSS refactoring
2023-04-19 15:30:18 +10:00
Damien Elmes
dd13e78eca
Add ability to restore a notetype to its original configuration (#2472)
* Store the original stock notetype kind in the notetype

Will allow us to provide a command to restore a notetype to its default
settings/templates.

* Add a new action to restore a notetype to its original state
2023-04-18 14:07:51 +10:00
Damien Elmes
26c640805c Update Rust deps
axum-client-ip excluded, as it will need further work.
2023-03-31 14:38:24 +10:00
Damien Elmes
0a0d17ff98 Update Python deps
- Black's formatting has changed
- Pylint has introduced a new lint
2023-03-31 14:04:05 +10:00
Mani
2bf134dc72
Feature image occlusion (#2367)
* add note types with occlusions and image fields

* generate image occlusion cloze div data
- generate div element with data-* atrributes for canvas shape generate for reviewer

* getting image data & deck id and adding notes
the implementation added into backend
- added service index in backend.proto for image occlusion request
- created image_occlusion.proto with required message and service
- implementation in backend for getting image and adding notes, also during editing return imagecloze note and update notes
- add notes to selected deck, if no notetype then add image occlusion notetypes
- reuse notetype from stock notetypes when not exist

* script for generating shapes using canvas api in reviewer
- the flash issues fixed by loading image and using image size to draw canvas, also when image get resized, calculate scale using natural width and canvas width to draw shape at right position
- limit size of canvas for safari

* init image occlusion page in ts and build page
with
- fabricjs for editing shapes
- panzoom for drag and zoom
- pickr for color picker
- build page using web.rs

* implement top toolbar for canvas shapes
- undo & redo tools
- zoom in, zoom out and zoom fit
- group & ungroup
- copy & paste
- set transparency of shapes
- align tools

* implement side toolbar for drawing shapes
add top toolbar and the side toolbar contains following tools
- cursor for selecting shapes
- zoom for drag and zoom shapes in mask editor
- rectangle for creating it
- ellipse for creating it
- polygon for creating it using points
- shape fill color
- question mask color (currently only single color can be added for all shapes)

* add maskeditor page for editing mask
- add side toolbar and sidebar include toptoolbar
- load maskeditor in two mode
     - for adding note using path to image
     - for editing note using note id

* implement note editor page for adding notes
- the note editor page have simple button (B/I/U) and option to toggle html view
- option to select deck for adding notes into that deck
- option to generate to hide all, guess one & hide one, guess one notes

* add image occlusion page
add side toolbar, top toolbar, mask editor and note editor
- option to switch between mask editor and note editor

* implement generates notes and save notes
implemention to show toast components for messages

* removed pickr & implemented color picker component
- remove pickr
- implemented using html5 canvas
- range input for changing color
- another range input for opacity changes
- hex and rgba value support

* rename methods name & rust unwrap safety
- change plural names to singular
- create respone message in proto and return response with imagecloze note or error if not found with note id
- remove image_occlusion from post handler list
- rename service name in mediasrv.py
- rename methods name for image occlusion in backend and image_occlusion
- update frontend also for update functions' names
- handle error in frontend mask-editor.ts, when error getting notes then toast message shown to frontend

* extract to function & add comments & remove global
- extract function in mask-editor.ts to reduce duplicate
- remove unused global from css
- add comments to store.ts explaining usage
- changes id to noteId in lib.ts
- add comments for limitSize, becuase of duplicate implementation

* remove image_occlusion notetype
- remove from stock notetype, stdmodels
- add implementation for notetype to image occlusion
- add i18n for errors

* update smooth scroll, always show cursor tools
- change questionmask to qmask
- make selectable for shape true in all tools to simplify edits and draw shapes
- update image occlusion in reviewer ts to load image properly

* add and get notetype else return errors

* fix: not showing occlusion

* Use a oneof for ImageClozeNoteResponse

Makes it clearer that only one of them can be returned

* Don't crash if image filename not provided

The second unwrap should be ok, as the input is utf8

* Refactor get_image_cloze_note

- fixes crash when note doesn't exist - Ok(None) case was not covered
- decouples business logic from native error->proto error conversion
- no need for original copy
- field[x] is more idiomatic than field.get(x).unwrap()
- don't need mutable access to fields

* Fix crash if image file unreadable

+ Use our read_file helper for better error context

* Add metadata() helper

* Fix crash if file metadata can't be read

* remove color picker, qmask and shape color
- remove strings from ftl
- remove color picker component
- remove from cloze generation
- remove icons for two buttons
- use constant color for shapes

* update color in reviewer and ftl strings

* fix shape position in canvas & add border to shape
- rename mask to inactive shape and active shape color
- border witdth and border color
- change decimal point deserializing string and toFixed(2)
- add thin border in mask editor, may be image background was transparent

* fix shape position in canvas after modified
- do not draw fixed ratio shapes by turn of uniformScaling
- fix rectangle width,height
- fix ellipse rx,ry,width,height
- fix polygon postion and points
- draw outside of canvas also

* fix border width and color in reviewer canvas
- rename variable

* refactor cloze div generate and remove angle

* fix origin when drawn outside of canvas from right

* fix shape at boundry & not include rx,ry rectangle
- move shapes at boundry when pointer is outside of canvas
- include rx, ry for ellipse only
- include points for polygon only

* fix lint errors & update image size in editor canvas based on height and width

* remove unsupported layerX & layerX for touchscreen
- fix shapes at edges

* implemented undo redo with canvas state

- implemented undo redo using fabric canvas events
- polygon is special case and implemented only added and modified event
- rectangle and ellipse have object:added, object:modified and object:removed case
- change id to undo and redo

* remove background image from canvas and used css to put image tag below canvas editor

- set image width and height after adding image

* fix for polygon points, add br in cloze strings, & toogle masks button

- fix shapes at edges
- toggle masks button to show/hide masks
- hide clozes string, it contains <br>
- set height for div container (used 'relative' in css)

* refactor top toolbar, add space and border radius
- rename cursor tools
- add left and right border

* fix undo after undo happen, use transparent color in draw mode
2023-03-29 12:33:19 +10:00
RumovZ
bd88c6d352
Ensure state mutator runs after card is rendered (#2421)
* Ensure state mutator runs after card is rendered

* Ensure ease buttons only show when states are ready

* Pass context into states mutator

* Revert queuing of state mutator hook

Now that context data is exposed users shouldn't rely on the question
having been rendered anymore.

* Use callbacks instead of signals and timeout

... to track whether the states mutator ran or failed.

* Make mutator async

* Remove State enum

* Reduce requests and compute seed on backend
2023-03-16 16:31:00 +10:00
RumovZ
cdfb84f19a
Implement TTS using windows crate (#2371)
* Implement TTS using windows crate

* Use API calls instead of SSML

* Properly stop player in case of TTS error

* Add context to WindowsErrors

* Validate available voices

* Remove TTS text from synthesize error

* Limit maximum buffer size

* Make validation optional and list it in tts filter

* We no longer need the winrt module (dae)

* Use a separate request object so the meaning of the bool is clear (dae)

* Slightly shorten runtime error message (dae)

The default message appears to clip slightly.

* Alternate buffer implementation (dae)

* Use array instead of vec

* Drop the max buffer size to 128k (dae)
2023-02-17 12:26:07 +10:00
Damien Elmes
19bbcb4cc0 Use backend for extracting cloze text to type
Closes #2311
2023-01-18 23:05:28 +10:00
Damien Elmes
943dddf28f
Update Rust deps (#2332)
* Temporarily disable hakari

* Upgrade compatible deps except Chrono

* Update semver-incompatible crates

* Re-enable hakari

* Update licenses & cargo-deny

* Fix new clippy lints

* Update to latest Rust
2023-01-18 22:24:29 +10:00
Damien Elmes
ded805b504
Switch Rust import style (#2330)
* Prepare to switch Rust import style

* Run nightly format

Closes #2320

* Clean up a few imports

* Enable comment wrapping

* Wrap comments
2023-01-18 21:39:55 +10:00
Damien Elmes
cf45cbf429
Rework syncing code, and replace local sync server (#2329)
This PR replaces the existing Python-driven sync server with a new one in Rust.
The new server supports both collection and media syncing, and is compatible
with both the new protocol mentioned below, and older clients. A setting has
been added to the preferences screen to point Anki to a local server, and a
similar setting is likely to come to AnkiMobile soon.

Documentation is available here: <https://docs.ankiweb.net/sync-server.html>

In addition to the new server and refactoring, this PR also makes changes to the
sync protocol. The existing sync protocol places payloads and metadata inside a
multipart POST body, which causes a few headaches:

- Legacy clients build the request in a non-deterministic order, meaning the
entire request needs to be scanned to extract the metadata.
- Reqwest's multipart API directly writes the multipart body, without exposing
the resulting stream to us, making it harder to track the progress of the
transfer. We've been relying on a patched version of reqwest for timeouts,
which is a pain to keep up to date.

To address these issues, the metadata is now sent in a HTTP header, with the
data payload sent directly in the body. Instead of the slower gzip, we now
use zstd. The old timeout handling code has been replaced with a new implementation
that wraps the request and response body streams to track progress, allowing us
to drop the git dependencies for reqwest, hyper-timeout and tokio-io-timeout.

The main other change to the protocol is that one-way syncs no longer need to
downgrade the collection to schema 11 prior to sending.
2023-01-18 12:43:46 +10:00
Damien Elmes
0eddb25287
Integrate AnkiDroid's backend patches into the repo (#2290)
* Relax chrono specification for AnkiDroid

https://github.com/ankidroid/Anki-Android-Backend/pull/251

* Add AnkiDroid service and AnkiDroid customizations

Most of the work here was done by David in the Backend repo; integrating
it into this repo for ease of future maintenance.

Based on 5d9f262f4c
with some tweaks:

- Protobuf imports have been fixed to match the recent refactor
- FatalError has been renamed to AnkidroidPanicError
- Tweaks to the desktop code to deal with the extra arg to open_collection,
and exclude AnkiDroid service methods from our Python code.

* Refactor AnkiDroid's DB code to avoid uses of unsafe
2023-01-03 13:11:23 +10:00
Damien Elmes
0570cfdf48 Migrate from slog to tracing
The Rust community appear to have converged on tracing - it's used by
the Rust compiler, and receives close to 10x the number of downloads
that slog does. Its API is more ergonomic, and it does a much nicer
job with async rust.

To make this change, we no longer pass around explicit loggers, and rely
on a globally-registered one. The log file location has been changed
from one in each profile folder to a single one in the base folder. This
will remain empty for most users, since only errors are logged by default,
but may be useful for debugging future changes.
2022-12-24 10:44:40 +10:00
TRIAEIOU
9901ae428a
Nested clozes and increased cloze meta data (#2141)
* Nested clozes and increased cloze meta data

* Update contributors

* This reverts commit 3423df73f89f04a606b1bff3542a68a49ca52e9f.

* Update CONTRIBUTORS

* Formating

* Formating

* Formating

* Formating

* Formating

* Formating

* Formating

* Formating

* Code refactor

* Formating

* Formating

* Formating

* Formating and dead code

* Correct test case

* Remove Hint and Close storage of token string

* Update

* Formating

* Formating

* Formating

* Use write! instead of .push_str(&format).

* Formating
2022-12-19 12:03:15 +10:00
Matthias Metelka
6481899454
Fix some more RTL issues (#2244)
* Swap flag and mark indicator position in RTL mode

* Make buttons of bottom toolbar align to edge of screen in RTL mode

* Use start instead of left and end instead of right
2022-12-08 22:29:56 +10:00
Yoshi
ef3cfc561c
Facilitate hook updating/replacement (#2213)
* Facilitate updating of hooks

- Add instructions in contributing.md
- Change addon_config_editor_will_update_json hook to work with the new
  hookslib code

* Fix typo in docs

* Always run replaced hook

* Use lowercase list for typing

* Forbid defining both a replaced and a legacy hook
2022-12-07 15:39:57 +10:00
Damien Elmes
b4f4df0030 Pin chrono; update compatible Rust crates
Also pin reqwest in our other crates
2022-11-30 12:38:10 +10:00
Damien Elmes
0ac7969e2a Use workspace package info in more crates; mark private for cargo-deny 2022-11-30 12:19:56 +10:00
Damien Elmes
e497a56f54 Re-enable formatting for .toml files 2022-11-28 09:16:28 +10:00
Damien Elmes
5e0a761b87
Move away from Bazel (#2202)
(for upgrading users, please see the notes at the bottom)

Bazel brought a lot of nice things to the table, such as rebuilds based on
content changes instead of modification times, caching of build products,
detection of incorrect build rules via a sandbox, and so on. Rewriting the build
in Bazel was also an opportunity to improve on the Makefile-based build we had
prior, which was pretty poor: most dependencies were external or not pinned, and
the build graph was poorly defined and mostly serialized. It was not uncommon
for fresh checkouts to fail due to floating dependencies, or for things to break
when trying to switch to an older commit.

For day-to-day development, I think Bazel served us reasonably well - we could
generally switch between branches while being confident that builds would be
correct and reasonably fast, and not require full rebuilds (except on Windows,
where the lack of a sandbox and the TS rules would cause build breakages when TS
files were renamed/removed).

Bazel achieves that reliability by defining rules for each programming language
that define how source files should be turned into outputs. For the rules to
work with Bazel's sandboxing approach, they often have to reimplement or
partially bypass the standard tools that each programming language provides. The
Rust rules call Rust's compiler directly for example, instead of using Cargo,
and the Python rules extract each PyPi package into a separate folder that gets
added to sys.path.

These separate language rules allow proper declaration of inputs and outputs,
and offer some advantages such as caching of build products and fine-grained
dependency installation. But they also bring some downsides:

- The rules don't always support use-cases/platforms that the standard language
tools do, meaning they need to be patched to be used. I've had to contribute a
number of patches to the Rust, Python and JS rules to unblock various issues.
- The dependencies we use with each language sometimes make assumptions that do
not hold in Bazel, meaning they either need to be pinned or patched, or the
language rules need to be adjusted to accommodate them.

I was hopeful that after the initial setup work, things would be relatively
smooth-sailing. Unfortunately, that has not proved to be the case. Things
frequently broke when dependencies or the language rules were updated, and I
began to get frustrated at the amount of Anki development time I was instead
spending on build system upkeep. It's now about 2 years since switching to
Bazel, and I think it's time to cut losses, and switch to something else that's
a better fit.

The new build system is based on a small build tool called Ninja, and some
custom Rust code in build/. This means that to build Anki, Bazel is no longer
required, but Ninja and Rust need to be installed on your system. Python and
Node toolchains are automatically downloaded like in Bazel.

This new build system should result in faster builds in some cases:

- Because we're using cargo to build now, Rust builds are able to take advantage
of pipelining and incremental debug builds, which we didn't have with Bazel.
It's also easier to override the default linker on Linux/macOS, which can
further improve speeds.
- External Rust crates are now built with opt=1, which improves performance
of debug builds.
- Esbuild is now used to transpile TypeScript, instead of invoking the TypeScript
compiler. This results in faster builds, by deferring typechecking to test/check
time, and by allowing more work to happen in parallel.

As an example of the differences, when testing with the mold linker on Linux,
adding a new message to tags.proto (which triggers a recompile of the bulk of
the Rust and TypeScript code) results in a compile that goes from about 22s on
Bazel to about 7s in the new system. With the standard linker, it's about 9s.

Some other changes of note:

- Our Rust workspace now uses cargo-hakari to ensure all packages agree on
available features, preventing unnecessary rebuilds.
- pylib/anki is now a PEP420 implicit namespace, avoiding the need to merge
source files and generated files into a single folder for running. By telling
VSCode about the extra search path, code completion now works with generated
files without needing to symlink them into the source folder.
- qt/aqt can't use PEP420 as it's difficult to get rid of aqt/__init__.py.
Instead, the generated files are now placed in a separate _aqt package that's
added to the path.
- ts/lib is now exposed as @tslib, so the source code and generated code can be
provided under the same namespace without a merging step.
- MyPy and PyLint are now invoked once for the entire codebase.
- dprint will be used to format TypeScript/json files in the future instead of
the slower prettier (currently turned off to avoid causing conflicts). It can
automatically defer to prettier when formatting Svelte files.
- svelte-check is now used for typechecking our Svelte code, which revealed a
few typing issues that went undetected with the old system.
- The Jest unit tests now work on Windows as well.

If you're upgrading from Bazel, updated usage instructions are in docs/development.md and docs/build.md. A summary of the changes:

- please remove node_modules and .bazel
- install rustup (https://rustup.rs/)
- install rsync if not already installed  (on windows, use pacman - see docs/windows.md)
- install Ninja (unzip from https://github.com/ninja-build/ninja/releases/tag/v1.11.1 and
  place on your path, or from your distro/homebrew if it's 1.10+)
- update .vscode/settings.json from .vscode.dist
2022-11-27 15:24:20 +10:00
Stefan Kangas
5551a37f03
Fix typos (#2210) 2022-11-24 20:18:57 +10:00
Damien Elmes
063623af3c Format .toml files with dprint 2022-11-09 20:03:49 +10:00
Damien Elmes
2504ad0b99 Fix mypy not picking up on missing attributes
Behaviour changed in recent releases:
https://github.com/python/mypy/issues/13319
2022-11-04 14:56:36 +10:00
Damien Elmes
f9aff548b7 Update protoc for new Python codegen
Pylint is no longer able to detect available members in the generated
protobuf files, so we need to exclude them from processing. This should
not be a problem as we get these checks from mypy already, and as a
bonus we no longer need to exclude protobuf classes manually.

The fact that anki.sync_pb2 needs to be added after the wildcard appears
to be a bug.
2022-10-21 20:23:31 +10:00
RumovZ
c521753057
Refactor error handling (#2136)
* Add crate snafu

* Replace all inline structs in AnkiError

* Derive Snafu on AnkiError

* Use snafu for card type errors

* Use snafu whatever error for InvalidInput

* Use snafu for NotFoundError and improve message

* Use snafu for FileIoError to attach context

Remove IoError.
Add some context-attaching helpers to replace code returning bare
io::Errors.

* Add more context-attaching io helpers

* Add message, context and backtrace to new snafus

* Utilize error context and backtrace on frontend

* Rename LocalizedError -> BackendError.
* Remove DocumentedError.
* Have all backend exceptions inherit BackendError.

* Rename localized(_description) -> message

* Remove accidentally committed experimental trait

* invalid_input_context -> ok_or_invalid

* ensure_valid_input! -> require!

* Always return `Err` from `invalid_input!`

Instead of a Result to unwrap, the macro accepts a source error now.

* new_tempfile_in_parent -> new_tempfile_in_parent_of

* ok_or_not_found -> or_not_found

* ok_or_invalid -> or_invalid

* Add crate convert_case

* Use unqualified lowercase type name

* Remove uses of snafu::ensure

* Allow public construction of InvalidInputErrors (dae)

Needed to port the AnkiDroid changes.

* Make into_protobuf() public (dae)

Also required for AnkiDroid. Not sure why it worked previously - possible
bug in older Rust version?
2022-10-21 18:02:12 +10:00
Damien Elmes
fb9c934ef2 Use protoc from Bazel if missing from path
Closes #2134
2022-10-17 09:58:51 +10:00
Damien Elmes
cfb309e6b3 Update Rust deps 2022-09-24 13:22:46 +10:00
Damien Elmes
a39a3b4d34 Update to latest rules_rust and Rust 1.64 2022-09-24 11:12:58 +10:00
Damien Elmes
9abc73360e Update Python deps
Addresses a protobuf CVE. Required some other patches due to changes
in latest mypy and pylint.
2022-09-24 09:46:43 +10:00
RumovZ
e39fb74e82
Enable state-dependent custom scheduling data (#2049)
* Enable state-dependent custom scheduling data

* Next(Card)States -> SchedulingStates

The fact that `current` was included in `next` always bothered me,
and custom data is part of the card state, so that was a bit confusing
too.

* Store custom_data in SchedulingState

* Make custom_data optional when answering

Avoids having to send it 4 extra times to the frontend, and avoids the
legacy answerCard() API clobbering the stored data.

Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-09-05 16:48:01 +10:00
Damien Elmes
0bcb3a3564 Add non-legacy backend interface for adding new decks 2022-09-04 14:12:29 +10:00
RumovZ
31b7464c67
Add card meta for persisting custom scheduling state (#2040)
* Add card meta for persisting custom scheduling state

* Rename meta -> custom_data

* Enforce limits on size of custom data

Large values will slow down table scans of the cards table, and it's
easier to be strict now and possibly relax things in the future than
the opposite.

* Pack card states and customData into a single message

+ default customData to empty if it can't be parsed

Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-09-02 11:22:49 +10:00
gnnoh
e4f1d9952b
Make dvipng use transparent background by default (#2036)
* make dvipng use transparent background by default

Note that if using all-lowercase 'transparent', it will give "a simple fully transparent background with non-transparent antialiased pixels", which is unnecessary.

* latex.py: format
2022-08-31 18:49:22 +10:00
Aristotelis
825c88b6e8
Make all Anki-native exceptions inherit from the same base class (#2028)
* Make all Anki-native exceptions inherit from same base class

Allows add-ons to easily catch all Anki-native exceptions without being coupled to the currently implemented exceptions.

* Satisfy pylint
2022-08-24 16:07:44 +10:00
phwoo
a338271d55
Narrow typehints within pylib (#2003)
* narrowed type hints

* narrowed type Any

* narrowed type Any

* narrow type Any
2022-08-05 14:48:07 +10:00
Damien Elmes
1e0be26b7e Partially migrate type answer to backend
Partially completes #1068, and will allow mobile clients to drop
their separate diff-match-patch imports. Does not yet try to handle
case folding or combining-char stripping, and leaves some of the outer
HTML wrapping up to the frontend for now.

The logic for rendering the provided string has changed: missing chars
are now only inserted if they follow a correct section, and the original
text is shown instead of hyphens. This is an experiment, and can be
changed if it's not well received.
2022-07-22 19:28:34 +10:00
RumovZ
cc929687ae
Deck-specific Limits (#1955)
* Add deck-specific limits to DeckNormal

* Add deck-specific limits to schema11

* Add DeckLimitsDialog

* deck_limits_qt6.py needs to be a symlink

* Clear duplicate deck setting keys on downgrade

* Export deck limits when exporting with scheduling

* Revert "deck_limits_qt6.py needs to be a symlink"

This reverts commit 4ee7be1e10c4e8c49bb20de3bf45ac18b5e2d4f6.

* Revert "Add DeckLimitsDialog"

This reverts commit eb0e2a62d33df0b518d9204a27b09e97966ce82a.

* Add day limits to DeckNormal

* Add deck and day limits mock to deck options

* Revert "Add deck and day limits mock to deck options"

This reverts commit 0775814989e8cb486483d06727b1af266bb4513a.

* Add Tabs component for daily limits

* Add borders to tabs component

* Revert "Add borders to tabs component"

This reverts commit aaaf5538932540f944d92725c63bb04cfe97ea14.

* Implement tabbed limits properly

* Add comment to translations

* Update rslib/src/decks/limits.rs

Co-authored-by: Damien Elmes <dae@users.noreply.github.com>

* Fix camel case in clear_other_duplicates()

* day_limit → current_limit

* Also import day limits

* Remember last used day limits

* Add day limits to schema 11

* Tweak comment (dae)

* Exclude day limit in export (dae)

* Tweak tab wording (dae)

* Update preset limits on preset change

* Explain tabs in tooltip (dae)

* Omit deck and today limits if v2 is enabled

* Preserve deck limit when switching to today limit
2022-07-19 18:27:25 +10:00
Matthias Metelka
d1ba48bc48
Expose cloze text as HTML attribute on question side (#1968)
* Expose cloze text as attribute on front side

* Update test_models.py

* Update template_filters.rs

* Escape HTML for data-attribute

* Use minimal HTML encoding in Rust

to match Python's html.escape and pass tests.

* Rename attribute to data-cloze

to make it more generic.

* Run formatter

* Revert to using Rust encode_attribute and add helper function for tests
2022-07-19 04:22:57 +10:00
Sam Penny
9dcceff4af
Remember previous choices in reposition dialog (#1950)
* remember previous choices in reposition dialog

* remember previous choice for randomize option as well

* fix failing test
2022-07-08 11:28:38 +10:00
Damien Elmes
5ea78e1c8e Since DupeResolution is in CsvMetadata, we don't need to pass it separately
Follow-up to #1930
2022-06-27 17:15:54 +10:00
RumovZ
6da5e5b042
CSV import/export fixes and features (#1898)
* Fix footer moving upwards

* Fix column detection

Was broken because escaped line breaks were not considered.
Also removes delimiter detection on `#columns:` line. User must use tabs
or set delimiter beforehand.

* Add CSV preview

* Parse `#tags column:`

* Optionally export deck and notetype with CSV

* Avoid clones in CSV export

* Prevent bottom of page appearing under footer (dae)

* Increase padding to 1em (dae)

With 0.5em, when a vertical scrollbar is shown, it sits right next to
the right edge of the content, making it look like there's no right
margin.

* Experimental changes to make table fit+scroll (dae)

- limit individual cells to 15em, and show ellipses when truncated
- limit total table width to body width, so that inner table is shown
with scrollbar
- use class rather than id - ids are bad practice in Svelte components,
as more than one may be displayed on a single page

* Skip importing foreign notes with filtered decks

Were implicitly imported into the default deck before.
Also some refactoring to fetch deck ids and names beforehand.

* Hide spacer below hidden field mapping

* Fix guid being replaced when updating note

* Fix dupe identity check

Canonify tags before checking if dupe is identical, but only add update
tags later if appropriate.

* Fix deck export for notes with missing card 1

* Fix note lines starting with `#`

csv crate doesn't support escaping a leading comment char. :(

* Support import/export of guids

* Strip HTML from preview rows

* Fix initially set deck if current is filtered

* Make isHtml toggle reactive

* Fix `html_to_text_line()` stripping sound names

* Tweak export option labels

* Switch to patched rust-csv fork

Fixes writing lines starting with `#`, so revert 5ece10ad05f331.

* List column options with first column field

* Fix flag for exports with HTML stripped
2022-06-09 10:28:01 +10:00
Damien Elmes
59ee399c5f Update regex crate for CVE 2022-06-07 08:39:07 +10:00
Damien Elmes
53c768e61c Reload scheduler after full download
Fixes invalid state where schedVer=2 but v1 scheduler active
https://forums.ankiweb.net/t/not-able-to-fix-database-error/20483
2022-06-02 16:42:43 +10:00
RumovZ
42cbe42f06
Plaintext import/export (#1850)
* Add crate csv

* Add start of csv importing on backend

* Add Menomosyne serializer

* Add csv and json importing on backend

* Add plaintext importing on frontend

* Add csv metadata extraction on backend

* Add csv importing with GUI

* Fix missing dfa file in build

Added compile_data_attr, then re-ran cargo/update.py.

* Don't use doubly buffered reader in csv

* Escape HTML entities if CSV is not HTML

Also use name 'is_html' consistently.

* Use decimal number as foreign ease (like '2.5')

* ForeignCard.ivl → ForeignCard.interval

* Only allow fixed set of CSV delimiters

* Map timestamp of ForeignCard to native due time

* Don't trim CSV records

* Document use of empty strings for defaults

* Avoid creating CardGenContexts for every note

This requires CardGenContext to be generic, so it works both with an
owned and borrowed notetype.

* Show all accepted file types  in import file picker

* Add import_json_file()

* factor → ease_factor

* delimter_from_value → delimiter_from_value

* Map columns to fields, not the other way around

* Fallback to current config for csv metadata

* Add start of new import csv screen

* Temporary fix for compilation issue on Linux/Mac

* Disable jest bazel action for import-csv

Jest fails with an error code if no tests are available, but this would
not be noticable on Windows as Jest is not run there.

* Fix field mapping issue

* Revert "Temporary fix for compilation issue on Linux/Mac"

This reverts commit 21f8a261408cdae49ec031aa21a1b659c4f66d82.

* Add HtmlSwitch and move Switch to components

* Fix spacing and make selectors consistent

* Fix shortcut tooltip

* Place import button at the top with path

* Fix meta column indices

* Remove NotetypeForString

* Fix queue and type of foreign cards

* Support different dupe resolution strategies

* Allow dupe resolution selection when importing CSV

* Test import of unnormalized text

Close  #1863.

* Fix logging of foreign notes

* Implement CSV exports

* Use db_scalar() in notes_table_len()

* Rework CSV metadata

- Notetypes and decks are either defined by a global id or by a column.
- If a notetype id is provided, its field map must also be specified.
- If a notetype column is provided, fields are now mapped by index
instead of name at import time. So the first non-meta column is used for
the first field of every note, regardless of notetype. This makes
importing easier and should improve compatiblity with files without a
notetype column.
- Ensure first field can be mapped to a column.
- Meta columns must be defined as `#[meta name]:[column index]` instead
of in the `#columns` tag.
- Column labels contain the raw names defined by the file and must be
prettified by the frontend.

* Adjust frontend to new backend column mapping

* Add force flags for is_html and delimiter

* Detect if CSV is HTML by field content

* Update dupe resolution labels

* Simplify selectors

* Fix coalescence of oneofs in TS

* Disable meta columns from selection

Plus a lot of refactoring.

* Make import button stick to the bottom

* Write delimiter and html flag into csv

* Refetch field map after notetype change

* Fix log labels for csv import

* Log notes whose deck/notetype was missing

* Fix hiding of empty log queues

* Implement adding tags to all notes of a csv

* Fix dupe resolution not being set in log

* Implement adding tags to updated notes of a csv

* Check first note field is not empty

* Temporary fix for build on Linux/Mac

* Fix inverted html check (dae)

* Remove unused ftl string

* Delimiter → Separator

* Remove commented-out line

* Don't accept .json files

* Tweak tag ftl strings

* Remove redundant blur call

* Strip sound and add spaces in csv export

* Export HTML by default

* Fix unset deck in Mnemosyne import

Also accept both numbers and strings for notetypes and decks in JSON.

* Make DupeResolution::Update the default

* Fix missing dot in extension

* Make column indices 1-based

* Remove StickContainer from TagEditor

Fixes line breaking, border and z index on ImportCsvPage.

* Assign different key combos to tag editors

* Log all updated duplicates

Add a log field for the true number of found notes.

* Show identical notes as skipped

* Split tag-editor into separate ts module (dae)

* Add progress for CSV export

* Add progress for text import

* Tidy-ups after tag-editor split (dae)

- import-csv no longer depends on editor
- remove some commented lines
2022-06-01 20:26:16 +10:00
Abdo
45b12ca104
Store rendered side in TemplateRenderContext (#1852)
* Store rendered side in TemplateRenderContext

* Move _question_side up with the other simple properties (dae)
2022-05-10 11:44:44 +10:00
RumovZ
890b28c1e8
Fix undisruptive formatting issues (#1848)
* Skip unparsable pylib files with black

This was causing black to return a non-zero code, preventing the
subsequent isort from running.

* Organise imports on save
2022-05-09 20:00:20 +10:00
Damien Elmes
c8a4e5ea22 Add more progress + abort points to export code
The old `media_files_did_export` hook has been kept around for use with
the legacy apkg exporter (an add-on uses it), and a new 
`legacy_export_progress` hook has been added so we can get progress
from the new colpkg exporter until we move over fully to the new code.
2022-05-06 15:32:23 +10:00
RumovZ
5f9451f547
Add apkg import/export on backend (#1743)
* Add apkg export on backend

* Filter out missing media-paths at write time

* Make TagMatcher::new() infallible

* Gather export data instead of copying directly

* Revert changes to rslib/src/tags/

* Reuse filename_is_safe/check_filename_safe()

* Accept func to produce MediaIter in export_apkg()

* Only store file folder once in MediaIter

* Use temporary tables for gathering

export_apkg() now accepts a search instead of a deck id. Decks are
gathered according to the matched notes' cards.

* Use schedule_as_new() to reset cards

* ExportData → ExchangeData

* Ignore ascii case when filtering system tags

* search_notes_cards_into_table →

search_cards_of_notes_into_table

* Start on apkg importing on backend

* Fix due dates in days for apkg export

* Refactor import-export/package

- Move media and meta code into appropriate modules.
- Normalize/check for normalization when deserializing media entries.

* Add SafeMediaEntry for deserialized MediaEntries

* Prepare media based on checksums

- Ensure all existing media files are hashed.
- Hash incoming files during preparation to detect conflicts.
- Uniquify names of conflicting files with hash (not notetype id).
- Mark media files as used while importing notes.
- Finally copy used media.

* Handle encoding in `replace_media_refs()`

* Add trait to keep down cow boilerplate

* Add notetypes immediately instaed of preparing

* Move target_col into Context

* Add notes immediately instaed of preparing

* Note id, not guid of conflicting notes

* Add import_decks()

* decks_configs → deck_configs

* Add import_deck_configs()

* Add import_cards(), import_revlog()

* Use dyn instead of generic for media_fn

Otherwise, would have to pass None with type annotation in the default
case.

* Fix signature of import_apkg()

* Fix search_cards_of_notes_into_table()

* Test new functions in text.rs

* Add roundtrip test for apkg (stub)

* Keep source id of imported cards (or skip)

* Keep source ids of imported revlog (or skip)

* Try to keep source ids of imported notes

* Make adding notetype with id undoable

* Wrap apkg import in transaction

* Keep source ids of imported deck configs (or skip)

* Handle card due dates and original due/did

* Fix importing cards/revlog

Card ids are manually uniquified.

* Factor out card importing

* Refactor card and revlog importing

* Factor out card importing

Also handle missing parents .

* Factor out note importing

* Factor out media importing

* Maybe upgrade scheduler of apkg

* Fix parent deck gathering

* Unconditionally import static media

* Fix deck importing edge cases

Test those edge cases, and add some global test helpers.

* Test note importing

* Let import_apkg() take a progress func

* Expand roundtrip apkg test

* Use fat pointer to avoid propogating generics

* Fix progress_fn type

* Expose apkg export/import on backend

* Return note log when importing apkg

* Fix archived collection name on apkg import

* Add CollectionOpWithBackendProgress

* Fix wrong Interrupted Exception being checked

* Add ClosedCollectionOp

* Add note ids to log and strip HTML

* Update progress when checking incoming media too

* Conditionally enable new importing in GUI

* Fix all_checksums() for media import

Entries of deleted files are nulled, not removed.

* Make apkg exporting on backend abortable

* Return number of notes imported from apkg

* Fix exception printing for QueryOp as well

* Add QueryOpWithBackendProgress

Also support backend exporting progress.

* Expose new apkg and colpkg exporting

* Open transaction in insert_data()

Was slowing down exporting by several orders of magnitude.

* Handle zstd-compressed apkg

* Add legacy arg to ExportAnkiPackage

Currently not exposed on the frontend

* Remove unused import in proto file

* Add symlink for typechecking of import_export_pb2

* Avoid kwargs in pb message creation, so typechecking is not lost

Protobuf's behaviour is rather subtle and I had to dig through the docs
to figure it out: set a field on a submessage to automatically assign 
the submessage to the parent, or call SetInParent() to persist a default
version of the field you specified.

* Avoid re-exporting protobuf msgs we only use internally

* Stop after one test failure

mypy often fails much faster than pylint

* Avoid an extra allocation when extracting media checksums

* Update progress after prepare_media() finishes

Otherwise the bulk of the import ends up being shown as "Checked: 0"
in the progress window.

* Show progress of note imports

Note import is the slowest part, so showing progress here makes the UI
feel more responsive.

* Reset filtered decks at import time

Before this change, filtered decks exported with scheduling remained
filtered on import, and maybe_remove_from_filtered_deck() moved cards
into them as their home deck, leading to errors during review.

We may still want to provide a way to preserve filtered decks on import,
but to do that we'll need to ensure we don't rewrite the home decks of
cards, and we'll need to ensure the home decks are included as part of
the import (or give an error if they're not).

https://github.com/ankitects/anki/pull/1743/files#r839346423

* Fix a corner-case where due dates were shifted by a day

This issue existed in the old Python code as well. We need to include
the user's UTC offset in the exported file, or days_elapsed falls back
on the v1 cutoff calculation, which may be a day earlier or later than
the v2 calculation.

* Log conflicting note in remapped nt case

* take_fields() → into_fields()

* Alias `[u8; 20]` with `Sha1Hash`

* Truncate logged fields

* Rework apkg note import tests

- Use macros for more helpful errors.
- Split monolith into unit tests.
- Fix some unknown error with the previous test along the way.
(Was failing after 969484de4388d225c9f17d94534b3ba0094c3568.)

* Fix sorting of imported decks

Also adjust the test, so it fails without the patch. It was only passing
before, because the parent deck happened to come before the
inconsistently capitalised child alphabetically. But we want all parent
decks to be imported before their child decks, so their children can
adopt their capitalisation.

* target[_id]s → existing_card[_id]s

* export_collection_extracting_media() → ...

export_into_collection_file()

* target_already_exists→card_ordinal_already_exists

* Add search_cards_of_notes_into_table.sql

* Imrove type of apkg export selector/limit

* Remove redundant call to mod_schema()

* Parent tooltips to mw

* Fix a crash when truncating note text

String::truncate() is a bit of a footgun, and I've hit this before
too :-)

* Remove ExportLimit in favour of separate classes

* Remove OpWithBackendProgress and ClosedCollectionOp

Backend progress logic is now in ProgressManager. QueryOp can be used
for running on closed collection.

Also fix aborting of colpkg exports, which slipped through in #1817.

* Tidy up import log

* Avoid QDialog.exec()

* Default to excluding scheuling for deck list deck

* Use IncrementalProgress in whole import_export code

* Compare checksums when importing colpkgs

* Avoid registering changes if hashes are not needed

* ImportProgress::Collection → ImportProgress::File

* Make downgrading apkgs depend on meta version

* Generalise IncrementableProgress

And use it in entire import_export code instead.

* Fix type complexity lint

* Take count_map for IncrementableProgress::get_inner

* Replace import/export env with Shift click

* Accept all args from update() for backend progress

* Pass fields of ProgressUpdate explicitly

* Move update_interval into IncrementableProgress

* Outsource incrementing into Incrementor

* Mutate ProgressUpdate in progress_update callback

* Switch import/export legacy toggle to profile setting

Shift would have been nice, but the existing shortcuts complicate things.
If the user triggers an import with ctrl+shift+i, shift is unlikely to
have been released by the time our code runs, meaning the user accidentally
triggers the new code. We could potentially wait a while before bringing
up the dialog, but then we're forced to guess at how long it will take the
user to release the key.

One alternative would be to use alt instead of shift, but then we need to
trigger our shortcut when that key is pressed as well, and it could
potentially cause a conflict with an add-on that already uses that
combination.

* Show extension in export dialog

* Continue to provide separate options for schema 11+18 colpkg export

* Default to colpkg export when using File>Export

* Improve appearance of combo boxes when switching between apkg/colpkg

+ Deal with long deck names

* Convert newlines to spaces when showing fields from import

Ensures each imported note appears on a separate line

* Don't separate total note count from the other summary lines

This may come down to personal preference, but I feel the other counts
are equally as important, and separating them feels like it makes it
a bit easier to ignore them.

* Fix 'deck not normal' error when importing a filtered deck for the 2nd time

* Fix [Identical] being shown on first import

* Revert "Continue to provide separate options for schema 11+18 colpkg export"

This reverts commit 8f0b2c175f4794d642823b60414d142a12768441.

Will use a different approach

* Move legacy support into a separate exporter option; add to apkg export

* Adjust 'too new' message to also apply to .apkg import case

* Show a better message when attempting to import new apkg into old code

Previously the user could end seeing a message like:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte

Unfortunately we can't retroactively fix this for older clients.

* Hide legacy support option in older exporting screen

* Reflect change from paths to fnames in type & name

* Make imported decks normal at once

Then skip special casing in update_deck(). Also skip updating
description if new one is empty.

Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-05-02 21:12:46 +10:00
Damien Elmes
e6d990a44e Fix cwd changing on export
No longer required after #1656
2022-04-21 16:50:05 +10:00
Damien Elmes
72c7d64876 Fix compatibility with older macOS versions
https://forums.ankiweb.net/t/anki-2-1-50-qt5-wont-open/19091
2022-04-15 17:35:05 +10:00
Damien Elmes
a9769813ba Add back support for custom mountpoint in card stats
The move to separate .html files broke our legacy card stats routine.

Related: d1d71ffdbb
2022-04-15 15:30:05 +10:00
Damien Elmes
d90608198f Don't rely on frontend to cap time taken in v3 2022-04-02 20:06:23 +10:00
RumovZ
f1488b5983
Card type error (#1749)
* TemplateSaveError -> CardTypeError

* Don't show success tooltip if export fails

* Attach help page to error

Show help link if export fails due to card type error.

* Add type (dae)

* Add shared show_exception() (dae)

- Use a shared routine for printing standard backend errors, so that
we can take advantage of the help links in eg. the card layout screen
as well.
- The truthiness check on help in showInfo() would have ignored the
enum 0 value.
- Close the exporting dialog on a documented failure as well

* Fix local variable help_page
2022-03-28 22:17:50 +10:00
RumovZ
dd16890c11
Add Deleted error and disable all bad browser rows (#1742)
* Add Deleted error and disable all bad browser rows

* Avoid error when opening the browse screen to a card with a missing note (dae)

* In cards mode, a missing note is NotFound, not Deleted (dae)

So we distinguish between referential integrity error, and explicit
deletion.

* Remove redundant try block
2022-03-28 19:06:19 +10:00
Damien Elmes
4515c41d2c
Backup improvements (#1728)
* Collection needs to be closed prior to backup even when not downgrading

* Backups -> BackupLimits

* Some improvements to backup_task

- backup_inner now returns the error instead of logging it, so that
the frontend can discover the issue when they await a backup (or create
another one)
- start_backup() was acquiring backup_task twice, and if another thread
started a backup between the two locks, the task could have been accidentally
overwritten without awaiting it

* Backups no longer require a collection close

- Instead of closing the collection, we ensure there is no active
transaction, and flush the WAL to disk. This means the undo history
is no longer lost on backup, which will be particularly useful if we
add a periodic backup in the future.
- Because a close is no longer required, backups are now achieved with
a separate command, instead of being included in CloseCollection().
- Full sync no longer requires an extra close+reopen step, and we now
wait for the backup to complete before proceeding.
- Create a backup before 'check db'

* Add File>Create Backup

https://forums.ankiweb.net/t/anki-mac-os-no-backup-on-sync/6157

* Defer checkpoint until we know we need it

When running periodic backups on a timer, we don't want to be fsync()ing
unnecessarily.

* Skip backup if modification time has not changed

We don't want the user leaving Anki open overnight, and coming back
to lots of identical backups.

* Periodic backups

Creates an automatic backup every 30 minutes if the collection has been
modified.

If there's a legacy checkpoint active, tries again 5 minutes later.

* Switch to a user-configurable backup duration

CreateBackup() now uses a simple force argument to determine whether
the user's limits should be respected or not, and only potentially
destructive ops (full download, check DB) override the user's configured
limit.

I considered having a separate limit for collection close and automatic
backups (eg keeping the previous 5 minute limit for collection close),
but that had two downsides:

- When the user closes their collection at the end of the day, they'd
get a recent backup. When they open the collection the next day, it
would get backed up again within 5 minutes, even though not much had
changed.
- Multiple limits are harder to communicate to users in the UI

Some remaining decisions I wasn't 100% sure about:

- If force is true but the collection has not been modified, the backup
will be skipped. If the user manually deleted their backups without
closing Anki, they wouldn't get a new one if the mtime hadn't changed.
- Force takes preference over the configured backup interval - should
we be ignored the user here, or take no backups at all?

Did a sneaky edit of the existing ftl string, as it hasn't been live
long.

* Move maybe_backup() into Collection

* Use a single method for manual and periodic backups

When manually creating a backup via the File menu, we no longer make
the user wait until the backup completes. As we continue waiting for
the backup in the background, if any errors occur, the user will get
notified about it fairly quickly.

* Show message to user if backup was skipped due to no changes

+ Don't incorrectly assert a backup will be created on force

* Add "automatic" to description

* Ensure we backup prior to importing colpkg if collection open

The backup doesn't happen when invoked from 'open backup' in the profile
screen, which matches Anki's previous behaviour. The user could
potentially clobber up to 30 minutes of their work if they exited to
the profile screen and restored a backup, but the alternative is we
create backups every time a backup is restored, which may happen a number
of times if the user is trying various ones. Or we could go back to a
separate throttle amount for this case, at the cost of more complexity.

* Remove the 0 special case on backup interval; minimum of 5 minutes

https://github.com/ankitects/anki/pull/1728#discussion_r830876833
2022-03-21 19:40:42 +10:00
RumovZ
16fe18d033
Refactor export-import code and resolve fixmes (#1723)
* Write media files in chunks

* Test media file writing

* Add iter `ReadDirFiles`

* Remove ImportMediaError, fail fatally instead

Partially reverts commit f8ed4d89ba.

* Compare hashes of media files to be restored

* Improve `MediaCopier::copy()`

* Restore media files atomically with tempfile

* Make downgrade flag an enum

* Remove SchemaVersion::Latest in favour of Option

* Remove sha1 comparison again

* Remove unnecessary repr(u8) (dae)
2022-03-18 19:31:55 +10:00
Damien Elmes
f8ed4d89ba Add new error variant to frontend; ensure errors are mapped appropriately 2022-03-17 20:58:35 +10:00
Damien Elmes
c2e8d89fc6
Colpkg fixes (#1722)
* Fix legacy colpkg import; disable v3 import/export; add roundtrip test

The test has revealed we weren't decompressing the media files on v3
import. That's easy to fix, but means all files need decompressing
even when they already exist, which is not ideal - it would be better
to store size/checksum in the metadata instead.

* Switch media and meta to protobuf; re-enable v3 import/export

- Fixed media not being decompressed on import
- The uncompressed size and checksum is now included for each media
entry, so that we can quickly check if a given file needs to be extracted.
We're still just doing a naive size comparison on colpkg import at the
moment, but we may want to use a checksum in the future, and will need
a checksum for apkg imports.
- Checksums can't be efficiently encoded in JSON, so the media list
has been switched to protobuf to reduce the the space requirements.
- The meta file has been switched to protobuf as well, for consistency.
This will mean any colpkg files exported with beta7 will be
unreadable.

* Avoid integer version comparisons

* Re-enable v3 test

* Apply suggestions from code review

Co-authored-by: RumovZ <gp5glkw78@relay.firefox.com>

* Add export_colpkg() method to Collection

More discoverable, and easier to call from unit tests

* Split import/export code out into separate folders

Currently colpkg/*.rs contain some routines that will be useful for
apkg import/export as well; in the future we can refactor them into a
separate file in the parent module.

* Return a proper error when media import fails

This tripped me up when writing the earlier unit test - I had called
the equivalent of import_colpkg()?, and it was returning a string error
that I didn't notice. In practice this should result in the same text
being shown in the UI, but just skips the tooltip.

* Automatically create media folder on import

* Move roundtrip test into separate file; check collection too

* Remove zstd version suffix

Prevents a warning shown each time Rust Analyzer is used to check the
code.

Co-authored-by: RumovZ <gp5glkw78@relay.firefox.com>
2022-03-17 15:11:23 +10:00
Damien Elmes
ed9476c856 Update Rust deps
Primarily for https://blog.rust-lang.org/2022/03/08/cve-2022-24713.html
2022-03-15 16:51:52 +10:00
RumovZ
e759885734
Backend colpkg exporting (#1719)
* Implement colpkg exporting on backend

* Use exporting logic in backup.rs

* Refactor exporting.rs

* Add backend function to export collection

* Refactor backend/collection.rs

* Use backend for colpkg exporting

* Don't use default zip compression for media

* Add exporting progress

* Refactor media file writing

* Write dummy collections

* Localize dummy collection note

* Minimize dummy db size

* Use `NamedTempFile::new()` instead of `new_in`

* Drop redundant v2 dummy collection

* COLLECTION_VERSION -> PACKAGE_VERSION

* Split `lock_collection()` into two to drop flag

* Expose new colpkg in GUI

* Improve dummy collection message

* Please type checker

* importing-colpkg-too-new -> exporting-...

* Compress the media map in the v3 package (dae)

On collections with lots of media, it can grow into megabytes.

Also return an error in extract_media_file_names(), instead of masking
it as an optional.

* Store media map as a vector in the v3 package (dae)

This compresses better (eg 280kb original, 100kb hashmap, 42kb vec)

In the colpkg import case we don't need random access. When importing
an apkg, we will need to be able to fetch file data for a given media
filename, but the existing map doesn't help us there, as we need
filename->index, not index->filename.

* Ensure folders in the media dir don't break the file mapping (dae)
2022-03-15 16:48:02 +10:00
Damien Elmes
2d6dd0630f Switch card.original_position to a proto3 optional 2022-03-10 20:48:26 +10:00
Damien Elmes
99cb6c616e Fix intermittent anki/backend_pb2.py build error on Windows
The _pb2 files are built for both the host and target architectures
(which seems superfluous - we may be able to fix that in the future).
Our script wrote the files into the build folder and then moved them
into the correct place, but because builds are not sandboxed on Windows,
the two actions were racy, and could cause each other to fail. Solved
by writing the files directly into their target locations.
2022-03-10 17:24:23 +10:00
Damien Elmes
f3e81c8a95 Move custom study tag and limit gathering+saving into the backend
Ideally this would have been in beta 6 :-) No add-ons appear to be
using customstudy.py/taglimit.py though, so it should hopefully not be
disruptive.

In the earlier custom study changes, we didn't get around to addressing
issue #1136. Now instead of trying to determine the maximum increase
to allow (which doesn't work correctly with nested decks), we just
present the total available to the user again, and let them decide. There's
plenty of room for improvement here still, but further work here might
be better done once we look into decoupling deck limits from deck presets.

Tags and available cards are fetched prior to showing the dialog now,
and will show a progress dialog if things take a while.

Tags are stored in an aux var now, so they don't inflate the deck
object size.
2022-03-10 16:23:03 +10:00
RumovZ
b9c3b12f71
Optionally restore original position and reset counts when forgetting (#1714)
* Add forget prompt with options

- Restore original position
- Reset reps and lapses

* Restore position when resetting for export

* Add config context to avoid passing keys

* Add routine to fetch defaults; use method-specific enum (dae)

* Keep original position by default (dae)

* Fix code completion for forget dialog (dae)

Needs to be a symbolic link to the generated file
2022-03-09 16:51:41 +10:00
RumovZ
f3c8857421
Backups (#1685)
* Add zstd dep

* Implement backend backup with zstd

* Implement backup thinning

* Write backup meta

* Use new file ending anki21b

* Asynchronously backup on collection close in Rust

* Revert "Add zstd dep"

This reverts commit 3fcb2141d2be15f907269d13275c41971431385c.

* Add zstd again

* Take backup col path from col struct

* Fix formatting

* Implement backup restoring on backend

* Normalize restored media file names

* Refactor `extract_legacy_data()`

A bit cumbersome due to borrowing rules.

* Refactor

* Make thinning calendar-based and gradual

* Consider last kept backups of previous stages

* Import full apkgs and colpkgs with backend

* Expose new backup settings

* Test `BackupThinner` and make it deterministic

* Mark backup_path when closing optional

* Delete leaky timer

* Add progress updates for restoring media

* Write restored collection to tempfile first

* Do collection compression in the background thread

This has us currently storing an uncompressed and compressed copy of
the collection in memory (not ideal), but means the collection can be
closed without waiting for compression to complete. On a large collection,
this takes a close and reopen from about 0.55s to about 0.07s. The old
backup code for comparison: about 0.35s for compression off, about
8.5s for zip compression.

* Use multithreading in zstd compression

On my system, this reduces the compression time of a large collection
from about 0.55s to 0.08s.

* Stream compressed collection data into zip file

* Tweak backup explanation

+ Fix incorrect tab order for ignore accents option

* Decouple restoring backup and full import

In the first case, no profile is opened, unless the new collection
succeeds to load.
In the second case, either the old collection is reloaded or the new one
is loaded.

* Fix number gap in Progress message

* Don't revert backup when media fails but report it

* Tweak error flow

* Remove native BackupLimits enum

* Fix type annotation

* Add thinning test for whole year

* Satisfy linter

* Await async backup to finish

* Move restart disclaimer out of backup tab

Should be visible regardless of the current tab.

* Write restored collection in chunks

* Refactor

* Write media in chunks and refactor

* Log error if removing file fails

* join_backup_task -> await_backup_completion

* Refactor backup.rs

* Refactor backup meta and collection extraction

* Fix wrong error being returned

* Call sync_all() on new collection

* Add ImportError

* Store logger in Backend, instead of creating one on demand

init_backend() accepts a Logger rather than a log file, to allow other
callers to customize the logger if they wish.

In the future we may want to explore using the tracing crate as an
alternative; it's a bit more ergonomic, as a logger doesn't need to be
passed around, and it plays more nicely with async code.

* Sync file contents prior to rename; sync folder after rename.

* Limit backup creation to once per 30 min

* Use zstd::stream::copy_decode

* Make importing abortable

* Don't revert if backup media is aborted

* Set throttle implicitly

* Change force flag to minimum_backup_interval

* Don't attempt to open folders on Windows

* Join last backup thread before starting new one

Also refactor.

* Disable auto sync and backup when restoring again

* Force backup on full download

* Include the reason why a media file import failed, and the file path

- Introduce a FileIoError that contains a string representation of
the underlying I/O error, and an associated path. There are a few
places in the code where we're currently manually including the filename
in a custom error message, and this is a step towards a more consistent
approach (but we may be better served with a more general approach in
the future similar to Anyhow's .context())
- Move the error message into importing.ftl, as it's a bit neater
when error messages live in the same file as the rest of the messages
associated with some functionality.

* Fix importing of media files

* Minor wording tweaks

* Save an allocation

I18n strings with replacements are already strings, so we can skip the
extra allocation. Not that it matters here at all.

* Terminate import if file missing from archive

If a third-party tool is creating invalid archives, the user should know
about it. This should be rare, so I did not attempt to make it
translatable.

* Skip multithreaded compression on small collections

Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-03-07 15:11:31 +10:00
Brayan Oliveira
b403f20cae
Fix html_media_regexps (#1710)
* Fix html_media_regexps

* Update CONTRIBUTORS
2022-03-07 11:02:30 +10:00
Damien Elmes
2c80bcb571 Update to latest rules_rust, which unblocks the tokio upgrade
Rust version has been pinned, as 1.59 gets stuck compiling pyo3_macros,
and appears to have other issues:
https://github.com/rust-lang/rust/issues/94390
2022-03-03 19:48:17 +10:00
RumovZ
508b5ab947
Remove top_deck_id arg in deck_tree() (#1702)
Counts don't propogate correctly anymore (#1678).
2022-03-02 15:30:32 +10:00
Damien Elmes
89bce6d829 Update Rust deps; pin Tokio
Tokio has had to be pinned, because the 1.17 release introduces
a dependency on windows_sys, which fails to build on Windows on
Bazel.

The issue appears to be the build script of a subcrate - it is using
CARGO_MANIFEST_DIR to update the linking path so windows.lib can be
found (it's contained in that crate), but the path is set incorrectly.

dfc25285a2/crates/targets/x86_64_msvc/build.rs

One way we might be able to work around it is to add to the link path
in our own build script.
2022-02-25 17:10:47 +10:00
Damien Elmes
1165939271 Update Python deps, including stable Black
Black 22.1 made some changes that required some minor reformatting.
2022-02-25 15:26:16 +10:00
RumovZ
a4d61fe7d9
Original position (#1677)
* Replace Card.data with .original_position

* Use and update original position in v3

* Show original position in card info

* Revert restoring original position for now

* Fix pb card to/from pylib card

* Try original_position as the last pb field

* minor wording tweaks (dae)
2022-02-22 22:48:21 +10:00
gnnoh
d29a0beb40
Fix ValueError when exported files have wrong mtime (#1666)
* Fix ValueError when exported files have wrong mtime

Set the `strict_timestamps` argument to `False`, so the media files which have a wrong mtime can be normally added to the zipfile.

* Update CONTRIBUTORS (ankitects#1666)

* Reformat exporting.py
2022-02-16 11:17:20 +10:00
Damien Elmes
bf1d81aacc fix cyclic import when importing anki.collection directly
Could be triggered with:
PYTHONPATH=.bazel/bin/qt/runanki.runfiles/ankidesktop/pylib python -c 'import anki.collection'
2022-02-15 19:08:29 +10:00
Damien Elmes
1253dc3a3b drop the bug: lines in v2.py
IIRC, it was triggered when the user doesn't have a valid current
deck selected. currentRevLimit() sets default=False, but it is ignored
and the default deck is returned instead.
2022-02-13 14:11:15 +10:00
RumovZ
9c54f85be6
Import submodules directly (#1662)
* Use submodule imports in aqt

* Use submodule imports in pylib

* More submodule imports in pylib

These required removing some direct imports to get rid of import cycles.
2022-02-13 13:40:47 +10:00
Damien Elmes
6ab221bda5 fix broken Windows tests 2022-02-11 19:46:38 +10:00
Damien Elmes
a8939e7938 serialize black invocations
On a Linux machine here, the tests consistently fail when two copies
of black are run at once:

% bazel test //qt:format_check //pylib:format_check --cache_test_results=no
==================== Test output for //qt:format_check:
Process SyncManager-1:
Traceback (most recent call last):
  File "/home/dae/.cache/bazel/_bazel_dae/fc22e40cbbf8b7d16ac57a00991b1ef1/external/python/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/dae/.cache/bazel/_bazel_dae/fc22e40cbbf8b7d16ac57a00991b1ef1/external/python/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/dae/.cache/bazel/_bazel_dae/fc22e40cbbf8b7d16ac57a00991b1ef1/external/python/lib/python3.9/multiprocessing/managers.py", line 583, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/home/dae/.cache/bazel/_bazel_dae/fc22e40cbbf8b7d16ac57a00991b1ef1/external/python/lib/python3.9/multiprocessing/managers.py", line 156, in __init__
    self.listener = Listener(address=address, backlog=16)
  File "/home/dae/.cache/bazel/_bazel_dae/fc22e40cbbf8b7d16ac57a00991b1ef1/external/python/lib/python3.9/multiprocessing/connection.py", line 453, in __init__
    self._listener = SocketListener(address, family, backlog)
  File "/home/dae/.cache/bazel/_bazel_dae/fc22e40cbbf8b7d16ac57a00991b1ef1/external/python/lib/python3.9/multiprocessing/connection.py", line 596, in __init__
    self._socket.bind(address)
OSError: [Errno 98] Address already in use

I dug briefly into Black's code, but suspect this is actually an issue
with the multiprocessing library. Didn't have time to investigate it
further; this workaround will do for now.

(One day I'll get around to merging those separate scripts into a single
one. One day. :-))
2022-02-11 14:47:05 +10:00
RumovZ
6c8cdc0a0c
Keep cwd and pass dir to player subprocess instead (#1656) 2022-02-11 10:35:48 +10:00
RumovZ
d55f080733
V3 parent limits (#1638)
* avoid repinning Rust deps by default

* add id_tree dependency

* Respect intermediate child limits in v3

* Test new behaviour of v3 counts

* Rework v3 queue building to respect parent limits

* Add missing did field to SQL query

* Fix `LimitTreeMap::is_exhausted()`

* Rework tree building logic

https://github.com/ankitects/anki/pull/1638#discussion_r798328734

* Add timer for build_queues()

* `is_exhausted()` -> `limit_reached()`

* Move context and limits into `QueueBuilder`

This allows for moving more logic into QueueBuilder, so less passing
around of arguments. Unfortunately, some tests will require additional
work to set up.

* Fix stop condition in new_cards_by_position

* Fix order gather order of new cards by deck

* Add scheduler/queue/builder/burying.rs

* Fix bad tree due to unsorted child decks

* Fix comment

* Fix `cap_new_to_review_rec()`

* Add test for new card gathering

* Always sort `child_decks()`

* Fix deck removal in `cap_new_to_review_rec()`

* Fix sibling ordering in new card gathering

* Remove limits for deck total count with children

* Add random gather order

* Remove bad sibling order handling

All routines ensure ascending order now.
Also do some other minor refactoring.

* Remove queue truncating

All routines stop now as soon as the root limit is reached.

* Move deck fetching into `QueueBuilder::new()`

* Rework new card gather and sort options

https://github.com/ankitects/anki/pull/1638#issuecomment-1032173013

* Disable new sort order choices ...

depending on set gather order.

* Use enum instead of numbers

* Ensure valid sort order setting

* Update new gather and sort order tooltips

* Warn about random insertion order with v3

* Revert "Add timer for build_queues()"

This reverts commit c9f5fc6ebe87953c17a0c842990b009b5596c69c.

* Update rslib/src/storage/card/mod.rs (dae)

* minor wording tweaks to the tooltips (dae)

+ move legacy strings to bottom
+ consistent capitalization (our leech action still needs fixing,
but that will require introducing a new 'suspend card' string as the
existing one is used elsewhere as well)
2022-02-10 09:55:43 +10:00
Damien Elmes
9a80524560 add Odia to language list 2022-02-07 22:31:41 +10:00
Arthur Milchior
8afe36b8e9
A field in __dict__ was not updated (#1632)
It should have been done in 2a93355824
2022-02-01 13:41:57 +10:00
Damien Elmes
369223ce8b fix generated code symlinks again
To work with f842ab7c9d
2022-01-24 16:59:37 +10:00
Arthur Milchior
d571ae29f0
Replace models by decks in models warning (#1619) 2022-01-24 11:47:58 +10:00
Damien Elmes
5efaf5a4be move Bazel convenience symlinks outside of repo folder
The default symlink location can cause slowdowns and wasted CPU cycles
in VS Code and PyCharm/IntelliJ, as they try to watch Bazel's (large)
build folder for changes. The issue can be mostly ameliorated in VS Code
by excluding the symlinks using globs in settings like watcherExclude,
but the Rust extension doesn't support globs, so each folder needs to be
listed out separately. And because the product name symlink depends on
the name of the directory you're building from, we can't just include
the excludes in .vscode - it will depend on the folder the user is storing
things.

PyCharm and IntelliJ behave even worse here - they continue to monitor
for changes in all folders of the repo, even if those folders have been
marked as excluded in the project settings. Placing the folders into the
IDE-global Editor>File Types>Ignored Files And Folders works around this,
but again we run into troubles making this work out of the box, especially
with the product name in the symlink.

One option would be to turn the symlinks off completely. They are not
required for building, and for scripting/debugging, we can get the folder
locations via 'bazel info'. But with that approach, we would no longer
be able to symlink build products into the source tree, as we do for
things like the generated backend methods and translations, so we'd lose
code completion for them that way.

Another option would be to place the symlinks in .bazel/ inside the repo.
That solves the VS Code case (in conjunction with a workspace config file),
but doesn't fully fix IntelliJ/PyCharm.

The only remaining option I can see is to place the symlinks outside the
repo. Bazel won't expand ~ in the symlink path, so we can't use something
like ~/.cache/bazel/anki to place the files near the other build files.
So we end up having to have the files written to ../bazel/anki, in the
repo's parent folder. Not very clean, but I don't see a better alternative
at the moment.

.gitignore is still ignoring bazel-*, as currently bazel-dist and
bazel-pkg will be created when building/packaging. They should be fairly
innocuous, but we may want to rename them at one point.

Other changes:

- add missing symlink for pylib hooks
- add a sample .user.bazelrc file
2022-01-23 19:18:44 +10:00
Henrik Giesel
a8d4774cdb
Add _raw methods for all methods in the backend (#1594)
* Add _bytes methods for all methods in the backend

Expose get_note in qt/aqt/mediasrv.py

* Satisfy formatter

* Rename _bytes function to _raw and have them bytes as input

* Fix backend generation

* Use lib/proto/deckOptions in deck-options

* Add exposed_backend to qt/aqt/mediasrv.py

* Move some more backend methods to exposed_backend_list

* Use protobufjs for congrats and i18n

* Use protobufjs for completeTag

* Use protobufjs services in change-notetype

* Reorder post handlers in alphabetical manner

* Satisfy tests

* Remove unused collection methods

* Rename access_backend to raw_backend_request

* Use _vendor.stringcase instead of creating a new function

* Remove SKIP_UNROLL_OUTPUT

* Directly call _run_command in non _raw methods

* Remove TranslateString, ChangeNotetype and CompleteTag from SKIP_UNROLL_INPUT

* Remove UpdateDeckConfigs from SKIP_UNROLL_INPUT

* Remove ChangeNotetype from SKIP_UNROLL_INPUT

* Remove SKIP_UNROLL_INPUT

* Fix typing issue with translate_string

- Adds typing support for Protobuf maps in genbackend.py

* Do not emit convenience method for protobuf TranslateString
2022-01-21 21:32:39 +10:00
RumovZ
9aca778a93
Backend Custom Study (#1600)
* Implement custom study on backend

* Switch frontend to backend custom study

* Skip typecheck for new pb classes

* Build tag search string on backend

Also fixes escaping of special characters in tag names.

* `cram.cards` -> `cram.card_limit`

* Assign more meaningful names in `TagLimit`

* Broaden rustfmt glob

* Use `invalid_input()` helper

* Assign `FilteredDeckForUpdate` to temp var

* Implement `SearchBuilder`

* Rewrite `custom_study()` with `SearchBuilder`

* Replace match macros with `SearchBuilder`

* Remove `into_nodes_list` & `concatenate_searches`
2022-01-20 14:25:22 +10:00
Damien Elmes
eba21780e1 add Belarusian to prefs 2022-01-16 15:58:57 +10:00
Damien Elmes
732c33c2b3 update to latest rules_python 2022-01-15 16:16:33 +10:00
Damien Elmes
1315a3a3bb update mypy and mypy-protobuf
- The way mypy gathers site packages has changed slightly, so we had to
update extendsitepkgs.py to work with it.
- Not sure if there's a way to avoid the ignore in
operations/__init__.py. mypy is still ensuring a provided argument has
a .changes attribute, so thankfully we don't seem to have lost much here.
2022-01-15 16:04:04 +10:00
Damien Elmes
bbc191711d update Rust deps 2022-01-15 14:59:43 +10:00
Abdo
35c8449ab2
Increment sched.reps for preview cards in V2 (#1573)
This is to make the timebox dialog report the correct count.
2022-01-04 08:56:02 +10:00
Damien Elmes
0c7a9be359 fix Windows build with latest rules_rust
the extra .lib file that is being output for .dll files was tripping
up the copy; we need to select only the .dll file
2021-12-20 18:59:55 +10:00
RumovZ
2221d0a520
Make Hard use current step's interval if it's not the first one (#1561)
* Make hard repeat the current step's interval in v3

Unless for the first step to avoid identical interval with Again.

* Make Hard repeat the current step's interval in v2

* Adjust test to new Hard behaviour
2021-12-16 22:02:13 +10:00
RumovZ
80ed94ed08
Cap steps (#1548)
* Fix steps being mistaken for seconds

* Cap steps at `u32::max` seconds

* Fix overflow of steps in Rust

* Prevent overflow of `IntervalKind`

* Prevent overflow in `revlod/mod.rs`

Also replace some `as` with `from` and `try_from` as is recommended to
highlight potential issues.

* Ensure v2 doesn't store overflowing revlog ivls

* Lower steps cap in deck options

Whereas large card intervals are converted to days, revlog intervals use
i32s to store large numbers of seconds.

* Format
2021-12-15 18:46:26 +10:00
Damien Elmes
8d90b6b061 run buildifier/buildozer to tidy up BUILD files 2021-12-14 09:18:24 +10:00
Damien Elmes
1b60f07e51 fix incorrect folder name in pylib mypy invocation 2021-12-10 10:50:57 +10:00
Damien Elmes
366d5ca9e3 experiment with mypy cache
Speeds things up in the small change case; remains to be seen whether
this will be robust or will introduce caching issues or other problems.
2021-12-10 10:46:38 +10:00
Arthur Milchior
08ad9f33b9
The importer list have a Hook (#1543)
* The importer list have a Hook

Previously, add-on 175027074 simply edited the list once. It became impossible
since the list became a function. Hence I need a filter to add the list here.

@kelciour (nice to meet you by the way), you may be interested by it too (at
least if I believe efb1ce46d4 )

I would have preferred to use `anki.importing.base.Importer` instead of
`Any`. However, this leads to
> Name "anki.importing.base.Importer" is not defined  [name-defined]
when I run test.

Helps to solve this would be welcomed

* mention the hook may not last too long (dae)
2021-12-09 11:20:29 +10:00
Arthur Milchior
69469c6428
Add py3.9 to hooks (#1542)
* Add py3.9 to hooks

This follows examples from efb1ce46d4 I assume the
hooks were missed because those were not considered types but strings.

I did not even try to run pyupgrade and did the change manually, then used bazel format

* remove wildcard import in find.py, and change Any to object (dae)
2021-12-09 09:11:22 +10:00
Damien Elmes
cad0c64105 add unbury_cards() op
https://forums.ankiweb.net/t/add-a-collection-operation-to-unbury-cards/15720
2021-12-08 09:44:47 +10:00