Commit Graph

9324 Commits

Author SHA1 Message Date
Henrik Giesel
bf20353b53
Add tooltip that you can double-click image for restoring its size (#1738)
* Add tooltip that you can double-click image for restoring its size

* Update ftl/core/editing.ftl

Lower case label
2022-03-24 19:53:57 +10:00
Henrik Giesel
9029c5f497
Fix cursor sometimes not being moved into mathjax editor (#1737) 2022-03-24 18:29:30 +10:00
Damien Elmes
7ab79aad8b
Update ts deps for minimist issue (#1739)
Presumably not an issue for our use case, but this removes the warning
from GitHub.

ef88b9325f
2022-03-24 10:24:56 +10:00
Damien Elmes
d2caab6d5a Update Python deps for Waitress fix
b28c9e8bda

Closes #1731
2022-03-23 22:22:23 +10:00
Damien Elmes
bbcab21f13 Start random card position from 1
Anki's DB schema unfortunately uses odid=0 instead of null to indicate
a lack of an original due date, so having a due position of 0 leads to
the temporary due date not being reset when the card is removed from
a filtered deck.

https://forums.ankiweb.net/t/anki-2-1-50-beta-6-9-stable-release/18181/52
2022-03-23 22:00:25 +10:00
Damien Elmes
6735b23e5b Force a schema change on colpkg import
If we don't force a full sync when restoring, any items that were added
since the backup may have already been sent to AnkiWeb, and they
won't have deletion records. After the user restores from a backup,
they'll end up in a state where their local and AnkiWeb collections
differ, and the changes will not sync. The old backup code forced a schema
change, but we weren't previously doing it via File>Import.
2022-03-22 13:10:32 +10:00
Damien Elmes
44342660d8 Update translations
Update translations
2022-03-21 21:49:28 +10:00
Damien Elmes
4515c41d2c
Backup improvements (#1728)
* Collection needs to be closed prior to backup even when not downgrading

* Backups -> BackupLimits

* Some improvements to backup_task

- backup_inner now returns the error instead of logging it, so that
the frontend can discover the issue when they await a backup (or create
another one)
- start_backup() was acquiring backup_task twice, and if another thread
started a backup between the two locks, the task could have been accidentally
overwritten without awaiting it

* Backups no longer require a collection close

- Instead of closing the collection, we ensure there is no active
transaction, and flush the WAL to disk. This means the undo history
is no longer lost on backup, which will be particularly useful if we
add a periodic backup in the future.
- Because a close is no longer required, backups are now achieved with
a separate command, instead of being included in CloseCollection().
- Full sync no longer requires an extra close+reopen step, and we now
wait for the backup to complete before proceeding.
- Create a backup before 'check db'

* Add File>Create Backup

https://forums.ankiweb.net/t/anki-mac-os-no-backup-on-sync/6157

* Defer checkpoint until we know we need it

When running periodic backups on a timer, we don't want to be fsync()ing
unnecessarily.

* Skip backup if modification time has not changed

We don't want the user leaving Anki open overnight, and coming back
to lots of identical backups.

* Periodic backups

Creates an automatic backup every 30 minutes if the collection has been
modified.

If there's a legacy checkpoint active, tries again 5 minutes later.

* Switch to a user-configurable backup duration

CreateBackup() now uses a simple force argument to determine whether
the user's limits should be respected or not, and only potentially
destructive ops (full download, check DB) override the user's configured
limit.

I considered having a separate limit for collection close and automatic
backups (eg keeping the previous 5 minute limit for collection close),
but that had two downsides:

- When the user closes their collection at the end of the day, they'd
get a recent backup. When they open the collection the next day, it
would get backed up again within 5 minutes, even though not much had
changed.
- Multiple limits are harder to communicate to users in the UI

Some remaining decisions I wasn't 100% sure about:

- If force is true but the collection has not been modified, the backup
will be skipped. If the user manually deleted their backups without
closing Anki, they wouldn't get a new one if the mtime hadn't changed.
- Force takes preference over the configured backup interval - should
we be ignored the user here, or take no backups at all?

Did a sneaky edit of the existing ftl string, as it hasn't been live
long.

* Move maybe_backup() into Collection

* Use a single method for manual and periodic backups

When manually creating a backup via the File menu, we no longer make
the user wait until the backup completes. As we continue waiting for
the backup in the background, if any errors occur, the user will get
notified about it fairly quickly.

* Show message to user if backup was skipped due to no changes

+ Don't incorrectly assert a backup will be created on force

* Add "automatic" to description

* Ensure we backup prior to importing colpkg if collection open

The backup doesn't happen when invoked from 'open backup' in the profile
screen, which matches Anki's previous behaviour. The user could
potentially clobber up to 30 minutes of their work if they exited to
the profile screen and restored a backup, but the alternative is we
create backups every time a backup is restored, which may happen a number
of times if the user is trying various ones. Or we could go back to a
separate throttle amount for this case, at the cost of more complexity.

* Remove the 0 special case on backup interval; minimum of 5 minutes

https://github.com/ankitects/anki/pull/1728#discussion_r830876833
2022-03-21 19:40:42 +10:00
Hikaru Y
704fb918c3
Fix NotFoundError when opening browser after switching profiles (#1725) 2022-03-21 14:45:22 +10:00
Damien Elmes
e2eff7deae Fix change notetype affecting cards outside selection
Accidentally introduced by the search refactoring in #1600, this lead
to a query that matched items outside of the selected notes, eg

(n.id in (1506029488152) and c.ord = 43 or c.ord = 4)

Closes #1727
2022-03-20 15:36:54 +10:00
Damien Elmes
d100f7a2c8 Don't fsync media files on import
I was seeing import speeds of only 10-20 files a second before this
change.
2022-03-19 23:31:11 +10:00
RumovZ
16fe18d033
Refactor export-import code and resolve fixmes (#1723)
* Write media files in chunks

* Test media file writing

* Add iter `ReadDirFiles`

* Remove ImportMediaError, fail fatally instead

Partially reverts commit f8ed4d89ba.

* Compare hashes of media files to be restored

* Improve `MediaCopier::copy()`

* Restore media files atomically with tempfile

* Make downgrade flag an enum

* Remove SchemaVersion::Latest in favour of Option

* Remove sha1 comparison again

* Remove unnecessary repr(u8) (dae)
2022-03-18 19:31:55 +10:00
RumovZ
5781e86995
Don't propogate limit extensions in v3 scheduler (#1724) 2022-03-18 19:16:31 +10:00
Damien Elmes
02c580e4f8 Update translations 2022-03-17 21:09:49 +10:00
Damien Elmes
5c1abb7f09 Add missing file header 2022-03-17 21:09:20 +10:00
Damien Elmes
dadb69a14b Fix some clippy lints + imports 2022-03-17 21:03:39 +10:00
Damien Elmes
bf8e70c70f Ensure partial colpkg file removed if export fails 2022-03-17 20:58:36 +10:00
Damien Elmes
f8ed4d89ba Add new error variant to frontend; ensure errors are mapped appropriately 2022-03-17 20:58:35 +10:00
Damien Elmes
bdfa8387a5 Handle export failures better
- Ensure collection reopened even on failure
- Don't display a traceback to the user
2022-03-17 17:59:47 +10:00
Damien Elmes
30e53c51ab Tiny cleanup missing from previous commit
Had intended to apply it prior to pushing.
2022-03-17 17:49:18 +10:00
Damien Elmes
4687620f5e Expand normalization checks on import/export
The old Python code was only checking for NFC encoding, but we should
check for other issues like special filenames on windows (eg con.mp3)

- On export, the user is told to use Check Media if their media has
invalid filenames.
- On import, legacy packages will be transparently normalized. Since we're
doing the checks on export as well, any invalid names in a v3 package
are an error.
2022-03-17 17:31:19 +10:00
Damien Elmes
57a4495d92 Check for attempted path traversal on import 2022-03-17 16:40:52 +10:00
Damien Elmes
c2e8d89fc6
Colpkg fixes (#1722)
* Fix legacy colpkg import; disable v3 import/export; add roundtrip test

The test has revealed we weren't decompressing the media files on v3
import. That's easy to fix, but means all files need decompressing
even when they already exist, which is not ideal - it would be better
to store size/checksum in the metadata instead.

* Switch media and meta to protobuf; re-enable v3 import/export

- Fixed media not being decompressed on import
- The uncompressed size and checksum is now included for each media
entry, so that we can quickly check if a given file needs to be extracted.
We're still just doing a naive size comparison on colpkg import at the
moment, but we may want to use a checksum in the future, and will need
a checksum for apkg imports.
- Checksums can't be efficiently encoded in JSON, so the media list
has been switched to protobuf to reduce the the space requirements.
- The meta file has been switched to protobuf as well, for consistency.
This will mean any colpkg files exported with beta7 will be
unreadable.

* Avoid integer version comparisons

* Re-enable v3 test

* Apply suggestions from code review

Co-authored-by: RumovZ <gp5glkw78@relay.firefox.com>

* Add export_colpkg() method to Collection

More discoverable, and easier to call from unit tests

* Split import/export code out into separate folders

Currently colpkg/*.rs contain some routines that will be useful for
apkg import/export as well; in the future we can refactor them into a
separate file in the parent module.

* Return a proper error when media import fails

This tripped me up when writing the earlier unit test - I had called
the equivalent of import_colpkg()?, and it was returning a string error
that I didn't notice. In practice this should result in the same text
being shown in the UI, but just skips the tooltip.

* Automatically create media folder on import

* Move roundtrip test into separate file; check collection too

* Remove zstd version suffix

Prevents a warning shown each time Rust Analyzer is used to check the
code.

Co-authored-by: RumovZ <gp5glkw78@relay.firefox.com>
2022-03-17 15:11:23 +10:00
Henrik Giesel
2d00b6659f
Fix add-on usages of addMedia (#1721)
* Expose old .addMedia again, and use a new resolve_media for built-in uses

* Add an explaining comment for addMedia

* Add some docstrings (dae)
2022-03-16 10:29:06 +10:00
Damien Elmes
02ba50f277 Update translations 2022-03-15 16:52:15 +10:00
Damien Elmes
ed9476c856 Update Rust deps
Primarily for https://blog.rust-lang.org/2022/03/08/cve-2022-24713.html
2022-03-15 16:51:52 +10:00
RumovZ
e759885734
Backend colpkg exporting (#1719)
* Implement colpkg exporting on backend

* Use exporting logic in backup.rs

* Refactor exporting.rs

* Add backend function to export collection

* Refactor backend/collection.rs

* Use backend for colpkg exporting

* Don't use default zip compression for media

* Add exporting progress

* Refactor media file writing

* Write dummy collections

* Localize dummy collection note

* Minimize dummy db size

* Use `NamedTempFile::new()` instead of `new_in`

* Drop redundant v2 dummy collection

* COLLECTION_VERSION -> PACKAGE_VERSION

* Split `lock_collection()` into two to drop flag

* Expose new colpkg in GUI

* Improve dummy collection message

* Please type checker

* importing-colpkg-too-new -> exporting-...

* Compress the media map in the v3 package (dae)

On collections with lots of media, it can grow into megabytes.

Also return an error in extract_media_file_names(), instead of masking
it as an optional.

* Store media map as a vector in the v3 package (dae)

This compresses better (eg 280kb original, 100kb hashmap, 42kb vec)

In the colpkg import case we don't need random access. When importing
an apkg, we will need to be able to fetch file data for a given media
filename, but the existing map doesn't help us there, as we need
filename->index, not index->filename.

* Ensure folders in the media dir don't break the file mapping (dae)
2022-03-15 16:48:02 +10:00
Bruce Harris
d7a101827a
Extend maximum answer time... (#1698)
* Extend maximum answer time...

Previously the time allowed to answer a question was capped at 10 minutes.
While this makes sense for fact recall, it limits the utility of Anki when
used for solving problems that can take more time to work through. This
extends the maximum answer time to 2 hours, which seems to be a reasonable
upper limit for solving a math or algorithm question.

* Add warning when max answer time exceeds 10 minutes

* Move warning below input field
2022-03-15 10:06:45 +10:00
Henrik Giesel
1ec3741934
Fix outstanding tag editor issues (#1717)
* Start using WithFloating for SelectedTagBadge

* Adjust arrow on WithFloating for all directions

* Move TagOptionsBadge to its own sub directory

* Show autocomplete menu via WithFloating

* Have WithFloating return asReference instead of initializing its own reference element

* Add html: overflow: hidden for editor

* Replace ButtonToolbar with generic div

* Move scroll logic into autocomplete item + restrict Popover width to 95vw

* Fix autocomplete menu after pressing enter after selecting

- should not trigger an autocomplete choose

* Overlap TagInput perfectly with Tag

* Satisfy formatter

* Fix autocompletion item scrolling too much

* Remove unused Tag.svelte focusable prop

* Remove console.log

* Fix floating arrow is a diamond in dark mode

* Set autocompletion menu to 80vw
2022-03-11 15:48:49 +10:00
Damien Elmes
2d6dd0630f Switch card.original_position to a proto3 optional 2022-03-10 20:48:26 +10:00
Damien Elmes
99cb6c616e Fix intermittent anki/backend_pb2.py build error on Windows
The _pb2 files are built for both the host and target architectures
(which seems superfluous - we may be able to fix that in the future).
Our script wrote the files into the build folder and then moved them
into the correct place, but because builds are not sandboxed on Windows,
the two actions were racy, and could cause each other to fail. Solved
by writing the files directly into their target locations.
2022-03-10 17:24:23 +10:00
Damien Elmes
f3e81c8a95 Move custom study tag and limit gathering+saving into the backend
Ideally this would have been in beta 6 :-) No add-ons appear to be
using customstudy.py/taglimit.py though, so it should hopefully not be
disruptive.

In the earlier custom study changes, we didn't get around to addressing
issue #1136. Now instead of trying to determine the maximum increase
to allow (which doesn't work correctly with nested decks), we just
present the total available to the user again, and let them decide. There's
plenty of room for improvement here still, but further work here might
be better done once we look into decoupling deck limits from deck presets.

Tags and available cards are fetched prior to showing the dialog now,
and will show a progress dialog if things take a while.

Tags are stored in an aux var now, so they don't inflate the deck
object size.
2022-03-10 16:23:03 +10:00
Damien Elmes
d1a43a2d42 Invoke mypy from within runfiles
tools/mypy-watch now prints .py paths relative to the workspace root,
which makes it easy to click on them to jump to the relevant file/line
in VS Code.
2022-03-10 12:02:04 +10:00
Damien Elmes
a495fbaa8d Try again if dmg detach fails 2022-03-09 19:01:20 +10:00
Damien Elmes
742e4a5cd2 Make build step optional in Linux docker containers 2022-03-09 18:20:13 +10:00
Damien Elmes
e7f99356e9 Update translations 2022-03-09 18:06:58 +10:00
RumovZ
b9c3b12f71
Optionally restore original position and reset counts when forgetting (#1714)
* Add forget prompt with options

- Restore original position
- Reset reps and lapses

* Restore position when resetting for export

* Add config context to avoid passing keys

* Add routine to fetch defaults; use method-specific enum (dae)

* Keep original position by default (dae)

* Fix code completion for forget dialog (dae)

Needs to be a symbolic link to the generated file
2022-03-09 16:51:41 +10:00
RumovZ
c21e6e2b97
Disable full screen on Windows with OpenGL (#1715) 2022-03-09 14:21:54 +10:00
Damien Elmes
e6028cd9a4 Preserve old translation
7f076ee92a..9f37a0245c (r820447434)
2022-03-07 18:14:12 +10:00
RumovZ
f3c8857421
Backups (#1685)
* Add zstd dep

* Implement backend backup with zstd

* Implement backup thinning

* Write backup meta

* Use new file ending anki21b

* Asynchronously backup on collection close in Rust

* Revert "Add zstd dep"

This reverts commit 3fcb2141d2be15f907269d13275c41971431385c.

* Add zstd again

* Take backup col path from col struct

* Fix formatting

* Implement backup restoring on backend

* Normalize restored media file names

* Refactor `extract_legacy_data()`

A bit cumbersome due to borrowing rules.

* Refactor

* Make thinning calendar-based and gradual

* Consider last kept backups of previous stages

* Import full apkgs and colpkgs with backend

* Expose new backup settings

* Test `BackupThinner` and make it deterministic

* Mark backup_path when closing optional

* Delete leaky timer

* Add progress updates for restoring media

* Write restored collection to tempfile first

* Do collection compression in the background thread

This has us currently storing an uncompressed and compressed copy of
the collection in memory (not ideal), but means the collection can be
closed without waiting for compression to complete. On a large collection,
this takes a close and reopen from about 0.55s to about 0.07s. The old
backup code for comparison: about 0.35s for compression off, about
8.5s for zip compression.

* Use multithreading in zstd compression

On my system, this reduces the compression time of a large collection
from about 0.55s to 0.08s.

* Stream compressed collection data into zip file

* Tweak backup explanation

+ Fix incorrect tab order for ignore accents option

* Decouple restoring backup and full import

In the first case, no profile is opened, unless the new collection
succeeds to load.
In the second case, either the old collection is reloaded or the new one
is loaded.

* Fix number gap in Progress message

* Don't revert backup when media fails but report it

* Tweak error flow

* Remove native BackupLimits enum

* Fix type annotation

* Add thinning test for whole year

* Satisfy linter

* Await async backup to finish

* Move restart disclaimer out of backup tab

Should be visible regardless of the current tab.

* Write restored collection in chunks

* Refactor

* Write media in chunks and refactor

* Log error if removing file fails

* join_backup_task -> await_backup_completion

* Refactor backup.rs

* Refactor backup meta and collection extraction

* Fix wrong error being returned

* Call sync_all() on new collection

* Add ImportError

* Store logger in Backend, instead of creating one on demand

init_backend() accepts a Logger rather than a log file, to allow other
callers to customize the logger if they wish.

In the future we may want to explore using the tracing crate as an
alternative; it's a bit more ergonomic, as a logger doesn't need to be
passed around, and it plays more nicely with async code.

* Sync file contents prior to rename; sync folder after rename.

* Limit backup creation to once per 30 min

* Use zstd::stream::copy_decode

* Make importing abortable

* Don't revert if backup media is aborted

* Set throttle implicitly

* Change force flag to minimum_backup_interval

* Don't attempt to open folders on Windows

* Join last backup thread before starting new one

Also refactor.

* Disable auto sync and backup when restoring again

* Force backup on full download

* Include the reason why a media file import failed, and the file path

- Introduce a FileIoError that contains a string representation of
the underlying I/O error, and an associated path. There are a few
places in the code where we're currently manually including the filename
in a custom error message, and this is a step towards a more consistent
approach (but we may be better served with a more general approach in
the future similar to Anyhow's .context())
- Move the error message into importing.ftl, as it's a bit neater
when error messages live in the same file as the rest of the messages
associated with some functionality.

* Fix importing of media files

* Minor wording tweaks

* Save an allocation

I18n strings with replacements are already strings, so we can skip the
extra allocation. Not that it matters here at all.

* Terminate import if file missing from archive

If a third-party tool is creating invalid archives, the user should know
about it. This should be rare, so I did not attempt to make it
translatable.

* Skip multithreaded compression on small collections

Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-03-07 15:11:31 +10:00
Damien Elmes
864904729b Fix an intermittent failure in one of the Rust unit tests 2022-03-07 14:18:17 +10:00
Damien Elmes
b8ce480f61 Don't warn about missing .py files in VS Code
Our _pb2.py files are available at runtime, and we only need the .pyi
files for editing.
2022-03-07 11:54:41 +10:00
RumovZ
8504bd67ed
Fix bury count (more) (#1712)
* Fix bury count for negative values

* Enhance bury count tooltip

* Please type checker
2022-03-07 11:03:14 +10:00
Brayan Oliveira
b403f20cae
Fix html_media_regexps (#1710)
* Fix html_media_regexps

* Update CONTRIBUTORS
2022-03-07 11:02:30 +10:00
Henrik Giesel
202207b765
Update prettier-plugin-svelte (#1711) 2022-03-05 14:27:47 +10:00
RumovZ
5426164ebf
Add regex tag search (#1707) 2022-03-04 18:43:27 +10:00
Damien Elmes
4066518808 Clean up webview after fields dialog closed
https://github.com/ankitects/anki/pull/1706#issuecomment-1058056161
2022-03-04 17:23:28 +10:00
Henrik Giesel
76bdfb3899
Add a flag to allow loading the fields dialog in a web view (#1706)
* Load fields_web from fields.py if appropriate flag is set

* Add FieldsPage as entry for new fields view

* Pass mypy

* Fix pylint

* Fix fields_web in Qt5 (dae)

May not be related to the CI error, but required for compatibility
with Qt5.
2022-03-03 22:45:26 +10:00
Damien Elmes
a06036b8aa Disable buried counts on overview when v2 scheduler in use
https://forums.ankiweb.net/t/fatal-bug-anki-2-1-50-beta5-buried-bug/17969
2022-03-03 20:12:54 +10:00
Damien Elmes
2c80bcb571 Update to latest rules_rust, which unblocks the tokio upgrade
Rust version has been pinned, as 1.59 gets stuck compiling pyo3_macros,
and appears to have other issues:
https://github.com/rust-lang/rust/issues/94390
2022-03-03 19:48:17 +10:00