anki/qt/aqt/progress.py

350 lines
12 KiB
Python
Raw Normal View History

2019-02-05 04:59:03 +01:00
# Copyright: Ankitects Pty Ltd and contributors
# License: GNU AGPL, version 3 or later; http://www.gnu.org/licenses/agpl.html
2020-02-10 08:58:54 +01:00
from __future__ import annotations
import time
Add apkg import/export on backend (#1743) * Add apkg export on backend * Filter out missing media-paths at write time * Make TagMatcher::new() infallible * Gather export data instead of copying directly * Revert changes to rslib/src/tags/ * Reuse filename_is_safe/check_filename_safe() * Accept func to produce MediaIter in export_apkg() * Only store file folder once in MediaIter * Use temporary tables for gathering export_apkg() now accepts a search instead of a deck id. Decks are gathered according to the matched notes' cards. * Use schedule_as_new() to reset cards * ExportData → ExchangeData * Ignore ascii case when filtering system tags * search_notes_cards_into_table → search_cards_of_notes_into_table * Start on apkg importing on backend * Fix due dates in days for apkg export * Refactor import-export/package - Move media and meta code into appropriate modules. - Normalize/check for normalization when deserializing media entries. * Add SafeMediaEntry for deserialized MediaEntries * Prepare media based on checksums - Ensure all existing media files are hashed. - Hash incoming files during preparation to detect conflicts. - Uniquify names of conflicting files with hash (not notetype id). - Mark media files as used while importing notes. - Finally copy used media. * Handle encoding in `replace_media_refs()` * Add trait to keep down cow boilerplate * Add notetypes immediately instaed of preparing * Move target_col into Context * Add notes immediately instaed of preparing * Note id, not guid of conflicting notes * Add import_decks() * decks_configs → deck_configs * Add import_deck_configs() * Add import_cards(), import_revlog() * Use dyn instead of generic for media_fn Otherwise, would have to pass None with type annotation in the default case. * Fix signature of import_apkg() * Fix search_cards_of_notes_into_table() * Test new functions in text.rs * Add roundtrip test for apkg (stub) * Keep source id of imported cards (or skip) * Keep source ids of imported revlog (or skip) * Try to keep source ids of imported notes * Make adding notetype with id undoable * Wrap apkg import in transaction * Keep source ids of imported deck configs (or skip) * Handle card due dates and original due/did * Fix importing cards/revlog Card ids are manually uniquified. * Factor out card importing * Refactor card and revlog importing * Factor out card importing Also handle missing parents . * Factor out note importing * Factor out media importing * Maybe upgrade scheduler of apkg * Fix parent deck gathering * Unconditionally import static media * Fix deck importing edge cases Test those edge cases, and add some global test helpers. * Test note importing * Let import_apkg() take a progress func * Expand roundtrip apkg test * Use fat pointer to avoid propogating generics * Fix progress_fn type * Expose apkg export/import on backend * Return note log when importing apkg * Fix archived collection name on apkg import * Add CollectionOpWithBackendProgress * Fix wrong Interrupted Exception being checked * Add ClosedCollectionOp * Add note ids to log and strip HTML * Update progress when checking incoming media too * Conditionally enable new importing in GUI * Fix all_checksums() for media import Entries of deleted files are nulled, not removed. * Make apkg exporting on backend abortable * Return number of notes imported from apkg * Fix exception printing for QueryOp as well * Add QueryOpWithBackendProgress Also support backend exporting progress. * Expose new apkg and colpkg exporting * Open transaction in insert_data() Was slowing down exporting by several orders of magnitude. * Handle zstd-compressed apkg * Add legacy arg to ExportAnkiPackage Currently not exposed on the frontend * Remove unused import in proto file * Add symlink for typechecking of import_export_pb2 * Avoid kwargs in pb message creation, so typechecking is not lost Protobuf's behaviour is rather subtle and I had to dig through the docs to figure it out: set a field on a submessage to automatically assign the submessage to the parent, or call SetInParent() to persist a default version of the field you specified. * Avoid re-exporting protobuf msgs we only use internally * Stop after one test failure mypy often fails much faster than pylint * Avoid an extra allocation when extracting media checksums * Update progress after prepare_media() finishes Otherwise the bulk of the import ends up being shown as "Checked: 0" in the progress window. * Show progress of note imports Note import is the slowest part, so showing progress here makes the UI feel more responsive. * Reset filtered decks at import time Before this change, filtered decks exported with scheduling remained filtered on import, and maybe_remove_from_filtered_deck() moved cards into them as their home deck, leading to errors during review. We may still want to provide a way to preserve filtered decks on import, but to do that we'll need to ensure we don't rewrite the home decks of cards, and we'll need to ensure the home decks are included as part of the import (or give an error if they're not). https://github.com/ankitects/anki/pull/1743/files#r839346423 * Fix a corner-case where due dates were shifted by a day This issue existed in the old Python code as well. We need to include the user's UTC offset in the exported file, or days_elapsed falls back on the v1 cutoff calculation, which may be a day earlier or later than the v2 calculation. * Log conflicting note in remapped nt case * take_fields() → into_fields() * Alias `[u8; 20]` with `Sha1Hash` * Truncate logged fields * Rework apkg note import tests - Use macros for more helpful errors. - Split monolith into unit tests. - Fix some unknown error with the previous test along the way. (Was failing after 969484de4388d225c9f17d94534b3ba0094c3568.) * Fix sorting of imported decks Also adjust the test, so it fails without the patch. It was only passing before, because the parent deck happened to come before the inconsistently capitalised child alphabetically. But we want all parent decks to be imported before their child decks, so their children can adopt their capitalisation. * target[_id]s → existing_card[_id]s * export_collection_extracting_media() → ... export_into_collection_file() * target_already_exists→card_ordinal_already_exists * Add search_cards_of_notes_into_table.sql * Imrove type of apkg export selector/limit * Remove redundant call to mod_schema() * Parent tooltips to mw * Fix a crash when truncating note text String::truncate() is a bit of a footgun, and I've hit this before too :-) * Remove ExportLimit in favour of separate classes * Remove OpWithBackendProgress and ClosedCollectionOp Backend progress logic is now in ProgressManager. QueryOp can be used for running on closed collection. Also fix aborting of colpkg exports, which slipped through in #1817. * Tidy up import log * Avoid QDialog.exec() * Default to excluding scheuling for deck list deck * Use IncrementalProgress in whole import_export code * Compare checksums when importing colpkgs * Avoid registering changes if hashes are not needed * ImportProgress::Collection → ImportProgress::File * Make downgrading apkgs depend on meta version * Generalise IncrementableProgress And use it in entire import_export code instead. * Fix type complexity lint * Take count_map for IncrementableProgress::get_inner * Replace import/export env with Shift click * Accept all args from update() for backend progress * Pass fields of ProgressUpdate explicitly * Move update_interval into IncrementableProgress * Outsource incrementing into Incrementor * Mutate ProgressUpdate in progress_update callback * Switch import/export legacy toggle to profile setting Shift would have been nice, but the existing shortcuts complicate things. If the user triggers an import with ctrl+shift+i, shift is unlikely to have been released by the time our code runs, meaning the user accidentally triggers the new code. We could potentially wait a while before bringing up the dialog, but then we're forced to guess at how long it will take the user to release the key. One alternative would be to use alt instead of shift, but then we need to trigger our shortcut when that key is pressed as well, and it could potentially cause a conflict with an add-on that already uses that combination. * Show extension in export dialog * Continue to provide separate options for schema 11+18 colpkg export * Default to colpkg export when using File>Export * Improve appearance of combo boxes when switching between apkg/colpkg + Deal with long deck names * Convert newlines to spaces when showing fields from import Ensures each imported note appears on a separate line * Don't separate total note count from the other summary lines This may come down to personal preference, but I feel the other counts are equally as important, and separating them feels like it makes it a bit easier to ignore them. * Fix 'deck not normal' error when importing a filtered deck for the 2nd time * Fix [Identical] being shown on first import * Revert "Continue to provide separate options for schema 11+18 colpkg export" This reverts commit 8f0b2c175f4794d642823b60414d142a12768441. Will use a different approach * Move legacy support into a separate exporter option; add to apkg export * Adjust 'too new' message to also apply to .apkg import case * Show a better message when attempting to import new apkg into old code Previously the user could end seeing a message like: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte Unfortunately we can't retroactively fix this for older clients. * Hide legacy support option in older exporting screen * Reflect change from paths to fnames in type & name * Make imported decks normal at once Then skip special casing in update_deck(). Also skip updating description if new one is empty. Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-05-02 13:12:46 +02:00
from dataclasses import dataclass
2019-12-20 10:19:03 +01:00
import aqt.forms
from anki._legacy import print_deprecation_warning
Add apkg import/export on backend (#1743) * Add apkg export on backend * Filter out missing media-paths at write time * Make TagMatcher::new() infallible * Gather export data instead of copying directly * Revert changes to rslib/src/tags/ * Reuse filename_is_safe/check_filename_safe() * Accept func to produce MediaIter in export_apkg() * Only store file folder once in MediaIter * Use temporary tables for gathering export_apkg() now accepts a search instead of a deck id. Decks are gathered according to the matched notes' cards. * Use schedule_as_new() to reset cards * ExportData → ExchangeData * Ignore ascii case when filtering system tags * search_notes_cards_into_table → search_cards_of_notes_into_table * Start on apkg importing on backend * Fix due dates in days for apkg export * Refactor import-export/package - Move media and meta code into appropriate modules. - Normalize/check for normalization when deserializing media entries. * Add SafeMediaEntry for deserialized MediaEntries * Prepare media based on checksums - Ensure all existing media files are hashed. - Hash incoming files during preparation to detect conflicts. - Uniquify names of conflicting files with hash (not notetype id). - Mark media files as used while importing notes. - Finally copy used media. * Handle encoding in `replace_media_refs()` * Add trait to keep down cow boilerplate * Add notetypes immediately instaed of preparing * Move target_col into Context * Add notes immediately instaed of preparing * Note id, not guid of conflicting notes * Add import_decks() * decks_configs → deck_configs * Add import_deck_configs() * Add import_cards(), import_revlog() * Use dyn instead of generic for media_fn Otherwise, would have to pass None with type annotation in the default case. * Fix signature of import_apkg() * Fix search_cards_of_notes_into_table() * Test new functions in text.rs * Add roundtrip test for apkg (stub) * Keep source id of imported cards (or skip) * Keep source ids of imported revlog (or skip) * Try to keep source ids of imported notes * Make adding notetype with id undoable * Wrap apkg import in transaction * Keep source ids of imported deck configs (or skip) * Handle card due dates and original due/did * Fix importing cards/revlog Card ids are manually uniquified. * Factor out card importing * Refactor card and revlog importing * Factor out card importing Also handle missing parents . * Factor out note importing * Factor out media importing * Maybe upgrade scheduler of apkg * Fix parent deck gathering * Unconditionally import static media * Fix deck importing edge cases Test those edge cases, and add some global test helpers. * Test note importing * Let import_apkg() take a progress func * Expand roundtrip apkg test * Use fat pointer to avoid propogating generics * Fix progress_fn type * Expose apkg export/import on backend * Return note log when importing apkg * Fix archived collection name on apkg import * Add CollectionOpWithBackendProgress * Fix wrong Interrupted Exception being checked * Add ClosedCollectionOp * Add note ids to log and strip HTML * Update progress when checking incoming media too * Conditionally enable new importing in GUI * Fix all_checksums() for media import Entries of deleted files are nulled, not removed. * Make apkg exporting on backend abortable * Return number of notes imported from apkg * Fix exception printing for QueryOp as well * Add QueryOpWithBackendProgress Also support backend exporting progress. * Expose new apkg and colpkg exporting * Open transaction in insert_data() Was slowing down exporting by several orders of magnitude. * Handle zstd-compressed apkg * Add legacy arg to ExportAnkiPackage Currently not exposed on the frontend * Remove unused import in proto file * Add symlink for typechecking of import_export_pb2 * Avoid kwargs in pb message creation, so typechecking is not lost Protobuf's behaviour is rather subtle and I had to dig through the docs to figure it out: set a field on a submessage to automatically assign the submessage to the parent, or call SetInParent() to persist a default version of the field you specified. * Avoid re-exporting protobuf msgs we only use internally * Stop after one test failure mypy often fails much faster than pylint * Avoid an extra allocation when extracting media checksums * Update progress after prepare_media() finishes Otherwise the bulk of the import ends up being shown as "Checked: 0" in the progress window. * Show progress of note imports Note import is the slowest part, so showing progress here makes the UI feel more responsive. * Reset filtered decks at import time Before this change, filtered decks exported with scheduling remained filtered on import, and maybe_remove_from_filtered_deck() moved cards into them as their home deck, leading to errors during review. We may still want to provide a way to preserve filtered decks on import, but to do that we'll need to ensure we don't rewrite the home decks of cards, and we'll need to ensure the home decks are included as part of the import (or give an error if they're not). https://github.com/ankitects/anki/pull/1743/files#r839346423 * Fix a corner-case where due dates were shifted by a day This issue existed in the old Python code as well. We need to include the user's UTC offset in the exported file, or days_elapsed falls back on the v1 cutoff calculation, which may be a day earlier or later than the v2 calculation. * Log conflicting note in remapped nt case * take_fields() → into_fields() * Alias `[u8; 20]` with `Sha1Hash` * Truncate logged fields * Rework apkg note import tests - Use macros for more helpful errors. - Split monolith into unit tests. - Fix some unknown error with the previous test along the way. (Was failing after 969484de4388d225c9f17d94534b3ba0094c3568.) * Fix sorting of imported decks Also adjust the test, so it fails without the patch. It was only passing before, because the parent deck happened to come before the inconsistently capitalised child alphabetically. But we want all parent decks to be imported before their child decks, so their children can adopt their capitalisation. * target[_id]s → existing_card[_id]s * export_collection_extracting_media() → ... export_into_collection_file() * target_already_exists→card_ordinal_already_exists * Add search_cards_of_notes_into_table.sql * Imrove type of apkg export selector/limit * Remove redundant call to mod_schema() * Parent tooltips to mw * Fix a crash when truncating note text String::truncate() is a bit of a footgun, and I've hit this before too :-) * Remove ExportLimit in favour of separate classes * Remove OpWithBackendProgress and ClosedCollectionOp Backend progress logic is now in ProgressManager. QueryOp can be used for running on closed collection. Also fix aborting of colpkg exports, which slipped through in #1817. * Tidy up import log * Avoid QDialog.exec() * Default to excluding scheuling for deck list deck * Use IncrementalProgress in whole import_export code * Compare checksums when importing colpkgs * Avoid registering changes if hashes are not needed * ImportProgress::Collection → ImportProgress::File * Make downgrading apkgs depend on meta version * Generalise IncrementableProgress And use it in entire import_export code instead. * Fix type complexity lint * Take count_map for IncrementableProgress::get_inner * Replace import/export env with Shift click * Accept all args from update() for backend progress * Pass fields of ProgressUpdate explicitly * Move update_interval into IncrementableProgress * Outsource incrementing into Incrementor * Mutate ProgressUpdate in progress_update callback * Switch import/export legacy toggle to profile setting Shift would have been nice, but the existing shortcuts complicate things. If the user triggers an import with ctrl+shift+i, shift is unlikely to have been released by the time our code runs, meaning the user accidentally triggers the new code. We could potentially wait a while before bringing up the dialog, but then we're forced to guess at how long it will take the user to release the key. One alternative would be to use alt instead of shift, but then we need to trigger our shortcut when that key is pressed as well, and it could potentially cause a conflict with an add-on that already uses that combination. * Show extension in export dialog * Continue to provide separate options for schema 11+18 colpkg export * Default to colpkg export when using File>Export * Improve appearance of combo boxes when switching between apkg/colpkg + Deal with long deck names * Convert newlines to spaces when showing fields from import Ensures each imported note appears on a separate line * Don't separate total note count from the other summary lines This may come down to personal preference, but I feel the other counts are equally as important, and separating them feels like it makes it a bit easier to ignore them. * Fix 'deck not normal' error when importing a filtered deck for the 2nd time * Fix [Identical] being shown on first import * Revert "Continue to provide separate options for schema 11+18 colpkg export" This reverts commit 8f0b2c175f4794d642823b60414d142a12768441. Will use a different approach * Move legacy support into a separate exporter option; add to apkg export * Adjust 'too new' message to also apply to .apkg import case * Show a better message when attempting to import new apkg into old code Previously the user could end seeing a message like: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte Unfortunately we can't retroactively fix this for older clients. * Hide legacy support option in older exporting screen * Reflect change from paths to fnames in type & name * Make imported decks normal at once Then skip special casing in update_deck(). Also skip updating description if new one is empty. Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-05-02 13:12:46 +02:00
from anki.collection import Progress
2019-12-20 10:19:03 +01:00
from aqt.qt import *
2021-03-26 04:48:26 +01:00
from aqt.utils import disable_help_button, tr
# Progress info
##########################################################################
2019-12-23 01:34:10 +01:00
class ProgressManager:
def __init__(self, mw: aqt.AnkiQt) -> None:
self.mw = mw
self.app = mw.app
self.inDB = False
self.blockUpdates = False
self._show_timer: QTimer | None = None
self._busy_cursor_timer: QTimer | None = None
self._win: ProgressDialog | None = None
self._levels = 0
Add apkg import/export on backend (#1743) * Add apkg export on backend * Filter out missing media-paths at write time * Make TagMatcher::new() infallible * Gather export data instead of copying directly * Revert changes to rslib/src/tags/ * Reuse filename_is_safe/check_filename_safe() * Accept func to produce MediaIter in export_apkg() * Only store file folder once in MediaIter * Use temporary tables for gathering export_apkg() now accepts a search instead of a deck id. Decks are gathered according to the matched notes' cards. * Use schedule_as_new() to reset cards * ExportData → ExchangeData * Ignore ascii case when filtering system tags * search_notes_cards_into_table → search_cards_of_notes_into_table * Start on apkg importing on backend * Fix due dates in days for apkg export * Refactor import-export/package - Move media and meta code into appropriate modules. - Normalize/check for normalization when deserializing media entries. * Add SafeMediaEntry for deserialized MediaEntries * Prepare media based on checksums - Ensure all existing media files are hashed. - Hash incoming files during preparation to detect conflicts. - Uniquify names of conflicting files with hash (not notetype id). - Mark media files as used while importing notes. - Finally copy used media. * Handle encoding in `replace_media_refs()` * Add trait to keep down cow boilerplate * Add notetypes immediately instaed of preparing * Move target_col into Context * Add notes immediately instaed of preparing * Note id, not guid of conflicting notes * Add import_decks() * decks_configs → deck_configs * Add import_deck_configs() * Add import_cards(), import_revlog() * Use dyn instead of generic for media_fn Otherwise, would have to pass None with type annotation in the default case. * Fix signature of import_apkg() * Fix search_cards_of_notes_into_table() * Test new functions in text.rs * Add roundtrip test for apkg (stub) * Keep source id of imported cards (or skip) * Keep source ids of imported revlog (or skip) * Try to keep source ids of imported notes * Make adding notetype with id undoable * Wrap apkg import in transaction * Keep source ids of imported deck configs (or skip) * Handle card due dates and original due/did * Fix importing cards/revlog Card ids are manually uniquified. * Factor out card importing * Refactor card and revlog importing * Factor out card importing Also handle missing parents . * Factor out note importing * Factor out media importing * Maybe upgrade scheduler of apkg * Fix parent deck gathering * Unconditionally import static media * Fix deck importing edge cases Test those edge cases, and add some global test helpers. * Test note importing * Let import_apkg() take a progress func * Expand roundtrip apkg test * Use fat pointer to avoid propogating generics * Fix progress_fn type * Expose apkg export/import on backend * Return note log when importing apkg * Fix archived collection name on apkg import * Add CollectionOpWithBackendProgress * Fix wrong Interrupted Exception being checked * Add ClosedCollectionOp * Add note ids to log and strip HTML * Update progress when checking incoming media too * Conditionally enable new importing in GUI * Fix all_checksums() for media import Entries of deleted files are nulled, not removed. * Make apkg exporting on backend abortable * Return number of notes imported from apkg * Fix exception printing for QueryOp as well * Add QueryOpWithBackendProgress Also support backend exporting progress. * Expose new apkg and colpkg exporting * Open transaction in insert_data() Was slowing down exporting by several orders of magnitude. * Handle zstd-compressed apkg * Add legacy arg to ExportAnkiPackage Currently not exposed on the frontend * Remove unused import in proto file * Add symlink for typechecking of import_export_pb2 * Avoid kwargs in pb message creation, so typechecking is not lost Protobuf's behaviour is rather subtle and I had to dig through the docs to figure it out: set a field on a submessage to automatically assign the submessage to the parent, or call SetInParent() to persist a default version of the field you specified. * Avoid re-exporting protobuf msgs we only use internally * Stop after one test failure mypy often fails much faster than pylint * Avoid an extra allocation when extracting media checksums * Update progress after prepare_media() finishes Otherwise the bulk of the import ends up being shown as "Checked: 0" in the progress window. * Show progress of note imports Note import is the slowest part, so showing progress here makes the UI feel more responsive. * Reset filtered decks at import time Before this change, filtered decks exported with scheduling remained filtered on import, and maybe_remove_from_filtered_deck() moved cards into them as their home deck, leading to errors during review. We may still want to provide a way to preserve filtered decks on import, but to do that we'll need to ensure we don't rewrite the home decks of cards, and we'll need to ensure the home decks are included as part of the import (or give an error if they're not). https://github.com/ankitects/anki/pull/1743/files#r839346423 * Fix a corner-case where due dates were shifted by a day This issue existed in the old Python code as well. We need to include the user's UTC offset in the exported file, or days_elapsed falls back on the v1 cutoff calculation, which may be a day earlier or later than the v2 calculation. * Log conflicting note in remapped nt case * take_fields() → into_fields() * Alias `[u8; 20]` with `Sha1Hash` * Truncate logged fields * Rework apkg note import tests - Use macros for more helpful errors. - Split monolith into unit tests. - Fix some unknown error with the previous test along the way. (Was failing after 969484de4388d225c9f17d94534b3ba0094c3568.) * Fix sorting of imported decks Also adjust the test, so it fails without the patch. It was only passing before, because the parent deck happened to come before the inconsistently capitalised child alphabetically. But we want all parent decks to be imported before their child decks, so their children can adopt their capitalisation. * target[_id]s → existing_card[_id]s * export_collection_extracting_media() → ... export_into_collection_file() * target_already_exists→card_ordinal_already_exists * Add search_cards_of_notes_into_table.sql * Imrove type of apkg export selector/limit * Remove redundant call to mod_schema() * Parent tooltips to mw * Fix a crash when truncating note text String::truncate() is a bit of a footgun, and I've hit this before too :-) * Remove ExportLimit in favour of separate classes * Remove OpWithBackendProgress and ClosedCollectionOp Backend progress logic is now in ProgressManager. QueryOp can be used for running on closed collection. Also fix aborting of colpkg exports, which slipped through in #1817. * Tidy up import log * Avoid QDialog.exec() * Default to excluding scheuling for deck list deck * Use IncrementalProgress in whole import_export code * Compare checksums when importing colpkgs * Avoid registering changes if hashes are not needed * ImportProgress::Collection → ImportProgress::File * Make downgrading apkgs depend on meta version * Generalise IncrementableProgress And use it in entire import_export code instead. * Fix type complexity lint * Take count_map for IncrementableProgress::get_inner * Replace import/export env with Shift click * Accept all args from update() for backend progress * Pass fields of ProgressUpdate explicitly * Move update_interval into IncrementableProgress * Outsource incrementing into Incrementor * Mutate ProgressUpdate in progress_update callback * Switch import/export legacy toggle to profile setting Shift would have been nice, but the existing shortcuts complicate things. If the user triggers an import with ctrl+shift+i, shift is unlikely to have been released by the time our code runs, meaning the user accidentally triggers the new code. We could potentially wait a while before bringing up the dialog, but then we're forced to guess at how long it will take the user to release the key. One alternative would be to use alt instead of shift, but then we need to trigger our shortcut when that key is pressed as well, and it could potentially cause a conflict with an add-on that already uses that combination. * Show extension in export dialog * Continue to provide separate options for schema 11+18 colpkg export * Default to colpkg export when using File>Export * Improve appearance of combo boxes when switching between apkg/colpkg + Deal with long deck names * Convert newlines to spaces when showing fields from import Ensures each imported note appears on a separate line * Don't separate total note count from the other summary lines This may come down to personal preference, but I feel the other counts are equally as important, and separating them feels like it makes it a bit easier to ignore them. * Fix 'deck not normal' error when importing a filtered deck for the 2nd time * Fix [Identical] being shown on first import * Revert "Continue to provide separate options for schema 11+18 colpkg export" This reverts commit 8f0b2c175f4794d642823b60414d142a12768441. Will use a different approach * Move legacy support into a separate exporter option; add to apkg export * Adjust 'too new' message to also apply to .apkg import case * Show a better message when attempting to import new apkg into old code Previously the user could end seeing a message like: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte Unfortunately we can't retroactively fix this for older clients. * Hide legacy support option in older exporting screen * Reflect change from paths to fnames in type & name * Make imported decks normal at once Then skip special casing in update_deck(). Also skip updating description if new one is empty. Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-05-02 13:12:46 +02:00
self._backend_timer: QTimer | None = None
# Safer timers
##########################################################################
# Custom timers which avoid firing while a progress dialog is active
# (likely due to some long-running DB operation)
def timer(
self,
ms: int,
func: Callable,
repeat: bool,
requiresCollection: bool = True,
*,
parent: QObject = None,
) -> QTimer:
"""Create and start a standard Anki timer. For an alternative see `single_shot()`.
If the timer fires while a progress window is shown:
- if it is a repeating timer, it will wait the same delay again
- if it is non-repeating, it will try again in 100ms
If requiresCollection is True, the timer will not fire if the
collection has been unloaded. Setting it to False will allow the
timer to fire even when there is no collection, but will still
only fire when there is no current progress dialog.
Issues and alternative
---
The created timer will only be destroyed when `parent` is destroyed.
This can cause memory leaks, because anything captured by `func` isn't freed either.
If there is no QObject that will get destroyed reasonably soon, and you have to
pass `mw`, you should call `deleteLater()` on the returned QTimer as soon as
it's served its purpose, or use `single_shot()`.
Also note that you may not be able to pass an adequate parent, if you want to
make a callback after a widget closes. If you passed that widget, the timer
would get destroyed before it could fire.
"""
if parent is None:
print_deprecation_warning(
"to avoid memory leaks, pass an appropriate parent to progress.timer()"
" or use progress.single_shot()"
)
parent = self.mw
qtimer = QTimer(parent)
if not repeat:
qtimer.setSingleShot(True)
qconnect(qtimer.timeout, self._get_handler(func, repeat, requiresCollection))
qtimer.start(ms)
return qtimer
def single_shot(
self,
ms: int,
func: Callable[[], None],
requires_collection: bool = True,
) -> None:
"""Create and start a one-off Anki timer. For an alternative and more
documentation, see `timer()`.
Issues and alternative
---
`single_shot()` cleans itself up, so a passed closure won't leak any memory.
However, if `func` references a QObject other than `mw`, which gets deleted before the
timer fires, an Exception is raised. To avoid this, either use `timer()` passing
that object as the parent, or check in `func` with `sip.isdeleted(object)` if
it still exists.
On the other hand, if a widget is supposed to make an external callback after it closes,
you likely want to use `single_shot()`, which will fire even if the calling
widget is already destroyed.
"""
QTimer.singleShot(ms, self._get_handler(func, False, requires_collection))
def _get_handler(
self, func: Callable[[], None], repeat: bool, requires_collection: bool
) -> Callable[[], None]:
2021-02-01 14:28:21 +01:00
def handler() -> None:
if requires_collection and not self.mw.col:
# no current collection; timer is no longer valid
print(f"Ignored progress func as collection unloaded: {repr(func)}")
return
if not self._levels:
# no current progress; safe to fire
func()
else:
if repeat:
# skip this time; we'll fire again
pass
else:
# retry in 100ms
self.single_shot(100, func, requires_collection)
2019-12-23 01:34:10 +01:00
return handler
# Creating progress dialogs
##########################################################################
2020-02-10 08:58:54 +01:00
def start(
self,
max: int = 0,
min: int = 0,
label: str | None = None,
parent: QWidget | None = None,
immediate: bool = False,
) -> ProgressDialog | None:
self._levels += 1
if self._levels > 1:
2020-02-10 08:58:54 +01:00
return None
# setup window
parent = parent or self.app.activeWindow()
if not parent and self.mw.isVisible():
parent = self.mw
2021-03-26 04:48:26 +01:00
label = label or tr.qt_misc_processing()
2020-02-10 08:58:54 +01:00
self._win = ProgressDialog(parent)
self._win.form.progressBar.setMinimum(min)
self._win.form.progressBar.setMaximum(max)
2018-01-14 09:05:43 +01:00
self._win.form.progressBar.setTextVisible(False)
self._win.form.label.setText(label)
self._win.setWindowTitle("Anki")
self._win.setWindowModality(Qt.WindowModality.ApplicationModal)
self._win.setMinimumWidth(300)
self._busy_cursor_timer = QTimer(self.mw)
self._busy_cursor_timer.setSingleShot(True)
self._busy_cursor_timer.start(300)
qconnect(self._busy_cursor_timer.timeout, self._set_busy_cursor)
self._shown: float = 0
self._counter = min
self._min = min
self._max = max
self._firstTime = time.time()
self._show_timer = QTimer(self.mw)
self._show_timer.setSingleShot(True)
self._show_timer.start(immediate and 100 or 600)
qconnect(self._show_timer.timeout, self._on_show_timer)
return self._win
Add apkg import/export on backend (#1743) * Add apkg export on backend * Filter out missing media-paths at write time * Make TagMatcher::new() infallible * Gather export data instead of copying directly * Revert changes to rslib/src/tags/ * Reuse filename_is_safe/check_filename_safe() * Accept func to produce MediaIter in export_apkg() * Only store file folder once in MediaIter * Use temporary tables for gathering export_apkg() now accepts a search instead of a deck id. Decks are gathered according to the matched notes' cards. * Use schedule_as_new() to reset cards * ExportData → ExchangeData * Ignore ascii case when filtering system tags * search_notes_cards_into_table → search_cards_of_notes_into_table * Start on apkg importing on backend * Fix due dates in days for apkg export * Refactor import-export/package - Move media and meta code into appropriate modules. - Normalize/check for normalization when deserializing media entries. * Add SafeMediaEntry for deserialized MediaEntries * Prepare media based on checksums - Ensure all existing media files are hashed. - Hash incoming files during preparation to detect conflicts. - Uniquify names of conflicting files with hash (not notetype id). - Mark media files as used while importing notes. - Finally copy used media. * Handle encoding in `replace_media_refs()` * Add trait to keep down cow boilerplate * Add notetypes immediately instaed of preparing * Move target_col into Context * Add notes immediately instaed of preparing * Note id, not guid of conflicting notes * Add import_decks() * decks_configs → deck_configs * Add import_deck_configs() * Add import_cards(), import_revlog() * Use dyn instead of generic for media_fn Otherwise, would have to pass None with type annotation in the default case. * Fix signature of import_apkg() * Fix search_cards_of_notes_into_table() * Test new functions in text.rs * Add roundtrip test for apkg (stub) * Keep source id of imported cards (or skip) * Keep source ids of imported revlog (or skip) * Try to keep source ids of imported notes * Make adding notetype with id undoable * Wrap apkg import in transaction * Keep source ids of imported deck configs (or skip) * Handle card due dates and original due/did * Fix importing cards/revlog Card ids are manually uniquified. * Factor out card importing * Refactor card and revlog importing * Factor out card importing Also handle missing parents . * Factor out note importing * Factor out media importing * Maybe upgrade scheduler of apkg * Fix parent deck gathering * Unconditionally import static media * Fix deck importing edge cases Test those edge cases, and add some global test helpers. * Test note importing * Let import_apkg() take a progress func * Expand roundtrip apkg test * Use fat pointer to avoid propogating generics * Fix progress_fn type * Expose apkg export/import on backend * Return note log when importing apkg * Fix archived collection name on apkg import * Add CollectionOpWithBackendProgress * Fix wrong Interrupted Exception being checked * Add ClosedCollectionOp * Add note ids to log and strip HTML * Update progress when checking incoming media too * Conditionally enable new importing in GUI * Fix all_checksums() for media import Entries of deleted files are nulled, not removed. * Make apkg exporting on backend abortable * Return number of notes imported from apkg * Fix exception printing for QueryOp as well * Add QueryOpWithBackendProgress Also support backend exporting progress. * Expose new apkg and colpkg exporting * Open transaction in insert_data() Was slowing down exporting by several orders of magnitude. * Handle zstd-compressed apkg * Add legacy arg to ExportAnkiPackage Currently not exposed on the frontend * Remove unused import in proto file * Add symlink for typechecking of import_export_pb2 * Avoid kwargs in pb message creation, so typechecking is not lost Protobuf's behaviour is rather subtle and I had to dig through the docs to figure it out: set a field on a submessage to automatically assign the submessage to the parent, or call SetInParent() to persist a default version of the field you specified. * Avoid re-exporting protobuf msgs we only use internally * Stop after one test failure mypy often fails much faster than pylint * Avoid an extra allocation when extracting media checksums * Update progress after prepare_media() finishes Otherwise the bulk of the import ends up being shown as "Checked: 0" in the progress window. * Show progress of note imports Note import is the slowest part, so showing progress here makes the UI feel more responsive. * Reset filtered decks at import time Before this change, filtered decks exported with scheduling remained filtered on import, and maybe_remove_from_filtered_deck() moved cards into them as their home deck, leading to errors during review. We may still want to provide a way to preserve filtered decks on import, but to do that we'll need to ensure we don't rewrite the home decks of cards, and we'll need to ensure the home decks are included as part of the import (or give an error if they're not). https://github.com/ankitects/anki/pull/1743/files#r839346423 * Fix a corner-case where due dates were shifted by a day This issue existed in the old Python code as well. We need to include the user's UTC offset in the exported file, or days_elapsed falls back on the v1 cutoff calculation, which may be a day earlier or later than the v2 calculation. * Log conflicting note in remapped nt case * take_fields() → into_fields() * Alias `[u8; 20]` with `Sha1Hash` * Truncate logged fields * Rework apkg note import tests - Use macros for more helpful errors. - Split monolith into unit tests. - Fix some unknown error with the previous test along the way. (Was failing after 969484de4388d225c9f17d94534b3ba0094c3568.) * Fix sorting of imported decks Also adjust the test, so it fails without the patch. It was only passing before, because the parent deck happened to come before the inconsistently capitalised child alphabetically. But we want all parent decks to be imported before their child decks, so their children can adopt their capitalisation. * target[_id]s → existing_card[_id]s * export_collection_extracting_media() → ... export_into_collection_file() * target_already_exists→card_ordinal_already_exists * Add search_cards_of_notes_into_table.sql * Imrove type of apkg export selector/limit * Remove redundant call to mod_schema() * Parent tooltips to mw * Fix a crash when truncating note text String::truncate() is a bit of a footgun, and I've hit this before too :-) * Remove ExportLimit in favour of separate classes * Remove OpWithBackendProgress and ClosedCollectionOp Backend progress logic is now in ProgressManager. QueryOp can be used for running on closed collection. Also fix aborting of colpkg exports, which slipped through in #1817. * Tidy up import log * Avoid QDialog.exec() * Default to excluding scheuling for deck list deck * Use IncrementalProgress in whole import_export code * Compare checksums when importing colpkgs * Avoid registering changes if hashes are not needed * ImportProgress::Collection → ImportProgress::File * Make downgrading apkgs depend on meta version * Generalise IncrementableProgress And use it in entire import_export code instead. * Fix type complexity lint * Take count_map for IncrementableProgress::get_inner * Replace import/export env with Shift click * Accept all args from update() for backend progress * Pass fields of ProgressUpdate explicitly * Move update_interval into IncrementableProgress * Outsource incrementing into Incrementor * Mutate ProgressUpdate in progress_update callback * Switch import/export legacy toggle to profile setting Shift would have been nice, but the existing shortcuts complicate things. If the user triggers an import with ctrl+shift+i, shift is unlikely to have been released by the time our code runs, meaning the user accidentally triggers the new code. We could potentially wait a while before bringing up the dialog, but then we're forced to guess at how long it will take the user to release the key. One alternative would be to use alt instead of shift, but then we need to trigger our shortcut when that key is pressed as well, and it could potentially cause a conflict with an add-on that already uses that combination. * Show extension in export dialog * Continue to provide separate options for schema 11+18 colpkg export * Default to colpkg export when using File>Export * Improve appearance of combo boxes when switching between apkg/colpkg + Deal with long deck names * Convert newlines to spaces when showing fields from import Ensures each imported note appears on a separate line * Don't separate total note count from the other summary lines This may come down to personal preference, but I feel the other counts are equally as important, and separating them feels like it makes it a bit easier to ignore them. * Fix 'deck not normal' error when importing a filtered deck for the 2nd time * Fix [Identical] being shown on first import * Revert "Continue to provide separate options for schema 11+18 colpkg export" This reverts commit 8f0b2c175f4794d642823b60414d142a12768441. Will use a different approach * Move legacy support into a separate exporter option; add to apkg export * Adjust 'too new' message to also apply to .apkg import case * Show a better message when attempting to import new apkg into old code Previously the user could end seeing a message like: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte Unfortunately we can't retroactively fix this for older clients. * Hide legacy support option in older exporting screen * Reflect change from paths to fnames in type & name * Make imported decks normal at once Then skip special casing in update_deck(). Also skip updating description if new one is empty. Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-05-02 13:12:46 +02:00
def start_with_backend_updates(
self,
progress_update: Callable[[Progress, ProgressUpdate], None],
start_label: str | None = None,
parent: QWidget | None = None,
) -> None:
self._backend_timer = QTimer()
self._backend_timer.setSingleShot(False)
self._backend_timer.setInterval(100)
if not (dialog := self.start(immediate=True, label=start_label, parent=parent)):
print("Progress dialog already running; aborting will not work")
def on_progress() -> None:
assert self.mw
user_wants_abort = dialog and dialog.wantCancel or False
update = ProgressUpdate(user_wants_abort=user_wants_abort)
progress = self.mw.backend.latest_progress()
progress_update(progress, update)
if update.abort:
self.mw.backend.set_wants_abort()
if update.has_update():
self.update(label=update.label, value=update.value, max=update.max)
qconnect(self._backend_timer.timeout, on_progress)
self._backend_timer.start()
2020-05-30 04:28:22 +02:00
def update(
self,
label: str | None = None,
value: int | None = None,
process: bool = True,
maybeShow: bool = True,
max: int | None = None,
2020-05-30 04:28:22 +02:00
) -> None:
2019-12-23 01:34:10 +01:00
# print self._min, self._counter, self._max, label, time.time() - self._lastTime
if not self.mw.inMainThread():
print("progress.update() called on wrong thread")
return
if maybeShow:
self._maybeShow()
if not self._shown:
return
if label:
self._win.form.label.setText(label)
2020-06-08 12:28:11 +02:00
self._max = max or 0
self._win.form.progressBar.setMaximum(self._max)
if self._max:
self._counter = value if value is not None else (self._counter + 1)
self._win.form.progressBar.setValue(self._counter)
2020-06-08 12:28:11 +02:00
def finish(self) -> None:
self._levels -= 1
self._levels = max(0, self._levels)
if self._levels == 0:
if self._win:
self._closeWin()
if self._busy_cursor_timer:
self._busy_cursor_timer.stop()
self._busy_cursor_timer = None
self._restore_cursor()
if self._show_timer:
self._show_timer.stop()
self._show_timer = None
Add apkg import/export on backend (#1743) * Add apkg export on backend * Filter out missing media-paths at write time * Make TagMatcher::new() infallible * Gather export data instead of copying directly * Revert changes to rslib/src/tags/ * Reuse filename_is_safe/check_filename_safe() * Accept func to produce MediaIter in export_apkg() * Only store file folder once in MediaIter * Use temporary tables for gathering export_apkg() now accepts a search instead of a deck id. Decks are gathered according to the matched notes' cards. * Use schedule_as_new() to reset cards * ExportData → ExchangeData * Ignore ascii case when filtering system tags * search_notes_cards_into_table → search_cards_of_notes_into_table * Start on apkg importing on backend * Fix due dates in days for apkg export * Refactor import-export/package - Move media and meta code into appropriate modules. - Normalize/check for normalization when deserializing media entries. * Add SafeMediaEntry for deserialized MediaEntries * Prepare media based on checksums - Ensure all existing media files are hashed. - Hash incoming files during preparation to detect conflicts. - Uniquify names of conflicting files with hash (not notetype id). - Mark media files as used while importing notes. - Finally copy used media. * Handle encoding in `replace_media_refs()` * Add trait to keep down cow boilerplate * Add notetypes immediately instaed of preparing * Move target_col into Context * Add notes immediately instaed of preparing * Note id, not guid of conflicting notes * Add import_decks() * decks_configs → deck_configs * Add import_deck_configs() * Add import_cards(), import_revlog() * Use dyn instead of generic for media_fn Otherwise, would have to pass None with type annotation in the default case. * Fix signature of import_apkg() * Fix search_cards_of_notes_into_table() * Test new functions in text.rs * Add roundtrip test for apkg (stub) * Keep source id of imported cards (or skip) * Keep source ids of imported revlog (or skip) * Try to keep source ids of imported notes * Make adding notetype with id undoable * Wrap apkg import in transaction * Keep source ids of imported deck configs (or skip) * Handle card due dates and original due/did * Fix importing cards/revlog Card ids are manually uniquified. * Factor out card importing * Refactor card and revlog importing * Factor out card importing Also handle missing parents . * Factor out note importing * Factor out media importing * Maybe upgrade scheduler of apkg * Fix parent deck gathering * Unconditionally import static media * Fix deck importing edge cases Test those edge cases, and add some global test helpers. * Test note importing * Let import_apkg() take a progress func * Expand roundtrip apkg test * Use fat pointer to avoid propogating generics * Fix progress_fn type * Expose apkg export/import on backend * Return note log when importing apkg * Fix archived collection name on apkg import * Add CollectionOpWithBackendProgress * Fix wrong Interrupted Exception being checked * Add ClosedCollectionOp * Add note ids to log and strip HTML * Update progress when checking incoming media too * Conditionally enable new importing in GUI * Fix all_checksums() for media import Entries of deleted files are nulled, not removed. * Make apkg exporting on backend abortable * Return number of notes imported from apkg * Fix exception printing for QueryOp as well * Add QueryOpWithBackendProgress Also support backend exporting progress. * Expose new apkg and colpkg exporting * Open transaction in insert_data() Was slowing down exporting by several orders of magnitude. * Handle zstd-compressed apkg * Add legacy arg to ExportAnkiPackage Currently not exposed on the frontend * Remove unused import in proto file * Add symlink for typechecking of import_export_pb2 * Avoid kwargs in pb message creation, so typechecking is not lost Protobuf's behaviour is rather subtle and I had to dig through the docs to figure it out: set a field on a submessage to automatically assign the submessage to the parent, or call SetInParent() to persist a default version of the field you specified. * Avoid re-exporting protobuf msgs we only use internally * Stop after one test failure mypy often fails much faster than pylint * Avoid an extra allocation when extracting media checksums * Update progress after prepare_media() finishes Otherwise the bulk of the import ends up being shown as "Checked: 0" in the progress window. * Show progress of note imports Note import is the slowest part, so showing progress here makes the UI feel more responsive. * Reset filtered decks at import time Before this change, filtered decks exported with scheduling remained filtered on import, and maybe_remove_from_filtered_deck() moved cards into them as their home deck, leading to errors during review. We may still want to provide a way to preserve filtered decks on import, but to do that we'll need to ensure we don't rewrite the home decks of cards, and we'll need to ensure the home decks are included as part of the import (or give an error if they're not). https://github.com/ankitects/anki/pull/1743/files#r839346423 * Fix a corner-case where due dates were shifted by a day This issue existed in the old Python code as well. We need to include the user's UTC offset in the exported file, or days_elapsed falls back on the v1 cutoff calculation, which may be a day earlier or later than the v2 calculation. * Log conflicting note in remapped nt case * take_fields() → into_fields() * Alias `[u8; 20]` with `Sha1Hash` * Truncate logged fields * Rework apkg note import tests - Use macros for more helpful errors. - Split monolith into unit tests. - Fix some unknown error with the previous test along the way. (Was failing after 969484de4388d225c9f17d94534b3ba0094c3568.) * Fix sorting of imported decks Also adjust the test, so it fails without the patch. It was only passing before, because the parent deck happened to come before the inconsistently capitalised child alphabetically. But we want all parent decks to be imported before their child decks, so their children can adopt their capitalisation. * target[_id]s → existing_card[_id]s * export_collection_extracting_media() → ... export_into_collection_file() * target_already_exists→card_ordinal_already_exists * Add search_cards_of_notes_into_table.sql * Imrove type of apkg export selector/limit * Remove redundant call to mod_schema() * Parent tooltips to mw * Fix a crash when truncating note text String::truncate() is a bit of a footgun, and I've hit this before too :-) * Remove ExportLimit in favour of separate classes * Remove OpWithBackendProgress and ClosedCollectionOp Backend progress logic is now in ProgressManager. QueryOp can be used for running on closed collection. Also fix aborting of colpkg exports, which slipped through in #1817. * Tidy up import log * Avoid QDialog.exec() * Default to excluding scheuling for deck list deck * Use IncrementalProgress in whole import_export code * Compare checksums when importing colpkgs * Avoid registering changes if hashes are not needed * ImportProgress::Collection → ImportProgress::File * Make downgrading apkgs depend on meta version * Generalise IncrementableProgress And use it in entire import_export code instead. * Fix type complexity lint * Take count_map for IncrementableProgress::get_inner * Replace import/export env with Shift click * Accept all args from update() for backend progress * Pass fields of ProgressUpdate explicitly * Move update_interval into IncrementableProgress * Outsource incrementing into Incrementor * Mutate ProgressUpdate in progress_update callback * Switch import/export legacy toggle to profile setting Shift would have been nice, but the existing shortcuts complicate things. If the user triggers an import with ctrl+shift+i, shift is unlikely to have been released by the time our code runs, meaning the user accidentally triggers the new code. We could potentially wait a while before bringing up the dialog, but then we're forced to guess at how long it will take the user to release the key. One alternative would be to use alt instead of shift, but then we need to trigger our shortcut when that key is pressed as well, and it could potentially cause a conflict with an add-on that already uses that combination. * Show extension in export dialog * Continue to provide separate options for schema 11+18 colpkg export * Default to colpkg export when using File>Export * Improve appearance of combo boxes when switching between apkg/colpkg + Deal with long deck names * Convert newlines to spaces when showing fields from import Ensures each imported note appears on a separate line * Don't separate total note count from the other summary lines This may come down to personal preference, but I feel the other counts are equally as important, and separating them feels like it makes it a bit easier to ignore them. * Fix 'deck not normal' error when importing a filtered deck for the 2nd time * Fix [Identical] being shown on first import * Revert "Continue to provide separate options for schema 11+18 colpkg export" This reverts commit 8f0b2c175f4794d642823b60414d142a12768441. Will use a different approach * Move legacy support into a separate exporter option; add to apkg export * Adjust 'too new' message to also apply to .apkg import case * Show a better message when attempting to import new apkg into old code Previously the user could end seeing a message like: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte Unfortunately we can't retroactively fix this for older clients. * Hide legacy support option in older exporting screen * Reflect change from paths to fnames in type & name * Make imported decks normal at once Then skip special casing in update_deck(). Also skip updating description if new one is empty. Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-05-02 13:12:46 +02:00
if self._backend_timer:
Refactor progress handling (#2549) Previously it was Backend's responsibility to store the last progress, and when calling routines in Collection, one had to construct and pass in a Fn, which wasn't the most ergonomic. This PR adds the last progress state to the collection, so that the routines no longer need a separate progress arg, and makes some other tweaks to improve ergonomics. ThrottlingProgressHandler has been tweaked so that it now stores the current state, so that callers don't need to store it separately. When a long-running routine starts, it calls col.new_progress_handler(), which automatically initializes the data to defaults, and updates the shared UI state, so we no longer need to manually update the state at the start of an operation. The backend shares the Arc<Mutex<>> with the collection, so it can get at the current state, and so we can update the state when importing a backup. Other tweaks: - The current Incrementor was awkward to use in the media check, which uses a single incrementing value across multiple method calls, so I've added a simpler alternative for such cases. The old incrementor method has been kept, but implemented directly on ThrottlingProgressHandler. - The full sync code was passing the progress handler in a complicated way that may once have been required, but no longer is. - On the Qt side, timers are now stopped before deletion, or they keep running for a few seconds. - I left the ChangeTracker using a closure, as it's used for both importing and syncing.
2023-06-19 05:48:32 +02:00
self._backend_timer.stop()
Add apkg import/export on backend (#1743) * Add apkg export on backend * Filter out missing media-paths at write time * Make TagMatcher::new() infallible * Gather export data instead of copying directly * Revert changes to rslib/src/tags/ * Reuse filename_is_safe/check_filename_safe() * Accept func to produce MediaIter in export_apkg() * Only store file folder once in MediaIter * Use temporary tables for gathering export_apkg() now accepts a search instead of a deck id. Decks are gathered according to the matched notes' cards. * Use schedule_as_new() to reset cards * ExportData → ExchangeData * Ignore ascii case when filtering system tags * search_notes_cards_into_table → search_cards_of_notes_into_table * Start on apkg importing on backend * Fix due dates in days for apkg export * Refactor import-export/package - Move media and meta code into appropriate modules. - Normalize/check for normalization when deserializing media entries. * Add SafeMediaEntry for deserialized MediaEntries * Prepare media based on checksums - Ensure all existing media files are hashed. - Hash incoming files during preparation to detect conflicts. - Uniquify names of conflicting files with hash (not notetype id). - Mark media files as used while importing notes. - Finally copy used media. * Handle encoding in `replace_media_refs()` * Add trait to keep down cow boilerplate * Add notetypes immediately instaed of preparing * Move target_col into Context * Add notes immediately instaed of preparing * Note id, not guid of conflicting notes * Add import_decks() * decks_configs → deck_configs * Add import_deck_configs() * Add import_cards(), import_revlog() * Use dyn instead of generic for media_fn Otherwise, would have to pass None with type annotation in the default case. * Fix signature of import_apkg() * Fix search_cards_of_notes_into_table() * Test new functions in text.rs * Add roundtrip test for apkg (stub) * Keep source id of imported cards (or skip) * Keep source ids of imported revlog (or skip) * Try to keep source ids of imported notes * Make adding notetype with id undoable * Wrap apkg import in transaction * Keep source ids of imported deck configs (or skip) * Handle card due dates and original due/did * Fix importing cards/revlog Card ids are manually uniquified. * Factor out card importing * Refactor card and revlog importing * Factor out card importing Also handle missing parents . * Factor out note importing * Factor out media importing * Maybe upgrade scheduler of apkg * Fix parent deck gathering * Unconditionally import static media * Fix deck importing edge cases Test those edge cases, and add some global test helpers. * Test note importing * Let import_apkg() take a progress func * Expand roundtrip apkg test * Use fat pointer to avoid propogating generics * Fix progress_fn type * Expose apkg export/import on backend * Return note log when importing apkg * Fix archived collection name on apkg import * Add CollectionOpWithBackendProgress * Fix wrong Interrupted Exception being checked * Add ClosedCollectionOp * Add note ids to log and strip HTML * Update progress when checking incoming media too * Conditionally enable new importing in GUI * Fix all_checksums() for media import Entries of deleted files are nulled, not removed. * Make apkg exporting on backend abortable * Return number of notes imported from apkg * Fix exception printing for QueryOp as well * Add QueryOpWithBackendProgress Also support backend exporting progress. * Expose new apkg and colpkg exporting * Open transaction in insert_data() Was slowing down exporting by several orders of magnitude. * Handle zstd-compressed apkg * Add legacy arg to ExportAnkiPackage Currently not exposed on the frontend * Remove unused import in proto file * Add symlink for typechecking of import_export_pb2 * Avoid kwargs in pb message creation, so typechecking is not lost Protobuf's behaviour is rather subtle and I had to dig through the docs to figure it out: set a field on a submessage to automatically assign the submessage to the parent, or call SetInParent() to persist a default version of the field you specified. * Avoid re-exporting protobuf msgs we only use internally * Stop after one test failure mypy often fails much faster than pylint * Avoid an extra allocation when extracting media checksums * Update progress after prepare_media() finishes Otherwise the bulk of the import ends up being shown as "Checked: 0" in the progress window. * Show progress of note imports Note import is the slowest part, so showing progress here makes the UI feel more responsive. * Reset filtered decks at import time Before this change, filtered decks exported with scheduling remained filtered on import, and maybe_remove_from_filtered_deck() moved cards into them as their home deck, leading to errors during review. We may still want to provide a way to preserve filtered decks on import, but to do that we'll need to ensure we don't rewrite the home decks of cards, and we'll need to ensure the home decks are included as part of the import (or give an error if they're not). https://github.com/ankitects/anki/pull/1743/files#r839346423 * Fix a corner-case where due dates were shifted by a day This issue existed in the old Python code as well. We need to include the user's UTC offset in the exported file, or days_elapsed falls back on the v1 cutoff calculation, which may be a day earlier or later than the v2 calculation. * Log conflicting note in remapped nt case * take_fields() → into_fields() * Alias `[u8; 20]` with `Sha1Hash` * Truncate logged fields * Rework apkg note import tests - Use macros for more helpful errors. - Split monolith into unit tests. - Fix some unknown error with the previous test along the way. (Was failing after 969484de4388d225c9f17d94534b3ba0094c3568.) * Fix sorting of imported decks Also adjust the test, so it fails without the patch. It was only passing before, because the parent deck happened to come before the inconsistently capitalised child alphabetically. But we want all parent decks to be imported before their child decks, so their children can adopt their capitalisation. * target[_id]s → existing_card[_id]s * export_collection_extracting_media() → ... export_into_collection_file() * target_already_exists→card_ordinal_already_exists * Add search_cards_of_notes_into_table.sql * Imrove type of apkg export selector/limit * Remove redundant call to mod_schema() * Parent tooltips to mw * Fix a crash when truncating note text String::truncate() is a bit of a footgun, and I've hit this before too :-) * Remove ExportLimit in favour of separate classes * Remove OpWithBackendProgress and ClosedCollectionOp Backend progress logic is now in ProgressManager. QueryOp can be used for running on closed collection. Also fix aborting of colpkg exports, which slipped through in #1817. * Tidy up import log * Avoid QDialog.exec() * Default to excluding scheuling for deck list deck * Use IncrementalProgress in whole import_export code * Compare checksums when importing colpkgs * Avoid registering changes if hashes are not needed * ImportProgress::Collection → ImportProgress::File * Make downgrading apkgs depend on meta version * Generalise IncrementableProgress And use it in entire import_export code instead. * Fix type complexity lint * Take count_map for IncrementableProgress::get_inner * Replace import/export env with Shift click * Accept all args from update() for backend progress * Pass fields of ProgressUpdate explicitly * Move update_interval into IncrementableProgress * Outsource incrementing into Incrementor * Mutate ProgressUpdate in progress_update callback * Switch import/export legacy toggle to profile setting Shift would have been nice, but the existing shortcuts complicate things. If the user triggers an import with ctrl+shift+i, shift is unlikely to have been released by the time our code runs, meaning the user accidentally triggers the new code. We could potentially wait a while before bringing up the dialog, but then we're forced to guess at how long it will take the user to release the key. One alternative would be to use alt instead of shift, but then we need to trigger our shortcut when that key is pressed as well, and it could potentially cause a conflict with an add-on that already uses that combination. * Show extension in export dialog * Continue to provide separate options for schema 11+18 colpkg export * Default to colpkg export when using File>Export * Improve appearance of combo boxes when switching between apkg/colpkg + Deal with long deck names * Convert newlines to spaces when showing fields from import Ensures each imported note appears on a separate line * Don't separate total note count from the other summary lines This may come down to personal preference, but I feel the other counts are equally as important, and separating them feels like it makes it a bit easier to ignore them. * Fix 'deck not normal' error when importing a filtered deck for the 2nd time * Fix [Identical] being shown on first import * Revert "Continue to provide separate options for schema 11+18 colpkg export" This reverts commit 8f0b2c175f4794d642823b60414d142a12768441. Will use a different approach * Move legacy support into a separate exporter option; add to apkg export * Adjust 'too new' message to also apply to .apkg import case * Show a better message when attempting to import new apkg into old code Previously the user could end seeing a message like: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte Unfortunately we can't retroactively fix this for older clients. * Hide legacy support option in older exporting screen * Reflect change from paths to fnames in type & name * Make imported decks normal at once Then skip special casing in update_deck(). Also skip updating description if new one is empty. Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-05-02 13:12:46 +02:00
self._backend_timer.deleteLater()
self._backend_timer = None
def clear(self) -> None:
"Restore the interface after an error."
if self._levels:
self._levels = 1
self.finish()
def _maybeShow(self) -> None:
if not self._levels:
return
if self._shown:
return
delta = time.time() - self._firstTime
if delta > 0.5:
self._showWin()
def _showWin(self) -> None:
self._shown = time.time()
self._win.show()
def _closeWin(self) -> None:
if self._shown:
while True:
# give the window system a second to present
# window before we close it again - fixes
# progress window getting stuck, especially
# on ubuntu 16.10+
elap = time.time() - self._shown
if elap >= 0.5:
break
self.app.processEvents(QEventLoop.ProcessEventsFlag.ExcludeUserInputEvents) # type: ignore #possibly related to https://github.com/python/mypy/issues/6910
clear_unused_tags and browser redraw improvements - clear_unused_tags() is now undoable, and returns the number of removed notes - add a new mw.query_op() helper for immutable queries - decouple "freeze/unfreeze ui state" hooks from the "interface update required" hook, so that the former is fired even on error, and can be made re-entrant - use a 'block_updates' flag in Python, instead of setUpdatesEnabled(), as the latter has the side-effect of preventing child windows like tooltips from appearing, and forces a full redrawn when updates are enabled again. The new behaviour leads to the card list blanking out when a long-running op is running, but in the future if we cache the cell values we can just display them from the cache instead. - we were indiscriminately saving the note with saveNow(), due to the call to saveTags(). Changed so that it only saves when the tags field is focused. - drain the "on_done" queue on main before launching a new background task, to lower the chances of something in on_done making a small query to the DB and hanging until a long op finishes - the duplicate check in the editor was executed after the webview loads, leading to it hanging until the sidebar finishes loading. Run it at set_note() time instead, so that the editor loads first. - don't throw an error when a long-running op started with with_progress() finishes after the window it was launched from has closed - don't throw an error when the browser is closed before the sidebar has finished loading
2021-03-17 12:27:42 +01:00
# if the parent window has been deleted, the progress dialog may have
# already been dropped; delete it if it hasn't been
if not sip.isdeleted(self._win):
self._win.cancel()
self._win = None
self._shown = 0
def _set_busy_cursor(self) -> None:
self.mw.app.setOverrideCursor(QCursor(Qt.CursorShape.WaitCursor))
def _restore_cursor(self) -> None:
self.app.restoreOverrideCursor()
def busy(self) -> int:
"True if processing."
return self._levels
2020-02-10 08:58:54 +01:00
def _on_show_timer(self) -> None:
if self.mw.app.focusWindow() is None:
# if no window is focused (eg app is minimized), defer display
self._show_timer.start(10)
return
self._show_timer = None
self._showWin()
2020-05-30 04:28:22 +02:00
def want_cancel(self) -> bool:
win = self._win
if win:
return win.wantCancel
else:
return False
def set_title(self, title: str) -> None:
win = self._win
if win:
win.setWindowTitle(title)
2020-02-10 08:58:54 +01:00
class ProgressDialog(QDialog):
def __init__(self, parent: QWidget) -> None:
2020-02-10 08:58:54 +01:00
QDialog.__init__(self, parent)
disable_help_button(self)
2020-02-10 08:58:54 +01:00
self.form = aqt.forms.progress.Ui_Dialog()
self.form.setupUi(self)
self._closingDown = False
self.wantCancel = False
2020-05-30 06:19:21 +02:00
# required for smooth progress bars
self.form.progressBar.setStyleSheet("QProgressBar::chunk { width: 1px; }")
2020-02-10 08:58:54 +01:00
def cancel(self) -> None:
2020-02-10 08:58:54 +01:00
self._closingDown = True
self.hide()
def closeEvent(self, evt: QCloseEvent) -> None:
2020-02-10 08:58:54 +01:00
if self._closingDown:
evt.accept()
else:
self.wantCancel = True
evt.ignore()
def keyPressEvent(self, evt: QKeyEvent) -> None:
if evt.key() == Qt.Key.Key_Escape:
2020-02-10 08:58:54 +01:00
evt.ignore()
self.wantCancel = True
Add apkg import/export on backend (#1743) * Add apkg export on backend * Filter out missing media-paths at write time * Make TagMatcher::new() infallible * Gather export data instead of copying directly * Revert changes to rslib/src/tags/ * Reuse filename_is_safe/check_filename_safe() * Accept func to produce MediaIter in export_apkg() * Only store file folder once in MediaIter * Use temporary tables for gathering export_apkg() now accepts a search instead of a deck id. Decks are gathered according to the matched notes' cards. * Use schedule_as_new() to reset cards * ExportData → ExchangeData * Ignore ascii case when filtering system tags * search_notes_cards_into_table → search_cards_of_notes_into_table * Start on apkg importing on backend * Fix due dates in days for apkg export * Refactor import-export/package - Move media and meta code into appropriate modules. - Normalize/check for normalization when deserializing media entries. * Add SafeMediaEntry for deserialized MediaEntries * Prepare media based on checksums - Ensure all existing media files are hashed. - Hash incoming files during preparation to detect conflicts. - Uniquify names of conflicting files with hash (not notetype id). - Mark media files as used while importing notes. - Finally copy used media. * Handle encoding in `replace_media_refs()` * Add trait to keep down cow boilerplate * Add notetypes immediately instaed of preparing * Move target_col into Context * Add notes immediately instaed of preparing * Note id, not guid of conflicting notes * Add import_decks() * decks_configs → deck_configs * Add import_deck_configs() * Add import_cards(), import_revlog() * Use dyn instead of generic for media_fn Otherwise, would have to pass None with type annotation in the default case. * Fix signature of import_apkg() * Fix search_cards_of_notes_into_table() * Test new functions in text.rs * Add roundtrip test for apkg (stub) * Keep source id of imported cards (or skip) * Keep source ids of imported revlog (or skip) * Try to keep source ids of imported notes * Make adding notetype with id undoable * Wrap apkg import in transaction * Keep source ids of imported deck configs (or skip) * Handle card due dates and original due/did * Fix importing cards/revlog Card ids are manually uniquified. * Factor out card importing * Refactor card and revlog importing * Factor out card importing Also handle missing parents . * Factor out note importing * Factor out media importing * Maybe upgrade scheduler of apkg * Fix parent deck gathering * Unconditionally import static media * Fix deck importing edge cases Test those edge cases, and add some global test helpers. * Test note importing * Let import_apkg() take a progress func * Expand roundtrip apkg test * Use fat pointer to avoid propogating generics * Fix progress_fn type * Expose apkg export/import on backend * Return note log when importing apkg * Fix archived collection name on apkg import * Add CollectionOpWithBackendProgress * Fix wrong Interrupted Exception being checked * Add ClosedCollectionOp * Add note ids to log and strip HTML * Update progress when checking incoming media too * Conditionally enable new importing in GUI * Fix all_checksums() for media import Entries of deleted files are nulled, not removed. * Make apkg exporting on backend abortable * Return number of notes imported from apkg * Fix exception printing for QueryOp as well * Add QueryOpWithBackendProgress Also support backend exporting progress. * Expose new apkg and colpkg exporting * Open transaction in insert_data() Was slowing down exporting by several orders of magnitude. * Handle zstd-compressed apkg * Add legacy arg to ExportAnkiPackage Currently not exposed on the frontend * Remove unused import in proto file * Add symlink for typechecking of import_export_pb2 * Avoid kwargs in pb message creation, so typechecking is not lost Protobuf's behaviour is rather subtle and I had to dig through the docs to figure it out: set a field on a submessage to automatically assign the submessage to the parent, or call SetInParent() to persist a default version of the field you specified. * Avoid re-exporting protobuf msgs we only use internally * Stop after one test failure mypy often fails much faster than pylint * Avoid an extra allocation when extracting media checksums * Update progress after prepare_media() finishes Otherwise the bulk of the import ends up being shown as "Checked: 0" in the progress window. * Show progress of note imports Note import is the slowest part, so showing progress here makes the UI feel more responsive. * Reset filtered decks at import time Before this change, filtered decks exported with scheduling remained filtered on import, and maybe_remove_from_filtered_deck() moved cards into them as their home deck, leading to errors during review. We may still want to provide a way to preserve filtered decks on import, but to do that we'll need to ensure we don't rewrite the home decks of cards, and we'll need to ensure the home decks are included as part of the import (or give an error if they're not). https://github.com/ankitects/anki/pull/1743/files#r839346423 * Fix a corner-case where due dates were shifted by a day This issue existed in the old Python code as well. We need to include the user's UTC offset in the exported file, or days_elapsed falls back on the v1 cutoff calculation, which may be a day earlier or later than the v2 calculation. * Log conflicting note in remapped nt case * take_fields() → into_fields() * Alias `[u8; 20]` with `Sha1Hash` * Truncate logged fields * Rework apkg note import tests - Use macros for more helpful errors. - Split monolith into unit tests. - Fix some unknown error with the previous test along the way. (Was failing after 969484de4388d225c9f17d94534b3ba0094c3568.) * Fix sorting of imported decks Also adjust the test, so it fails without the patch. It was only passing before, because the parent deck happened to come before the inconsistently capitalised child alphabetically. But we want all parent decks to be imported before their child decks, so their children can adopt their capitalisation. * target[_id]s → existing_card[_id]s * export_collection_extracting_media() → ... export_into_collection_file() * target_already_exists→card_ordinal_already_exists * Add search_cards_of_notes_into_table.sql * Imrove type of apkg export selector/limit * Remove redundant call to mod_schema() * Parent tooltips to mw * Fix a crash when truncating note text String::truncate() is a bit of a footgun, and I've hit this before too :-) * Remove ExportLimit in favour of separate classes * Remove OpWithBackendProgress and ClosedCollectionOp Backend progress logic is now in ProgressManager. QueryOp can be used for running on closed collection. Also fix aborting of colpkg exports, which slipped through in #1817. * Tidy up import log * Avoid QDialog.exec() * Default to excluding scheuling for deck list deck * Use IncrementalProgress in whole import_export code * Compare checksums when importing colpkgs * Avoid registering changes if hashes are not needed * ImportProgress::Collection → ImportProgress::File * Make downgrading apkgs depend on meta version * Generalise IncrementableProgress And use it in entire import_export code instead. * Fix type complexity lint * Take count_map for IncrementableProgress::get_inner * Replace import/export env with Shift click * Accept all args from update() for backend progress * Pass fields of ProgressUpdate explicitly * Move update_interval into IncrementableProgress * Outsource incrementing into Incrementor * Mutate ProgressUpdate in progress_update callback * Switch import/export legacy toggle to profile setting Shift would have been nice, but the existing shortcuts complicate things. If the user triggers an import with ctrl+shift+i, shift is unlikely to have been released by the time our code runs, meaning the user accidentally triggers the new code. We could potentially wait a while before bringing up the dialog, but then we're forced to guess at how long it will take the user to release the key. One alternative would be to use alt instead of shift, but then we need to trigger our shortcut when that key is pressed as well, and it could potentially cause a conflict with an add-on that already uses that combination. * Show extension in export dialog * Continue to provide separate options for schema 11+18 colpkg export * Default to colpkg export when using File>Export * Improve appearance of combo boxes when switching between apkg/colpkg + Deal with long deck names * Convert newlines to spaces when showing fields from import Ensures each imported note appears on a separate line * Don't separate total note count from the other summary lines This may come down to personal preference, but I feel the other counts are equally as important, and separating them feels like it makes it a bit easier to ignore them. * Fix 'deck not normal' error when importing a filtered deck for the 2nd time * Fix [Identical] being shown on first import * Revert "Continue to provide separate options for schema 11+18 colpkg export" This reverts commit 8f0b2c175f4794d642823b60414d142a12768441. Will use a different approach * Move legacy support into a separate exporter option; add to apkg export * Adjust 'too new' message to also apply to .apkg import case * Show a better message when attempting to import new apkg into old code Previously the user could end seeing a message like: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte Unfortunately we can't retroactively fix this for older clients. * Hide legacy support option in older exporting screen * Reflect change from paths to fnames in type & name * Make imported decks normal at once Then skip special casing in update_deck(). Also skip updating description if new one is empty. Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-05-02 13:12:46 +02:00
@dataclass
class ProgressUpdate:
label: str | None = None
value: int | None = None
max: int | None = None
user_wants_abort: bool = False
abort: bool = False
def has_update(self) -> bool:
return self.label is not None or self.value is not None or self.max is not None