- backend routines should contain minimal logic, and should call
into a routine on the collection
- instead of copying the giant-string approach the Python code was taking,
we use a HashSet to keep track of seen tags as we loop through the
notes, which should be more efficient
See https://github.com/ankitects/anki/pull/900#issuecomment-758284016
- Leave tag names alone and add the collapsed and config columns to the tags table.
- Update The DB check code to preserve the collapse state of used tags.
- Add a simple test for clearing tags and their children
- use the TimestampSecs newtype instead of raw i64s
- use FixedOffset instead of a minutes_west offset
- check localOffset each time the timing is calculated, and set it
if it's stale - even for v1.
- check for and fix missing rollover when calculating timing
- stop explicitly passing localOffset in the sync/start call
Running and testing should be working on the three platforms, but
there's still a fair bit that needs to be done:
- Wheel building + testing in a venv still needs to be implemented.
- Python requirements still need to be compiled with piptool and pinned;
need to compile on all platforms then merge
- Cargo deps in cargo/ and rslib/ need to be cleaned up, and ideally
unified into one place
- Currently using rustls to work around openssl compilation issues
on Linux, but this will break corporate proxies with custom SSL
authorities; need to conditionally use openssl or use
https://github.com/seanmonstar/reqwest/pull/1058
- Makefiles and docs still need cleaning up
- It may make sense to reparent ts/* to the top level, as we don't
nest the other modules under a specific language.
- rspy and pylib must always be updated in lock-step, so merging
rspy into pylib as a private module would simplify things.
- Merging desktop-ftl and mobile-ftl into the core ftl would make
managing and updating translations easier.
- Obsolete scripts need removing.
- And probably more.
The previous implementation had some slightly questionable memory safety
properties (older versions of PyO3 didn't uphold the Rust aliasing rules
and would thus create multiple &mut references to #[pyclass] objects).
This explains why Backend has internal Mutex<T>s even though all of its
methods took &mut self.
The solution is to simply make all methods take &self, which luckily
doesn't pose too make issues -- most of the code inside Backend already
has sufficient locking. The only two things which needed to be
explicitly handled where:
1. "self.runtime" which was fairly easy to handle. All usages of
the Runtime only require an immutable reference to create a new
Handle, so we could switch to OnceCell which provides
lazy-initialisation semantics without needing a more heavy-handed
Mutex<tokio::runtime::Handle>.
2. "self.sync_abort" was simply wrapped in a Mutex<>, though some of the
odd semantics of sync_abort (not being able to handle multiple
processes synchronising at the same time) become pretty obvious with
this change (for now we just log a warning in that case). In
addition, switch to an RAII-style guard to make sure we don't forget
to clear the abort_handle.
As a result, we now no longer break Rust's aliasing rules and we can
build with newer versions of PyO3 which have runtime checks for these
things (and build on stable Rust).
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
- blue for normal sync, red for full sync required
- refactor status fetching code so we don't hold a collection lock
during the network request, which slows things down
- fix sync spinner restarting when returning to deck list
- Use a separate abort handle, as the media sync is running
in the background and we need to be able to target them separately.
The current progress handling is going to need a rethink if we introduce
any other background tasks in the future.
- Roll back the transaction when interrupting.
Also:
- provide a way for the progress handler to skip the throttling so that
we can ensure progress is updated at the end of a stage
- show 'checking' at the end of full sync
- .update() should update a single deck and preserve usn by default,
as that's what existing code expects
- decks are automatically renamed when they conflict with an existing
name
- changes are now committed in bulk when closing the dialog,
and can be canceled
- it's not necessary to save the note to the database to preview it
- duplicate fields are now shown as duplicates in the top list
- redraw preview more quickly
- use + instead of _ when deduplicating names, as the latter is a
glob character
Saves having to serialize the note fields and q/a templates, which
is particularly a win when rendering question/answer in the browse
screen.
Also some work towards being able to preview notes without having to
commit them to the database.
- notes with wrong field count are now recovered instead of
being deleted
- notes with missing note types are now recovered
- notes with missing cards are now recovered
- recover_missing_deck() still needs implementing
- checks required
changes to note:
- add_note() now takes a provided deck id instead of looking it up
in the notetype
- note type use counts fetched using a single table scan
- make sure note type changes are persisted
- expose optionalness of ords in templates and fields json
In the cloze deletion case, we already created a dummy card 0 when
no cloze deletions were found. This change makes normal note types
behave the same way - if no cards would be generated, a dummy card
0 is added to allow the note to be added.
This also applies when modifying note types - it is now possible
to delete card templates even if some notes only use that template,
as a dummy card 0 will be generated for notes that end up with no
cards left.
Still a prototype at this stage - we'll likely want a caching layer
for note types, and I'm not sure of the merit of having fields in
a separate table, since they're almost always required.
- mtime is tracked on each key individually, which will allow
merging of config changes when syncing in the future
- added col.(get|set|remove)_config()
- in order to support existing code that was mutating returned
values (eg col.conf["something"]["another"] = 5), the returned list/dict
will be automatically wrapped so that when the value is dropped, it
will save the mutated item back to the DB if it's changed. Code that
is fetching lists/dicts from the config like so:
col.conf["foo"]["bar"] = baz
col.setMod()
will continue to work in most case, but should be gradually updated to:
conf = col.get_config("foo")
conf["bar"] = baz
col.set_config("foo", conf)
Disabled for now; when enabled it will allow faster collection
open and close in the normal case, while continuing to downgrade
when exporting or doing a full sync.
Also, when downgrading is disabled, the journal mode is no longer
changed back to delete.
- tag list stored in a separate DB table
- non-wildcard searches now do full unicode case folding
(eg tag:masse matches 'Maße')
- wildcard matches do simple unicode case folding
- some functions haven't been updated yet, so ascii folding will
continue to be used in some operations
The progress handling code needs a rethink, as we now have two separate
ways to flag that the media sync should abort. In the future, it may
make sense to switch to polling the backend for progress, instead of
passing a callback in.
- on collection load, the schema is upgraded to 12
- on collection close, the changes are reversed so older clients
can continue to open the collection
- in the future, we could potentially skip the reversal except
when exporting/doing a full sync
- the same approach should work for decks, note types and tags in the
future too
- the deck list code needs updating to cache the deck confs for the
life of the call
This is safer than just dropping the backend, as .close() will
block if something else is holding the mutex. Also means we can
drop the extra I18nBackend code.
Media syncing still needs fixing.
Some initial testing with orjson indicates performance varies from
slightly better than pysqlite to about 2x slower depending on the type
of query.
Performance could be improved by building the Python list in rspy
instead of sending back json that needs to be decoded, but it may make
more sense to rewrite the hotspots in Rust instead. More testing is
required in any case.
committing the Protobuf implementation for posterity, but will replace
it with json, as Protobuf measures about 6x slower for some workloads
like 'select * from notes'
