anki/rslib/build/protobuf.rs

154 lines
4.6 KiB
Rust
Raw Normal View History

// Copyright: Ankitects Pty Ltd and contributors
// License: GNU AGPL, version 3 or later; http://www.gnu.org/licenses/agpl.html
use std::{
env,
fmt::Write,
path::{Path, PathBuf},
};
struct CustomGenerator {}
fn write_method_trait(buf: &mut String, service: &prost_build::Service) {
buf.push_str(
r#"
pub trait Service {
fn run_method(&self, method: u32, input: &[u8]) -> Result<Vec<u8>> {
match method {
"#,
);
for (idx, method) in service.methods.iter().enumerate() {
write!(
buf,
concat!(" ",
refactor protobuf handling for split/import In order to split backend.proto into a more manageable size, the protobuf handling needed to be updated. This took more time than I would have liked, as each language handles protobuf differently: - The Python Protobuf code ignores "package" directives, and relies solely on how the files are laid out on disk. While it would have been nice to keep the generated files in a private subpackage, Protobuf gets confused if the files are located in a location that does not match their original .proto layout, so the old approach of storing them in _backend/ will not work. They now clutter up pylib/anki instead. I'm rather annoyed by that, but alternatives seem to be having to add an extra level to the Protobuf path, making the other languages suffer, or trying to hack around the issue by munging sys.modules. - Protobufjs fails to expose packages if they don't start with a capital letter, despite the fact that lowercase packages are the norm in most languages :-( This required a patch to fix. - Rust was the easiest, as Prost is relatively straightforward compared to Google's tools. The Protobuf files are now stored in /proto/anki, with a separate package for each file. I've split backend.proto into a few files as a test, but the majority of that work is still to come. The Python Protobuf building is a bit of a hack at the moment, hard-coding "proto" as the top level folder, but it seems to get the job done for now. Also changed the workspace name, as there seems to be a number of Bazel repos moving away from the more awkward reverse DNS naming style.
2021-07-10 09:50:18 +02:00
"{idx} => {{ let input = super::{input_type}::decode(input)?;\n",
"let output = self.{rust_method}(input)?;\n",
"let mut out_bytes = Vec::new(); output.encode(&mut out_bytes)?; Ok(out_bytes) }}, "),
idx = idx,
input_type = method.input_type,
rust_method = method.name
)
.unwrap();
}
buf.push_str(
r#"
_ => crate::invalid_input!("invalid command"),
}
}
"#,
);
for method in &service.methods {
write!(
buf,
concat!(
refactor protobuf handling for split/import In order to split backend.proto into a more manageable size, the protobuf handling needed to be updated. This took more time than I would have liked, as each language handles protobuf differently: - The Python Protobuf code ignores "package" directives, and relies solely on how the files are laid out on disk. While it would have been nice to keep the generated files in a private subpackage, Protobuf gets confused if the files are located in a location that does not match their original .proto layout, so the old approach of storing them in _backend/ will not work. They now clutter up pylib/anki instead. I'm rather annoyed by that, but alternatives seem to be having to add an extra level to the Protobuf path, making the other languages suffer, or trying to hack around the issue by munging sys.modules. - Protobufjs fails to expose packages if they don't start with a capital letter, despite the fact that lowercase packages are the norm in most languages :-( This required a patch to fix. - Rust was the easiest, as Prost is relatively straightforward compared to Google's tools. The Protobuf files are now stored in /proto/anki, with a separate package for each file. I've split backend.proto into a few files as a test, but the majority of that work is still to come. The Python Protobuf building is a bit of a hack at the moment, hard-coding "proto" as the top level folder, but it seems to get the job done for now. Also changed the workspace name, as there seems to be a number of Bazel repos moving away from the more awkward reverse DNS naming style.
2021-07-10 09:50:18 +02:00
" fn {method_name}(&self, input: super::{input_type}) -> ",
"Result<super::{output_type}>;\n"
),
method_name = method.name,
input_type = method.input_type,
output_type = method.output_type
)
.unwrap();
}
buf.push_str("}\n");
}
impl prost_build::ServiceGenerator for CustomGenerator {
fn generate(&mut self, service: prost_build::Service, buf: &mut String) {
write!(
buf,
"pub mod {name}_service {{
use prost::Message;
2021-04-01 08:06:24 +02:00
use crate::error::Result;
",
name = service.name.replace("Service", "").to_ascii_lowercase()
)
.unwrap();
write_method_trait(buf, &service);
buf.push('}');
}
}
fn service_generator() -> Box<dyn prost_build::ServiceGenerator> {
Box::new(CustomGenerator {})
}
pub fn write_backend_proto_rs() {
maybe_add_protobuf_to_path();
refactor protobuf handling for split/import In order to split backend.proto into a more manageable size, the protobuf handling needed to be updated. This took more time than I would have liked, as each language handles protobuf differently: - The Python Protobuf code ignores "package" directives, and relies solely on how the files are laid out on disk. While it would have been nice to keep the generated files in a private subpackage, Protobuf gets confused if the files are located in a location that does not match their original .proto layout, so the old approach of storing them in _backend/ will not work. They now clutter up pylib/anki instead. I'm rather annoyed by that, but alternatives seem to be having to add an extra level to the Protobuf path, making the other languages suffer, or trying to hack around the issue by munging sys.modules. - Protobufjs fails to expose packages if they don't start with a capital letter, despite the fact that lowercase packages are the norm in most languages :-( This required a patch to fix. - Rust was the easiest, as Prost is relatively straightforward compared to Google's tools. The Protobuf files are now stored in /proto/anki, with a separate package for each file. I've split backend.proto into a few files as a test, but the majority of that work is still to come. The Python Protobuf building is a bit of a hack at the moment, hard-coding "proto" as the top level folder, but it seems to get the job done for now. Also changed the workspace name, as there seems to be a number of Bazel repos moving away from the more awkward reverse DNS naming style.
2021-07-10 09:50:18 +02:00
let proto_dir = if let Ok(proto) = env::var("PROTO_TOP") {
2021-07-12 08:15:38 +02:00
PathBuf::from(proto).parent().unwrap().to_owned()
} else {
refactor protobuf handling for split/import In order to split backend.proto into a more manageable size, the protobuf handling needed to be updated. This took more time than I would have liked, as each language handles protobuf differently: - The Python Protobuf code ignores "package" directives, and relies solely on how the files are laid out on disk. While it would have been nice to keep the generated files in a private subpackage, Protobuf gets confused if the files are located in a location that does not match their original .proto layout, so the old approach of storing them in _backend/ will not work. They now clutter up pylib/anki instead. I'm rather annoyed by that, but alternatives seem to be having to add an extra level to the Protobuf path, making the other languages suffer, or trying to hack around the issue by munging sys.modules. - Protobufjs fails to expose packages if they don't start with a capital letter, despite the fact that lowercase packages are the norm in most languages :-( This required a patch to fix. - Rust was the easiest, as Prost is relatively straightforward compared to Google's tools. The Protobuf files are now stored in /proto/anki, with a separate package for each file. I've split backend.proto into a few files as a test, but the majority of that work is still to come. The Python Protobuf building is a bit of a hack at the moment, hard-coding "proto" as the top level folder, but it seems to get the job done for now. Also changed the workspace name, as there seems to be a number of Bazel repos moving away from the more awkward reverse DNS naming style.
2021-07-10 09:50:18 +02:00
PathBuf::from("../proto")
};
let subfolders = &["anki"];
let mut paths = vec![];
for subfolder in subfolders {
for entry in proto_dir.join(subfolder).read_dir().unwrap() {
let entry = entry.unwrap();
let path = entry.path();
if path
.file_name()
.unwrap()
.to_str()
.unwrap()
.ends_with(".proto")
{
println!("cargo:rerun-if-changed={}", path.to_str().unwrap());
paths.push(path);
}
}
}
2021-07-12 08:15:38 +02:00
let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap());
let mut config = prost_build::Config::new();
config
.out_dir(&out_dir)
.service_generator(service_generator())
.type_attribute(
2021-04-04 13:15:30 +02:00
"Deck.Filtered.SearchTerm.Order",
"#[derive(strum::EnumIter)]",
)
.type_attribute(
"Deck.Normal.DayLimit",
"#[derive(Copy, serde_derive::Deserialize, serde_derive::Serialize)]",
)
.type_attribute("HelpPageLinkRequest.HelpPage", "#[derive(strum::EnumIter)]")
Plaintext import/export (#1850) * Add crate csv * Add start of csv importing on backend * Add Menomosyne serializer * Add csv and json importing on backend * Add plaintext importing on frontend * Add csv metadata extraction on backend * Add csv importing with GUI * Fix missing dfa file in build Added compile_data_attr, then re-ran cargo/update.py. * Don't use doubly buffered reader in csv * Escape HTML entities if CSV is not HTML Also use name 'is_html' consistently. * Use decimal number as foreign ease (like '2.5') * ForeignCard.ivl → ForeignCard.interval * Only allow fixed set of CSV delimiters * Map timestamp of ForeignCard to native due time * Don't trim CSV records * Document use of empty strings for defaults * Avoid creating CardGenContexts for every note This requires CardGenContext to be generic, so it works both with an owned and borrowed notetype. * Show all accepted file types in import file picker * Add import_json_file() * factor → ease_factor * delimter_from_value → delimiter_from_value * Map columns to fields, not the other way around * Fallback to current config for csv metadata * Add start of new import csv screen * Temporary fix for compilation issue on Linux/Mac * Disable jest bazel action for import-csv Jest fails with an error code if no tests are available, but this would not be noticable on Windows as Jest is not run there. * Fix field mapping issue * Revert "Temporary fix for compilation issue on Linux/Mac" This reverts commit 21f8a261408cdae49ec031aa21a1b659c4f66d82. * Add HtmlSwitch and move Switch to components * Fix spacing and make selectors consistent * Fix shortcut tooltip * Place import button at the top with path * Fix meta column indices * Remove NotetypeForString * Fix queue and type of foreign cards * Support different dupe resolution strategies * Allow dupe resolution selection when importing CSV * Test import of unnormalized text Close #1863. * Fix logging of foreign notes * Implement CSV exports * Use db_scalar() in notes_table_len() * Rework CSV metadata - Notetypes and decks are either defined by a global id or by a column. - If a notetype id is provided, its field map must also be specified. - If a notetype column is provided, fields are now mapped by index instead of name at import time. So the first non-meta column is used for the first field of every note, regardless of notetype. This makes importing easier and should improve compatiblity with files without a notetype column. - Ensure first field can be mapped to a column. - Meta columns must be defined as `#[meta name]:[column index]` instead of in the `#columns` tag. - Column labels contain the raw names defined by the file and must be prettified by the frontend. * Adjust frontend to new backend column mapping * Add force flags for is_html and delimiter * Detect if CSV is HTML by field content * Update dupe resolution labels * Simplify selectors * Fix coalescence of oneofs in TS * Disable meta columns from selection Plus a lot of refactoring. * Make import button stick to the bottom * Write delimiter and html flag into csv * Refetch field map after notetype change * Fix log labels for csv import * Log notes whose deck/notetype was missing * Fix hiding of empty log queues * Implement adding tags to all notes of a csv * Fix dupe resolution not being set in log * Implement adding tags to updated notes of a csv * Check first note field is not empty * Temporary fix for build on Linux/Mac * Fix inverted html check (dae) * Remove unused ftl string * Delimiter → Separator * Remove commented-out line * Don't accept .json files * Tweak tag ftl strings * Remove redundant blur call * Strip sound and add spaces in csv export * Export HTML by default * Fix unset deck in Mnemosyne import Also accept both numbers and strings for notetypes and decks in JSON. * Make DupeResolution::Update the default * Fix missing dot in extension * Make column indices 1-based * Remove StickContainer from TagEditor Fixes line breaking, border and z index on ImportCsvPage. * Assign different key combos to tag editors * Log all updated duplicates Add a log field for the true number of found notes. * Show identical notes as skipped * Split tag-editor into separate ts module (dae) * Add progress for CSV export * Add progress for text import * Tidy-ups after tag-editor split (dae) - import-csv no longer depends on editor - remove some commented lines
2022-06-01 12:26:16 +02:00
.type_attribute("CsvMetadata.Delimiter", "#[derive(strum::EnumIter)]")
Backups (#1685) * Add zstd dep * Implement backend backup with zstd * Implement backup thinning * Write backup meta * Use new file ending anki21b * Asynchronously backup on collection close in Rust * Revert "Add zstd dep" This reverts commit 3fcb2141d2be15f907269d13275c41971431385c. * Add zstd again * Take backup col path from col struct * Fix formatting * Implement backup restoring on backend * Normalize restored media file names * Refactor `extract_legacy_data()` A bit cumbersome due to borrowing rules. * Refactor * Make thinning calendar-based and gradual * Consider last kept backups of previous stages * Import full apkgs and colpkgs with backend * Expose new backup settings * Test `BackupThinner` and make it deterministic * Mark backup_path when closing optional * Delete leaky timer * Add progress updates for restoring media * Write restored collection to tempfile first * Do collection compression in the background thread This has us currently storing an uncompressed and compressed copy of the collection in memory (not ideal), but means the collection can be closed without waiting for compression to complete. On a large collection, this takes a close and reopen from about 0.55s to about 0.07s. The old backup code for comparison: about 0.35s for compression off, about 8.5s for zip compression. * Use multithreading in zstd compression On my system, this reduces the compression time of a large collection from about 0.55s to 0.08s. * Stream compressed collection data into zip file * Tweak backup explanation + Fix incorrect tab order for ignore accents option * Decouple restoring backup and full import In the first case, no profile is opened, unless the new collection succeeds to load. In the second case, either the old collection is reloaded or the new one is loaded. * Fix number gap in Progress message * Don't revert backup when media fails but report it * Tweak error flow * Remove native BackupLimits enum * Fix type annotation * Add thinning test for whole year * Satisfy linter * Await async backup to finish * Move restart disclaimer out of backup tab Should be visible regardless of the current tab. * Write restored collection in chunks * Refactor * Write media in chunks and refactor * Log error if removing file fails * join_backup_task -> await_backup_completion * Refactor backup.rs * Refactor backup meta and collection extraction * Fix wrong error being returned * Call sync_all() on new collection * Add ImportError * Store logger in Backend, instead of creating one on demand init_backend() accepts a Logger rather than a log file, to allow other callers to customize the logger if they wish. In the future we may want to explore using the tracing crate as an alternative; it's a bit more ergonomic, as a logger doesn't need to be passed around, and it plays more nicely with async code. * Sync file contents prior to rename; sync folder after rename. * Limit backup creation to once per 30 min * Use zstd::stream::copy_decode * Make importing abortable * Don't revert if backup media is aborted * Set throttle implicitly * Change force flag to minimum_backup_interval * Don't attempt to open folders on Windows * Join last backup thread before starting new one Also refactor. * Disable auto sync and backup when restoring again * Force backup on full download * Include the reason why a media file import failed, and the file path - Introduce a FileIoError that contains a string representation of the underlying I/O error, and an associated path. There are a few places in the code where we're currently manually including the filename in a custom error message, and this is a step towards a more consistent approach (but we may be better served with a more general approach in the future similar to Anyhow's .context()) - Move the error message into importing.ftl, as it's a bit neater when error messages live in the same file as the rest of the messages associated with some functionality. * Fix importing of media files * Minor wording tweaks * Save an allocation I18n strings with replacements are already strings, so we can skip the extra allocation. Not that it matters here at all. * Terminate import if file missing from archive If a third-party tool is creating invalid archives, the user should know about it. This should be rare, so I did not attempt to make it translatable. * Skip multithreaded compression on small collections Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-03-07 06:11:31 +01:00
.type_attribute(
Backup improvements (#1728) * Collection needs to be closed prior to backup even when not downgrading * Backups -> BackupLimits * Some improvements to backup_task - backup_inner now returns the error instead of logging it, so that the frontend can discover the issue when they await a backup (or create another one) - start_backup() was acquiring backup_task twice, and if another thread started a backup between the two locks, the task could have been accidentally overwritten without awaiting it * Backups no longer require a collection close - Instead of closing the collection, we ensure there is no active transaction, and flush the WAL to disk. This means the undo history is no longer lost on backup, which will be particularly useful if we add a periodic backup in the future. - Because a close is no longer required, backups are now achieved with a separate command, instead of being included in CloseCollection(). - Full sync no longer requires an extra close+reopen step, and we now wait for the backup to complete before proceeding. - Create a backup before 'check db' * Add File>Create Backup https://forums.ankiweb.net/t/anki-mac-os-no-backup-on-sync/6157 * Defer checkpoint until we know we need it When running periodic backups on a timer, we don't want to be fsync()ing unnecessarily. * Skip backup if modification time has not changed We don't want the user leaving Anki open overnight, and coming back to lots of identical backups. * Periodic backups Creates an automatic backup every 30 minutes if the collection has been modified. If there's a legacy checkpoint active, tries again 5 minutes later. * Switch to a user-configurable backup duration CreateBackup() now uses a simple force argument to determine whether the user's limits should be respected or not, and only potentially destructive ops (full download, check DB) override the user's configured limit. I considered having a separate limit for collection close and automatic backups (eg keeping the previous 5 minute limit for collection close), but that had two downsides: - When the user closes their collection at the end of the day, they'd get a recent backup. When they open the collection the next day, it would get backed up again within 5 minutes, even though not much had changed. - Multiple limits are harder to communicate to users in the UI Some remaining decisions I wasn't 100% sure about: - If force is true but the collection has not been modified, the backup will be skipped. If the user manually deleted their backups without closing Anki, they wouldn't get a new one if the mtime hadn't changed. - Force takes preference over the configured backup interval - should we be ignored the user here, or take no backups at all? Did a sneaky edit of the existing ftl string, as it hasn't been live long. * Move maybe_backup() into Collection * Use a single method for manual and periodic backups When manually creating a backup via the File menu, we no longer make the user wait until the backup completes. As we continue waiting for the backup in the background, if any errors occur, the user will get notified about it fairly quickly. * Show message to user if backup was skipped due to no changes + Don't incorrectly assert a backup will be created on force * Add "automatic" to description * Ensure we backup prior to importing colpkg if collection open The backup doesn't happen when invoked from 'open backup' in the profile screen, which matches Anki's previous behaviour. The user could potentially clobber up to 30 minutes of their work if they exited to the profile screen and restored a backup, but the alternative is we create backups every time a backup is restored, which may happen a number of times if the user is trying various ones. Or we could go back to a separate throttle amount for this case, at the cost of more complexity. * Remove the 0 special case on backup interval; minimum of 5 minutes https://github.com/ankitects/anki/pull/1728#discussion_r830876833
2022-03-21 10:40:42 +01:00
"Preferences.BackupLimits",
Backups (#1685) * Add zstd dep * Implement backend backup with zstd * Implement backup thinning * Write backup meta * Use new file ending anki21b * Asynchronously backup on collection close in Rust * Revert "Add zstd dep" This reverts commit 3fcb2141d2be15f907269d13275c41971431385c. * Add zstd again * Take backup col path from col struct * Fix formatting * Implement backup restoring on backend * Normalize restored media file names * Refactor `extract_legacy_data()` A bit cumbersome due to borrowing rules. * Refactor * Make thinning calendar-based and gradual * Consider last kept backups of previous stages * Import full apkgs and colpkgs with backend * Expose new backup settings * Test `BackupThinner` and make it deterministic * Mark backup_path when closing optional * Delete leaky timer * Add progress updates for restoring media * Write restored collection to tempfile first * Do collection compression in the background thread This has us currently storing an uncompressed and compressed copy of the collection in memory (not ideal), but means the collection can be closed without waiting for compression to complete. On a large collection, this takes a close and reopen from about 0.55s to about 0.07s. The old backup code for comparison: about 0.35s for compression off, about 8.5s for zip compression. * Use multithreading in zstd compression On my system, this reduces the compression time of a large collection from about 0.55s to 0.08s. * Stream compressed collection data into zip file * Tweak backup explanation + Fix incorrect tab order for ignore accents option * Decouple restoring backup and full import In the first case, no profile is opened, unless the new collection succeeds to load. In the second case, either the old collection is reloaded or the new one is loaded. * Fix number gap in Progress message * Don't revert backup when media fails but report it * Tweak error flow * Remove native BackupLimits enum * Fix type annotation * Add thinning test for whole year * Satisfy linter * Await async backup to finish * Move restart disclaimer out of backup tab Should be visible regardless of the current tab. * Write restored collection in chunks * Refactor * Write media in chunks and refactor * Log error if removing file fails * join_backup_task -> await_backup_completion * Refactor backup.rs * Refactor backup meta and collection extraction * Fix wrong error being returned * Call sync_all() on new collection * Add ImportError * Store logger in Backend, instead of creating one on demand init_backend() accepts a Logger rather than a log file, to allow other callers to customize the logger if they wish. In the future we may want to explore using the tracing crate as an alternative; it's a bit more ergonomic, as a logger doesn't need to be passed around, and it plays more nicely with async code. * Sync file contents prior to rename; sync folder after rename. * Limit backup creation to once per 30 min * Use zstd::stream::copy_decode * Make importing abortable * Don't revert if backup media is aborted * Set throttle implicitly * Change force flag to minimum_backup_interval * Don't attempt to open folders on Windows * Join last backup thread before starting new one Also refactor. * Disable auto sync and backup when restoring again * Force backup on full download * Include the reason why a media file import failed, and the file path - Introduce a FileIoError that contains a string representation of the underlying I/O error, and an associated path. There are a few places in the code where we're currently manually including the filename in a custom error message, and this is a step towards a more consistent approach (but we may be better served with a more general approach in the future similar to Anyhow's .context()) - Move the error message into importing.ftl, as it's a bit neater when error messages live in the same file as the rest of the messages associated with some functionality. * Fix importing of media files * Minor wording tweaks * Save an allocation I18n strings with replacements are already strings, so we can skip the extra allocation. Not that it matters here at all. * Terminate import if file missing from archive If a third-party tool is creating invalid archives, the user should know about it. This should be rare, so I did not attempt to make it translatable. * Skip multithreaded compression on small collections Co-authored-by: Damien Elmes <gpg@ankiweb.net>
2022-03-07 06:11:31 +01:00
"#[derive(Copy, serde_derive::Deserialize, serde_derive::Serialize)]",
)
Plaintext import/export (#1850) * Add crate csv * Add start of csv importing on backend * Add Menomosyne serializer * Add csv and json importing on backend * Add plaintext importing on frontend * Add csv metadata extraction on backend * Add csv importing with GUI * Fix missing dfa file in build Added compile_data_attr, then re-ran cargo/update.py. * Don't use doubly buffered reader in csv * Escape HTML entities if CSV is not HTML Also use name 'is_html' consistently. * Use decimal number as foreign ease (like '2.5') * ForeignCard.ivl → ForeignCard.interval * Only allow fixed set of CSV delimiters * Map timestamp of ForeignCard to native due time * Don't trim CSV records * Document use of empty strings for defaults * Avoid creating CardGenContexts for every note This requires CardGenContext to be generic, so it works both with an owned and borrowed notetype. * Show all accepted file types in import file picker * Add import_json_file() * factor → ease_factor * delimter_from_value → delimiter_from_value * Map columns to fields, not the other way around * Fallback to current config for csv metadata * Add start of new import csv screen * Temporary fix for compilation issue on Linux/Mac * Disable jest bazel action for import-csv Jest fails with an error code if no tests are available, but this would not be noticable on Windows as Jest is not run there. * Fix field mapping issue * Revert "Temporary fix for compilation issue on Linux/Mac" This reverts commit 21f8a261408cdae49ec031aa21a1b659c4f66d82. * Add HtmlSwitch and move Switch to components * Fix spacing and make selectors consistent * Fix shortcut tooltip * Place import button at the top with path * Fix meta column indices * Remove NotetypeForString * Fix queue and type of foreign cards * Support different dupe resolution strategies * Allow dupe resolution selection when importing CSV * Test import of unnormalized text Close #1863. * Fix logging of foreign notes * Implement CSV exports * Use db_scalar() in notes_table_len() * Rework CSV metadata - Notetypes and decks are either defined by a global id or by a column. - If a notetype id is provided, its field map must also be specified. - If a notetype column is provided, fields are now mapped by index instead of name at import time. So the first non-meta column is used for the first field of every note, regardless of notetype. This makes importing easier and should improve compatiblity with files without a notetype column. - Ensure first field can be mapped to a column. - Meta columns must be defined as `#[meta name]:[column index]` instead of in the `#columns` tag. - Column labels contain the raw names defined by the file and must be prettified by the frontend. * Adjust frontend to new backend column mapping * Add force flags for is_html and delimiter * Detect if CSV is HTML by field content * Update dupe resolution labels * Simplify selectors * Fix coalescence of oneofs in TS * Disable meta columns from selection Plus a lot of refactoring. * Make import button stick to the bottom * Write delimiter and html flag into csv * Refetch field map after notetype change * Fix log labels for csv import * Log notes whose deck/notetype was missing * Fix hiding of empty log queues * Implement adding tags to all notes of a csv * Fix dupe resolution not being set in log * Implement adding tags to updated notes of a csv * Check first note field is not empty * Temporary fix for build on Linux/Mac * Fix inverted html check (dae) * Remove unused ftl string * Delimiter → Separator * Remove commented-out line * Don't accept .json files * Tweak tag ftl strings * Remove redundant blur call * Strip sound and add spaces in csv export * Export HTML by default * Fix unset deck in Mnemosyne import Also accept both numbers and strings for notetypes and decks in JSON. * Make DupeResolution::Update the default * Fix missing dot in extension * Make column indices 1-based * Remove StickContainer from TagEditor Fixes line breaking, border and z index on ImportCsvPage. * Assign different key combos to tag editors * Log all updated duplicates Add a log field for the true number of found notes. * Show identical notes as skipped * Split tag-editor into separate ts module (dae) * Add progress for CSV export * Add progress for text import * Tidy-ups after tag-editor split (dae) - import-csv no longer depends on editor - remove some commented lines
2022-06-01 12:26:16 +02:00
.type_attribute(
"CsvMetadata.DupeResolution",
Plaintext import/export (#1850) * Add crate csv * Add start of csv importing on backend * Add Menomosyne serializer * Add csv and json importing on backend * Add plaintext importing on frontend * Add csv metadata extraction on backend * Add csv importing with GUI * Fix missing dfa file in build Added compile_data_attr, then re-ran cargo/update.py. * Don't use doubly buffered reader in csv * Escape HTML entities if CSV is not HTML Also use name 'is_html' consistently. * Use decimal number as foreign ease (like '2.5') * ForeignCard.ivl → ForeignCard.interval * Only allow fixed set of CSV delimiters * Map timestamp of ForeignCard to native due time * Don't trim CSV records * Document use of empty strings for defaults * Avoid creating CardGenContexts for every note This requires CardGenContext to be generic, so it works both with an owned and borrowed notetype. * Show all accepted file types in import file picker * Add import_json_file() * factor → ease_factor * delimter_from_value → delimiter_from_value * Map columns to fields, not the other way around * Fallback to current config for csv metadata * Add start of new import csv screen * Temporary fix for compilation issue on Linux/Mac * Disable jest bazel action for import-csv Jest fails with an error code if no tests are available, but this would not be noticable on Windows as Jest is not run there. * Fix field mapping issue * Revert "Temporary fix for compilation issue on Linux/Mac" This reverts commit 21f8a261408cdae49ec031aa21a1b659c4f66d82. * Add HtmlSwitch and move Switch to components * Fix spacing and make selectors consistent * Fix shortcut tooltip * Place import button at the top with path * Fix meta column indices * Remove NotetypeForString * Fix queue and type of foreign cards * Support different dupe resolution strategies * Allow dupe resolution selection when importing CSV * Test import of unnormalized text Close #1863. * Fix logging of foreign notes * Implement CSV exports * Use db_scalar() in notes_table_len() * Rework CSV metadata - Notetypes and decks are either defined by a global id or by a column. - If a notetype id is provided, its field map must also be specified. - If a notetype column is provided, fields are now mapped by index instead of name at import time. So the first non-meta column is used for the first field of every note, regardless of notetype. This makes importing easier and should improve compatiblity with files without a notetype column. - Ensure first field can be mapped to a column. - Meta columns must be defined as `#[meta name]:[column index]` instead of in the `#columns` tag. - Column labels contain the raw names defined by the file and must be prettified by the frontend. * Adjust frontend to new backend column mapping * Add force flags for is_html and delimiter * Detect if CSV is HTML by field content * Update dupe resolution labels * Simplify selectors * Fix coalescence of oneofs in TS * Disable meta columns from selection Plus a lot of refactoring. * Make import button stick to the bottom * Write delimiter and html flag into csv * Refetch field map after notetype change * Fix log labels for csv import * Log notes whose deck/notetype was missing * Fix hiding of empty log queues * Implement adding tags to all notes of a csv * Fix dupe resolution not being set in log * Implement adding tags to updated notes of a csv * Check first note field is not empty * Temporary fix for build on Linux/Mac * Fix inverted html check (dae) * Remove unused ftl string * Delimiter → Separator * Remove commented-out line * Don't accept .json files * Tweak tag ftl strings * Remove redundant blur call * Strip sound and add spaces in csv export * Export HTML by default * Fix unset deck in Mnemosyne import Also accept both numbers and strings for notetypes and decks in JSON. * Make DupeResolution::Update the default * Fix missing dot in extension * Make column indices 1-based * Remove StickContainer from TagEditor Fixes line breaking, border and z index on ImportCsvPage. * Assign different key combos to tag editors * Log all updated duplicates Add a log field for the true number of found notes. * Show identical notes as skipped * Split tag-editor into separate ts module (dae) * Add progress for CSV export * Add progress for text import * Tidy-ups after tag-editor split (dae) - import-csv no longer depends on editor - remove some commented lines
2022-06-01 12:26:16 +02:00
"#[derive(serde_derive::Deserialize, serde_derive::Serialize)]",
)
2021-07-12 08:15:38 +02:00
.compile_protos(paths.as_slice(), &[proto_dir])
.unwrap();
}
/// If PROTOC is not defined, and protoc is not on path, use the protoc
/// fetched by Bazel so that Rust Analyzer does not fail.
fn maybe_add_protobuf_to_path() {
if std::env::var("PROTOC").is_ok() {
return;
}
if which::which("protoc").is_ok() {
return;
}
let path = if cfg!(target_os = "windows") {
let base = std::fs::read_link("../.bazel/out").unwrap();
base.join("../external/protoc_bin_windows/bin/protoc.exe")
} else {
let base = Path::new("../.bazel/out/../external");
let subpath = if cfg!(target_os = "macos") {
"protoc_bin_macos/bin/protoc"
} else {
"protoc_bin_linux_x86_64/bin/protoc"
};
base.join(subpath)
};
std::env::set_var("PROTOC", path.to_str().unwrap());
}