anki/ts/lib/proto.ts

84 lines
2.4 KiB
TypeScript
Raw Normal View History

refactor protobuf handling for split/import In order to split backend.proto into a more manageable size, the protobuf handling needed to be updated. This took more time than I would have liked, as each language handles protobuf differently: - The Python Protobuf code ignores "package" directives, and relies solely on how the files are laid out on disk. While it would have been nice to keep the generated files in a private subpackage, Protobuf gets confused if the files are located in a location that does not match their original .proto layout, so the old approach of storing them in _backend/ will not work. They now clutter up pylib/anki instead. I'm rather annoyed by that, but alternatives seem to be having to add an extra level to the Protobuf path, making the other languages suffer, or trying to hack around the issue by munging sys.modules. - Protobufjs fails to expose packages if they don't start with a capital letter, despite the fact that lowercase packages are the norm in most languages :-( This required a patch to fix. - Rust was the easiest, as Prost is relatively straightforward compared to Google's tools. The Protobuf files are now stored in /proto/anki, with a separate package for each file. I've split backend.proto into a few files as a test, but the majority of that work is still to come. The Python Protobuf building is a bit of a hack at the moment, hard-coding "proto" as the top level folder, but it seems to get the job done for now. Also changed the workspace name, as there seems to be a number of Bazel repos moving away from the more awkward reverse DNS naming style.
2021-07-10 09:50:18 +02:00
// Copyright: Ankitects Pty Ltd and contributors
// License: GNU AGPL, version 3 or later; http://www.gnu.org/licenses/agpl.html
/* eslint
@typescript-eslint/no-explicit-any: "off",
*/
import type { Message, rpc, RPCImpl, RPCImplCallback } from "protobufjs";
refactor protobuf handling for split/import In order to split backend.proto into a more manageable size, the protobuf handling needed to be updated. This took more time than I would have liked, as each language handles protobuf differently: - The Python Protobuf code ignores "package" directives, and relies solely on how the files are laid out on disk. While it would have been nice to keep the generated files in a private subpackage, Protobuf gets confused if the files are located in a location that does not match their original .proto layout, so the old approach of storing them in _backend/ will not work. They now clutter up pylib/anki instead. I'm rather annoyed by that, but alternatives seem to be having to add an extra level to the Protobuf path, making the other languages suffer, or trying to hack around the issue by munging sys.modules. - Protobufjs fails to expose packages if they don't start with a capital letter, despite the fact that lowercase packages are the norm in most languages :-( This required a patch to fix. - Rust was the easiest, as Prost is relatively straightforward compared to Google's tools. The Protobuf files are now stored in /proto/anki, with a separate package for each file. I've split backend.proto into a few files as a test, but the majority of that work is still to come. The Python Protobuf building is a bit of a hack at the moment, hard-coding "proto" as the top level folder, but it seems to get the job done for now. Also changed the workspace name, as there seems to be a number of Bazel repos moving away from the more awkward reverse DNS naming style.
2021-07-10 09:50:18 +02:00
import { anki } from "./backend_proto";
Refactor i18n (#1405) Merging note: the typing changes were fixed in a separate PR. * Put rootDirs into subprojects - typings do not work for any ts or svelte files - if we set the 'rootDirs' in ts/tsconfig.json to '../bazel-bin/ts' and then inherit them from e.g. editor, the root will be changed to '../../bazel-bin/ts', however editor needs look in '../../bazel-bin/ts/editor' instead. * Rename i18n and i18n_helpers to i18n-generated and i18n - This way, we can restrict the awkwardness of importing files outside the ts directory within lib * Fix missing typing of i18n and backend_proto by adding back symlinks * Split up i18n-generated into i18n-{translate,modules} * Change i18n from singleton to functions * Revert "Put rootDirs into subprojects" This partially reverts commit e1d4292ce3979e7b7ee21bf3951b8a462d45c29c. It seems like this might not be necessary after all. However some other change made on this branch seems to have fixed the .svelte.d.ts imports * Introduce i18n-bundles to remove circular import There was a circular import i18n.ts <-> i18n-translate.ts * Create own directory for i18n * Move lib/i18n/translate to lib/translate * This restores tree shaking * Update tsconfig libs and module * es2018-2020 have wide support on all modern browsers including * Switch bundles and langs inside i18n to variables again * Add missing copyright header * Rename translate.ts to ftl.ts * Remove the symlinks again I added them to fix to have completion for tr, however this would have also have meant to abandon the tree shaking. As we want to have tree shaking, it's also not necessary to have the symlinks anymore * Revert "Update tsconfig libs and module" This reverts commit 0a96776a475e9901c1f9f3407c726d1d002fb9ef. * move withCollapsedWhitespace back to i18n/utils * Add back /ts as in rootDirs
2021-10-07 15:31:49 +02:00
2021-07-10 11:52:31 +02:00
import Cards = anki.cards;
import Collection = anki.collection;
import DeckConfig = anki.deckconfig;
import Decks = anki.decks;
import Generic = anki.generic;
import I18n = anki.i18n;
Plaintext import/export (#1850) * Add crate csv * Add start of csv importing on backend * Add Menomosyne serializer * Add csv and json importing on backend * Add plaintext importing on frontend * Add csv metadata extraction on backend * Add csv importing with GUI * Fix missing dfa file in build Added compile_data_attr, then re-ran cargo/update.py. * Don't use doubly buffered reader in csv * Escape HTML entities if CSV is not HTML Also use name 'is_html' consistently. * Use decimal number as foreign ease (like '2.5') * ForeignCard.ivl → ForeignCard.interval * Only allow fixed set of CSV delimiters * Map timestamp of ForeignCard to native due time * Don't trim CSV records * Document use of empty strings for defaults * Avoid creating CardGenContexts for every note This requires CardGenContext to be generic, so it works both with an owned and borrowed notetype. * Show all accepted file types in import file picker * Add import_json_file() * factor → ease_factor * delimter_from_value → delimiter_from_value * Map columns to fields, not the other way around * Fallback to current config for csv metadata * Add start of new import csv screen * Temporary fix for compilation issue on Linux/Mac * Disable jest bazel action for import-csv Jest fails with an error code if no tests are available, but this would not be noticable on Windows as Jest is not run there. * Fix field mapping issue * Revert "Temporary fix for compilation issue on Linux/Mac" This reverts commit 21f8a261408cdae49ec031aa21a1b659c4f66d82. * Add HtmlSwitch and move Switch to components * Fix spacing and make selectors consistent * Fix shortcut tooltip * Place import button at the top with path * Fix meta column indices * Remove NotetypeForString * Fix queue and type of foreign cards * Support different dupe resolution strategies * Allow dupe resolution selection when importing CSV * Test import of unnormalized text Close #1863. * Fix logging of foreign notes * Implement CSV exports * Use db_scalar() in notes_table_len() * Rework CSV metadata - Notetypes and decks are either defined by a global id or by a column. - If a notetype id is provided, its field map must also be specified. - If a notetype column is provided, fields are now mapped by index instead of name at import time. So the first non-meta column is used for the first field of every note, regardless of notetype. This makes importing easier and should improve compatiblity with files without a notetype column. - Ensure first field can be mapped to a column. - Meta columns must be defined as `#[meta name]:[column index]` instead of in the `#columns` tag. - Column labels contain the raw names defined by the file and must be prettified by the frontend. * Adjust frontend to new backend column mapping * Add force flags for is_html and delimiter * Detect if CSV is HTML by field content * Update dupe resolution labels * Simplify selectors * Fix coalescence of oneofs in TS * Disable meta columns from selection Plus a lot of refactoring. * Make import button stick to the bottom * Write delimiter and html flag into csv * Refetch field map after notetype change * Fix log labels for csv import * Log notes whose deck/notetype was missing * Fix hiding of empty log queues * Implement adding tags to all notes of a csv * Fix dupe resolution not being set in log * Implement adding tags to updated notes of a csv * Check first note field is not empty * Temporary fix for build on Linux/Mac * Fix inverted html check (dae) * Remove unused ftl string * Delimiter → Separator * Remove commented-out line * Don't accept .json files * Tweak tag ftl strings * Remove redundant blur call * Strip sound and add spaces in csv export * Export HTML by default * Fix unset deck in Mnemosyne import Also accept both numbers and strings for notetypes and decks in JSON. * Make DupeResolution::Update the default * Fix missing dot in extension * Make column indices 1-based * Remove StickContainer from TagEditor Fixes line breaking, border and z index on ImportCsvPage. * Assign different key combos to tag editors * Log all updated duplicates Add a log field for the true number of found notes. * Show identical notes as skipped * Split tag-editor into separate ts module (dae) * Add progress for CSV export * Add progress for text import * Tidy-ups after tag-editor split (dae) - import-csv no longer depends on editor - remove some commented lines
2022-06-01 12:26:16 +02:00
import ImportExport = anki.import_export;
import Notes = anki.notes;
import Notetypes = anki.notetypes;
import Scheduler = anki.scheduler;
import Stats = anki.stats;
import Tags = anki.tags;
update to latest rules_nodejs & switch to ts_project ts_library() is deprecated and will presumably be dropped from a future rules_nodejs, and it wasn't working with the jest tests after updating, so we switch over to ts_project(). There are some downsides: - It's a bit slower, as the worker mode doesn't appear to function at the moment. - Getting it working with a mix of source files and generated files was quite tricky, especially as things behave differently on Windows, and differently when editing with VS Code. Solved with a small patch to the rules, and a wrapper script that copies everything into the bin folder first. To keep VS Code working correctly as well, the built files are symlinked into the source folder. - TS libraries are not implicitly linked to node_modules, so they can't be imported with an absolute name like "lib/proto" - we need to use relative paths like "../lib/proto" instead. Adjusting "paths" in tsconfig.json makes it work for TS compilation, but then it fails at the esbuild stage. We could resolve it by wrapping the TS libraries in a subsequent js_library() call, but that has the downside of losing the transient dependencies, meaning they need to be listed again. Alternatively we might be able to solve it in the future by adjusting esbuild, but for now the paths have been made relative to keep things simple. Upsides: - Along with updates to the Svelte tooling, Svelte typing has improved. All exports made in a Svelte file are now visible to other files that import them, and we no longer rebuild the Svelte files when TS files are updated, as the Svelte files do no type checking themselves, and are just a simple transpilation. Svelte-check now works on Windows again, and there should be no errors when editing in VS Code after you've built the project. The only downside seems to be that cmd+clicking on a Svelte imports jumps to the .d.ts file instead of the original now; presumably they'll fix that in a future plugin update. - Each subfolder now has its own tsconfig.json, and tsc can be called directly for testing purposes (but beware it will place build products in the source tree): ts/node_modules/.bin/tsc -b ts - We can drop the custom esbuild_toolchain, as it's included in the latest rules_nodejs. Other changes: - "image_module_support" is moved into lib/, and imported with <reference types=...> - Images are now imported directly from their npm package; the extra copy step has been removed. Windows users may need to use "bazel clean" before building this, due to old files lying around in the build folder.
2021-09-30 14:16:29 +02:00
export { Cards, Collection, Decks, Generic, Notes };
export const empty = Generic.Empty.create();
async function serviceCallback(
method: rpc.ServiceMethod<Message<any>, Message<any>>,
requestData: Uint8Array,
callback: RPCImplCallback,
): Promise<void> {
const headers = new Headers();
headers.set("Content-type", "application/octet-stream");
const methodName = method.name[0].toLowerCase() + method.name.substring(1);
const path = `/_anki/${methodName}`;
try {
const result = await fetch(path, {
method: "POST",
headers,
body: requestData,
});
const blob = await result.blob();
const respBuf = await new Response(blob).arrayBuffer();
const uint8Array = new Uint8Array(respBuf);
callback(null, uint8Array);
} catch (error) {
console.log("error caught");
callback(error as Error, null);
}
}
Plaintext import/export (#1850) * Add crate csv * Add start of csv importing on backend * Add Menomosyne serializer * Add csv and json importing on backend * Add plaintext importing on frontend * Add csv metadata extraction on backend * Add csv importing with GUI * Fix missing dfa file in build Added compile_data_attr, then re-ran cargo/update.py. * Don't use doubly buffered reader in csv * Escape HTML entities if CSV is not HTML Also use name 'is_html' consistently. * Use decimal number as foreign ease (like '2.5') * ForeignCard.ivl → ForeignCard.interval * Only allow fixed set of CSV delimiters * Map timestamp of ForeignCard to native due time * Don't trim CSV records * Document use of empty strings for defaults * Avoid creating CardGenContexts for every note This requires CardGenContext to be generic, so it works both with an owned and borrowed notetype. * Show all accepted file types in import file picker * Add import_json_file() * factor → ease_factor * delimter_from_value → delimiter_from_value * Map columns to fields, not the other way around * Fallback to current config for csv metadata * Add start of new import csv screen * Temporary fix for compilation issue on Linux/Mac * Disable jest bazel action for import-csv Jest fails with an error code if no tests are available, but this would not be noticable on Windows as Jest is not run there. * Fix field mapping issue * Revert "Temporary fix for compilation issue on Linux/Mac" This reverts commit 21f8a261408cdae49ec031aa21a1b659c4f66d82. * Add HtmlSwitch and move Switch to components * Fix spacing and make selectors consistent * Fix shortcut tooltip * Place import button at the top with path * Fix meta column indices * Remove NotetypeForString * Fix queue and type of foreign cards * Support different dupe resolution strategies * Allow dupe resolution selection when importing CSV * Test import of unnormalized text Close #1863. * Fix logging of foreign notes * Implement CSV exports * Use db_scalar() in notes_table_len() * Rework CSV metadata - Notetypes and decks are either defined by a global id or by a column. - If a notetype id is provided, its field map must also be specified. - If a notetype column is provided, fields are now mapped by index instead of name at import time. So the first non-meta column is used for the first field of every note, regardless of notetype. This makes importing easier and should improve compatiblity with files without a notetype column. - Ensure first field can be mapped to a column. - Meta columns must be defined as `#[meta name]:[column index]` instead of in the `#columns` tag. - Column labels contain the raw names defined by the file and must be prettified by the frontend. * Adjust frontend to new backend column mapping * Add force flags for is_html and delimiter * Detect if CSV is HTML by field content * Update dupe resolution labels * Simplify selectors * Fix coalescence of oneofs in TS * Disable meta columns from selection Plus a lot of refactoring. * Make import button stick to the bottom * Write delimiter and html flag into csv * Refetch field map after notetype change * Fix log labels for csv import * Log notes whose deck/notetype was missing * Fix hiding of empty log queues * Implement adding tags to all notes of a csv * Fix dupe resolution not being set in log * Implement adding tags to updated notes of a csv * Check first note field is not empty * Temporary fix for build on Linux/Mac * Fix inverted html check (dae) * Remove unused ftl string * Delimiter → Separator * Remove commented-out line * Don't accept .json files * Tweak tag ftl strings * Remove redundant blur call * Strip sound and add spaces in csv export * Export HTML by default * Fix unset deck in Mnemosyne import Also accept both numbers and strings for notetypes and decks in JSON. * Make DupeResolution::Update the default * Fix missing dot in extension * Make column indices 1-based * Remove StickContainer from TagEditor Fixes line breaking, border and z index on ImportCsvPage. * Assign different key combos to tag editors * Log all updated duplicates Add a log field for the true number of found notes. * Show identical notes as skipped * Split tag-editor into separate ts module (dae) * Add progress for CSV export * Add progress for text import * Tidy-ups after tag-editor split (dae) - import-csv no longer depends on editor - remove some commented lines
2022-06-01 12:26:16 +02:00
export const decks = Decks.DecksService.create(serviceCallback as RPCImpl);
export { DeckConfig };
export const deckConfig = DeckConfig.DeckConfigService.create(
serviceCallback as RPCImpl,
);
export { I18n };
export const i18n = I18n.I18nService.create(serviceCallback as RPCImpl);
Plaintext import/export (#1850) * Add crate csv * Add start of csv importing on backend * Add Menomosyne serializer * Add csv and json importing on backend * Add plaintext importing on frontend * Add csv metadata extraction on backend * Add csv importing with GUI * Fix missing dfa file in build Added compile_data_attr, then re-ran cargo/update.py. * Don't use doubly buffered reader in csv * Escape HTML entities if CSV is not HTML Also use name 'is_html' consistently. * Use decimal number as foreign ease (like '2.5') * ForeignCard.ivl → ForeignCard.interval * Only allow fixed set of CSV delimiters * Map timestamp of ForeignCard to native due time * Don't trim CSV records * Document use of empty strings for defaults * Avoid creating CardGenContexts for every note This requires CardGenContext to be generic, so it works both with an owned and borrowed notetype. * Show all accepted file types in import file picker * Add import_json_file() * factor → ease_factor * delimter_from_value → delimiter_from_value * Map columns to fields, not the other way around * Fallback to current config for csv metadata * Add start of new import csv screen * Temporary fix for compilation issue on Linux/Mac * Disable jest bazel action for import-csv Jest fails with an error code if no tests are available, but this would not be noticable on Windows as Jest is not run there. * Fix field mapping issue * Revert "Temporary fix for compilation issue on Linux/Mac" This reverts commit 21f8a261408cdae49ec031aa21a1b659c4f66d82. * Add HtmlSwitch and move Switch to components * Fix spacing and make selectors consistent * Fix shortcut tooltip * Place import button at the top with path * Fix meta column indices * Remove NotetypeForString * Fix queue and type of foreign cards * Support different dupe resolution strategies * Allow dupe resolution selection when importing CSV * Test import of unnormalized text Close #1863. * Fix logging of foreign notes * Implement CSV exports * Use db_scalar() in notes_table_len() * Rework CSV metadata - Notetypes and decks are either defined by a global id or by a column. - If a notetype id is provided, its field map must also be specified. - If a notetype column is provided, fields are now mapped by index instead of name at import time. So the first non-meta column is used for the first field of every note, regardless of notetype. This makes importing easier and should improve compatiblity with files without a notetype column. - Ensure first field can be mapped to a column. - Meta columns must be defined as `#[meta name]:[column index]` instead of in the `#columns` tag. - Column labels contain the raw names defined by the file and must be prettified by the frontend. * Adjust frontend to new backend column mapping * Add force flags for is_html and delimiter * Detect if CSV is HTML by field content * Update dupe resolution labels * Simplify selectors * Fix coalescence of oneofs in TS * Disable meta columns from selection Plus a lot of refactoring. * Make import button stick to the bottom * Write delimiter and html flag into csv * Refetch field map after notetype change * Fix log labels for csv import * Log notes whose deck/notetype was missing * Fix hiding of empty log queues * Implement adding tags to all notes of a csv * Fix dupe resolution not being set in log * Implement adding tags to updated notes of a csv * Check first note field is not empty * Temporary fix for build on Linux/Mac * Fix inverted html check (dae) * Remove unused ftl string * Delimiter → Separator * Remove commented-out line * Don't accept .json files * Tweak tag ftl strings * Remove redundant blur call * Strip sound and add spaces in csv export * Export HTML by default * Fix unset deck in Mnemosyne import Also accept both numbers and strings for notetypes and decks in JSON. * Make DupeResolution::Update the default * Fix missing dot in extension * Make column indices 1-based * Remove StickContainer from TagEditor Fixes line breaking, border and z index on ImportCsvPage. * Assign different key combos to tag editors * Log all updated duplicates Add a log field for the true number of found notes. * Show identical notes as skipped * Split tag-editor into separate ts module (dae) * Add progress for CSV export * Add progress for text import * Tidy-ups after tag-editor split (dae) - import-csv no longer depends on editor - remove some commented lines
2022-06-01 12:26:16 +02:00
export { ImportExport };
export const importExport = ImportExport.ImportExportService.create(
serviceCallback as RPCImpl,
);
export { Notetypes };
export const notetypes = Notetypes.NotetypesService.create(serviceCallback as RPCImpl);
export { Scheduler };
export const scheduler = Scheduler.SchedulerService.create(serviceCallback as RPCImpl);
export { Stats };
export const stats = Stats.StatsService.create(serviceCallback as RPCImpl);
export { Tags };
export const tags = Tags.TagsService.create(serviceCallback as RPCImpl);