anki/rslib/BUILD.bazel
RumovZ 42cbe42f06
Plaintext import/export (#1850)
* Add crate csv

* Add start of csv importing on backend

* Add Menomosyne serializer

* Add csv and json importing on backend

* Add plaintext importing on frontend

* Add csv metadata extraction on backend

* Add csv importing with GUI

* Fix missing dfa file in build

Added compile_data_attr, then re-ran cargo/update.py.

* Don't use doubly buffered reader in csv

* Escape HTML entities if CSV is not HTML

Also use name 'is_html' consistently.

* Use decimal number as foreign ease (like '2.5')

* ForeignCard.ivl → ForeignCard.interval

* Only allow fixed set of CSV delimiters

* Map timestamp of ForeignCard to native due time

* Don't trim CSV records

* Document use of empty strings for defaults

* Avoid creating CardGenContexts for every note

This requires CardGenContext to be generic, so it works both with an
owned and borrowed notetype.

* Show all accepted file types  in import file picker

* Add import_json_file()

* factor → ease_factor

* delimter_from_value → delimiter_from_value

* Map columns to fields, not the other way around

* Fallback to current config for csv metadata

* Add start of new import csv screen

* Temporary fix for compilation issue on Linux/Mac

* Disable jest bazel action for import-csv

Jest fails with an error code if no tests are available, but this would
not be noticable on Windows as Jest is not run there.

* Fix field mapping issue

* Revert "Temporary fix for compilation issue on Linux/Mac"

This reverts commit 21f8a261408cdae49ec031aa21a1b659c4f66d82.

* Add HtmlSwitch and move Switch to components

* Fix spacing and make selectors consistent

* Fix shortcut tooltip

* Place import button at the top with path

* Fix meta column indices

* Remove NotetypeForString

* Fix queue and type of foreign cards

* Support different dupe resolution strategies

* Allow dupe resolution selection when importing CSV

* Test import of unnormalized text

Close  #1863.

* Fix logging of foreign notes

* Implement CSV exports

* Use db_scalar() in notes_table_len()

* Rework CSV metadata

- Notetypes and decks are either defined by a global id or by a column.
- If a notetype id is provided, its field map must also be specified.
- If a notetype column is provided, fields are now mapped by index
instead of name at import time. So the first non-meta column is used for
the first field of every note, regardless of notetype. This makes
importing easier and should improve compatiblity with files without a
notetype column.
- Ensure first field can be mapped to a column.
- Meta columns must be defined as `#[meta name]:[column index]` instead
of in the `#columns` tag.
- Column labels contain the raw names defined by the file and must be
prettified by the frontend.

* Adjust frontend to new backend column mapping

* Add force flags for is_html and delimiter

* Detect if CSV is HTML by field content

* Update dupe resolution labels

* Simplify selectors

* Fix coalescence of oneofs in TS

* Disable meta columns from selection

Plus a lot of refactoring.

* Make import button stick to the bottom

* Write delimiter and html flag into csv

* Refetch field map after notetype change

* Fix log labels for csv import

* Log notes whose deck/notetype was missing

* Fix hiding of empty log queues

* Implement adding tags to all notes of a csv

* Fix dupe resolution not being set in log

* Implement adding tags to updated notes of a csv

* Check first note field is not empty

* Temporary fix for build on Linux/Mac

* Fix inverted html check (dae)

* Remove unused ftl string

* Delimiter → Separator

* Remove commented-out line

* Don't accept .json files

* Tweak tag ftl strings

* Remove redundant blur call

* Strip sound and add spaces in csv export

* Export HTML by default

* Fix unset deck in Mnemosyne import

Also accept both numbers and strings for notetypes and decks in JSON.

* Make DupeResolution::Update the default

* Fix missing dot in extension

* Make column indices 1-based

* Remove StickContainer from TagEditor

Fixes line breaking, border and z index on ImportCsvPage.

* Assign different key combos to tag editors

* Log all updated duplicates

Add a log field for the true number of found notes.

* Show identical notes as skipped

* Split tag-editor into separate ts module (dae)

* Add progress for CSV export

* Add progress for text import

* Tidy-ups after tag-editor split (dae)

- import-csv no longer depends on editor
- remove some commented lines
2022-06-01 20:26:16 +10:00

173 lines
4.4 KiB
Python

# Copyright: Ankitects Pty Ltd and contributors
# License: GNU AGPL, version 3 or later; http://www.gnu.org/licenses/agpl.html
load("@rules_rust//rust:defs.bzl", "rust_library", "rust_test")
load("@rules_rust//cargo:cargo_build_script.bzl", "cargo_build_script")
load(":rustfmt.bzl", "rustfmt_fix", "rustfmt_test")
load("//ts/sql_format:defs.bzl", "sql_format")
# Build script
#######################
cargo_build_script(
name = "build_script",
srcs = glob(["build/*.rs"]),
build_script_env = {
"PROTO_TOP": "$(location //proto:.top_level)",
"PROTOC": "$(location @com_google_protobuf//:protoc)",
"RSLIB_FTL_ROOT": "$(location @rslib_ftl//:l10n.toml)",
"EXTRA_FTL_ROOT": "$(location @extra_ftl//:l10n.toml)",
"BAZEL": "1",
},
crate_root = "build/main.rs",
data = [
"//ftl",
"//proto",
"@com_google_protobuf//:protoc",
# bazel requires us to list these out separately
"//proto:.top_level",
"@rslib_ftl//:l10n.toml",
"@extra_ftl//:l10n.toml",
],
deps = [
"//rslib/cargo:prost_build",
],
)
# Library
#######################
_anki_compile_data = glob([
"src/**/*.sql",
"src/**/*.html",
"src/**/*.css",
"src/**/*.tex",
]) + [
"Cargo.toml", # prevents a warning about num_enum
"//:buildinfo.txt",
]
_anki_features = [
"translations",
]
_anki_rustc_env = {
"BUILDINFO": "$(location //:buildinfo.txt)",
}
rust_library(
name = "anki",
srcs = glob([
"src/**/*.rs",
]),
compile_data = _anki_compile_data,
crate_features = _anki_features,
proc_macro_deps = [
"//rslib/cargo:async_trait",
"//rslib/cargo:serde_derive",
"//rslib/cargo:serde_repr",
],
rustc_env = _anki_rustc_env,
visibility = ["//visibility:public"],
deps = [
":build_script",
"//rslib/cargo:ammonia",
"//rslib/cargo:blake3",
"//rslib/cargo:bytes",
"//rslib/cargo:chrono",
"//rslib/cargo:coarsetime",
"//rslib/cargo:csv",
"//rslib/cargo:flate2",
"//rslib/cargo:fluent",
"//rslib/cargo:fnv",
"//rslib/cargo:futures",
"//rslib/cargo:hex",
"//rslib/cargo:htmlescape",
"//rslib/cargo:id_tree",
"//rslib/cargo:intl_memoizer",
"//rslib/cargo:itertools",
"//rslib/cargo:lazy_static",
"//rslib/cargo:nom",
"//rslib/cargo:num_cpus",
"//rslib/cargo:num_enum",
"//rslib/cargo:num_integer",
"//rslib/cargo:once_cell",
"//rslib/cargo:pin_project",
"//rslib/cargo:prost",
"//rslib/cargo:pulldown_cmark",
"//rslib/cargo:rand",
"//rslib/cargo:regex",
"//rslib/cargo:rusqlite",
"//rslib/cargo:scopeguard",
"//rslib/cargo:serde",
"//rslib/cargo:serde_aux",
"//rslib/cargo:serde_json",
"//rslib/cargo:serde_tuple",
"//rslib/cargo:sha1",
"//rslib/cargo:slog",
"//rslib/cargo:slog_async",
"//rslib/cargo:slog_envlogger",
"//rslib/cargo:slog_term",
"//rslib/cargo:strum",
"//rslib/cargo:tempfile",
"//rslib/cargo:tokio",
"//rslib/cargo:tokio_util",
"//rslib/cargo:unic_langid",
"//rslib/cargo:unicase",
"//rslib/cargo:unicode_normalization",
"//rslib/cargo:unic_ucd_category",
"//rslib/cargo:utime",
"//rslib/cargo:zip",
"//rslib/cargo:zstd",
"//rslib/cargo:pct_str",
"//rslib/i18n:anki_i18n",
] + select({
# rustls on Linux
"//platforms:linux_x86_64": ["@reqwest_rustls//:reqwest"],
"//platforms:linux_arm64": ["@reqwest_rustls//:reqwest"],
# native tls on other platforms
"//conditions:default": ["//rslib/cargo:reqwest"],
}),
)
# Tests
#######################
rust_test(
name = "anki_tests",
compile_data = _anki_compile_data,
crate = ":anki",
crate_features = _anki_features,
data = glob([
"tests/support/**",
]),
rustc_env = _anki_rustc_env,
rustc_flags = [
"-Dclippy::dbg_macro",
],
deps = [
"//rslib/cargo:env_logger",
"//rslib/cargo:linkcheck",
"//rslib/cargo:utime",
],
)
rustfmt_test(
name = "format_check",
srcs = glob([
"**/*.rs",
]),
)
rustfmt_fix(
name = "format",
srcs = glob([
"**/*.rs",
]),
)
sql_format(
name = "sql_format",
srcs = glob(["**/*.sql"]),
)