Skip to content

Bulk-scan single-line string bodies - #491

Merged
frostming merged 1 commit into
python-poetry:masterfrom
AstekGroup:perf/string-scan
Jun 10, 2026
Merged

Bulk-scan single-line string bodies#491
frostming merged 1 commit into
python-poetry:masterfrom
AstekGroup:perf/string-scan

Conversation

@tfoutrein

@tfoutrein tfoutrein commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Stacked on #489 and #490 — the first two commits here are those PRs (index-based Source + bulk run scanning). Best reviewed/merged after them; GitHub will collapse this to just the string-scan commit once they land. Happy to rebase.

What

Parsing a single-line string appended its body one character at a time (value += current; inc()). For long string values (the bulk of most config / lockfile content) this dominates.

This scans the run of ordinary characters up to the next delimiter, backslash or control character in a single pass (Source.advance_until) and appends the whole slice at once; the stop character is then handled by the existing branch on the next iteration. Multiline strings keep the per-character loop (CRLF handling).

The stop-set is exactly the control characters the per-character loop rejects, so InvalidControlChar / escape / delimiter handling is unchanged, and a mid-string EOF raises UnexpectedEofError just as the per-char inc(exception=...) did.

Benchmarks

Parsing speedup across document shapes (median, interleaved A/B vs master, includes #489+#490):

document speedup
large flat, single-line string values (~90 KB) 4.9×
pyproject.toml 2.1×
poetry.lock-like (~64 KB) 2.1×
typical mixed (~4 KB) 1.5×

No regression on any shape (multiline-heavy and nested docs are unchanged).

Tests

Full suite passes (972 tests, incl. the toml-test conformance submodule). On top of that, a 4135-input adversarial differential (random escapes valid+invalid, every control byte 0x00–0x1F+DEL in basic & literal, unicode/astral/combining, the other-quote char inside strings, truncated/malformed inputs for error parity) is byte-identical in output and exception type to the per-character loop. No public API or behaviour change.

@tfoutrein
tfoutrein force-pushed the perf/string-scan branch 2 times, most recently from 421172e to e08a656 Compare June 5, 2026 17:39
This was referenced Jun 6, 2026
Closed
Merged
@frostming

frostming commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

@tfoutrein please resolve the conflicts

@tfoutrein
Bulk-scan single-line string bodies
1ec5f04 
Parsing a single-line string appended its body one character at a time
(`value += current; inc()`). For long string values this dominates.

Scan the run of ordinary characters up to the next delimiter, backslash
or control character in a single pass (`Source.advance_until`) and append
the whole slice at once; the stop character is then handled by the
existing branch on the next iteration. Multiline strings keep the
per-character loop (CRLF handling). The stop-set is exactly the control
characters the per-character loop rejects, so InvalidControlChar / escape
/ delimiter handling is unchanged.

No behaviour change (972 tests incl. the toml-test conformance submodule;
plus a 4135-input adversarial differential — output and error-type
byte-identical to the per-char loop). Up to ~5x faster parsing on
string-heavy single-line documents.
@tfoutrein

tfoutrein commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto master now that #490 has merged — this is a single commit on top of current master, no conflicts. CI is green. Ready for squash-merge whenever you'd like. Thanks!

@frostming
frostming merged commit 4623b45 into python-poetry:master Jun 10, 2026
25 checks passed
@tfoutrein tfoutrein mentioned this pull request Jun 15, 2026
Merged
tfoutrein added a commit to AstekGroup/tomlkit that referenced this pull request Jun 15, 2026
@tfoutrein
Bulk-scan multiline string bodies
1147a02 
python-poetry#491 bulk-scanned single-line string bodies but left multiline ("""/''')
bodies on the per-character loop because of \r\n handling. Extend the same
fast path to multiline: append the run of ordinary characters in one slice up
to the next delimiter, backslash, carriage return or control character.

Raw line feeds and tabs are valid inside a multiline body, so they are NOT
stop characters -- a whole multi-line body is consumed in a single pass. A
carriage return still stops the scan, so the existing branch keeps validating
the \r\n pair (and rejecting a lone \r) and \r\n is preserved byte-for-byte.
Invalid control characters and DEL also stop the scan, so they are still
rejected at the same position.

No behaviour change: full suite incl. toml-test conformance passes; verified
by a differential over ~122k generated multiline strings (LF / CRLF / bare CR,
tabs, lone quotes, escapes, control bytes, unicode/astral, EOF truncation) --
byte-identical round-trip and identical exception class/line/col. ~23x faster
parsing an LF multiline-heavy document (~11x with CRLF).
frostming added a commit that referenced this pull request Jun 18, 2026
@tfoutrein @frostming
Bulk-scan multiline string bodies (#518)
c468c97 
#491 bulk-scanned single-line string bodies but left multiline ("""/''')
bodies on the per-character loop because of \r\n handling. Extend the same
fast path to multiline: append the run of ordinary characters in one slice up
to the next delimiter, backslash, carriage return or control character.

Raw line feeds and tabs are valid inside a multiline body, so they are NOT
stop characters -- a whole multi-line body is consumed in a single pass. A
carriage return still stops the scan, so the existing branch keeps validating
the \r\n pair (and rejecting a lone \r) and \r\n is preserved byte-for-byte.
Invalid control characters and DEL also stop the scan, so they are still
rejected at the same position.

No behaviour change: full suite incl. toml-test conformance passes; verified
by a differential over ~122k generated multiline strings (LF / CRLF / bare CR,
tabs, lone quotes, escapes, control bytes, unicode/astral, EOF truncation) --
byte-identical round-trip and identical exception class/line/col. ~23x faster
parsing an LF multiline-heavy document (~11x with CRLF).

Co-authored-by: Frost Ming <me@frostming.com>
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Jul 26, 2026
@0-wiz-0
py-tomlkit: update to 0.15.1.
cdc2158 
## [0.15.1] - 2026-07-17

### Changed

- Speed up membership tests (`key in ...`) on `Container`, `Table` and `InlineTable` with native `__contains__` implementations, avoiding the inherited `MutableMapping` round-trip through `__getitem__` (which resolves the value and builds an exception on every absent key). ([#483](python-poetry/tomlkit#483))
- Speed up parsing by making `Source` index-based: it now tracks an integer position over the input string instead of materializing a list of `(index, char)` tuples up front, so construction is O(1) and state save/restore no longer copies an iterator. ([#489](python-poetry/tomlkit#489))
- Speed up parsing by scanning character runs in bulk: `Source.advance_while`/`advance_until` consume a whole run of whitespace, bare-key or number characters in a single pass over the input string instead of one `inc()` call per character. ([#490](python-poetry/tomlkit#490))
- Speed up parsing of single-line strings by bulk-appending the run of ordinary characters up to the next delimiter, backslash or control character in one pass, instead of one character at a time. ([#491](python-poetry/tomlkit#491))
- Speed up parsing by removing the internal `TOMLChar` wrapper: the parser now reads plain `str` characters from `Source` and detects end-of-input positionally, avoiding a per-character object construction and method dispatch. ([#492](python-poetry/tomlkit#492))
- Speed up parsing by comparing `StringType` members by identity (`is`) instead of building a set on every `is_basic`/`is_literal`/`is_singleline`/`is_multiline` call, avoiding millions of enum hashes while parsing. ([#502](python-poetry/tomlkit#502))
- Speed up merging super tables by merging in place instead of deep-copying the growing target on every merge, turning the parse of documents with many subtables under a shared super table (e.g. consecutive `[a.b.c]` / `[a.b.d]` headers) from O(n²) into O(n). ([#503](python-poetry/tomlkit#503))
- Speed up membership tests (`key in ...`) on out-of-order tables with a native `OutOfOrderTableProxy.__contains__`, completing [#483](python-poetry/tomlkit#483) for the last mapping type that still inherited the slow `MutableMapping` mixin (which resolves the value and builds an exception on every absent key). ([#515](python-poetry/tomlkit#515))
- Speed up parsing documents with many dotted keys or table headers sharing a prefix by validating out-of-order tables incrementally: each new fragment is merged into a cached validation container once, instead of re-merging (and deep-copying) every earlier fragment on each append, turning a super-cubic worst case into linear time (80 shared-prefix dotted keys: ~8 s → ~10 ms). ([#479](python-poetry/tomlkit#479))
- Speed up parsing of arrays that close right after a value (e.g. the `files = [...]` blocks that dominate lock files): the parser no longer attempts to read a value while sitting on the closing `]`, which previously built an `UnexpectedCharError` just to discard it — and constructing that exception eagerly computes a line/column by scanning the whole document, making it O(document size) per such array. ([#517](python-poetry/tomlkit#517))
- Speed up parsing of multiline strings by bulk-appending the run of ordinary characters — across raw line feeds and tabs — up to the next delimiter, backslash, carriage return or control character, instead of one character at a time. This extends to `"""`/`'''` bodies the single-line fast path added in [#491](python-poetry/tomlkit#491); a `\r` still stops the scan so `\r\n` stays validated and byte-for-byte preserved. ([#518](python-poetry/tomlkit#518))
- Speed up `unwrap()` (converting a parsed document to a plain `dict`) by resolving each key directly from the container's key map instead of iterating the inherited `MutableMapping` view, which rebuilt a `SingleKey` from the bare string for every key just to re-look-up the value. Out-of-order tables still resolve through their proxy, so their validation is unchanged. ([#521](python-poetry/tomlkit#521))
- Speed up rendering (`as_string()` / `dumps()`) of inline tables with many keys by precomputing the last-key and last-deleted-element indices in a single pass, instead of rescanning the remaining body on every separator comma — turning an O(n²) render into O(n). ([#525](python-poetry/tomlkit#525))
- Raise on malformed array element instead of dropping it, ([#527](python-poetry/tomlkit#527))

### Fixed

- Fix `string()` dropping a leading newline of a multiline string on round-trip: a value beginning with a newline is now rendered with an extra leading newline (the one the parser trims after the opening delimiter) so it survives re-parsing.
- Fix invalid serialization with a duplicated comma when removing a non-edge element from a parsed inline table. ([#486](python-poetry/tomlkit#486))
- Fix invalid serialization with a duplicated comma when appending or inserting into a comma-first formatted array. ([#499](python-poetry/tomlkit#499))
- Fix `ParseError` when a sub-table extends the last element of an array of tables after an unrelated table. ([#261](python-poetry/tomlkit#261))
- Fix unparseable serialization when adding a key to a dotted-key table inside an inline table. ([#500](python-poetry/tomlkit#500))
- Fix a table replaced by a plain value being serialized inside the preceding table's body when other tables follow; the value now moves before the first table like other root-level values. ([#504](python-poetry/tomlkit#504))
- Fix assigning a table over a dotted key (e.g. `doc["a"] = {...}` where `a` came from `a.b = ...`): the dotted prefix was duplicated onto the new `[a]` header, and the header then swallowed any sibling that follows it on round-trip. The replacement now renders as a plain table and, when needed, moves before the inline entries (values and dotted keys) it would otherwise capture. ([#513](python-poetry/tomlkit#513), [#524](python-poetry/tomlkit#524))
- Restore `dumps()` rendering mapping-like wrappers around a parsed document (e.g. `dotty_dict`'s `Dotty`) through their delegated `as_string`, preserving the original table order and layout instead of re-encoding through a plain dict — a 0.15.0 regression. ([#482](python-poetry/tomlkit#482))
- Fix uncontrolled recursion when parsing deeply nested documents: crafted input could crash the process with a `RecursionError`. Values nested more than 100 levels deep and keys with more than 100 dotted fragments now raise `ParseError`. ([#459](python-poetry/tomlkit#459))
- Fix `comment()` producing invalid TOML for a multiline string by prefixing every line with `#`, not just the first. ([#449](python-poetry/tomlkit#449))
- Fix the separator comma being swallowed by a trailing comment when appending a key to a multiline inline table, leaving the new key without a separator so the result no longer round-trips. ([#512](python-poetry/tomlkit#512))
- Fix a `KeyAlreadyPresent` error when parsing or accessing an out-of-order table whose array-of-tables elements are split across the table's parts. ([#505](python-poetry/tomlkit#505))
- Out-of-order value-vs-table and dotted-key-vs-table redefinitions are now rejected at parse time instead of being silently accepted or raising only on access. The parser also detects when a non-dotted key is a prefix of an existing dotted key, matching the stdlib `tomllib` behaviour. ([#523](python-poetry/tomlkit#523))
- Reject tables inserted into inline tables instead of serializing invalid TOML. ([#531](python-poetry/tomlkit#531))
- Fix assigning an array of tables over a dotted key (e.g. `doc["a"] = aot(...)` where `a` came from `a.b = ...`): the new `[[a]]` header kept the dotted key's inline position and swallowed the following dotted sibling on round-trip. The array of tables now renders past the inline entries it would otherwise capture, mirroring the table fix for [#513](python-poetry/tomlkit#513). ([#542](python-poetry/tomlkit#542))
- Fix a new top-level scalar being captured by a table rendered from a dotted key: appending a scalar after a dotted-key entry (e.g. `a.b = 1`) whose table had gained a `[a.c]`-style child placed the scalar inside that table's scope, silently re-nesting it on round-trip. Scalars now move before such an entry, like they do before regular tables. ([#543](python-poetry/tomlkit#543))
- Fix invalid serialization with a duplicated `[table]` header when adding a key to an out-of-order table whose concrete header is declared after its sub-tables; the new key now lands in the existing concrete part instead of giving the header-less super part a second header. ([#545](python-poetry/tomlkit#545))
- Fix a table's display name (its exact header spelling, including whitespace and quoting) being normalised when the table is assigned onto itself, e.g. `doc[k] = doc[k]` rewriting `[keys .'a'.'c']` to `[keys.a.'c']`. ([#291](python-poetry/tomlkit#291))
- Fix missing newlines when appending a key after a dotted inline table, including when the original document has no trailing newline. ([#533](python-poetry/tomlkit#533))
- Preserve trailing whitespace when replacing a super table, including assigning it onto itself. ([#534](python-poetry/tomlkit#534))
- Fix `str()` and `repr()` of out-of-order table proxies to show their merged values. ([#536](python-poetry/tomlkit#536))
- Reject decimal integer literals that exceed Python's integer-string conversion limit instead of coercing them to infinity. ([#538](python-poetry/tomlkit#538))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@frostming frostming frostming approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tfoutrein @frostming