Skip to content

gh-141336: Fix Unicode escape assertion failure in error handler #141344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

gh-141336: Fix Unicode escape assertion failure in error handler #141344

wants to merge 2 commits into from
+12 −2

Conversation

@mohsinm-dev
Copy link
Contributor

@mohsinm-dev mohsinm-dev commented Nov 10, 2025 •
edited by bedevere-app bot
Loading

Fixes assertion failure in Unicode escape decoding when using custom error handlers that return single-character replacements and rewind input position.

Root Cause

The issue occurs in unicode_decode_call_errorhandler_writer() when:

  1. Error handler returns a replacement string of length 1 (replen == 1)
  2. Error handler rewinds input position (increasing remaining input bytes)
  3. Original code failed to account for replacement length in buffer calculations

This led to assertion failures in debug builds.

Fix

  • Always include full replacement length in writer.min_length calculations
  • Change condition from replen > 1 to replen > 0
  • Add full replen instead of replen - 1 to maintain buffer invariant

Testing

  • Reproduction case now works without assertion failure
  • All codec callback tests pass (43/43)
  • Specific mutating decode handler test passes
  • No regressions detected in broader codec tests

Fixes #141336

@mohsinm-dev
pythongh-141336: Fix Unicode escape assertion failure in error handler
4f20106 
When unicode_decode_call_errorhandler_writer() processes a replacement
string of length 1 and the error handler rewinds the input position
(increasing remaining input), the original code failed to account for
the replacement length in writer.min_length calculations.

This resulted in assertion failures when writer capacity was insufficient
after writing the replacement: assert(end - s <= writer.size - writer.pos).

The fix ensures the full replacement length is always included in
writer.min_length, regardless of replacement size, maintaining the
buffer capacity invariant for all decoders using this error helper.
@bedevere-app bedevere-app bot mentioned this pull request Nov 10, 2025
Open
@serhiy-storchaka serhiy-storchaka self-requested a review November 10, 2025 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka Awaiting requested review from serhiy-storchaka

Assignees

No one assigned

Labels

awaiting review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Assertion failure in Objects/unicodeobject.c _PyUnicode_DecodeUnicodeEscapeInternal2: Assertion 'end - s <= writer.size - writer.pos' failed

1 participant

@mohsinm-dev