⚠ This page is served via a proxy. Original site: https://github.com
This service does not collect credentials or authentication data.
Skip to content

Conversation

@jeffhuen
Copy link

Summary

This PR modernizes excoding while maintaining full backwards compatibility:

  • Switch to encoding_rs - The same encoding library used by Firefox. Better performance, actively maintained, and WHATWG compliant
  • Update dependencies - rustler 0.29 → 0.37, rustler_precompiled 0.5 → 0.8, Rust edition 2018 → 2021
  • Add dirty schedulers - Operations on binaries >64KB automatically use dirty CPU schedulers to avoid blocking the BEAM
  • Better error handling - All Rust .unwrap() calls replaced with proper error propagation (no more panics)

Issues Fixed

New Functions

# Tuple-based error handling (new)
Excoding.safe_encode(string, encoding)  # => {:ok, binary} | {:error, :unknown_encoding}
Excoding.safe_decode(binary, encoding)  # => {:ok, string} | {:error, :unknown_encoding}

# Utilities (new)
Excoding.encoding_exists?("utf-8")       # => true
Excoding.canonical_name("latin1")        # => {:ok, "windows-1252"}
Excoding.list_encodings()                # => ["UTF-8", "Shift_JIS", ...]
Excoding.dirty_threshold()               # => 65536

Backwards Compatibility

  • encode/2 and decode/2 work exactly as before (return raw value, raise on error)
  • No breaking changes to existing API

Test Plan

  • All existing tests pass
  • Added tests for new functions
  • Roundtrip tests for various encodings (UTF-8, Shift_JIS, EUC-KR, Windows-125x)
  • mix format passes
  • mix compile --warnings-as-errors passes
  • cargo clippy passes

Happy to adjust anything based on feedback!

🤖 Generated with Claude Code

jeffhuen and others added 15 commits January 22, 2026 10:51
Major update that modernizes the codebase while maintaining backwards
compatibility:

- Switch from `encoding` crate to `encoding_rs` (Firefox's encoding library)
- Update rustler 0.29.1 -> 0.37 (fixes OTP-26+ compilation, closes #40)
- Update rustler_precompiled 0.5 -> 0.8
- Update Rust edition 2018 -> 2021
- Remove unused deps (lazy_static, rustler_codegen)
- Rename .cargo/config to .cargo/config.toml (cargo deprecation warning)

New features:
- Dirty CPU schedulers for binaries >64KB (non-blocking for large data)
- safe_encode/2, safe_decode/2 returning {:ok, _} | {:error, _} tuples
- encoding_exists?/1 to check if encoding is supported
- canonical_name/1 to get WHATWG canonical name for aliases
- list_encodings/0 to list all supported encodings

Bug fixes:
- No more panics in Rust code - proper error handling throughout (closes #24)
- All .unwrap() calls replaced with proper error propagation

Backwards compatible:
- encode/2 and decode/2 maintain same behavior as v0.1.x

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- encode/2 and decode/2 now return {:ok, result} | {:error, reason}
- encode!/2 and decode!/2 return raw value or raise
- Removed safe_encode/safe_decode (redundant with new API)

This follows standard Elixir conventions for fallible operations.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update base_url to jeffhuen/excoding for precompiled binaries
- Modernize GitHub Actions workflow (ubuntu-22.04, macos-14, actions v4)
- Add CI workflow for tests across Elixir 1.15-1.18 and OTP 26-27
- Add lint checks for Elixir and Rust formatting

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add NIF version features to Cargo.toml (required since Rustler 0.30.0)
- Update macOS x86_64 runner from macos-13 to macos-15

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove checksum-* from .gitignore (checksum file must be committed)
- Add checksums for all 20 precompiled binaries (10 targets × 2 NIF versions)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The checksums were generated before the final release binaries were uploaded.
This updates them to match the actual release artifacts.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
GitHub API digest field is NOT the sha256 of the tar.gz file.
Computed checksums by downloading each file and running shasum -a 256.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Generated from release binaries after CI build completed.
DO NOT move the v0.2.0 tag - that would trigger a rebuild.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Streaming Decoder (Excoding.Decoder):
- Stateful decoder for chunked/streaming data
- Properly handles multibyte characters split across chunk boundaries
- Essential for File.stream!/3 with Shift_JIS, GBK, Big5, etc.
- New functions: new/1, decode_chunk/3, stream/2, stream_with_errors/2

BOM Detection:
- detect_bom/1 - Detect encoding from Byte Order Mark
- detect_and_strip_bom/1 - Detect and strip BOM in one step
- Supports UTF-8, UTF-16LE, UTF-16BE

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Allows triggering NIF builds before tagging, so tags can include checksums.
- Manual builds create draft releases
- Verifies version matches mix.exs
- Keeps tag-triggered builds as fallback
@jeffhuen
Copy link
Author

Closing this PR—after further development, the changes became substantial enough (switching to the encoding_rs crate, new streaming API, different architecture) that it made more sense to publish as a separate package.

Available as encoding_rs on Hex for anyone interested. Thanks for the original work on excoding!

@jeffhuen jeffhuen closed this Jan 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Error when compiling against OTP-26 Rust decode<'a> and encode<'a> should not panic, instead they should return {:error, ...} tuple to elixir

1 participant