-
Notifications
You must be signed in to change notification settings - Fork 0
Update to encoding_rs, rustler 0.37, add dirty schedulers #41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
+1,860
−307
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Major update that modernizes the codebase while maintaining backwards compatibility: - Switch from `encoding` crate to `encoding_rs` (Firefox's encoding library) - Update rustler 0.29.1 -> 0.37 (fixes OTP-26+ compilation, closes #40) - Update rustler_precompiled 0.5 -> 0.8 - Update Rust edition 2018 -> 2021 - Remove unused deps (lazy_static, rustler_codegen) - Rename .cargo/config to .cargo/config.toml (cargo deprecation warning) New features: - Dirty CPU schedulers for binaries >64KB (non-blocking for large data) - safe_encode/2, safe_decode/2 returning {:ok, _} | {:error, _} tuples - encoding_exists?/1 to check if encoding is supported - canonical_name/1 to get WHATWG canonical name for aliases - list_encodings/0 to list all supported encodings Bug fixes: - No more panics in Rust code - proper error handling throughout (closes #24) - All .unwrap() calls replaced with proper error propagation Backwards compatible: - encode/2 and decode/2 maintain same behavior as v0.1.x Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- encode/2 and decode/2 now return {:ok, result} | {:error, reason}
- encode!/2 and decode!/2 return raw value or raise
- Removed safe_encode/safe_decode (redundant with new API)
This follows standard Elixir conventions for fallible operations.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update base_url to jeffhuen/excoding for precompiled binaries - Modernize GitHub Actions workflow (ubuntu-22.04, macos-14, actions v4) - Add CI workflow for tests across Elixir 1.15-1.18 and OTP 26-27 - Add lint checks for Elixir and Rust formatting Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add NIF version features to Cargo.toml (required since Rustler 0.30.0) - Update macOS x86_64 runner from macos-13 to macos-15 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove checksum-* from .gitignore (checksum file must be committed) - Add checksums for all 20 precompiled binaries (10 targets × 2 NIF versions) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The checksums were generated before the final release binaries were uploaded. This updates them to match the actual release artifacts. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
GitHub API digest field is NOT the sha256 of the tar.gz file. Computed checksums by downloading each file and running shasum -a 256. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Generated from release binaries after CI build completed. DO NOT move the v0.2.0 tag - that would trigger a rebuild. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Streaming Decoder (Excoding.Decoder): - Stateful decoder for chunked/streaming data - Properly handles multibyte characters split across chunk boundaries - Essential for File.stream!/3 with Shift_JIS, GBK, Big5, etc. - New functions: new/1, decode_chunk/3, stream/2, stream_with_errors/2 BOM Detection: - detect_bom/1 - Detect encoding from Byte Order Mark - detect_and_strip_bom/1 - Detect and strip BOM in one step - Supports UTF-8, UTF-16LE, UTF-16BE Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Allows triggering NIF builds before tagging, so tags can include checksums. - Manual builds create draft releases - Verifies version matches mix.exs - Keeps tag-triggered builds as fallback
Author
|
Closing this PR—after further development, the changes became substantial enough (switching to the encoding_rs crate, new streaming API, different architecture) that it made more sense to publish as a separate package. Available as encoding_rs on Hex for anyone interested. Thanks for the original work on excoding! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR modernizes excoding while maintaining full backwards compatibility:
encoding_rs- The same encoding library used by Firefox. Better performance, actively maintained, and WHATWG compliant.unwrap()calls replaced with proper error propagation (no more panics)Issues Fixed
New Functions
Backwards Compatibility
encode/2anddecode/2work exactly as before (return raw value, raise on error)Test Plan
mix formatpassesmix compile --warnings-as-errorspassescargo clippypassesHappy to adjust anything based on feedback!
🤖 Generated with Claude Code