Prism.lex_compat creates wrong on_sp token when used with heredoc and unclosed embexpr

[tompng]

Prism.lex_compat("<<A+B\n\#{C").value
# => [[[1, 0], :on_heredoc_beg, "<<A", BEG], [[1, 3], :on_sp, "+B\n\#{C", BEG]]

Found in IRB's truffleruby ci failure. Actual ruby code:
https://github.com/ruby/irb/blob/342f57b2d0a39c519a53e6d5a3938f6235119c9c/test/irb/test_ruby_lex.rb#L17-L21

①+<<A-②
#{③*<<B/④
#{⑤&<<C|⑥

This failing test is not important in IRB anymore and I think it will be removed soon (in ruby/irb#1160).

[Earlopain]

Generally I would say that syntax invalid code not matching up is somewhat expected but the prism output doesn't look all that correct:

Prism.lex("<<A+B\n\#{C").value
=> 
[[HEREDOC_START(1,0)-(1,3)("<<A"), 1],
 [EMBEXPR_BEGIN(2,0)-(2,2)("\#{"), 1],
 [CONSTANT(2,2)-(2,3)("C"), 32],
 [EOF(2,3)-(2,3)(""), 32],
 [EOF(2,3)-(2,3)(""), 32]]

There's clearly some stuff missing and EOF appears twice. Seems like it isn't jumping back to the right place after looking for the heredoc end.