GForce fails to do overflow checks for integer64

[MichaelChirico]

Related to #7571.

DT = data.table(i = c(1L, 1L), x = lim.integer64()[2L])

DT[, sum(x), by=i]
#        i    V1
#    <int> <i64>
# 1:     1    -2
DT[, base::sum(x), by=i]
#        i    V1
#    <int> <i64>
# 1:     1  <NA>
# Warning message:
# In sum.integer64(c(NaN, NaN), na.rm = FALSE) :
#   NAs produced by integer64 overflow

[ashish-um]

I have reproduced this on the latest dev version.

It seems the overflow happens in src/gsumm.c because the int64 path lacks the "fallback to numeric" logic that the standard integer path has.

I traced the execution to the gsum64 logic. Specifically, in the case REALSXP block (handling int64), the addition happens here without an overflow check:

// src/gsumm.c (~line 620 depending on version)
// inside the else { // int64 block
_ans[my_low[i]] += my_gx[i];

Unlike the 32-bit integer block above it (which checks if (overflow) and restarts using REALSXP), this loop blindly adds, leading to the wraparound behavior.