WIP: Adjust Resampling Logic and Defaults

[malteschaaf]

This PR introduces three key changes to the resampling logic and configuration. These changes aim to improve the clarity and behavior of the resampling process.
Note: Tests have not been updated yet, and this PR is intended for discussion.

Changes:

1. Use bisect_left for min_index calculation in _ResamplingHelper

   • Replaced bisect with bisect_left to ensure the correct lower bound index is calculated for relevant samples.
   • This ensures that the minimum relevant timestamp is included in the range of relevant samples, improving accuracy.
   • Example:
     Suppose we have a sorted list of timestamps: timestamps = [10, 20, 30, 40, 50]
     If we search for the position of 30:
     Using bisect: It would return the position after 30 (index 3), which excludes 30 itself.
     Using bisect_left: It would return the position at 30 (index 2), ensuring 30 is included in the range.
     This change ensures that the lower bound of the relevant range is correctly calculated and includes the minimum timestamp.

2. Change default value of max_data_age_in_periods in ResamplerConfig

   • Updated the default value of max_data_age_in_periods from 3.0 to 1.0.
   • This change is proposed because, when debugging the MicrogridDataSourceNode, it was observed that values were being resampled twice:
     The node receives already resampled samples (to 1s). However, these 1s samples were based on 3-second windows. For example, the sample with timestamp 12:00:01 resampled the values from 11:59:58 to 12:00:01. This behavior likely stems from not explicitly setting the max_data_age_in_periods value in the configuration, leaving it at the default of 3.0.
     When resampling these 1s samples again inside the node to a desired period (e.g., 10s), the resampling process would also include original samples from 11:59:58 to 12:00:00 in the resampled sample representing 12:00:00 to 12:00:10. This leads to unintended overlaps and inaccuracies.
   • The new default ensures that only the most recent period is resampled unless explicitly configured.

3. Adjust timestamp handling in resampling to represent the start of the period

   • Changed the sample_time in _ResamplingHelper's resampling function to timestamp - conf.resampling_period.
   • This ensures that the timestamp of each resampled data point represents the beginning of the resampling period instead of the end.

[cwasicki]

I agree that this value is leading to unexpected behavior, but this would be a very intrusive change which would result in many more NaN values in deployments. If we want to change it we could make it a required parameter for migration purpose, and later introduce the new default.

[malteschaaf]

Sounds good.

[cwasicki]

I think the behavior how to resample, i.e. left or right open and the labeling should be config parameters.

[malteschaaf]

I think we should make left or right opened configurable with the corresponding label, such as:
right open: [t,t+1) -> labeled as t
left open: (t-1,t] -> labeled as t (the current behavior)
Or do you want to allow the user to additionally do something like:
right open, label in the end: [t,t+1) -> labeled as t+1
left open, label in the beginning:(t-1,t] -> labeled as t-1
Because I think the later could lead to a lot of confusion (not sure if its ever needed)

[cwasicki]

I think the latter is also reasonable options (see e.g. https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.resample.html), but don't see a strong reason to implement this now if not needed. If it's well-documented, the users can also adjust the timestamps trivially. So your proposal sounds good to me.