feat(zero_shot): enhance evaluation pipeline with rerun-judge and cha…

[XiaoBoAI]

…rt improvements

• Add --rerun-judge flag to re-run pairwise evaluation with different judge model
• Enhance chart_generator with additional visualization options
• Update schema with new configuration options
• Improve zero_shot_pipeline with better checkpoint management
• Fix minor issue in openai_chat_model

OpenJudge Version

[The version of OpenJudge you are working on, e.g. import openjudge; print(openjudge.__version__)]

Description

[Please describe the background, purpose, changes made, and how to test this PR]

Checklist

Please check the following items before code is ready to be reviewed.

• Code has been formatted with pre-commit run --all-files command
• All tests are passing
• Docstrings are in Google style
• Related documentation has been updated (e.g. links, examples, etc.)
• Code is ready for review

[gemini-code-assist]

Summary of Changes

Hello @XiaoBoAI, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly upgrades the zero-shot evaluation pipeline by introducing more granular control over the evaluation process and expanding its visualization capabilities. The addition of a dedicated flag for re-running judge evaluations and improvements to incremental data collection streamline the workflow for model comparison. Furthermore, new chart types offer richer insights into model performance, making the evaluation process more efficient and informative.

Highlights

• Enhanced Evaluation Control: Added a new --rerun-judge flag to the CLI, allowing users to re-run only the pairwise evaluation step with a different judge model without re-generating queries or collecting responses. This significantly speeds up judge model experimentation.
• Advanced Charting Options: The chart generator now supports vertical bar charts and introduces a new win rate matrix heatmap, providing more comprehensive and flexible visualization of evaluation results.
• Improved Checkpoint Management: The evaluation pipeline's checkpointing mechanism has been enhanced to support incremental response collection and pairwise evaluation. It can now detect changes in the judge model and automatically trigger a re-evaluation of only the affected steps, saving time and resources.
• Schema Updates: New configuration options for chart orientation and win rate matrix generation have been added to the schema, along with an increased default DPI for higher quality image outputs.
• Gemini Model Compatibility Fix: A minor fix in the OpenAI chat model integration ensures that Gemini models correctly handle JSON object response formats.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request introduces significant enhancements to the zero-shot evaluation pipeline, primarily focusing on improved checkpoint management, incremental evaluation capabilities, and advanced visualization options. The new --rerun-judge flag allows for efficient re-evaluation with different judge models without re-running the entire pipeline. The chart_generator now supports both vertical bar charts and win rate matrix heatmaps, providing more comprehensive visualization of results. The zero_shot_pipeline has been refactored to support incremental collection of responses and pairwise evaluation, making it more robust and efficient for large-scale evaluations. Additionally, a minor fix for Gemini model compatibility in openai_chat_model is included. Overall, these changes greatly improve the usability, efficiency, and analytical depth of the evaluation framework.

[gemini-code-assist]

The configuration is loaded twice: once here to get the output_dir and again inside _run_evaluation. To improve efficiency and avoid redundant operations, consider passing effective_output_dir directly to _run_evaluation and modifying _run_evaluation to accept it, thus removing the need for _run_evaluation to load the config itself.