[Community Submission] Model: Isaac-AI-OS/isaac-20b, Username: petermantell

[pedroMantAmp]

ISAAC 20B — Deterministic (Agentic-Lite) Evaluation

Org/Model: https://huggingface.co/Isaac-AI-OS/isaac-20b
Username: petermantell

Benchmarks (subset, N=5 each)

• HumanEval pass@1 = 0.60
• MBPP pass@1 = 0.80

Eval mode: temperature=0, top_p=0, top_k=1, n=1, seed=7
Deterministic tools only, no_concurrency, max_steps=6
Code-only normalization (fences/prose stripped)
Docker digest: isaac-hf@sha256:6fc9f0d85dfe56daba8fc92496718226f056014b3e84ee7a823df1d9271a57c0

Artifacts:

• Code summary: https://huggingface.co/datasets/Isaac-AI-OS/isaac-20b-eval-artifacts/resolve/main/eval/artifacts/code/summary.json
• HumanEval: https://huggingface.co/datasets/Isaac-AI-OS/isaac-20b-eval-artifacts/resolve/main/eval/artifacts/code/humaneval_results.json
• MBPP: https://huggingface.co/datasets/Isaac-AI-OS/isaac-20b-eval-artifacts/resolve/main/eval/artifacts/code/mbpp_results.json
• Manifest: https://huggingface.co/datasets/Isaac-AI-OS/isaac-20b-eval-artifacts/resolve/main/eval/artifacts/manifest.json

Folder added: community_results/Isaac-AI-OS_isaac-20b_petermantell

• grouped_scores.json
• generations/* (humaneval)
• metrics/* (mbpp)

[pedroMantAmp]

cc: @bigcode-project maintainers — community submission per README.
HumanEval N=5 pass@1=0.60, MBPP N=5 pass@1=0.80 (deterministic Agentic-Lite).
Artifacts & manifest linked above; folder includes grouped_scores + generations + metrics.
Thanks!