Preprint: Small Open LLMs Fine-Tuned for Eclipse Bug Severity Match CodeBERT but Lag Zero-Shot
An arXiv preprint released October 25, 2026 tests Llama 3 8B and Qwen 2.5 7B zero-shot and after QLoRA fine-tuning against TF-IDF and CodeBERT baselines on 20,000…
- This abstract-only preprint finds fine-tuned Llama 3 8B shows no statistically detectable macro-F1 difference from CodeBERT on four-level bug severity prediction.
- Zero-shot Llama 3 and Qwen 2.5 reach macro-F1 of 0.305 and 0.321, below every TF-IDF baseline; fine-tuning narrows the gap but the setups also differ in checkpoint, runtime and decoding.
- Per-class recall varies sharply and 8% test-train overlap inflates some baseline scores; authors release splits, predictions and a local serving tool called CyberTriage.
What the preprint measured
According to its abstract, researchers used a public dump of 361,006 Eclipse Bugzilla reports, created a class-balanced sample of 20,000 rows, and evaluated eleven configurations on the same 2,000 held-out reports.
The models compared were six TF-IDF classifiers, a fine-tuned CodeBERT encoder, and Llama 3 8B plus Qwen 2.5 7B in both zero-shot and QLoRA fine-tuned forms. They report macro-F1 scores with bootstrap intervals and McNemar tests.
The authors note the original dump contains about 60,000 duplicate rows and that 8% of test reports duplicate a training report. They have released split identifiers, per-report predictions, analysis scripts and CyberTriage, a tool that serves the fine-tuned classifier locally.
Performance results
Zero-shot runs of the open models reached macro-F1 0.305 for Llama 3 and 0.321 for Qwen 2.5, below all TF-IDF models which scored 0.430–0.489.
After fine-tuning, Llama 3 reached 0.526 and Qwen 2.5 reached 0.485. Fine-tuned Llama 3 showed no detectable difference from CodeBERT at 0.518 (difference +0.007, 95% CI -0.017 to +0.032). Fine-tuned Qwen 2.5 scored below CodeBERT on aggregate.
Class-level differences were larger: fine-tuned Llama 3 recalled 36% of Critical reports while Qwen 2.5 recalled 69% of them but only 21% of Minor ones. Removing duplicate overlap lowered TF-IDF scores by up to 0.031 but left the LLMs almost unchanged.
Why it matters for uncensored AI
Our analysis: this preprint starts from the fact that small open models can be fine-tuned and run locally on a laptop for a practical task, as shown by the released CyberTriage tool. That supports the value of model ownership and private deployment for teams who want full control over their data and inference.
The same practical checklist of verifying data similarity, checking exact checkpoints and testing on held-out data applies when fine-tuning open models for any private use case. When you also want hosted multimodal companions that combine uncensored chat, photo exchange, voice interactions and phone calls, we recommend considering Muah AI.
