Red Hat enterprise-specific benchmarks for coding agents: Why we built RH SWE-bench

Coding agents promise to revolutionize software development, but how do we know if they’re any good at the kind of work enterprise teams do every day? SWE-bench, a benchmark for evaluating large language models (LLMs) on real-world software issues, established the paradigm: give the agent a bug report, let it fix the code, check whether the tests pass. 

Since then, variants like SWE-bench Verified, SWE-bench Pro, SWE-Smith, Multi-SWE-bench, and Terminal-Bench have expanded the scope by adding curated subsets, multilingual support, and new task types. The problem is that the majority of tasks across these benchmarks are still in Python, and none of them target the specific repositories and ecosystems that enterprise teams actually work on.

Why enterprise software needs more

Enterprise software looks different. Kubernetes controllers are written in Go. Ansible modules are in Python but with very specific testing patterns. Performance Co-Pilot (PCP) is C with custom build systems. If you want to know whether a coding agent can actually help with the software your organization maintains, existing benchmarks won’t tell you.

That’s the gap our team set out to close. We built RH SWE-bench, a benchmark of 357 quality-verified coding tasks drawn from 25 Red Hat ecosystem repositories across Go, Python, and C.

What we built

The journey to 357 tasks required a long funnel. We started with ~6,800 candidate instances generated through SWE-Smith’s various mutation strategies. After validation (checking that tests actually fail pre-fix and pass post-fix), only ~7% survived, yielding 531 verified instances. We then quality-filtered aggressively, removing 233 low-quality tasks (trivial single-function mutations, cheap solves, single-file patches) and adding 78 real PR-derived tasks from REPO2RL, a tool for turning a repository into a verifiable reinforcement learning environment. arriving at 357 verified tasks spanning:

• Go: 277 instances (78%) across 21 repositories including kubectl, etcd, podman, kustomize, buildah, and more
• Python: 73 instances (20%) across 4 repositories including Ansible, Ansible Molecule, Insights Core, and osbuild
• C: 7 instances (2%) from PCP

Each task is a self-contained coding challenge. The agent receives a problem description, gets dropped into a Docker container with the repository at a specific commit, and has to produce a fix that makes failing tests pass without breaking anything else.

The dataset is published on Github and  Hugging Face and is available on Harbor Hub for direct use with the Harbor evaluation framework.

Release versions

v1.0 is the original 357-task release across 25 repositories. v1.1 is the revised 337-task release across 24 repositories, with a smaller task set after review, updated instructions, and flaky or privileged-access P2P checks removed. No full benchmark has been rerun on the revised v1.1 prompts.

Two ways to generate bugs

We used a hybrid approach to build the dataset, combining synthetic mutations with real bugs extracted from pull requests.

Synthetic mutations (SWE-Smith)

SWE-Smith is a toolkit for generating synthetic SWE-bench instances. It takes working code and introduces bugs through various mutation strategies, including:

• LLM rewriting: An LLM rewrites a function to introduce a subtle bug while keeping the code plausible (66 instances in the final dataset).
• Combined mutations: Multiple single-function mutations merge into one task, creating more complex multi-site bugs (105 instances, the largest synthetic category).
• Procedural mutations: Automated transforms like inverting conditionals, swapping operators, and removing loop conditions (18 instances).

We generated thousands of synthetic instances across all our target repositories, but only ~7% survived validation. The strength is volume: you can generate many candidates per repository. The weakness is that many mutations don’t produce meaningful test failures, which is why aggressive quality filtering was essential.

Final dataset composition

Real PR pipeline: the breakthrough

The more interesting approach was extracting real bugs from merged pull requests using our real PR pipeline. The idea is simple: if a PR fixes a bug and adds tests for it, you can reverse-engineer a SWE-bench instance by separating the fix from the tests.

Getting this to work reliably took three iterations. First, we tried to reverse-apply patches: we took the fixed code, undid the fix, and checked that tests failed. This was fragile. Patches don’t reverse cleanly when there are merge conflicts, and test changes are tangled with code changes.

Here was the breakthrough. Instead of reversing patches, we:

1. Collected merged PRs that modify both source code and test files.
2. Built a Docker image at the PR’s base commit before the fix.
3. Applied only the test changes from the PR, so we have new tests but no fix.
4. Verified the tests. New tests failed because the bug exists and existing tests passed.
5. Applied the code changes as the solution, and now all tests passed.

This forward-apply approach achieved near-100% validation rates on real PR instances, compared to ~0% with the reverse-apply methods.

We also incorporated REPO2RL-generated tasks, which are real PR-derived instances from the REPO2RL tool, adding kubevirt, operator-lifecycle-manager, and other repositories to the dataset.

What a task looks like

Each task in the dataset follows the Harbor format:

task-0042/
  instruction.md             # Problem description for the agent
  environment/
    Dockerfile               # Reproducible build environment
    docker-compose.yaml      # Container configuration
  tests/
    test.sh                  # Validation script (F2P + P2P tests)
  solution/
    solve.sh                 # Oracle solution (base64 encoded)
    gold.patch               # Human-readable correct fix
  task.toml                  # Metadata (difficulty, timeouts, repo info)

The agent sees only the instruction and the repo state inside the container. It has to figure out the fix on its own. The validation script checks two things:

• Fail-to-pass (F2P): The specific tests that should start failing and become passing after the fix
• Pass-to-pass (P2P): Existing tests that should continue passing with no regressions

The validation journey

Building the dataset was one thing. Making sure every instance actually works was another, and it took eight rounds of validation to get right.

The unexpected hurdles

Some of the most interesting challenges were in the validation pipeline itself:

• grep substring matching: Our test script used grep to check Go test results, but Go test output includes subtests, and grep matched those too, giving false passes. Where Python’s Pytest modules handle it by unique test IDs, Go tests do not. We fixed this by adding a trailing space to the pattern.
• Multi-module Go repositories: etcd has separate Go modules for server/, client/, and api/. Running go test ./... from the root doesn’t work. Tests must be executed directly within each module’s subdirectory. Each repo needed its own test scoping strategy.
• Vendor directory challenges: kubectl’s mirror repo had go.mod replace directives pointing to vendored paths that didn’t exist in the container. We had to strip the replaces and pin real module versions.
• PCP zombie processes: PCP tests spawn performance metric domain agents (PMDAs) that become zombies if not properly reaped. We fixed it by running containers with init: true.

How we validate each task

Every task goes through a two-phase verification.

1. Pre-fix (F2P check): Apply only the test patch to the base commit. The F2P tests must fail, confirming the bug exists and the tests catch it.
2. Post-fix (P2F check): Apply both the test patch and the solution patch. The F2P tests must now pass, and all P2P tests must still pass, confirming the fix works without regressions.

A task is only valid if pre-fix tests fail and post-fix tests pass. Tasks where pre-fix tests already pass (P2P violation) or post-fix tests still fail are rejected, as they indicate the bug isn’t real or the fix doesn’t work.

What didn’t make the cut

We started with 531 candidate instances (before adding REPO2RL tasks) and removed 25 during validation:

• 15 buildah vendor mutations where tests didn’t actually exercise the mutated vendored code
• 2 storage mutations that affected only Windows-only or SELinux-specific files
• 2 kustomize PRs with network-dependent tests (flaky in containers)
• The remaining exclusions included etcd candidates with corrupted protobuf patches. We did not track that category separately.
• Others with empty patches or test violations

After adding 78 REPO2RL tasks and removing low-quality synthetic instances, the final dataset was 357 verified tasks.

Infrastructure

Running hundreds of coding tasks in isolated containers requires serious infrastructure. We used Red Hat OpenShift clusters for both development/validation and production benchmark runs.

• Docker-in-Docker isolation: Each task runs in its own Docker-in-Docker pod with an isolated Docker daemon, ensuring tasks don’t interfere with each other.
• Harbor orchestration: Harbor handles task scheduling, agent execution, and result collection across concurrent pods. We built a job queue service on top for managing benchmark runs at scale.
• Pre-built container images: Building repo environments from scratch takes 5 to 20 minutes per task. We pre-built and pushed images to GitHub Container Registry, cutting startup time to seconds.

Results

We’ve benchmarked several models and harnesses on RH SWE-bench:

A few observations:

• Go is harder than Python for coding agents. Go’s strict type system, interface patterns, and multi-module structure trip up agents that are trained primarily on Python.
• Harness matters. The same model (Nemotron-120B) scored 30.8% with OpenCode vs 21.6% with Pi, a 9 point gap just from the agent harness (the framework that directs the agent’s actions).
• Smaller models can punch above their weight. Qwen3.6-27B-FP8 at 49.3% outperforms the much larger Nemotron-120B and GPT-OSS-120B, showing that parameter count isn’t everything. Architecture and training data matter more.
• Enterprise repositories are more challenging. The top models solve 80%+ of SWE-bench Verified (general Python), but only 55 to 63% on RH SWE-bench. Enterprise code has more complex build systems, deeper dependency chains, and domain-specific testing patterns.

Findings from running benchmarks

Running RH SWE-bench across multiple models and harnesses revealed issues that wouldn’t surface in Python-only benchmarks.

• Base image diversity matters: Our PCP tasks use Alpine-based Docker images, which use apk instead of apt-get. Some harnesses assume Debian/Ubuntu and hardcode apt-get for agent setup, causing exit 127 failures on every Alpine task. Harnesses that detect the package manager (apk, apt-get, yum) work fine. This is the real-world problem: enterprise containers use diverse base images and benchmarks need to surface these compatibility gaps.
• Go test suites are heavier than expected: Kubevirt tasks run ~200 separate go test commands for verification, taking 20 to 25 minutes. Python repository verification typically finishes in under a minute. Initial timeout values calibrated for Python caused false errors on Go repositories. We had to profile actual run times and set timeouts per-task based on real data.

What we learned

Synthetic + real is better than either alone

Synthetic mutations give you volume and control over difficulty. Real PRs give you authentic bug patterns that synthetic generation misses. REPO2RL adds another dimension by extracting instances from PRs automatically. The combination produces a more representative benchmark.

Validation is the hardest part

Generating candidate instances is relatively easy. Verifying that each one has a clear bug, a correct fix, proper test coverage, and a reproducible environment is where most of the effort went: eight validation rounds over several weeks.

Go testing is a different world

Python’s pytest identifies tests with unique node IDs (module::class::test_name), so you can target specific tests, collect results by ID, and validate pass/fail with precision. SWE-bench’s original validation was built around this. Go has none of those advantages. Tests are identified by function name and package path, but the same test name can exist in multiple packages (e.g. TestFilter in 5 different packages). Go test output doesn’t include package paths in --- PASS: lines, so matching results to specific tests requires scoping each go test command to the right package. Multi-module repositories like etcd need separate Go test invocations per module. And Go test suites are significantly heavier. kubevirt’s test suite runs ~200 separate Go test commands and takes 25 minutes just for verification, compared to Python repositories where verification typically finishes in under a minute.

Infrastructure is underrated

A benchmark isn’t useful if people can’t run it. Docker-in-Docker isolation, pre-built images, and a standard task format (Harbor) make the difference between a research artifact and a usable tool.

Timeouts need real data

Initial timeout values (600s) caused false errors on kubevirt tasks that needed 25 minutes for verification. We calibrated all timeouts from actual Sonnet run data: verifier at 1800s minimum and agent timeouts at 1800 to 3600s depending on task complexity.

Ultimately, building RH SWE-bench demonstrated that evaluating coding agents for the enterprise requires enterprise-grade tools. By combining real-world pull requests with synthetic data across languages like Go and C, we’ve created a more realistic proving ground that reflects the challenges our developers, and our customers, face every day.

What’s Next

RH SWE-bench is a starting point. We’re working on:

• Leaderboard integration: Results are published to the coding agent leaderboard alongside SWE-bench Verified and SWE-bench Pro results.
• More languages and task types: Adding Java and Rust repositories, and exploring multi-file tasks that require cross-package changes.

Get RH SWE-bench from Harbor Hub, which we recommend for running benchmarks,  or download it from GitHub or Hugging Face. Then evaluate it with your preferred coding-agent model and harness.

Note: Red Hat’s Emerging Technologies blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless otherwise stated the technologies and how-tos shared here aren’t part of supported products, nor promised to be in the future.