Software Developer Intern • May 2026 - Aug 2026 • San Jose, California
Worked inside an internal LLM evaluation platform that stress-tests IBM's AI coding assistants on mainframe modernization tasks: code explanation, COBOL to Java transformation, and generation. Shipped an automated grading layer, collapsed two assistant surfaces into one, and took the evaluation service from crashing at 4 concurrent users to 512 users with zero failures.
A Streamlit wizard on the front, an evaluation service on the back, talking over HTTP. The wizard owns pipeline construction; the service owns execution.
The HTTP boundary is what let the wizard and the service evolve independently, and what made the service's load behaviour measurable in isolation later on.
Judging whether an AI coding assistant is actually getting better at mainframe work is only useful if the benchmark is repeatable. It was not, yet.
Constructing a benchmark meant wiring together the task, the dataset, the models and the scoring config yourself, so two people benchmarking the same thing did not necessarily benchmark the same thing.
Explaining COBOL, transforming COBOL into Java, and generating fresh code are different problems with different failure modes. A single flat benchmark hid that.
Even once a run finished, the output still had to be read by a person, which capped how many samples anyone was willing to evaluate.
Extend the platform with a pipeline builder, so a benchmark becomes a thing you configure rather than a thing you assemble.
A stepped interface that walks through every decision a benchmark needs, in order, with the invalid combinations closed off.
The same selections produce the same pipeline, so a result can be re-run and compared rather than taken on trust.
The builder hands off over HTTP, keeping heavy evaluation work off the UI process and out of the user's session.
Three families of mainframe modernization task, each benchmarked on its own terms.
Can the assistant describe what a piece of legacy code actually does, accurately enough to be trusted by someone who did not write it?
The core modernization path. Does the translated Java preserve the behaviour of the COBOL it came from?
Writing new code against a mainframe context, where the surrounding conventions matter as much as the syntax.
A strong LLM grades the code that the candidate assistants produced. The grader is deliberately not one of the models under test.
Candidate output
Code from the assistants under test
Rubric applied
Explicit criteria, not vibes
Judge model scores
Grade, comparison, risk flag
Run-level result
Comparable across runs
Because the rubric and the judge are fixed for a run, two runs are comparable to each other. That is the property manual review could never offer.
Manual review of AI-generated code is slow, subjective, and does not scale. All three are disqualifying for a benchmark.
Three grading modes, because "is this code good" and "is this code better than that code" are different questions.
The judge grades a single output against explicit written criteria, producing an absolute score that means the same thing across runs.
Two competing models answer the same prompt and the judge picks a winner. This catches quality differences that absolute scoring rounds away.
Separate from "how good is it", a judgement on how dangerous the output would be if a developer trusted it. On mainframe code paths, that distinction matters.
1000s
Samples gradeable per run
3
Independent grading modes
Reproducible
Same inputs, same verdict
The shift that mattered was not speed on its own. It was that evaluation stopped being an opinion and started being a measurement, which is what makes model regressions visible before they ship.
Two assistant surfaces each had their own page in the benchmarking UI, and each page had grown to roughly ten thousand lines carrying its own private copy of the same helper and render logic.
The same date formatters, run selectors and render helpers were defined independently in both files. Every change had to be made twice, and staying in sync was a matter of discipline rather than structure.
One implementation, two callers. A correctness fix lands once and applies to both surfaces.
Duplication is cheap to create and expensive to keep. At ten thousand lines per page, the cost had already arrived.
The two copies were close enough to look interchangeable and different enough that you could not assume they were. That combination is the worst case for anyone reading the code, because the diff between them carried no signal about which differences were deliberate.
With no single source of truth, the two surfaces could render the same benchmark differently and nothing in the system would flag it.
Every fix, every new task type and every scoring change was two edits and two reviews, with a standing chance of only landing one of them.
Done as an incremental arc rather than one heroic commit, so that each step was reviewable and behaviour could be checked as it moved.
Stand up the shared module and move one page's helpers into it, proving the seam before committing to it.
Migrate the other surface onto the same module, which is where the near-duplicates had to be reconciled against each other.
The large step: roughly 3,200 lines deleted across the two pages against about 1,836 added to the shared module, which is the deduplication showing up as a net negative diff.
Where the surfaces genuinely differ, the difference became a parameter at the call site rather than a fork in a copied file.
The goal was structural, not functional. Both surfaces keep behaving the way their users expect.
~3,200
Duplicated lines removed
102
Helpers now shared
2 → 1
Sources of truth
Net −
Deletions exceeded insertions
The measurable win is the deleted code. The durable win is that the two surfaces can no longer quietly disagree about what a benchmark means, because there is now exactly one place where that is defined.
The evaluation service died under load. "It crashes sometimes" is not a bug report, so the first job was building something that made it crash on demand, the same way, every time.
Starts the service itself
Launches the API as a subprocess and waits for the readiness endpoint, so a run needs zero manual setup.
Seeds once, reuses after
Only seeds if the table is below a row floor, so the old and new code are measured against byte-identical data.
Ramps concurrency
Escalates users from 2 up to 512, crossed with rising requests per user, rather than firing one fixed load.
Samples RSS every 0.25s
Tracks resident memory across the whole process tree, parent plus children, and keeps the peak alongside p50 and p95 latency.
After every step it evaluates break conditions and stops on the first one that trips, in priority order:
Every run is tagged with the code branch it measured and emits its step log, report and chart side by side, so old versus new is a diff of artifacts rather than a recollection. Requests are sent with an identical URL on both sides, leaving the query code as the only variable.
The memory curve pointed at one thing: list endpoints that fetched entire tables and then deserialized every row.
Four list endpoints accepted no pagination parameters at
all. Each one issued a query with no LIMIT, materialized the full result
set into a Python list, then ran every row through model validation. The rows themselves
held large JSON text columns, so a single request could deserialize one to three unbounded
blobs per row for the entire table.
Request arrives
no limit given
Whole table
into memory
Every row
JSON parsed
Under concurrency
OOM
Memory was a function of how much data existed, multiplied by how many requests were in flight, and entirely independent of how much the caller actually wanted.
The readiness endpoint returned a hardcoded healthy response without ever touching the database, so a dying worker stayed in rotation while returning 100% errors.
That combination is why the failure looked load-dependent rather than request-dependent. No individual request was obviously wrong. The ceiling was simply missing, and nothing upstream noticed.
Three layers, shipped in that order: bound the pages, then stream them, then stop lying about health.
Added optional limit and offset parameters to all four endpoints, clamped to a maximum page size at the API layer so a caller cannot ask for an unbounded page. Made opt-in first, which kept existing callers working while the migration happened.
Offset pagination still makes the database read and discard every skipped row, so deep pages stay expensive. Replaced it with seek pagination on the monotonic primary key.
Seek, do not scan
Filter on id greater than the last id seen, so the database jumps straight to the page through the index.
Stream one batch
A server-side cursor yields rows per batch, so the full result set is never materialized at once.
Fetch one extra row
Requesting page size plus one reveals whether a next page exists without a separate count query.
Rewrote the readiness endpoint to actually open a connection and run a trivial query, returning 503 when it fails. Now an orchestrator pulls a dying instance out of rotation instead of continuing to route traffic at it.
Same machine, same seeded data, same request URL, code as the only variable. Both sides ran the full ramp to 512 users.
Roughly a 5.5x reduction on the whole service. Measured against an idle baseline of about 95 MB, growth over idle fell from about 62x to about 12x.
~5.5x
Lower peak memory
~61x
On the heaviest endpoint
100% → 0%
Peak error rate
4 → 512
Concurrent users survived
Isolating a single endpoint made the mechanism starkest: with a full scan its peak reached 13.4 GB at 100% errors, and with cursor pagination the same endpoint under the same load held flat at 220 MB at zero errors. That is the ~61x figure, and it is the difference between memory scaling with your data and memory scaling with your page size.
The concurrency-ramping, RSS-sampling harness built to find the leak outlived the investigation and became a repeatable way to check the service under load.
Consolidating the duplicated benchmarking logic gave the platform one definition of what a benchmark is, which every later change builds on.
Moving grading from manual review to rubric-driven automation changed model comparison from a discussion into a number that can be tracked over time.