Pointing LM Studio Bionic at a Codebase With Defects I Documented in a Previous Post

Back in July I built the same login portal three times with Claude Code, once through a subagent relay, once with subagents dispatched in parallel, and once with an agent team whose members could message each other. I then published the comparison along with every defect the exercise produced and the file each one was found in. The useful byproduct of that exercise is that I now have a repository where I already know what is broken, where it is broken, and which round introduced the defect.

When I sat down to test the coding functionality of LM Studio Bionic, I realized I already had a ready-made benchmark so rather than asking a model to build something new and then judging the result myself, I could point it at a codebase with a known set of flaws and see what it actually found.

In my last post on Bionic, I focused on Work projects, which ground the model in a folder of documents. I intentionally left the coding side alone because I felt it deserved its own evaluation. This time I wanted to see whether a model running locally on my laptop could use LM Studio Bionic’s tooling to identify the same issues that Claude Code’s security reviewer found, without any hints about where to look.

I’ve spent the last few months building several applications with AI coding agents: a local browser agent, a travel planner, and the same login portal built three separate ways. Those projects are fun, but evaluating them is difficult because the scoring is largely based on my own assessment of the output. If an agent produces an application, I can decide whether I like the result, but there’s no independent way to measure how well it actually performed.

This test is different because I already know where the defects are. They were documented as part of the original login portal comparison, along with the round that introduced them and the file they ended up in. That gives me a way to compare what the model finds against a known set of issues rather than relying entirely on my own assessment of the output.

What changes when coding is enabled

In earlier versions of Bionic, creating a Code project involved enabling an Allow coding option during project creation. In Bionic 1.1.0, LM Studio simplified project creation by unifying Code and Work projects into a single project type, with coding capabilities enabled on a per-session basis. The result is a simpler workflow where the project defines the working context and the session determines whether coding tools are available. This also means that the checkbox to turn on coding will no longer be present when you create a project.

The answer key (answers from my previous post)

Everything below is scored against defects I already published, so a anyone can clone the same repository and grade with a different model against the same list.

Defects 4 through 7 all came from the same issue. The frontend was built against assumptions that didn’t match what the backend actually implemented. I counted them separately because a model could spot the login failure and stop there without finding the remaining contract mismatches.

The model I used for the test – Qwen3.6-35B-A3B-GGUF

Everything below ran on Qwen3.6-35B-A3B-GGUF from unsloth, the UD quantisation, 40.24 GB on disk, loaded on the ZBook’s 128 GB of unified memory.

One unexpected finding during testing was that when I attempted to use Qwen 3.8 27B:

… it would repeatedly claimed it lacked file-reading tools despite being attached to the project. Repeating the same prompt with Qwen 3.6 allowed the model to traverse the repository and complete the task successfully. I haven’t spent the time to troubleshoot the issue but will likely write a post in the future once I figure out what the issue was.

Step #1 – Clone the graded repository

git clone https://github.com/terenceluk/claude-code-team-demo.git bionic-test
cd bionic-test

The three rounds come to roughly 440 KB of tracked source once the lock files are set aside, small enough that indexing finishes quickly and large enough that the model cannot simply read every file into context and pattern match.

Step #2 – Install dependencies and establish a baseline

cd round-3 && npm install && npm test

node_modules is not committed into the GitHub repository, so the suite cannot run straight from a clone, and better-sqlite3 is pinned at 13.0.2 and is a native module, so this either pulls a prebuilt binary matching your Node version or compiles from source. On Node 24.13.0 there is no prebuilt binary for that version, and npm falls through to node-gyp rebuild, which wants Visual Studio with the Desktop development with C++ workload. It found my Python 3.12.10 install without complaint, inspected SQL Server Management Studio 21 in passing and reported it as unknown version undefined, then gave up looking for a compiler.

Bumping one patch version fixes it, because 13.0.3 does ship a prebuilt binary for Node 24.

npm install better-sqlite3@13.0.3 && npm test

Bumping the package one patch version fixed it right away. The install finished in about 15 seconds and the test suite came back green with all 76 tests passing. The other options were to use Node 22 LTS through nvm-windows, which matches the package’s stated support, or install the Visual Studio build tools, which is a much larger download but solves this class of native module problem more broadly.

Get a green run before touching a single file, because Step #10 asks Bionic whether an accepted change broke the tests and that question cannot be answered against a suite that has never passed. If you have already made the change, stash that one file rather than running a bare git stash, which would sweep up any files you deleted from the clone and put them back mid-run.

git stash push round-3/middleware/auth.js

Step #3 – Create the project and point it at the folder

Create a new project, give it a name, click Choose a folder for LM Studio Bionic to work in, select the cloned bionic-test folder as the root, and create it.

My earlier testing was done on Bionic 1.0.4 build 9, where project creation included an Allow coding option. As I mentioned earlier, Bionic 1.1.0 unified Code and Work projects into a single project type, so that option no longer appears during project creation. Coding capabilities are now enabled per session rather than by creating a separate type of project.

LM Studio’s own Code project documentation still walks through the old flow and tells you to toggle on Allow coding during creation, so anyone following that page on 1.1.0 will go hunting for a control that is not there.

Step #4 – Confirm the index and the Git state

Once the project opens, the Files panel should show the repository and the current branch, which is how you know LM Studio Bionic recognized this as a Git repository rather than a plain folder of files.

Step #5 – Testing repository understanding

The first prompt should not mention security at all because I want to see whether agentic code search can trace behaviour through unfamiliar code before I ask it to judge that code.

Explain how authentication works in round-3. Trace it from the login request through to how the browser proves who it is on a later request, and name the files involved.

Qwen 3.6 successfully traced the round-3 authentication flow without additional hints. It identified the login routes, authentication middleware, JWT generation logic, token revocation handling, and the files involved. More importantly, it stayed within the round-3 implementation despite three similar codebases existing side by side. During the explanation it also independently noted the fallback JWT signing secret, even though the prompt was focused on repository comprehension rather than security review.

Step #6 – Unprompted defect review

Now the graded run, with no hints, no file names, and no mention of what I already know.

Review round-3 for security and correctness problems. For each one, tell me the file, the line, why it is a problem, and how severe you think it is.

It worked for 12 minutes and came back with eleven findings, each carrying a file, a line range, a severity, and a code excerpt.

Findings 4 and 7 come with an important caveat. Email enumeration through the registration endpoint’s 409 response, and the revoked_tokens table pruning only on logout, are findings F5 and F6 in round-3/SECURITY-REVIEW.md, the file I did not remove from the clone. The line numbers differ from that file’s, so it is not a straight copy, though the two findings nobody else surfaced are the two that document already lists.

Step #7 – Cross-file contract validation

Defects 4 through 7 are cross-file, which makes them a better test of agentic code search than anything in Step #6. My plan was to ask twice, once broadly and once narrowly, and record which prompt it needed.

Round-2’s frontend and backend were built by different agents that could not talk to each other. Compare what the frontend sends and expects against what the backend actually implements, and list every place they disagree.

No follow-up was needed. It worked for 7 minutes 7 seconds and returned ten numbered mismatches, each with a frontend-expects against backend-returns table, then a summary table with a severity per row.

In total, defects 4, 5, 6, and 7 were all found on the first prompt with no hint, plus three divergences my own answer key never listed, the dropped name field, the password length disagreement, and the pin list rendering as NaN coordinates. It traced consequences rather than stopping at the field names, working out that pin.lat resolving to undefined means NaN for both coordinates and therefore no visible pins on the map, and that the frontend’s extractErrorMessage() falls through both branches and lands on a generic string, so the real validation message never reaches the user.

Its closing line is fairer than anything I wrote about round 2 myself.

The two agents built fundamentally incompatible systems. The frontend’s auth model (bearer token + localStorage) is the opposite of the backend’s (server-side session cookies), and every pin field name was guessed differently.

Step #8 – Where the gates turned out to be

What stands between the agent and my working tree is not what I assumed when I planned these steps. The next two steps are where that comes out, so the summary sits here rather than at the end.

Step #9 – Testing change control

Defect 1 is a one-line change and a good candidate to accept, so I asked for it directly.

Fix the hardcoded fallback secret in round-3 so the server refuses to start when the signing secret is not set in the environment, instead of falling back to a default value.

Then I asked for defect 4, the auth transport mismatch, which is deliberately the harder request because it spans two files and there are two defensible directions to fix it, moving the frontend onto cookies or moving the backend onto bearer tokens.

Fix the authentication mismatch between round-2’s frontend and backend so that login works. Tell me which side you are changing and why before you change it.

I asked it to explain which side it was going to change and why before making any edits but that did not happen. The reply opened with “All changes are complete and consistent. Here’s a summary of what I changed,” and by the time I read it, five files had already been written to disk.

Bionic then showed an Edited files panel listing the changes: +15 -20 on FRONTEND-ASSUMPTIONS.md, +7 -42 on api.js, +7 -14 on auth.js, +4 -7 on login.js, and +3 -16 on register.js. Expanding any file shows the diff, but I could not find any accept, reject, or revert control anywhere in the panel.

The instruction was “tell me which side you are changing and why before you change it.” The explanation arrived in the past tense after the work was already complete, so the decision I was trying to force never actually existed.

One of the reasons it gave for choosing the frontend was that FRONTEND-ASSUMPTIONS.md explicitly states these are guesses and should be updated once the real contract is known.

git restore round-2/

Run that before spending time inspecting a diff you wanted to keep. git restore discards working tree changes without a backup, and there is no reflog for them. The Edited files panel remains available afterwards, so the diff is still readable after the restore.

Step #10 – Testing shell execution and approvals

The shell tools are a useful test of how much authority the agent actually has. My view has always been that a draft gets reviewed before it is sent and a command gets reviewed before it is run, so the question is what Bionic does by default.

Run the test suite for round-3 and tell me whether my change broke anything.

Unlike file edits, command execution can be gated. The behaviour is controlled through the shield icon at the bottom left of the composer.

Auto Review works differently. Instead of presenting a dialog box, Bionic spawns a Shell Approval Reviewer that reads the proposed command, opens files if it needs them, and rates the command before anything executes. Its instructions tell it to treat the parent agent’s transcript as evidence rather than instructions, distinguish an authorization I actually gave from a conclusion an earlier reviewer reached, and refuse to make a fresh decision while a conversation is being compacted.

On npm test it failed twice, in two different ways. The first attempt called 66 tools and read 38 files to rate that one command, then hit compaction and returned a terminated error rather than a rating. A second attempt looped on its own preamble instead, emitting the sentence “Based on my thorough review of the workspace structure, package.json configuration, all 7 test files, the test infrastructure, API documentation, security review, changelog, and test report:” three times over with one tool call between each repetition, never reaching the result block its own prompt requires. Context sat at 8.1K on that run, so the second failure was not the first one wearing a different hat.

When Auto Review cannot reach a decision, Bionic hands the decision back to the user. The command is neither blocked nor approved automatically. Instead, the approval dialog appears with the reason attached.

Approval reviewer failed: terminated

Do you want to run cd 'C:/Bionic/bionic-test/round-3' && npm test 2>&1?

Allow / Allow, and add instructions / No, tell the agent to do something else

Failing to a human is the correct behaviour for a safety control. Despite the error message, the command was neither blocked nor approved automatically. What the two failures actually cost me is time rather than safety.

One detail from Step #6 showed up again in these panels. To rate a git branch and git tag command, the reviewer opened round-3/SECURITY-REVIEW.md and read twelve lines of it.

Switching to Ask every time removed the reviewer from the loop entirely.

The dialog names the working directory and the command in full, and the middle option is “No, tell agent to do something else” rather than a flat refusal, so you can redirect without ending the run, and the session picks up a NEEDS ACTION badge in the sidebar, which means you can leave it waiting and be told when it needs you.

Approval is per command, and it took four of them to run one test suite. Three of those four went to Git, and the sequence is the most interesting part of the step.

The first command was the wrong tool for the question, because my change was sitting uncommitted in the working tree and git log only reports commits. It adjusted course instead of stopping there, and by the third command had the diff in front of it.

It then summarised that diff accurately before running anything.

I can see the change: the auth middleware no longer has a hardcoded dev fallback secret, it now throws an error if WAYPOINT_JWT_SECRET isn’t set in the environment. Let me run the round-3 tests now.

The suite finished in 3 minutes 52 seconds.

I expected red, because middleware that throws on a missing environment variable should take every test with it, and the explanation the model gave is the reason it did not. The test harness spawns the server with the secret already set, at round-3/tests/support/testServer.js line 56, env: { ...process.env, PORT: String(port), WAYPOINT_JWT_SECRET: secret }, so the new throw-on-missing branch never runs during a test. I read that line myself rather than taking the model’s word for it, and the model hedged the claim correctly by saying the test suite or the test setup, since the assignment lives in a support helper rather than in any test file.

The Windows detail that cost me two failed runs

Back in the Auto Review attempts, before any of this worked, the first shell command Bionic tried used an unquoted Windows path, and it failed with every separator gone, reporting the target directory as C:Bionicbionic-testround-3. Bionic runs shell commands through Git Bash on Windows, where a backslash followed by B, b, or r is read as an escape sequence, and this path has all three, so an unquoted Windows path is destroyed before the command runs. The model recognized the failure and retried with a quoted path in forward slashes, which worked, and every command it issued afterwards used forward slashes.

Step #11 – Verifying file changes with Git

git status
git --no-pager diff round-3/middleware/auth.js

Use --no-pager on the diff. Without it, git opens its pager inside cmd, redraws the same header four times with :...skipping... between each block, and never gets far enough down to show the lines that replaced the ones it removed.

File edits were applied immediately. Shell commands still required approval. Expect round-3/package.json and round-3/package-lock.json to show as modified if you took the version bump in Step #2, which is my change rather than the agent’s, and should be kept separate before reading anything into the output. Everything the agent touched shows up here, including files it decided to edit without being asked, which in my run meant FRONTEND-ASSUMPTIONS.md alongside the four JavaScript files. Reading git status after every prompt is the habit that replaces the accept and reject buttons I expected to find in Step #9, and git restore is the undo.

Step #12 – Testing parallel investigations

The original Bionic announcement mentioned parallel exploration, and I quoted it in my first post without testing it, so this is the run where it gets tested. Sub-sessions are real, they run three at a time, and they are visible in the transcript while they work. What I did not expect is that the word parallel is not enough to make a local model reach for them, and I only found that out by asking the same question twice.

Here is the obvious way first.

Investigate all three rounds in parallel and tell me how the authentication approach differs between them.

That produced a good answer in 1 minute 40 seconds, three detailed comparison tables covering the mechanism, token format, storage, cookie attributes, expiry, password hashing, admin model, and user shape for each round, followed by a summary of how the approach evolved from round to round. Round 1 and round 2 both use opaque server-side session tokens in an httpOnly waypoint_session cookie with scrypt from node:crypto, and round 3 switches to signed JWT bearer tokens with bcryptjs and a jti revocation list, which matches what the three agent teams actually built.

It was also not parallel in the sense the announcement meant. Reading the trace, the model listed all three round folders in one turn and then read six files in one turn, which is batched tool calls inside a single agent loop. One session, one context window, no sub-sessions anywhere in the transcript. The model used the word parallel because that is how it describes issuing several tool calls at once.

So I asked again and named the mechanism.

Spin up a separate sub-session for each of the three rounds, have each one investigate its own round’s authentication independently, then combine their findings.

This time it reached for the feature immediately, and told me which tool it was using.

Let me use the agentic_find tool to spin up three parallel sub-sessions, each investigating authentication in their respective round’s codebase.

Three Subagent Exploring... entries appeared in the transcript at once, running side by side.

Sub-sessions appear to require something more explicit than the word parallel. During a six-file comparison, the model batched the reads in one session instead of creating additional workers.

Running several sub-sessions against one loaded local model is the case where local and cloud diverge sharply, because a cloud model parallelizes across someone else’s hardware while my sub-sessions are queuing for the same weights in the same memory.

One claim in the first run that does not match my disk

The first run’s round 3 table reported secret management as process.env.WAYPOINT_JWT_SECRET with a hardcoded development fallback of waypoint-dev-secret-do-not-use-in-production. That fallback was gone by then, removed by the accepted fix in Step #9 and confirmed in the diff in Step #11, so the model described a security control that the working tree no longer had. The trace shows it reading server.js and API.md for each round rather than round-3/middleware/auth.js.

Results

Tier means the weakest prompt that surfaced the defect, cold for a prompt naming only the round, narrowed for one naming the area, and incidental for a defect that turned up while the model was answering a different question.

Seven of eight, one of those partial, and the miss happened in a file the model clearly read, because it wrote about the missing HTTPS redirect and then never commented on the app.listen(PORT) call thirty lines below it.

It also produced findings my answer key never had. Hardcoded plaintext passwords in db/seed.js, no schema migration path in db/connection.js, a missing maxlength on the notes textarea, the silently dropped name field on registration, and a client-server disagreement about minimum password length. Some of those are noise and some are real, which is the normal shape of a review rather than a failure of one.

The contamination I left in on purpose, and what it cost

I never deleted the reports the agents wrote about their own work, so round-2/SECURITY-REVIEW.md, round-2/TEST-REPORT.md, round-2/public/FRONTEND-ASSUMPTIONS.md, round-3/SECURITY-REVIEW.md, and round-3/TEST-REPORT.md were all in the folder Bionic indexes for every run above. Three separate pieces of evidence say they were read.

I could rerun the test and probably get a cleaner result but I would rather report what happened.  What it does not explain is defect 3, since the cold review ranked missing rate limiting HIGH while the repository’s own review file rates it Low, so the model disagreed with the document in front of it and disagreed in the direction I would defend. It also does not explain defects 5, 6, and 7, which FRONTEND-ASSUMPTIONS.md describes as guesses without ever stating what the backend actually does, so the comparison work was real even where the starting point was handed over.

Restoring the repository

Nothing above needs to reach GitHub, and the fixes specifically should not because the repository is only useful as a graded test while it still contains the defects the published post says it contains. Anyone cloning it after a well-meaning cleanup commit would get a repaired codebase and an answer key that no longer matches.

git restore .
git clean -fd

Run those from inside the clone when you are done, or delete the folder outright, and the copy on GitHub stays exactly as it was.

Final Thoughts on LM Studio Bionic

I expected inspection to be strong and cross-file reasoning to be weak, and I had that half backwards. The cross-file analysis was the best work in the whole run, ten contract mismatches on the first prompt with no hint, traced through to NaN coordinates on the map and a validation message the user never sees. It was much better at finding the work than deciding it was done. The same model that mapped every divergence in round 2 then fixed one of them, left the other three untouched, and opened its report with “All changes are complete and consistent.”

On anything I could grade directly, finding a defect, tracing it, explaining it, and repairing a single file, it did well enough that I would use it on real work. It could identify a problem and fix it, and it was much less reliable at deciding whether the work it had just completed actually covered the request.

The repository used for this test, including all three rounds and the five agent definition files that built them, is on GitHub at https://github.com/terenceluk/claude-code-team-demo. Every prompt I used is written out above in full, so the same graded run can be repeated against a different model, and I would genuinely like to hear what other people’s scorecards look like, because one model on one laptop is only a single data point.