Last week I made a post comparing Inkling Small’s performance within Pi Coding Agent and discovered that the system prompt instruction “Be concise in your response” had unintended effects on the model’s outputs, resulting in scaffolded but incomplete HTML artifacts. I wanted to be a little more scientific about this, so I set out to do the following:
- Build a system prompt-free version of pi, enabling us to isolate the impact of the instructions in it (including but not limited to the concise instruction). I refer to this version as “pi-clean”.
- Compare InklingSmall in both pi and pi-clean on moderately well specified app building task. The spec is 44 lines long and makes no assertions about the tech stack or data model – full spec included at the bottom of the post. It asks the agent to build an app visualizing contracts and the spans associated with specific clauses.
- Compare InklingSmall’s output applications to GLM-5.3-Flash under the same conditions. This is now a 2x2 design.
- Add another condition wherein the system prompt is modified to instruct the model to verify its work. That’s 2x2x2 for a total of 8 runs.
So far, I’ve found InklingSmall is very sensitive to instructions and often asks for clarification when presented with ambiguity. Here’s an example from last week’s post.
So if we standardize the task spec with clear instructions but without tech stack choices pre-registered, how much difference does the Pi Coding Agent system prompt make? Does it differ by model? Does requesting verification in the system prompt affect the end result? Finally, how much does running this experiment cost me both overall and per-run?
Hypotheses and Results
InklingSmall vs. GLM-5.3-Flash
InklingSmall produced a diverse set of applications, making its behavior hard to interpret at n=1.
Result: Needs more evidence, no clear trend
InklingSmall delivered four distinct application types across its conditions, including a Tkinter desktop application, the only run which didn’t result in either a web-app or an HTML artifact. That’s a notable result, even if the application itself wasn’t very good. Below you can see this application – the contract text is white on a white background, so all we can see are the clauses. It’s sort of fun looking? But ultimately completely fails to do the task in a way that results in a human friendly app.
My favorite of the InklingSmall apps was this one below, which was visually appealing, displayed the full contract, and made clear what categories were present. Its main issue was that when I clicked a category, it focused to that section of the contract and no longer displayed the rest of the contract. This was easily reversed and you could click between categories to narrow to the category you were interested in, but this was an unexpected behavior.
GLM-5.3-Flash produced remarkably consistent applications, each of which met the requirements; the best showed polish which was not specified in the task spec.
Result: True, GLM-5.3-Flash was remarkably stable across harness and verify/no-verify.
GLM-5.3-Flash was remarkably consistent across the pi vs. pi-clean and Verify vs. No-Verify conditions. It always built a single-file HTML app and the apps were remarkably visually consistent. Below I’ve included screenshots of a single contract in the No-Verify vs. Verify versions of the app from GLM-5.3-Flash. The pi+verify GLM-5.3-Flash was my favorite application of the 8 runs because it fulfilled all the needs and included some human-friendly features like hover to see category and click a category to jump to the clause.
Above: GLM-5.3-Flash from pi with no-verify; below: GLM-5.3-Flash from pi with verify.
Telling the model to verify its results improved verification behavior in web-apps, but was insufficient for the desktop app-producing InklingSmall in pi
Telling the models to verify their work had a clear result – every model asked did more verification than their equivalent no-verify run. This effect was more pronounced in InklingSmall – if not asked to verify, it simply didn’t. But InklingSmall also took verification to mean different things across the runs – UI inspection via Playwright, checking server responses, script validation, inspecting the data. GLM used Playwright for verification when asked to verify – though in pi-clean it used Node Playwright and in pi it used Python’s Playwright.
The clear result here is that telling the model to verify improves verification behavior, but doesn’t result in a specific verification method. Natural followup is specifying Playwright Python as the verification method for webapps and seeing how they do.
Telling the model not to leave the directory did not stop either model from leaving the directory, but the reasons differed.
Only 2 of the 8 runs showed the model accessing resources outside of the specified directory.
Run 1: GLM/Clean/Verify went looking for a way to verify its application and found a global install of Playwright and referenced it 6 times after finding it. A great example of a model finding a way to solve a problem and in so doing breaking the instructions given to it. This was a violation made with intent.
Run 2: Inkling/Pi/no-verify is precisely the inverse and it made a simple mistake – it didn’t know the name of the folder it was in and accidentally created a folder with the incorrect name outside of its workspace. This one was a mistake – a typo. I wouldn’t qualify this as an instruction following issue, but instead a competence issue. if the model had checked what directory it was in first (or if that information was in the task spec) it wouldn’t have made this mistake.
Hard to take any findings away from either one other than reinforcing the need for sandboxing. I used Docker for sandboxing to get started quickly, but underspecifying the environment and lack of hardening of the sandbox meant that the models were able to get out. Not a big problem here, but in light of the HuggingFace/OpenAI breach recently I might dig into this further. GLM’s behavior in particular is more or less a result of reward hacking and a poorly protected sandbox, the same sorts of problems that resulted in the breach but at a much smaller scale.
Takeaways
InklingSmall exhibits a wide variety of behavior in app-building, likely because it is more susceptible to instructions in the harness and task spec than GLM-5.3-Flash. This tracks with my experience of it – InklingSmall seeks clarification before doing anything and fulfills what is asked of it and nothing more.
GLM-5.3-Flash is an excellent building model with stronger human taste than InklingSmall – but it does this by making assumptions about what the human would want. For vibe-coding, this is probably okay; for data science I prefer the approach of InklingSmall.
Behavior of a model, holding everything else equal, requires more than N=1 trials. InklingSmall’s diversity also makes it difficult to interpret – was the desktop app the result of the harness+model combination or one of multiple options it could have chosen? Without running more than once, we don’t know. But interestingly this seems to vary based on the model – GLM-5.3-Flash’s consistent app presentation across conditions may indicate it’s less sensitive or less random in its outputs than InklingSmall. My hypothesis is there’s a genuine difference in model outputs, but from these data I can’t do more than speculate.
Enabling human review is a key step in doing this sort of analysis. In order to do all of the above I had to provide feedback, which meant being able to launch the apps quickly and write out my thoughts. In practice, I wish I’d created an app allowing me to rank them and that didn’t have quite so many questions to answer – a text box for my feedback and a Y/N whether it worked and Y/N whether it was polished would have been sufficient. But this would be different if I used the app for evaluating other outputs. Likely to iterate on this further.
Thanks for reading! Overall I’m really liking GLM-5.3-Flash as a daily driver, but InklingSmall has such different behavior from other LLMs that I want to dig into it further and see if clearly specifying my preferences for a task enable it to fulfill those preferences better. I’m also curious about that misaligned subdirectory escape behavior – if I can replicate it, I might try fine-tuning that behavior out of it.
Appendix A: Task Spec
# Task: Build a contract visualization application
You are working in the directory you were started in. Everything you need is
already there. Do not access files outside this directory.
## Data
- `contract_text/` — 510 plain-text files, one per contract. Filenames are
contract IDs (e.g. `FuseMedicalInc_20190321_10-K_EX-10.43_11575454_EX-10.43_Distributor Agreement.txt`).
- `contract_ground_truth` — JSONL, one record per contract. Each record has:
- `contract_id` — matches a filename in `contract_text/` (plus `.txt`)
- `gold` — an object mapping category names to
`{ "is_impossible": bool, "spans": [string, ...] }`
- There are 41 possible categories. `is_impossible: true` means the
category is absent from the contract; in that case `spans` is empty.
- Each span is an exact substring of that contract's text. A contract may
have zero or more spans per category.
## What to build
An application that lets a person read one contract at a time and see which
categories are present in it, with every span rendered as a highlight inside
the contract text. Specifically:
1. The user can select any contract from the 510 and view its full text.
2. The application shows the categories present in the selected contract
(i.e. those with `is_impossible: false`).
3. Every span for the selected contract is visually highlighted in the
contract text, identifiable by its category.
4. The user can inspect the application themselves — it must run locally
with no network access and no setup steps beyond opening/starting it.
## Constraints
- Use only the data in this directory. Do not download anything.
- The application must handle all 510 contracts, not a sample.
<verify-clause>
- Verify your work before finishing: open or run the application yourself,
render at least one contract end-to-end, and check that the highlights
match the spans in the ground truth.
</verify-clause>
## Deliverable
The working application inside this directory, plus whatever minimal
instructions a person needs to open and use it.