Recently I’ve been working on creating a prompt-learning autoresearcher – a system wherein one LLM acts as a data scientist to understand flaws in a prompt that lead to errors in model outputs as compared to some ground truth dataset. More on that another time. I’m using it as an excuse to try out OpenCode and Pi using open source models instead of my usual Claude Code subscription and the first model I tested was InklingSmall.
InklingSmall is Quite Honest
InklingSmall on paper has some interesting qualities. It’s just about as high quality across a wide variety of benchmarks as its larger cousin, Inkling. Both models were trained with an emphasis on epistemics – how well calibrated, resistant to censorship, and good at following instructions it is. I like this philosophy – it’s a consistent issue with LLMs that that express a lot of confidence in answers that turn out to be incorrect. If they just said “hey, I don’t know this, can you tell me more or send me a source?” we’d like them a lot more. It’s one of my main frustrations with Opus 5 and Fable, but that’s for another post.
See below, where I asked InklingSmall to create a survey-llms Skill.
Instead of immediately rushing to do what I asked (eager builder behavior like we find in Fable) InklingSmall noted my request was ambiguous and requested clarification. Which was really novel! It made me way more interested in using InklingSmall because in my data science work I really don’t want LLMs making assumptions about my intent. Yes I have instructions my CLAUDE.md indicating it should ask, but in practice this is the first model I’ve tested that does it this consistently and thoroughly.
I’ve found it really nice for small scoped questions and smaller edits of my codebase. I could see it working extremely well as a subagent with a smarter orchestrator because it does such a good job in asking questions and requesting more information instead of making assumptions. It’s also quite good at instruction following.
…Most of the Time
Unfortunately, at least in Pi InklingSmall did really poorly once I got to moderately sized jobs. I was excited by Thinking Machine’s demos of its visual capabilities, so I asked it the following, looking for a quick visual of some contract data I’m working with overlaid with the key clauses according to the ground truth vs. the model’s outputs:
In @studies/008-principled-extraction-cuad/investigations/008-principle-improvement-loop/n1-visibility/ we have a single iteration of our principle extractor. I'd like us to create an application which renders (1) the full contract, (2) a comparison mode which shows up to two outputs side by side where an output can be from the base model, the ground truth, or a model with added principles, and (3) I'd like this to be reusable for different contracts such that a user can click between contracts (though we currently would only have one option. Really what I'm looking for is insight that's both human and agent readable -- did the model improve? Did it match more of the ground truth? Are we extracting data well, but not highlighting well? Etc.
It did produce an artifact and I could interact with it. But it didn’t have the full contract, only the first few lines. And it didn’t show me what the ground truth was or what the model outputs were.
So I asked it to try again, giving it some info about what I didn’t like about the artifact.
So question for you. There are several issues with this app. First, it doesn't contain all the data even for a single contract. Second, it doesn't display any highlights, despite having the dropdowns correct. Third, I asked for an app and you gave me a signle HTML page. My question is how can we enable you to correct these issues out of the box next time? Do you have instructions to keep it brief or do the minimum solution? Can I provide instrucitons in another manner that would allow you to do a more thorough job? Do you need a more detailed plan?
Based on my expectations working with InklingSmall, I fully expected it to tell me something like “I can’t do a good job without clearer instructions on where to find the data. Also, can I have a screenshot tool?” Instead, it gave me what appeared to be actionable insights which proved more or less useless.
It built the app, so I asked how to run it.
And it completely missed the target again. This time the Javascript rendering didn’t work and after another turn I gave up. InklingSmall was great at the small tasks I’d given it to that point, but an HTML artifact visualizing text data proved too much for it.
And here you can see what it told me was the reason for merely scaffolding and not implementing. Note the consistency in the reasoning and the actual response – it refers to a system prompt instructing it to be concise which overrides my direct ask. That feels like a problem and it also makes me suspicious – how could a model simultaneously score 64.7% on TerminalBench 2.1 and also struggle to build a simple HTML artifact? I repeated the exercise using HuggingFace as the inference provider (perhaps Thinking Machines includes a system prompt in their OAI) but observed the same result.
But Why?
Well, in prep for this post I was digging through the Pi Github and found a suspicious looking phrase in the: “Be concise in your responses.” And sure enough, when I looked at my Pi export log there it was: “Be concise in your responses.”
I tested it again with OpenCode and InklingSmall. I was a little surprised to see it dove right in instead of asking me questions. It didn’t look right at first (a minor syntax error in the javascript renderer), but after I prompted it again it fixed it quickly.
Observe the result:
That’s the full contract, fully present and scrollable.
Formatting could use some work, but it managed both the comparison of ground truth to the base model (very basic prompt) and (below) an overlap analysis which provides some actionable insights (Most Favored Nation and Non-Transferable License clauses need prompt refinement).
I definitely wouldn’t call it perfect, but for a simple one-off artifact it’s exactly what I need. If I’d tried this OpenCode experiment a few days earlier it would have saved me quite a bit of frustration and testing other models. Despite Pi promising a customizable experience it is rather opinionated in its system prompt and that had an impact on my use case. I found an issue closed without fix on their Github suggesting a solution to exactly this: https://github.com/earendil-works/pi/issues/8391 and a I’m disappointed to see it’s not something they’re planning on updating. Perhaps the only harness that’s truly customizable is one you build… or that’s empirically derived to be suitable.
(Look out for another post on Better Harness, Smaller Models from Yang et al (2026), fantastic paper that’s flown under the radar).
Takeaways
InklingSmall is a better model than I initially gave it credit for in Pi and I’ll be using it as my daily driver for the next week or two at least to see if I can better understand its strengths and limitations. It’s fairly obvious that the harness has an impact on output quality, but this is one of the first times I’ve seen such a simple line in the system prompt for a harness have this large of an impact. It turns me off from Pi a little frankly – if it’s going to be opinionated by default, at least let me turn them off with a parameter.
I suspect what made InklingSmall so susceptible to this issue is the same training that improved its instruction following. The problem is it followed a harness instruction over the user’s instruction and I didn’t have any way to revert that behavior (at least not without modifying Pi, which I may well do). I’d be curious if you could fine tune the model to respond differently to instructions by source – effectively allowing the user to overrule system prompt instructions coming from a harness. Maybe that’ll be a new research direction for me.
Either way, I’m excited to see how far InklingSmall can take me. Anyone looking for an inexpensive generalist which asks for clarification instead of assuming your intent should definitely give it a spin.