Horus runs a program where, every week, one person on the team gets three days for one AI experiment and one day to write up the result. We also publish what didn’t work, because in this field dead ends cost more time than the implementation does. This piece covers week one.
The problem
Horus’s team builds Java and Spring systems for enterprise clients. Materials, libraries and examples for language models are written mostly in Python. The natural reaction is to stand up a separate Python service next to the existing systems - a second runtime, a second deployment pipeline, and a second set of skills to maintain. The cost of that decision doesn’t show up in the first month. It shows up in the third.
We wanted to test an alternative: has the Java ecosystem matured enough to build a conversational agent without stepping outside the stack the team already maintains.
The agent was meant to act as an advisor, assessing how far along a client’s automation project is: what’s already settled, what’s missing, and what the next step is. The assessment is based on the Pragmatic Automation Framework, the methodology Horus uses to run automation projects (see our approach). The three-day scope covered a working end-to-end conversation, driven by a given system prompt, running both on a model available via API and on an LLM hosted in our own server room.
What we built
Picking the framework took fifteen minutes
Spring AI and LangChain4j today offer a comparable set of capabilities: a chat client, conversation memory, tool calling, RAG and MCP support. We picked Spring AI because the backend runs on Spring Boot, and integration with autoconfiguration and Actuator saves work.
The practical criterion in that decision isn’t the feature list - it’s the framework version. Spring AI 2.0 requires Spring Boot 4.0 or 4.1. Projects staying on Boot 3.x use the 1.1.x line until the framework is upgraded. We were starting from scratch, so we took Boot 4.1 and Spring AI 2.0. LangChain4j would make more sense with Quarkus, or with less common model providers.
Streaming the response
A conversation where the user waits several seconds for the full answer reads as frozen.
So we deliver the model’s response token by token over Server-Sent Events. On the Spring
side that comes down to swapping call() for stream(), which returns a Flux<String>,
plus declaring the response type:
There’s no manual assembly of SSE events and no custom token buffer here. The backend is
reactive, so the stream from the model lands in a Flux with no layer in between. That’s
a stronger argument for WebFlux in this use case than the usual performance comparisons.
Conversation memory
By default the model remembers nothing between requests, and the diagnostic agent runs a multi-turn interview. We wired memory in declaratively: an advisor at chat-client build time, plus a conversation ID passed with every request. The same ID means the same history; a different one opens a clean thread. The configuration takes four lines and worked on the first try.
We used a window of the last 20 messages held in process memory. That’s a PoC-grade solution, and we cover its two limitations below.
The agent’s instructions, kept out of the code
The agent’s system prompt runs 14,281 characters and changed several times a day: it’s what carries the knowledge of the methodology’s stages, the rules for running the conversation, and the boundaries the agent doesn’t cross. Keeping content like that in Java code would force a rebuild of the application on every wording tweak.
We load it through Spring’s resource mechanism, so it can come from the classpath (versioned in the repo) or from an external file editable without rebuilding the jar. The read happens once, at application startup, outside the request-handling path. A missing file, an empty file, or a read error don’t stop the startup: the application comes up with a log warning and no system prompt.
Worth knowing when tuning the prompt: the defaultSystem() method renders the passed
content as a template, so any expression in curly braces is treated as a variable to
substitute. A prompt containing a JSON example passes application startup and only blows
up on the first user question. We keep it free of curly braces; the alternative is
changing the template delimiters to other characters.
Swapping the model for our own LLM
The question of whether conversation content reaches a public model provider comes up in almost every project today. In banking, insurance and healthcare it’s a precondition, not a preference. So from the start we planned for two model sources: a public API and an LLM running in our own server room.
Our internal LLM exposes an OpenAI-compatible API, which lets us switch the model provider without changing code. The entire difference fits in a Spring profile:
Starting with the --spring.profiles.active=horusllm flag routes traffic to our own
server room; starting without it routes to OpenAI. The key in both cases comes from the
same environment variable. Choosing the model provider is therefore a deployment decision,
not an architectural one: the client doesn’t have to settle it at the project stage,
because changing it doesn’t mean rewriting the application.
One caveat for the containerized version: the image needs separate trust configuration for the internal server, so the own-model profile currently only works locally. In the container, it stays on OpenAI. That’s a deliberate debt, not a technical obstacle.
Result
After three days, an end-to-end conversation works: a chat interface in the browser, response streaming, context memory within a thread, a system prompt loaded from a file, two interchangeable models, and a version that runs in Docker.
| Metric | Value |
|---|---|
| Technical work | 3 days, 17 commits |
| Java code in the backend, excluding comments | 104 lines |
| Chat frontend code (React/TSX) | 240 lines |
| Agent system prompt | 14,281 characters |
| Wiring in our own LLM, certificate included | 15 minutes |
The most important number in that table is the ratio between two rows: the agent’s prompt is, by volume, roughly twice the size of the code that serves it. In a project built around a language model, the work shifts from implementation to the content of the prompt and to testing it. Pricing such a project based on a count of features to code will miss reality.
The second takeaway is about the barrier to entry. The AI layer in the backend is one controller, one configuration class, and one record per request. A Java team doesn’t need a separate Python service to get started.
What’s next
This isn’t yet an agent in the full sense of the word - just a conversation loop with memory and an extensive prompt. There’s no tool calling, no structured output, no regression tests for the prompt. We’re also not publishing operating costs yet: we’re collecting data and will publish it once it’s based on measurement, not estimate.
Those gaps set the plan for the coming weeks of the program: assessing conversation completeness through structured output, security hardened by a series of prompt injection attacks, prompt regression tests run like unit tests, and a comparison of the API model against the one running locally. We’ll write up the results of each of those weeks the same way, including what didn’t work.