Building an AI Agent From Scratch (No Frameworks) — Lessons Learned
I recently experimented with AI agents, but instead of using frameworks like
LangChain or Autopen, I decided to implement a minimal agent in plain Python.
The goal wasn’t production-ready tooling — it was understanding the mechanics.
I wanted to see what an agent is really doing under the hood.
Key takeaways:
- Agents are mostly structured orchestration around an LLM.
- The core loop is Think → Act → Observe.
- Tool invocation, iterative reasoning, and memory handling can all be implemented in a few dozen lines of code.
- Frameworks like LangChain are convenient, but they hide the underlying mechanics.
Here’s a discussion I’d love to start: for those of you experimenting with AI agents, do you usually start from scratch to understand the logic, or do you rely on frameworks?
Trending on Indie Hackers
This resonates. I went through the same exercise building a tech news aggregator with AI summaries — started with LangChain, then stripped it out once I realized I was fighting the abstraction more than using it.
What I've found:
The Think → Act → Observe loop is deceptively simple in concept but the real complexity lives in:
Context management — what goes into the prompt, what gets summarized, what gets dropped. Frameworks often have opinions here that don't match your use case.
Error recovery — when a tool call fails or returns unexpected output, handcrafted logic is often clearer than framework retry policies.
Observability — understanding why the agent took a path. With raw code, you can log exactly what you need. Frameworks sometimes bury this.
To your question: I usually start with a framework for prototyping (fast iteration), then rewrite the core loop in plain code once I know what I actually need. The framework teaches me the patterns; the rewrite teaches me the edge cases.
What was the hardest part of the implementation for you — the tool invocation plumbing, or the reasoning/memory piece?
If you are interested, this are my findings on building agents from scratch https://medium.com/@kamil.tustanowski/ai-agents-101-from-concept-to-code-no-frameworks-required-2dfdaf66b6c1