Hi everyone,
I’m the founder of Heshware LLC, and I’m currently developing an AI project called Marven.
Marven started as an experiment in building a more personal AI assistant, but over time the project has grown into a larger research and engineering question:
What would it take to build a local-first AI system that doesn’t simply respond to prompts, but develops meaningful continuity with a user over long periods of time?
The goal is not to claim that Marven is conscious or to build another wrapper around an LLM.
I’m interested in developing an architecture where an AI can maintain persistent memory, understand changing human context, learn from long-term interactions, preserve continuity across sessions, and operate primarily on infrastructure controlled by the user.
What I’m Trying to Build
The long-term vision for Marven is a Personal Sovereign AI — an AI system whose memory, models, identity data, and personal context can remain under the user’s control rather than existing entirely inside a centralized cloud platform.
Some of the capabilities I’m exploring include:
- Persistent long-term memory
- Episodic and semantic memory
- Memory consolidation and archival
- Retrieval-augmented generation
- Continual learning without blindly retraining on every interaction
- User-specific context modeling
- Emotional and conversational context
- Voice-based human-state detection
- Long-term identity and behavioral continuity
- Local inference
- Hybrid local/cloud model routing
- AI tool use and agent capabilities
- Permission and memory boundaries
- A persistent digital-human interface
One of the core questions behind Marven is:
Can an AI maintain continuity with someone for years while still giving that person ownership and control over what the AI remembers?
Current Architecture
Marven is still experimental and the architecture continues to evolve.
The current/local stack has included:
LLM inference
- Ollama
- Mistral 7B
- Llama 3 8B
- Phi-3 experimentation
- Exploring vLLM for improved inference infrastructure
- Experimenting with LoRA/fine-tuning for Marven-specific behavior
Memory
- SQLite-based persistent memory storage
- FAISS/vector retrieval
- RAG-based contextual memory
- Structured memory objects
- Conversation history
- Long-term memory archives
I’ve also been experimenting with what I call the MetaMirror system.
MetaMirror is intended to represent more than conversation history. The idea is to maintain structured observations about interaction context and how that context changes over time.
For example:
Interaction
↓
Conversation / Voice Input
↓
Context Analysis
↓
Working Memory
↓
Memory Importance Evaluation
↓
Episodic / Semantic Memory
↓
Long-Term Memory Archive
↓
Retrieval + Context Reconstruction
↓
Future Marven Interaction
The architecture is moving toward separating memory into different layers instead of continuously dumping conversations into a vector database.
That distinction is becoming increasingly important to me.
Remembering everything is not necessarily intelligence.
Knowing what should be remembered, what should be forgotten, what should be summarized, and when something should be retrieved is a much harder problem.
Human-State Awareness
Another part of the project is what I’m currently thinking of as a Human-State Engine.
The idea is to give Marven additional context regarding how someone is communicating rather than only processing the literal words they say.
Potential signals include:
- Text sentiment
- Linguistic tone
- Speaking speed
- Changes in vocal pitch
- Pauses
- Energy level
- Conversational patterns
- Long-term deviations from an individual’s normal communication style
Future experiments could potentially include visual signals, but I want this architecture to be privacy-conscious and based around explicit user consent.
For example, instead of simply receiving:
"I'm fine."
the system might internally receive contextual information closer to:
Text: "I'm fine."
Voice indicators:
- lower energy than personal baseline
- longer pause before response
- reduced speaking speed
Conversation context:
- previously discussing stressful event
Confidence:
- uncertain emotional state
The LLM still determines how to respond, but it receives richer contextual information.
Importantly, I do not want Marven confidently diagnosing someone’s emotions.
Human-state detection should be probabilistic and uncertain by design.
Digital Human / Presence Layer
I’ve also been working on a persistent visual representation of Marven.
The current direction uses a 3D digital-human interface with states such as:
- Idle
- Listening
- Thinking
- Speaking
- Eye contact
- Facial movement
- Environmental changes based on interaction mode
The goal isn’t simply to create an animated chatbot.
I’m interested in whether maintaining a consistent visual and behavioral presence contributes to the feeling of continuity in long-term human-AI interaction.
Voice
The voice pipeline has experimented with technologies including:
- Whisper
- faster-whisper
- Vosk
- Local speech recognition
- Neural TTS experimentation
Eventually I want voice interaction to operate with sufficiently low latency that Marven feels less like:
Speak → wait → AI responds
and more like a natural conversational system.
That requires work across:
Wake word
↓
Voice activity detection
↓
Streaming STT
↓
Context + memory retrieval
↓
Model inference
↓
Streaming TTS
↓
Digital-human animation
Latency across that entire pipeline is one of the engineering challenges I’m currently investigating.
The Hardest Problem: Memory
The part of Marven I’m currently most interested in improving is the memory architecture.
Most current AI assistants effectively have some combination of:
Conversation History
+
Vector Database
+
Retrieval
That works, but I’m interested in something more structured.
I’m experimenting conceptually with a system closer to:
Sensory / Interaction Memory
↓
Working Memory
↓
Importance Scoring
↓
┌────────┴─────────┐
↓ ↓
Episodic Semantic
Memory Memory
↓ ↓
└────────┬─────────┘
↓
Memory Consolidation
↓
Long-Term Archive
↓
Contextual Retrieval
Some questions I’m investigating:
How should memory importance be calculated?
Something mentioned once may be extremely important, while something repeated every day may not be.
How should memory decay work?
Human memory isn’t an infinitely growing database.
How should contradictions be handled?
If someone’s preferences or beliefs change, the AI shouldn’t continue retrieving outdated information as if it were current.
How do you prevent memory poisoning?
If an attacker can manipulate persistent memory, prompt injection becomes much more dangerous.
How should memories be consolidated?
Hundreds of interactions about the same subject should potentially become a higher-level semantic representation rather than hundreds of individual memories.
How can the model know why a memory matters?
Similarity alone may not be enough for good retrieval.
Continual Learning
Another challenge is determining how much of Marven’s learning should happen through:
- Prompt/context learning
- Retrieval
- Structured memory
- Adapter training
- LoRA
- Periodic fine-tuning
- Preference models
- User-specific models
I don’t think continuously fine-tuning the base model on every user interaction is necessarily the correct solution.
My current thinking is that most personalization should initially happen through memory and contextual systems, while actual model-weight updates happen much more deliberately.
I’d be especially interested in hearing from anyone researching continual learning or catastrophic forgetting.
AI Security
Persistent memory also creates an entirely new security problem.
I’m actively thinking about threats including:
- Prompt injection
- Persistent-memory poisoning
- RAG poisoning
- Malicious document ingestion
- Tool/agent permission escalation
- Sensitive-memory exfiltration
- Cross-user memory leakage
- Model supply-chain risks
A personal AI could eventually know more sensitive contextual information about someone than almost any other piece of software they use.
That means security cannot be something added after the architecture is finished.
I’m particularly interested in architectures where memories have permissions, provenance, confidence scores, timestamps, and potentially cryptographic integrity verification.
Local-First Infrastructure
One of Marven’s fundamental design principles is that users should eventually have the option to run significant portions of the system locally.
That could mean:
Personal Device
↓
Local Marven Node
↓
Local Memory Database
↓
Local / Home LLM Server
↓
Optional Cloud Models
The cloud should ideally become an optional compute resource rather than the permanent owner of someone’s AI memory.
I’m also exploring home-server/NAS-style architectures where an individual’s long-term AI memory could exist on storage they physically control.
What I’ve Built So Far
Marven isn’t only a concept.
I’ve already experimented with or built portions of:
- Local LLM inference
- Persistent SQLite memory
- Vector-based retrieval
- RAG
- Marven-specific structured memory
- Voice recognition pipelines
- Text-to-speech
- Digital-human presence
- React Native companion interfaces
- User authentication
- Local/cloud architecture experiments
- Memory/context engines
- Emotional-context experiments
- LoRA dataset preparation
- Multi-component AI pipelines
A lot of these pieces are currently separate experiments.
One of my current priorities is turning them into a cleaner unified architecture.
What I’m Looking For
I’m posting here because I would really like feedback from people who work in areas such as:
- LLM architecture
- RAG
- Vector databases
- Knowledge graphs
- Continual learning
- Memory architectures
- Fine-tuning / LoRA
- Local inference
- Quantization
- Speech AI
- Multimodal AI
- AI security
- Agent architecture
- Human-computer interaction
- Affective computing
- Digital humans
I’m especially interested in contributors who enjoy experimenting with research ideas and turning them into working prototypes.
You don’t need to be an expert in every area.
Even feedback like:
“Your memory architecture will eventually break because of X.”
or
“There’s a paper/project you should study that already tackles part of this.”
would be extremely useful.
Questions for the Hugging Face Community
I’d particularly like opinions on a few questions:
1. If you were designing long-term memory for an AI intended to interact with someone for 5–10+ years, what architecture would you use?
2. Would you combine vector retrieval with a knowledge graph, or use another architecture entirely?
3. What is the best way to separate episodic memory from semantic/user-profile memory?
4. How would you design memory consolidation and forgetting?
5. How would you defend persistent AI memory against poisoning and prompt injection?
6. What parts of personalization actually deserve model fine-tuning versus external memory?
7. Are there open-source projects or papers exploring similar long-duration human-AI relationships that you think I should study?
Collaboration
I’m also open to collaborating with developers or researchers who find these problems interesting.
Right now I’m particularly interested in people who could help experiment with:
Memory / RAG
- memory ranking
- memory consolidation
- knowledge graphs
- retrieval architectures
Local AI
- inference optimization
- quantization
- vLLM
- local model routing
Continual Learning
- LoRA/adapters
- preference learning
- safe personalization
Voice / Multimodal
- streaming speech recognition
- conversational latency
- voice-state analysis
AI Security
- prompt-injection defenses
- memory provenance
- RAG security
- permission systems
This is still an early-stage research and development project, so I’m interested in people who enjoy exploring difficult questions rather than only implementing a predetermined product specification.
Marven is being developed through Heshware LLC, but I’m also interested in keeping portions of the research and experimentation open enough that other developers can learn from it and contribute ideas.
Ultimately, the question I’m trying to explore with Marven is pretty simple:
What happens when an AI stops being something you repeatedly start a new conversation with and instead becomes a system capable of maintaining years of meaningful, user-controlled continuity?
There are a lot of unanswered engineering, security, privacy, and research questions between here and there.
That’s exactly why I’m interested in building it.
I’d love to hear what the Hugging Face community thinks.