One of the limitations of LLM agents such as ChatGPT is that (currently) they are fairly restricted in what personal data they can access and manipulate. Open AI is doing first steps with Gmail, GitHub and Google Calendar integrations but for me this doesn’t go far enough. I want an agent that can act on my behalf, use tools and interact with my second brain.
I’ve already had a taste of this vision by using Cursor. When I open my second brain repo in Cursor I have an agent that can answer questions using all my personal information for context. And I can also use that same agent to edit & improve my notes. But with Cursor I’m limited to doing this on a computer. I feel like it would be incredibly helpful if I could do all of those things on the go with my phone. So that’s why I built my personal agent Ariyaa.
Hello Ariyaa
Ariyaa is a chat agent similar to ChatGPT but with a feature set personalized for me. She runs in a docker container on my home server and has access to various tools such as:
- Shell: Alpine linux shell and system tools, e.g. git, grep, python3, curl, rsync, imagemagick and many more. When I see that she wants to use a tool that is not installed yet I can easily provide it by adding it to her docker container. By simply giving a agent shell access and thereby full use of a computer it becomes instantly very versatile.
- N8N MCPs: Via n8n I can easily expose various MCP tools to Ariyaa such as Google Calendar, Gmail, Google Contacts and others. By using N8N I don’t need to make this integration individually in the application but instead I can configure it on web UI (set developer tokens etc.)
- Scheduler: Ariyaa can schedule tasks for herself such as reminding herself to check in with me after a doctors visit or checking the news and providing a summary at a specific time.
- Web Search: Ariyaa can search the web and ingest the results.
- Image Generation: Ariyaa can generate images.
- Image Viewer: Ariyaa can view images attached to a message by the user but she can also view images stored on the server.
- Push Notification: Ariyaa can send a push notification via a PWA mechanism to my phone or browser. This is useful if she’s reacting to a scheduled task (e.g. send me the news at a specific time) or if she has a problem where she needs help (e.g. unable to commit & push a file).
- Search conversation: Only the last 20 messages are included as context for Ariyaa by default. However with the search conversation tool she can search the entire conversation history.
- Handoff: Ariyaa can handoff the conversation to a different agent that is more specialized to deal with a certain task. So far I’ve only implemented this as a PoC with an Elrond agent that is specialized to answer questions about Middle Earth. In the future as Ariyaa’s prompt grows too big it may be worthwhile to split her into specialized agents.
Tech Stack
Ariyaa is built in NodeJS with the Open AI typescript agents framework. The interaction between the frontend and backend is done via WebSockets. Large parts of the application were prototyped with Cursor and later refactored & refined by me. Ariyaa is compiled into a docker container and deployed automatically via a GitHub workflow.