|
@@ -1,19 +1,21 @@
|
|
|
---
|
|
---
|
|
|
type: workflow
|
|
type: workflow
|
|
|
title: AI auto-reply workflow
|
|
title: AI auto-reply workflow
|
|
|
-description: End-to-end trace of the AI autoreply — enabling it per chat (F3) or globally (F4 / web panel) for private chats and groups, the ReplyGate rules and cooldowns, building the AI context, deferred replies, the startup catch-up, the vision path with its courtesy fallback, retries, and sending the reply.
|
|
|
|
|
-tags: [ai, autoreply, workflow, vision, reply-gate, cooldown, catch-up]
|
|
|
|
|
|
|
+description: End-to-end trace of the AI autoreply — enabling it per chat (F3) or globally (F4 / web panel) for private chats and groups, the ReplyGate rules, the reply delay that batches messages into one reply, the drop when the owner answered by hand, image descriptions stored once and reused, the startup catch-up, building the AI context from SOUL.md, the no_reply tool, retries and sending the reply.
|
|
|
|
|
+tags: [ai, autoreply, workflow, vision, reply-gate, reply-delay, catch-up, image-description]
|
|
|
|
|
+verified:
|
|
|
|
|
+ - by: openwiki/0.7.2
|
|
|
|
|
+ at: 2026-10-21T21:49:12.291Z
|
|
|
sources:
|
|
sources:
|
|
|
- id: openwiki-source-9fec9d1af408501c9c455161
|
|
- id: openwiki-source-9fec9d1af408501c9c455161
|
|
|
resource: repo://src/app.rs
|
|
resource: repo://src/app.rs
|
|
|
- id: openwiki-source-043e6252badc57a1ee6bfb48
|
|
- id: openwiki-source-043e6252badc57a1ee6bfb48
|
|
|
resource: repo://src/autoreply.rs
|
|
resource: repo://src/autoreply.rs
|
|
|
|
|
+ - id: openwiki-source-2a737474d86fc75cc9d9694f
|
|
|
|
|
+ resource: repo://src/config.rs
|
|
|
- id: openwiki-source-ce0c4ea8cd3b15431fa691ce
|
|
- id: openwiki-source-ce0c4ea8cd3b15431fa691ce
|
|
|
resource: repo://src/history.rs
|
|
resource: repo://src/history.rs
|
|
|
-generated: { by: "claude-code", at: "2026-10-20T20:27:06.515Z" }
|
|
|
|
|
-verified:
|
|
|
|
|
- - by: openwiki/0.7.2
|
|
|
|
|
- at: 2026-10-20T20:27:06.515Z
|
|
|
|
|
|
|
+generated: { by: "claude-code", at: "2026-10-21T21:49:12.291Z" }
|
|
|
---
|
|
---
|
|
|
|
|
|
|
|
# AI auto-reply workflow
|
|
# AI auto-reply workflow
|
|
@@ -22,154 +24,145 @@ The autoreply answers incoming WhatsApp messages with an AI-generated text,
|
|
|
only for chats where the user turned it on — one by one, or all at once with
|
|
only for chats where the user turned it on — one by one, or all at once with
|
|
|
the global switch. Decision rules and context building are pure functions in
|
|
the global switch. Decision rules and context building are pure functions in
|
|
|
`src/autoreply.rs` (no I/O, `src/autoreply.rs:1-4`); orchestration lives in
|
|
`src/autoreply.rs` (no I/O, `src/autoreply.rs:1-4`); orchestration lives in
|
|
|
-[`App`](../architecture/app-state.md); the HTTP call is done by the
|
|
|
|
|
|
|
+[`App`](../architecture/app-state.md); the HTTP calls are done by the
|
|
|
[AI client](../integrations/ai-client.md).
|
|
[AI client](../integrations/ai-client.md).
|
|
|
|
|
|
|
|
```
|
|
```
|
|
|
-WaEvent::Message { message, image, addressed_to }
|
|
|
|
|
|
|
+WaEvent::Message { message, image }
|
|
|
-> History::save_message (new only)
|
|
-> History::save_message (new only)
|
|
|
-> App::maybe_autoreply (enabled = global OR per-chat flag)
|
|
-> App::maybe_autoreply (enabled = global OR per-chat flag)
|
|
|
- ReplyGate::should_reply? --yes--> start_reply
|
|
|
|
|
- else ReplyGate::deferral_due? --yes--> mark_pending + timer
|
|
|
|
|
- -> AppEvent::CooldownElapsed
|
|
|
|
|
- -> take_pending && enabled? -> start_reply (text-only)
|
|
|
|
|
- else stop
|
|
|
|
|
-WaEvent::OfflineSyncCompleted (first time only)
|
|
|
|
|
- -> App::catch_up -> start_reply (text-only) per due chat
|
|
|
|
|
-
|
|
|
|
|
-start_reply:
|
|
|
|
|
- ReplyGate::record_reply (clears pending)
|
|
|
|
|
- History::messages_since(context_start(now), 200) -> build_turns
|
|
|
|
|
- image + vision model? -> vision request (fallback: text request)
|
|
|
|
|
- else -> text request (each with up to ai_retries retries)
|
|
|
|
|
|
|
+ ReplyGate::should_schedule? --yes--> mark_pending + timer(reply_delay_seconds)
|
|
|
|
|
+ image from an allowed sender? ----> keep_image (pending_images / backlog_images)
|
|
|
|
|
+ ... AppEvent::ReplyDue
|
|
|
|
|
+ -> prepare_reply: reply_context (still enabled? AI? owner not answered last?)
|
|
|
|
|
+ --no--> finish_reply (drop)
|
|
|
|
|
+ images without description? --yes--> download + describe_image (parallel, max 5)
|
|
|
|
|
+ -> AppEvent::ImagesDescribed -> store descriptions
|
|
|
|
|
+ -> start_reply: messages_since(yesterday 00:00, 200) -> mark @[te] -> build_turns
|
|
|
|
|
+ -> AiClient::complete (tools: [no_reply], up to ai_retries retries)
|
|
|
-> AppEvent::AiReply
|
|
-> AppEvent::AiReply
|
|
|
- -> still enabled (global OR per-chat)? --no--> discard
|
|
|
|
|
- -> WaHandle::send_text -> AppEvent::Sent -> stored with is_ai = true
|
|
|
|
|
|
|
+ NoReply -> status "AI: nessuna risposta necessaria"
|
|
|
|
|
+ Text -> still enabled AND owner did not answer meanwhile? -> send_text -> stored is_ai = true
|
|
|
|
|
+ -> finish_reply: newer message arrived meanwhile? -> schedule one new reply
|
|
|
|
|
+WaEvent::OfflineSyncCompleted (first time only)
|
|
|
|
|
+ -> catch_up -> mark_pending + prepare_reply (no delay) per due private chat
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
## 1. Enabling
|
|
## 1. Enabling
|
|
|
|
|
|
|
|
There are two independent switches, and a chat is **enabled** when either is
|
|
There are two independent switches, and a chat is **enabled** when either is
|
|
|
-on (`autoreply_enabled`, `src/app.rs:366-372`):
|
|
|
|
|
|
|
+on (`autoreply_enabled`, `src/app.rs:521-527`):
|
|
|
|
|
|
|
|
- **Per chat** — F3 toggles the flag of the open chat (`toggle_autoreply` /
|
|
- **Per chat** — F3 toggles the flag of the open chat (`toggle_autoreply` /
|
|
|
- `toggle_chat_autoreply`, `src/app.rs:763-794`); the
|
|
|
|
|
|
|
+ `toggle_chat_autoreply`, `src/app.rs:976-1007`); the
|
|
|
[web control panel](../integrations/web-control-panel.md) can toggle any
|
|
[web control panel](../integrations/web-control-panel.md) can toggle any
|
|
|
listed chat the same way. Turning it off is always allowed; turning it on
|
|
listed chat the same way. Turning it off is always allowed; turning it on
|
|
|
- requires a configured AI (API key and `system_prompt`, see
|
|
|
|
|
- [Configuration](../operations/configuration-and-data.md)) and a chat that
|
|
|
|
|
- supports autoreply (`can_enable_autoreply`, `src/app.rs:1127-1140`),
|
|
|
|
|
- otherwise the status shows
|
|
|
|
|
|
|
+ requires a configured AI (API key in `config.toml` and a non-blank
|
|
|
|
|
+ `SOUL.md`, see [Configuration](../operations/configuration-and-data.md))
|
|
|
|
|
+ and a chat that supports autoreply (`can_enable_autoreply`,
|
|
|
|
|
+ `src/app.rs:1403-1416`), otherwise the status shows
|
|
|
`Autoreply disponibile solo nelle chat private e nei gruppi`. The flag is
|
|
`Autoreply disponibile solo nelle chat private e nei gruppi`. The flag is
|
|
|
persisted in the `chats` table of the
|
|
persisted in the `chats` table of the
|
|
|
[history store](../concepts/history-store.md) (default off).
|
|
[history store](../concepts/history-store.md) (default off).
|
|
|
- **Global ("AI tutte")** — F4 or the web panel toggles
|
|
- **Global ("AI tutte")** — F4 or the web panel toggles
|
|
|
- `global_autoreply` (`toggle_global_autoreply`, `src/app.rs:839-856`);
|
|
|
|
|
|
|
+ `global_autoreply` (`toggle_global_autoreply`, `src/app.rs:1052-1069`);
|
|
|
enabling requires an AI client. It is persisted in the `settings` table,
|
|
enabling requires an AI client. It is persisted in the `settings` table,
|
|
|
loaded at startup and does not change any per-chat flag, so turning it off
|
|
loaded at startup and does not change any per-chat flag, so turning it off
|
|
|
restores the per-chat selection. The TUI marks it in the list title and
|
|
restores the per-chat selection. The TUI marks it in the list title and
|
|
|
footer ([Terminal UI](../architecture/terminal-ui.md)).
|
|
footer ([Terminal UI](../architecture/terminal-ui.md)).
|
|
|
|
|
|
|
|
-The global switch does not bypass the chat-type rule: `should_reply` still
|
|
|
|
|
-requires a supported chat, so broadcasts, status and newsletters are never
|
|
|
|
|
-answered.
|
|
|
|
|
|
|
+The global switch does not bypass the chat-type rule: `meets_content_rules`
|
|
|
|
|
+still requires a supported chat, so broadcasts, status and newsletters are
|
|
|
|
|
+never answered.
|
|
|
|
|
|
|
|
-`supports_autoreply` (`src/autoreply.rs:89-97`) accepts private chats and
|
|
|
|
|
|
|
+`supports_autoreply` (`src/autoreply.rs:99-107`) accepts private chats and
|
|
|
groups:
|
|
groups:
|
|
|
|
|
|
|
|
- private: JID server `s.whatsapp.net` or `lid` (`is_private_chat`,
|
|
- private: JID server `s.whatsapp.net` or `lid` (`is_private_chat`,
|
|
|
- `src/autoreply.rs:66-76`);
|
|
|
|
|
-- group: JID server `g.us` (`is_group_chat`, `src/autoreply.rs:78-87`);
|
|
|
|
|
|
|
+ `src/autoreply.rs:76-86`);
|
|
|
|
|
+- group: JID server `g.us` (`is_group_chat`, `src/autoreply.rs:88-97`);
|
|
|
- broadcasts (including `status@broadcast`), newsletters and malformed JIDs
|
|
- broadcasts (including `status@broadcast`), newsletters and malformed JIDs
|
|
|
are excluded.
|
|
are excluded.
|
|
|
|
|
|
|
|
-## 2. Trigger
|
|
|
|
|
|
|
+## 2. Trigger and ReplyGate rules
|
|
|
|
|
|
|
|
Only messages that `History::save_message` reports as new reach
|
|
Only messages that `History::save_message` reports as new reach
|
|
|
-`maybe_autoreply` (`on_message`, `src/app.rs:304-313`;
|
|
|
|
|
-`maybe_autoreply`, `src/app.rs:324-346`), so duplicates and history-sync
|
|
|
|
|
-re-deliveries of stored messages never trigger. `maybe_autoreply` needs an
|
|
|
|
|
-AI client and a system prompt (`ai_setup`, `src/app.rs:430-436`), refreshes
|
|
|
|
|
-the own JIDs if unknown and computes `enabled` (global or per chat). It also
|
|
|
|
|
-receives `addressed_to`, the raw JIDs the message mentions or quotes (filled
|
|
|
|
|
-by `addressed_jids` in the [WhatsApp client](../integrations/whatsapp-client.md)).
|
|
|
|
|
-
|
|
|
|
|
-## 3. ReplyGate rules
|
|
|
|
|
|
|
+`maybe_autoreply` (`on_message`, `src/app.rs:333-342`; `maybe_autoreply`,
|
|
|
|
|
+`src/app.rs:344-366`), so duplicates and history-sync re-deliveries of
|
|
|
|
|
+stored messages never trigger. `maybe_autoreply` needs an AI client and a
|
|
|
|
|
+system prompt (`ai_setup`, `src/app.rs:628-634`), refreshes the own JIDs if
|
|
|
|
|
+unknown and computes `enabled` (global or per chat).
|
|
|
|
|
|
|
|
-`ReplyGate::should_reply(message, enabled, addressed_to, now)`
|
|
|
|
|
-(`src/autoreply.rs:176-194`) returns `true` only when **all** hold
|
|
|
|
|
-(`is_reply_candidate` and `meets_content_rules`,
|
|
|
|
|
-`src/autoreply.rs:258-282`):
|
|
|
|
|
|
|
+`ReplyGate::should_schedule(message, enabled)` (`src/autoreply.rs:182-191`)
|
|
|
|
|
+returns `true` only when **all** hold (`is_reply_candidate` and
|
|
|
|
|
+`meets_content_rules`, `src/autoreply.rs:241-265`):
|
|
|
|
|
|
|
|
| Rule | Detail |
|
|
| Rule | Detail |
|
|
|
|---|---|
|
|
|---|---|
|
|
|
|
|
+| Not pending | the chat has no reply already scheduled or in progress |
|
|
|
| Enabled | global autoreply on, or the per-chat flag on |
|
|
| Enabled | global autoreply on, or the per-chat flag on |
|
|
|
-| Sender allowed | not sent by the user — except human-written messages in the self-chat ("Messaggi a te stesso"); AI messages never trigger (`src/autoreply.rs:166-174`) |
|
|
|
|
|
|
|
+| Sender allowed | not sent by the user — except human-written messages in the self-chat ("Messaggi a te stesso"); AI messages never trigger (`sender_allowed`, `src/autoreply.rs:172-180`) |
|
|
|
| Supported chat | private chat or group (`supports_autoreply`) |
|
|
| Supported chat | private chat or group (`supports_autoreply`) |
|
|
|
| No backlog | `timestamp >= started_at`, the process start time |
|
|
| No backlog | `timestamp >= started_at`, the process start time |
|
|
|
| Has content | text not blank and not `[media]`/`[messaggio non supportato]`; voice notes and images do trigger |
|
|
| Has content | text not blank and not `[media]`/`[messaggio non supportato]`; voice notes and images do trigger |
|
|
|
-| Cooldown | enough time since the last reply in that chat (see below) |
|
|
|
|
|
-
|
|
|
|
|
-### Cooldowns
|
|
|
|
|
-
|
|
|
|
|
-`Cooldowns { private, group }` (`src/autoreply.rs:99-107`) comes from the
|
|
|
|
|
-configuration keys `cooldown_seconds` (default 30) and
|
|
|
|
|
-`group_cooldown_seconds` (default 300), built in `App::new`
|
|
|
|
|
-(`src/app.rs:197-200`); `0` disables a cooldown. The last reply instant is
|
|
|
|
|
-tracked per chat. `cooldown_for` (`src/autoreply.rs:334-346`) picks:
|
|
|
|
|
-
|
|
|
|
|
-- `group` for a group message **not** addressed to the user;
|
|
|
|
|
-- `private` for private chats and for group messages addressed to the user.
|
|
|
|
|
-
|
|
|
|
|
-A group message is addressed to the user when one of its `addressed_to`
|
|
|
|
|
-JIDs — an `@`-mention or the author of a quoted message — equals one of the
|
|
|
|
|
-own JIDs, ignoring a `:N` device suffix (`is_addressed`,
|
|
|
|
|
-`src/autoreply.rs:322-332`). Mentioning someone else does not shorten the
|
|
|
|
|
-cooldown. In private chats mentions make no difference.
|
|
|
|
|
|
|
|
|
|
The self-chat is detected by comparing the chat JID with the account's own
|
|
The self-chat is detected by comparing the chat JID with the account's own
|
|
|
-PN/LID JIDs, ignoring a device suffix (`src/autoreply.rs:145-164`,
|
|
|
|
|
-`strip_device_suffix`, `src/autoreply.rs:349-365`).
|
|
|
|
|
-
|
|
|
|
|
-`record_reply` is called **before** the AI request is spawned (in
|
|
|
|
|
-`start_reply`, `src/app.rs:438-457`), so a second message arriving during
|
|
|
|
|
-the request cannot cause a double reply; it also clears any pending
|
|
|
|
|
-deferral of that chat (`src/autoreply.rs:246-256`).
|
|
|
|
|
-
|
|
|
|
|
-## 4. Deferred reply (private chats)
|
|
|
|
|
-
|
|
|
|
|
-When a message in a private chat (including the self-chat) passes every
|
|
|
|
|
-rule except the cooldown, it is not dropped: `deferral_due`
|
|
|
|
|
-(`src/autoreply.rs:196-224`) returns the instant the cooldown ends, provided
|
|
|
|
|
-a reply was already recorded for the chat, the cooldown is still running
|
|
|
|
|
-and no deferral is pending. `maybe_autoreply` then calls `mark_pending` and
|
|
|
|
|
-`spawn_cooldown_timer` (`src/app.rs:986-997`), a tokio task that sleeps
|
|
|
|
|
-until that instant and sends `AppEvent::CooldownElapsed { chat_jid }`.
|
|
|
|
|
-
|
|
|
|
|
-`handle_cooldown_elapsed` (`src/app.rs:348-364`) consumes the deferral with
|
|
|
|
|
-`take_pending`; if it was still pending, the chat is still enabled (global
|
|
|
|
|
-or per chat) and the AI is configured, it starts **one** text-only reply (no
|
|
|
|
|
-image is downloaded) built from the latest context, so several messages
|
|
|
|
|
-received during the cooldown get a single answer. If a live reply happened
|
|
|
|
|
-in the meantime, `record_reply` already removed the pending mark and nothing
|
|
|
|
|
-is sent. Deleting the chat locally also drops its pending deferral.
|
|
|
|
|
-
|
|
|
|
|
-Groups never defer: a group message blocked by the cooldown is simply not
|
|
|
|
|
-answered.
|
|
|
|
|
-
|
|
|
|
|
-## 5. Startup catch-up
|
|
|
|
|
|
|
+PN/LID JIDs, ignoring a `:N` device suffix (`is_self_chat`,
|
|
|
|
|
+`src/autoreply.rs:148-157`; `strip_device_suffix`,
|
|
|
|
|
+`src/autoreply.rs:293-309`). Groups follow exactly the same rules as private
|
|
|
|
|
+chats; mentions and quotes no longer change anything.
|
|
|
|
|
+
|
|
|
|
|
+## 3. Reply delay and batching
|
|
|
|
|
+
|
|
|
|
|
+When `should_schedule` passes, `schedule_reply` (`src/app.rs:368-376`) marks
|
|
|
|
|
+the chat pending in the gate (`mark_pending`, `src/autoreply.rs:202-210`)
|
|
|
|
|
+and `spawn_reply_timer` (`src/app.rs:1199-1209`) emits
|
|
|
|
|
+`AppEvent::ReplyDue { chat_jid }` after `reply_delay_seconds` (default 30;
|
|
|
|
|
+`0` answers at once). The chat stays pending until `finish_reply`, so every
|
|
|
|
|
+message arriving during the wait — and while the AI is generating — gets no
|
|
|
|
|
+timer of its own: the reply is built from the context loaded when it starts,
|
|
|
|
|
+so it covers everything received during the wait.
|
|
|
|
|
+
|
|
|
|
|
+`handle_reply_due` (`src/app.rs:417-428`) takes the images kept for the chat
|
|
|
|
|
+and, if the chat is still pending (a local deletion removes the pending
|
|
|
|
|
+mark), calls `prepare_reply` (`src/app.rs:430-451`). Its `reply_context`
|
|
|
|
|
+(`src/app.rs:529-547`) drops the reply — and `finish_reply` ends the pending
|
|
|
|
|
+state — when:
|
|
|
|
|
+
|
|
|
|
|
+- the autoreply was turned off (both switches) during the wait;
|
|
|
|
|
+- the AI is not configured;
|
|
|
|
|
+- the chat has no message in the context window;
|
|
|
|
|
+- the latest message was written by hand by the owner outside the self-chat
|
|
|
|
|
+ (`answered_by_owner`, `src/autoreply.rs:231-239`): the owner already
|
|
|
|
|
+ answered, so the AI stays silent.
|
|
|
|
|
+
|
|
|
|
|
+Otherwise it describes the new images first (see §5) or calls
|
|
|
|
|
+`start_reply` at once.
|
|
|
|
|
+
|
|
|
|
|
+### After the reply
|
|
|
|
|
+
|
|
|
|
|
+`AiReply` is always followed by `finish_reply` (`src/app.rs:499-519`), which
|
|
|
|
|
+clears the pending state and the `answered_up_to` snapshot. If a reply was
|
|
|
|
|
+started and the latest stored message is a different one
|
|
|
|
|
+(`arrived_after`, `src/app.rs:1307-1314`) that itself passes
|
|
|
|
|
+`should_schedule`, a **new** reply is scheduled (with a fresh delay, keeping
|
|
|
|
|
+the images received meanwhile); otherwise the kept images are dropped. So
|
|
|
|
|
+messages that arrive while a reply is being generated are answered once,
|
|
|
|
|
+afterwards, rather than being lost or triggering parallel replies.
|
|
|
|
|
+
|
|
|
|
|
+## 4. Startup catch-up
|
|
|
|
|
|
|
|
Messages received while the app was off are delivered on connection and are
|
|
Messages received while the app was off are delivered on connection and are
|
|
|
older than `started_at`, so the live rules treat them as backlog. When
|
|
older than `started_at`, so the live rules treat them as backlog. When
|
|
|
WhatsApp signals that the offline queue has been delivered
|
|
WhatsApp signals that the offline queue has been delivered
|
|
|
(`WaEvent::OfflineSyncCompleted`), `on_offline_sync_completed`
|
|
(`WaEvent::OfflineSyncCompleted`), `on_offline_sync_completed`
|
|
|
-(`src/app.rs:374-384`) runs `catch_up` once per process (`catch_up_done`;
|
|
|
|
|
|
|
+(`src/app.rs:567-577`) runs `catch_up` once per process (`catch_up_done`;
|
|
|
later reconnections skip it).
|
|
later reconnections skip it).
|
|
|
|
|
|
|
|
-`catch_up` / `catch_up_chats` (`src/app.rs:386-428`) look at every enabled
|
|
|
|
|
|
|
+`catch_up` / `catch_up_chats` (`src/app.rs:579-626`) look at every enabled
|
|
|
chat — all chats when the global autoreply is on, otherwise those with the
|
|
chat — all chats when the global autoreply is on, otherwise those with the
|
|
|
per-chat flag — and take its last stored message.
|
|
per-chat flag — and take its last stored message.
|
|
|
-`ReplyGate::catch_up_due` (`src/autoreply.rs:284-307`) accepts it only when:
|
|
|
|
|
|
|
+`ReplyGate::catch_up_due` (`src/autoreply.rs:267-290`) accepts it only when:
|
|
|
|
|
|
|
|
- `catch_up_seconds` (default 7200) is positive — `0` disables the catch-up;
|
|
- `catch_up_seconds` (default 7200) is positive — `0` disables the catch-up;
|
|
|
- the chat is not a group;
|
|
- the chat is not a group;
|
|
@@ -178,94 +171,126 @@ per-chat flag — and take its last stored message.
|
|
|
- it passes `meets_content_rules`, so a last message written by the user
|
|
- it passes `meets_content_rules`, so a last message written by the user
|
|
|
(already answered) never qualifies.
|
|
(already answered) never qualifies.
|
|
|
|
|
|
|
|
-Each due chat gets one text-only reply through `start_reply`; when at least
|
|
|
|
|
-one was started the status shows `Risposte di recupero: N`.
|
|
|
|
|
|
|
+Each due chat is marked pending and goes through `prepare_reply` **without**
|
|
|
|
|
+the reply delay, together with the images of its offline messages
|
|
|
|
|
+(`backlog_images`); offline images of other chats are dropped. When at least
|
|
|
|
|
+one reply went ahead the status shows `Risposte di recupero: N`. Offline
|
|
|
|
|
+images that arrive after the catch-up already ran are described on their own,
|
|
|
|
|
+without a reply (`describe_late_image`, `src/app.rs:400-415`).
|
|
|
|
|
+
|
|
|
|
|
+## 5. Image descriptions
|
|
|
|
|
+
|
|
|
|
|
+Images are never sent to the text model. Each image sent by an allowed
|
|
|
|
|
+sender in an enabled chat is kept by `keep_image` (`src/app.rs:378-398`) and,
|
|
|
|
|
+just before the reply that covers it, described **once** by the vision model
|
|
|
|
|
+(`AiClient::describe_image`, request text from `vision.md` or the built-in
|
|
|
|
|
+one, see [AI client](../integrations/ai-client.md)). The description is
|
|
|
|
|
+stored in the `image_descriptions` table and reused in every later context.
|
|
|
|
|
+
|
|
|
|
|
+- `select_images_to_describe` (`src/app.rs:1265-1292`) keeps only the images
|
|
|
|
|
+ whose message is in the reply context and that have neither a stored
|
|
|
|
|
+ description nor one in progress (`describing`), and at most the **5 most
|
|
|
|
|
+ recent** (`MAX_IMAGES_PER_REPLY`); older images of the same batch are
|
|
|
|
|
+ dropped for good, to cap the cost of albums. Without `openai_vision_model`
|
|
|
|
|
+ nothing is described (`vision_applies`).
|
|
|
|
|
+- `spawn_image_descriptions` (`src/app.rs:1231-1263`) describes all selected
|
|
|
|
|
+ images in parallel. Each download (`download_image_with_retry`,
|
|
|
|
|
+ `src/app.rs:1336-1362`) makes up to 3 attempts, each abandoned after 60 s,
|
|
|
|
|
+ waiting 2 s × attempt between them; the bytes stay in memory.
|
|
|
|
|
+- `handle_images_described` (`src/app.rs:465-497`) stores each description
|
|
|
|
|
+ whose message still exists (`History::message_exists`), so a chat deleted
|
|
|
|
|
+ meanwhile gets no orphan rows. Failed images stay without description, the
|
|
|
|
|
+ others are unaffected, and the status shows
|
|
|
|
|
+ `Descrizione immagine fallita (N): <first error>`. If the chat is still
|
|
|
|
|
+ pending, the reply then starts (re-checking `reply_context`).
|
|
|
|
|
|
|
|
## 6. Building the context
|
|
## 6. Building the context
|
|
|
|
|
|
|
|
-`start_reply` loads the context with
|
|
|
|
|
|
|
+`start_reply` (`src/app.rs:645-673`) loads the context with
|
|
|
`History::messages_since(chat_jid, context_start(now), CONTEXT_MESSAGES)`:
|
|
`History::messages_since(chat_jid, context_start(now), CONTEXT_MESSAGES)`:
|
|
|
|
|
|
|
|
-- `context_start` (`src/autoreply.rs:476-497`) is local midnight of
|
|
|
|
|
|
|
+- `context_start` (`src/autoreply.rs:459-468`) is local midnight of
|
|
|
**yesterday** (falls back to `now` minus two days if local time cannot be
|
|
**yesterday** (falls back to `now` minus two days if local time cannot be
|
|
|
resolved);
|
|
resolved);
|
|
|
- `CONTEXT_MESSAGES` = 200 (`src/autoreply.rs:13-15`) caps the window,
|
|
- `CONTEXT_MESSAGES` = 200 (`src/autoreply.rs:13-15`) caps the window,
|
|
|
keeping the most recent messages in ascending order
|
|
keeping the most recent messages in ascending order
|
|
|
([History store](../concepts/history-store.md)).
|
|
([History store](../concepts/history-store.md)).
|
|
|
|
|
|
|
|
-`build_turns(system_prompt, messages, context)`
|
|
|
|
|
-(`src/autoreply.rs:499-526`):
|
|
|
|
|
-
|
|
|
|
|
-- first a `System` turn: the configured prompt followed by a role note
|
|
|
|
|
- (`role_note`, `src/autoreply.rs:20-27`, `src/autoreply.rs:467-474`).
|
|
|
|
|
- `ROLE_NOTE_CHAT` tells the model that `assistant` messages were written by
|
|
|
|
|
- the account owner it answers for and `user` messages by the contact or,
|
|
|
|
|
- in groups, by the participants named in front; `ROLE_NOTE_SELF_CHAT`
|
|
|
|
|
- says that in the self-chat `user` is the owner and `assistant` its own
|
|
|
|
|
- previous replies;
|
|
|
|
|
-- then the last `CONTEXT_MESSAGES` messages, skipping blank texts; the last
|
|
|
|
|
- one is the newest message of the chat;
|
|
|
|
|
-- roles (`turn_role`, `src/autoreply.rs:367-377`): in normal chats and
|
|
|
|
|
|
|
+It records the last context message in `answered_up_to`, replaces every
|
|
|
|
|
+`@<own number or LID>` in the texts with `@[te]` (`mark_own_mentions` /
|
|
|
|
|
+`replace_mention`, `src/autoreply.rs:159-170`, `src/autoreply.rs:438-457`;
|
|
|
|
|
+a longer number is never matched) and calls
|
|
|
|
|
+`build_turns(system_prompt, messages, TurnContext)`
|
|
|
|
|
+(`src/autoreply.rs:482-503`) with the chat name and the stored image
|
|
|
|
|
+descriptions:
|
|
|
|
|
+
|
|
|
|
|
+- first a `System` turn (`system_turn_content`,
|
|
|
|
|
+ `src/autoreply.rs:420-436`): the `SOUL.md` prompt as written, then — in
|
|
|
|
|
+ normal chats and groups — a line `Chat privata con «name».` or
|
|
|
|
|
+ `Chat di gruppo «name».` and `ROLE_NOTE_CHAT` (`assistant` messages were
|
|
|
|
|
+ written by the account owner it answers for, `user` messages by the
|
|
|
|
|
+ contact or, in groups, by the participants named in front); in the
|
|
|
|
|
+ self-chat only `ROLE_NOTE_SELF_CHAT` (`user` is the owner, `assistant` its
|
|
|
|
|
+ own previous replies). Parts are separated by a blank line;
|
|
|
|
|
+- then the last `CONTEXT_MESSAGES` messages, skipping blank texts;
|
|
|
|
|
+- roles (`turn_role`, `src/autoreply.rs:311-321`): in normal chats and
|
|
|
groups own messages are `Assistant`; in the self-chat, where every message
|
|
groups own messages are `Assistant`; in the self-chat, where every message
|
|
|
is `from_me`, only `is_ai` messages are `Assistant`;
|
|
is `from_me`, only `is_ai` messages are `Assistant`;
|
|
|
- in groups every `User` turn is prefixed with `"<sender>: "`, where the
|
|
- in groups every `User` turn is prefixed with `"<sender>: "`, where the
|
|
|
sender is the trimmed push name or else the user part of the sender JID
|
|
sender is the trimmed push name or else the user part of the sender JID
|
|
|
- (`sender_label`, `message_to_turn`, `src/autoreply.rs:427-465`);
|
|
|
|
|
-- only `User` turns are rewritten (`src/autoreply.rs:388-425`):
|
|
|
|
|
- - voice note: the triggering one becomes an instruction to reply
|
|
|
|
|
- courteously that it cannot be listened to and ask to write it; older ones
|
|
|
|
|
- become a short history note;
|
|
|
|
|
- - image: the triggering one becomes "reply to the attached image" when
|
|
|
|
|
- vision is used, otherwise an instruction to say it cannot be seen and
|
|
|
|
|
- ask for a description; older ones become a history note; a caption is
|
|
|
|
|
- appended as `Didascalia: ...`;
|
|
|
|
|
|
|
+ (`sender_label`, `message_to_turn`, `src/autoreply.rs:381-418`);
|
|
|
|
|
+- only `User` turns are rewritten (`user_content`,
|
|
|
|
|
+ `src/autoreply.rs:364-379`), with plain facts — how to react is left to
|
|
|
|
|
+ `SOUL.md`:
|
|
|
|
|
+ - voice note → `[Il contatto ha inviato un messaggio vocale che non puoi ascoltare]`;
|
|
|
|
|
+ - image with a stored description →
|
|
|
|
|
+ `[Il contatto ha inviato un'immagine. Descrizione: <text>]`, without one →
|
|
|
|
|
+ `[Il contatto ha inviato un'immagine che non puoi vedere]`; a caption is
|
|
|
|
|
+ appended as `\nDidascalia: <caption>`;
|
|
|
- any other text is kept as is.
|
|
- any other text is kept as is.
|
|
|
|
|
|
|
|
-`TurnContext { self_chat, vision }` (`src/autoreply.rs:379-386`) carries
|
|
|
|
|
-the two switches.
|
|
|
|
|
|
|
+## 7. Request, `no_reply` and retries
|
|
|
|
|
|
|
|
-## 7. Vision path, fallback and retries
|
|
|
|
|
|
|
+`spawn_ai_request` (`src/app.rs:1211-1229`) calls `AiClient::complete`, which
|
|
|
|
|
+offers the model a single parameterless tool, `no_reply`. When the model
|
|
|
|
|
+calls it, the outcome is `Reply::NoReply`: nothing is sent and the status
|
|
|
|
|
+shows `AI: nessuna risposta necessaria`. The app never says when to use it —
|
|
|
|
|
+the cases belong in `SOUL.md`.
|
|
|
|
|
|
|
|
-The image is kept only for a live triggering image when a vision model is
|
|
|
|
|
-configured (`vision_applies`); deferred and catch-up replies never carry
|
|
|
|
|
-one. In that case `spawn_reply` (`src/app.rs:459-490`) builds two turn
|
|
|
|
|
-lists — with `vision: true` and, as fallback, `vision: false` — and runs
|
|
|
|
|
-one background task that:
|
|
|
|
|
-
|
|
|
|
|
-1. downloads the image into memory via the
|
|
|
|
|
- [WhatsApp client](../integrations/whatsapp-client.md);
|
|
|
|
|
-2. calls `complete_with_image` with the vision model;
|
|
|
|
|
-3. on any failure, tries plain `complete` using the fallback turns
|
|
|
|
|
- (the courtesy "cannot see it" reply).
|
|
|
|
|
-
|
|
|
|
|
-If both fail, the error combines both messages
|
|
|
|
|
-(`<vision error>; Fallback cortesia fallito: <error>`,
|
|
|
|
|
-`src/app.rs:1035-1064`). Without a usable image, a plain text request is
|
|
|
|
|
-spawned.
|
|
|
|
|
-
|
|
|
|
|
-Every AI call (text, vision and the courtesy fallback) is retried by the
|
|
|
|
|
|
|
+Every AI call (text completion and image description) is retried by the
|
|
|
[AI client](../integrations/ai-client.md) up to `ai_retries` times (default
|
|
[AI client](../integrations/ai-client.md) up to `ai_retries` times (default
|
|
|
2, waiting 2 s, 4 s, …) on transient errors — HTTP 429, 5xx, timeouts,
|
|
2, waiting 2 s, 4 s, …) on transient errors — HTTP 429, 5xx, timeouts,
|
|
|
transport errors, malformed or empty replies — while other 4xx errors fail
|
|
transport errors, malformed or empty replies — while other 4xx errors fail
|
|
|
at once. A reply can therefore arrive noticeably later than the trigger;
|
|
at once. A reply can therefore arrive noticeably later than the trigger;
|
|
|
-the "still enabled" check in step 8 happens after all retries. When
|
|
|
|
|
-`ai_log = true`, every attempt is also appended to `ai_requests.log`.
|
|
|
|
|
|
|
+the checks in §8 happen after all retries. The request task is awaited by a
|
|
|
|
|
+second task, so `AiReply` is always delivered and the chat never stays
|
|
|
|
|
+pending. When `ai_log = true`, every attempt is also appended to
|
|
|
|
|
+`ai_requests.log`.
|
|
|
|
|
|
|
|
## 8. Sending
|
|
## 8. Sending
|
|
|
|
|
|
|
|
-`AppEvent::AiReply` is handled by `send_ai_reply_if_enabled`
|
|
|
|
|
-(`src/app.rs:519-534`), which re-checks `autoreply_enabled`: if both the
|
|
|
|
|
-global and the per-chat switch are off by then, the reply is discarded with
|
|
|
|
|
-status `Risposta AI scartata: autoreply disattivato`. Otherwise the text is
|
|
|
|
|
-sent with `spawn_send(..., is_ai = true)`; on success `handle_sent`
|
|
|
|
|
-(`src/app.rs:536-564`) stores it as an own message with `is_ai = true`,
|
|
|
|
|
-which keeps it from re-triggering the self-chat. Errors appear in the status
|
|
|
|
|
-line (`Errore AI: ...`, `Invio fallito: ...`).
|
|
|
|
|
|
|
+A text reply is handled by `send_ai_reply_if_enabled`
|
|
|
|
|
+(`src/app.rs:703-722`), which re-checks two things:
|
|
|
|
|
+
|
|
|
|
|
+- `autoreply_enabled`: if both the global and the per-chat switch are off by
|
|
|
|
|
+ then, the reply is discarded with status
|
|
|
|
|
+ `Risposta AI scartata: autoreply disattivato`;
|
|
|
|
|
+- `owner_answered_meanwhile` (`src/app.rs:724-735`): if a hand-written owner
|
|
|
|
|
+ message follows the last message of the reply context
|
|
|
|
|
+ (`manual_reply_after`), the reply is discarded with
|
|
|
|
|
+ `Risposta AI scartata: hai già risposto`.
|
|
|
|
|
+
|
|
|
|
|
+Otherwise the text is sent with `spawn_send(..., is_ai = true)`; on success
|
|
|
|
|
+`handle_sent` (`src/app.rs:737-765`) stores it as an own message with
|
|
|
|
|
+`is_ai = true`, which keeps it from re-triggering the self-chat and from
|
|
|
|
|
+counting as a manual answer. Errors appear in the status line
|
|
|
|
|
+(`Errore AI: ...`, `Invio fallito: ...`).
|
|
|
|
|
|
|
|
## Tests
|
|
## Tests
|
|
|
|
|
|
|
|
-The rules, cooldowns, deferral, catch-up and turn building are covered by
|
|
|
|
|
-77 tests in `src/autoreply.rs:528+`; enable rules, `web_chats` and vision
|
|
|
|
|
-applicability by `src/app.rs:1167+`; the global flag storage by
|
|
|
|
|
-`src/history.rs`; the retry policy by `src/ai.rs`. See
|
|
|
|
|
-[Unit tests](../testing/unit-tests.md).
|
|
|
|
|
|
|
+The gate rules, pending tracking, catch-up, mention marking and turn building
|
|
|
|
|
+are covered by `src/autoreply.rs:505+`; enable rules, `web_chats`, image
|
|
|
|
|
+selection, `arrived_after`, `manual_reply_after` and the description failure
|
|
|
|
|
+status by `src/app.rs:1443+`; image descriptions and the global flag storage
|
|
|
|
|
+by `src/history.rs`; the `no_reply` tool, descriptions and the retry policy by
|
|
|
|
|
+`src/ai.rs`. See [Unit tests](../testing/unit-tests.md).
|