開口點KAIKŌ

Case study · Voice ordering

KAIKŌ 開口點

Multi-tenant voice ordering: customers order by talking, merchants set up a store in minutes.

A web app on Cloudflare Pages Functions and D1. Each store gets its own menu, customer page and kitchen display. The voice waiter runs on OpenAI Realtime, and every change it makes to an order goes through the server first.

Recording

Recorded from a local run of the app with a template store (巷口食堂, “Xiangkou Diner”) and a synthetic customer voice (text-to-speech). The waiter is the real OpenAI Realtime model, speaking Mandarin; the English captions translate both sides and mark each tool call. 90 seconds with sound. Only waits on the recording side (connecting, pauses before each scripted line, page navigation) were trimmed, 8.67 s in total; the model's response times are left as they were.

What it does

Customer scans a QR code, opens the store page

  • Talk to a waiter. A Conversation button starts an OpenAI Realtime voice session. The waiter asks for required options before adding an item; in the recording it asked for the milk tea size first.
  • Fallbacks. Without Realtime, hold-to-talk uses the browser's speech recognition. Typing and tapping the menu always work, on the same order state.
  • A receipt, not a cart. The order is a paper-style ticket; items added by voice carry a mic badge, and quantities stay editable by hand.
  • Checkout and status. Dine-in or takeout, table number, optional discount code. The status page shows the pickup number and progress, and lets the customer cancel while the order is still at “received” or print a receipt.
  • Four languages. Traditional Chinese, Simplified Chinese, English and Japanese, for the interface, the menu and the waiter.

Merchant no account system, one admin key per store

  • Three-step setup. Store name and URL slug, then a menu from a template, pasted CSV or manual entry, then a printable QR code and the store's admin key.
  • Menu editor. Import from a menu photo (needs an LLM key) or CSV, edit option groups and spoken aliases in Chinese and English, toggle items sold out.
  • Kitchen display. New, In progress and Ready columns, refreshed every 4 seconds. Cards show dine-in or takeout, table, a voice tag and a paid or unpaid tag, with Start, Void and Mark paid.
  • Keys and history. The admin key is kept only as a SHA-256 hash; orders keep a snapshot of each line's price, so later menu edits do not change past orders.

How the voice path works

Three input tiers share one order. The model never writes to the ticket directly: it proposes tool calls, the browser hands each one to the server, and only the cart operations the server returns are applied.

KAIKŌ voice ordering data flow Customer input has three tiers: conversation over OpenAI Realtime, hold-to-talk with browser speech recognition, and typing. The browser gets a short-lived client secret from the server, then talks to OpenAI Realtime over WebRTC. Each model tool call goes to the server's realtime-tool endpoint for validation against the menu. Speech-recognition and typed text go to the interpret endpoint, which uses an LLM if a key is set and otherwise a deterministic parser. Validated cart operations update the ticket; on submit the server recomputes prices and the order appears on the kitchen display. CUSTOMER INPUT TICKET → KITCHEN 1 · Conversation OpenAI Realtime over WebRTC half-duplex mic, live captions gpt-realtime-2 2 · Hold to talk browser SpeechRecognition 3 · Type and tap always available if unavailable if unavailable 2 OpenAI Realtime audio both ways + oai-events channel KAIKŌ SERVER 1 POST voice-session mints a 300 s client secret 3 POST realtime-tool checks each tool call against the menu before the ticket changes 5 POST interpret LLM parse if a key is set, checked field by field; otherwise a deterministic zh/en parser 4 Ticket (receipt) changes only on validated cart operations 6 Submit → order prices recomputed by the server Kitchen display polls every 4 s secret KAIKŌ voice ordering data flow Three input tiers: conversation, hold-to-talk and typing. Conversation gets a client secret from the server, talks to OpenAI Realtime, and sends tool calls to the server for validation. Hold-to-talk and typing go to the interpret endpoint, which uses an LLM if a key is set and otherwise a deterministic parser. Both paths update the ticket only with validated cart operations; on submit the server recomputes prices and the order reaches the kitchen display. CUSTOMER INPUT 1 · Talk Realtime 2 · Hold speech API 3 · Type always on 1 voice-session 300 s client secret 2 OpenAI Realtime WebRTC audio + events 3 realtime-tool each tool call checked against the menu 5 interpret LLM parse if a key is set (Claude or gpt-4.1-mini), checked field by field otherwise, or when the daily LLM budget is spent: deterministic zh/en parser 4 Ticket (receipt) changes only on validated cart operations 6 Submit → order prices recomputed by the server Kitchen display polls every 4 s
  1. The browser asks the server for a voice session. The server calls OpenAI's client-secret endpoint and returns a secret that expires after 300 seconds, so the API key never reaches the browser. Minting is rate limited per store and per IP.
  2. The browser connects to OpenAI Realtime over WebRTC with that secret: audio both ways, plus an oai-events data channel that carries transcripts and tool calls.
  3. When the model calls a tool (add_items, set_choice, confirm_order and six others), the browser posts it to the server with the count of customer speech turns heard and the turn's transcript.
  4. The server checks it against the store's menu: the item exists and is not sold out, required options are chosen, allergens the customer named are avoided, and the customer has actually spoken. It returns cart operations, the browser applies them to the ticket, and the result goes back to the model.
  5. Without Realtime, recognised speech or typed text goes to interpret. With a key, an LLM (Claude when its key is set, otherwise gpt-4.1-mini) parses it and the server validates the result; any item id not on the menu discards the whole reply. With no key, a failed check or the daily budget spent, the built-in parser handles Chinese and English.
  6. The voice flow ends by asking the customer to tap Submit; it never submits on its own. The server recomputes every price from the menu, and the order shows up on the kitchen display.
4 / 0
tool calls checked / rejected
1.9–3.7 s
end of customer speech to first waiter audio
118
tests passing in 7 files
4
languages for the interface, menu and waiter

Response times per turn were 3.2, 1.9, 3.7 and 2.3 s, measured on the recording's audio. They include the 600 ms silence window the voice detector waits for, and turns 1 and 3 include a tool call and a second response before the waiter speaks.

Engineering notes

  • The model speaks only when asked

    An earlier build had a waiter that answered nobody in a quiet room. The cause was server-side auto-response: OpenAI's turn detection defaults create_response to true. KAIKŌ sets create_response:false and interrupt_response:false when it mints the session, then sends them again in a session.update once the call is live, because the settings in the token were only partly applied.

    The browser creates a response in three places only: the greeting, each committed customer turn, and each tool result. The mic track is switched off while the waiter is audible and reopened 600 ms after it goes quiet. A “let me talk” button cancels the current response and opens the mic, so interrupting does not depend on echo cancellation.

  • A transcription model that does not invent phrases

    whisper-1 produced stock phrases from near-silence. Input transcription now uses gpt-realtime-whisper with a language hint for the session, plus near-field noise reduction. As a last check, the server refuses add_items when the customer has not spoken yet in the session, or when the turn's transcript has fewer than two meaningful characters.

  • Rate limits and daily budgets

    Fixed-window limits are stored in D1 and set in wrangler.toml and the server code. Voice sessions: 20 a minute per store and 10 per IP, with a global REALTIME_DAILY_BUDGET of 2000 a day. Text parsing: 30 a minute per IP and a global LLM_DAILY_BUDGET of 5000 a day; past the budget it falls back to the local parser instead of failing. Orders: 10 a minute per IP and 120 per store. Creating a store: 5 a minute per IP.

  • The server owns the money

    Prices are recomputed from the menu on every order, and unknown item ids are rejected. If the merchant changes a price or an option while someone is ordering, submit returns a line-by-line diff and asks the customer to confirm again rather than charging the new price.

  • Tests

    118 vitest tests in 7 files pass: parser 38, API 26, tools 24, ask 14, money 11, translate 3, i18n 2. The API tests run the routes against an in-memory D1 stand-in on better-sqlite3 and cover auth, isolation between stores, price recomputation, duplicate orders, invalid state changes and rate limits. Live WebRTC speech cannot run in a unit test, which is why the voice path is shown here as a recording.

Screens

Captured from the same local run. The app's interface was set to Traditional Chinese, matching the recording.

Customer store page for 巷口食堂: menu sections for noodles, rice, sides and drinks with prices, an empty ticket on the right, and a bottom dock with a text button, a hold-to-talk mic and a Conversation button.
01Store page. Menu on the left, an empty ticket on the right, and the input dock: type, hold to talk, or start a conversation.
Mid-conversation: the caption panel shows the customer's transcribed request for bubble milk tea and the waiter asking whether it should be medium or large. The beef noodle soup is already on the ticket at NT$180.
02A voice turn. The customer's line and the waiter's reply as captions. The milk tea needs a size, so the waiter asks before adding it.
Ticket with three voice-added lines, each with a small mic badge: beef noodle soup NT$180, bubble milk tea NT$75, braised egg NT$15, total NT$270. The waiter's caption reads the order back and asks the customer to tap submit.
03Read-back. Three lines added by voice (mic badge), NT$270, and the waiter asking the customer to tap Submit.
Checkout sheet listing the three items and the NT$270 total, with dine-in selected, table number 5, an empty discount code field and a submit button reading NT$270. A note says to pay at the counter.
04Checkout. Dine-in, table 5, an optional discount code, pay at the counter.
Order status receipt: order received, pickup number #001, dine-in table 5, a four-step progress line from received to done, the three items and NT$270. Buttons to cancel the order, print a receipt and order again.
05Order status. Pickup number #001 and its progress. The customer can cancel while it is still at “received”, print a receipt, or order again.
Kitchen display with three columns, New, In progress and Ready. Order #001 sits under New, tagged dine-in, table 5, voice and unpaid, with its three items, a Start button, a Void button and Mark paid.
06Kitchen display, new order. #001 arrives under New, tagged voice and unpaid.
Kitchen display after Start: order #001 has moved to the In progress column and its main button now reads Ready.
07After Start. The order moves to In progress; the next action marks it Ready.
Merchant menu editor: buttons to import from a menu photo with AI and to import CSV, then menu items grouped by section, each with option count, Chinese and English aliases, a price and an available toggle.
08Menu editor. Photo or CSV import, spoken aliases in Chinese and English, and an availability toggle per item.
Setup wizard, step 2 of 3 (build the menu): buttons to load the template menu or fill a CSV example, a CSV text area with columns for section, name, price, aliases and English name, and the loaded template items below.
09Setup, step 2. The template menu loaded; CSV paste and manual entry are on the same step. Steps 1 and 3 are not shown.
The store page on a phone: store name, section tabs, the menu with dotted price leaders, a hint suggesting what to say, and a bottom dock with a text button, a large mic button, a Conversation button and the ticket total.
10On a phone. At 390 px the ticket collapses into the dock, next to the text, hold-to-talk and Conversation buttons. A hint suggests a first sentence to say.