Route a request to the right tool

What the router will do

The router suggests a handler; it does not execute a refund or change an order. “Where is my parcel?” belongs with shipping. “I was charged twice” belongs with billing. A request that mixes a late delivery with a refund may need review. Write that policy before tuning the question.

Run the local example

Download the ZIP above, unzip it and open a terminal in that folder. You need Python 3.10 or later; there are no packages to install. dry-run writes request payloads. demo reads the supplied, hand-written responses and runs the application logic. Use a new output folder for each run.

python jev_lab.py --mode dry-run --out requests
python jev_lab.py --mode demo --out demo-run

Open the six test messages

Read samples.json. It includes delivery, payment and exchange requests, plus a mixed request, a poem request and an attempt to name an unregistered tool. fixture-responses.json supplies the answers for demo mode. The mixed request deliberately gets low confidence so you can exercise the review branch.

Choose a handler or ask for review

Reject a category that is absent from the handler list. Send other and confidence below the cutoff to human_review. Otherwise suggest the selected handler. The demo uses 0.7 to make the branch easy to inspect; choose your own cutoff using labeled messages before a live rollout.

if label not in allowed_handlers:
    raise ResponseError('Unknown category')
if label == 'other' or answer['confidence'] < cutoff:
    return {'action': 'human_review'}
return {'action': 'suggest_handler', 'handler': label}

Check three suggestions and three reviews

In demo-run/results.json, the six fixtures should produce three handler suggestions and three reviews. Now run the command below with a cutoff of 0.2. The overlap case changes from review to shipping. You are changing application policy while holding the supplied answer fixed.

python jev_lab.py --mode demo --confidence 0.2 --out lower-cutoff
# Compare overlap in demo-run/results.json and lower-cutoff/results.json.

Try your own messages with the API

Copy the structure of samples.json into own-data.json and replace the examples and expected labels. Set TYPESAFE_API_KEY in your terminal, then run the command below. Use --model with an available version identifier to repeat an experiment against the same model. Check failures.json before interpreting the results.

python jev_lab.py --mode live --samples own-data.json --out live-run

Measure the work the router creates

Compare with your current routing rules on new, labeled messages. Count wrong handlers and the number sent to review. Looking only at accuracy can hide a router that reviews almost everything. Before connecting actions, keep permission checks and the handler allowlist in application code.