CLI fails locally
A real failing fixture run exits non-zero and shows the exact check that broke.
v0.1.0 - open source - MIT
Fail the demo pull request
when the LLM regresses.
Aici is a small CLI and GitHub Action that runs your prompts, JSON contracts, and tool calls against a few test cases, then exits non-zero when anything drifts. No dashboard. No tracing proxy. No sales call. It works the way npm test works.
npx @mgicloud/aici init --config aici.yml
ok10-minute setup
okruns in your repo
okno account required
PR #142
main
prompt/refund-tone
aici - 4 checks
refund-policy-json
JSON schema valid - contains "approved"
order-lookup-tool-call
expected tool lookup_order, model returned plain text
support-classifier-contains
all 6 fixtures returned a known label
cost-budget
run cost $0.087 exceeded maxCostUsd: 0.05
2 failed - merge blocked
posted by aici-bot - 4s ago
how it works
Edit one line. Watch the PR turn red.
Aici is just a YAML file and a CLI. Try a change below to see the same workflow you would run in .github/workflows/aici.yml, but live on this page.
edit:
aici.yml
no changes
run:
aici run
exit 0
product proof
The failure state is the product.
The launch page now carries concrete product states: local CLI failure, pull-request failure, and the HTML report artifact a developer gets after a run.
PR gets blocked
The GitHub Action turns prompt regressions into normal pull-request evidence.
Report artifact
The same run writes Markdown, JSON, and HTML reports for review and CI artifacts.
is this for you?
Aici is for one specific person. Probably you. Maybe not.
Better to know fit before you install. Aici is intentionally a release gate, not a general AI platform.
You will feel at home if yes
- +You ship LLM features in a real repo, on real branches, through pull requests.
- +You have watched a prompt edit silently break a JSON field downstream.
- +You already trust CI to gate your code. AI behavior should clear the same bar.
- +You want a YAML file in your repo, not another dashboard tab.
- +Your team is small enough that platform migration is a real cost.
Look elsewhere if no
- -You need full production tracing of every LLM call. Use a tracing platform.
- -You want non-engineers to label outputs in annotation queues.
- -You want hosted prompt management with versioning and rollout controls.
- -You need enterprise governance, SSO, audit logs, or compliance claims in v1.
- -You do not write tests for normal code. Aici will not fix that culture.
what we do not do
The boring promises that make this safe to install on a Friday.
Most AI quality tools want to sit between you and your model. Aici sits next to your test runner instead.
01 / code
Open source. MIT.
The CLI, GitHub Action, YAML schema, examples, and templates are in one repo you can inspect and fork.
02 / data
No prompt storage.
Your prompts, fixtures, and outputs stay in your repo and CI. Aici has no hosted prompt database in v0.1.
03 / network
No proxy. No SDK rewrite.
Live checks call OpenAI or Anthropic with your key during a run. Production traffic does not pass through Aici.
04 / price
Free is the real product.
The CLI and Action are free. Hosted history may become paid later, but the release gate is not a fake free tier.
05 / sales
No book-a-demo wall.
There is a README, npm package, GitHub Action, examples, and docs. You can evaluate it without a call.
06 / scope
One job.
Catch LLM regressions before deploy. Hosted tracing, annotation, and generation are deliberately out of v1.
07 / exit
Easy to leave.
Delete aici.yml and the workflow file. No data export or offboarding email.
08 / model
Provider-light.
OpenAI, Anthropic, and OpenAI-compatible endpoints are supported without coupling tests to one vendor.
packaging
Free is the whole v0.1 product. Paid is future convenience.
You can ship Aici in production CI without paying. Future paid surfaces should add hosted memory and team visibility, not take away the local release gate.
Open source
EUR 0/ forever
- CLI and GitHub Action
- JSON Schema, text, regex, tool-call, cost, and latency checks
- Markdown, JSON, and HTML reports
- Template packs for support, extraction, and agents
- OpenAI, Anthropic, and OpenAI-compatible providers
Hosted history
planned/ not sold yet
- Run history across pull requests
- Pass rate, cost, and latency trends
- Flake detection for unstable checks
- Private projects if demand exists
- Built only after user validation
Team controls
later/ demand-gated
- Shared projects across a team
- Slack or email regression alerts
- Larger fixture suites
- Policy presets for sensitive repos
- No enterprise claims in v0.1
try it now
Add a release gate to your LLM in the time it takes to read this page.
Initialize a config, commit the workflow, push a branch. The next time someone changes the prompt, the PR tells them what broke before users do.
npx @mgicloud/aici init && npx @mgicloud/aici run
okexits non-zero on regression
okwrites Markdown report
okworks offline with mock fixtures