How Do You Evaluate an AI Agency or Vendor Before Signing a Contract?

Quick answer

Before signing, check four things in the contract itself, not just the pitch: who owns the code, model artifacts, and any fine-tuned weights (it should be you, explicitly, not implied), what "done" means (measurable evaluation criteria, not vague deliverable language, what happens if you want to leave) a clear handoff and exit clause, not a system only the agency can maintain, and who's liable for what the AI does in production, especially around data handling and incorrect outputs. A strong portfolio doesn't protect you from a weak contract.

This is the contract-terms sequel to vetting the agency itself

Our agency vetting checklist covers how to evaluate an agency's capability, references, and technical approach before you get to paperwork. This post assumes you've done that and picked a finalist: the question here is narrower and just as important: what does the contract itself need to say before you sign it.

The four contract terms that matter most

1. IP and artifact ownership, stated explicitly. The contract should say, in plain terms, that you own the code, any fine-tuned model weights, prompts, and evaluation datasets produced during the engagement. "Work product" language that doesn't explicitly name model artifacts and prompts can leave ambiguity about whether you actually own the parts of the system that make it work. Get this named specifically, not assumed.

2. A measurable definition of "done." Vague deliverable language ("build a working AI agent") is unenforceable in practice. What does "working" mean, measured how, against what test cases? A contract with an attached evaluation criteria document (accuracy thresholds, specific test scenarios, latency requirements) gives you something concrete to hold the agency to, and gives the agency something concrete to build against instead of guessing at your bar.

3. An exit and handoff clause. Ask directly: if this relationship ends in six months, what do we get, and can we maintain it ourselves? A good contract specifies documentation requirements, a knowledge-transfer period, and confirms the system doesn't depend on the agency's proprietary infrastructure to keep running. If the answer is vague, that's a sign the agency's business model depends on lock-in, not on the system being genuinely yours.

4. Liability and data-handling terms. If the agent will touch customer data, the contract should specify who's responsible for data handling compliance, what happens if the AI produces a harmful or incorrect output in production, and what security standards the agency's own development process follows. Our security guardrails post covers the technical side of this: the contract should require those guardrails exist, not just hope the agency builds them in.

Red flags in a proposed contract

• "Work product" language with no explicit mention of model artifacts, prompts, or fine-tuned weights.
• No attached evaluation criteria: just a narrative description of the deliverable.
• No handoff or documentation clause, or one that's vague about what "documentation" includes.
• Ongoing dependency on the agency's proprietary tooling to run the system, with no path to run it independently.
• No mention of data handling, security, or liability, especially for a project touching customer or regulated data.

What a strong contract actually protects you from

Not just cost overruns: the more expensive failure mode is a system that ships, works well enough to pass a demo, and then can't be maintained, extended, or even fully understood once the agency's engagement ends. A contract that nails ownership, evaluation criteria, and handoff terms is what prevents that outcome; a contract that's silent on those points leaves you exposed even if the initial build goes well.

How this fits the build-vs-buy decision overall

If you're still deciding whether to build in-house instead of hiring an agency at all, our build-in-house checklist covers that upstream decision: the contract terms above matter most when the answer is "bring in outside help," which is the more common path for a first AI build.

FAQ

Should we get a lawyer to review an AI development contract specifically? Yes, ideally one with some familiarity with software/IP contracts generally: the specific AI-related additions (model artifact ownership, evaluation criteria) are new enough that a generic services contract template often misses them.

What if the agency insists on keeping the model weights or fine-tuned artifacts? That's a significant red flag for a customer-facing or business-critical system. It means your product's core capability lives outside your control, and you should treat it as a dealbreaker unless there's a specific, well-understood reason (e.g., a shared foundation model they license, not something built specifically for you).

How detailed should the evaluation criteria be before signing? Detailed enough that both sides would agree, without argument, whether a given test case passed or failed. Vague criteria like "high quality responses" aren't enforceable and lead to disputes at delivery.

Is a shorter initial contract with an option to extend a good idea? Often yes: a smaller first engagement with clear success criteria, followed by an extension decision based on real results, reduces risk on both sides compared to a large upfront commitment based on a pitch alone.

Accelate's own contracts lead with explicit IP ownership, attached evaluation criteria, and a documented handoff plan: the same terms we'd tell a prospective client to insist on from anyone else.