1 Choose a task with a known answer
Pick a question you can check against a signed agreement or an approved company record. Say what you want back and which places are involved. Keep it small enough that one person can read every source and every decision.
2 Ask which records the product uses
A good answer names the company facts and documents it used. Ask what happens when one is missing. Does the product ask for it, or carry on without it?
3 Test conflicting records
Put an older draft and the current signed agreement in the same test. Ask which one the product used, and why. Your reviewer should be able to say why one version wins.
4 Read the cited source yourself
Check that the cited clause or record exists and says what the answer says. A link is not proof. Ask the reviewer to point to the exact words that support the proposed action.
A citation has to say what the answer says.
5 Separate training from review of your task
Ask what “trained by lawyers” means. Then ask which of your outputs a lawyer reviews, when, and which work waits for one. Those are different questions. A model trained with lawyer feedback is not a model whose every answer a lawyer checked.
6 Ask what happens when facts are missing
The product should show the gap. Does it ask the right person, pause where it must, and record the answer? Do not score an unfinished task as done because the draft looks finished.
7 Test who approves an exception
Pick a business term above a named person's limit. Ask who receives it and what happens before approval. A business approver and a lawyer decide different things. The test should keep both.
8 Confirm what each channel supports
Test email and Slack separately. Does the draft wait at Send, get a warning, or get reviewed afterwards? For calls, ask what the speaker sees and what everyone else is told. Spoken words cannot be held.
9 Inspect the record after the decision
Look for the excerpt, the source, the decision, and the reason. Ask who can see that record, how long it is kept, and how it fits your legal-hold process. Ask separately about data the product reads but does not keep.
10 Ask for current security materials
Ask about encryption, who can access what, how company data is used, support access, export, deletion, and when backups expire. For calls: capture, notice and retention. For certifications, hosting locations and contract terms, ask for current evidence, not a promise.
11 Check the price against the task
Ask what is charged, when you approve the cost, and which fees sit outside the plan. Ask what happens when the included hours run out. Compare the work and review included, not just the rate.
12 Agree the counting method before testing
For FineShield, agree what counts before you start: issues fixed before sending, review time saved, time from flag to decision, flags that turned out to be nothing, and coverage. Name who checks each count and which team, channels and dates are in.
Count what you saw, not the lawsuit you avoided
A flagged promise does not prove a lawsuit was avoided. Report the fix you saw. If you estimate money, state the assumptions and keep it apart from measured review time, product cost and setup cost.
Write the finding with its limits
Write down the task, sources, versions, scope, dates, reviewer, outputs, and open questions. Say what was left out of the test. The next buyer should know exactly what the result supports.
- Task
- Sources
- Versions
- Scope
- Dates
- Reviewer
- Outputs
- Unresolved questions
- Work excluded