How to evaluate coding QA AI without starting with production PHI
A useful first evaluation can test workflow logic, evidence links and reviewer control before a vendor receives protected health information. Real-data access should be a later, explicit decision - not the price of seeing a demo.
Separate four different stages
Discovery, synthetic demonstration, de-identified evaluation and production processing are different operating states. Treating them as one step makes it harder to define who can see data, what is retained and what decision the system is allowed to influence.
Discovery: map the workflow, roles, baseline and expensive failures without document content.
Synthetic test: demonstrate inputs, evidence coordinates, exceptions and approval using fictional records.
Controlled evaluation: use data approved by the customer under agreed access, hosting, retention and deletion terms.
Production: proceed only after security, legal, clinical or coding responsibility and monitoring have been addressed for the use case.
Do not use “de-identified” as a casual label
The U.S. HHS guidance describes two HIPAA Privacy Rule methods for de-identification: Expert Determination and Safe Harbor. A vendor should not decide informally that a sample is safe because obvious names were removed. The organization responsible for the data should make and document the applicable determination, with legal or privacy expertise where needed.
What can be validated with synthetic data
Whether every material finding points to a visible source passage.
Whether missing, conflicting or low-confidence evidence becomes an explicit exception.
Whether reviewers can accept, reject, edit or defer a finding.
Whether the output contract supports the intended downstream workflow.
Whether the team can define acceptance criteria before viewing model results.
The decision gate before real data
Expand access only when the synthetic evaluation identifies a credible use case and the parties have agreed data flow, purpose, minimum necessary fields, hosting, access, logging, retention, deletion and incident responsibilities. This is an operating checklist, not a compliance certification.