RSG#333: How To Audit Secret Government Watchlists and Risk Scores

Secret government watchlists and risk scores can influence who gets stopped, searched, questioned, investigated, denied a benefit, or quietly assigned extra scrutiny. Yet the affected person may never see the record, learn the rule, or receive a useful explanation. Nothing says procedural fairness quite like being judged by a spreadsheet nobody admits exists.

Auditing these systems requires reconstructing the machinery around them. This guide explains how to audit government watchlists and risk scores by identifying their data sources, rules, users, vendors, consequences, error controls, and correction processes. The goal is accountability without exposing private information or helping anyone evade lawful screening. Secrecy may protect particular intelligence. It should not become a scented candle placed over every bad database decision.

Understand What You Are Auditing

A watchlist may be an identity database feeding several screening programs, a local gang database, a threat file, or flags inside another system. A risk score is not necessarily artificial intelligence. It may come from a statistical model, vendor formula, point system, or ordinary rules wearing an expensive technology badge.

Keep the layers separate: source information, inclusion criteria, matching or scoring, and the final government action. An error at any layer can travel downstream and acquire official confidence. The FBI says its renamed Threat Screening Center maintains the consolidated federal terrorism watchlist and shares information with screening partners. Still, an airport delay does not prove someone is listed, and unrelated government scores do not automatically belong to that system.

Step by Step Guide

Step One: Define the Government Action

Begin with the consequence you can observe. Record the date, agency, location, notice, questions, delay, denial, referral, or added condition. Save letters, portal messages, screenshots, receipts, and appeal instructions. Begin with the decision. Evidence enjoys arriving before adjectives.

Step Two: Identify the System and Its Owner

Search budgets, meeting records, manuals, audits, procurement files, and job postings for the program name. Record the data owner, users, vendors, former names, and abbreviations. The office that delivered the bad news may not control the underlying data.

Step Three: Find the Legal Authority

Locate the law or written policy authorizing the system and its consequences. Identify the permitted purpose, covered population, decision maker, review schedule, and sharing limits. If the agency invokes public safety but cannot identify authority for the actual use, flag the gap.

Step Four: Read the Privacy Notices

For federal systems, search privacy pages and the Federal Register for a System of Records Notice. The Justice Department Privacy Act overview explains that these notices describe systems retrieving records by a personal identifier. Read the populations, sources, routine uses, retention rules, safeguards, and access procedures. Find Privacy Impact Assessments and later amendments, then compare the disclosed purpose with actual use.

Step Five: Map Every Data Source

Map where each field originates, including government records, tips, interviews, social media, commercial databases, sensors, and partner agencies. Record whether each source is verified, updated, and correctable. A model can perform flawless mathematics on rotten inputs and still produce premium grade nonsense.

Step Six: Trace the Vendor and Contract

Search SAM.gov contracting records and USAspending.gov for the product, vendor, parent company, award number, and subcontractors. Locally, search board packets, bid portals, payment registers, and sole source justifications. Request the contract, statement of work, data rights clauses, validation reports, training materials, and amendments. Proprietary software does not turn public authority into private weather.

Step Seven: Reconstruct Inclusion and Scoring Rules

Request policies describing who may nominate a person, what facts qualify, what thresholds mean, and whether a human reviews the result. Seek field definitions, manuals, decision tables, validation plans, version histories, and change logs. If operational details are protected, request non sensitive portions and aggregate descriptions. Your audit needs accountability rules, not a road map for evasion.

Step Eight: Follow the Score Into the Decision

Determine whether the score informs an official or automatically triggers action. Compare policies with case records, hearing transcripts, and notices. A score may be marketed as advisory while functioning as an order because nobody wants to disagree with the computer.

Step Nine: Test Accuracy and Disparate Impact

Request aggregate data on entries, removals, false matches, overrides, complaints, outcomes, and demographics. Seek validation studies separating false positives from false negatives across relevant groups. The NIST AI Risk Management Framework treats validity, transparency, accountability, privacy, and harmful bias as distinct concerns. Overall accuracy can conceal unequal mistakes.

Step Ten: Audit Retention and Sharing

Identify how long records remain active, what triggers review, and whether corrections reach every recipient. Request retention schedules, purge logs, access logs, sharing agreements, and audits. The California State Auditor found unsupported entries and defective purge dates in its CalGang audit, reminding us that deletion rules matter only when databases obey them.

Step Eleven: Test Notice and Redress

Document the notice, reviewable evidence, appeal official, deadlines, and correction process. For repeated travel screening problems, the DHS Traveler Redress Inquiry Program is the relevant federal route, not a universal watchlist appeal. The Privacy Act may permit some United States citizens and lawful permanent residents to seek access or amendment, but exemptions can restrict those rights.

Step Twelve: File Focused Records Requests

Ask for existing records, not explanations the agency would have to create. Identify the system, office, dates, document types, contract numbers, and policy versions. Seek aggregate statistics, native electronic files, segregable portions, and the exemption applied to each withholding. The official FOIA guide says a request must reasonably describe agency records. Appeal inadequate searches and unsupported secrecy on time.

Step Thirteen: Reproduce and Publish the Audit

Maintain a source log for every claim. Label confirmed facts, comparisons, expert interpretations, and unresolved inferences. Publish your data dictionary, calculations, exclusions, and limitations without exposing private people. The Electronic Frontier Foundation and American Civil Liberties Union offer independent research on transparency and due process. Use it to supplement primary records.

Red Flags That Deserve Closer Attention

Watch for undocumented criteria, missing privacy notices, unexplained sources, unrecorded model changes, permanent consequences from temporary information, repeated false matches, inaccessible appeals, and corrections that never reach partner agencies. Pay special attention when the agency cannot state how many people are affected or how the tool performs after deployment. A dashboard with three green circles is office decor with electricity.

What Secrecy Does and Does Not Prove

Agencies may lawfully withhold classified information, active investigative material, protected methods, or private data. Withholding does not prove the system is accurate or authorized. Request policies, aggregates, audits, contracts, retention rules, and segregable portions. Never publish names merely because they appear in a leaked or disputed database. Inclusion is not proof of wrongdoing, and accountability reporting should not recreate the harm it investigates.

Turning Findings Into Accountability

Present the audit as a chain from source data to inclusion, score, action, notice, appeal, correction, and deletion. Send findings to the agency privacy office, inspector general, oversight body, public defender, civil liberties group, or independent newsroom. Ask for independent validation, meaningful notice, correction deadlines, review dates, vendor transparency, and public performance reports. A misspelled name needs correction. A structurally biased process may need suspension.

Closing RK Thoughts

Secret government watchlists and risk scores gain power from fragmentation. One office supplies data. Another buys software. A third applies the score. A fourth delivers the consequence. Each points toward the next building until accountability develops shin splints.

A serious audit reconnects those pieces and tests whether mistakes can be fixed before they become permanent government folklore. Resistance Kitty does not expect every sensitive record to be public. She does expect the government to prove that secret systems are lawful, tested, reviewable, and capable of recognizing when they are wrong.

RSG#333: How To Audit Secret Government Watchlists and Risk Scores

Understand What You Are Auditing