United KingdomPublic sector2023 – ongoingDepartment for Transport
Consultation Analysis Tool for thematic analysis
DfT's consultation tool scored 0.93 F1 against human-adjusted theme mappings in live pilots
Problem
Manual analysis of free-text consultation responses required substantial staff time. Large consultations could force analysts to work from samples.
Approach
DfT and The Alan Turing Institute developed CAT using Google's Gemini models. It generated themes, which researchers reviewed before the tool classified responses against them. The team evaluated the system using blind benchmarks and live pilots.
Results
Takeaway
The live-pilot comparison used human reviewers who could see and amend the AI classifications. Its score cannot be treated as a blind-test result.
Categories
- Size
- Large enterprise
- Industry
- Public sector
- Business function
- Internal work and knowledge management
- Innovation type
- Process innovation
- AI technology
- Generative AI
- Adoption
- Partner co-development
- Evidence
- Company's own claim
Sources
Source summaries
DfT reported results from live consultations and blind benchmarks. Theme-mapping F1 was 0.93 against human-adjusted pilot classifications and 0.75 in blind evaluation.
Full details are in the original sources.