United StatesPublic sector2025 – ongoingNational Institute of Standards and Technology
NCCoE internal cybersecurity guidance chatbot
NIST tested an internal cybersecurity chatbot with about 100 reference questions and found flaws in automated grading
Problem
Engineers needed to retrieve specific guidance from NCCoE cybersecurity publications. General language-model answers could lack the required detail or supporting references.
Approach
NIST and MITRE contributors built a retrieval-augmented chatbot with page-level citations. NCCoE hosted it internally for engineers' testing and business use. The team evaluated answers using a prepared question set and limited manual review.
Results
Takeaway
Automated grading sometimes awarded credit to empty answers. The team concluded that it could support development but could not replace rigorous human assessment.
Categories
- Industry
- Public sector
- Business function
- Internal work and knowledge management
- Innovation type
- Process innovation
- AI technology
- Generative AI
- Adoption
- Partner co-development
- Evidence
- Company's own claim
Sources
Source summaries
NIST's initial public draft documented an internally deployed prototype. Testing exposed answer limitations and weaknesses in model-based grading; broader human evaluation remained necessary.
Full details are in the original sources.