United StatesProfessional services2024 – ongoingDDI
Automated scoring of leadership behavioral simulations
DDI cut simulation report time from 48 hours to 10 seconds by automating the scoring of leadership assessments
Problem
DDI assesses leaders with behavioral simulations, in which candidates respond to work scenarios. Human assessors scored the responses, and a report typically took 24 to 48 hours. DDI wanted faster and cheaper evaluation.
Approach
DDI tested prompt engineering with GPT-4, including few-shot, chain-of-thought and self-ask prompts. It then used DSPy for prompt optimization and instruction fine-tuning of Llama3-8b on the Databricks platform. MLflow tracked the experiments, and the models were deployed as serving endpoints under Unity Catalog governance.
Results
Takeaway
Scoring work that depended on trained human assessors became a task a model could finish in seconds once prompts were tuned and a model was fine-tuned. The page gives no figure for scoring accuracy against human assessors.
Categories
- Industry
- Professional services
- Business function
- Internal work and knowledge management
- Innovation type
- Process innovationProduct and service innovation
- AI technology
- Generative AI
- Adoption
- Off-the-shelf tool or platform
- Evidence
- Vendor case study
Sources
Source summaries
Databricks describes how DDI, a leadership development and assessment firm, automated the scoring of behavioral simulations with prompt engineering and a fine-tuned Llama3-8b model. It reports a simulation report time of 10 seconds against 48 hours before.
Full details are in the original sources.