ThinkActuary

Who marks
the markers?

Developing Automated Marking

2017–2026: making expert-informed
evaluation and feedback more accessible.

Burt Verster   /   Matthew Smith

ThinkActuary

Expert feedback on students’
practice exam answers is
scarce, slow and expensive.

  1. Student answerPractising is possible
  2. Awaiting expert reviewAccess is uneven
  3. Independent feedbackWhat earned credit? How can I improve?
ThinkActuary

2017: a first attempt at automated marking

The first Mark prototype tested whether technology could help.

  1. Candidate answerRepresent the wording
  2. Memorandum pointsCompare using cosine similarity
  3. Suggested marksApply a threshold and add credit
78%of marks within one standardised mark
23pilot attempts
Promising credit estimates, but similar wording can express incorrect reasoning.
ThinkActuary

ThinkActuary connected students with actuaries

Students gained access to independent feedback. We began building a valuable dataset of actuarial judgement, applied through marking.

  1. StudentAttempts a past-paper question
  2. ActuaryMarks the answer
  3. StudentReceives a mark and feedback
>73kcumulative question submissions
>6kactuary reviews
>110actuary markers

Through 30 June 2026.

ThinkActuary

In 2023, GPT-4 caught our ‘attention’

LLMs might offer a path beyond text matching.

01

Could it follow the student’s reasoning?

02

Could it judge what deserved credit?

03

Could it explain how to improve?

Artificial Analysis logo

Artificial Analysis Intelligence Index

Selected models by release date
Artificial Analysis intelligence timeline: OpenAI GPT-3.5 approximately 5, OpenAI GPT-4 approximately 7 highlighted, with OpenAI’s launch claim of “human-level performance on various professional and academic benchmarks”, OpenAI o1 estimated 15, Google Gemini 3.1 Pro 30, Anthropic Claude Opus 5.5 Max 58.

Current index snapshot, plotted by release date. *Approximate / estimated legacy scores.

A capable model wasn’t yet a reliable marker.
ThinkActuary
Hi, I’m Mark · the new markerNew starter

Meet Mark, an enthusiastic grad

Keen to become a marker. Well-read, but new to marking actuarial exams.

Brain The model

OpenAI
Anthropic
Gemini
Grok
Z.ai

Method The harness

123
SKILL.md
SKILL.md
✓ Check
Agent Skills

Marked scripts The evaluation

Actuary marking
QA Score
Confusion
MAE
Stability
The ingredients were there. Now we needed sound marking judgement.
ThinkActuary
Hi, I’m Mark · the new markerIn training

Learning from senior markers

A three-step, human-controlled improvement loop Evaluate Mark against actuary marks on the same cases, with repeat runs to check stability. Diagnose where and why Mark diverges from senior markers, and whether it is a one-off or a pattern. Teach Mark by writing the marking doctrine into its skills and workflow, or upgrade the model. Then repeat the loop. New: the Anthropic Hillclimb Skill. Change one thing at a time. When repeated changes bring no meaningful progress, wait for a stronger model. Every reviewed script is feedback. Every change is checked before it sticks. 01EvaluateScore Mark against actuary markson the same cases. Repeat runsto check stability.SAME CASESREPEAT RUNS 02DiagnoseFind where and why Mark divergesfrom senior markers: a one-off,or a pattern?PATTERNS 03TeachHarness: write the doctrineinto skills and workflow.Model: upgrade when needed.HARNESSMODEL Repeat NEWAnthropic Hillclimb SkillChange one thing at a time.Progress stalled? Wait for a stronger model.
How reliable was the reference we were learning from?
ThinkActuary
Hi, I’m Mark · the new markerIn training

Even senior markers disagree

Our eval set was valuable, but platform reviews showed the human reference wasn’t perfect either.

Mean student rating and auto QA / 5

● Student rating● Auto QA
12345Marker A4.69 / 1.88n=35 / 94Marker B4.59 / 2.29n=17 / 111Marker C5.00 / 2.89n=15 / 63Marker D5.00 / 2.90n=18 / 29Marker E4.75 / 2.91n=16 / 35

2025 Session 2 · Markers with ≥15 student ratings
Auto QA range: 1.88–2.91/5, despite high student ratings.

Final-score distribution

2025 S2Mean 65.1% · n=685
0%Final score100%

Slightly left-skewed, with a longer tail towards lower scores.

Full session cohort; shape alone does not establish bias.

Agreement alone wasn’t enough. We needed a clear, consistent, inspectable method.
ThinkActuary
Hi, I’m Mark · the new markerIn training

We taught Mark a method

The Marking Engine: four stages powered by a versioned marking skillset A candidate answer enters Map and Split, then Award marks, Audit and Grade, and Write feedback. The output is a mark, grade and feedback. Core judgement, exam-level expectations and subject expertise form a shared versioned skillset feeding the marking and audit stages. Evidence rules underpin marking and grading. Candidateanswer Mark + grade+ feedback MARKING ENGINE ONE ANSWER. FOUR STAGES. 01Map & Split 02Award marks ½ 03Audit & Grade Pass 04Write feedback VERSIONED MARKING SKILLSET Evidence rules underpin marking and grading Core judgementSKILL.md Exam-level expectationsSKILL.md Subject expertiseSKILL.md
Powered by Gemini
More than a score: marks, grades and feedback an actuary could check.
ThinkActuary
Hi, I’m Mark · the new markerTrainee (under supervision)

By Feb 2026, Mark was ready for supervised marking

Gemini 3.1 Pro powered all workflow steps. Marking doctrine stabilised,
with decent accuracy and consistency.

71.3%

Pass/fail agreement

72 agreements from 101 scripts.
29 disagreements.

Enough promise to explore

Inspectable proposed marks and feedback supported supervised marking.

Still important errors

Average scores hid individual differences. Context and duplicate credit needed work.

The next step: give actuaries a first pass to inspect and correct.
ThinkActuary

First, help the actuary

Shared guidance and in-line recommendations could help structure marking and feedback.

Existing guidance

Lower awardsHigher awards
Expected improvement

In-line recommendations

Lower awardsHigher awards

Conceptual illustration, not measured results. No same-script, multiple-marker experiment is shown.

The hypothesis: more structured marking and feedback, with the actuary retaining the final decision.
ThinkActuary

Grader brought the engine into the workflow

2026 S1. The engine proposes. The actuary reviews, changes and approves.

  1. Student attemptAnswer and question
  2. Marking Engine
  3. Actuary reviewSegment decisions and final sign-off
  4. Student feedbackApproved mark and guidance

Actuaries stay in control

Inspect each segment’s rationale and confidence. Edit awards, review the overall feedback and approve the result.

A real review decision

“Consider the tax implications if the structure were to change”
Engine: 0 marksActuary: 0.5 marks

Anonymised F103 review, 2026 S1. Record excerpt, not a UI screenshot.

Grader

Engine-supported marking.
Actuary-approved feedback.

grader.thinkactuary.com
ThinkActuary

Early signs were encouraging

The session before Grader (2025 S2) compared with the first session using it (2026 S1).

Final scores

Before GraderMean 65.1% · n=685
0%Final score100%
With GraderMean 53.7% · n=292
0%Final score100%

More concentrated with Grader; different cohorts, not causal evidence.

Written-feedback QA

Mean score / 5

Before Grader · 30 reviews
3.03
With Grader · 30 reviews
4.20

+1.17 points in mean QA.
Same prompt, balanced samples.

Student feedback

Mean voluntary rating / 5

Before Grader · 1,169 ratings
4.5
With Grader · 101 ratings
4.76

Zero-to-five scale.
Before Grader: 2020–25.

Review timing

Median elapsed time

Before GraderWith Grader

Final ownership interval

31 min23 min
2025 S2 · n=682Early S1 · n=66

Early estimate, not active work.

Overall turnaround

5.3 h2.0 h
2025 S2 · n=6352026 S1 · n=276

62% shorter, including queue time.
Post-review QA proxy.

Higher feedback QA is encouraging, but these comparisons do not establish marking accuracy or a causal improvement.
ThinkActuary

Actuaries accepted nine in ten
segment awards unchanged

2026 S1 Grader: paired segment awards, not whole-script decisions.

91.2%

8,683 of 9,517
paired segment awards unchanged

834 changes mattered

Overrides exposed missed credit, over-credit and borderline decisions.

Acceptance needs interpretation

It may reflect good recommendations, human anchoring, or both.

Could the engine already provide useful feedback before an actuary reviewed it?
ThinkActuary

Auto Mark removed the individual review step

Students get feedback in a matter of minutes, at a quarter of the cost. A311 and selected F100 subjects only.

Grader and Auto Mark share the same marking engine Grader: student attempt, marking engine, actuary review and final approval, human-approved feedback. Auto Mark Beta: student attempt, marking engine, no individual actuary review, engine-generated feedback. Grader Auto Mark (Beta) Student attempt Student attempt Marking Engine Marking Engine Actuary review Review and final approval Student feedback Human-approved Student feedback Engine-generated No individual actuary review

Student price, relative to actuary marking

25% Engine-only route75% Additional price for actuary review
75%cheaper with Auto Mark
ThinkActuary

Students rated Auto Mark highly

Voluntary ratings in the same six supported subjects, across different cohorts.

Average student feedback rating / 5

Grader2026 S1 · 41 ratings
4.8
Auto Mark2026 S1 · 169 ratings
4.7
Actuary marking before Grader2020–25 · 1,169 ratings
4.5

2026 S1 beta use

10.8% of submitted marks in supported subjects, per the session report.

Limited scope

A311, F102, F103, F105, F106 and F108. No F200 subjects in this beta cohort.

What still needs proof

Monitor failures and feedback. Compare matched scripts with human review and test repeatability.

Encouraging student-value evidence. Different cohorts and subject weights prevent a superiority claim.
ThinkActuary

Nine years of progress

With the release of Auto Mark (Beta),
we made meaningful progress
on the original objective.

Scarce→Accessible

Students can access engine-generated feedback without individual actuary review.

Slow→Quick

Human sign-off is no longer a prerequisite for returning feedback.

Expensive→Affordable

A lower-priced route: 25% of the standard credit rate before rounding.

ThinkActuary

Beyond marking: helping students learn

Better feedback can support better learning. The next steps are independent evaluation and stronger feedback.

Evaluate with ASSA

Proposed retrospective collaboration using scripts already marked by ASSA examiners. We are also working to understand the impact on pass rates.

Improve the engine with Grader

Use actuary decisions and overrides to refine skills and model choices. Include difficult and borderline cases, and test changes before widening use.

Technology, actuarial judgement, and people willing to help.
ThinkActuary

The people who made it possible

In memory of

Wim Els

Where it all began

His contribution started the journey
that brought us here.

Our sincere thanks to

Paul Lewis
Imraan Mohamed
Siobhain Mahoney
Arne Verster
Kavi Pather

Instrumental in helping us get here.

ThinkActuary

Questions

2017

Expert evaluation and feedback were difficult to access.

Now

Auto Mark (Beta) makes useful feedback more accessible.

Future

Exciting possibilities for personalised learning and the rise of the “Agentic Actuary”.