The AI Practicum · Level 0

Measure what you actually get

Continue

0.1 What the productivity studies actually measured

Level 0 · 21 min · 2 knowledge checks

0 of 10 gates cleared0 of 17 visible items read

You already use AI. This level is not about starting. It is about finding out what your current use actually produces, on your own recurring task, judged by somebody other than you.

Three counts, taken this week and again in week eight. What proportion of the model's output you kept unchanged. What your judge says about the result. And how many claims in it you checked.

Everything else in the course is judged against the change in those three numbers.

The reason for measuring rather than reflecting is in this level. Sixteen experienced developers, working in code they had known for five years, believed AI had made them 20 per cent faster after it had made them 19 per cent slower. They had the recordings. They still could not see it.

Level 0 lessons

0 of 5 done

Take the two numbers this course is judged against, and find out that you cannot read either of them off your own experience.

0.1What the productivity studies actually measured21 min 0.2The jagged frontier19 min 0.3The three counts17 min 0.4Why your own judgement will not do19 min 0.5When two is worse than one20 min

Level 0 exercises

0 of 2 done

Done against a real organisation you choose, not a case study. The gates below are about what these produce.

Exercise 0.A - The baseline

Take the three counts on one real piece of AI-assisted work from your named recurring task.

Do the task the way you normally do it. Do not improve your practice for the measurement; the point of a baseline is that it records what you actually do. If you would normally paste and send, paste and send.

Then, before you send it, take the counts. Retention: what proportion of the sentences you are sending are word-for-word what the model produced. Checkable claims: how many statements in the piece could in principle be confirmed against a source. Verified claims: how many you actually opened something to confirm.

Keep the model's original output and your final version, both, in a place you will find them in week eight.

Takes about forty minutes on top of the task itself, most of it counting sentences.

It fails in two ways. Improving your process because you are being measured, which produces a baseline you cannot reproduce. And estimating the retention proportion instead of counting it, which is the judgement this level exists to distrust. If your retention estimate and your count differ by more than ten points, the count is the number.

Exercise 0.B - The outside judge

Give the finished piece to your named judge and ask four questions.

Do not tell them AI was involved, do not describe the task first, and do not send it with a covering note explaining what you were trying to do. Any of those three converts a measurement into a rating.

The questions: what is this saying, in one sentence. What in here would you act on. What in here would you want checked before acting. And what is missing.

Write their answers verbatim before you respond to any of them. Then answer the same four questions yourself, separately, and put both sets away.

Takes about thirty minutes.

The third question is the one that matters and it is the one people forget to ask. A judge who says "this looks fine" has told you about fluency. A judge who names something they would want checked has told you where the work is load-bearing, and Level 4 runs against exactly that list.

Expect your own fourth answer to be shorter than theirs. What is missing is the thing you cannot see, and the consulting experiment is the reason: on the task where correctness fell nineteen points, everything observable about the work improved.

Level 0 gates

0 of 10 cleared

You have one named recurring professional task, fixed for eight weeks, that produces an artefact somebody reads or acts on.

Not started

You have a named judge who normally receives or reviews this kind of work.

Not started

You have named one task where you find out afterwards whether you were right, or recorded that you have none.

Not started

You have written down what you are not permitted to put into a tool your organisation does not control, and where that rule comes from.

Evidence needed: Attach the artefact, or attest with a date if it lives somewhere you cannot upload from.

Not started

You have kept both the model's original output and your final version for one piece of work.

Not started

You can state your retention proportion for that piece, counted rather than estimated.

Not started

You can state how many claims in it were checkable and how many you verified.

Not started

You have your judge's four answers recorded verbatim, given without telling them AI was involved.

Not started

You have your own four answers to the same questions, written separately.

Evidence needed: Attach the artefact, or attest with a date if it lives somewhere you cannot upload from.

Not started

You do not describe your AI results as good or bad on the basis of how the work felt to produce.

Not started

Tools in this level

The three counts

Reads retention, the judge’s answers and your verification count from one real piece of work. It refuses an estimated retention proportion and it will not accept a judge who was told AI was involved.

Open and get it reviewed

The ten statement check

Ten statements, marked true or false about you, before Level 0 and again after Level 5. It reports the individual lines rather than a total, because the total hides the only movement worth reading.

Open and get it reviewed

Desktop only

Drag-and-drop tools stay on a wider screen. Lessons and checks work anywhere.