Test-Retest Validation of using Skills to Produce Decision Models

Purpose and scope

The purpose of the HEmodelR Skill is to be able to reproduce models from natural language descriptions of their design and parameterization. I developed this Skill through repeated replications of publicly available documentation of decision models used in HTA decision-making. During each replication exercise, I fed information and syntax back into the Skill. This expanded the Skill’s capabilities while also ensuring that a consistent set of methods and assumptions are used across models.

LLMs are inherently probabilistic. Given the same set of prompts and the same information to work from, AI is likely to produce outputs that are similar but nevertheless different. The magnitude of these differences scales with the complexity of the task. Skills, a feature of Claude AI, allow users to provide Claude with specific step-by-step instructions to guide its responses. In so doing, the differences are reduced.

HEmodelR should therefore be able, in theory, to create consistent models using the same natural language description. While the Skill was built with replication in mind, there are obvious useful applications for a Skill that can also do de novo model generation.

This blog post describes an exercise in which I used the HEmodelR Skill to duplicate the Disease Analogue Models.

Approach

In the documentation of each of the Disease Analogue Models there is a fictionalized cost-effectiveness report. These reports, also generated by Claude, are meant to replicate the kind of information that a submitter would include in their economic evidence dossier. There are descriptions of each parameter, as well as the logic of the model structure and assumptions. These reports are written with the aim of making the models reproducible by others who are conducting their own experiments in AI model generation.

I removed the results from these reports. Then I uploaded them back into Claude and asked it to use the HEmodelR Skill to generate models that match what is described in the reports’ methods sections. I then compared the deterministic results of each duplicated model against its original. Results were disaggregated by cost parameter and health state.

Important caveat

The original models adjusted cohort survival for background all-cause mortality. The models did not use a consistent dataset - an approximation of the mortality data was carried over from the replication exercise. This data was not published in the Original reports, and thus was not available for the test-retest duplication.

To address this issue, I asked Claude to reparameterize the Disease Analogue models to use a consistent definition of background mortality. This created a like-for-like comparison of any other differences in the Duplicate model output.

The remaining differences were found by reading the code and the reports side by side, and their effects were measured by attribution. Starting from the recalculated Original, the differences were applied one at a time, cumulatively, until the model reproduced the Duplicate.

Results

A Shiny app is available in the associated Github repository.

CAIS

The Original lets infection be acquired from the stable exacerbation state and lets patients in the infected exacerbation states clear back to stable states; the Duplicate allows neither, and caps death probabilities at 1 rather than 0.999. This accounts for essentially all of the gap: +0.066 and +0.031 life years, +0.101 and +0.047 QALYs, and costs lower by $3,720 and $1,625 for usual care and Infectaguard. The cost reduction sits almost entirely in chronic infection costs (−$3,981 and −$1,813), consistent with fewer patient-years in infected states.

the Duplicate model production also swapped exacerbation risk multipliers for the two infection pathways due to an ambiguity in the language about how to interpret language saying acquiring chronic infection “further elevates exacerbation risk by 25–30%”. This ambiguity resulted in two parameters getting their values swapped. The impact of these results turned out to be negligible (0 life years, +0.001 QALYs and −$13).

CISS

The Duplicate applies a half-cycle correction, weighting the first and last cycles at 0.5, which the Original does not. With only 17 cycles of 16 weeks (about 5.2 years), this removes about 0.31 years of follow-up and cuts discounted life years by 0.285 (from 4.805 to 4.546 in every arm, since survival is the same across arms). QALYs fall by 0.16 to 0.18 and costs by $4,592 to $14,334. Drug arms lose proportionally more cost because cycle 1 is the one cycle in which every alive patient is on drug, including non-responders. Two smaller differences offset a tenth of the effect: the Duplicate applies no background deaths in the induction cycle (+0.005 life years) and discounts from the start of each cycle rather than the middle (+0.022).

CNRD

Ten differences were identified, but three drive the cost gap. The no-treatment relapse probability used after cycle 80 is 0.40 in the Original and 0.12 in the Duplicate (a placeholder), lowering costs by about $9,500 per arm and raising QALYs by about 0.077. This value was not included in the report, making it unavailable for the test-retest duplication.

Second- and third-line drug costs are $100.64 and $445.94 per year in the Original and a flat $38 in the Duplicate, lowering costs by $5,060 (Velonix) and $5,356 (Doralene). Hospitalisation is charged differently: the Original applies $30,000 times expected admissions at cycle 1 only, while the Duplicate charges one admission per patient plus one at each later switch to line 2 or 3, which moves cost by +$4,206 for Velonix and −$9,076 for Doralene.

DMSS

Six differences were identified. The largest for cost is that the Duplicate omits background beta-blocker and calcium-channel-blocker costs, because the report gives no rates, removing $9,915 to $11,585 per arm. The Duplicate also lags mortality by one cycle: the transition into cycle i uses the age at the start of cycle i−1 (four weeks earlier), adding 0.027 to 0.028 life years.

Disease grade in the model is ‘frozen’ after a certain time period. Before the freeze cycle, the Original applies the frozen grade split from cycle 2 for all arms, although its own code comment says Cardivex and SoC should be frozen at cycle 8; the Duplicate implements cycle 8, changing QALYs by −0.035 to +0.008 and costs by −$1,006 to +$4,328.

Three minor differences (Zonapride cycle-1 drug cost, the Septoplasty disutility window, and rounding of the surgical-arm grade split) change costs by under $100 and QALYs by under 0.001. Net costs are lower by $4,996 to $10,493, but only 0.3% for Cardivex because its drug cost exceeds $1.1 million.

NCD

Outputs are identical in both versions to numerical precision. This confirms that the shared life table and comparison method introduce no differences of their own.

Discussion

By and large, the Duplicate models matched the Originals. One model was reproduced exactly, all models replicated the Markov trace to within ~1%, and the fifth model was accurate to within 5%. Costs varied much more widely than LYs or QALYs, but differences never exceeded 8% of the Original. However, the magnitude of differences was greater than I’d hoped especially as related to cost.

Where the differences came from

Three themes explain almost all of the differences.

1. Imprecise Report Language: the Duplicate filled gaps in the source reports with placeholders or assumptions that the Original resolved differently: a no-treatment relapse probability, a flat second- and third-line drug cost, omitted background medication costs and a half-cycle correction. These are examples of the kinds of ambiguous language choices that can have more than one interpretation.

2. True Misinterpretations: the AI did seem to misunderstand a few things, most notably in the CAIS example where two parameters swapped values because Claude read the report backwards from the way it was itended.

3. Undeclared assumptions: Timing conventions differed between the models: start-of-cycle rather than mid-cycle discounting, a mortality age lag in DMSS and an added stabilisation cycle in CNRD.

What all of these themes have in common is that they are caused by issues with the report, rather than issues with the Skill. When I asked Claude to generate the model reports, the prompt included instructions to write it in sufficient detail for it to be duplicated. Even with that instruction, natural language runs up against hard limits when it acts as a stand-in for computer code. Most of the problems Claude encountered would have been solved if the report had included a little more technical detail.

There were also some important lessons learned here that can be fed back into the Skill:

• Strengthening the language instructing Claude to check in with the user when the language is ambiguous.

• Keeping a running registration of all decisions made for the user to approve at Step 1.

• A few bugs related to the half-cycle correction.

These changes are minor. The real difference will come with greater clarity in the source reports.

Half-cycle correction in the Original documentation

The Original CISS documentation does not describe a half-cycle correction; it says only that costs and outcomes are “discounted at 3% per annum using mid-cycle convention”. That wording appears in the report’s model structure section, the README and the validation results. The Original code implements it as mid-cycle discounting alone: each cycle is discounted at (cycle − 0.5) × cycle length and counts in full, with no weighting of the first or last cycle.

The Duplicate’s parameter file shows where its correction came from. A code comment reads the report’s “mid-cycle convention” as a half-cycle correction on the trace (0.5 weight on the first and last rows), flags this as an assumption, and provides a switch to turn it off. The documentation therefore does not support the correction, and the Duplicate’s start-of-cycle discounting also departs from the stated mid-cycle convention. The Original packages are not consistent with one another: CAIS applies a half-cycle correction in code and NCD states that it uses none.

Limitations

The most obvious limitation of this exercise is that I am using the model’s training data to replicate its outputs. HEmodelR contains explicit rules about how to build models that were learned from building the Originals. The reports were produced with that Skill in its context window, and it was explicitly asked to write in such a way as to make replication easier. I definitely set myself up for success, and so it is little surprise that the duplication results were so close.

A more conclusive test would require me to have access to a model description written by someone else. The Skill was trained on exactly such descriptions, but was anchored against the published results to ensure accuracy. A future experiment could involve trying to replicate a model without having the results available.

Another future experiment could involve comparing HEmodelR to other methods for developing AI-generated models. I know there are a few different folks out there working on this same problem. It would be interesting to conduct some parallel duplications and see what the results are.

Conclusion

HEmodelR is capable of reproducing its own models from natural language descriptions. This suggests that the Skill successfully manages the probabilistic nature of LLMs to a degree that makes it reliable for producing decision models in a consistent way. Its efficacy is highly contingent on clearly written instructions, and the Skill still made important errors. The best use of the Skill continues to be in replicating models whose parameters, structure, and results are known. However, this experiment suggests that it has some degree of usefulness in producing de novo models as well.

The results of this duplication exercise, as well as the replicated models themselves, are available on Github. There is also a Shiny app for those who’d like to explore the results in greater detail.

Written with StackEdit.