Question 1

What is "cumulative prompting" and how is it structured in this script?

Accepted Answer

Cumulative prompting is a method of building an AI's context in stages, where each subsequent level of instruction retains all previous information while adding new, more complex layers. In this specific script, the process is divided into four conditions: Condition A provides the baseline study and ID; Condition B adds a specific JSON schema and risk-of-bias criteria; Condition C introduces "training material" or methodological textbooks; and Condition D adds a "worked example" featuring a completed assessment of a separate study to serve as a gold-standard reference for the model.

Question 2

How does the script ensure the AI's output is scientifically accurate and valid?

Accepted Answer

The script employs a "guilty until proven innocent" approach to AI responses through a strict validation gate. After the model generates a response, the script parses the JSON to ensure it matches the required schema and verifies that the `study_id` in the output matches the input study to prevent hallucinations. Furthermore, the script requires the AI to provide a "verbatim quote" from the PDF for every judgment made, forcing the model to anchor its scientific claims in the actual text of the study.

Question 3

Why does the script use a "PROMPT_BRIDGE" and temperature set to zero?

Accepted Answer

The `PROMPT_BRIDGE` is a specific string used as a linguistic marker to help the model transition from reading training materials or examples to performing the actual task on the target study, which is crucial for maintaining focus in long-context windows. Setting the temperature to zero ensures the model's output is deterministic and consistent rather than creative. In a research context, this "boring" consistency is preferred because it prioritizes accuracy and reproducibility over varied or "fun" responses.

Question 4

How does the script handle failures or errors during the assessment process?

Accepted Answer

The script uses a robust two-strike system for errors. If a response fails to parse or validate on the first attempt, the script logs the error and automatically tries a second time. If it fails again, the script records the failure in a detailed CSV log (`parse_failures.csv`) including the study ID, model used, and the specific error message. This allows researchers to audit the process and identify if certain models or prompts are consistently struggling with specific types of data.

Question 5

What is the "PromptAssetResolver" and why is it used?

Accepted Answer

The `PromptAssetResolver` is a dynamic discovery tool that allows the script to find necessary files—like training PDFs or example JSONs—without the researcher having to hardcode every file path. It uses a scoring system to search through directories for keywords like "mulder" or "green" to identify the best-fit materials for the prompt. This makes the script flexible and portable, allowing different researchers to use the same logic even if their file naming conventions vary slightly, while still "failing loudly" if it finds ambiguous or conflicting files.

Automating Risk-of-Bias Assessments with Gemini

How This Personalized Podcast Was Made

More like this