TxAgent Portal: AI Agent Evaluation

Sign Up

Click Next to start the study. Your progress will be saved after you submit each question. For questions or concerns, contact us directly. Thank you for participating!

Instructions:

Please review these instructions and enter your information to begin:

  • Each session requires at least 5-10 minutes per question.
  • You can evaluate multiple questions; you will not repeat evaluations.
  • For each question, compare responses from two models and rate them (scale: 1-5).
  • If a question is unclear or irrelevant to biomedicine, click the RED BUTTON at the top of the comparison page.
  • Use the Back and Next buttons to edit responses before submission.
  • Use the Home Page button to return to the homepage; progress will save but not submit.
  • Submit answers to the current question before moving to the next.
  • You can pause between questions and return later; ensure current answers are submitted to save them.

Part 1/2: Pairwise Comparison

Model A Response:

Model B Response:

Task success
Which response more fully and correctly accomplishes the therapeutic task—providing the intended recommendation accurately and without substantive errors or omissions?
Model A Response - Did the model successfully complete the therapeutic task it was given?
Model B Response - Did the model successfully complete the therapeutic task it was given?

Helpfulness of rationale
Which response offers a clearer, more detailed rationale that genuinely aids you in judging whether the answer is correct?
Model A Response - Is the model’s rationale helpful in determining whether the answer is correct?
Model B Response - Is the model’s rationale helpful in determining whether the answer is correct?

Cognitive traceability
In which response are the intermediate reasoning steps and decision factors laid out more transparently and logically, making it easy to follow how the final recommendation was reached?
Model A Response - Are the intermediate reasoning steps and decision factors interpretable and traceable?
Model B Response - Are the intermediate reasoning steps and decision factors interpretable and traceable?

Possibility of harm
Which response presents a lower likelihood of causing clinical harm, based on the safety and soundness of its recommendations and rationale?
Model A Response - Based on the model’s output and rationale, is there a risk that the recommendation could cause clinical harm?
Model B Response - Based on the model’s output and rationale, is there a risk that the recommendation could cause clinical harm?

Alignment with clinical consensus
Which response aligns better with clinical guidelines and practice standards?
Model A Response - Does the answer reflect established clinical practices and guidelines?
Model B Response - Does the answer reflect established clinical practices and guidelines?

Accuracy of content
Which response is more factually accurate and relevant, containing fewer (or no) errors or extraneous details?
Model A Response - Are there any factual inaccuracies or irrelevant information in the response?
Model B Response - Are there any factual inaccuracies or irrelevant information in the response?

Completeness
Which response is more comprehensive, covering all necessary therapeutic considerations without significant omissions?
Model A Response - Does the model provide a complete response covering all necessary elements?
Model B Response - Does the model provide a complete response covering all necessary elements?

Clinical relevance
Which response stays focused on clinically meaningful issues—such as appropriate drug choices, pertinent patient subgroups, and key outcomes—while minimizing tangential or less useful content?
Model A Response - Does the model focus on clinically meaningful aspects of the case (e.g., appropriate drug choices, patient subgroups, relevant outcomes)?
Model B Response - Does the model focus on clinically meaningful aspects of the case (e.g., appropriate drug choices, patient subgroups, relevant outcomes)?

You have no questions left to evaluate. Thank you for your participation!