修改指标中的提示词
Ragas 中每个使用 LLM 的指标都使用一个或多个提示词来得出用于计算分数的中间结果。在使用基于 LLM 的指标时，提示词可以被视为超参数。适合您的领域和用例的优化提示词可以将基于 LLM 的指标的准确性提高 10-20%。最优提示词也取决于您正在使用的 LLM，因此作为用户，您可能需要调整驱动每个指标的提示词。
Ragas 中的每个提示词都使用 [Prompt Object][ragas.prompts.PydanticPrompt] 编写，请在继续之前确保您对其有所了解。
理解指标的提示词
由于 Ragas 将提示词视为指标中的超参数，因此我们提供了一个统一的接口 get_prompts 来访问任何指标背后使用的提示词。
from ragas.metrics._simple_criteria import SimpleCriteriaScoreWithReference
scorer = SimpleCriteriaScoreWithReference(name="random", definition="some definition")
scorer.get_prompts()
{'multi_turn_prompt': <ragas.metrics._simple_criteria.MultiTurnSimpleCriteriaWithReferencePrompt at 0x7f8c41410970>,
'single_turn_prompt': <ragas.metrics._simple_criteria.SingleTurnSimpleCriteriaWithReferencePrompt at 0x7f8c41412590>}
Your task is to judge the faithfulness of a series of statements based on a given context. For each statement you must return verdict as 1 if the statement can be directly inferred based on the context or 0 if the statement can not be directly inferred based on the context.
修改默认提示词中的指令
您很有可能希望根据自己的需求修改提示词。Ragas 提供了 set_prompts 方法来允许您这样做。让我们修改 FactualCorrectness 指标中使用的其中一个提示词。
prompt = scorer.get_prompts()["single_turn_prompt"]
prompt.instruction += "\nOnly output valid JSON."
scorer.set_prompts(**{"single_turn_prompt": prompt})
Given a input, system response and reference. Evaluate and score the response against the reference only using the given criteria.
Only output valid JSON.
修改默认提示词中的示例
少样本示例可以极大地影响任何 LLM 的结果。默认提示词中的示例很可能无法反映您的领域或用例。因此，使用您自己的自定义示例进行修改始终是很好的做法。让我们在这里尝试一个。
输出
[(SingleTurnSimpleCriteriaWithReferenceInput(user_input='Who was the director of Los Alamos Laboratory?', response='Einstein was the director of Los Alamos Laboratory.', criteria='Score responses in range of 0 (low) to 5 (high) based similarity with reference.', reference='The director of Los Alamos Laboratory was J. Robert Oppenheimer.'),
SimpleCriteriaOutput(reason='The response and reference have two very different answers.', score=0))]
from ragas.metrics._simple_criteria import (
SingleTurnSimpleCriteriaWithReferenceInput,
SimpleCriteriaOutput,
)
new_example = [
(
SingleTurnSimpleCriteriaWithReferenceInput(
user_input="Who was the first president of the United States?",
response="Thomas Jefferson was the first president of the United States.",
criteria="Score responses in range of 0 (low) to 5 (high) based similarity with reference.",
reference="George Washington was the first president of the United States.",
),
SimpleCriteriaOutput(
reason="The response incorrectly states Thomas Jefferson instead of George Washington. While both are significant historical figures, the answer does not match the reference.",
score=2,
),
)
]
prompt.examples = new_example
scorer.set_prompts(**{"single_turn_prompt": prompt})
print(scorer.get_prompts()["single_turn_prompt"].examples)
输出
[(SingleTurnSimpleCriteriaWithReferenceInput(user_input='Who was the first president of the United States?', response='Thomas Jefferson was the first president of the United States.', criteria='Score responses in range of 0 (low) to 5 (high) based similarity with reference.', reference='George Washington was the first president of the United States.'), SimpleCriteriaOutput(reason='The response incorrectly states Thomas Jefferson instead of George Washington. While both are significant historical figures, the answer does not match the reference.', score=2))]
现在让我们查看并验证修改了指令和示例的完整新提示词。