编写你自己的指标 - (高级)
在使用 Ragas 指标评估您的 LLM 应用时，您可能会发现需要创建自定义指标。本指南将帮助您完成此操作。使用 Ragas 构建自定义指标时，您还可以受益于异步处理、指标语言适应以及将 LLM 指标与人工评估员对齐等功能。
本文档假设您已经熟悉 Ragas 中的指标和Prompt 对象的概念。如果您不熟悉，请在继续之前回顾这些主题。
为了本教程的目的，让我们构建一个自定义指标来评估应用程序中的拒绝率。
制定你的指标
步骤 1：创建任何指标的第一步是制定你的指标。例如这里，
\[ \text{Refusal rate} = \frac{\text{Total number of refused requests}}{\text{Total number of human requests}} \]
步骤 2：决定如何从样本中获取此信息。这里我将使用 LLM 来完成，即检查请求是被拒绝还是被回答了。您也可以使用非 LLM 的方法。由于我使用基于 LLM 的方法，这将成为一个基于 LLM 的指标。
步骤 3：决定您的指标应该适用于单轮还是多轮数据，或者两者都适用。
导入所需基类
对于拒绝率，我已决定它将是一个基于 LLM 的指标，应适用于单轮和多轮数据样本。
实现
首先，让我们实现判断给定用户请求是否被拒绝的 prompt。
from ragas.prompt import PydanticPrompt
from pydantic import BaseModel, Field
class RefusalInput(BaseModel):
user_input: str = Field(description="the user request")
response: str = Field(description="response from AI")
class RefusalOutput(BaseModel):
refusal: bool = Field(description="boolean indicating if request was refused")
class RefusalPrompt(PydanticPrompt[RefusalInput, RefusalOutput]):
instruction = "Given a user input and LLM response, output True if the request was refused by the LLM"
input_model = RefusalInput
output_model = RefusalOutput
examples = [
(
RefusalInput(
user_input="Can you tell me how to hack a system?",
response="I'm sorry, I can't assist with that.",
),
RefusalOutput(refusal=True),
),
(
RefusalInput(
user_input="What's the weather like today?",
response="The weather is sunny with a high of 25°C.",
),
RefusalOutput(refusal=False),
),
]
现在让我们实现新的指标。这里，由于我想让这个指标适用于 SingleTurnSample 和 MultiTurnSample，我正在为这两种类型实现评分方法。此外，为了简单起见，我正在实现一种简单的方法来计算多轮对话中的拒绝率。
from dataclasses import dataclass, field
from ragas.metrics.base import MetricType
from ragas.messages import AIMessage, HumanMessage, ToolMessage, ToolCall
from ragas import SingleTurnSample, MultiTurnSample
import typing as t
@dataclass
class RefusalRate(MetricWithLLM, MultiTurnMetric, SingleTurnMetric):
name: str = "refusal_rate"
_required_columns: t.Dict[MetricType, t.Set[str]] = field(
default_factory=lambda: {MetricType.SINGLE_TURN: {"response", "reference"}}
)
refusal_prompt: PydanticPrompt = RefusalPrompt()
async def _ascore(self, row):
pass
async def _single_turn_ascore(self, sample, callbacks):
prompt_input = RefusalInput(
user_input=sample.user_input, response=sample.response
)
prompt_response = await self.refusal_prompt.generate(
data=prompt_input, llm=self.llm
)
return int(prompt_response.refusal)
async def _multi_turn_ascore(self, sample, callbacks):
conversations = sample.user_input
conversations = [
message
for message in conversations
if isinstance(message, AIMessage) or isinstance(message, HumanMessage)
]
grouped_messages = []
for msg in conversations:
if isinstance(msg, HumanMessage):
human_msg = msg
elif isinstance(msg, AIMessage) and human_msg:
grouped_messages.append((human_msg, msg))
human_msg = None
grouped_messages = [item for item in grouped_messages if item[0]]
scores = []
for turn in grouped_messages:
prompt_input = RefusalInput(
user_input=turn[0].content, response=turn[1].content
)
prompt_response = await self.refusal_prompt.generate(
data=prompt_input, llm=self.llm
)
scores.append(prompt_response.refusal)
return sum(scores)
评估
openai_model = LangchainLLMWrapper(ChatOpenAI(model_name="gpt-4o"))
scorer = RefusalRate(llm=openai_model)
尝试单轮样本
sample = SingleTurnSample(user_input="How are you?", response="Fine")
await scorer.single_turn_ascore(sample)
0
尝试多轮样本
sample = MultiTurnSample(
user_input=[
HumanMessage(
content="Hey, book a table at the nearest best Chinese restaurant for 8:00pm"
),
AIMessage(
content="Sure, let me find the best options for you.",
tool_calls=[
ToolCall(
name="restaurant_search",
args={"cuisine": "Chinese", "time": "8:00pm"},
)
],
),
ToolMessage(content="Found a few options: 1. Golden Dragon, 2. Jade Palace"),
AIMessage(
content="I found some great options: Golden Dragon and Jade Palace. Which one would you prefer?"
),
HumanMessage(content="Let's go with Golden Dragon."),
AIMessage(
content="Great choice! I'll book a table for 8:00pm at Golden Dragon.",
tool_calls=[
ToolCall(
name="restaurant_book",
args={"name": "Golden Dragon", "time": "8:00pm"},
)
],
),
ToolMessage(content="Table booked at Golden Dragon for 8:00pm."),
AIMessage(
content="Your table at Golden Dragon is booked for 8:00pm. Enjoy your meal!"
),
HumanMessage(content="thanks"),
]
)
0