跳转到内容

Prompt API 参考

Ragas 中的提示系统提供了一种灵活且类型安全的方式来为基于 LLM 的指标和其他组件定义提示。本页文档介绍了核心提示类及其用法。

概述

Ragas 使用基于 BasePrompt 类的模块化提示架构。提示可以是:

  • 输入/输出模型:定义提示输入和输出结构的 Pydantic BaseModel 类
  • 提示类:继承自 BasePrompt 来定义指令、示例和提示生成逻辑
  • 字符串提示:为实现向后兼容而设的简单文本提示

核心类

InputModel module-attribute

InputModel = TypeVar('InputModel', bound=BaseModel)

OutputModel module-attribute

OutputModel = TypeVar('OutputModel', bound=BaseModel)

BasePrompt

BasePrompt(name: Optional[str] = None, language: str = 'english', original_hash: Optional[str] = None)

Bases: ABC

源代码位于 src/ragas/prompt/base.py
def __init__(
    self,
    name: t.Optional[str] = None,
    language: str = "english",
    original_hash: t.Optional[str] = None,
):
    if name is None:
        self.name = camel_to_snake(self.__class__.__name__)

    self.language = language
    self.original_hash = original_hash

generate abstractmethod async

generate(llm: BaseRagasLLM, data: Any, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Callbacks = []) -> Any

根据提示生成单个补全结果。

源代码位于 src/ragas/prompt/base.py
@abstractmethod
async def generate(
    self,
    llm: BaseRagasLLM,
    data: t.Any,
    temperature: t.Optional[float] = None,
    stop: t.Optional[t.List[str]] = None,
    callbacks: Callbacks = [],
) -> t.Any:
    """
    Generate a single completion from the prompt.
    """
    pass

generate_multiple abstractmethod

generate_multiple(llm: BaseRagasLLM, data: Any, n: int = 1, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Callbacks = []) -> Any

根据提示生成多个补全结果。

源代码位于 src/ragas/prompt/base.py
@abstractmethod
def generate_multiple(
    self,
    llm: BaseRagasLLM,
    data: t.Any,
    n: int = 1,
    temperature: t.Optional[float] = None,
    stop: t.Optional[t.List[str]] = None,
    callbacks: Callbacks = [],
) -> t.Any:
    """
    Generate multiple completions from the prompt.
    """
    pass

StringPrompt

StringPrompt(name: Optional[str] = None, language: str = 'english', original_hash: Optional[str] = None)

Bases: BasePrompt

一个可以使用 f-string 语法和附加数据进行格式化的简单提示。

对于那些偏好更灵活方法而不需要 Pydantic 模型的用户,此提示是 PydanticPrompt 的一个更简单的替代方案。

参数

名称 类型 描述 默认值
instruction str

可以与附加数据进行格式化的指令字符串。

必需

示例

>>> from ragas.prompt import string_prompt
>>> await prompt.generate(llm=llm, data={"category": "commerce"})
源代码位于 src/ragas/prompt/base.py
def __init__(
    self,
    name: t.Optional[str] = None,
    language: str = "english",
    original_hash: t.Optional[str] = None,
):
    if name is None:
        self.name = camel_to_snake(self.__class__.__name__)

    self.language = language
    self.original_hash = original_hash

generate async

generate(llm: BaseRagasLLM, data: str, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Callbacks = []) -> str

根据指令和提供的数据生成文本。

参数

名称 类型 描述 默认值
llm BaseRagasLLM

用于文本生成的语言模型。

必需
data Optional[Dict[str, Any]]

用于格式化指令的数据,默认为 None。

必需
n int

要生成的补全数量,默认为 1。

必需
temperature Optional[float]

用于文本生成的温度参数,默认为 None。

None
stop Optional[List[str]]

用于文本生成的停止序列,默认为 None。

None
callbacks Callbacks

在文本生成期间使用的回调函数,默认为 []。

[]

返回

类型 描述
str

生成的文本。

源代码位于 src/ragas/prompt/base.py
async def generate(
    self,
    llm: BaseRagasLLM,
    data: str,
    temperature: t.Optional[float] = None,
    stop: t.Optional[t.List[str]] = None,
    callbacks: Callbacks = [],
) -> str:
    """
    Generate text based on the instruction and provided data.

    Parameters
    ----------
    llm : BaseRagasLLM
        The language model to use for text generation.
    data : Optional[Dict[str, Any]], optional
        The data to format the instruction with, by default None.
    n : int, optional
        The number of completions to generate, by default 1.
    temperature : Optional[float], optional
        The temperature for text generation, by default None.
    stop : Optional[List[str]], optional
        The stop sequences for text generation, by default None.
    callbacks : Callbacks, optional
        The callbacks to use during text generation, by default [].

    Returns
    -------
    str
        The generated text.
    """
    llm_result = await llm.agenerate_text(
        StringPromptValue(text=data),
        n=1,
        temperature=temperature,
        stop=stop,
        callbacks=callbacks,
    )
    return llm_result.generations[0][0].text

generate_multiple async

generate_multiple(llm: BaseRagasLLM, data: str, n: int = 1, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Callbacks = []) -> List[str]

根据指令和提供的数据生成多个不同的文本输出。

参数

名称 类型 描述 默认值
llm BaseRagasLLM

用于文本生成的语言模型。

必需
data str

用于格式化指令的数据。

必需
n int

要生成的补全数量,默认为 1。

1
temperature Optional[float]

用于文本生成的温度参数,默认为 None。

None
stop Optional[List[str]]

用于文本生成的停止序列,默认为 None。

None
callbacks Callbacks

在文本生成期间使用的回调函数,默认为 []。

[]

返回

类型 描述
List[str]

一个包含 n 个生成输出的列表。

备注
  • 启用缓存时,每个输出都会被唯一缓存以防止重复。
  • 这确保了同一输入的多个输出是不同的。
  • 之前缓存返回重复输出的问题已得到修复。
源代码位于 src/ragas/prompt/base.py
async def generate_multiple(
    self,
    llm: BaseRagasLLM,
    data: str,
    n: int = 1,
    temperature: t.Optional[float] = None,
    stop: t.Optional[t.List[str]] = None,
    callbacks: Callbacks = [],
) -> t.List[str]:
    """
    Generate multiple distinct text outputs based on the instruction and provided data.

    Parameters
    ----------
    llm : BaseRagasLLM
        The language model to use for text generation.
    data : str
        The data to format the instruction with.
    n : int, optional
        The number of completions to generate, by default 1.
    temperature : Optional[float], optional
        The temperature for text generation, by default None.
    stop : Optional[List[str]], optional
        Stop sequences for text generation, by default None.
    callbacks : Callbacks, optional
        Callbacks to use during text generation, by default [].

    Returns
    -------
    List[str]
        A list containing `n` generated outputs.

    Notes
    -----
    - When caching is enabled, each output is uniquely cached to prevent duplicates.
    - This ensures that multiple outputs for the same input are distinct.
    - Previous issues where caching returned duplicate outputs have been fixed.
    """
    llm_result = await llm.agenerate_text(
        StringPromptValue(text=data),
        n=n,
        temperature=temperature,
        stop=stop,
        callbacks=callbacks,
    )

    # flatten the generations
    return [gen.text for gen in llm_result.generations[0]]

PydanticPrompt

PydanticPrompt(name: Optional[str] = None, language: str = 'english', original_hash: Optional[str] = None)

Bases: BasePrompt, Generic[InputModel, OutputModel]

源代码位于 src/ragas/prompt/base.py
def __init__(
    self,
    name: t.Optional[str] = None,
    language: str = "english",
    original_hash: t.Optional[str] = None,
):
    if name is None:
        self.name = camel_to_snake(self.__class__.__name__)

    self.language = language
    self.original_hash = original_hash

generate async

generate(llm: Union[BaseRagasLLM, InstructorBaseRagasLLM, BaseLanguageModel], data: InputModel, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Optional[Callbacks] = None, retries_left: int = 3) -> OutputModel

使用提供的语言模型和输入数据生成单个输出。

此方法是 generate_multiple 的一个特例,只生成一个输出。

参数

名称 类型 描述 默认值
llm BaseRagasLLM

用于生成的语言模型。

必需
data InputModel

用于生成的输入数据。

必需
temperature float

用于控制生成随机性的温度参数。

None
stop List[str]

用于结束生成的停止序列列表。

None
callbacks Callbacks

在生成过程中调用的回调函数。

None
retries_left int

对无效 LLM 响应的重试次数。

3

返回

类型 描述
OutputModel

生成的输出。

备注

此方法内部调用 generate_multiple 并设置 n=1,然后返回第一个(也是唯一一个)结果。

源代码位于 src/ragas/prompt/pydantic_prompt.py
async def generate(
    self,
    llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM, BaseLanguageModel],
    data: InputModel,
    temperature: t.Optional[float] = None,
    stop: t.Optional[t.List[str]] = None,
    callbacks: t.Optional[Callbacks] = None,
    retries_left: int = 3,
) -> OutputModel:
    """
    Generate a single output using the provided language model and input data.

    This method is a special case of `generate_multiple` where only one output is generated.

    Parameters
    ----------
    llm : BaseRagasLLM
        The language model to use for generation.
    data : InputModel
        The input data for generation.
    temperature : float, optional
        The temperature parameter for controlling randomness in generation.
    stop : List[str], optional
        A list of stop sequences to end generation.
    callbacks : Callbacks, optional
        Callback functions to be called during the generation process.
    retries_left : int, optional
        Number of retry attempts for an invalid LLM response

    Returns
    -------
    OutputModel
        The generated output.

    Notes
    -----
    This method internally calls `generate_multiple` with `n=1` and returns the first (and only) result.
    """
    callbacks = callbacks or []

    # this is just a special case of generate_multiple
    output_single = await self.generate_multiple(
        llm=llm,
        data=data,
        n=1,
        temperature=temperature,
        stop=stop,
        callbacks=callbacks,
        retries_left=retries_left,
    )
    return output_single[0]

generate_multiple async

generate_multiple(llm: Union[BaseRagasLLM, InstructorBaseRagasLLM, BaseLanguageModel], data: InputModel, n: int = 1, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Optional[Callbacks] = None, retries_left: int = 3) -> List[OutputModel]

使用提供的语言模型和输入数据生成多个输出。

参数

名称 类型 描述 默认值
llm BaseRagasLLM

用于生成的语言模型。

必需
data InputModel

用于生成的输入数据。

必需
n int

要生成的输出数量。默认为 1。

1
temperature float

用于控制生成随机性的温度参数。

None
stop List[str]

用于结束生成的停止序列列表。

None
callbacks Callbacks

在生成过程中调用的回调函数。

None
retries_left int

对无效 LLM 响应的重试次数。

3

返回

类型 描述
List[OutputModel]

一个生成输出的列表。

抛出

类型 描述
RagasOutputParserException

如果解析输出时出错。

源代码位于 src/ragas/prompt/pydantic_prompt.py
async def generate_multiple(
    self,
    llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM, BaseLanguageModel],
    data: InputModel,
    n: int = 1,
    temperature: t.Optional[float] = None,
    stop: t.Optional[t.List[str]] = None,
    callbacks: t.Optional[Callbacks] = None,
    retries_left: int = 3,
) -> t.List[OutputModel]:
    """
    Generate multiple outputs using the provided language model and input data.

    Parameters
    ----------
    llm : BaseRagasLLM
        The language model to use for generation.
    data : InputModel
        The input data for generation.
    n : int, optional
        The number of outputs to generate. Default is 1.
    temperature : float, optional
        The temperature parameter for controlling randomness in generation.
    stop : List[str], optional
        A list of stop sequences to end generation.
    callbacks : Callbacks, optional
        Callback functions to be called during the generation process.
    retries_left : int, optional
        Number of retry attempts for an invalid LLM response

    Returns
    -------
    List[OutputModel]
        A list of generated outputs.

    Raises
    ------
    RagasOutputParserException
        If there's an error parsing the output.
    """
    callbacks = callbacks or []

    processed_data = self.process_input(data)
    prompt_rm, prompt_cb = new_group(
        name=self.name,
        inputs={"data": processed_data},
        callbacks=callbacks,
        metadata={"type": ChainType.RAGAS_PROMPT},
    )
    prompt_value = PromptValue(text=self.to_string(processed_data))

    # Handle different LLM types with different interfaces
    # 1. LangChain LLMs have agenerate_prompt() for async with specific signature
    # 2. BaseRagasLLM have generate() with n, temperature, stop, callbacks
    # 3. InstructorLLM has generate()/agenerate() with only prompt and response_model
    if is_langchain_llm(llm):
        # This is a LangChain LLM - use agenerate_prompt() with batch for multiple generations
        langchain_llm = t.cast(BaseLanguageModel, llm)
        # LangChain doesn't support n parameter directly, so we batch multiple prompts
        prompts = t.cast(t.List[t.Any], [prompt_value for _ in range(n)])
        resp = await langchain_llm.agenerate_prompt(
            prompts,
            stop=stop,
            callbacks=prompt_cb,
        )
    elif isinstance(llm, InstructorBaseRagasLLM):
        # This is an InstructorLLM - use its generate()/agenerate() method
        # InstructorLLM.generate()/agenerate() only takes prompt and response_model parameters
        from ragas.llms.base import InstructorLLM

        instructor_llm = t.cast(InstructorLLM, llm)
        if instructor_llm.is_async:
            result = await llm.agenerate(
                prompt=prompt_value.text,
                response_model=self.output_model,
            )
        else:
            result = llm.generate(
                prompt=prompt_value.text,
                response_model=self.output_model,
            )
        # Wrap the single response in an LLMResult-like structure for consistency
        from langchain_core.outputs import Generation, LLMResult

        generation = Generation(text=result.model_dump_json())
        resp = LLMResult(generations=[[generation]])
    else:
        # This is a standard BaseRagasLLM - use generate()
        ragas_llm = t.cast(BaseRagasLLM, llm)
        resp = await ragas_llm.generate(
            prompt_value,
            n=n,
            temperature=temperature,
            stop=stop,
            callbacks=prompt_cb,
        )

    output_models = []
    parser = RagasOutputParser(pydantic_object=self.output_model)

    # Handle cases where LLM returns fewer generations than requested
    if is_langchain_llm(llm) or isinstance(llm, InstructorBaseRagasLLM):
        available_generations = len(resp.generations)
    else:
        available_generations = len(resp.generations[0]) if resp.generations else 0

    actual_n = min(n, available_generations)

    if actual_n == 0:
        logger.error(
            f"LLM returned no generations when {n} were requested. Cannot proceed."
        )
        raise ValueError(f"LLM returned no generations when {n} were requested")

    if actual_n < n:
        logger.warning(
            f"LLM returned {actual_n} generations instead of requested {n}. "
            f"Proceeding with {actual_n} generations."
        )

    for i in range(actual_n):
        if is_langchain_llm(llm) or isinstance(llm, InstructorBaseRagasLLM):
            # For LangChain LLMs and InstructorLLM, each generation is in a separate batch result
            output_string = resp.generations[i][0].text
        else:
            # For Ragas LLMs, all generations are in the first batch
            output_string = resp.generations[0][i].text
        try:
            # For the parser, we need a BaseRagasLLM, so if it's a LangChain LLM, we need to handle this
            if is_langchain_llm(llm) or isinstance(llm, InstructorBaseRagasLLM):
                # Skip parsing retry for LangChain LLMs since parser expects BaseRagasLLM
                answer = self.output_model.model_validate_json(output_string)
            else:
                ragas_llm = t.cast(BaseRagasLLM, llm)
                answer = await parser.parse_output_string(
                    output_string=output_string,
                    prompt_value=prompt_value,
                    llm=ragas_llm,
                    callbacks=prompt_cb,
                    retries_left=retries_left,
                )
            processed_output = self.process_output(answer, data)  # type: ignore
            output_models.append(processed_output)
        except RagasOutputParserException as e:
            prompt_rm.on_chain_error(error=e)
            logger.error("Prompt %s failed to parse output: %s", self.name, e)
            raise e

    prompt_rm.on_chain_end({"output": output_models})

    # Track prompt usage
    track(
        PromptUsageEvent(
            prompt_type="pydantic",
            has_examples=len(self.examples) > 0,
            num_examples=len(self.examples),
            has_response_model=True,  # PydanticPrompt always has response model
            language=self.language,
        )
    )

    return output_models

adapt async

adapt(target_language: str, llm: Union[BaseRagasLLM, InstructorBaseRagasLLM], adapt_instruction: bool = False) -> 'PydanticPrompt[InputModel, OutputModel]'

将提示调整为新语言。

源代码位于 src/ragas/prompt/pydantic_prompt.py
async def adapt(
    self,
    target_language: str,
    llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM],
    adapt_instruction: bool = False,
) -> "PydanticPrompt[InputModel, OutputModel]":
    """
    Adapt the prompt to a new language.
    """

    strings = get_all_strings(self.examples)
    translated_strings = await translate_statements_prompt.generate(
        llm=llm,
        data=ToTranslate(target_language=target_language, statements=strings),
    )

    translated_examples = update_strings(
        obj=self.examples,
        old_strings=strings,
        new_strings=translated_strings.statements,
    )

    new_prompt = copy.deepcopy(self)
    new_prompt.examples = translated_examples
    new_prompt.language = target_language

    if adapt_instruction:
        translated_instruction = await translate_statements_prompt.generate(
            llm=llm,
            data=ToTranslate(
                target_language=target_language, statements=[self.instruction]
            ),
        )
        new_prompt.instruction = translated_instruction.statements[0]

    new_prompt.original_hash = hash(new_prompt)

    return new_prompt

save

save(file_path: str)

将提示保存到文件。

源代码位于 src/ragas/prompt/pydantic_prompt.py
def save(self, file_path: str):
    """
    Save the prompt to a file.
    """
    data = {
        "ragas_version": __version__,
        "original_hash": (
            hash(self) if self.original_hash is None else self.original_hash
        ),
        "language": self.language,
        "instruction": self.instruction,
        "examples": [
            {"input": example[0].model_dump(), "output": example[1].model_dump()}
            for example in self.examples
        ],
    }
    if os.path.exists(file_path):
        raise FileExistsError(f"The file '{file_path}' already exists.")
    with open(file_path, "w", encoding="utf-8") as f:
        json.dump(data, f, indent=2, ensure_ascii=False)
        print(f"Prompt saved to {file_path}")

BoolIO

Bases: BaseModel

StringIO

Bases: BaseModel

PromptMixin

用于包含提示的类的 Mixin 类。例如:BaseSynthesizer, MetricWithLLM

get_prompts

get_prompts() -> Dict[str, PydanticPrompt]

返回该类的提示字典。

源代码位于 src/ragas/prompt/mixin.py
def get_prompts(self) -> t.Dict[str, PydanticPrompt]:
    """
    Returns a dictionary of prompts for the class.
    """
    prompts = {}
    for _, value in self._get_prompts().items():
        prompts.update({value.name: value})
    return prompts

set_prompts

set_prompts(**prompts)

设置该类的提示。

抛出

类型 描述
ValueError

如果提示不是 PydanticPrompt 的实例。

源代码位于 src/ragas/prompt/mixin.py
def set_prompts(self, **prompts):
    """
    Sets the prompts for the class.

    Raises
    ------
    ValueError
        If the prompt is not an instance of `PydanticPrompt`.
    """
    available_prompts = self.get_prompts()
    name_to_var = {v.name: k for k, v in self._get_prompts().items()}
    for key, value in prompts.items():
        if key not in available_prompts:
            raise ValueError(
                f"Prompt with name '{key}' does not exist. Use get_prompts() to see available prompts."
            )
        if not isinstance(value, PydanticPrompt):
            raise ValueError(
                f"Prompt with name '{key}' must be an instance of 'ragas.prompt.PydanticPrompt'"
            )
        setattr(self, name_to_var[key], value)

adapt_prompts async

adapt_prompts(language: str, llm: Union[BaseRagasLLM, InstructorBaseRagasLLM], adapt_instruction: bool = False) -> Dict[str, PydanticPrompt]

将类中的提示调整到给定语言,并使用给定的 LLM。

备注

请确保使用最佳可用 LLM 来调整提示,然后使用 save_promptsload_prompts 方法保存和加载提示。

源代码位于 src/ragas/prompt/mixin.py
async def adapt_prompts(
    self,
    language: str,
    llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM],
    adapt_instruction: bool = False,
) -> t.Dict[str, PydanticPrompt]:
    """
    Adapts the prompts in the class to the given language and using the given LLM.

    Notes
    -----
    Make sure you use the best available LLM for adapting the prompts and then save and load the prompts using
    [save_prompts][ragas.prompt.mixin.PromptMixin.save_prompts] and [load_prompts][ragas.prompt.mixin.PromptMixin.load_prompts]
    methods.
    """
    prompts = self.get_prompts()
    adapted_prompts = {}
    for name, prompt in prompts.items():
        adapted_prompt = await prompt.adapt(language, llm, adapt_instruction)
        adapted_prompts[name] = adapted_prompt

    return adapted_prompts

save_prompts

save_prompts(path: str)

将提示以 {name}_{language}.json 的格式保存到目录中。

源代码位于 src/ragas/prompt/mixin.py
def save_prompts(self, path: str):
    """
    Saves the prompts to a directory in the format of {name}_{language}.json
    """
    # check if path is valid
    if not os.path.exists(path):
        raise ValueError(f"Path {path} does not exist")

    prompts = self.get_prompts()
    for prompt_name, prompt in prompts.items():
        # hash_hex = f"0x{hash(prompt) & 0xFFFFFFFFFFFFFFFF:016x}"
        if self.name == "":
            file_name = os.path.join(path, f"{prompt_name}_{prompt.language}.json")
        else:
            file_name = os.path.join(
                path, f"{self.name}_{prompt_name}_{prompt.language}.json"
            )
        prompt.save(file_name)

load_prompts

load_prompts(path: str, language: Optional[str] = None)

从路径加载提示。文件格式应为 {name}_{language}.json。

源代码位于 src/ragas/prompt/mixin.py
def load_prompts(self, path: str, language: t.Optional[str] = None):
    """
    Loads the prompts from a path. File should be in the format of {name}_{language}.json
    """
    # check if path is valid
    if not os.path.exists(path):
        raise ValueError(f"Path {path} does not exist")

    # check if language is supported, defaults to english
    if language is None:
        language = "english"
        logger.info(
            "Language not specified, loading prompts for default language: %s",
            language,
        )

    loaded_prompts = {}
    for prompt_name, prompt in self.get_prompts().items():
        if self.name == "":
            file_name = os.path.join(path, f"{prompt_name}_{language}.json")
        else:
            file_name = os.path.join(
                path, f"{self.name}_{prompt_name}_{language}.json"
            )
        loaded_prompt = prompt.__class__.load(file_name)
        loaded_prompts[prompt_name] = loaded_prompt
    return loaded_prompts

指标集合提示

Ragas 中的现代指标使用专门的提示类。每个指标模块包含:

  • 输入模型:定义提示所需的数据(例如,FaithfulnessInput
  • 输出模型:定义预期的 LLM 响应结构(例如,FaithfulnessOutput
  • 提示类:继承自 BasePrompt,用于生成包含示例和指令的提示字符串

示例:忠实度指标提示

from ragas.metrics.collections.faithfulness.util import (
    FaithfulnessPrompt,
    FaithfulnessInput,
    FaithfulnessOutput,
)

# The prompt class combines input/output models with instructions and examples
prompt = FaithfulnessPrompt()

# Create input data
input_data = FaithfulnessInput(
    response="The capital of France is Paris.",
    context="Paris is the capital and most populous city of France."
)

# Generate the prompt string for the LLM
prompt_string = prompt.to_string(input_data)

# The output will be structured according to FaithfulnessOutput model

可用指标提示

有关其提示的详细信息,请参阅各个指标的文档

自定义

有关自定义指标提示的详细指导,请参阅修改指标中的提示