Rhesis Synthesizers

This module contains the synthesizer classes used throughout the Rhesis SDK.

Base Synthesizer

class ABC[source]

Bases: object

Helper class that provides a standard way to create an ABC using inheritance.

abstractmethod(funcobj)[source]

A decorator indicating abstract methods.

Requires that the metaclass is ABCMeta or derived from it. A class that has a metaclass derived from ABCMeta cannot be instantiated unless all of its abstract methods are overridden. The abstract methods can be called using any of the normal ‘super’ call mechanisms. abstractmethod() may be used to declare abstract methods for properties and descriptors.

Usage:

class C(metaclass=ABCMeta):

@abstractmethod def my_abstract_method(self, …):

…

cast(typ, val)[source]

Cast a value to a type.

This returns the value unchanged. To the type checker this signals that the return value has the designated type, but at runtime we intentionally don’t check anything (we want this to be as fast as possible).

class BaseModel(**data)[source]

Bases: object

!!! abstract “Usage Documentation”

[Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__

The names of the class variables defined on the model.

__private_attributes__

Metadata about the private attributes of the model.

__signature__

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__

The core schema of the model.

__pydantic_custom_init__

Whether the model has a custom __init__ function.

__pydantic_decorators__

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__

The name of the post-init method for the model, if defined.

__pydantic_root_model__

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__

A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__

A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to 'allow'.

__pydantic_fields_set__

The names of fields explicitly set during instantiation.

__pydantic_private__

Values of private attributes set on the model instance.

Parameters:

data (typing.Any)

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

__init__(**data)[source]

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

data (typing.Any)

model_fields = {}
model_computed_fields = {}
property model_extra: dict[str, Any] | None

Get extra fields set during validation.

Returns:

A dictionary of extra fields, or None if config.extra is not set to "allow".

property model_fields_set: set[str]

Returns the set of fields that have been explicitly set on this model instance.

Returns:

A set of strings representing the fields that have been set,

i.e. that were not filled from defaults.

classmethod model_construct(_fields_set=None, **values)[source]

Creates a new instance of the Model class with validated data.

Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.

!!! note

model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == 'allow', then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == 'ignore' (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == 'forbid' does not result in an error if extra values are passed, but they will be ignored.

Parameters:

  • _fields_set (Optional[set[str]], default: None) – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.

  • values (typing.Any) – Trusted or pre-validated data dictionary.

Return type:

Self

Returns:

A new instance of the Model class with validated data.

model_copy(*, update=None, deep=False)[source]
!!! abstract “Usage Documentation”

[model_copy](../concepts/models.md#model-copy)

Returns a copy of the model.

!!! note

The underlying instance’s [__dict__][object.__dict__] attribute is copied. This might have unexpected side effects if you store anything in it, on top of the model fields (e.g. the value of [cached properties][functools.cached_property]).

Parameters:

  • update (Optional[Mapping[str, typing.Any]], default: None) – Values to change/add in the new model. Note: the data is not validated before creating the new model. You should trust this data.

  • deep (bool, default: False) – Set to True to make a deep copy of the model.

Return type:

Self

Returns:

New model instance.

model_dump(*, mode='python', include=None, exclude=None, context=None, by_alias=None, exclude_unset=False, exclude_defaults=False, exclude_none=False, exclude_computed_fields=False, round_trip=False, warnings=True, fallback=None, serialize_as_any=False)[source]
!!! abstract “Usage Documentation”

[model_dump](../concepts/serialization.md#python-mode)

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

Parameters:

  • mode (default: 'python') – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.

  • include (default: None) – A set of fields to include in the output.

  • exclude (default: None) – A set of fields to exclude from the output.

  • context (default: None) – Additional context to pass to the serializer.

  • by_alias (default: None) – Whether to use the field’s alias in the dictionary key if defined.

  • exclude_unset (default: False) – Whether to exclude fields that have not been explicitly set.

  • exclude_defaults (default: False) – Whether to exclude fields that are set to their default value.

  • exclude_none (default: False) – Whether to exclude fields that have a value of None.

  • exclude_computed_fields (default: False) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.

  • round_trip (default: False) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].

  • warnings (default: True) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].

  • fallback (default: None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.

  • serialize_as_any (default: False) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A dictionary representation of the model.

model_dump_json(*, indent=None, ensure_ascii=False, include=None, exclude=None, context=None, by_alias=None, exclude_unset=False, exclude_defaults=False, exclude_none=False, exclude_computed_fields=False, round_trip=False, warnings=True, fallback=None, serialize_as_any=False)[source]
!!! abstract “Usage Documentation”

[model_dump_json](../concepts/serialization.md#json-mode)

Generates a JSON representation of the model using Pydantic’s to_json method.

Parameters:

  • indent (default: None) – Indentation to use in the JSON output. If None is passed, the output will be compact.

  • ensure_ascii (default: False) – If True, the output is guaranteed to have all incoming non-ASCII characters escaped. If False (the default), these characters will be output as-is.

  • include (default: None) – Field(s) to include in the JSON output.

  • exclude (default: None) – Field(s) to exclude from the JSON output.

  • context (default: None) – Additional context to pass to the serializer.

  • by_alias (default: None) – Whether to serialize using field aliases.

  • exclude_unset (default: False) – Whether to exclude fields that have not been explicitly set.

  • exclude_defaults (default: False) – Whether to exclude fields that are set to their default value.

  • exclude_none (default: False) – Whether to exclude fields that have a value of None.

  • exclude_computed_fields (default: False) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.

  • round_trip (default: False) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].

  • warnings (default: True) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].

  • fallback (default: None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.

  • serialize_as_any (default: False) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A JSON string representation of the model.

classmethod model_json_schema(by_alias=True, ref_template='#/$defs/{model}', schema_generator=<class 'pydantic.json_schema.GenerateJsonSchema'>, mode='validation', *, union_format='any_of')[source]

Generates a JSON schema for a model class.

Parameters:

  • by_alias (bool, default: True) – Whether to use attribute aliases or not.

  • ref_template (str, default: '#/$defs/{model}') – The reference template.

  • union_format (Literal['any_of', 'primitive_type_array'], default: 'any_of') –

    The format to use when combining schemas from unions together. Can be one of:

    keyword to combine schemas (the default). - 'primitive_type_array': Use the [type](https://json-schema.org/understanding-json-schema/reference/type) keyword as an array of strings, containing each type of the combination. If any of the schemas is not a primitive type (string, boolean, null, integer or number) or contains constraints/metadata, falls back to any_of.

  • schema_generator (type[GenerateJsonSchema], default: <class 'pydantic.json_schema.GenerateJsonSchema'>) – To override the logic used to generate the JSON schema, as a subclass of GenerateJsonSchema with your desired modifications

  • mode (Literal['validation', 'serialization'], default: 'validation') – The mode in which to generate the schema.

Return type:

dict[str, typing.Any]

Returns:

The JSON schema for the given model class.

classmethod model_parametrized_name(params)[source]

Compute the class name for parametrizations of generic classes.

This method can be overridden to achieve a custom naming scheme for generic BaseModels.

Parameters:

params (tuple[type[typing.Any], ...]) – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.

Return type:

str

Returns:

String representing the new class where params are passed to cls as type variables.

Raises:

TypeError – Raised when trying to generate concrete names for non-generic models.

model_post_init(context, /)[source]

Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.

Parameters:

context (typing.Any)

Return type:

None

classmethod model_rebuild(*, force=False, raise_errors=True, _parent_namespace_depth=2, _types_namespace=None)[source]

Try to rebuild the pydantic-core schema for the model.

This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.

Parameters:

  • force (bool, default: False) – Whether to force the rebuilding of the model schema, defaults to False.

  • raise_errors (bool, default: True) – Whether to raise errors, defaults to True.

  • _parent_namespace_depth (int, default: 2) – The depth level of the parent namespace, defaults to 2.

  • _types_namespace (Optional[Mapping[str, Any]], default: None) – The types namespace, defaults to None.

Return type:

bool | None

Returns:

Returns None if the schema is already “complete” and rebuilding was not required. If rebuilding _was_ required, returns True if rebuilding was successful, otherwise False.

classmethod model_validate(obj, *, strict=None, extra=None, from_attributes=None, context=None, by_alias=None, by_name=None)[source]

Validate a pydantic model instance.

Parameters:

  • obj – The object to validate.

  • strict (default: None) – Whether to enforce types strictly.

  • extra (default: None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • from_attributes (default: None) – Whether to extract data from object attributes.

  • context (default: None) – Additional context to pass to the validator.

  • by_alias (default: None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (default: None) – Whether to use the field’s name when validating against the provided input data.

Raises:

ValidationError – If the object could not be validated.

Returns:

The validated model instance.

classmethod model_validate_json(json_data, *, strict=None, extra=None, context=None, by_alias=None, by_name=None)[source]
!!! abstract “Usage Documentation”

[JSON Parsing](../concepts/json.md#json-parsing)

Validate the given JSON data against the Pydantic model.

Parameters:

  • json_data – The JSON data to validate.

  • strict (default: None) – Whether to enforce types strictly.

  • extra (default: None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • context (default: None) – Extra variables to pass to the validator.

  • by_alias (default: None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (default: None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Raises:

ValidationError – If json_data is not a JSON string or the object could not be validated.

classmethod model_validate_strings(obj, *, strict=None, extra=None, context=None, by_alias=None, by_name=None)[source]

Validate the given object with string data against the Pydantic model.

Parameters:

  • obj – The object containing string data to validate.

  • strict (default: None) – Whether to enforce types strictly.

  • extra (default: None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • context (default: None) – Extra variables to pass to the validator.

  • by_alias (default: None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (default: None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

dict(*, include=None, exclude=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False)[source]
Parameters:

  • include (IncEx | None, default: None)

  • exclude (IncEx | None, default: None)

  • by_alias (bool, default: False)

  • exclude_unset (bool, default: False)

  • exclude_defaults (bool, default: False)

  • exclude_none (bool, default: False)

Return type:

Dict[str, Any]

json(*, include=None, exclude=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False, encoder=PydanticUndefined, models_as_dict=PydanticUndefined, **dumps_kwargs)[source]
Parameters:
Return type:

str

classmethod parse_obj(obj)[source]
Parameters:

obj (typing.Any)

Return type:

Self

classmethod parse_raw(b, *, content_type=None, encoding='utf8', proto=None, allow_pickle=False)[source]
Parameters:

  • b (str | bytes)

  • content_type (Optional[str], default: None)

  • encoding (str, default: 'utf8')

  • proto (Optional[Protocol], default: None)

  • allow_pickle (bool, default: False)

Return type:

Self

classmethod parse_file(path, *, content_type=None, encoding='utf8', proto=None, allow_pickle=False)[source]
classmethod from_orm(obj)[source]
Parameters:

obj (typing.Any)

Return type:

Self

classmethod construct(_fields_set=None, **values)[source]
Parameters:
Return type:

Self

copy(*, include=None, exclude=None, update=None, deep=False)[source]

Returns a copy of the model.

!!! warning “Deprecated”

This method is now deprecated; use model_copy instead.

If you need include or exclude, use:

`python {test="skip" lint="skip"} data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `

Parameters:

  • include (AbstractSetIntStr | MappingIntStrAny | None, default: None) – Optional set or mapping specifying which fields to include in the copied model.

  • exclude (AbstractSetIntStr | MappingIntStrAny | None, default: None) – Optional set or mapping specifying which fields to exclude in the copied model.

  • update (Dict[str, typing.Any] | None, default: None) – Optional dictionary of field-value pairs to override field values in the copied model.

  • deep (bool, default: False) – If True, the values of fields that are Pydantic models will be deep-copied.

Return type:

Self

Returns:

A copy of the model with included, excluded and updated fields as specified.

classmethod schema(by_alias=True, ref_template='#/$defs/{model}')[source]
Parameters:

  • by_alias (bool, default: True)

  • ref_template (str, default: '#/$defs/{model}')

Return type:

Dict[str, Any]

classmethod schema_json(*, by_alias=True, ref_template='#/$defs/{model}', **dumps_kwargs)[source]
Parameters:

  • by_alias (bool, default: True)

  • ref_template (str, default: '#/$defs/{model}')

  • dumps_kwargs (typing.Any)

Return type:

str

classmethod validate(value)[source]
Parameters:

value (typing.Any)

Return type:

Self

classmethod update_forward_refs(**localns)[source]
Parameters:

localns (typing.Any)

Return type:

None

tqdm

alias of tqdm_asyncio

class TestType(value)[source]

Bases: str, Enum

Enum for test types.

These values align with the backend TypeLookup table: - SINGLE_TURN: Traditional single request-response tests - MULTI_TURN: Agentic multi-turn conversation tests using Penelope

SINGLE_TURN = 'Single-Turn'
MULTI_TURN = 'Multi-Turn'
get_model(provider=None, model_name=None, api_key=None, config=None, **kwargs)[source]

Create a model instance with smart defaults and comprehensive error handling.

This function provides multiple ways to create a model instance:

  1. Minimal: get_model() - uses all defaults

  2. Provider only: get_model("rhesis") - uses default model for provider

  3. Provider + Model: get_model("rhesis", "rhesis-llm-v1")

  4. Shorthand: get_model("rhesis/rhesis-llm-v1")

  5. Full config: get_model(config=ModelConfig(...))

Parameters:

  • provider (Optional[str], default: None) – Provider name (e.g., “rhesis”, “anthropic”, “gemini”, “openai”, “mistral”, “ollama”)

  • model_name (Optional[str], default: None) – Specific model name

  • api_key (Optional[str], default: None) – API key for authentication

  • config (Optional[ModelConfig], default: None) – Complete configuration object

  • **kwargs – Additional parameters passed to ModelConfig

Returns:

Configured model instance

Return type:

BaseLLM

Raises:

  • ValueError – If configuration is invalid or provider not supported

  • ImportError – If required dependencies are missing

Examples

>>> # Basic usage with defaults
>>> model = get_model()

>>> # Specify provider and model
>>> model = get_model("rhesis", "rhesis-llm-v1")

>>> # Use provider/model shorthand
>>> model = get_model("rhesis/rhesis-llm-v1")

>>> # Use different providers
>>> model = get_model("anthropic", "claude-4")
>>> model = get_model("openai", "gpt-4o")
>>> model = get_model("mistral/mistral-medium-latest")

>>> # With custom configuration
>>> config = ModelConfig(
...     provider="gemini",
...     model_name="gemini-pro",
...     api_key="your-api-key"
... )
>>> model = get_model(config=config)

>>> # With extra parameters
>>> model = get_model(
...     "rhesis",
...     "rhesis-llm-v1",
...     extra_params={"temperature": 0.5}
... )
class BaseLLM(model_name, *args, **kwargs)[source]

Bases: ABC

PROVIDER: str = ''
__init__(model_name, *args, **kwargs)[source]
abstract load_model(*args, **kwargs)[source]

Loads a model

Returns:

A model object

abstract generate(*args, **kwargs)[source]

Runs the model to output LLM response.

Return type:

Union[str, Dict[str, Any]]

Returns:

A string or dict (if schema provided).

abstract generate_batch(*args, **kwargs)[source]

Run model on multiple prompts to output LLM responses.

Return type:

List[Union[str, Dict[str, Any]]]

Returns:

A list of strings or dicts (if schema provided).

get_model_name(*args, **kwargs)[source]
Return type:

str

get_available_models()[source]
Return type:

List[str]

push(name, description=None)[source]

Save this LLM configuration to the Rhesis platform as a Model entity.

Creates a Model entity with this LLM’s provider, model name, and API key, then saves it to the platform.

Parameters:

  • name (str) – Name for the saved model configuration (required)

  • description (Optional[str], default: None) – Optional description for the model

Returns:

The created Model entity (can be used for set_default_generation, etc.)

Return type:

Model

Raises:

ValueError – If provider is not set on this LLM class

Example

>>> from rhesis.sdk.models.factory import get_model
>>> llm = get_model("openai", "gpt-4", api_key="sk-...")
>>> model = llm.push(name="My GPT-4 Production")
>>> model.set_default_generation()
class ChunkingService(sources, strategy)[source]

Bases: object

Chunk sources using a selected chunking strategy.

Parameters:
__init__(sources, strategy)[source]
Parameters:
chunk()[source]
Return type:

List[Chunk]

class ChunkingStrategy[source]

Bases: ABC

Abstract base class for chunkers.

abstract chunk(text)[source]

Chunk the text into a list of chunks.

Parameters:

text (str)

Return type:

List[str]

class SemanticChunker(max_tokens_per_chunk=1500)[source]

Bases: ChunkingStrategy

Service for generating chunks of text from various sources using intelligent semantic chunking.

Parameters:

max_tokens_per_chunk (int, default: 1500)

__init__(max_tokens_per_chunk=1500)[source]

Initialize the context generator.

Parameters:

  • max_context_tokens – Maximum tokens per context (user preference)

  • max_tokens_per_chunk (int, default: 1500)

chunk(text)[source]

Generate contexts using intelligent semantic chunking with hard size limits.

Strategy: 1. Identify semantic boundaries (headers, sections, paragraphs) 2. Create contexts that respect these boundaries 3. Enforce hard token limit; if a semantic span exceeds the context limit, split it abruptly 4. If there are no internal boundaries, slice the text into token-capped windows

Parameters:

text (str)

Return type:

List[str]

class ExtractionService[source]

Bases: object

Service for extracting text from sources.

static extract(sources)[source]
Parameters:

sources (list[SourceSpecification])

Return type:

list[ExtractedSource]

class SourceSpecification(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

type: SourceType
name: Optional[str]
description: Optional[str]
metadata: Optional[dict]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

create_test_set(tests, model, **metadata_kwargs)[source]

Create and configure a TestSet with metadata.

Parameters:
Return type:

TestSet

load_prompt_template(prompt_template_file)[source]

Load prompt template from assets or use custom prompt.

Parameters:

prompt_template_file (str)

Return type:

jinja2.environment.Template

class Prompt(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

content: str
expected_response: str
language_code: str
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class Test(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

prompt: Prompt
behavior: str
category: str
topic: str
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class Tests(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

tests: List[Test]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class FlatTest(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

prompt_content: str
prompt_expected_response: str
prompt_language_code: str
behavior: str
category: str
topic: str
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class FlatTests(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

tests: List[FlatTest]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class TestSetSynthesizer(batch_size=5, model=None, sources=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Bases: ABC

Base class for all test set synthesizers.

Parameters:
prompt_template_file: str
__init__(batch_size=5, model=None, sources=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Initialize the base synthesizer.

Parameters:

  • batch_size (int, default: 5) – Maximum number of items to process in a single LLM call

  • model (Union[str, BaseLLM, None], default: None) – The model to use for generation (string name or BaseLLM instance)

  • sources (Optional[List[SourceSpecification]], default: None) – Optional list of source specifications to extract content from

  • chunking_strategy (Optional[ChunkingStrategy], default: <rhesis.sdk.services.chunker.SemanticChunker object at 0x7a8775d39c90>) – Strategy for chunking source content

generate(num_tests=5, **kwargs)[source]

Generate test cases.

Parameters:

  • num_tests (int, default: 5)

  • kwargs (Any)

Return type:

TestSet

Config Synthesizer

A synthesizer that generates test cases based on a prompt using LLM.

class BaseModel(**data)[source]

Bases: object

!!! abstract “Usage Documentation”

[Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__

The names of the class variables defined on the model.

__private_attributes__

Metadata about the private attributes of the model.

__signature__

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__

The core schema of the model.

__pydantic_custom_init__

Whether the model has a custom __init__ function.

__pydantic_decorators__

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__

The name of the post-init method for the model, if defined.

__pydantic_root_model__

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__

A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__

A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to 'allow'.

__pydantic_fields_set__

The names of fields explicitly set during instantiation.

__pydantic_private__

Values of private attributes set on the model instance.

Parameters:

data (typing.Any)

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

__init__(**data)[source]

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

data (typing.Any)

model_fields = {}
model_computed_fields = {}
property model_extra: dict[str, Any] | None

Get extra fields set during validation.

Returns:

A dictionary of extra fields, or None if config.extra is not set to "allow".

property model_fields_set: set[str]

Returns the set of fields that have been explicitly set on this model instance.

Returns:

A set of strings representing the fields that have been set,

i.e. that were not filled from defaults.

classmethod model_construct(_fields_set=None, **values)[source]

Creates a new instance of the Model class with validated data.

Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.

!!! note

model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == 'allow', then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == 'ignore' (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == 'forbid' does not result in an error if extra values are passed, but they will be ignored.

Parameters:

  • _fields_set (Optional[set[str]], default: None) – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.

  • values (typing.Any) – Trusted or pre-validated data dictionary.

Return type:

Self

Returns:

A new instance of the Model class with validated data.

model_copy(*, update=None, deep=False)[source]
!!! abstract “Usage Documentation”

[model_copy](../concepts/models.md#model-copy)

Returns a copy of the model.

!!! note

The underlying instance’s [__dict__][object.__dict__] attribute is copied. This might have unexpected side effects if you store anything in it, on top of the model fields (e.g. the value of [cached properties][functools.cached_property]).

Parameters:

  • update (Optional[Mapping[str, typing.Any]], default: None) – Values to change/add in the new model. Note: the data is not validated before creating the new model. You should trust this data.

  • deep (bool, default: False) – Set to True to make a deep copy of the model.

Return type:

Self

Returns:

New model instance.

model_dump(*, mode='python', include=None, exclude=None, context=None, by_alias=None, exclude_unset=False, exclude_defaults=False, exclude_none=False, exclude_computed_fields=False, round_trip=False, warnings=True, fallback=None, serialize_as_any=False)[source]
!!! abstract “Usage Documentation”

[model_dump](../concepts/serialization.md#python-mode)

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

Parameters:

  • mode (default: 'python') – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.

  • include (default: None) – A set of fields to include in the output.

  • exclude (default: None) – A set of fields to exclude from the output.

  • context (default: None) – Additional context to pass to the serializer.

  • by_alias (default: None) – Whether to use the field’s alias in the dictionary key if defined.

  • exclude_unset (default: False) – Whether to exclude fields that have not been explicitly set.

  • exclude_defaults (default: False) – Whether to exclude fields that are set to their default value.

  • exclude_none (default: False) – Whether to exclude fields that have a value of None.

  • exclude_computed_fields (default: False) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.

  • round_trip (default: False) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].

  • warnings (default: True) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].

  • fallback (default: None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.

  • serialize_as_any (default: False) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A dictionary representation of the model.

model_dump_json(*, indent=None, ensure_ascii=False, include=None, exclude=None, context=None, by_alias=None, exclude_unset=False, exclude_defaults=False, exclude_none=False, exclude_computed_fields=False, round_trip=False, warnings=True, fallback=None, serialize_as_any=False)[source]
!!! abstract “Usage Documentation”

[model_dump_json](../concepts/serialization.md#json-mode)

Generates a JSON representation of the model using Pydantic’s to_json method.

Parameters:

  • indent (default: None) – Indentation to use in the JSON output. If None is passed, the output will be compact.

  • ensure_ascii (default: False) – If True, the output is guaranteed to have all incoming non-ASCII characters escaped. If False (the default), these characters will be output as-is.

  • include (default: None) – Field(s) to include in the JSON output.

  • exclude (default: None) – Field(s) to exclude from the JSON output.

  • context (default: None) – Additional context to pass to the serializer.

  • by_alias (default: None) – Whether to serialize using field aliases.

  • exclude_unset (default: False) – Whether to exclude fields that have not been explicitly set.

  • exclude_defaults (default: False) – Whether to exclude fields that are set to their default value.

  • exclude_none (default: False) – Whether to exclude fields that have a value of None.

  • exclude_computed_fields (default: False) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.

  • round_trip (default: False) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].

  • warnings (default: True) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].

  • fallback (default: None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.

  • serialize_as_any (default: False) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A JSON string representation of the model.

classmethod model_json_schema(by_alias=True, ref_template='#/$defs/{model}', schema_generator=<class 'pydantic.json_schema.GenerateJsonSchema'>, mode='validation', *, union_format='any_of')[source]

Generates a JSON schema for a model class.

Parameters:

  • by_alias (bool, default: True) – Whether to use attribute aliases or not.

  • ref_template (str, default: '#/$defs/{model}') – The reference template.

  • union_format (Literal['any_of', 'primitive_type_array'], default: 'any_of') –

    The format to use when combining schemas from unions together. Can be one of:

    keyword to combine schemas (the default). - 'primitive_type_array': Use the [type](https://json-schema.org/understanding-json-schema/reference/type) keyword as an array of strings, containing each type of the combination. If any of the schemas is not a primitive type (string, boolean, null, integer or number) or contains constraints/metadata, falls back to any_of.

  • schema_generator (type[GenerateJsonSchema], default: <class 'pydantic.json_schema.GenerateJsonSchema'>) – To override the logic used to generate the JSON schema, as a subclass of GenerateJsonSchema with your desired modifications

  • mode (Literal['validation', 'serialization'], default: 'validation') – The mode in which to generate the schema.

Return type:

dict[str, typing.Any]

Returns:

The JSON schema for the given model class.

classmethod model_parametrized_name(params)[source]

Compute the class name for parametrizations of generic classes.

This method can be overridden to achieve a custom naming scheme for generic BaseModels.

Parameters:

params (tuple[type[typing.Any], ...]) – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.

Return type:

str

Returns:

String representing the new class where params are passed to cls as type variables.

Raises:

TypeError – Raised when trying to generate concrete names for non-generic models.

model_post_init(context, /)[source]

Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.

Parameters:

context (typing.Any)

Return type:

None

classmethod model_rebuild(*, force=False, raise_errors=True, _parent_namespace_depth=2, _types_namespace=None)[source]

Try to rebuild the pydantic-core schema for the model.

This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.

Parameters:

  • force (bool, default: False) – Whether to force the rebuilding of the model schema, defaults to False.

  • raise_errors (bool, default: True) – Whether to raise errors, defaults to True.

  • _parent_namespace_depth (int, default: 2) – The depth level of the parent namespace, defaults to 2.

  • _types_namespace (Optional[Mapping[str, Any]], default: None) – The types namespace, defaults to None.

Return type:

bool | None

Returns:

Returns None if the schema is already “complete” and rebuilding was not required. If rebuilding _was_ required, returns True if rebuilding was successful, otherwise False.

classmethod model_validate(obj, *, strict=None, extra=None, from_attributes=None, context=None, by_alias=None, by_name=None)[source]

Validate a pydantic model instance.

Parameters:

  • obj – The object to validate.

  • strict (default: None) – Whether to enforce types strictly.

  • extra (default: None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • from_attributes (default: None) – Whether to extract data from object attributes.

  • context (default: None) – Additional context to pass to the validator.

  • by_alias (default: None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (default: None) – Whether to use the field’s name when validating against the provided input data.

Raises:

ValidationError – If the object could not be validated.

Returns:

The validated model instance.

classmethod model_validate_json(json_data, *, strict=None, extra=None, context=None, by_alias=None, by_name=None)[source]
!!! abstract “Usage Documentation”

[JSON Parsing](../concepts/json.md#json-parsing)

Validate the given JSON data against the Pydantic model.

Parameters:

  • json_data – The JSON data to validate.

  • strict (default: None) – Whether to enforce types strictly.

  • extra (default: None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • context (default: None) – Extra variables to pass to the validator.

  • by_alias (default: None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (default: None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Raises:

ValidationError – If json_data is not a JSON string or the object could not be validated.

classmethod model_validate_strings(obj, *, strict=None, extra=None, context=None, by_alias=None, by_name=None)[source]

Validate the given object with string data against the Pydantic model.

Parameters:

  • obj – The object containing string data to validate.

  • strict (default: None) – Whether to enforce types strictly.

  • extra (default: None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • context (default: None) – Extra variables to pass to the validator.

  • by_alias (default: None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (default: None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

dict(*, include=None, exclude=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False)[source]
Parameters:

  • include (IncEx | None, default: None)

  • exclude (IncEx | None, default: None)

  • by_alias (bool, default: False)

  • exclude_unset (bool, default: False)

  • exclude_defaults (bool, default: False)

  • exclude_none (bool, default: False)

Return type:

Dict[str, Any]

json(*, include=None, exclude=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False, encoder=PydanticUndefined, models_as_dict=PydanticUndefined, **dumps_kwargs)[source]
Parameters:
Return type:

str

classmethod parse_obj(obj)[source]
Parameters:

obj (typing.Any)

Return type:

Self

classmethod parse_raw(b, *, content_type=None, encoding='utf8', proto=None, allow_pickle=False)[source]
Parameters:

  • b (str | bytes)

  • content_type (Optional[str], default: None)

  • encoding (str, default: 'utf8')

  • proto (Optional[Protocol], default: None)

  • allow_pickle (bool, default: False)

Return type:

Self

classmethod parse_file(path, *, content_type=None, encoding='utf8', proto=None, allow_pickle=False)[source]
classmethod from_orm(obj)[source]
Parameters:

obj (typing.Any)

Return type:

Self

classmethod construct(_fields_set=None, **values)[source]
Parameters:
Return type:

Self

copy(*, include=None, exclude=None, update=None, deep=False)[source]

Returns a copy of the model.

!!! warning “Deprecated”

This method is now deprecated; use model_copy instead.

If you need include or exclude, use:

`python {test="skip" lint="skip"} data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `

Parameters:

  • include (AbstractSetIntStr | MappingIntStrAny | None, default: None) – Optional set or mapping specifying which fields to include in the copied model.

  • exclude (AbstractSetIntStr | MappingIntStrAny | None, default: None) – Optional set or mapping specifying which fields to exclude in the copied model.

  • update (Dict[str, typing.Any] | None, default: None) – Optional dictionary of field-value pairs to override field values in the copied model.

  • deep (bool, default: False) – If True, the values of fields that are Pydantic models will be deep-copied.

Return type:

Self

Returns:

A copy of the model with included, excluded and updated fields as specified.

classmethod schema(by_alias=True, ref_template='#/$defs/{model}')[source]
Parameters:

  • by_alias (bool, default: True)

  • ref_template (str, default: '#/$defs/{model}')

Return type:

Dict[str, Any]

classmethod schema_json(*, by_alias=True, ref_template='#/$defs/{model}', **dumps_kwargs)[source]
Parameters:

  • by_alias (bool, default: True)

  • ref_template (str, default: '#/$defs/{model}')

  • dumps_kwargs (typing.Any)

Return type:

str

classmethod validate(value)[source]
Parameters:

value (typing.Any)

Return type:

Self

classmethod update_forward_refs(**localns)[source]
Parameters:

localns (typing.Any)

Return type:

None

class BaseLLM(model_name, *args, **kwargs)[source]

Bases: ABC

PROVIDER: str = ''
__init__(model_name, *args, **kwargs)[source]
abstract load_model(*args, **kwargs)[source]

Loads a model

Returns:

A model object

abstract generate(*args, **kwargs)[source]

Runs the model to output LLM response.

Return type:

Union[str, Dict[str, Any]]

Returns:

A string or dict (if schema provided).

abstract generate_batch(*args, **kwargs)[source]

Run model on multiple prompts to output LLM responses.

Return type:

List[Union[str, Dict[str, Any]]]

Returns:

A list of strings or dicts (if schema provided).

get_model_name(*args, **kwargs)[source]
Return type:

str

get_available_models()[source]
Return type:

List[str]

push(name, description=None)[source]

Save this LLM configuration to the Rhesis platform as a Model entity.

Creates a Model entity with this LLM’s provider, model name, and API key, then saves it to the platform.

Parameters:

  • name (str) – Name for the saved model configuration (required)

  • description (Optional[str], default: None) – Optional description for the model

Returns:

The created Model entity (can be used for set_default_generation, etc.)

Return type:

Model

Raises:

ValueError – If provider is not set on this LLM class

Example

>>> from rhesis.sdk.models.factory import get_model
>>> llm = get_model("openai", "gpt-4", api_key="sk-...")
>>> model = llm.push(name="My GPT-4 Production")
>>> model.set_default_generation()
class ChunkingStrategy[source]

Bases: ABC

Abstract base class for chunkers.

abstract chunk(text)[source]

Chunk the text into a list of chunks.

Parameters:

text (str)

Return type:

List[str]

class SemanticChunker(max_tokens_per_chunk=1500)[source]

Bases: ChunkingStrategy

Service for generating chunks of text from various sources using intelligent semantic chunking.

Parameters:

max_tokens_per_chunk (int, default: 1500)

__init__(max_tokens_per_chunk=1500)[source]

Initialize the context generator.

Parameters:

  • max_context_tokens – Maximum tokens per context (user preference)

  • max_tokens_per_chunk (int, default: 1500)

chunk(text)[source]

Generate contexts using intelligent semantic chunking with hard size limits.

Strategy: 1. Identify semantic boundaries (headers, sections, paragraphs) 2. Create contexts that respect these boundaries 3. Enforce hard token limit; if a semantic span exceeds the context limit, split it abruptly 4. If there are no internal boundaries, slice the text into token-capped windows

Parameters:

text (str)

Return type:

List[str]

class SourceSpecification(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

type: SourceType
name: Optional[str]
description: Optional[str]
metadata: Optional[dict]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class TestSetSynthesizer(batch_size=5, model=None, sources=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Bases: ABC

Base class for all test set synthesizers.

Parameters:
prompt_template_file: str
__init__(batch_size=5, model=None, sources=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Initialize the base synthesizer.

Parameters:

  • batch_size (int, default: 5) – Maximum number of items to process in a single LLM call

  • model (Union[str, BaseLLM, None], default: None) – The model to use for generation (string name or BaseLLM instance)

  • sources (Optional[List[SourceSpecification]], default: None) – Optional list of source specifications to extract content from

  • chunking_strategy (Optional[ChunkingStrategy], default: <rhesis.sdk.services.chunker.SemanticChunker object at 0x7a8775d39c90>) – Strategy for chunking source content

generate(num_tests=5, **kwargs)[source]

Generate test cases.

Parameters:

  • num_tests (int, default: 5)

  • kwargs (Any)

Return type:

TestSet

class GenerationConfig(**data)[source]

Bases: BaseModel

Dataclass representing generation config information.

Parameters:

data (typing.Any)

generation_prompt: Optional[str]
behaviors: Optional[List[str]]
categories: Optional[List[str]]
topics: Optional[List[str]]
additional_context: Optional[str]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class ConfigSynthesizer(config, batch_size=20, model=None, sources=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Bases: TestSetSynthesizer

A synthesizer that generates test cases based on a generation config using LLM.

Parameters:
prompt_template_file: str = 'config_synthesizer.jinja'
__init__(config, batch_size=20, model=None, sources=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Initialize the ConfigSynthesizer. :type config: GenerationConfig :param config: The generation config to use :type batch_size: int, default: 20 :param batch_size: Maximum number of tests to generate in a single LLM call :type model: Union[str, BaseLLM, None], default: None :param model: The model to use for generation :type sources: Optional[List[SourceSpecification]], default: None :param sources: Optional list of source specifications to use :type chunking_strategy: Optional[ChunkingStrategy], default: <rhesis.sdk.services.chunker.SemanticChunker object at 0x7a8775d38070> :param chunking_strategy: Strategy for chunking source content

Context Synthesizer

A synthesizer that generates test cases based on a prompt using LLM.

class BaseLLM(model_name, *args, **kwargs)[source]

Bases: ABC

PROVIDER: str = ''
__init__(model_name, *args, **kwargs)[source]
abstract load_model(*args, **kwargs)[source]

Loads a model

Returns:

A model object

abstract generate(*args, **kwargs)[source]

Runs the model to output LLM response.

Return type:

Union[str, Dict[str, Any]]

Returns:

A string or dict (if schema provided).

abstract generate_batch(*args, **kwargs)[source]

Run model on multiple prompts to output LLM responses.

Return type:

List[Union[str, Dict[str, Any]]]

Returns:

A list of strings or dicts (if schema provided).

get_model_name(*args, **kwargs)[source]
Return type:

str

get_available_models()[source]
Return type:

List[str]

push(name, description=None)[source]

Save this LLM configuration to the Rhesis platform as a Model entity.

Creates a Model entity with this LLM’s provider, model name, and API key, then saves it to the platform.

Parameters:

  • name (str) – Name for the saved model configuration (required)

  • description (Optional[str], default: None) – Optional description for the model

Returns:

The created Model entity (can be used for set_default_generation, etc.)

Return type:

Model

Raises:

ValueError – If provider is not set on this LLM class

Example

>>> from rhesis.sdk.models.factory import get_model
>>> llm = get_model("openai", "gpt-4", api_key="sk-...")
>>> model = llm.push(name="My GPT-4 Production")
>>> model.set_default_generation()
class TestSetSynthesizer(batch_size=5, model=None, sources=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Bases: ABC

Base class for all test set synthesizers.

Parameters:
prompt_template_file: str
__init__(batch_size=5, model=None, sources=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Initialize the base synthesizer.

Parameters:

  • batch_size (int, default: 5) – Maximum number of items to process in a single LLM call

  • model (Union[str, BaseLLM, None], default: None) – The model to use for generation (string name or BaseLLM instance)

  • sources (Optional[List[SourceSpecification]], default: None) – Optional list of source specifications to extract content from

  • chunking_strategy (Optional[ChunkingStrategy], default: <rhesis.sdk.services.chunker.SemanticChunker object at 0x7a8775d39c90>) – Strategy for chunking source content

generate(num_tests=5, **kwargs)[source]

Generate test cases.

Parameters:

  • num_tests (int, default: 5)

  • kwargs (Any)

Return type:

TestSet

class ContextSynthesizer(prompt, batch_size=20, model=None)[source]

Bases: TestSetSynthesizer

A synthesizer that generates test cases based on a prompt using LLM.

Parameters:
prompt_template_file: str = 'context_synthesizer.jinja'
__init__(prompt, batch_size=20, model=None)[source]

Initialize the context synthesizer. :type prompt: str :param prompt: The generation prompt to use :type batch_size: int, default: 20 :param batch_size: Maximum number of tests to generate in a single LLM call :type model: Union[str, BaseLLM, None], default: None :param model: The model to use for generation

Prompt Synthesizer

A synthesizer that generates test cases based on a prompt using LLM.

class BaseLLM(model_name, *args, **kwargs)[source]

Bases: ABC

PROVIDER: str = ''
__init__(model_name, *args, **kwargs)[source]
abstract load_model(*args, **kwargs)[source]

Loads a model

Returns:

A model object

abstract generate(*args, **kwargs)[source]

Runs the model to output LLM response.

Return type:

Union[str, Dict[str, Any]]

Returns:

A string or dict (if schema provided).

abstract generate_batch(*args, **kwargs)[source]

Run model on multiple prompts to output LLM responses.

Return type:

List[Union[str, Dict[str, Any]]]

Returns:

A list of strings or dicts (if schema provided).

get_model_name(*args, **kwargs)[source]
Return type:

str

get_available_models()[source]
Return type:

List[str]

push(name, description=None)[source]

Save this LLM configuration to the Rhesis platform as a Model entity.

Creates a Model entity with this LLM’s provider, model name, and API key, then saves it to the platform.

Parameters:

  • name (str) – Name for the saved model configuration (required)

  • description (Optional[str], default: None) – Optional description for the model

Returns:

The created Model entity (can be used for set_default_generation, etc.)

Return type:

Model

Raises:

ValueError – If provider is not set on this LLM class

Example

>>> from rhesis.sdk.models.factory import get_model
>>> llm = get_model("openai", "gpt-4", api_key="sk-...")
>>> model = llm.push(name="My GPT-4 Production")
>>> model.set_default_generation()
class ChunkingStrategy[source]

Bases: ABC

Abstract base class for chunkers.

abstract chunk(text)[source]

Chunk the text into a list of chunks.

Parameters:

text (str)

Return type:

List[str]

class SemanticChunker(max_tokens_per_chunk=1500)[source]

Bases: ChunkingStrategy

Service for generating chunks of text from various sources using intelligent semantic chunking.

Parameters:

max_tokens_per_chunk (int, default: 1500)

__init__(max_tokens_per_chunk=1500)[source]

Initialize the context generator.

Parameters:

  • max_context_tokens – Maximum tokens per context (user preference)

  • max_tokens_per_chunk (int, default: 1500)

chunk(text)[source]

Generate contexts using intelligent semantic chunking with hard size limits.

Strategy: 1. Identify semantic boundaries (headers, sections, paragraphs) 2. Create contexts that respect these boundaries 3. Enforce hard token limit; if a semantic span exceeds the context limit, split it abruptly 4. If there are no internal boundaries, slice the text into token-capped windows

Parameters:

text (str)

Return type:

List[str]

class SourceSpecification(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

type: SourceType
name: Optional[str]
description: Optional[str]
metadata: Optional[dict]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class TestSetSynthesizer(batch_size=5, model=None, sources=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Bases: ABC

Base class for all test set synthesizers.

Parameters:
prompt_template_file: str
__init__(batch_size=5, model=None, sources=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Initialize the base synthesizer.

Parameters:

  • batch_size (int, default: 5) – Maximum number of items to process in a single LLM call

  • model (Union[str, BaseLLM, None], default: None) – The model to use for generation (string name or BaseLLM instance)

  • sources (Optional[List[SourceSpecification]], default: None) – Optional list of source specifications to extract content from

  • chunking_strategy (Optional[ChunkingStrategy], default: <rhesis.sdk.services.chunker.SemanticChunker object at 0x7a8775d39c90>) – Strategy for chunking source content

generate(num_tests=5, **kwargs)[source]

Generate test cases.

Parameters:

  • num_tests (int, default: 5)

  • kwargs (Any)

Return type:

TestSet

class PromptSynthesizer(prompt, batch_size=20, sources=None, model=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Bases: TestSetSynthesizer

A synthesizer that generates test cases based on a prompt.

Parameters:
prompt_template_file: str = 'prompt_synthesizer.jinja'
__init__(prompt, batch_size=20, sources=None, model=None, chunking_strategy=<rhesis.sdk.services.chunker.SemanticChunker object>)[source]

Initialize the prompt synthesizer. :type prompt: str :param prompt: The generation prompt to use :type batch_size: int, default: 20 :param batch_size: Maximum number of tests to generate in a single LLM call :type sources: Optional[List[SourceSpecification]], default: None :param sources: Optional list of source specifications to use :type model: Union[str, BaseLLM, None], default: None :param model: The model to use for generation :type chunking_strategy: Optional[ChunkingStrategy], default: <rhesis.sdk.services.chunker.SemanticChunker object at 0x7a8775d3ef50> :param chunking_strategy: Strategy for chunking source content

Multi-Turn Synthesizer

class Environment(block_start_string='{%', block_end_string='%}', variable_start_string='{{', variable_end_string='}}', comment_start_string='{#', comment_end_string='#}', line_statement_prefix=None, line_comment_prefix=None, trim_blocks=False, lstrip_blocks=False, newline_sequence='\\n', keep_trailing_newline=False, extensions=(), optimized=True, undefined=<class 'jinja2.runtime.Undefined'>, finalize=None, autoescape=False, loader=None, cache_size=400, auto_reload=True, bytecode_cache=None, enable_async=False)[source]

Bases: object

The core component of Jinja is the Environment. It contains important shared variables like configuration, filters, tests, globals and others. Instances of this class may be modified if they are not shared and if no template was loaded so far. Modifications on environments after the first template was loaded will lead to surprising effects and undefined behavior.

Here are the possible initialization parameters:

block_start_string

The string marking the beginning of a block. Defaults to '{%'.

block_end_string

The string marking the end of a block. Defaults to '%}'.

variable_start_string

The string marking the beginning of a print statement. Defaults to '{{'.

variable_end_string

The string marking the end of a print statement. Defaults to '}}'.

comment_start_string

The string marking the beginning of a comment. Defaults to '{#'.

comment_end_string

The string marking the end of a comment. Defaults to '#}'.

line_statement_prefix

If given and a string, this will be used as prefix for line based statements. See also Line Statements.

line_comment_prefix

If given and a string, this will be used as prefix for line based comments. See also Line Statements.

Added in version 2.2.

trim_blocks

If this is set to True the first newline after a block is removed (block, not variable tag!). Defaults to False.

lstrip_blocks

If this is set to True leading spaces and tabs are stripped from the start of a line to a block. Defaults to False.

newline_sequence

The sequence that starts a newline. Must be one of '\r', '\n' or '\r\n'. The default is '\n' which is a useful default for Linux and OS X systems as well as web applications.

keep_trailing_newline

Preserve the trailing newline when rendering templates. The default is False, which causes a single newline, if present, to be stripped from the end of the template.

Added in version 2.7.

extensions

List of Jinja extensions to use. This can either be import paths as strings or extension classes. For more information have a look at the extensions documentation.

optimized

should the optimizer be enabled? Default is True.

undefined

Undefined or a subclass of it that is used to represent undefined values in the template.

finalize

A callable that can be used to process the result of a variable expression before it is output. For example one can convert None implicitly into an empty string here.

autoescape

If set to True the XML/HTML autoescaping feature is enabled by default. For more details about autoescaping see Markup. As of Jinja 2.4 this can also be a callable that is passed the template name and has to return True or False depending on autoescape should be enabled by default.

Changed in version 2.4: autoescape can now be a function

loader

The template loader for this environment.

cache_size

The size of the cache. Per default this is 400 which means that if more than 400 templates are loaded the loader will clean out the least recently used template. If the cache size is set to 0 templates are recompiled all the time, if the cache size is -1 the cache will not be cleaned.

Changed in version 2.8: The cache size was increased to 400 from a low 50.

auto_reload

Some loaders load templates from locations where the template sources may change (ie: file system or database). If auto_reload is set to True (default) every time a template is requested the loader checks if the source changed and if yes, it will reload the template. For higher performance it’s possible to disable that.

bytecode_cache

If set to a bytecode cache object, this object will provide a cache for the internal Jinja bytecode so that templates don’t have to be parsed if they were not changed.

See Bytecode Cache for more information.

enable_async

If set to true this enables async template execution which allows using async functions and generators.

Parameters:

  • block_start_string (str, default: '{%')

  • block_end_string (str, default: '%}')

  • variable_start_string (str, default: '{{')

  • variable_end_string (str, default: '}}')

  • comment_start_string (str, default: '{#')

  • comment_end_string (str, default: '#}')

  • line_statement_prefix (Optional[str], default: None)

  • line_comment_prefix (Optional[str], default: None)

  • trim_blocks (bool, default: False)

  • lstrip_blocks (bool, default: False)

  • newline_sequence (te.Literal[‘n’, ‘rn’, ‘r’], default: '\\n')

  • keep_trailing_newline (bool, default: False)

  • extensions (Sequence[Union[str, Type[Extension]]], default: ())

  • optimized (bool, default: True)

  • undefined (Type[Undefined], default: <class 'jinja2.runtime.Undefined'>)

  • finalize (Optional[Callable[..., Any]], default: None)

  • autoescape (Union[bool, Callable[[Optional[str]], bool]], default: False)

  • loader (Optional[BaseLoader], default: None)

  • cache_size (int, default: 400)

  • auto_reload (bool, default: True)

  • bytecode_cache (Optional[BytecodeCache], default: None)

  • enable_async (bool, default: False)

sandboxed = False

if this environment is sandboxed. Modifying this variable won’t make the environment sandboxed though. For a real sandboxed environment have a look at jinja2.sandbox. This flag alone controls the code generation by the compiler.

overlayed = False

True if the environment is just an overlay

linked_to: Optional[Environment] = None

the environment this environment is linked to if it is an overlay

shared = False

shared environments have this set to True. A shared environment must not be modified

code_generator_class

alias of CodeGenerator

concat()

Concatenate any number of strings.

The string whose method is called is inserted in between each given string. The result is returned as a new string.

Example: ‘.’.join([‘ab’, ‘pq’, ‘rs’]) -> ‘ab.pq.rs’

context_class

alias of Context

template_class

alias of Template

__init__(block_start_string='{%', block_end_string='%}', variable_start_string='{{', variable_end_string='}}', comment_start_string='{#', comment_end_string='#}', line_statement_prefix=None, line_comment_prefix=None, trim_blocks=False, lstrip_blocks=False, newline_sequence='\\n', keep_trailing_newline=False, extensions=(), optimized=True, undefined=<class 'jinja2.runtime.Undefined'>, finalize=None, autoescape=False, loader=None, cache_size=400, auto_reload=True, bytecode_cache=None, enable_async=False)[source]
Parameters:

  • block_start_string (str, default: '{%')

  • block_end_string (str, default: '%}')

  • variable_start_string (str, default: '{{')

  • variable_end_string (str, default: '}}')

  • comment_start_string (str, default: '{#')

  • comment_end_string (str, default: '#}')

  • line_statement_prefix (Optional[str], default: None)

  • line_comment_prefix (Optional[str], default: None)

  • trim_blocks (bool, default: False)

  • lstrip_blocks (bool, default: False)

  • newline_sequence (te.Literal[‘n’, ‘rn’, ‘r’], default: '\\n')

  • keep_trailing_newline (bool, default: False)

  • extensions (Sequence[Union[str, Type[Extension]]], default: ())

  • optimized (bool, default: True)

  • undefined (Type[Undefined], default: <class 'jinja2.runtime.Undefined'>)

  • finalize (Optional[Callable[..., Any]], default: None)

  • autoescape (Union[bool, Callable[[Optional[str]], bool]], default: False)

  • loader (Optional[BaseLoader], default: None)

  • cache_size (int, default: 400)

  • auto_reload (bool, default: True)

  • bytecode_cache (Optional[BytecodeCache], default: None)

  • enable_async (bool, default: False)

add_extension(extension)[source]

Adds an extension after the environment was created.

Added in version 2.5.

Parameters:

extension (Union[str, Type[Extension]])

Return type:

None

extend(**attributes)[source]

Add the items to the instance of the environment if they do not exist yet. This is used by extensions to register callbacks and configuration values without breaking inheritance.

Parameters:

attributes (Any)

Return type:

None

overlay(block_start_string=missing, block_end_string=missing, variable_start_string=missing, variable_end_string=missing, comment_start_string=missing, comment_end_string=missing, line_statement_prefix=missing, line_comment_prefix=missing, trim_blocks=missing, lstrip_blocks=missing, newline_sequence=missing, keep_trailing_newline=missing, extensions=missing, optimized=missing, undefined=missing, finalize=missing, autoescape=missing, loader=missing, cache_size=missing, auto_reload=missing, bytecode_cache=missing, enable_async=missing)[source]

Create a new overlay environment that shares all the data with the current environment except for cache and the overridden attributes. Extensions cannot be removed for an overlayed environment. An overlayed environment automatically gets all the extensions of the environment it is linked to plus optional extra extensions.

Creating overlays should happen after the initial environment was set up completely. Not all attributes are truly linked, some are just copied over so modifications on the original environment may not shine through.

Changed in version 3.1.5: enable_async is applied correctly.

Changed in version 3.1.2: Added the newline_sequence, keep_trailing_newline, and enable_async parameters to match __init__.

Parameters:

  • block_start_string (str, default: missing)

  • block_end_string (str, default: missing)

  • variable_start_string (str, default: missing)

  • variable_end_string (str, default: missing)

  • comment_start_string (str, default: missing)

  • comment_end_string (str, default: missing)

  • line_statement_prefix (Optional[str], default: missing)

  • line_comment_prefix (Optional[str], default: missing)

  • trim_blocks (bool, default: missing)

  • lstrip_blocks (bool, default: missing)

  • newline_sequence (te.Literal[‘n’, ‘rn’, ‘r’], default: missing)

  • keep_trailing_newline (bool, default: missing)

  • extensions (Sequence[Union[str, Type[Extension]]], default: missing)

  • optimized (bool, default: missing)

  • undefined (Type[Undefined], default: missing)

  • finalize (Optional[Callable[..., Any]], default: missing)

  • autoescape (Union[bool, Callable[[Optional[str]], bool]], default: missing)

  • loader (Optional[BaseLoader], default: missing)

  • cache_size (int, default: missing)

  • auto_reload (bool, default: missing)

  • bytecode_cache (Optional[BytecodeCache], default: missing)

  • enable_async (bool, default: missing)

Return type:

te.Self

property lexer: Lexer

The lexer for this environment.

iter_extensions()[source]

Iterates over the extensions by priority.

Return type:

Iterator[Extension]

getitem(obj, argument)[source]

Get an item or attribute of an object but prefer the item.

Parameters:
Return type:

Union[Any, Undefined]

getattr(obj, attribute)[source]

Get an item or attribute of an object but prefer the attribute. Unlike getitem() the attribute must be a string.

Parameters:
Return type:

Any

call_filter(name, value, args=None, kwargs=None, context=None, eval_ctx=None)[source]

Invoke a filter on a value the same way the compiler does.

This might return a coroutine if the filter is running from an environment in async mode and the filter supports async execution. It’s your responsibility to await this if needed.

Added in version 2.7.

Parameters:
Return type:

Any

call_test(name, value, args=None, kwargs=None, context=None, eval_ctx=None)[source]

Invoke a test on a value the same way the compiler does.

This might return a coroutine if the test is running from an environment in async mode and the test supports async execution. It’s your responsibility to await this if needed.

Changed in version 3.0: Tests support @pass_context, etc. decorators. Added the context and eval_ctx parameters.

Added in version 2.7.

Parameters:
Return type:

Any

parse(source, name=None, filename=None)[source]

Parse the sourcecode and return the abstract syntax tree. This tree of nodes is used by the compiler to convert the template into executable source- or bytecode. This is useful for debugging or to extract information from templates.

If you are developing Jinja extensions this gives you a good overview of the node tree generated.

Parameters:
Return type:

Template

lex(source, name=None, filename=None)[source]

Lex the given sourcecode and return a generator that yields tokens as tuples in the form (lineno, token_type, value). This can be useful for extension development and debugging templates.

This does not perform preprocessing. If you want the preprocessing of the extensions to be applied you have to filter source through the preprocess() method.

Parameters:
Return type:

Iterator[Tuple[int, str, str]]

preprocess(source, name=None, filename=None)[source]

Preprocesses the source with all extensions. This is automatically called for all parsing and compiling methods but not for lex() because there you usually only want the actual source tokenized.

Parameters:
Return type:

str

compile(source, name=None, filename=None, raw=False, defer_init=False)[source]

Compile a node or template source code. The name parameter is the load name of the template after it was joined using join_path() if necessary, not the filename on the file system. the filename parameter is the estimated filename of the template on the file system. If the template came from a database or memory this can be omitted.

The return value of this method is a python code object. If the raw parameter is True the return value will be a string with python code equivalent to the bytecode returned otherwise. This method is mainly used internally.

defer_init is use internally to aid the module code generator. This causes the generated code to be able to import without the global environment variable to be set.

Added in version 2.4: defer_init parameter added.

Parameters:
Return type:

Union[str, CodeType]

compile_expression(source, undefined_to_none=True)[source]

A handy helper method that returns a callable that accepts keyword arguments that appear as variables in the expression. If called it returns the result of the expression.

This is useful if applications want to use the same rules as Jinja in template “configuration files” or similar situations.

Example usage:

>>> env = Environment()
>>> expr = env.compile_expression('foo == 42')
>>> expr(foo=23)
False
>>> expr(foo=42)
True

Per default the return value is converted to None if the expression returns an undefined value. This can be changed by setting undefined_to_none to False.

>>> env.compile_expression('var')() is None
True
>>> env.compile_expression('var', undefined_to_none=False)()
Undefined

Added in version 2.1.

Parameters:

  • source (str)

  • undefined_to_none (bool, default: True)

Return type:

TemplateExpression

compile_templates(target, extensions=None, filter_func=None, zip='deflated', log_function=None, ignore_errors=True)[source]

Finds all the templates the loader can find, compiles them and stores them in target. If zip is None, instead of in a zipfile, the templates will be stored in a directory. By default a deflate zip algorithm is used. To switch to the stored algorithm, zip can be set to 'stored'.

extensions and filter_func are passed to list_templates(). Each template returned will be compiled to the target folder or zipfile.

By default template compilation errors are ignored. In case a log function is provided, errors are logged. If you want template syntax errors to abort the compilation you can set ignore_errors to False and you will get an exception on syntax errors.

Added in version 2.4.

Parameters:
Return type:

None

list_templates(extensions=None, filter_func=None)[source]

Returns a list of templates for this environment. This requires that the loader supports the loader’s list_templates() method.

If there are other files in the template folder besides the actual templates, the returned list can be filtered. There are two ways: either extensions is set to a list of file extensions for templates, or a filter_func can be provided which is a callable that is passed a template name and should return True if it should end up in the result list.

If the loader does not support that, a TypeError is raised.

Added in version 2.4.

Parameters:
Return type:

List[str]

handle_exception(source=None)[source]

Exception handling helper. This is used internally to either raise rewritten exceptions or return a rendered traceback for the template.

Parameters:

source (Optional[str], default: None)

Return type:

te.NoReturn

join_path(template, parent)[source]

Join a template with the parent. By default all the lookups are relative to the loader root so this method returns the template parameter unchanged, but if the paths should be relative to the parent template, this function can be used to calculate the real template name.

Subclasses may override this method and implement template path joining here.

Parameters:
Return type:

str

get_template(name, parent=None, globals=None)[source]

Load a template by name with loader and return a Template. If the template does not exist a TemplateNotFound exception is raised.

Parameters:

  • name (Union[str, jinja2.environment.Template]) – Name of the template to load. When loading templates from the filesystem, “/” is used as the path separator, even on Windows.

  • parent (Optional[str], default: None) – The name of the parent template importing this template. join_path() can be used to implement name transformations with this.

  • globals (Optional[MutableMapping[str, Any]], default: None) – Extend the environment globals with these extra variables available for all renders of this template. If the template has already been loaded and cached, its globals are updated with any new items.

Return type:

jinja2.environment.Template

Changed in version 3.0: If a template is loaded from cache, globals will update the template’s globals instead of ignoring the new values.

Changed in version 2.4: If name is a Template object it is returned unchanged.

select_template(names, parent=None, globals=None)[source]

Like get_template(), but tries loading multiple names. If none of the names can be loaded a TemplatesNotFound exception is raised.

Parameters:

  • names (Iterable[Union[str, jinja2.environment.Template]]) – List of template names to try loading in order.

  • parent (Optional[str], default: None) – The name of the parent template importing this template. join_path() can be used to implement name transformations with this.

  • globals (Optional[MutableMapping[str, Any]], default: None) – Extend the environment globals with these extra variables available for all renders of this template. If the template has already been loaded and cached, its globals are updated with any new items.

Return type:

jinja2.environment.Template

Changed in version 3.0: If a template is loaded from cache, globals will update the template’s globals instead of ignoring the new values.

Changed in version 2.11: If names is Undefined, an UndefinedError is raised instead. If no templates were found and names contains Undefined, the message is more helpful.

Changed in version 2.4: If names contains a Template object it is returned unchanged.

Added in version 2.3.

get_or_select_template(template_name_or_list, parent=None, globals=None)[source]

Use select_template() if an iterable of template names is given, or get_template() if one name is given.

Added in version 2.3.

Parameters:
Return type:

jinja2.environment.Template

from_string(source, globals=None, template_class=None)[source]

Load a template from a source string without using loader.

Parameters:

  • source (Union[str, Template]) – Jinja source to compile into a template.

  • globals (Optional[MutableMapping[str, Any]], default: None) – Extend the environment globals with these extra variables available for all renders of this template. If the template has already been loaded and cached, its globals are updated with any new items.

  • template_class (Optional[Type[jinja2.environment.Template]], default: None) – Return an instance of this Template class.

Return type:

jinja2.environment.Template

make_globals(d)[source]

Make the globals map for a template. Any given template globals overlay the environment globals.

Returns a collections.ChainMap. This allows any changes to a template’s globals to only affect that template, while changes to the environment’s globals are still reflected. However, avoid modifying any globals after a template is loaded.

Parameters:

d (Optional[MutableMapping[str, Any]]) – Dict of template-specific globals.

Return type:

MutableMapping[str, Any]

Changed in version 3.0: Use collections.ChainMap to always prevent mutating environment globals.

class FileSystemLoader(searchpath, encoding='utf-8', followlinks=False)[source]

Bases: BaseLoader

Load templates from a directory in the file system.

The path can be relative or absolute. Relative paths are relative to the current working directory.

loader = FileSystemLoader("templates")

A list of paths can be given. The directories will be searched in order, stopping at the first matching template.

loader = FileSystemLoader(["/override/templates", "/default/templates"])
Parameters:

  • searchpath (Union[str, PathLike[str], Sequence[Union[str, PathLike[str]]]]) – A path, or list of paths, to the directory that contains the templates.

  • encoding (str, default: 'utf-8') – Use this encoding to read the text from template files.

  • followlinks (bool, default: False) – Follow symbolic links in the path.

Changed in version 2.8: Added the followlinks parameter.

__init__(searchpath, encoding='utf-8', followlinks=False)[source]
Parameters:
get_source(environment, template)[source]

Get the template source, filename and reload helper for a template. It’s passed the environment and template name and has to return a tuple in the form (source, filename, uptodate) or raise a TemplateNotFound error if it can’t locate the template.

The source part of the returned tuple must be the source of the template as a string. The filename should be the name of the file on the filesystem if it was loaded from there, otherwise None. The filename is used by Python for the tracebacks if no loader extension is used.

The last item in the tuple is the uptodate function. If auto reloading is enabled it’s always called to check if the template changed. No arguments are passed so the function must store the old state somewhere (for example in a closure). If it returns False the template will be reloaded.

Parameters:

  • environment (Environment)

  • template (str)

Return type:

Tuple[str, str, Callable[[], bool]]

list_templates()[source]

Iterates over all templates. If the loader does not support that it should raise a TypeError which is the default behavior.

Return type:

List[str]

class BaseModel(**data)[source]

Bases: object

!!! abstract “Usage Documentation”

[Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__

The names of the class variables defined on the model.

__private_attributes__

Metadata about the private attributes of the model.

__signature__

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__

The core schema of the model.

__pydantic_custom_init__

Whether the model has a custom __init__ function.

__pydantic_decorators__

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__

The name of the post-init method for the model, if defined.

__pydantic_root_model__

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__

A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__

A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to 'allow'.

__pydantic_fields_set__

The names of fields explicitly set during instantiation.

__pydantic_private__

Values of private attributes set on the model instance.

Parameters:

data (typing.Any)

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

__init__(**data)[source]

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

data (typing.Any)

model_fields = {}
model_computed_fields = {}
property model_extra: dict[str, Any] | None

Get extra fields set during validation.

Returns:

A dictionary of extra fields, or None if config.extra is not set to "allow".

property model_fields_set: set[str]

Returns the set of fields that have been explicitly set on this model instance.

Returns:

A set of strings representing the fields that have been set,

i.e. that were not filled from defaults.

classmethod model_construct(_fields_set=None, **values)[source]

Creates a new instance of the Model class with validated data.

Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.

!!! note

model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == 'allow', then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == 'ignore' (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == 'forbid' does not result in an error if extra values are passed, but they will be ignored.

Parameters:

  • _fields_set (Optional[set[str]], default: None) – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.

  • values (typing.Any) – Trusted or pre-validated data dictionary.

Return type:

Self

Returns:

A new instance of the Model class with validated data.

model_copy(*, update=None, deep=False)[source]
!!! abstract “Usage Documentation”

[model_copy](../concepts/models.md#model-copy)

Returns a copy of the model.

!!! note

The underlying instance’s [__dict__][object.__dict__] attribute is copied. This might have unexpected side effects if you store anything in it, on top of the model fields (e.g. the value of [cached properties][functools.cached_property]).

Parameters:

  • update (Optional[Mapping[str, typing.Any]], default: None) – Values to change/add in the new model. Note: the data is not validated before creating the new model. You should trust this data.

  • deep (bool, default: False) – Set to True to make a deep copy of the model.

Return type:

Self

Returns:

New model instance.

model_dump(*, mode='python', include=None, exclude=None, context=None, by_alias=None, exclude_unset=False, exclude_defaults=False, exclude_none=False, exclude_computed_fields=False, round_trip=False, warnings=True, fallback=None, serialize_as_any=False)[source]
!!! abstract “Usage Documentation”

[model_dump](../concepts/serialization.md#python-mode)

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

Parameters:

  • mode (default: 'python') – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.

  • include (default: None) – A set of fields to include in the output.

  • exclude (default: None) – A set of fields to exclude from the output.

  • context (default: None) – Additional context to pass to the serializer.

  • by_alias (default: None) – Whether to use the field’s alias in the dictionary key if defined.

  • exclude_unset (default: False) – Whether to exclude fields that have not been explicitly set.

  • exclude_defaults (default: False) – Whether to exclude fields that are set to their default value.

  • exclude_none (default: False) – Whether to exclude fields that have a value of None.

  • exclude_computed_fields (default: False) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.

  • round_trip (default: False) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].

  • warnings (default: True) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].

  • fallback (default: None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.

  • serialize_as_any (default: False) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A dictionary representation of the model.

model_dump_json(*, indent=None, ensure_ascii=False, include=None, exclude=None, context=None, by_alias=None, exclude_unset=False, exclude_defaults=False, exclude_none=False, exclude_computed_fields=False, round_trip=False, warnings=True, fallback=None, serialize_as_any=False)[source]
!!! abstract “Usage Documentation”

[model_dump_json](../concepts/serialization.md#json-mode)

Generates a JSON representation of the model using Pydantic’s to_json method.

Parameters:

  • indent (default: None) – Indentation to use in the JSON output. If None is passed, the output will be compact.

  • ensure_ascii (default: False) – If True, the output is guaranteed to have all incoming non-ASCII characters escaped. If False (the default), these characters will be output as-is.

  • include (default: None) – Field(s) to include in the JSON output.

  • exclude (default: None) – Field(s) to exclude from the JSON output.

  • context (default: None) – Additional context to pass to the serializer.

  • by_alias (default: None) – Whether to serialize using field aliases.

  • exclude_unset (default: False) – Whether to exclude fields that have not been explicitly set.

  • exclude_defaults (default: False) – Whether to exclude fields that are set to their default value.

  • exclude_none (default: False) – Whether to exclude fields that have a value of None.

  • exclude_computed_fields (default: False) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.

  • round_trip (default: False) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].

  • warnings (default: True) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].

  • fallback (default: None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.

  • serialize_as_any (default: False) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A JSON string representation of the model.

classmethod model_json_schema(by_alias=True, ref_template='#/$defs/{model}', schema_generator=<class 'pydantic.json_schema.GenerateJsonSchema'>, mode='validation', *, union_format='any_of')[source]

Generates a JSON schema for a model class.

Parameters:

  • by_alias (bool, default: True) – Whether to use attribute aliases or not.

  • ref_template (str, default: '#/$defs/{model}') – The reference template.

  • union_format (Literal['any_of', 'primitive_type_array'], default: 'any_of') –

    The format to use when combining schemas from unions together. Can be one of:

    keyword to combine schemas (the default). - 'primitive_type_array': Use the [type](https://json-schema.org/understanding-json-schema/reference/type) keyword as an array of strings, containing each type of the combination. If any of the schemas is not a primitive type (string, boolean, null, integer or number) or contains constraints/metadata, falls back to any_of.

  • schema_generator (type[GenerateJsonSchema], default: <class 'pydantic.json_schema.GenerateJsonSchema'>) – To override the logic used to generate the JSON schema, as a subclass of GenerateJsonSchema with your desired modifications

  • mode (Literal['validation', 'serialization'], default: 'validation') – The mode in which to generate the schema.

Return type:

dict[str, typing.Any]

Returns:

The JSON schema for the given model class.

classmethod model_parametrized_name(params)[source]

Compute the class name for parametrizations of generic classes.

This method can be overridden to achieve a custom naming scheme for generic BaseModels.

Parameters:

params (tuple[type[typing.Any], ...]) – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.

Return type:

str

Returns:

String representing the new class where params are passed to cls as type variables.

Raises:

TypeError – Raised when trying to generate concrete names for non-generic models.

model_post_init(context, /)[source]

Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.

Parameters:

context (typing.Any)

Return type:

None

classmethod model_rebuild(*, force=False, raise_errors=True, _parent_namespace_depth=2, _types_namespace=None)[source]

Try to rebuild the pydantic-core schema for the model.

This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.

Parameters:

  • force (bool, default: False) – Whether to force the rebuilding of the model schema, defaults to False.

  • raise_errors (bool, default: True) – Whether to raise errors, defaults to True.

  • _parent_namespace_depth (int, default: 2) – The depth level of the parent namespace, defaults to 2.

  • _types_namespace (Optional[Mapping[str, Any]], default: None) – The types namespace, defaults to None.

Return type:

bool | None

Returns:

Returns None if the schema is already “complete” and rebuilding was not required. If rebuilding _was_ required, returns True if rebuilding was successful, otherwise False.

classmethod model_validate(obj, *, strict=None, extra=None, from_attributes=None, context=None, by_alias=None, by_name=None)[source]

Validate a pydantic model instance.

Parameters:

  • obj – The object to validate.

  • strict (default: None) – Whether to enforce types strictly.

  • extra (default: None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • from_attributes (default: None) – Whether to extract data from object attributes.

  • context (default: None) – Additional context to pass to the validator.

  • by_alias (default: None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (default: None) – Whether to use the field’s name when validating against the provided input data.

Raises:

ValidationError – If the object could not be validated.

Returns:

The validated model instance.

classmethod model_validate_json(json_data, *, strict=None, extra=None, context=None, by_alias=None, by_name=None)[source]
!!! abstract “Usage Documentation”

[JSON Parsing](../concepts/json.md#json-parsing)

Validate the given JSON data against the Pydantic model.

Parameters:

  • json_data – The JSON data to validate.

  • strict (default: None) – Whether to enforce types strictly.

  • extra (default: None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • context (default: None) – Extra variables to pass to the validator.

  • by_alias (default: None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (default: None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Raises:

ValidationError – If json_data is not a JSON string or the object could not be validated.

classmethod model_validate_strings(obj, *, strict=None, extra=None, context=None, by_alias=None, by_name=None)[source]

Validate the given object with string data against the Pydantic model.

Parameters:

  • obj – The object containing string data to validate.

  • strict (default: None) – Whether to enforce types strictly.

  • extra (default: None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • context (default: None) – Extra variables to pass to the validator.

  • by_alias (default: None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (default: None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

dict(*, include=None, exclude=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False)[source]
Parameters:

  • include (IncEx | None, default: None)

  • exclude (IncEx | None, default: None)

  • by_alias (bool, default: False)

  • exclude_unset (bool, default: False)

  • exclude_defaults (bool, default: False)

  • exclude_none (bool, default: False)

Return type:

Dict[str, Any]

json(*, include=None, exclude=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False, encoder=PydanticUndefined, models_as_dict=PydanticUndefined, **dumps_kwargs)[source]
Parameters:
Return type:

str

classmethod parse_obj(obj)[source]
Parameters:

obj (typing.Any)

Return type:

Self

classmethod parse_raw(b, *, content_type=None, encoding='utf8', proto=None, allow_pickle=False)[source]
Parameters:

  • b (str | bytes)

  • content_type (Optional[str], default: None)

  • encoding (str, default: 'utf8')

  • proto (Optional[Protocol], default: None)

  • allow_pickle (bool, default: False)

Return type:

Self

classmethod parse_file(path, *, content_type=None, encoding='utf8', proto=None, allow_pickle=False)[source]
classmethod from_orm(obj)[source]
Parameters:

obj (typing.Any)

Return type:

Self

classmethod construct(_fields_set=None, **values)[source]
Parameters:
Return type:

Self

copy(*, include=None, exclude=None, update=None, deep=False)[source]

Returns a copy of the model.

!!! warning “Deprecated”

This method is now deprecated; use model_copy instead.

If you need include or exclude, use:

`python {test="skip" lint="skip"} data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `

Parameters:

  • include (AbstractSetIntStr | MappingIntStrAny | None, default: None) – Optional set or mapping specifying which fields to include in the copied model.

  • exclude (AbstractSetIntStr | MappingIntStrAny | None, default: None) – Optional set or mapping specifying which fields to exclude in the copied model.

  • update (Dict[str, typing.Any] | None, default: None) – Optional dictionary of field-value pairs to override field values in the copied model.

  • deep (bool, default: False) – If True, the values of fields that are Pydantic models will be deep-copied.

Return type:

Self

Returns:

A copy of the model with included, excluded and updated fields as specified.

classmethod schema(by_alias=True, ref_template='#/$defs/{model}')[source]
Parameters:

  • by_alias (bool, default: True)

  • ref_template (str, default: '#/$defs/{model}')

Return type:

Dict[str, Any]

classmethod schema_json(*, by_alias=True, ref_template='#/$defs/{model}', **dumps_kwargs)[source]
Parameters:

  • by_alias (bool, default: True)

  • ref_template (str, default: '#/$defs/{model}')

  • dumps_kwargs (typing.Any)

Return type:

str

classmethod validate(value)[source]
Parameters:

value (typing.Any)

Return type:

Self

classmethod update_forward_refs(**localns)[source]
Parameters:

localns (typing.Any)

Return type:

None

class TestConfiguration(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

goal: str
instructions: str
restrictions: str
scenario: str
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class TestType(value)[source]

Bases: str, Enum

Enum for test types.

These values align with the backend TypeLookup table: - SINGLE_TURN: Traditional single request-response tests - MULTI_TURN: Agentic multi-turn conversation tests using Penelope

SINGLE_TURN = 'Single-Turn'
MULTI_TURN = 'Multi-Turn'
get_model(provider=None, model_name=None, api_key=None, config=None, **kwargs)[source]

Create a model instance with smart defaults and comprehensive error handling.

This function provides multiple ways to create a model instance:

  1. Minimal: get_model() - uses all defaults

  2. Provider only: get_model("rhesis") - uses default model for provider

  3. Provider + Model: get_model("rhesis", "rhesis-llm-v1")

  4. Shorthand: get_model("rhesis/rhesis-llm-v1")

  5. Full config: get_model(config=ModelConfig(...))

Parameters:

  • provider (Optional[str], default: None) – Provider name (e.g., “rhesis”, “anthropic”, “gemini”, “openai”, “mistral”, “ollama”)

  • model_name (Optional[str], default: None) – Specific model name

  • api_key (Optional[str], default: None) – API key for authentication

  • config (Optional[ModelConfig], default: None) – Complete configuration object

  • **kwargs – Additional parameters passed to ModelConfig

Returns:

Configured model instance

Return type:

BaseLLM

Raises:

  • ValueError – If configuration is invalid or provider not supported

  • ImportError – If required dependencies are missing

Examples

>>> # Basic usage with defaults
>>> model = get_model()

>>> # Specify provider and model
>>> model = get_model("rhesis", "rhesis-llm-v1")

>>> # Use provider/model shorthand
>>> model = get_model("rhesis/rhesis-llm-v1")

>>> # Use different providers
>>> model = get_model("anthropic", "claude-4")
>>> model = get_model("openai", "gpt-4o")
>>> model = get_model("mistral/mistral-medium-latest")

>>> # With custom configuration
>>> config = ModelConfig(
...     provider="gemini",
...     model_name="gemini-pro",
...     api_key="your-api-key"
... )
>>> model = get_model(config=config)

>>> # With extra parameters
>>> model = get_model(
...     "rhesis",
...     "rhesis-llm-v1",
...     extra_params={"temperature": 0.5}
... )
class BaseLLM(model_name, *args, **kwargs)[source]

Bases: ABC

PROVIDER: str = ''
__init__(model_name, *args, **kwargs)[source]
abstract load_model(*args, **kwargs)[source]

Loads a model

Returns:

A model object

abstract generate(*args, **kwargs)[source]

Runs the model to output LLM response.

Return type:

Union[str, Dict[str, Any]]

Returns:

A string or dict (if schema provided).

abstract generate_batch(*args, **kwargs)[source]

Run model on multiple prompts to output LLM responses.

Return type:

List[Union[str, Dict[str, Any]]]

Returns:

A list of strings or dicts (if schema provided).

get_model_name(*args, **kwargs)[source]
Return type:

str

get_available_models()[source]
Return type:

List[str]

push(name, description=None)[source]

Save this LLM configuration to the Rhesis platform as a Model entity.

Creates a Model entity with this LLM’s provider, model name, and API key, then saves it to the platform.

Parameters:

  • name (str) – Name for the saved model configuration (required)

  • description (Optional[str], default: None) – Optional description for the model

Returns:

The created Model entity (can be used for set_default_generation, etc.)

Return type:

Model

Raises:

ValueError – If provider is not set on this LLM class

Example

>>> from rhesis.sdk.models.factory import get_model
>>> llm = get_model("openai", "gpt-4", api_key="sk-...")
>>> model = llm.push(name="My GPT-4 Production")
>>> model.set_default_generation()
create_test_set(tests, model, **metadata_kwargs)[source]

Create and configure a TestSet with metadata.

Parameters:
Return type:

TestSet

class GenerationConfig(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

generation_prompt: str
behaviors: Optional[list[str]]
categories: Optional[list[str]]
topics: Optional[list[str]]
additional_context: Optional[str]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class Test(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

test_configuration: TestConfiguration
behavior: str
category: str
topic: str
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class Tests(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

tests: List[Test]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class FlatTest(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

test_configuration_goal: str
test_configuration_instructions: str
test_configuration_restrictions: str
test_configuration_scenario: str
behavior: str
category: str
topic: str
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class FlatTests(**data)[source]

Bases: BaseModel

Parameters:

data (typing.Any)

tests: List[FlatTest]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class MultiTurnSynthesizer(config, model=None, batch_size=10)[source]

Bases: object

Parameters:
prompt_template_file: str = 'base.jinja'
__init__(config, model=None, batch_size=10)[source]
Parameters:
load_prompt_template(prompt_template_file)[source]

Load prompt template from assets or use custom prompt.

Parameters:

prompt_template_file (str)

Return type:

jinja2.environment.Template

generate(num_tests=5)[source]
Parameters:

num_tests (int, default: 5)

Return type:

TestSet