API Reference

This page contains the auto-generated API reference for Modern-BERT-Score.

BertScore Module

class modern_bert_score.bert_score.BertScore(model_id: str = 'answerdotai/ModernBERT-base', idf_weighting: bool = False, baseline_rescaling: bool = False, custom_baseline: Tuple[float, float, float] | None = None, device: str = 'cpu', backend: str = 'default', sentence_transformers_args: Dict[str, Any] | None = None, vllm_args: Dict[str, Any] | None = None)[source]

Bases: object

Computes BERTScore for text generation evaluation.

inference_engine

The backend engine used for computing embeddings.

Type:: modern_bert_score.inference.STInference | modern_bert_score.inference.VLLMInference | None

inference_engine: STInference | VLLMInference | None

bert_score(candidates: Tensor, references: Tensor, idf_dict_ref: Dict[int, float] | None = None, input_ids_cand: List[int] | None = None, input_ids_ref: List[int] | None = None) → Tuple[float, float, float][source]

Computes the BERTScore for a single pair of token embedding matrices.

Parameters:

candidates – Tensor of candidate token embeddings.
references – Tensor of reference token embeddings.
idf_dict_ref – Dictionary mapping token IDs to IDF weights (optional).
input_ids_cand – List of token IDs for the candidate text (optional).
input_ids_ref – List of token IDs for the reference text (optional).

Returns:

A tuple containing (Precision, Recall, F1).

get_idf_dict(corpus: List[str], nthreads: int = 4, batch_size: int = 100000) → Tuple[Dict[int, float], List[List[int]]][source]

Builds an IDF (Inverse-Document-Frequency) dictionary for a corpus.

Parameters:

corpus – The list of strings to build the IDF dictionary from.
nthreads – The number of threads to use for parallel processing.
batch_size – The number of samples per batch.

Returns:

A tuple containing the IDF dictionary mapping token IDs to weights, and the tokenized input IDs for the corpus.

modern_bert_score.bert_score.ModernBERTBaseScore(**kwargs: Any) → BertScore[source]

Initializes BertScore with the LazerLambda/ModernBERT-base-ModBERTScore-12 model.

Parameters:: **kwargs – Arbitrary keyword arguments passed to the BertScore constructor.
Returns:: A BertScore instance configured with the ModernBERT base model.

modern_bert_score.bert_score.ModernBERTLargeScore(**kwargs: Any) → BertScore[source]

Initializes BertScore with the LazerLambda/ModernBERT-large-ModBERTScore-19 model.

Parameters:: **kwargs – Arbitrary keyword arguments passed to the BertScore constructor.
Returns:: A BertScore instance configured with the ModernBERT large model.

modern_bert_score.bert_score.RobertaBaseScore(**kwargs: Any) → BertScore[source]

Initializes BertScore with the LazerLambda/roberta-base-ModBERTScore-10 model.

Parameters:: **kwargs – Arbitrary keyword arguments passed to the BertScore constructor.
Returns:: A BertScore instance configured with the RoBERTa base model.

modern_bert_score.bert_score.RobertaLargeScore(**kwargs: Any) → BertScore[source]

Initializes BertScore with the LazerLambda/roberta-large-ModBERTScore-17 model.

Parameters:: **kwargs – Arbitrary keyword arguments passed to the BertScore constructor.
Returns:: A BertScore instance configured with the RoBERTa large model.

modern_bert_score.bert_score.RobertaLargeMNLIScore(**kwargs: Any) → BertScore[source]

Initializes BertScore with the LazerLambda/roberta-large-mnli-ModBERTScore-19 model.

Parameters:: **kwargs – Arbitrary keyword arguments passed to the BertScore constructor.
Returns:: A BertScore instance configured with the RoBERTa large MNLI model.

Inference Module

class modern_bert_score.inference.Inference[source]

Bases: object

Abstract base class for inference backends.

model: Any = None

inference(candidates: List[str], references: List[str], **kwargs: Any) → tuple[List[Tensor], List[Tensor]][source]

Computes embeddings for candidates and references.

Parameters:

candidates – A list of candidate strings.
references – A list of reference strings.
**kwargs – Additional arguments passed to the underlying model.

Returns:

A tuple containing two lists of tensors (candidate_embeddings, reference_embeddings).

class modern_bert_score.inference.STInference(model_id: str, device: str = 'cpu', batch_size: int = 64, **kwargs: Any)[source]

Bases: Inference

Inference backend using SentenceTransformers.

eps: float

inference(candidates: List[str], references: List[str], **kwargs: Any) → tuple[List[Tensor], List[Tensor]][source]

Computes embeddings using SentenceTransformers.

Parameters:

candidates – A list of candidate strings.
references – A list of reference strings.
**kwargs – Additional arguments passed to model.encode.

Returns:

A tuple containing two lists of tensors (candidate_embeddings, reference_embeddings).

class modern_bert_score.inference.VLLMInference(**kwargs: Any)[source]

Bases: Inference

Inference backend using vLLM.

eps: float

inference(candidates: List[str], references: List[str], **kwargs: Any) → tuple[List[Tensor], List[Tensor]][source]

Computes embeddings using vLLM.

Parameters:

candidates – A list of candidate strings.
references – A list of reference strings.
**kwargs – Additional arguments passed to model.encode.

Returns:

A tuple containing two lists of tensors (candidate_embeddings, reference_embeddings).

cleanup() → None[source]: Cleans up the vLLM model and frees GPU memory.