^Nj"ddlZddlmZmZmZmZddlmZddlm Z m Z m Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZGd deZy)N)AnyIterableSequenceType)asdict) NumpyArray OnnxProviderDevice)CLIPOnnxEmbedding)CustomTextEmbedding)PooledNormalizedEmbedding)PooledEmbedding)JinaEmbeddingV3)OnnxTextEmbedding)TextEmbeddingBase)DenseModelDescription ModelSource PoolingTypec&eZdZUeeeeeegZ e e e e d<ede eeeffdZede efdZe d'dededed ed ed ed ed edede edzddfdZddddej8ddfdededzdedzdeedzdeezde edzdedeffd Ze defdZ!ededefdZ" d(dee#ezded edzdede#e$f d!Z%d"ee#ezdede#e$fd#Z&d$e#edede#e$fd%Z' d)d$ee#ezdededefd&Z(xZ)S)* TextEmbeddingEMBEDDINGS_REGISTRYreturncZ|jDcgc] }t|c}Scc}w)zLists the supported models. Returns: list[dict[str, Any]]: A list of dictionaries containing the model information. )_list_supported_modelsr)clsmodels n/Users/ahmed/devFolder/Ultron/claude-voice/.venv/lib/python3.12/site-packages/fastembed/text/text_embedding.pylist_supported_modelsz#TextEmbedding.list_supported_modelss+,/+E+E+GH+G%u +GHHHs(cjg}|jD]!}|j|j#|SN)rextendr)rresult embeddings rrz$TextEmbedding._list_supported_models#s1.000I MM)::< =1 Nrpooling normalizationsourcesdim model_file descriptionlicense size_in_gbadditional_filesc |j} | D]<} |j| jjk(s/td|dt j t ||||||| | xsg||y)NModel zk is already registered in TextEmbedding, if you still want to add this model, please use another model name)rr'r(r)r*r+ size_in_GBr-)r%r&)rlowerr ValueErrorr add_modelr) rrr%r&r'r(r)r*r+r,r-registered_modelsregistered_models radd_custom_modelzTextEmbedding.add_custom_model*s 668 1 {{} 0 6 6 < < >> UG$45!2 %% !%'%!1!7R ' r$zBAAI/bge-small-en-v1.5F model_name cache_dirthreads providerscuda device_ids lazy_loadkwargsc 6t |||fi|jdjk(rtjdt djdjdjdjdjhvr tjd d t d|j D]=} | j} tfd | Ds(| d||||||d ||_ ytd d)Nz nomic-ai/nomic-embed-text-v1.5-QzThe model 'nomic-ai/nomic-embed-text-v1.5-Q' has been updated on HuggingFace. Please review the latest documentation on HF and release notes to ensure compatibility with your workflow. ) stacklevelz;sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2zthenlper/gte-largezintfloat/multilingual-e5-largez;sentence-transformers/paraphrase-multilingual-mpnet-base-v2z The model z now uses mean pooling instead of CLS embedding. In order to preserve the previous behaviour, consider either pinning fastembed version to 0.5.1 or using `add_custom_model` functionality.c3tK|]/}j|jjk(1ywr )r1r).0rr7s r z)TextEmbedding.__init__..qs/[JZ:##%):):)<EMBEDDING_MODEL_TYPEsupported_models __class__s ` rrGzTextEmbedding.__init__OsE YB6B    !C!I!I!K K MMp       I O O Q & & ( , 2 2 4 I O O Q "  MMZL)::  %)$<$< 3JJL [JZ[[1 )'#')'   %=ZL!^ ^  r$cr|j |j|j|_|jS)z+Get the embedding size of the current model)_embedding_sizeget_embedding_sizer7)rLs rembedding_sizezTextEmbedding.embedding_sizes3    '#'#:#:4??#KD ###r$c |j}d}|D];}|jj|jk(s/|j}n|*|Dcgc]}|j}}t d|d||Scc}w)a0Get the embedding size of the passed model Args: model_name (str): The name of the model to get embedding size for. Returns: int: The size of the embedding. Raises: ValueError: If the model name is not found in the supported models. NzEmbedding size for model z" was None. Available model names: )rrr1r(r2)rr7 descriptionsrSr* model_namess rrRz TextEmbedding.get_embedding_sizes113 %)'K  &&(J,<,<,>>!,(  !@LM ;,, KM+J<8**58  NsB documents batch_sizeparallelc+^K|jj|||fi|Ed{y7w)a Encode a list of documents into list of embeddings. We use mean pooling with attention so that the model can handle variable-length inputs. Args: documents: Iterator of documents or single document to embed batch_size: Batch size for encoding -- higher values will use more memory, but be faster parallel: If > 1, data-parallel encoding will be used, recommended for offline encoding of large datasets. If 0, use all available cores. If None, don't use data-parallel processing, use default onnxruntime threading instead. Returns: List of embeddings, one per document N)rembed)rLrWrXrYr>s rr[zTextEmbedding.embeds+,$4::##Iz8NvNNNs #-+-queryc+ZK|jj|fi|Ed{y7w)z Embeds queries Args: query (Union[str, Iterable[str]]): The query to embed, or an iterable e.g. list of queries. Returns: Iterable[NumpyArray]: The embeddings. N)r query_embed)rLr\r>s rr^zTextEmbedding.query_embeds'*4::))%:6::: !+)+textsc+ZK|jj|fi|Ed{y7w)a7 Embeds a list of text passages into a list of embeddings. Args: texts (Iterable[str]): The list of texts to embed. **kwargs: Additional keyword argument to pass to the embed method. Yields: Iterable[SparseEmbedding]: The sparse embeddings. N)r passage_embed)rLr`r>s rrbzTextEmbedding.passage_embeds',4::++Es rrdzTextEmbedding.token_counts%&tzz%%eM MfMMr$)zonnx/model.onnxregN)N)i)*__name__ __module__ __qualname__rr r rrr rlistrr__annotations__ classmethoddictstrrrrrrboolrintfloatr6r AUTOrr rGpropertyrSrRrrr[r^rbrd __classcell__)rOs@rrrs! :d#456Id4S>&:IIt,A'B ,-1" " "  "  "  " " " " " s)d*"  " " L3 $"37$kk'+2 2 :2 t 2 L)D0 2 Vm 2 I$2 2 2 h$$$ CC:# O#&OO* O  O *  O0 ;x}!4 ; ;Q[H\ ; =8C= =C =HZDX ==A N8C=( N69 NLO N  Nr$r)rHtypingrrrr dataclassesrfastembed.common.typesrr r fastembed.text.clip_embeddingr $fastembed.text.custom_text_embeddingr *fastembed.text.pooled_normalized_embeddingr fastembed.text.pooled_embeddingr"fastembed.text.multitask_embeddingrfastembed.text.onnx_embeddingr"fastembed.text.text_embedding_baser"fastembed.common.model_descriptionrrrrrEr$rrsC00CC;DP;>;@^^TN%TNr$