BNjdZddlmZddlZddlZddlmZddlmZm Z m Z ddl m Z ddl ZddlZddlmZmZmZmZmZmZmZ ddlmZ dd lmZmZej@e!Z"d d d d d Z#dd Z$dd Z%ddZ&ddZ'ddZ(ddZ)ddZ*ddZ+GddZ, d d dZ- d d!dZ.d"dZ/ d#dZ0y#e$rdZY~wxYw#eef$rdZdZYwxYw)$zAModality detection, input parsing, and message format conversion.) annotationsN) defaultdict)AnyLiteralNoReturn)urlparse)MULTIMODAL_DICT_KEYS AudioInput MessageFormatModality PairInput SingleInput VideoInput)Image) AudioDecoder VideoDecoderflat)apertus deepseek_v3gpt_ossseed_ossc0|jdxrd|vS)zQCheck if a string looks like a valid URL (starts with http(s) and has no spaces).)zhttp://zhttps:// ) startswithtexts t/Users/ahmed/devFolder/Ultron/claude-voice/.venv/lib/python3.12/site-packages/sentence_transformers/base/modality.py_looks_like_urlr/s ??2 3 G4Gc t|r5 t|jj}|j |S|jj |xrt jj|S#t$rYywxYw)zOCheck if a string is a URL or local file path with one of the given extensions.F)rrpathlower ValueErrorendswithosisfile)r extensionsr!s r_is_media_url_or_pathr(4sut D>&&,,.D}}Z(( ::<  , E1EE   s#B B  B c>|jdryt|dS)z:Check if a string is an image URL, file path, or data URI.z data:image/T)z.jpgz.jpegz.pngz.gifz.bmpz.tiffz.webp)rr(rs ris_image_url_or_pathr*@s }% 'b ccrct|dryt|sy t|j}|dvS#t$rYywxYw)z.Check if a string is a video URL or file path.)z.mp4z.aviz.movz.wmvz.flvz.mkvTF)zwww.youtube.comz youtube.comzyoutu.bez m.youtube.com)r(rrnetlocr#)rr,s ris_video_url_or_pathr-GsWT#ST 4 $&&   s4 AAct|dS)z/Check if a string is an audio URL or file path.)z.mp3z.wavz.oggz.flacz.aac)r(rs ris_audio_url_or_pathr/Ys 'P QQrc0t|ttfrt|dk7ryt|dtrt|dtry|D]E}t|t r d|vrd|vryt|ts.|s1t|dt sEyy)aCheck if a sample is a non-text pair (2-element tuple/list with at least one non-string element). Text pairs ``(str, str)`` are handled natively by tokenizers and detected as ``"text"`` modality by :func:`infer_modality`. This helper detects pairs that contain at least one non-string element (e.g. an image, audio array, or dict), which require conversion to message format. FrrolecontentT) isinstancetuplelistlenstrdict)sampleelems r_is_non_text_pairr=^s fudm ,F q0@&)S!jC&@ dD !fnd9J dD !dz$q'4/H  rct|trd|vr |d|dd<|dSt\t|trL|j}|j|dd<|j j djS|S)zUnwrap dict-wrapped audio or an ``AudioDecoder`` into a raw array, collecting ``sampling_rate``. Passes through unchanged if ``audio_value`` is already a raw array/tensor/URL/path. sampling_rateaudioarrayr)dim)r5r:rget_all_samples sample_ratedatameannumpy) audio_valueextra_modality_kwargssampless r _unwrap_audiorKss +t$ k )>I/>Z !' *? ;7##J{L$I--/:A:M:Mg&7||  Q '--// rc t|tr0d|vr'|djdgj|d|dStt|tr|j dt |}|djdgj|jj|jjtt|jjdd|jS|S)zUnwrap dict-wrapped video or a ``VideoDecoder`` into a raw array, collecting ``video_metadata``. Passes through unchanged if ``video_value`` is already a raw array/tensor/URL/path. video_metadatavideorAr)fpstotal_num_framesframes_indices)r5r: setdefaultappendrget_frames_in_ranger8metadata average_fps num_framesr7rangerEshape) video_valuerI frame_batchs r _unwrap_videor\s +t$ { * !' * 5 56F K R RS^_oSp q7##J{L$I!55a[9IJ g&112BBGNN"++77$/$8$8$C$C"&u[-=-=-C-CA-F'G"H   rceZdZdZ d ddZddZ ddZddZdddZ ddZ e dd Z dd Z dd Z dd Zy)InputFormatteraHandles input parsing, modality detection, and message format conversion. This class manages the complete input preprocessing pipeline: 1. Parsing raw inputs to detect their modality (text, image, audio, video, message) 2. Converting inputs to different chat template formats 3. Normalizing mixed-modality inputs Different models require different message/chat template formats: - **Structured format**: Content is a list of dicts with type annotations [{"role": "user", "content": [{"type": "text", "text": "hello"}]}] - **Flat format**: Content is the direct value [{"role": "user", "content": "hello"}] Args: model_type: The model type string (e.g. from ``config.model_type``). message_format: Message format to use. Options: - ``"structured"``: Content is a list of dicts with type/modality keys - ``"flat"``: Content is the direct value - ``"auto"``: Automatically infer from processor (default) processor: Optional processor to infer format from when ``message_format="auto"``. supported_modalities: Optional list of modalities supported by the model. When provided, string inputs that look like media URLs/paths are only classified as non-text if the model actually supports that modality. This prevents text-only models from misclassifying text containing media URLs. Nc||_||_||_|dk(r!|r|j||_yd|_y||_y)Nauto structured) model_type processorsupported_modalities _infer_formatmessage_format)selfrbrfrcrds r__init__zInputFormatter.__init__sG%"$8! V #CL$"4"4Y"?D R^D "0D rc|jtvrt|jSt|ddttrsygd}t fd|Dryy)aInfer the message format expected by the processor. Checks known model types first, then inspects the processor's chat template for patterns indicating structured format. Defaults to ``"structured"`` if neither approach is conclusive. Args: processor: The processor/tokenizer to inspect. Returns: ``"structured"`` or ``"flat"`` message format. chat_templateNra)z content[0]z.typez'type'z"type"z item.typezmessage.content[c3&K|]}|v ywN).0patterntemplates r z/InputFormatter._infer_format..sF2Eww("2Er)rb!KNOWN_MODEL_TYPES_MESSAGE_FORMATSgetattrr5r9any)rgrcstructured_patternsrps @rrezInputFormatter._infer_formats] ??? ?4T__E E9ot<(C(  F2EF Frc |sddgittfSg}tt}d}|D]}t|r|jd|fd}%t ||j }t |tr!|tvr|j|hk(r||}|dk(r t||}nu|dk(r t||}nct |tr8t|}d|vrt|d||d<d|vr.t|d||d<n|dk(rt |tr|g}n|}|j||f |rg}|D]\} }| dk(r!|j|j|,| dk(rEt |ttfr/t|d k(r!|j|j|v| dk(r|j|t | tr|n| |i} |j|j| dd|i|fSt!|\} } t| } t#| } t| d k(r\| j%}t |t&r|| i} n| d j}|D cic]} | | Dcgc]}||  c}} } }nUt(j+d | d d|Dcgc]*\}}|jt |tr|n||i,c}}i} d}|| |fScc}wcc}} wcc}}w)a-Parse inputs and group by modality. Analyzes a list of inputs to detect their modality (text, image, audio, video, message) and groups them appropriately for the processor. Handles mixed modalities by converting to message format when necessary. Non-text pairs (e.g. ``(image, text)`` or ``(image, image)``) are detected and converted to message format with ``"query"``/``"document"`` roles via :meth:`pair_to_messages`. Args: inputs: List of inputs to parse. Can be: - str: Text inputs - tuple/list of str: Text pairs (for cross-encoders) - tuple/list of mixed types: Non-text pairs (e.g. image + text) - dict: Chat messages, audio data, or multimodal inputs - PIL.Image.Image: Image inputs - np.ndarray/torch.Tensor: Audio (1-2D) or video (3-5D) inputs Returns: Tuple of (modality, processor_inputs_dict, extra_modality_kwargs) where: - modality: Detected modality string (``"text"``, ``"image"``, etc.) or tuple of modalities - processor_inputs_dict: Dictionary mapping modality names to input lists - extra_modality_kwargs: Extra kwargs per modality (e.g. ``sampling_rate`` for audio) rFpairTrdr@rNmessager1r2rzMixed modalities detected: z!. Converting to 'message' format.)rr:r=rSinfer_modalityrdr5r keysrKr\r6pair_to_messagesr7r8 to_messagezipsetpopr9loggerdebug)rginputs typed_inputsrI has_pairsitemmodalityvaluemessagesmodtyped modalitiesprocessed_inputsunique_modalities ordered_keysentrys r parse_inputszInputFormatter.parse_inputss+8FB<T):: :EG +D 1 D!&##VTN3 %dAZAZ[H$%(6J*Jtyy{_g^hOhH~ 7"%d,ABW$%d,ABHe,T e#%25>CX%YE'Ne#%25>CX%YE'NY&:dD+A   5 1 2CL H* U&=OOD$9$9%$@AF]z%%'GCPUJZ[OOOD$9$9%$@AI%OOE*%/U%;E#uEOODOOE$:;+y(35JJ J'*L'9$ $ 01 O  !Q &(,,.H(C($,.>#?  02779 _k#l_kX[CBR)SBR%*BR)S$S_k #l LL67H6IIjk l+7+7%OOZ%-HExY^N_`+7   !H)+@@@*T#ls4 K2> K- K22/K8-K2c|\}}t|}t|}|jdk(r d|dd|dgSd}d|||dd|||dgS)aConvert a pair of inputs to query/document message format. Each element of the pair is wrapped in a message with role ``"query"`` (first element) or ``"document"`` (second element). The modality of each element is inferred individually via :func:`infer_modality`. Args: pair: A 2-element tuple or list of inputs (e.g. ``(image, text)``). Returns: List of two message dictionaries with ``"query"`` and ``"document"`` roles. rqueryr3r4documentct|tr,t|tr|Dcgc]}||vsd||||ic}St|tr!|tvr|j |hk(r||}d|||igScc}w)Ntype)r5r6r:r r|)rrrs r _to_contentz4InputFormatter.pair_to_messages.._to_contentms(E*z$/EAIY#STX[c495YY$%(6J*Jtyy{_g^hOhH~Xx67 7 Zs A5 A5)r{rf)rgrx query_itemdoc_itemquery_modality doc_modalityrs rr}zInputFormatter.pair_to_messagesVsy $ H' 3%h/   & ( Z8#9   8^Z)P QK h,O P  rc |jdk(rNt|dk(r+tt|j \}}||dgSt j d||j Dcgc] \}}d|||i c}}dgScc}}w)aConvert a typed input dictionary to message format. Produces a single message with the given ``role``. For pair/multi-value inputs, use :meth:`pair_to_messages` instead (which is called automatically by :meth:`parse_inputs`). Args: typed_input: Dictionary mapping modality to input value (single value per modality). role: Role for the message (default: ``"user"``). Returns: List of message dictionaries (single message). rr2rzbFlat message format requested but multiple modalities detected. Falling back to structured format.r)rfr8nextiteritemsrwarning)rg typed_inputr3_rrs rr~zInputFormatter.to_message|s   & (;1$[%6%6%8 9:5!%%8999WbWhWhWjkWjOHeVXx?Wjk   ls2B cP|sddgifSt|tr|fn|}ttt |j }g}t |D]}|Dcic]}||vs||||}}t|dk(rytt |j } t| ttfrAt| dk(r3td| Dr!|j|j| |j|j|dd|ifScc}w)acConvert a batch of modality-specific inputs into the unified message format. Args: modality: The modality key (string) or tuple of modality keys. processor_inputs: Dictionary mapping modality names to lists of inputs. Returns: Tuple of ``("message", {"message": [messages_per_sample, ...]})`` rzr2r1c3<K|]}t|tywrl)r5r9)rnvs rrqz2InputFormatter.batch_to_message..sOrlqghPZ[\^aPblqs) r5r9r8rrvaluesrXr6r7allrSr}r~) rgrprocessor_inputsr batch_sizerirrrs rbatch_to_messagezInputFormatter.batch_to_messages y"o- -$.x$=h[8 d#3#:#:#<=>? z"ADTjDTSX[_iXi3 0 5a 88DTKj;1$T+"4"4"678eeT]3E aCOrlqOrLrOOD$9$9%$@A OODOOK8 9#9h///ks  D#' D#c|D]S}|D]L}|jd}t|tr%t|trt d|DsHyyUy)aCheck whether all messages in a batch contain only text content. Works with both flat format (``{"content": "hello"}``) and structured format (``{"content": [{"type": "text", "text": "hello"}]}``). Args: messages_batch: List of message lists, one per sample. Returns: True if every message contains only text, False if any contain non-text content. r4c3FK|]}|jdddk7yw)rrN)get)rnrs rrqz7InputFormatter.is_text_only_messages..s!R'$488FF3v='s!FT)rr5r9r7ru)messages_batchrrzr4s ris_text_only_messagesz$InputFormatter.is_text_only_messagessV'H#!++i0gs+gt,R'RR$ $'rcpg}|D]-}d|vsd|vrtjd|d&|d}t|txr|xrt|dt}|j dk(ri|rgt |dk(r%d|dvr|ji|d|dditjd t |d |j||j d k(rB|s@t|tr|ji|dd|d gi |j||j|0|S) a;Normalize messages to the target format (``self.message_format``). Extra keys beyond ``"role"`` and ``"content"`` are preserved during conversion. Args: messages: List of message dictionaries to normalize. Returns: Normalized list of message dictionaries. r3r4zInvalid message format: z . Skipping.rrr2rz;Cannot convert structured message to flat format: contains z# content items. Keeping structured.rarr) rrr5r7r:rfr8rSr9)rgr normalizedrzr4is_currently_structureds rnormalize_messagesz!InputFormatter.normalize_messagessN GW$ (@!9'+NOi(G&0$&?&lG&lPZ[bcd[egkPl #""f,1Hw<1$71:)=%%&P&P)WQZ=O&PQNN$$'L>1TV%%g.$$ 4=Tgs+%%&a&a)vW^>_=`&ab%%g.!!'*/ 2rc|jdk(r|Dcgc] }d|dg|z c}S|Dcgc]}dd|dgdg|zc}Scc}wcc}w)a'Prepend a system prompt to message format inputs. Args: messages: List of message lists (each message list represents one input). prompt: System prompt to prepend. Returns: Messages with system prompt prepended to each message list. rsystemrrr)rf)rgrprompt message_lists rprepend_prompt_to_messagesz)InputFormatter.prepend_prompt_to_messagessy   & (_gh_g|h6:;lJ_gh h!) ( VV,L+M N OR^ ^ (  i s AAc g}|D]N}t|tr|j||z(|j||dzgt|ddzP|S)abPrepend a prompt to text format inputs. For single texts, prepends the prompt directly. For text pairs (cross-encoder inputs), prepends only to the first text. Args: texts: List of text inputs (strings or pairs) prompt: Prompt to prepend Returns: Texts with prompt prepended rr2N)r5r9rSr7)rgtextsrresultrs rprepend_prompt_to_textsz&InputFormatter.prepend_prompt_to_texts s[D$$ ftm, vQ/04QR>AB   r)r`NN)rbr9rfr rdlist[Modality] | NonereturnNone)rzLiteral['structured', 'flat'])rlist[SingleInput | PairInput]rzBtuple[Modality, dict[str, list], defaultdict[str, dict[str, Any]]])rxz tuple | listrlist[dict[str, Any]])user)rzdict[Modality, Any]r3r9rr)rr rr:rz*tuple[Literal['message'], dict[str, list]])rlist[list[dict[str, Any]]]rbool)rrrr)rrrr9rr)rz'list[str | tuple[str, str] | list[str]]rr9rzlist[str | list[str]])__name__ __module__ __qualname____doc__rhrerr}r~r staticmethodrrrrrmrrr^r^s<)/6: 1 1& 1 4 1  1 DmA-mA LmA^$ L <0 0480 30:0%N 2 RSbiJj":!3:>> BBks>cHt|trdj|S|S)zSFormat a modality for display, e.g. ``("text", "image")`` becomes ``"text+image"``.+)r5r6join)rs rformat_modalityrs (E"xx!! Orc 2djdD}|dk(r |Dchc]}t|}}|dhk(rtd|d|dd|vrtd|d|d|rdjd t |t D}|D chc]}t|tr|n|fD]} |  } }} t fd | D} | r'td |d |ddj| d|d td |d|dt|tr5tfd|Dr!td|ddj|dtdt |d|d|dcc}w#ttf$rt }YUwxYwcc} }w)aRaise a clear ``ValueError`` explaining why ``modality`` is not supported. Shared by :meth:`BaseModel.preprocess` and :meth:`Transformer.preprocess` so both raise the same, accurate guidance. ``source`` is a human-readable description of what rejected the input, e.g. ``"SentenceTransformer model"`` or ``"Transformer module"``. Args: inputs: The original batch of inputs, re-inspected per sample to distinguish explicit chat-style ``"message"`` inputs from inferred mixed-modality batches (both collapse to the ``"message"`` modality at the batch level). modality: The inferred (and unsupported) batch modality. supported_modalities: The modalities the source actually supports. source: Human-readable name of the rejecting component, used verbatim in the message. Raises: ValueError: Always, with a message tailored to the specific unsupported-modality scenario. z, c32K|]}t|ywrlrrnms rrqz3raise_unsupported_modality_error..sO:NQoa0:NrzryzThis zy does not support chat-style 'message' inputs (dicts or lists of dicts with 'role' and 'content'). Supported modalities: .z does not support chat-style 'message' inputs (dicts or lists of dicts with 'role' and 'content'), which this batch mixes with other modalities. Supported modalities: c32K|]}t|ywrlrrs rrqz3raise_unsupported_modality_error..s#o@n1OA$6@nr)keyc3,K|] }|vs| ywrlrmrnpartrds rrqz3raise_unsupported_modality_error..s f/$TQeEe/s z&This batch mixes multiple modalities (z ), but this z does not support z. Supported modalities: z), which this zZ cannot encode in a single batch. Encode each modality separately (one call per modality).c3&K|]}|v ywrlrmrs rrqz3raise_unsupported_modality_error..s*]T\D43G+GT\rrz supports z and zr individually, but cannot combine them in a single input. Encode each modality separately (one call per modality).z Modality 'z' is not supported by this ) rr{r# TypeErrorrrrr5r6r) rrrdsource supported_strr;sample_modalities present_strrrbase_modalities unsupporteds ` r raise_unsupported_modality_errorrs7.IIO:NOOM9 &`f0`fV\vQ[\]_dQeAlmkoLo4tLot->Oq f/ ffK <[MV\U]^##'99[#9"::RS`Raabd8 ^TZS[\ef (E"s*]T\*]']F8:gll8&<%=>g h   _X.//J6(S!!.q 2 W0I& & #  &(rs'E3E.E3#F.E33FF)rr9rr)rr9r'ztuple[str, ...]rr)r;rrr)rHr rIdict[str, dict[str, Any]]rr)rZrrIrrrrl)r;zSingleInput | PairInput | Anyrdrrr )rJrrdrrr )rr rr9) rrrr rdzlist[Modality]rr9rr)1r __future__rloggingr% collectionsrtypingrrr urllib.parserrGrr)sentence_transformers.base.modality_typesr r r r r rr PIL.Imagerr ImportErrortorchcodec.decodersrrOSError getLoggerrrrsrr(r*r-r/r=rKr\r^r{rrrrmrrrsjG" #))! +>   8 $  %!H Fd$R *",EET 37\ )\/\\B37C *C/CC0L )LL)L  L  LMH WLLs$B= C =CC CC