GJj? ddlZddlmZddlmZddlmZddlm Z dZ eejddZ eejdd Z eejdd Zeejdd Zeejdd[d Zeejdd\d Zeejdd]dZeejdd\dZeejddZeejddZeejdd^defdZeejdd]dZeejddZeejddZeejddej6fdZeejddZeejddZd\dej6dedej6fdZ eejdd_dej6defdZ!eejdd Z"eejddej6d!ej6dej6fd"Z#eejddej6dej6fd#Z$eejdd$Z%eejdd`d%Z&eejdd^d&Z'eejdd\d'Z(d(Z)Gd)d*e Z*e e Gd+d,e Z+e e$Gd-d.e Z,e e Gd/d0e Z-e eGd1d2e Z.e eGd3d4e Z/Gd5d6e Z0Gd7d8e Z1e eGd9d:e Z2e eGd;de Z4Gd?d@e Z5GdAdBe Z6e eGdCdDe Z7e eGdEdFe Z8e eGdGdHe Z9GdIdJe Z:GdKdLe Z;e e)GdMdNe Z<e e%GdOdPe Z=GdQdRe Z>e e"GdSdTe Z?e e&GdUdVe Z@e e'GdWdXe ZAe e(GdYdZe ZBy)aN)partial)Any)Modulecfd}|S)Ncfd|_|S)Nc|SN)_xfs c/Users/ahmed/devFolder/claude-voice/.venv/lib/python3.12/site-packages/mlx/nn/layers/activations.pyz<_make_activation_module..decorator.. s ad)__call__)klassr s r decoratorz*_make_activation_module..decorator s* rr )r rs` r_make_activation_moduler s rT) shapelessc,tj|S)zpApplies the sigmoid function. .. math:: \text{Sigmoid}(x) = \sigma(x) = \frac{1}{1 + \exp(-x)} mxsigmoidr s rrrs ::a=rc.tj|dS)zIApplies the Rectified Linear Unit. Simply ``mx.maximum(x, 0)``. rrmaximumrs rrelurs ::a rcTtjtj|dS)u\Applies the ReLU² activation function. Applies :math:`\max(0, x)^2` element wise. r)rsquarerrs rrelu2r!&s 99RZZ1% &&rcVtjtj|ddS)z`Applies the Rectified Linear Unit 6. Applies :math:`\min(\max(x, 0), 6)` element wise. rg@rminimumrrs rrelu6r%/s ::bjjA& ,,rc4tj||z|S)z`Applies the Leaky Rectified Linear Unit. Simply ``mx.maximum(negative_slope * x, x)``. r)r negative_slopes r leaky_relur(8s ::nq(! ,,rc8|tj||dz S)zaApplies the Log Softmax function. Applies :math:`x + \log \sum_i e^{x_i}` element wise. T)axiskeepdims)r logsumexpr r*s r log_softmaxr.As r||AD48 88rchtj|dkD||tj|dz zS)zfApplies the Exponential Linear Unit. Simply ``mx.where(x > 0, x, alpha * (mx.exp(x) - 1))``. r)rwhereexpr alphas relur5Js, 88AE1ervvay1}5 66rc0tj||S)zdApplies the Softmax function. Applies :math:`\frac{e^{x_i}}{\sum_j e^{x_j}}` element wise. r*rsoftmaxr-s rr9r9Ss ::ad ##rc.tj|dS)zXApplies the Softplus function. Applies :math:`\log(1 + \exp(x))` element wise. r)r logaddexprs rsoftplusr<\s <<1 rcZtj|dtj|zS)zXApplies the Softsign function. Applies :math:`\frac{x}{1 + |x|}` element wise. r0)rdivideabsrs rsoftsignr@es! 99QBFF1I &&rlambdctjtj||kD|tj||zz dS)zApplies the Softshrink activation function. .. math:: \text{softshrink}(x) = \begin{cases} x - \lambda & \text{if } x > \lambda \\ x + \lambda & \text{if } x < -\lambda \\ 0 & \text{otherwise} \end{cases} r)rr1r?signr rAs r softshrinkrEns6 88BFF1I%q2771:+='=q AArctj|d|tjtj|d|z dz zzS)zApplies the Continuously Differentiable Exponential Linear Unit. Applies :math:`\max(0, x) + \min(0, \alpha * (\exp(x / \alpha) - 1))` element wise. r0)rrr2r$r3s rcelurH|s= ::a  1c0BU0J)Ka)O P PPrc2|tj|zS)zApplies the Sigmoid Linear Unit. Also known as Swish. Applies :math:`x \sigma(x)` element wise, where :math:`\sigma(\cdot)` is the logistic sigmoid. rrs rsilurJs rzz!} rct|  S)zmApplies the Log Sigmoid function. Applies :math:`\log(\sigma(x)) = -\log(1 + e^{-x})` element wise. )r<rs r log_sigmoidrLs aRL=rreturncj|dtj|tjdz zzdz S)zApplies the Gaussian Error Linear Units function. .. math:: \textrm{GELU}(x) = x * \Phi(x) where :math:`\Phi(x)` is the Gaussian CDF. See also :func:`gelu_approx` and :func:`gelu_fast_approx` for faster approximations. r0)rerfmathsqrtrs rgelurSs/ BFF1tyy|+,, - 11rc d|zdtjtjdtjz |d|dzzzzzzS)atAn approximation to Gaussian Error Linear Unit. See :func:`gelu` for the exact computation. This function approximates ``gelu`` with a maximum absolute error :math:`< 0.0005` in the range :math:`[-6, 6]` using the following .. math:: x = 0.5 * x * \left(1 + \text{Tanh}\left((\sqrt{2 / \pi} * \left(x + 0.044715 * x^3\right)\right)\right) ?r0rOgHm?)rtanhrQrRpirs r gelu_approxrYsD 7a"''$))AK"8A1a4 0 \\ \lambda \alpha (\exp(x) - 1) & \text{if } x \leq 0 \end{cases} where :math:`\lambda = 1.0507` and :math:`\alpha = 1.67326`. See also :func:`elu`. gGG?g䃞ͪ?)r5rs rselurhs q'?V ##rr4cbtjd||tjd|zzS)zApplies the element-wise parametric ReLU. .. math:: \text{PReLU}(x) = \max(0,x) + a * \min(0,x) where :math:`a` is an array. rrrr$r3s rprelurks* ::a ebjjA&66 66rcD|tjt|zS)zApplies the Mish function, element-wise. Mish: A Self Regularized Non-Monotonic Neural Activation Function. Reference: https://arxiv.org/abs/1908.08681 .. math:: \text{Mish}(x) = x * \text{Tanh}(\text{Softplus}(x)) )rrWr<rs rmishrms rwwx{# ##rcltj|dzd}|tj|dzdz S)zApplies the hardswish function, element-wise. .. math:: \text{Hardswish}(x) = x * \min(\max(x + 3, 0), 6) / 6 rVrrj)r max_x_3s r hardswishrqs3jjQ"G rzz'1% % ))rcVtjtj|||S)zzApplies the HardTanh function. Applies :math:`\max(\min(x, \mathrm{max\_val}), \mathrm{min\_val})` element-wise. r#)r min_valmax_vals r hard_tanhru*s ::bjjG,g 66rc\tjtj||kD|dS)zApplies the HardShrink activation function. .. math:: \text{hardshrink}(x) = \begin{cases} x & \text{if } x > \lambda \\ x & \text{if } x < -\lambda \\ 0 & \text{otherwise} \end{cases} r)rr1r?rDs r hard_shrinkrw3s$ 88BFF1I%q! ,,rc2tj| |S)zfApplies the Softmin function. Applies :math:`\frac{e^{-x_i}}{\sum_j e^{-x_j}}` element-wise. r7r8r-s rsoftminryAs ::qbt $$rc,tj|S)zIApplies the hyperbolic tangent function. Simply ``mx.tanh(x)``. )rrWrs rrWrWJs 771:rc6eZdZdZddeffd ZdefdZxZS)GLUr]r*c0t|||_yr )super__init__r*)selfr* __class__s rrz GLU.__init___s  rrMc0t||jS)Nr-)rbr*rr s rrz GLU.__call__csQTYY''r) __name__ __module__ __qualname____doc__intrrr __classcell__rs@rr|r|Rs  S(S(rr|ceZdZdZy)Sigmoidz~Applies the sigmoid function, element-wise. .. math:: \text{Sigmoid}(x) = \sigma(x) = \frac{1}{1 + \exp(-x)} Nrrrrr rrrrgrrceZdZdZy)MishzApplies the Mish function, element-wise. Reference: https://arxiv.org/abs/1908.08681 .. math:: \text{Mish}(x) = x * \text{Tanh}(\text{Softplus}(x)) Nrr rrrrpsrrceZdZdZy)ReLUzApplies the Rectified Linear Unit. Simply ``mx.maximum(x, 0)``. See :func:`relu` for the functional equivalent. Nrr rrrr|rrrceZdZdZy)ReLU2ubApplies the ReLU² activation function. See :func:`relu2` for the functional equivalent. Nrr rrrrrrceZdZdZy)ReLU6z_Applies the Rectified Linear Unit 6. See :func:`relu6` for the functional equivalent. Nrr rrrrrrrc*eZdZdZdfd ZdZxZS) LeakyReLUzApplies the Leaky Rectified Linear Unit. Simply ``mx.maximum(negative_slope * x, x)``. Args: negative_slope: Controls the angle of the negative slope. Default: ``1e-2`` c0t|||_yr )r~r_negative_slope)rr'rs rrzLeakyReLU.__init__s -rc.t||jSr )r(rrs rrzLeakyReLU.__call__s!T1122rg{Gz?rrrrrrrrs@rrrs.3rrc*eZdZdZdfd ZdZxZS)ELUzApplies the Exponential Linear Unit. Simply ``mx.where(x > 0, x, alpha * (mx.exp(x) - 1))``. See :func:`elu` for the functional equivalent. Args: alpha: the :math:`\alpha` value for the ELU formulation. Default: ``1.0`` c0t|||_yr r~r_alpharr4rs rrz ELU.__init__  rc.t||jSr )r5rrs rrz ELU.__call__s1dkk""r?rrs@rrrs#rrceZdZdZy)SoftmaxzZApplies the Softmax function. See :func:`softmax` for the functional equivalent. Nrr rrrrrrrceZdZdZy)Softplusz\Applies the Softplus function. See :func:`softplus` for the functional equivalent. Nrr rrrrrrrceZdZdZy)Softsignz\Applies the Softsign function. See :func:`softsign` for the functional equivalent. Nrr rrrrrrrc*eZdZdZdfd ZdZxZS) SoftshrinkzApplies the Softshrink function. See :func:`softshrink` for the functional equivalent. Args: lambd: the :math:`\lambda` value for Softshrink. Default: ``0.5`` c0t|||_yr )r~rrA)rrArs rrzSoftshrink.__init__s  rc.t||jSr )rErArs rrzSoftshrink.__call__s!TZZ((rrUrrs@rrrs)rrc*eZdZdZdfd ZdZxZS)CELUa<Applies the Continuously Differentiable Exponential Linear Unit. Applies :math:`\max(0, x) + \min(0, \alpha * (\exp(x / \alpha) - 1))` element wise. See :func:`celu` for the functional equivalent. Args: alpha: the :math:`\alpha` value for the CELU formulation. Default: ``1.0`` c0t|||_yr rrs rrz CELU.__init__rrc.t||jSr )rHrrs rrz CELU.__call__sAt{{##rrrrs@rrrs$rrceZdZdZy)SiLUzoApplies the Sigmoid Linear Unit. Also known as Swish. See :func:`silu` for the functional equivalent. Nrr rrrrrrrceZdZdZy) LogSoftmaxzbApplies the Log Softmax function. See :func:`log_softmax` for the functional equivalent. Nrr rrrrrrrceZdZdZy) LogSigmoidzbApplies the Log Sigmoid function. See :func:`log_sigmoid` for the functional equivalent. Nrr rrrrrrrcDeZdZdZdfd Zdej fdZxZS)PReLUa[Applies the element-wise parametric ReLU. Applies :math:`\max(0, x) + a * \min(0, x)` element wise, where :math:`a` is an array. See :func:`prelu` for the functional equivalent. Args: num_parameters: number of :math:`a` to learn. Default: ``1`` init: the initial value of :math:`a`. Default: ``0.25`` cZt|tj|g||_yr )r~rrfullweight)rnum_parametersinitrs rrzPReLU.__init__s# gg~.5 rr c.t||jSr )rkrrs rrzPReLU.__call__sQ $$r)r0g?) rrrrrrarrayrrrs@rrr s 6%"((%rrc*eZdZdZdfd ZdZxZS)GELUa0Applies the Gaussian Error Linear Units. .. math:: \textrm{GELU}(x) = x * \Phi(x) where :math:`\Phi(x)` is the Gaussian CDF. However, if ``approx`` is set to 'precise' or 'fast' it applies .. math:: \textrm{GELUApprox}(x) &= 0.5 * x * \left(1 + \text{Tanh}\left((\sqrt{2 / \pi} * \left(x + 0.044715 * x^3\right)\right)\right) \\ \textrm{GELUFast}(x) &= x * \sigma\left(1.702 * x\right) respectively. .. note:: For compatibility with the PyTorch API, 'tanh' can be used as an alias for 'precise'. See :func:`gelu`, :func:`gelu_approx` and :func:`gelu_fast_approx` for the functional equivalents and information regarding error bounds. Args: approx ('none' | 'precise' | 'fast'): Which approximation to gelu to use if any. cdt|||_gd}||vrtd|d|dy)N)nonepreciserWfastzThe approximation should be in z but 'z ' was given)r~r_approx ValueError)rapproxallowedrs rrz GELU.__init__<sF  5  1'& T  !rc~|jdk(r t|S|jdvr t|St|S)Nr)rrW)rrSrYr[rs rrz GELU.__call__Es9 <<6 !7N \\0 0q> !""r)rrrs@rrr s6#rrceZdZdZy)TanhzbApplies the hyperbolic tangent function. See :func:`tanh` for the functional equivalent. Nrr rrrrMrrrceZdZdZy) HardswishzlApplies the hardswish function, element-wise. See :func:`hardswish` for the functional equivalent. Nrr rrrrUrrrcJeZdZdZddeffd ZdejfdZxZ S)Steprercc0t|||_yr )r~rrc)rrcrs rrz Step.__init__ms "rr c.t||jSr )rfrcrs rrz Step.__call__qsAt~~&&rrG) rrrrfloatrrrrrrs@rrr]s$ #%#'"(('rrceZdZdZy)SELUzeApplies the Scaled Exponential Linear Unit. See :func:`selu` for the functional equivalent. Nrr rrrrurrrceZdZdZy)HardTanhz]Applies the HardTanh function. See :func:`hard_tanh` for the functional equivalent. Nrr rrrr}rrrceZdZdZy) HardShrinkzApplies the HardShrink function. See :func:`hard_shrink` for the functional equivalent. Args: lambd: the :math:`\lambda` value for Hardshrink. Default: ``0.5`` Nrr rrrrsrrceZdZdZy)SoftminzZApplies the Softmin function. See :func:`softmin` for the functional equivalent. Nrr rrrrrrrrrrrr)gr)CrQ functoolsrtypingrmlx.corecorermlx.nn.layers.baserrcompilerrr!r%r(r.r5r9r<r@rrErHrJrLrrSrYr[rrbrfrhrkrmrqrurwryrWr|rrrrrrrrrrrrrrrrrrrrrrrrr rrrs, % t$% t$% t$'%' t$-%- t$-%- t$9%9 t$7%7 t$$%$ t$% t$'%' t$ B B% B t$Q%Q t$% t$% t$ 2rxx 2% 2 t$ S% S  t$%%%* 288 3    t$)BHH))%)& t$ $% $  t$7RXX7bhh72887%7 t$ $BHH $ $% $ t$*%* t$7%7 t$ -% - t$%%%(&(*!f"66F F 33"#&#$!f""v#"v#))"$6$&6%&%&%F%(*#6*#Z6#$'6'06#v$%&!f"r