9jY#ddlmZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z m Z mZmZmZmZmZddgZGddeZd d e d ed e d ed ze_ d$deedeedeedeededzdedzdedzdededzdedededededededdf"dZd Zdeedeedeedeededzdedzdededededededededdfd!Zdeedeedeedeededzdedzdededededededededdfd"Zdeedeedeedeededzdedzdeezdedededededededdfd#Zy)%)castN)Tensor)_default_to_fused_or_foreach_device_dtype_check_for_fused_differentiable_doc _foreach_doc_get_scalar_dtype _get_value _maximize_doc _params_doc _to_scalar_use_grad_for_differentiable _view_as_real DeviceDict OptimizerParamsTAdagradadagradceZdZ ddddddedeezdededed ed edzd ed ed edzddffdZfdZddZ dZ e ddZ xZ S)rNF)maximizedifferentiablefusedparamslrlr_decay weight_decayinitial_accumulator_valueepsforeachrrrreturnc Dt|tr|jdk7r tdd|kstd|d|kstd|d|kstd|d|kstd|d|kstd||||||||| | d } t||| | r(| r t d |r t d d |_d |_|jD]} | d D]} |j| }| dr/tjdt| d| jntjdt|d<tj | r t#||n|}tj$| |tj&|d<y)NrzTensor lr must be 1-elementzInvalid learning rate: zInvalid lr_decay value: zInvalid weight_decay value: z)Invalid initial_accumulator_value value: zInvalid epsilon value: ) rrrrrr rrrz)`fused` does not support `differentiable`z0`fused` and `foreach` cannot be `True` together.Trris_fuseddtypedevicer(step memory_formatsum) isinstancernumel ValueErrorsuper__init__ RuntimeError"_need_device_dtype_check_for_fused_step_supports_amp_scaling param_groupsstatetorchzerosr r)tensor is_complexcomplex full_likepreserve_format)selfrrrrrrr rrrdefaultsgrouppr8 init_value __class__s S/media/conek/DATA/Code/OCR/venv/lib/python3.12/site-packages/torch/optim/adagrad.pyr3zAdagrad.__init__s b& !bhhjAo:; ;by6rd;< <h7zBC Cl";L>JK K//;  $"3U"C#$88 #\\(:K:MNFO   (DJJ--/0 l+q0 eoo OF #7 ! !LL!F)$,=u,M&  rGc~|jD].}|dD]$}|j|}|dj&0y)z6Calls tensor.share_memory_() on the state sum tensors.rr.N)r7r8 share_memory_)r@rBrCr8s rF share_memoryzAdagrad.share_memorysG&& -E8_ - 1 e **, - -rGcTd\}}|dD]}|j|drt|ddrt|d|_||jjz}|t j |z}|j||j|j|j|} t| dk(r|dr t||dr/t jdt|d |j nt jd t | d <|jd} t j |r t| | n| } t j || t j"| d<|j| d|j| d ||fS)N)FFrrr5TFrr$r%r'r#r*r+rr,r.)gradgetattrrr5 is_sparser9r<appendr8rLr:r r)r;rAr=r>r?) r@rBparams_with_gradgrads state_sums state_stepshas_sparse_grad has_complexrCr8rrDs rF _init_groupzAdagrad._init_groups'3$x( 2Avv!>g8' 2!4>CD;166#3#33u//22  ''* QVV$ 1 u:?W~5a8!>  "3U7^"L#$88 #\\#5F5HI&M15 31- !++A. 9;TU6 $)??:U5J5J$E%L!!%,/""5=1Q( 2T ++rGcbd}|$tj5|}ddd|jD]k}g}g}g}g}|j|||||\}} t |||||d|d|d|d||d|d|d| |d t |d dt |d d m|S#1swYxYw) zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrr rrr grad_scale found_inf) rrrrrbr rrrcrrfrg)r9 enable_gradr7rdrr[) r@closurelossrBr^r_r`rarbrcs rFr+z Adagrad.steps  ""$ !y !&& E-/ "$E')J(*K+/+;+;' K, (O[  ;">2z*%L /i(z*$%56'Gn"4t<!$ T:!  : A ! !s B%%B.)g{Gz?rrrg|=N)r!NN)__name__ __module__ __qualname__rrNrboolr3rIrXrdrr+ __classcell__)rEs@rFrrs"+,#E$!EE FNE E  E $) EEEEEd{E EN!F-,,\"*"*rGa[Implements Adagrad algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)}, \: \lambda \text{ (weight decay)}, \\ &\hspace{12mm} \tau \text{ (initial accumulator value)}, \: \eta\text{ (lr decay)}\\ &\textbf{initialize} : state\_sum_0 \leftarrow \tau \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm} \tilde{\gamma} \leftarrow \gamma / (1 +(t-1) \eta) \\ &\hspace{5mm} \textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}state\_sum_t \leftarrow state\_sum_{t-1} + g^2_t \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1}- \tilde{\gamma} \frac{g_t}{\sqrt{state\_sum_t}+\epsilon} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adaptive Subgradient Methods for Online Learning and Stochastic Optimization`_. z Args: a lr (float, Tensor, optional): learning rate (default: 1e-2) lr_decay (float, optional): learning rate decay (default: 0) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) initial_accumulator_value (float, optional): initial value of the sum of squares of gradients (default: 0) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-10) z a fused (bool, optional): whether the fused implementation (CPU and CUDA only) is used. Currently, `torch.float64`, `torch.float32`, `torch.float16`, and `torch.bfloat16` are supported. (default: None). Please note that the fused implementations does not support sparse or complex gradients. .. _Adaptive Subgradient Methods for Online Learning and Stochastic Optimization: http://jmlr.org/papers/v12/duchi11a.html rr_r`rarrfrgrbr rrcrrrrrr!c td|Ds td||t|| d\}}|d}|d}|r)tjj r td|r)tjj r td|r%tjj st }n-|r%tjj st}nt}|||||| | | |||| | ||y) ztFunctional API that performs Adagrad algorithm computation. See :class:`~torch.optim.Adagrad` for details. c3PK|]}t|tj ywrk)r/r9r).0ts rF zadagrad..6s@qz!U\\*@s$&zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF) use_fusedz6torch.jit.script not supported with foreach optimizersz4torch.jit.script not supported with fused optimizers rrrrrbrrrcrfrg) allr4rr9jit is_scripting_fused_adagrad_multi_tensor_adagrad_single_tensor_adagrad)rr_r`rarrfrgrbr rrcrrrrr_funcs rFrrs2 @K@ @ ^   }1 Ne 7 }599))+STT '')QRR UYY++- //1$%  ! '%rGcP|j}tj|||Srk)sizer9sparse_coo_tensor)rZ grad_indicesrPrs rF _make_sparsergs" 99;D  " "< >>rGc || tdtjjs t |}t ||||dD]\}}}}|dz }t |}| s|n| }|dk7r*|jr td|j||}|d|dz |zzz }|jr|j}|j}|j}|jt|||jd|j!|}|jj#j| }|jt||||z | &tj$|}|r?tj&|}tj&|}tj&|}|j)||d | r|j+| z}n|j+j| }|j-||| |stj.|}tj.|} y) N,Expected grad_scale and found_inf to be NoneT)strictrrz;weight_decay option is not compatible with sparse gradientsalphavalue)AssertionErrorr9ryrzrzipr r\r4addcoalesce_indices_valuesadd_rpow sparse_masksqrt_r< view_as_realaddcmul_sqrtaddcdiv_view_as_complex)rr_r`rarfrgrrrrrbrrrcparamrZ state_sumstep_tr+clrr grad_valuesstd std_valuesr<s rFr}r}ls!"!6KLL 99 ! ! # ^*-z;t+*=&tY ! &!#t$ 1 ~~"Q88E86DAX--. >>==?D==?L,,.K NN<lKOOA.s9 #DNN9 sTrwg?cpu)r)rrr)rrLrr"_group_tensors_by_device_and_dtyperPrrOranyr}rr9 _foreach_negcompiler is_compilingis_cpu _foreach_add_r; _foreach_addr _foreach_addcmul_ _foreach_sqrt _foreach_mul_ _foreach_mul_foreach_addcdiv_)rr_r`rarfrgrrrrrbrrrcgrouped_tensorlistsdevice_params_ device_grads_device_state_sums_device_state_steps_r~ device_params device_gradsdevice_state_sumsdevice_state_stepsdevice_has_sparse_gradr+ minus_clrr numerators rFr|r|s"BCC!6KLL 6{a BB#FF  K0 & & ( M?  T&\>: DL-8  f/AB!$v,0CD!0" S9 '39 6  " "!")! $!-'%#    -7H I  --l;L ~~**,1CA1F1M1M   "ELLU$C3     2A 6 1 ##L-|T$11 -|  GY >BRC1 4(1,88 9    1<UVW!!"34 C% 1     i 8$I**<CI  y#>[M?z s Jc|sy| s| r td| r td||j|ini}||j|ini}t|tr&t |jdk7r|j|ind}t j ||||g}|jD]:\\}}\\}}}}}ttt|}ttt|}ttt|}ttt|}d\}}|#|j||j|d}|#|j||j|d}|||vr|j|d||<||}tj|dtj|||||||| | || |tj||gt!|z=y) Nz5`fused` does not support sparse grad or complex paramz  9%B&b&1c"))n6MBSW  BB  K0O    +       T&\>: DL-8  f/AB!$v,0CD.8++  ! / : : f4 @!   -88 V$ ?    6#8 ee6eEGFOB .2     %(&   '   "%5$6=O9P$P S+rG)NNNFNFF)typingrr9r optimizerrrrr r r r r rrrrrr__all__r__doc__rOrorNrrr}r|r{r$rGrFrs $ i KiK^4      5.p $#" G LG <GV Gf G $; G  G}GGD[GGG G !G"#G$ %G&'G( )GT? A= LA= <A=V A=f A=  A= } A= A=A=A= A=A=A=A=A=  !A=Hl? Ll? <l?V l?f l?  l? } l? l?l?l? l?l?l?l?l?  !l?^S LS <SV Sf S  S } S SSS SSSSS  !SrG