9jGddlZddlZddlmZddlmZddlmZmZdddZ dZ ejfdZ dejfdZ dejfd Zejejfd Zejejfd Zejfd Zejfd ZejfdZddejfdZejejfdZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGdd eZGd!d"eZGd#d$eZ y)%N)Function)groupReduceOp suggestionc:d|d}|r |d|dz }t|)N torch.distributed.nn.functional.z& is not supported under torch.compile.z Use instead.) RuntimeError)namermsgs _/media/conek/DATA/Code/OCR/venv/lib/python3.12/site-packages/torch/distributed/nn/functional.py_not_supported_under_compilers7 *4&0VW zl),, s cJtjd|d|dtdy)Nr z is deprecated, use r )category stacklevel)warningswarn FutureWarning)r rs r _deprecatedrs, MM *4&1l) % rctjjr tddt ddt j |||S)a Broadcasts the tensor to the whole group. ``tensor`` must have the same number of elements in all processes participating in the collective. Arguments: tensor (Tensor): Data to be sent if ``src`` is the rank of current process. src (int): Source rank. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Received tensor from the broadcast op. broadcastz3torch.distributed._functional_collectives.broadcastr)torchcompiler is_compilingrr _Broadcastapply)tensorsrcrs rrr sD" ~~""$$ L  RS   C //rctjjr tdtj |||S)aT Gathers a list of tensors in a single process. Arguments: tensor (Tensor): Input tensor. dst (int, optional): Destination rank (default is 0). group (ProcessGroup, optional): The process group to work on. Returns: tuple[Tensor]: List of appropriately-sized tensors with the gathered data. gather)rrrr_Gatherr)r dstrs rr#r#:s/ ~~""$$X. ==eV ,,rctjjr tdt j ||g|S)a Scatters a list of tensors to all processes in a group. Each process will receive exactly one tensor and store its data in the ``tensor`` argument. Arguments: tensors (list[Tensor]): List of tensors to scatter on the source rank. Receivers must pass ``None`. src (int, optional): Source rank (default is 0). group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output tensor from the scatter operation. scatter)rrrr_Scatterr)tensorsr!rs rr'r'Ks2" ~~""$$Y/ >>#u /w //rctjjr tdtj ||||S)a Reduces the tensor data across all machines. Only the process with rank ``dst`` is going to receive the final result. Arguments: tensor (Tensor): Input of the collective. dst (int): Destination rank. op (optional): One of the values from ``torch.distributed.ReduceOp`` enum. Specifies an operation used for element-wise reductions. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output of the collective. reduce)rrrr_Reducer)r r%oprs rr+r+as1$ ~~""$$X. ==b% 00rctjjr tddt ddt j |||g|S)a Reduces, then scatters a list of tensors to all processes in a group. Arguments: output (Tensor): Output tensor. input_list (list[Tensor]): List of tensors to reduce and scatter. op (optional): One of the values from ``torch.distributed.ReduceOp`` enum. Specifies an operation used for element-wise reductions. group (ProcessGroup, optional): The process group to work on. Returns: Tensor: Output of the collective. reduce_scatterz?torch.distributed._functional_collectives.reduce_scatter_tensorr)rrrrr_Reduce_Scatterr)output input_listr-rs rr/r/xsN  ~~""$$ X I  UF @Z @@rctjjr tddt ddt j ||S)a Gathers tensors from the whole group in a list. Arguments: tensor (Tensor): Tensor to be broadcast from current process. group (ProcessGroup, optional): The process group to work on. Returns: tuple([Tensor]): Output of the collective. all_gatherz;torch.distributed._functional_collectives.all_gather_tensorr)rrrrr _AllGatherr)r rs rr4r4sG ~~""$$ T S   E6 **rctjjr tdtj |||S)a Single tensor all gather. Gathers a single tensor from all ranks, and puts them in a single output tensor. Args: output_tensor (Tensor): Output tensor. It should contain correctly-sized tensors to be used for output of the collective. input_tensor (Tensor): Tensor to be broadcast from current process. group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Examples: >>> # All tensors below are of torch.int64 dtype. >>> # We have 2 process groups, 2 ranks. >>> # xdoctest: +SKIP("incorrect want text") >>> output_tensor = torch.zeros(2, dtype=torch.int64) >>> output_tensor [tensor([0, 0])] # Rank 0 and 1 >>> tensor = torch.arange(1, dtype=torch.int64) + 1 + rank >>> tensor tensor([1]) # Rank 0 tensor([2]) # Rank 1 >>> dist.all_gather_base(output_tensor, tensor) >>> output_tensor tensor([1,2]) # Rank 0 tensor([1,2]) # Rank 1 .. warning:: `_all_gather_base` is experimental and subject to change. It is the caller's responsibility to ensure the output_tensor is correctly sized. _all_gather_base)rrrr_AllGatherBaser) output_tensor input_tensorrs rr7r7s3B ~~""$$%78    |U CCrctjjr tdt j ||g|S)a Each process scatters list of input tensors to all processes in a group and return gathered list of tensors in output list. Arguments: output_tensor_list (list[Tensor]): list of tensors to gather one per rank. input_tensor_list (list[Tensor]): List of tensors to scatter one per rank. group (ProcessGroup, optional): The process group to work on. Returns: tuple([Tensor]): Output of the collective. all_to_all)rrrr _AlltoAllr)output_tensor_listinput_tensor_listrs rr<r<s4 ~~""$$\2 ??5"4 I7H IIrctjjr tddt ddt j |||||S)a Each process splits input tensor and then scatters the split list to all processes in a group. Then concatenate the received tensors from all the processes in the group and return single output tensor. Arguments: output (Tensor): Gathered concatenated output tensor. input (Tensor): Input tensor to scatter. output_split_sizes: (list[Int], optional): Output split sizes for dim 0 if specified None or empty, dim 0 of ``output`` tensor must divide equally by ``world_size``. input_split_sizes: (list[Int], optional): Input split sizes for dim 0 if specified None or empty, dim 0 of ``input`` tensor must divide equally by ``world_size``. Returns: Tensor: Output of the collective. all_to_all_singlez;torch.distributed._functional_collectives.all_to_all_singler)rrrrr_AlltoAllSingler)r1inputoutput_split_sizesinput_split_sizesrs rrArAsT4 ~~""$$ T E   v)+[!! KKc 7[!! sB?c`dtj|j|jg|fzSNNN)r(rr%r)rQ grad_outputss rrYz_Gather.backwardQs(x~~cggsyyP<PRRRrNrZr_rrr$r$<s*""$SSrr$c,eZdZedZedZy)r(c4||_||_tfdDstt j d}t j||k(r$t j|t|||St j|d|||S)Nc3fK|](}|jdjk(*yw)rN)size).0tr)s r z#_Scatter.forward..\s'BQ1668wqz00Bs.1rrL) r!rallAssertionErrorrrdrMrNr'list)rQr!rr)r1s ` rrRz_Scatter.forwardWs B'BB !!'!*- ==u % , LLg5 A  LLs% 8 rc^dtj|j|j|zSrj)r$rr!rrQrWs rrYz_Scatter.backwardes$gmmCGGSYY LLLrNrZr_rrr(r(Vs*  MMrr(c,eZdZedZedZy)r,ct||_||_|j}tj|||||S)Nr-r)r!rrPrMr+)rQr!r-rr s rrRz_Reduce.forwardls4  FCBe4 rc`dtj|j|j|fzSN)NNN)rrr!rrxs rrYz_Reduce.backwardus)"Z%5%5cggsyy+%V$XXXrNrZr_rrr,r,ks*YYrr,c,eZdZedZedZy)r0c||_|j}td|D}tj|t ||||S)Nc3<K|]}|jywrTrerqrrs rrsz*_Reduce_Scatter.forward..s!LQ!,,.!Lr{)rrerfrMr/rv)rQr-rr r?s rrRz_Reduce_Scatter.forward|sJ ""$!!L:K!LL FD):$;%P rcHdtj|j|zSr})r5rrrxs rrYz_Reduce_Scatter.backwards "J$4$4SYY $LLLrNrZr_rrr0r0{s*MMrr0c,eZdZedZedZy)r5c|j}||_ttj|Dcgc]}t j |}}tj|||t|Scc}wrK) rerrbrMrcr empty_liker4rf)rQrr _out_tensor_lists rrRz_AllGather.forwardsp""$ .3D4G4Ge4T.U )*E  V $   u=_%%  sA:cntj|jtjjtjj fvrltj |j}tj||}tjtj|j|g|}d|fS|Dcgc]}tj|}}tj|j|g|}tjtj|d}d|fScc}w)NrLr)dim)rM get_backendrBackendNCCLXCCLrNrrr0rrrUr=sumstack)rQrlrOrXr rhgxss rrYz_AllGather.backwards   #)) ,1B1BDLLDUDU0V V==syy1D!!,t"45B &&x||SYYR\RBbzCOO5++F3OKO//#))[H<HC5;;s+3BbzPsD2NrZr_rrr5r5s( & &  rr5c,eZdZedZedZy)r8c`||_tj||j||SrK)rrMr7re)rQr9r:rs rrRz_AllGatherBase.forwards,  m\-D-D-FeTrctj|jtjjtjj fvrtj |j}t|j}|d|zdk7rtd|d||dtj |jz|d<tj||j|j}tj||tj |jn tdd|dfS)NrLrzTensor with dimensions: z8 does not have first dimension divisible by world_size: devicedtypezBackend not supported!)rMrrrrrrcrvrpr remptyrr_reduce_scatter_baserrU)rQrW world_sizeout_sizerXs rrYz_AllGatherBase.backwards    #)) ,1B1BDLLDUDU0V V,,399=JK,,./H{Z'1,".xj9IIS V#1+)<)<399)MMHQK!3!3;;L;LB  % %b+x||SYY O78 8b$rNrZr_rrr8r8s(   rr8c,eZdZedZedZy)r=cb||_ttj|Dcgc]}||j c}|_tj |}td|D}tj|tjjur]ttj|D]0}d}||k(r t|}tj|||||2t|Stj|t||t|Scc}w)NrLc3<K|]}|jywrTrrs rrsz$_AlltoAll.forward..s81 8r)rrbrMrcrpinput_tensor_size_listrNrfrrGLOOrvr'r<)rQrrr)rgmy_rankto_sends rrRz_AlltoAll.forwards ',T-@-@u-M'N& "#GAJOO & "--e,888   % (DLL,=,= =4..U;< J<"7mG _Q/!5I  J_%% OOW   _%%%& sD,c |jDcgc]4}tj||dj|dj6}}dt j |j|g|zScc}w)Nrrrk)rrrrrr=rr)rQrlrprhs rrYz_AlltoAll.backwardss 22   KK\!_33<?;P;P   ioociiT|TTT  s9A.NrZr_rrr=r=s*&&,UUrr=c,eZdZedZedZy)rBc||_|j|_||_||_t j ||||||S)N)rDrEr)rrp input_sizerDrErMrA)rQrr1rDrErCs rrRz_AlltoAllSingle.forwardsL !2 2   1/   rc tj|j|j|j}dt j |j||j|j|jfzS)Nr)NNNN) rrrrrrBrrrDrEre)rQrWr s rrYz_AlltoAllSingle.backwardss NN;#5#5[=N=N (  ! ! &&%%&&(  +   rNrZr_rrrBrBs(      rrBc,eZdZedZedZy)rHc||_||_|jtj}t j ||||S)N) memory_formatr{)rr-rPrcontiguous_formatrMrG)rQr-rr s rrRz_AllReduce.forwards= E,C,CD 2U3 rc`dtj|j|j|fzSrj)rHrr-rrxs rrYz_AllReduce.backwards)z// ;OQQQrNrZr_rrrHrH s*RRrrH)!rrtorch.distributed distributedrMtorch.autogradrrrrrWORLDrr#r'rUr+r/r4r7r<rArGrr$r(r,r0r5r8r=rBrHr_rrrsY # .6:"'04 -"%++0,$< #DL=BKKJ* ++ %P#,,ekk/6  ,ShS4MxM* Yh Y MhM"< X :!U!UH h D R Rr