9jHhUddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlZddlmcmZddlmcmcmZddlmZddlmZddlmZmZmZmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*dd l+m,Z,dd l-m.Z.ej^e0Z1da2e3dze4d<da5e3e4d<ejldKde3fdZ7ejldKde3fdZ8ejrddGddZ:ejrddGdde:Z;de:deefddfdZ?d e#d!e@e>d"fde#dzfd#ZAd e#d!e@e>d"fde#dzfd$ZBeCZDeCe@e>e@e>d"feEfe4d%<d&e#d!e@e>d"fd'e@e(d"fd(e@e(d"fd)e>d*eEd+eEddfd,ZFd-ee3d7e3dzd?e3dejfd@ZP dNdAdBdCedDejdzdEejdzd>e3f dFZRGdGdHejjZUGdIdJejjZVy)ON) defaultdict)Sequence)cache)cast)_are_we_tracingone_step_redistribute_cost)_StridedShardNotDecodableError DTensorSpec ShardOrderShardOrderEntry TensorMeta)assert_no_mixed_partial_types) DeviceMesh)_is_shard_like _StridedShardPartial Placement ReplicateShard) IntLikeType)get_active_debug_mode#_FORCE_MIN_COST_REDISTRIBUTION_PLANF,_DISABLE_REDISTRIBUTE_TRANSFORM_OPTIMIZATIONTenabledc#8Kt}|a d|ay#|awxYww)uX Context manager to control the redistribution planning strategy for DTensor operations. This context manager allows you to choose between two algorithms for computing the sequence of collective operations needed to redistribute a DTensor from one placement to another: - **Graph-based**: Uses Dijkstra's algorithm to find the minimum-cost path through all possible placement transformations. This approach considers the global cost of all collective operations and finds the optimal sequence. Best for complex redistribution patterns where reducing communication cost and memory overhead is critical. - **Greedy**: Uses a heuristic approach that makes locally optimal choices at each step. This is faster to compute but may not produce the globally optimal transformation sequence. Best for simple redistribution patterns or when planning speed is more important than optimal communication. **Default Behavior (without this context manager):** When this context manager is NOT used, the algorithm selection follows this priority: 1. **Non-default shard orders** → Always use graph-based algorithm (required for correctness) 2. **Explicit `use_graph_based_transform` parameter** to `_gen_transform_infos_non_cached` → Use the specified algorithm (True = graph-based, False = greedy) 3. **No explicit parameter** (default case) → Use greedy algorithm for faster planning **Behavior with this context manager:** This context manager overrides the default selection by setting the global flag `_FORCE_MIN_COST_REDISTRIBUTION_PLAN`, which takes precedence over the explicit `use_graph_based_transform` parameter (but not over non-default shard order requirements). **Cache Considerations:** The redistribution planner caches transform info for performance via the `@cache` decorator on `_gen_transform_infos`. If you need to change the algorithm selection for the same input specs, clear the cache using `_gen_transform_infos.cache_clear()` to ensure the new setting takes effect and doesn't reuse cached results from a previous run. Args: enabled (bool): If True, forces the use of the graph-based algorithm. If False, forces the use of the greedy algorithm. Default: True N)r)r old_values f/media/conek/DATA/Code/OCR/venv/lib/python3.12/site-packages/torch/distributed/tensor/_redistribute.py use_min_cost_redistribution_planr7s(j4I*1'8 .7+i+ disabledc#8Kt}|a d|ay#|awxYww)a Context manager to disable the transform optimization pass that merges consecutive same-type collectives into single flattened operations. When the optimization is disabled, ``_optimize_transform_infos`` becomes a no-op and returns the original list of ``_TransformInfo`` objects unchanged. This is useful for debugging or isolating issues related to the flattened collective merging logic. The flag can also be set directly:: torch.distributed.tensor._redistribute._DISABLE_REDISTRIBUTE_TRANSFORM_OPTIMIZATION = True Args: disabled (bool): If True (default), disables the optimization. If False, explicitly enables it (the normal default). N)r)r!rs r+disable_redistribute_transform_optimizationr#ts(*=I3;0A 7@4y4r frozenslotscTeZdZUeed<eeefed<eeed<dZ de dzfdZ y)_TransformInfomesh_dimsrc_dst_placements logical_shapec|jdkrt|jd|jdk(r tdy)NrzQTransformInfo should only be created if it is an op with some effect, not a no-op)r)AssertionErrorr*selfs r __post_init__z_TransformInfo.__post_init__sH ==1   " "1 %)@)@)C C c  DreturnNc|j\}}|jr|jry|jr t|ryt|r|jryt|r t|ryy)z Return a key for grouping transforms by communication type. Returns None for local ops (no communication needed), or a string that identifies the collective type for potential grouping/merging. all_reducereduce_scatter all_gather all_to_allN)r* is_partial is_replicater)r0srcdsts r_comm_type_keyz_TransformInfo._comm_type_keysl**S >>  0 0 2 ^^ ."5# C S%5%5%7 C ^C%8r2) __name__ __module__ __qualname__int__annotations__tuplerrrr1strr=r2rr(r(s7Mi233K((d r2r(cLeZdZUdZeed<eedfed<dZedzed<ddZ y) _FlattenedTransformInfoa4 Represents a flattened transform that combines multiple mesh dimensions into a single collective operation using a flattened DeviceMesh. Note: inherits the fields from _TransformInfo. Gets an __init__ with parent fields, followed by child fields, and runs parent validation (post_init) mesh.original_mesh_dimsN avg_scalectj||j(|jdkDstd|jyy)Nr-z"avg_scale must be > 1 if set, got )r(r1rJr.r/s rr1z%_FlattenedTransformInfo.__post_init__sL$$T* >> %>>A%$88HI& &r2r3N) r>r?r@__doc__rrBrCrArJr1rEr2rrGrGs2 c3h'!IsTz r2rGtransform_infocurrent_placementsshard_order_dictr3c z|j\}}t|tr |j}n |jf}t|t t zr|j}t}|D]L}t||dk(rtd|d|d|d||j||jNt||k(s#tdt|d|d|d|d| t|t t zr0|j} | |vrg|| <|D]} || j| |D]} ||| < y) zw Update current_placements and shard_order_dict in-place to reflect the effect of a single transform step. rz?Invalid shard_order update. No entries left to pop for src_dim z. transform_info=z, current_placements=z, shard_order=z^Mismatch between expected and removed mesh dims during shard_order update. Expected to remove z, but removed N)r* isinstancerGrIr)rrdimsetlen ValueErroraddpopappend) rNrOrP src_placement dst_placement mesh_dimssrc_dim removed_dim_dst_dimr)s r"_update_shard_order_and_placementsras$2#D#D M=."9:"55 #,,. -!67##e  =A#G,-2 Ui00@A**<)=>##3"46 OO,W599; < =9~,.y>". >""0!12&&8%9:/0 2 -!67## * *(* W %! 7H W % , ,X 6 75'48$5r2rHr\.cN|j}|jytfd|D}|j|}|j }t |dkDr|j }|jjD]}|j|k(s|cSy)z Query for an explicitly created flattened mesh using layout comparison. Searches root_mesh._flatten_mapping for a mesh whose layout matches the expected flattened layout for the given dims. Pure Python layout math. Nc3(K|] }| ywNrE).0imesh_dim_namess r z5_get_flattened_mesh_by_layout_impl..s;AnQ';sr-) _get_root_meshrgrC_get_slice_mesh_layoutcoalescerUnest_flatten_mappingvalues_layout)rHr\ root_mesh dim_names sliced_layoutexpected_layoutflattened_meshrgs @r"_get_flattened_mesh_by_layout_implrus##%I((N;;;I// :M#,,.O ?a)..0$44;;="  ! !_ 4! !" r2ctrjtjjjrFt ||yddlm}tjjj|t|St ||S)aB Query for an explicitly created flattened mesh using layout comparison. When tracing with compile_on_one_rank, delegates to a custom op so the flattened mesh appears as a call_function node derived from mesh (a graph input) rather than as a get_attr constant holding an unpicklable ProcessGroup. Nr) device_mesh) rtorch distributedconfigcompile_on_one_rankrutorch.distributed._opsrwops_get_flattened_submeshlist)rHr\r_s r_get_flattened_mesh_by_layoutr"saU..55II .dI > F;yy$$;;D$y/RR -dI >>r2_warned_flatten_issuesrwsrc_placementsdst_placementsnum_ops comm_typereasonct|||f}|tvrytj||j}|+|D cgc]} ||  } } dj d| D} nddj d|Dd|} d} |dk(rd | d } n"|d k(rd } n|d k(rd|d} nt d|t j| ||||| ycc} w)a- Warn once per (mesh, dims, reason) about inability to flatten operations. Args: device_mesh: The device mesh being used mesh_dims: Tuple of mesh dimensions that could not be flattened src_placements: Source placements for the redistribution dst_placements: Target placements for the redistribution num_ops: Number of sequential operations that will be performed comm_type: Type of collective operation (e.g., "reduce_scatter") reason: Either "no_flattened_mesh" or "uneven_tensor_shape" Nz, c3(K|] }d|d yw)"NrE)renames rrhz:_warn_flatten_optimization_not_possible..Zs?Tqa[?szdims c32K|]}t|ywrd)rD)reds rrhz:_warn_flatten_optimization_not_possible..\s$?SV$?sz of a%While redistributing from %s to %s, %d sequential %s operations will be performed. This is suboptimal: multiple collective operations have higher latency (separate kernel launches and synchronization points) and may give inconsistent results between ranks due to different reduction orders. %sno_flattened_meshz&To optimize, flatten mesh dimensions [z0] so DTensor can use a single operation instead.uneven_tensor_shape)z Unfortunately, because the tensor dimension is not evenly divisible by the product of the mesh dim sizes that would need to be flattened for the optimization to work, it can not be optimized.non_ascending_mesh_dimsz0it is not possible to merge non-ascending order z operations.zUnexpected reason: )hashrrWrgjoinr.loggerwarning)rwr\rrrrr cache_keyrgrrqdims_strcommon_warning reason_msgs r'_warn_flatten_optimization_not_possibler=s*k"Iv6I**y) //N!09:1^A&: :99?Y??499$?Y$??@[MR `$$=hZGwx ( (  , ,>yk V 26(;<< NNJ7;s Ctransform_infosc <t|dkr|Str|Sthddtdzdtffd }dt t t fdt t t fdtfd d ttdt tdztdzfffd }g}d }|t|krW||}|j} || s|j||d z }C|j} |g} |d z} | t|kr||| jrm|| j| rW| j|| | d z } | t|kr0||| jr|| j| rW|| \} }| |j| nJ|j| |dvr5t td| D}t|||t| | || }|t|krWt j#d|||S)a Optimize transform infos by merging consecutive same-type collectives into a single flattened operation when a matching flattened DeviceMesh exists. Merging requirements: - Operations must be consecutive in the transform list (no reordering). Notably, redistributing from P, P, P -> R, S, R is not optimized here and cannot be optimized due to optimization needing to fuse non-contiguous reductions, leaving this pattern vulnerable to numerics issues and suboptimal perf - Operations must have the same comm type (e.g., all allgather or all reduce_scatter) - Operations must have identical src_dst_placements (e.g., can't merge Partial->Shard(0) with Partial->Shard(1)) - A flattened mesh covering the relevant dimensions must exist - For reduce_scatter, tensor dim must be evenly divisible by flattened mesh size For nested sharding, the merged operation uses the logical_shape from the outermost mesh dimension (smallest mesh_dim index) which represents the global tensor shape needed for correct padding/unpadding. TODO: - all_to_all operations are excluded from merging, but it may be possible to merge them in some cases. >r7r5r6keyNr3c |vS)z:Check if a comm type key represents a mergeable operation.rE)rMERGEABLE_COMM_TYPESs r is_mergeablez/_optimize_transform_infos..is_mergeables***r2p1p2c||k(ry|\}}|\}}||k7ry|jr|jsytt|}tt|}ddh}|j|vxr|j|vS)z Check if two src_dst_placements can be merged. Allows merging of Partial("sum") and Partial("avg") since they can be combined: perform sum reduction, then scale by avg mesh dims afterward. TFsumavg)r9rr reduce_op) rrsrc1dst1src2dst2partial1partial2mergeable_reduce_opss rare_placements_mergeablez;_optimize_transform_infos..are_placements_mergeables 8 d d 4<!doo&7&& %u~   "6 6 ;""&:: r2infosc|t|dkry|dj|dj}tfd|Ds t dt d|D}t t |}|dk(r||k7ry|d k(r ||d d d k7ryt|}|y \}}|d k(r+tt|jt|fd }n~|dk(r`tt|jt|fd}|j}tjfd|D} || zdk7ry|dk(r|d}ntd|d } |} |j!r0tjfd|D} | dkDr | } t#d} | |f} t%d| |j||| d fS)a Try to create a flattened transform from 2+ same-type transforms. Returns (result, failure_reason) where: - result is the FlattenedTransformInfo if successful, None otherwise - failure_reason is None if successful, or one of: - "too_few_transforms": Less than 2 transforms provided - "no_flattened_mesh": No flattened mesh exists for the required dimensions - "uneven_tensor_shape": For reduce_scatter, tensor dim not evenly divisible r)Ntoo_few_transformsrc3DK|]}|jywrd)r*)reinforfirst_placementss rrhzJ_optimize_transform_infos..try_create_flattened..s'  %T%<%<>N O s z5All transforms must have mergeable src_dst_placementsc34K|]}|jywrdr))rers rrhzJ_optimize_transform_infos..try_create_flattened..s:D$--:r6)Nrr7N)Nrc"|jSrdr+x affected_dims rzI_optimize_transform_infos..try_create_flattened..aool6Sr2)rc"|jSrdrrs rrzI_optimize_transform_infos..try_create_flattened..rr2c3TK|]}j|j!ywrd)sizer)rerrws rrhzJ_optimize_transform_infos..try_create_flattened..s%248   /2s%()Nrr5z0Unsupported comm type for try_create_flattened: c3K|]I}tt|jdjdk(rj |j Kyw)rrN)rrr*rrr)rs rrhzJ_optimize_transform_infos..try_create_flattened..sH!8!8!;<FF%O  /sAAr-r)r)r*r+rHrIrJ)rUr*r=allr.rCsortedrrrrSmaxr+mathprodNotImplementedErrorr9rrG)rrr\sorted_mesh_dimsrtr;r<outermost_infotensor_dim_sizeeffective_shard_mesh_sizerJ merged_srcscalemerged_placementsrrrrws @@rtry_create_flattenedz7_optimize_transform_infos..try_create_flatteneds u:>-!866!H++-    !G :E::   !23 ( (,,6 , &,TrT2266{DTU  !,$S  $s+//L ,STN * *s+//L ,STN,::<HO(, 2> II!E qy! $U^ '- $#4,::##3#     r2rr-)rrrc34K|]}|jywrdr)regs rrhz,_optimize_transform_infos..Ys(C(Crz5_optimize_transform_infos original: %s, optimized: %s)rUr frozensetrDboolrCrrr(rGr=rYr*extendrrrdebug)rrwrrrrresultrfr current_keyrOgroupj flattenedfailure_reasonr\rrs ` @@r_optimize_transform_infosrys]: ?a3%%ST+#*++ )Y& ' -29i3G-H  @i N#i &-sTz9 :i X>@F A c/" "q!))+ K( MM$  FA  "44'+f E O$ $_Q/>>@A("557I LL+ , FA O$ $_Q/>>@A("557I%9$?! >  MM) $ MM% " "&(CU(C"CD 7""J" e c/" "f LL?RX Mr2DTensorRedistributePlanner_planner_cache dtensor_metactr t||Stj||f}|tvrt||}|t|<t|S)a Factory function to get or create a DTensorRedistributePlanner instance. This function provides transparent caching of planner instances based on device mesh and dtensor meta. Multiple calls with the same parameters will return the same cached instance for better performance. Args: device_mesh: The device mesh for the planner dtensor_meta: TensorMeta of the DTensor to redistribute Returns: A DTensorRedistributePlanner instance (potentially cached) )rrweakrefrefr)rwrrplanners rget_redistribute_plannerrssS)+|DD[)<8I&,[,G$+y! ) $$r2c,tjy)z8Clear the cache of DTensorRedistributePlanner instances.N)rclearrEr2r clear_redistribute_planner_cachersr2ceZdZdZej ddGddZdZede e e e fde fd Z ede de e e e ffd Ze d%d ed eedeedfde d zdedef dZdededd fdZ d&dZdeedfde de deffdZdedede dfdZddde dee dfde efd Zd!ed"edee dfde efd#Z d!ed"ede efd$Z!y )'ra This class is used to plan the collective calls to transform the local shard of the DTensor from its current spec to the target spec. Suppose there are N tensor dimensions and M mesh dimensions, the total possible state size will be (N+2)*M*M!. Note: Use get_redistribute_planner() factory function instead of direct instantiation for automatic caching. Tr$ceZdZUeedfed<eed<ejddddZ e dzed<dZ d Z d Z d e fd Zd e fd Zded efdZy)$DTensorRedistributePlanner.DistState. placementstensor_dim_to_mesh_dimNF)defaultinitreprcompare_hashcVtj|j|jSrd)r format_shard_order_strrrr/s r__str__z,DTensorRedistributePlanner.DistState.__str__s%55++ r2c"|jSrd)rr/s r__repr__z-DTensorRedistributePlanner.DistState.__repr__s<<> !r2cNtj|d|jy)Nr)object __setattr__ _compute_hashr/s rr1z2DTensorRedistributePlanner.DistState.__post_init__s"   ""$ r2r3cR|j |jS|jSrd)rrr/s r__hash__z-DTensorRedistributePlanner.DistState.__hash__s#!%!74:: QT=O=O=Q Qr2cDt|j|jfSrd)rrrr/s rrz2DTensorRedistributePlanner.DistState._compute_hashs$OO// r2otherct|tjsy|j|jk7ry|j|j f|j|j fk(SNF)rRr DistStaterrr)r0rs r__eq__z+DTensorRedistributePlanner.DistState.__eq__sae%?%I%IJzzU[[(++  ,, r2)r>r?r@rCrrBr dataclassesfieldrrArrr1rrrrrrEr2rrrsy)S.)) **-K--u5% sTz    "  Rc R 3   4 r2rc^t|ttzrtfd|DS|S)z.s<$-.s- U seEl C C s%')rCritems)rs r_dict_to_ShardOrderz.DTensorRedistributePlanner._dict_to_ShardOrders) $QWWY/   r2cvtt}|D]$}t|j||j<&|S)z1Convert ShardOrder to dict with tensor dim as key)rrr\r )rtensor_mesh_dim_dictentrys r_ShardOrder_to_dictz.DTensorRedistributePlanner._ShardOrder_to_dicts@ +40 KE59%//5J !1!1 2 K##r2NrHrrZ.src_shard_order use_strided_shard_as_shard_orderc.t||jk7r$tdt|d|j|rtj||d\}}|tj |}t |}tj|}tjt||}|g}g} |D]s} t| t} t| ||tjt|tj|} |j| | j| ut!|dg} t#| D]<\}} | rdnd}| j|| jt!||dz>d j%| S) a Generate a string representation of the sequence of state transitions (placements and shard orders) as described by the given transform_info. Args: mesh: The DeviceMesh used for the redistribution. transform_infos: A sequence of _TransformInfo objects describing each transformation step. src_placement: The initial tuple of Placement objects. src_shard_order: (Optional) The initial ShardOrder representing the mapping of tensor dimensions to mesh dimensions. If None, the default shard order is computed from src_placement and mesh. use_strided_shard_as_shard_order: If True, normalize _StridedShard placements into regular Shard placements with an explicit shard_order before stringifying. Returns: A string showing the sequence of DistState transitions, separated by '->'. zdimensions mismatch z vs Trrz-->z->r-)rUndimr.r &_normalize_placements_into_shard_ordercompute_default_shard_orderrrrrrCrRrGrarrYrD enumerater)rHrrZrr cur_placementrP cur_state state_listis_flattened_listrN is_flattened new_state trace_partsrf separators rstringify_transform_infosz4DTensorRedistributePlanner.stringify_transform_infoss6 }  * &s='9&:$tyykJ  ,BB!4$ +M?  ")EEmTO]+ 5II  /88 - /   )+- 3N%n6MNL . /? 3<<m$*>>?OPI   i (  $ $\ 2 3:a=)* ():; 7OA|!-4I   y )   s:a!e#45 6 7ww{##r2rwrc||_|jst|t||_t |j |_t|_t|_ |jy)z Initialize DTensorRedistributePlanner. Args: device_mesh: The device mesh for this planner dtensor_meta: TensorMeta of the DTensor to redistribute N) rw_is_current_rank_part_of_meshr.rrUshapetensor_dimensionrT"strided_shard_placements_in_targetpartial_reduce_ops_in_targetsetup_cost_callbacks)r0rwrs r__init__z#DTensorRedistributePlanner.__init__.se'88:   ( #L$6$6 7FIe/69e) !!#r2cXdtjdtffd fd}|_y)z Set up the cost function for different collective operations. Uses communication time estimation based on actual tensor sizes and mesh topology for accurate cost modeling. stater3cttj|jj|jdS)NF)rHr tensor_meta shard_orderr)r rwrrr)r0r0s r state_to_speczFDTensorRedistributePlanner.setup_cost_callbacks..state_to_specNs9%% ++ --!8816  r2c4t||Srdr) src_state dst_stater4s r cost_functionzFDTensorRedistributePlanner.setup_cost_callbacks..cost_functionYs-i(- *B r2N)rrr r8)r0r8r4s` @rr-z/DTensorRedistributePlanner.setup_cost_callbacksEs- -77     +r2rtensor_mesh_dim_tuplerc  i}tj|}|j|j||}|D]}|j}||d}t ||t s+t|jD]} || k(r ||j} || j| t|} t | | | <|j|j| tj|} |j|| || <||j| || j|D]}|j}||d}t ||t s+||j} t|} t| | <|j|j| tj|} ||j| |j|| || <t|D]d\}} t | t st|} t| |<|j|j| |} |j|| || <ft|D]\}} t | tst|jD]} t|} t | | |<|| j||j|j| tj|} |j|| || <|| jt|D]\}} t | t st|jD]} t|} t | | |<|| j||j|j| tj|} |j|| || <|| jt|D]\}} t | ts|j"D]}t|} t!|| |<| Dchc]}t |t s|j$!}}t'|dkDr|ddhk7r\|j|j| |} |j|| || <|D]}|j}||d}t ||t(s+||j} t|} t| | <|j|j| tj|} ||j| |j|| || <|j*s|St|D]\}} t | ts|j*D]}|j,} t|} || |<|| j||j|j| tj|} |j|| || <|| j|Scc}w)Nrr-rr)rrrr r rRrranger*rXrYrrr8rrrr,rrUrr+rS)r0rr9all_next_statercur_dist_statersrc_tensor_dim src_mesh_dimdst_tensor_dim move_mesh_dimnew_placements dist_state placementr)rppartial_reduce_opsstrided_shard_objs rget_next_statez)DTensorRedistributePlanner.get_next_state`sHMO9MM !  NN: & ! + ;E"--N/?CLj6>"'(=(="> ;!^3!5^ D H H J $^4;;MJ!%j!105n0E}-!^^NN>2.BB, .2-?-?".z* %^4;;MJ$^488:+ ;  ;B+ E"--N/?CLj6>0@DDFM!*-N,5KN= )~.*>>?STJ ! 0 7 7 F)-););*N: & *(1'<  #L)i1!*-N+4;N< (~.0EJ*.););*N: & $-Z#8 ; Hii3"'(=(="> ;!%j!1+0+@x($^4;;HE!^^NN>2.BB, .2-?-?".z*%^488: ; ;,$-Z#8 ; Hii1"'(=(="> ;!%j!1+0+@x($^4;;HE!^^NN>2.BB, .2-?-?".z*%^488: ; ;0$-Z#8  Hii3!>>  !%j!1+29+=x(*8&$%:a;QAKK&"&)*Q.3E%QV3W!^^NN>24I .2-?-?".z*  :+ E"--N/?CLj6 F0@DDFM!*-N,5KN= )~.*>>?STJ ! 0 7 7 F)-););*N: & &66! !$-Z#8 ; Hii3%)%L%L ;!!2!6!6!%j!1+<x($^4;;HE!^^NN>2.BB, .2-?-?".z*%^488:! ; ;*Q&s =X;X;r6r7cddl}d}d|||gfg}t}|r|j|\}}} } | |k(r| S| |vr$|j| |j | j | j } | jD]0\} } | |vs || z}| | gz}|dz }|j|||| |f2|rtd|d|)aB Find the min cost path from src_state to dst_state using Dijkstra's algorithm. Args: src_state: The source state dst_state: The destination state Returns: A list of states representing the min cost path from src_state to dst_state rNr-zNo path found from src_state z to dst_state ) heapqrTheappoprWrHrrrheappushr.)r0r6r7rJcounterpqvisitedcostr_ current_statepath next_states next_statetransition_costnew_costnew_paths rfind_min_cost_pathz-DTensorRedistributePlanner.find_min_cost_pathzs )i[ 1 2 %+0==+< (D!]D ) ' KK &--((-*N*NK0;/@/@/B R+ OW,#o5H#zl2HqLGNN2':x'PQ  R"+I;nYK P  r2r)full_tensor_shapec lt|}|jD]}|j}|j}t |dkrt |D]}||k(r |j |} t| trM| j|||jj||jj|\} } nkt| trM| j|||jj||jj|\} } ntd| | ||<|S)NrrzUnsupported placement type: )rrr r\rUr.rrRrlocal_shard_size_and_offsetrwr_sym_get_coordinaterrV) r0r6r)rYnew_logical_shaperr r\mdimrDnew_sizer_s rget_logical_shapez,DTensorRedistributePlanner.get_logical_shapesA 044E/F55 9E))JI9~"$$! 98#%006 i/"+"G"G)*5((--t-<((<._try_normalize_specsv44FF 9=, &{22##+$'DTF%KLL(8(888r2rz@Multiple mesh_dims are different between cur_state and nxt_stater)r*r+)r rCrr rRrr+rW itertoolschainrr,rrrXpairwiserrzipr.r`rYr()r0rarbrYrfrrrdst_shard_orderrDr6r7r state_pathr nxt_stateupdate_mesh_dimr)r nxt_placementr+s r$generate_graph_based_transform_infosz?DTensorRedistributePlanner.generate_graph_based_transform_infoss 9 9 5C(*4 5 9"+>h*G'*=h*G' ( GI)]377;;IF G#H KI)W-1155i6I6IJ KNN>?C NN>?C 02,,Y B $-$6$6z$B  Iy##y';';;"$@I ,,i.B.BCA(>%x1B) (..*)84A=3Q.;  0r2c t|j}|g}g}|jjdk(rZ|jd|jdk7r9|j t d|jd|jdf||St|jD]\}}||}t|tr||jjdz ks8|jj|} |j||j| |jj|\} } t|} | | |j<|j | |j |t|j} t|j}|jdkDrtt!t#| D]}| |}||}t|tr|j}gg}}tt%| |D]T\}\}}||k\rnG|j'|r|j ||j'|sD|j |V||k7r t)}||k7s|j t |||f|||| |<tt%| |D]5\}\}}||k7s|j t |||f|||| |<7|S)a Generate the transform infos from the source placements to the target placements. To transform from source to target placement it might have multiple steps, i.e. it might decompose Si -> Sj into Si -> R -> Sj. This would detect if there're mis-aligned/nested shardings between src/dst placements. E.g. Suppose the redistribution to perform is (Shard(0), Shard(0)) -> (Replicate(), Shard(0)), in this case Shard(0) -> Shard(0) for mesh dimension 1 actually needs resharding, because in the former is a nested-sharding of a tensor already already sharded dimension 0, whereas the latter is the first sharding on tensor dimension 0. r-rrgr)rr)rwrrrYr(rrRrr_local_shard_size_and_offsetrSr\ num_shardsreversedr;rUrkis_shardr)r0rarbinitial_logical_shapemesh_dims_to_logical_shaperrfr;current_logical_shape mesh_dim_sizelocal_shard_sizer_r]rOtarget_placementsr)currenttarget shard_dimcurrent_mesh_shardingtarget_mesh_shardingsrEs rgenerate_greedy_transform_infosz:DTensorRedistributePlanner.generate_greedy_transform_infoss2$!%X^^ 4&;%<"02    A %""1%)<)q$A !#u%t'',,q00$($4$4$9$91$9$EM*-*J*J-cgg6%((<r?r@rMr dataclassrr  staticmethoddictrArr rrrrr(rCrrrDr&rr.r-floatrHrXrr`r rqrrEr2rrrs0[$d3..4.`  tCcN3   $z$d3S >.B$$ .216 G$G$!.1G$Y^,G$$d* G$ +/ G$ G$G$R$$!$  $.+ +6V)S.)V *V 4e; < Vt0 "0 /80 4 50 d!9!!!c? ! k  !BGGG!c? G n  GR~~~ n  ~r2rarbuse_graph_based_transformc|j}|j}|j}td||fD }tdg|j|jD}|s|rd}nt t }n|d}|j tt||j }|r |j|||j} | S|j||} | S#t$r|j||} Y| SwxYw)Nc3FK|]}tj|ywrd)r is_default_device_order)reorders rrhz2_gen_transform_infos_non_cached..s#$  ++E2$s!c3<K|]}t|tywrd)rRr)rerEs rrhz2_gen_transform_infos_non_cached..s  1m$sTF) rwr3ranyrrr2r.rrqr)r r) rarbrrwrrlhas_non_default_orderhas_strided_sharddrprs r_gen_transform_infos_non_cachedrs< &&K**O**O!$$%7$!=8&&=)<)<= 1$(! , 8$G! " *$)!# " C! V!FF(HNNO ==hQ  . V!AA(HUO   Vs(CC:9C:ct|||Srd)r)rarbrs r_gen_transform_infosrs +(5 r2)async_opr is_explicit local_tensor current_spec target_specrrc  |j|jk7r td|j tdt |j t |j |}|j}|j s|Strt|||}n t|||}t|||j |j } t} | _| j||j |j tj|| |j |j|j|nt!j"} | 5| D] } t%| t&r | j} n|} | j(}| j*\}}t%|t,st%|t,r | |usJd| j/|}||k(r|}|dk(r|}|j1r|j3rRt5t6|}|j9|| |}t%| t&r| j:|| j:z }n|j=r0t5t>|}|jA|| || jB}nt%|t,r |jA|||| jB}nrtEd|d |d |j=rqt5t>|}|j3rSt5t6|}|jG|| ||}t%| t&r| j:|| j:z }n|j1r%|jI|| || jK|}n|j=rUt5t>|}|jL|jLk7rm|jO|| || jB|jL}nBt%|t,rC|jA|||| jB}|jI|| || jK|}ntd |d ||j3rm|j1r%t5t6|}|jQ|| |}ntS|rtEd|d |d ||k7rtUd |d|d|}nat%|t,rP|j3rGt5t6|}|j9|| |}|jW||||jK|}n|j1r$|jW||||jK|}n|j=rRt5t>|}|jA|| || jB}|jW||||jK|}nct%|t,rB|jA|||| jB}|jW||||jK|}ntd |d ||s*t%|tXjZr|j]}|} ddd|S#1swY|SxYw)z This redistribute the local tensor (torch.Tensor) from the current DTensorSpec to the target DTensorSpec, which involves the necessary collective calls to transform the local shard of the DTensor from its current spec to the target spec. z)Cross device mesh comm not supported yet!NzUuse_strided_shard_as_shard_order should be initialized in DTensorSpec.__post_init__())rz:_StridedShard redistribute assumes no flattened transformsrr-zredistribute from z to z not supported yetzUnexpected placement z& for redistribute to target placement z&Redistribution from one partial type (z) to another (z) is unsupported.)/rHrrrVrrr(rrrrrrecord_redistribute_callsrr&r3 contextlib nullcontextrRrGr)r*rrr:r9rr _reduce_valuerJrvr_to_replicate_tensorr+ RuntimeError_reduce_shard_value_replicate_to_shardr\rS_to_new_shard_dim_partition_valuerr._replicate_to_strided_shardfuncolAsyncCollectiveTensorwait)rrrrrrnew_local_tensorrwroptimized_transform_infos debug_moderedistribute_contextrN mesh_to_userfr}r~ num_chunks partial_speccurrent_placementtarget_placement shard_spec replicateds rredistribute_local_tensorrs K,,,!"MNN44< c  ","9"9:!+"8"89###K  4 4 69 +'@ / +'@  !: !'(J  ! ,,   # #  " " & @ @)''((==  $ -  # # %$ b,7a ,N.*AB,11 ) ''A,??OGV '=1Z 5V"k1P1%))1)5J& #/ Q$0 ""$%%'#'#9L'3'A'A$k1($ #>3JK*44@+;n>V>V+V(%%'(,UG(<%'8'M'M$k1n6R6R($ 7'.'C'C$k1n6R6R($',WIT&AST"#'v#6 %%'#'#9L'3'G'G$k16F($ #>3JK*44@+;n>V>V+V())+'7'K'K$k1k6U6UVW6X($%%'!%eW!5J!~~)9)=)==+5+G+G('*88,00 ,( 7!(!=!=$k1n6R6R"J(8'K'K"##77: ($%/y8^_e^fg""$'')#'#8L'3'D'D$k1($$G,&,WIT&AST&(,DWI^\b[cctu(4$FM2%%'#'#9L!-!;!;$k1"J(.'I'I"KK4S4STU4V($))+'-'I'I$k1k6U6UVW6X($%%'(,UG(<%!2!G!G$k1n6R6R"J(.'I'I"KK4S4STU4V($ 7")!=!=$k1n6R6R"J(.'I'I"KK4S4STU4V($%/y8^_e^fg  &">">!$4#8#8#: +LCa ,b,F Gb,F s TY,,Y6 grad_outputdtensor.DTensor previous_specoriginal_dtypebackward_dtypec t|||jjk7r|jj|}t|jj |jj t|j|j||jj}t|j |j |j|j}n|j}|j}g}t|j |j D][\}} t|s|jr*| jr|j!t#K|j!| ]t|j t%||j|j}t'||||} | j|k7r| j|} t|j t%|t|j|j| j|j} | | fS)am Common function for redistributing a distributed tensor during backward and twice-backward backpropagation steps. Args: grad_output: The output gradient tensor. previous_spec: DTensorSpec prior to redistribution. original_dtype: Original output tensor dtype from forward pass (for type checking) backward_dtype: Desired data type for backwards output. async_op: whether to perform the DTensor redistribute operation asynchronously or not. Default: False Returns: A :class:`torch.Tensor` object. A :class:`DTensorSpec` object. dtyper)striderrHrr2r)rr2r)r)r2r) _local_tensorrtor _specrwrrr)rrr2rkrr:r9rYrrCr) rrrrrrrnormalized_placementsr}r~outputrds r_redistribute_backwardrs.!n 8Q8Q8W8W&W"0033.3I """.."((33"!''"))+$  .9->->-_-_  $**$//$00-:-[-[  #00 "(( .0|66 8P8PQ1 7 #w';';'=6CTCTCV ! ( ( 5 ! ( ( 0 1  !!./!--)6)W)W M' F||~%>* !! #$##%%',, *7)W)W D 4<r2ceZdZe d dddedeedfdedejdzd ejdzf d Z ed d Z y) RedistributeNinputrrwr.r forward_dtyperc t||_||_|jj|_|||jjk7rx|jj |}t ||jjt|j|j||jj}n|j}|j}||_ |j|k7r)t |||j} t||| |d} n|} |} t!j"| | |j$S)Nrrr)r2T)rr requires_grad)rrrrrrr rrrr)rrrr2rdtensorDTensorr) ctxrrwrrrrrrrrs rforwardzRedistribute.forward+s)  +"0066  $%:M:M:S:S)S ..11 1FL&  ;;11&++ <<>' 271]1] L!..L ;;L'  " "j 0%Z\5M5MK/! F"F&K  --   r2c|j}tj|||j|j|j }|dddddfSrd)rNestedRedistributeapplyrrr)rrroutput_dtensors rbackwardzRedistribute.backwardfsZ(( +11   LL                 r2FNN)rr) r>r?r@rrrCrrrxrrrrEr2rrr*s,0-18 !8  8 )S.) 8  8 {{T)8  d*8 8 t  r2rc eZdZdZe d dddededejdzdejdzf d Z ed d Z y) ra This class is used to make the redistribution of a DTensor twice-differentiable. This is called during the `Redistribute.forward`. Therefore, `NestedRedistribute.forward` is called during the first backward pass, and `NestedRedistribute.backward` is called during the second backward pass. Note: `NestedRedistribute.backward` is not differentiable, and therefore triple backward is not yet supported. Nrrrrrrc||_|jj|_|xs |j|_t |||j||\}}||_tj|||jS)Nr) rrrrrrrrrr)rrrrrrrrds rrzNestedRedistribute.forwards  (66<<+As/A/A-            %33   r2c|j}|j}|jxs |j}tj |||||j}|dddddfSrd)rrrrrr)r grad2_outputrrrrs rrzNestedRedistribute.backwardso(( <<++As/A/A+11                  r2r)rr) r>r?r@rMrr rrxrrrrEr2rrrzs ,0-1 ' #    {{T)   d*  @  r2r)TrLrd)NNF)Wrrrhloggingrr collectionsrcollections.abcr functoolsrtypingrrx)torch.distributed._functional_collectivesry_functional_collectivesrtorch.distributed.tensor._apitensor_apirr*torch.distributed.tensor._collective_utilsr &torch.distributed.tensor._dtensor_specr r r r rtorch.distributed.tensor._utilsr$torch.distributed.tensor.device_meshr(torch.distributed.tensor.placement_typesrrrrrr torch.typesrtorch.utils._debug_moder getLoggerr>rrrrBrcontextmanagerrr#rr(rGrrrArarCrurrTrrDrrr ReferenceTyperrrrrTensorrrrautogradFunctionrrrEr2rrs #$ :://EQJ;$9   8 $ 48#TD[7 6;,d: 98d9898x A$AA8d$/  0 Fd$/n04.5".5Y.53S >*.5 .5b !&sCx$B? ?!&sCx?$?0ADE#uS#X";<=E99S#X9)S.)9)S.) 9  9  9 9 9xp.)pp)S.)p)S.) p  .2 23 pn '   +Z 78 " %%%"%4 B B P.2666 $d{6 . 6r.2 $d{ . -1o,,ooo  o $d{ oo \\oj*.)- Z"ZZKK$&ZKK$& Z  ZzM 5>>**M `A 00A r2