9j<3"dZddlmZddlZddlmZmZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZmZmZdd lmZdd lmZddZddl m!Z!m"Z"m#Z#m$Z$er ddl%m&Z&ddl'm(Z(ddZ)GddeZ*GddZ+y)a Decomposition-based sharding propagation for DTensor. When an operator doesn't have a registered sharding strategy, we derive one by tracing through its decomposition. The decomposed ops (which do have strategies) determine how placements propagate through the original op. ) annotationsN)Any TYPE_CHECKING)decomposition_table)_are_we_tracing) DeviceMesh) DTensorSpec)OpSchema OpStrategyRuntimeSchemaInfo)try_find_mesh_from_args) _StridedShard Placement ReplicateShard)GuardOnDataDependentSymNode)TorchDispatchModec|j}d}t|jD]7\}}|jrn&|jj dk7s2|5|}ng}|jD]H}|js|jj dk7s.|j |jJi}|||d<|r||d<tdi|S)zGInfer RuntimeSchemaInfo from an operator's schema for decomposition opsN TensorType static_argnumstatic_kwargkey) _schema enumerate arguments kwarg_onlytypekindappendnamer )opschemariargkwarg_only_nameskwargss h/media/conek/DATA/Code/OCR/venv/lib/python3.12/site-packages/torch/distributed/tensor/_decompositions.py_infer_schema_info_from_opr( s ZZFMF,,-3 >>  88==?l *}/DM  . >>chhmmo=  # #CHH -.F "/$4 !  &v &&)tree_any tree_flattentree_map tree_map_only) OpOverload)ShardingPropagatorcb|jt|jjzSN) args_schematuple kwargs_schemavalues) op_schemas r'_extract_input_specsr7Fs'  5)@)@)G)G)I#J JJr)c4eZdZdZdfd ZddZddZxZS)PlacementTrackingModez TorchDispatchMode that tracks DTensor placements through op execution. Used during decomposition tracing: intercepts each op, propagates sharding via the ShardingPropagator, and records output placements on the result tensors. c>t|||_||_yr1)super__init__ sharding_propmesh)selfr=r> __class__s r'r<zPlacementTrackingMode.__init__Rs * r)ctd||xsif\}}td||fstd|t|||}|jj j |}|%|jjj |}|||_|jtr|jj|} n|jj|} | jr^| jx} Ot|j | j D],\} } | j"| j"k7s t%d|||i|} |j'| | j(| S)NcTt|tjr t|d|S|S)N_spec) isinstancetorchTensorgetattrxs r'z:PlacementTrackingMode.__torch_dispatch__..Ys! 1ell0Kga!,QRr)c"t|tSr1rDr rHs r'rJz:PlacementTrackingMode.__torch_dispatch__..^ jK0r)z(No DTensorSpec found in args/kwargs for z*Decomposition requires redistribution for )r,r*NotImplementedErrorr r=op_to_schema_infoget)op_to_schema_info_for_single_dim_strategy schema_info_recompute_comparison_keyr propagate_op_sharding_non_cachedpropagate_op_shardingneeds_redistributeredistribute_schemazip args_spec placements RuntimeError_record_output_specs output_spec)r?functypesargsr&r2r4r6rRoutput_shardingrWorigdesiredouts r'__torch_dispatch__z(PlacementTrackingMode.__torch_dispatch__Ws%- R 6 ((::>>tD  ""LLPPQUV   "$/I !  / / 1  "00QQO#00FFyQO  . .#667# "%###--"  g??g&8&88&DTFK  D#F# !!#'B'BC r)ct|tjr |||_yt|tt fr>t|tt fr't ||D]\}}|j||yyyr1)rDrErFrCr3listrXr\)r?outputr]tss r'r\z*PlacementTrackingMode._record_output_specssn fell + 0G&FL  .: %4 FK0 01))!Q/ 04 .r))r=r/r>r)rN)rhrr]zDTensorSpec | AnyreturnNone)__name__ __module__ __qualname____doc__r<rer\ __classcell__)r@s@r'r9r9Js 5n0r)r9c|eZdZdZd dZd dZed dZd dZ ddZ ddZ e ddZ y )DecompShardingStrategyaA Generates sharding strategies for ops by tracing through their decompositions. For each candidate input placement combination, runs the decomposition on meta tensors under PlacementTrackingMode to determine the output placement. These single-dimension strategies are then expanded to the full mesh. c ||_i|_yr1)r= _fake_meshes)r?r=s r'r<zDecompShardingStrategy.__init__s* 46r)c~|jj|}|t|dgdd}||j|<|S)NrF) _init_backend_rank)rurPr)r? device_type fake_meshs r'_get_fake_meshz%DecompShardingStrategy._get_fake_meshsG%%))+6  ";5PQRI-6D  k *r)c6|tvxs|jSr1)r_can_decompose)r!s r' has_decompz!DecompShardingStrategy.has_decomps((?B,=,=,??r)c~||jjvr%t|}||jj|<yy)z Register schema_info for decomposition op on first invocation. Needed for correct shard prop cache key. N)r=rOr()r?r!rRs r'ensure_schema_infoz)DecompShardingStrategy.ensure_schema_infos= T''99 94R8K7BD   0 0 4 :r)ctd|j|jfsy|j|}t |j |jt |jjz}|j|j}g}g}|D]Q} |j|||}t!|t s|gn t#|}|j%|t#|zS|s t'dt)|} |j*j-|} ddlm} | || || S#t$rYyt$rYytttf$rYwxYw)Nc"t|tSr1rLrHs r'rJz;DecompShardingStrategy.propagate_strategy..rMr)zGSharding propagation should have produced at least Replicate() strategyr)expand_to_full_mesh_op_strategy) input_index)r*r2r4_get_candidate_placementsr r!r3r5r{ry_propagate_through_decomprNrr[KeyError IndexErrorrDrgrAssertionErrorlenr=_wrap_with_op_strategy#torch.distributed.tensor._ops.utilsr) r?r6candidate_placementsr>rzsingle_dim_strategiesoutput_placementsinput_placementsrh n_outputsstrategy_schemars r'propagate_strategyz)DecompShardingStrategy.propagate_strategys 0  " "I$;$; < #==iH& LL  ! !E)*A*A*H*H*J$K K  ''(8(89 "EG 4 U  77$!+65 9tF|  " ( ():TBR=S)S T' U*% Y )* ,,CCIN / /#8i  5' .  (J7  sD44 E!E! E! E!ct|j}|tvr t|}n+|jr |j}nt d|t |fd}ddlm}|5t||j}t||j} t|j5||i| } ddddddd} t|  } t| ttfr0| D cgc]} | |  }} t!|dk(r|dSt|S| S#1swYhxYw#1swYlxYwcc} w)NzNo decomposition found for ct}t|trVtj|j |j jd}t|f|j |_|S|S)Nmeta)dtypedevice) tensor_meta) nextrDr rEemptyshaperrrC)rIprr>placement_iters r'to_metazADecompShardingStrategy._propagate_through_decomp..to_metasW^$A![){{177!--2E2EfU(tO  Hr)r)maybe_disable_local_tensor_modecxt|tjr t|dd}|r|jdSdSy)NrCr)rDrErFrGrZ)rispecs r' get_placementzGDecompShardingStrategy._propagate_through_decomp..get_placements7!U\\*q'40-1tq);t;r))r!rr} decomposerNitertorch.distributed._local_tensorrr,r2r4r9r=rDr3rgr)r?r6 placementr>r! decomp_fnrr args_meta kwargs_metarhrresultrflatrs ` @r'rz0DecompShardingStrategy._propagate_through_decomps. \\ $ $+B/I     I%(CB4&HI Ii  T , . > )*?*?@I"7I,C,CDK&t'9'94@ >"I== >  >  -0 fudm ,%7!A7D7!$i1n47 =%+ =  > >  > > 8s1*AD)- D6D)/D57D5D& "D))D2c  t|}tt|\}}th t t fd|g}|D]}t |t s|jdg&t } D]}t |tr<|t|jDchc]}t||jc}z}Ot |ts`|t|jDchc] }t|c}z}|jt|ttj|Scc}wcc}w)Nc:j|jSr1)updaterZ)rall_placementss r'rJzBDecompShardingStrategy._get_candidate_placements..4s..t?r)) split_factor)r7r+rgrr-r rDrsetrrangendimrr itertoolsproduct) r6 tensor_specs flat_specs_ candidatesroptionsrr#rs @r'rz0DecompShardingStrategy._get_candidate_placements)s,,I6 $T,%78 A+4+  ?  46  1DdK0!!4&)n-'HA!!]3%*499%5$ !*!!..I$$Au-eDII6F#GE!H#GGH!!$w-0 1 I%%z233$ $Hs %E 2E N)r=r/)rystrrkr)r!r.rkbool)r!r.rkrl)r6r rkzOpStrategy | None)r6r rztuple[Placement | None]r>rrkz!Placement | tuple[Placement, ...])r6r rkzlist[tuple[Placement | None]]) rmrnrorpr<r{ staticmethodr~rrrrrr)r'rsrss6@@C6 6  6 p00+0 0 + 0d 4 4 & 4 4r)rs)r!r.rkr )r6r rkz tuple[DTensorSpec | object, ...]),rp __future__rrtypingrrrE torch._decompr)torch.distributed._functional_collectivesrtorch.distributed.device_meshr&torch.distributed.tensor._dtensor_specr #torch.distributed.tensor._op_schemar r r torch.distributed.tensor._utilsr (torch.distributed.tensor.placement_typesrrrr%torch.fx.experimental.symbolic_shapesrtorch.utils._python_dispatchrr(torch.utils._pytreer*r+r,r- torch._opsr.'torch.distributed.tensor._sharding_propr/r7r9rsrr)r'rs{#% -E4>WWC N:'<PO%JKK0-K0\r4r4r)