9j:UdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z ddl mZddl mZmZmZmZmZmZmZddlmZmZmZmZmZddlZddlmZdd lmZ dd l!m"Z"dd l#m$Z$dd l%m&Z&m'Z'm(Z(m)Z)m*Z*e jVr dd l,m-Z-ddl.m/Z/dZ0e1Z2de3d<e4f dSdZ5gdZ6e eeeeegefZ7e eeeegefZ8edZ9de3d<edZ:GddedZ;Gdde;dZ<GddeZ=Gdd eZ>Gd!d"eZ? dTd#Z@ dU dVd%ZA dWd&ZB dXd'ZC dYd(ZDd)ZEd*ZFdZd+ZGd[d,ZH d\d-ZI d]d.ZJe'd$d/ZKGd0d1ZLeLZMGd2d3e j$ZNGd4d5e j$ZO d^ d_d6ZPd7ZQGd8d9ZRGd:d;ZSd`d<ZTdad=ZUeEeEdf dbd>ZVdcd?ZWdcd@ZXeEeEf dddAZYeEeEf dedBZZ d^ dfdCZ[deEdf dgdDZ\dhdEZ] d^ didFZ^djdGZ_djdHZ` dkdIZae dldJdK dmdLZeedMebN dldJdK dndOZe dl dodPZe dl dpdQZ dqddK drdRZy)szIThis module implements the user facing API for flex_attention in PyTorch.) annotationsN)Callable)Enum)AnycastLiteral NamedTupleoverload TypeAliasTypeVar) deprecatedNever NotRequiredSelf TypedDict)Tensor)flex_attention)setup_compilation_env)_validate_sdpa_input) GetAttrKey tree_flatten tree_map_onlytree_unflattenTreeSpec)DeviceLikeType)BaseArgumentTypesFzset[str]_WARNINGS_SHOWNc|tvrLtjjst j ||dtj |yy)z=Helper to ensure each warning is shown only once per process. stacklevelN)rtorchcompiler is_compilingwarningswarnadd) warning_idmessagecategorys a/media/conek/DATA/Code/OCR/venv/lib/python3.12/site-packages/torch/nn/attention/flex_attention.py _warn_oncer,<s?(~~**, MM'8 :J')) BlockMaskr AuxOutput AuxRequestFlexKernelOptionscreate_block_mask create_maskor_masks and_masks noop_mask)AUTOTRITONFLASH TRITON_DECODEr _Backend_RceZdZUdZded< ded< ded< ded< ded< ded< ded < ded < d ed < d ed < d ed< d ed< d ed< d ed< ded< ded< ded< ded<y)r1aOptions for controlling the behavior of FlexAttention kernels. These options are passed to the underlying Triton kernels to control performance and numerical behavior. Most users will not need to specify these options as the default autotuning provides good performance. The options can be prefixed with ``fwd_`` or ``bwd_`` to apply only to forward or backward pass respectively. For example: ``fwd_BLOCK_M`` and ``bwd_BLOCK_M1``. Note: We currently do not provide any backward compatibility guarantees for these options. That being said most of these have remained pretty stable since their introduction. But We do not consider this part of the public API just yet. We think that some documentation Is better than secret hidden flags, but we may change these options in the future. Example Usage: .. code-block:: python # Using dictionary (backward compatible) kernel_opts = {"BLOCK_M": 64, "BLOCK_N": 64, "PRESCALE_QK": True} output = flex_attention(q, k, v, kernel_options=kernel_opts) # Using TypedDict (recommended for type safety) from torch.nn.attention.flex_attention import FlexKernelOptions kernel_opts: FlexKernelOptions = { "BLOCK_M": 64, "BLOCK_N": 64, "PRESCALE_QK": True, } output = flex_attention(q, k, v, kernel_options=kernel_opts) # Forward/backward specific options kernel_opts: FlexKernelOptions = { "fwd_BLOCK_M": 64, "bwd_BLOCK_M1": 32, "PRESCALE_QK": False, } output = flex_attention(q, k, v, kernel_options=kernel_opts) zNotRequired[int] num_warps num_stagesBLOCK_MBLOCK_NBLOCK_M1BLOCK_N1BLOCK_M2BLOCK_N2zNotRequired[bool] PRESCALE_QKROWS_GUARANTEED_SAFEBLOCKS_ARE_CONTIGUOUSWRITE_DQFORCE_USE_FLEX_ATTENTIONUSE_TMAkpackmatrix_instr_nonkdim waves_per_euzNotRequired[_Backend]BACKENDN__name__ __module__ __qualname____doc____annotations__r-r+r1r1Ys'V L! Nbb ,,,,#"6,+Q -,A G0/-/ 1**;""1 "" r-r1)totalc"eZdZUded<ded<y)_KernelOptionsWithInternalsboolOUTPUT_LOGSUMEXP OUTPUT_MAXN)rQrRrSrUrVr-r+rYrYsr-rYc.eZdZUdZdZded<dZded<y)r0zRequest which auxiliary outputs to compute from flex_attention. Each field is a boolean indicating whether that auxiliary output should be computed. FrZlse max_scoresNrQrRrSrTr^rUr_rVr-r+r0r0s CJr-r0c.eZdZUdZdZded<dZded<y)r/zAuxiliary outputs from flex_attention operation. Fields will be None if not requested, or contain the tensor if requested. N Tensor | Noner^r_r`rVr-r+r/r/s C $J $r-r/ceZdZdZdZdZdZy)_ModificationTypezEnum for the type of modification function. - SCORE_MOD: score_mod function which accepts a score as the first argument - mask_mod: mask function which does not accept a score and is only used for generating block mask rN)rQrRrSrT SCORE_MODMASK_MODUNKNOWNrVr-r+rdrds IHGr-rdct|drG|j}|j}d}t|dr|jxsd}t |}||z }n=t dt j|jjD}|dk7r|dk7rtd||dk(rtjS|dk(rtjStjS)a\Get the type of modification function. This function inspects the number of positional arguments of the function to determine the type of modification function. If the function has 5 positional arguments, it is considered as a score_mod function. If the function has 4 positional arguments, it is considered as a mask function. __code__rV __defaults__c3lK|],}|jtjjurd.yw)reN)defaultinspect Parameterempty).0params r+ z _get_mod_type..s0" }} 1 1 7 77 " s24z%Expected 4 or 5 positional args, got )hasattrrk co_argcountrllensumro signature parametersvaluesAssertionErrorrdrgrhri)fncodenum_positional_totaldefaults num_defaultsnum_positional_argss r+ _get_mod_typersr:{{#// 2~ &,"H8} 2\A!"  **2.99@@B"   a$71$<34G3H I  a ***  ! ))) (((r-rVcxg}gd}|r|dgz }|dgz }|D] }tj|||z|z|}"|S)aUsed to vmap both score_mods and mask_mods over 4-dimensional/5-dimension inputs. Mapping over the [b, hq, q_idx, kv_idx] or [b, hkv, g, q_idx, kv_idx] dimensions. Args: fn (callable): The function to vmap. prefix (tuple): The prefix of the vmap. For score mod functions, this should be set to (0,). For mask_mods = () suffix (tuple): We need to add (0,) if gradOut is being mapped over, and (None,) * len(other_buffers). out_dims (tuple): For forward cases, keep this as the default 0 since we are only returning 1 output. For backwards, the joint graph returns grads for B, H, Q_idx, KV_idx and other_buffers, so we set this to (0, None, None, None, None) + (None,) * len(other_buffers). Returns: callable: The vmapped function. ))NNNr)NNrNNrNNr)rNNN)in_dimsout_dims)r"vmap)rprefixsuffixr group_dim dimensionsdimss r+_vmap_for_bhqkvr(sp2OQJJ  !  JO ZZFTMF$:X NO Ir-c|SNrV)scorebatchheadtoken_qtoken_kvs r+ _identityrVs  Lr-cZ|jdtj|jS)zReturns a noop mask_modrV)sizedtypedevice)new_onesr"rZrrrrrs r+r6r6`s! >>rELL> IIr-ctd)z Raises helpful error when using mask_mod from a sliced BlockMask. After slicing a BlockMask, the mask_mod is reset and cannot be used directly. Users must reassign mask_mod from the original (unsliced) BlockMask. a&Cannot use mask_mod from a sliced BlockMask. When you slice a BlockMask using [], the mask_mod attribute is reset. You must set it from the original BlockMask's mask_mod. Incorrect usage: base_mask = create_block_mask(my_mask_fn, ...) sliced_mask = base_mask[:, :, block_idx] sliced_mask.mask_mod = apply_offset(sliced_mask.mask_mod, offset) # WRONG! Correct usage: base_mask = create_block_mask(my_mask_fn, ...) sliced_mask = base_mask[:, :, block_idx] sliced_mask.mask_mod = apply_offset(base_mask.mask_mod, offset) # Use base_mask!) RuntimeErrorrs r+_sliced_mask_mod_errorrjs  `  r-i@c |jd |jd|jdd}|j fd}|}tt|D]}t j |d}|||}|S)Nc|j dztj}tj tjj d}tjtj}||j dk}tj ||}|jd|||f<|dddfjS)NrerrrrrV) new_zerosr"int32arangeint unsqueezewherer contiguous) kv_num_blocks kv_indices dense_mask row_indices col_range index_mask valid_indicesrnum_colsnum_rowss r+create_dense_onez+_ordered_to_dense..create_dense_ones))(HqL )T ll8599VLVV  LL6J !8!8!<<  J HE 2<1D1DR1H ; -.!YhY,'2244r-)rr)r)shaperrangeryr"r) num_blocks_in_row col_indices batch_dimsrcreate_dense_batched_outrrrs @@@r+_ordered_to_densers  $H  $H"(("-J  % %F5 , 3z? #P$zz*>OP 0+ >C Jr-cT|jtj}|jd}tj|ddd}|jtjtj |jtjtj fS)NrrdimT)r descendingstable) memory_format)tor"rrzargsortcontiguous_format)rrrs r+_dense_to_orderedrs|U[[1J"2.-- tDQKU[[8O8OPu{{%2I2IJ r-cPt||}t|jddS)Nrr)rr transpose)rrdenses r+_transpose_orderedrs' / =E U__R4 55r-c|ddddd|d|f}|ddddd|f}tj||k||}tj||dddddddfkdjtj}||fS)Nrr)r"rrzrr) num_blocksindices new_num_rows new_num_colss r+_adjust_num_blocks_and_indicesrs aM\M=L=89GAq-<-/0JZ,6 LQJ7Z1a %>>BGJJ5;;WJ w r-receZdZdZdZddZy)_ExtractedLeafzSentinel in _StrippedClosure.leaf_entries marking a position that is filled from the extracted pytree leaves list during reconstruction.rVcy)N_EXTRACTED_LEAFrVselfs r+__repr__z_ExtractedLeaf.__repr__s r-Nreturnstr)rQrRrSrT __slots__rrVr-r+rrsKI!r-rc0eZdZUdZded<ded<ded<y) _FunctionLeafzEntry in _StrippedClosure.leaf_entries for a recursively processed function. Stores enough information to reconstruct the function from the extracted leaves during unflattening.z%_StrippedClosure | Callable[..., Any]strippedr closure_specr n_extractedNrPrVr-r+rrs143r-rcbeZdZUdZded<ded<ded<ded<d ed <d ed <ded <ded<y)_StrippedClosureu(Data container holding the parts of a function needed for reconstruction. Created by _extract_closure_pytree when closure tensors are lifted into pytree leaves. Unlike a FunctionType with None-filled cells, this is not callable — it is pure data stored in the pytree context. ztypes.CodeTyperzdict[str, Any] globals_dictrnamequalnameztuple[Any, ...] | Nonerzdict[str, Any] | None kwdefaults extra_dictz*tuple[_ExtractedLeaf | _FunctionLeaf, ...] leaf_entriesNrPrVr-r+rrs9   IM$$%%=.s@++@s)rrrrrrrr)ro isfunctionr"r#r$_EMPTY_CLOSURE_SPECsetidr' __closure__tuple ValueErrorr_extract_closure_pytreeextendappendrryrrrk __globals__rQrSrl__kwdefaults____dict__dict) r_seenclosurecontentsclosure_leavesr extractedrleafchild_extracted child_specchild_strippedrs r+rrs6   b !U^^%@%@%B&** } "v&** IIbfnnG &**+@@@ $0#9 NL)+I9;L 1   d #:Qe; 7OZ   _ -   nj#o:NO    T "    0 1 [[^^ [[$$(* 4 $<( H  \8 33A +&**+sGGGct|ts|Sg}d}|jD]}}t|trRt |j ||||j z|j}|j|||j z }e|j|||dz }t||}td|D}tj|j|j|j|j |} |j"| _|j&r|j&| _|j*r%| j,j/|j*| S)zJRebuild a function from a _StrippedClosure and flattened extracted leaves.rrec3FK|]}tj|ywr)typesCellType)rrvs r+rtz*_reconstruct_closure_fn..Ys:AennQ':s!) isinstancerrr_reconstruct_closure_fnrrrrrrr  FunctionTyperrrrrrSrrrrupdate) rextracted_leavesr all_leavesidxentrychild_fnr new_cellsrestoreds r+rrCsI h 0 1?AJ C&&  e] +. sU->->'>?""H   h ' 5$$ $C   .s3 4 1HC j,7H:::I!!   H%--H"*"5"5  !4!45 Or-c>eZdZdZdZd d dZd dZd dZd dZddZ y)_MaskModWrapperuWraps a mask_mod or _StrippedClosure with value-based equality. BlockMask stores an arbitrary callable (mask_mod) in its pytree context. The default __eq__ for functions uses identity comparison, which is too strict when the same closure is recreated (e.g., defined inside forward()). When closure tensors have been extracted (by _extract_closure_pytree), fn is a _StrippedClosure (pure data, not callable). Equality compares the code objects + closure_spec — no tensor dispatch is triggered. When extraction is skipped (e.g., under Dynamo), fn is the original callable and equality compares code objects + closure contents (for plain functions) or delegates to __eq__ (for callable objects). rrNc ||_||_yrr)rrrs r+__init__z_MaskModWrapper.__init__}s(r-ctt|jtr td|j||||S)Nus_MaskModWrapper with _StrippedClosure is not callable — use _reconstruct_closure_fn to rebuild the function first)rrrr)rbhq_idxkv_idxs r+__call__z_MaskModWrapper.__call__s; dgg/ 0L wwq!UF++r-ct|tsy|j|jur|j|juryt|jtrbt|jtrH|jj |jj k(xr|j|jk(St j|jrLt j|jr-|jj|jjk(St|jts3t|jts|j|jk(Sy)NFT) rrrrrrrorrk)rothers r+__eq__z_MaskModWrapper.__eq__s %1 77ehh 4#4#48J8J#J dgg/ 0Z HH&6   -<%%););;    dgg &7+=+=ehh+G77##uxx'8'88 8$''#34Z HH&> 77ehh& &r-ct|jtrt|jjSt j |jrt|jjSt|jSr)rrrhashrrorrkrs r+__hash__z_MaskModWrapper.__hash__sY dgg/ 0 % %   dgg &(() )DGG}r-c"d|jdS)Nz_MaskModWrapper())rrs r+rz_MaskModWrapper.__repr__s!$''!,,r-r)rNone r rr!rr"rr#rrr)r&objectrrZrrr) rQrRrSrTrrr$r'r*rrVr-r+rrks' 'I),.-r-rceZdZUdZded<ded<ded<ded<ded <ded <ded <ded <ded <ded<ded<gdZgdZ d)dZedde dddf d*dZ e d+ d,dZ e d-dZ d.d/dZ e d0dZd1dZ d2dZd1dZd3dZd4dZd5d Zd6d!Z d7 d8d"Zd9d#Zed:d$Zed:d%Z d;d&Ze dr.a/ BlockMask is our format for representing a block-sparse attention mask. It is somewhat of a cross in-between BCSR and a non-sparse format. **Basics** A block-sparse mask means that instead of representing the sparsity of individual elements in the mask, a KV_BLOCK_SIZE x Q_BLOCK_SIZE block is considered sparse only if every element within that block is sparse. This aligns well with hardware, which generally expects to perform contiguous loads and computation. This format is primarily optimized for 1. simplicity, and 2. kernel efficiency. Notably, it is *not* optimized for size, as this mask is always reduced by a factor of KV_BLOCK_SIZE * Q_BLOCK_SIZE. If the size is a concern, the tensors can be reduced in size by increasing the block size. The essentials of our format are: num_blocks_in_row: Tensor[ROWS]: Describes the number of blocks present in each row. col_indices: Tensor[ROWS, MAX_BLOCKS_IN_COL]: `col_indices[i]` is the sequence of block positions for row i. The values of this row after `col_indices[i][num_blocks_in_row[i]]` are undefined. For example, to reconstruct the original tensor from this format: .. code-block:: python dense_mask = torch.zeros(ROWS, COLS) for row in range(ROWS): for block_idx in range(num_blocks_in_row[row]): dense_mask[row, col_indices[row, block_idx]] = 1 Notably, this format makes it easier to implement a reduction along the *rows* of the mask. **Details** The basics of our format require only kv_num_blocks and kv_indices. But, we have up to 8 tensors on this object. This represents 4 pairs: 1. (kv_num_blocks, kv_indices): Used for the forwards pass of attention, as we reduce along the KV dimension. 2. [OPTIONAL] (full_kv_num_blocks, full_kv_indices): This is optional and purely an optimization. As it turns out, applying masking to every block is quite expensive! If we specifically know which blocks are "full" and don't require masking at all, then we can skip applying mask_mod to these blocks. This requires the user to split out a separate mask_mod from the score_mod. For causal masks, this is about a 15% speedup. 3. [GENERATED] (q_num_blocks, q_indices): Required for the backwards pass, as computing dKV requires iterating along the mask along the Q dimension. These are autogenerated from 1. 4. [GENERATED] (full_q_num_blocks, full_q_indices): Same as above, but for the backwards pass. These are autogenerated from 2. tuple[int, int] seq_lengthsrrrrbfull_kv_num_blocksfull_kv_indices q_num_blocks q_indicesfull_q_num_blocksfull_q_indices BLOCK_SIZE_mask_mod_signaturemask_mod)rrr4r5r6r7r8r9)r3r:r<c ^|jdkr td| td| td|du|duk7r td|du| duk7r td||_||_||_||_||_||_||_ ||_ | |_ | |_ | |_ y)Nr)BlockMask must have at least 2 dimensionszkv_num_blocks must be providedzkv_indices must be providedGfull_kv_num_blocks and full_kv_indices must be both provided or omittedzEfull_q_num_blocks and full_q_indices must be both provided or omitted)rrr~r3rrr4r5r6r7r8r9r:r<) rr3rrr4r5r6r7r8r9r:r<s r+rzBlockMask.__init__s >> a JK K   !AB B   !>? ? $ &Ot,C D Y   %>T+A B W '*$"4.("!2,$  r-NTc |jdkr td|du|duk7r td|r4t||\} } || tdt||\} } nd\} } n d\} } d\} } t |t r||f}||nt }|.|jd|dz} |jd |d z}| |f}||||||| | | | || S) a Creates a BlockMask instance from key-value block information. Args: kv_num_blocks (Tensor): Number of kv_blocks in each Q_BLOCK_SIZE row tile. kv_indices (Tensor): Indices of key-value blocks in each Q_BLOCK_SIZE row tile. full_kv_num_blocks (Optional[Tensor]): Number of full kv_blocks in each Q_BLOCK_SIZE row tile. full_kv_indices (Optional[Tensor]): Indices of full key-value blocks in each Q_BLOCK_SIZE row tile. BLOCK_SIZE (Union[int, tuple[int, int]]): Size of KV_BLOCK_SIZE x Q_BLOCK_SIZE tiles. mask_mod (Optional[Callable]): Function to modify the mask. Returns: BlockMask: Instance with full Q information generated via _transposed_ordered Raises: RuntimeError: If kv_indices has < 2 dimensions. AssertionError: If only one of full_kv_* args is provided. rr>Nr? full_kv_indices must not be NoneNNrrrre) r3rrr4r5r6r7r8r9r:r<)rrr~rrrr6r)clsrrr4r5r:r<r3compute_q_blocksr6r7r8r9q_length kv_lengths r+from_kv_blockszBlockMask.from_kv_blocks/s6< >> a JK K $ &Ot,C D Y  &8 &S #L)!-"*()KLL4F&51!>5?1!>&0 #L)0: - ~ j# &$j1J'38  !''+jm;H"((,z!}>#))+ X  U r-c 6t|ts|fn|}g|tdtdtddd}|jjdd}tdt ||dD}|j|}|j |}|j6|j td|j|}|j|}nd}d}tj|||||jt|j|jduS)ae Returns a new BlockMask instance by getting the mask for the given index position. Args: index: Index to apply to all attributes. Example Usage: .. code-block:: python def causal_mask(b, h, q_idx, kv_idx): return q_idx >= kv_idx block_mask = create_block_mask( causal_mask, 4, 2, 512, 512, device="cuda" ) assert block_mask.kv_num_blocks.shape == (4, 2, 4) assert block_mask.kv_indices.shape == (4, 2, 4, 4) # Index on batch dimension new_block_mask = block_mask[0] assert new_block_mask.kv_num_blocks.shape == (2, 4) assert new_block_mask.kv_indices.shape == (2, 4, 4) # Index on batch and head dimension new_block_mask = block_mask[0, 1] assert new_block_mask.kv_num_blocks.shape == (4,) assert new_block_mask.kv_indices.shape == (4, 4) # slicing on batch and head dimension new_block_mask = block_mask[0:2, 1:2] assert new_block_mask.kv_num_blocks.shape == (2, 1, 4) assert new_block_mask.kv_indices.shape == (2, 1, 4, 4) # slicing on batch, head, and query dimension new_block_mask = block_mask[ 0:2, 1:2, torch.tensor([1], dtype=torch.int32) ] assert new_block_mask.kv_num_blocks.shape == (2, 1, 1) assert new_block_mask.kv_indices.shape == (2, 1, 1, 4) Nrfc3K|]L\}}t|tr3| |cxkrdkrnnt||z||zdznt||dzn|Nyw)rreN)rrslice)rrins r+rtz(BlockMask.__getitem__..s] 1!S!*+a ! U1q5!a%!) $q!a%  sAAT)strictrA)r:r<r3rD)rrr[rrziprr4r5r~r.rGr:rr3r7)rindexpaddedsizesnew_kv_num_blocksnew_kv_indicesnew_full_kv_num_blocksnew_full_kv_indicess r+ __getitem__zBlockMask.__getitem__s3X!+5% 8e@5@%+@uT{@E$K@!D""((!, FE$7    !..u5/  " " .##+$%GHH%)%<%.shape_or_nonesm177 5 5r-zBlockMask( kv_num_blocks=z, kv_indices=z, full_kv_num_blocks=z, full_kv_indices=z, q_num_blocks=z, q_indices=z, full_q_num_blocks=z, full_q_indices=z, BLOCK_SIZE=z , shape=z, sparsity=rQz%, mask_mod=rQrR)rkztorch.Tensor | None)rrrr4r5r6r7r8r9r:rSrwr<rQ)rrls r+rzBlockMask.__repr__s< 6!!%!3!3!9!9 :;"oo3345&&3D4K4K&L%MN##01E1E#F"GH -d.?.? @AB*4>>:;<%%243I3I%J$KL""/0C0C"D!EF"oo./ % MMOC016=dmmZ6XDMM22lm  _c^k^klm  r-c||jdzdz |jdz}||jdzdz |jdz}t|j|j||\}}|j=|j t dt|j|j ||\}}nd}d}|j|||||j|jS)NrrerA) r:rrrr4r5r~rGr<) r new_q_len new_kv_lenrrrcrdrerfs r+_adjustzBlockMask._adjust's!DOOA$66:tq?QQ "T__Q%77!;PQ@RR ,J   |- )>  " " .##+$%GHH/''$$  &#&* ""& ""   "  OO MM   r-c0|j}d}||S)zIReturns the number of elements (not accounting for sparsity) in the mask.cLtjtj|dSNre) functoolsreduceoperatormul)xss r+_prodzBlockMask.numel.._prodIs##HLL"a8 8r-rj)rrrys r+numelzBlockMask.numelEs  9U|r-c"|j}|jj}|j||jjz }|j |j dz|j dz}||z }dd|z zS)zEComputes the percentage of blocks that are sparse (i.e. not computed)rred)rzrrzr4itemr:)r total_sizecomputed_blocks computed_size dense_ratios r+rSzBlockMask.sparsityNsZZ\ ,,002  " " . t66::< .create_block_vis..summarize_section~s6$]]_11388: ? 1_ r-c||dz z|zSrsrV)ar s r+cdivz;BlockMask.to_string..create_block_vis..cdivsQU ))r-rerrr)rjoinreversedmaxr) batch_idx descriptorsvisrrrow_stepcol_steprccur_maskrcharrmax_colsmax_rowsrrs r+create_block_visz-BlockMask.to_string..create_block_viswsK   ) .))H[12T9C ! *1d8X67H1d8X67H1h1 q(H5$A)H(1#+C=1, Q\!11q8|3C!CDD4!8OC$t  Jr-z...z3To print out more, set BlockMask.to_string(limit=N)zNYou can also index (BlockMask[batch, head]) to choose a specific batch or headr) rrrr enumerate itertoolsproductrrr)r grid_sizelimitrr total_visr\rr block_visrrrrrs @@@@@r+rTzBlockMask.to_stringes]]_ *4*:*:'Xx i % H H "_HH!* Hh  D '   *=Qa= > (NCe|  '  !VW  d()4I   Y ' (yy## >s-C0 cpttjfd|jd}t |S)aMoves the BlockMask to the specified device. Args: device (torch.device or str): The target device to move the BlockMask to. Can be a torch.device object or a string (e.g., 'cpu', 'cuda:0'). Returns: BlockMask: A new BlockMask instance with all tensor components moved to the specified device. Note: This method does not modify the original BlockMask in-place. Instead, it returns a new BlockMask instance where individual tensor attributes may or may not be moved to the specified device, depending on their current device placement. c&|jSr)r)rkrs r+zBlockMask.to..sadd6lr-F)rJ)rr"rrKr.)rrmapped_attributess ` r+rz BlockMask.tos6"* LL " MM%M (  +,,r-c&|dk(r t|S|S)Nr<)rattrvalues r+_wrap_context_valuezBlockMask._wrap_context_values : "5) ) r-cv|dk(r3t|tstdt||jS|S)Nr<zExpected _MaskModWrapper, got )rrr~typerrs r+_unwrap_context_valuezBlockMask._unwrap_context_values9 : e_5$'Ed5k]%STT88O r-ctfdjD}tj\}||z}tfdjD}||fS)aFlatten BlockMask into a list of tensors and context. Closure tensors from mask_mod are extracted into the leaves via _extract_closure_pytree so they are visible to the tracing infrastructure (instead of being hidden in the pytree context). c36K|]}t|ywr)getattrrrrrs r+rtz%BlockMask._flatten..sKd+Ksc3xK|]1}|dk7rj|t|n t3ywr<N)rrrrrrrrrs r+rtz%BlockMask._flatten..sH z!  $ $T74+> ? <8 9 s7:)r _TENSOR_ATTRSrr<_CONTEXT_ATTRS)rtensorsrrcontextrrs` @@r+_flattenzBlockMask._flattensfK8J8JKK1H1W. h~-  ++    7""r-cxt|j}|d|}||d}i}t|j|D]T\}}|dk(r5t |t r%t |j||j||<@|j||||<V|jt|j||di|S)z3Unflatten leaves and context back into a BlockMask.Nr<rV) ryrr_rrrrrrrr) rCleavesr n_regularregular_leavesrkwargsrvals r+ _unflattenzBlockMask._unflattens))*  + +S//9 DID#z!jo&F6FFNC,<,< t  #88sCt  D  c#++^<=}V}r-ctfdjD}tj\}tdt |D}||z}tfdj D}||fS)a Flatten BlockMask with keys for better tracing. Closure tensors from mask_mod are extracted into the leaves via _extract_closure_pytree so they are visible to the tracing infrastructure (instead of being hidden in the pytree context). c3LK|]}t|t|fywr)rrrs r+rtz/BlockMask._flatten_with_keys..s' 8.s)" 4;AtZ)A3 ($ /" sc 3K|]G}|dk7r't|j|t|fnt|tfIywr)rrrrrs r+rtz/BlockMask._flatten_with_keys.. s] z! t77gdD>QR ST"OHl$KL M sA A)rrrr<rr)rrrclosure_with_keysrrrrs` @@r+_flatten_with_keyszBlockMask._flatten_with_keyss @D@R@R  2I1W. h!" ?H?X"  00  ++    7""r-)r3r2rrrrr4rbr5rbr6rbr7rbr8rbr9rbr:r2r<r;rr-)rrrrr4rbr5rbr:int | tuple[int, int]r<_mask_mod_signature | Noner3ztuple[int, int] | NonerDrZrr).)rJ Literal[True]rztuple[int, int, Tensor, Tensor, Tensor | None, Tensor | None, Tensor | None, Tensor | None, Tensor | None, Tensor | None, int, int, _mask_mod_signature])rJLiteral[False]rztuple[tuple[int, int], Tensor, Tensor, Tensor | None, Tensor | None, Tensor | None, Tensor | None, Tensor | None, Tensor | None, int | tuple[int, int], _mask_mod_signature])T)rJrZrtuple[Any, ...])rztuple[int, ...]r)r`z7int | slice | Tensor | tuple[int | slice | Tensor, ...]rr)rnrrorrrr0)rr)rr))rrv)rrrrrr)rztorch.device | strrr.)rrrrrr)rz.padding_needed_for_multiple&s$Q1A55r-rrrzQ (z%) must be divisible by Q_BLOCK_SIZE (r,zKV (z&) must be divisible by KV_BLOCK_SIZE (rerrfrurr)rr"rZr~rnn functionalpadrviewpermuterzrint8) mask Q_BLOCK_SIZE KV_BLOCK_SIZEseparate_full_blocksrBHQKVmask_block_sumfull_block_sum full_blockspartial_blockss r+_convert_mask_to_block_maskrs  zzUZZB4::,OPP T1 %D6 88   " " ' 2 F ' 2 E   D**KAq!R<1!9,q I   MQ2$<]O1 M   99 1a<r]/BM D << 1aAq DXX HN% 5$6 (1,.1PQ'***<!nn5::n6 {**'!+'***<t##r-cTtdDstddfd }|S)z9Returns a mask_mod that's the union of provided mask_modsc32K|]}t|ywrcallablerrargs r+rtzor_masks..S2x}2)All inputs should be callable mask_mods: ct|jdtj}D]}||||||z}|SNrVr)rr"rZr r!r"r#resultr mask_modss r+or_maskzor_masks..or_maskVsBRuzz2 8Dd1a77F 8 r-r.allr)rrs` r+r4r4Qs0 2 2 2FykRSS Nr-cTtdDstddfd }|S)z@Returns a mask_mod that's the intersection of provided mask_modsc32K|]}t|ywrrrs r+rtzand_masks..arrrct|jdtj}D]}||||||z}|Sr)rr"rZrs r+and_maskzand_masks..and_maskdsBBejj1 8Dd1a77F 8 r-r.r)rrs` r+r5r5_s0 2 2 2FykRSS Or-c|jdk7rtd|j|j\}}}}|j||g|j}|j ddddddj ||||z||z}|S)Nrvz block_mask.dim() must be 4, got rrfrrure)rr~rexpandrreshape) block_maskrrrrrrs r+_convert_block_mask_to_maskr ms ~~1? @P?QRSS""KAq!R"""<RAQAQRJ##Aq!Q15== 1a,] 2J r-c |\}}t|}| t|}nd}tj|d|d|d|d||f||S)NrBrre)r:r<r3)rr.rG) r r<r3rrrr partial_bmfull_bms r+$_create_sparse_block_from_block_maskr|so#-NK">2J7H7U  # #1 1    -0 $ r-c |"tjjxsd}|d}|d}tjd||}tjd||}tjd||}tjd||} t |} ddlm} | 5| tjk(rh|} t| d} | tj||||||||| } tjtj| d d }|cdddS| tjk(r%|}t|d }||||| }|cdddSt#1swYyxYw) aThis function creates a mask tensor from a mod_fn function. Args: mod_fn (Union[_score_mod_signature, _mask_mod_signature]): Function to modify attention scores. B (int): Batch size. H (int): Number of query heads. Q_LEN (int): Sequence length of query. KV_LEN (int): Sequence length of key/value. device (str): Device to run the mask creation on. Returns: mask (Tensor): A mask tensor with shape (B, H, M, N). Ncpurer)r)TransformGetItemToIndex)r)rFTrV)r" acceleratorcurrent_acceleratorrr,torch._dynamo._trace_wrapped_higher_order_oprrdrgrzerosrisneginfrhr~)mod_fnrrQ_LENKV_LENrr r!mr]mod_typer score_modrrr<s r+r3r3sV*~""668AEy y  Q&)A Q&)A Qf-A Qv.AV$HT " ! (22 2I' $?IEKK1eVFKQPQSTVWXC;;u~~c2E4@D ! !*33 3H&x;HAq!Q'D ! !!  ! !s%A2E!.EEE(c |"tjjxsd}t|}|tj k7rt d||d}|d}t|tr|} |} n|\} } |rAtjdtdtjt|||||||St||||||} t| | | d\} } t!| | f|||f| | }|S) aThis function creates a block mask tuple from a mask_mod function. Args: mask_mod (Callable): mask_mod function. This is a callable that defines the masking pattern for the attention mechanism. It takes four arguments: b (batch size), h (number of heads), q_idx (query index), and kv_idx (key/value index). It should return a boolean tensor indicating which attention connections are allowed (True) or masked out (False). B (int): Batch size. H (int): Number of query heads. Q_LEN (int): Sequence length of query. KV_LEN (int): Sequence length of key/value. device (str): Device to run the mask creation on. BLOCK_SIZE (int or tuple[int, int]): Block size for the block mask. If a single int is provided it is used for both query and key/value. Returns: BlockMask: A BlockMask object that contains the block mask information. Example Usage: .. code-block:: python def causal_mask(b, h, q_idx, kv_idx): return q_idx >= kv_idx block_mask = create_block_mask(causal_mask, 1, 1, 8192, 8192, device="cuda") query = torch.randn(1, 1, 8192, 64, device="cuda", dtype=torch.float16) key = torch.randn(1, 1, 8192, 64, device="cuda", dtype=torch.float16) value = torch.randn(1, 1, 8192, 64, device="cuda", dtype=torch.float16) output = flex_attention(query, key, value, block_mask=block_mask) rz4create-block_mask requires a mask_mod function! Got rea_compile flag on create_block_mask was originally added to work around a torch.compile limitation. That limitation has since been addressed. So, to compile create_block_mask, we suggest doing torch.compile(create_block_mask). This still works for now, but will be removed in the future.rr T)rrr)r"rrrrdrhr~rrr%r&DeprecationWarningcompiler2r3rr)r<rrrrrr:_compilerrr mask_tensorpartial_block_maskfull_block_maskr s r+r2r2s*R~""668AEX&H$---B8* M   y y *c"! " &0# m  m  0u}}./ aE66:  h1eVVDK*E!#! +' 6 _-  J r-c|j}tjtjgdtj |tj gdtj |tdS)zDefault block mask for flex attention. If users don't specify any block sparse mask info, we create this empty block sparse mask. Which creates a BlockMask with 1 block that is the full length of the query and key tensors. )rererer)rererere)rere)rrr:r3)rr.rGr"onesrr_LARGE_SPARSE_BLOCK_SIZE)querykeyrs r+_create_empty_block_maskr,sS \\F  # #jj%++fM;;|5;;vN+ $ r-cXtt|in t|}d|vr|jddr t dd|vr4t j t}|d|vrtd|dd||jdd|jdd|jd d|jd d|jd d |jjd k(xs4|jjd k(xs|jjd k(}|}d} ||j}|j} d|vr tdd |d<|stj |d<|rd|d<d|vr td|ddk(r | r t#d| |d<|r | r t#d|S)NrOrJFzBACKEND cannot be combined with legacy FORCE_USE_FLEX_ATTENTION. BACKEND supersedes the legacy knob; please drop FORCE_USE_FLEX_ATTENTION and only specify the desired BACKEND.zInvalid BACKEND value 'z'. Must be one of r7rFrGrHrITrr[z.OUTPUT_LOGSUMEXP must not be in kernel_optionsr\z(OUTPUT_MAX must not be in kernel_optionsr9zsReturning max scores is not supported with BACKEND='FLASH'. Use return_aux=AuxRequest(lse=True) or omit max_scores.z-Returning max scores is not supported on CPU.)rrYrgetrtypingget_argsr;r setdefaultrrr^r_r~r"is_grad_enabledNotImplementedError) r*r+r return_lsekernel_options return_auxvalid_backendsany_inputs_on_cpu_device output_lse output_maxs r+_apply_kernel_optionsr;-s#$$~*>N N"~'9'9"E( 4  N"2 ) $N :).*C)DE""0!13  i0mU34e<5u=j$/  U" & ::??e # & <<   %JJ^^ ** ^+MNN)-N%& .3-B-B-D)* #27N- .~%GHHi G+ ! F  $.N< J""QRR r-c|jd|jdk7r0td|jdd|jddy)NrzJExpect query and key/value to have the same embedding dimension but got E=z and E=.)rr)r*r+rs r+_validate_embed_dimr>sU zz"~"%B( ~Q @  &r-c|jjdk(r/|js|js |jr tdhd}|jj|vr#t d|jjdy)zTODO: Remove once non cuda/cpu devices support is added We only need to check query since we have already that q,k,v are on the same device rzrFlexAttention does not support backward on CPU. Please set the input requires_grad to False or use another device.>rhpuxpucudazTFlexAttention is only supported on CUDA, CPU or HPU devices. Found input tensors on z device.N)rr requires_gradr3r)r*r+rsupported_devicess r+_validate_devicerEs ||E! s00E4G4G! A  6 || 11 &&+ll&7&7%8 B  2r-c&d d}d d}tjtjf}|j}||vxrdtjj duxrFtj j ddk\xr"tj j ddk}|s|||fS||s|j}||s|j}||s0|jddjjdd}|||fS) a Enforce memory layouts for query, key, and value tensors. For non-FP8 dtypes, no action is taken. For FP8 dtypes, we enforce the following memory layouts: - Query tensor must be in row-major memory layout, as it will be the left-operand in the FP8 GEMM `q @ k.T`. - Key tensor must be in row-major memory layout, as it will be transposed when used as the right-operand in the FP8 GEMM `q @ k.T`, meaning it will correctly be in column-major memory layout for the GEMM. - Value tensor must be in column-major memory layout, as it will be the right-operand in the FP8 GEMM `softmax_scores @ v`. Returns the query, key, and value tensors with the enforced memory layouts. c.|jddk(S)Nrrestridetensors r+ is_row_majorz*_enforce_mem_layouts..is_row_major}}r"a''r-c.|jddk(S)NrrerHrJs r+ is_col_majorz*_enforce_mem_layouts..is_col_majorrMr-NrB) ) rrr)rKrrrZ) r" float8_e4m3fn float8_e5m2rversionrBget_device_capabilityrr)r*r+rrLrO fp8_dtypesgemm_precisionshould_enforce_mem_layouts r+_enforce_mem_layoutsrZs"((  J[[N *$ ? MM  d * ? JJ , ,V 4 > ? JJ , ,V 4w >  %c5      "  nn  B'224>>r2F #u r-.)r6c yrrV r*r+rrr scale enable_gqar4r5r6s r+rrsr-zcreturn_lse is deprecated and will be removed in v2.10. Use return_aux=AuxRequest(lse=True) instead.r*c yrrVr\s r+rrs$ r-c yrrVr\s r+rrs #r-c yrrVr\s r+rr s r-c  r t|||dt|||t|||t|||\}}}|j dk7s&|j dk7s|j dk7r t d|sS|j d|j dk7r0td|j dd|j dd|r<|j d } |j d } | | zd k7rtd | d| d |j d |j d k7r|0td|j d d|j d d |jj d |j d k7rLtd|j d d|j d d|jj d d |t}| t||}t|dd tur td|jd tk(r|jd tk(rnz|j d} |j d} |j d| kDs|j d| kDr=td|j d|j dd|j dd|j d| kr|j d| ks(|j d| krQ|j d| kr=td|j d|j dd|j dd|j d| k7r!t#d|j dd| d|j d| k7r!t#d|j dd | d|'d!t%j&|j dz }|j(|jj(k7r0td"|j(d#|jj(d |r | td$|r| t+d%d&t,'t/|||||| } d0d(}t0j2j5rz|||fD]B}t0j6j9|dt0j6j9|dDt;|||||j=||\}}}||||| |)St>s t+d*d+,t0j6jAs td-d.}tC5}t>r|}nt1jD||d/}||||||j=||\}}}d d d || |)S#1swYxYw)1aThis function implements scaled dot product attention with an arbitrary attention score modification function described in the `Flex Attention `_ paper. See also the `blog post `_. This function computes the scaled dot product attention between query, key, and value tensors with a user-defined attention score modification function. The attention score modification function will be applied after the attention scores have been calculated between the query and key tensors. The attention scores are calculated as follows: The ``score_mod`` function should have the following signature: .. code-block:: python def score_mod( score: Tensor, batch: Tensor, head: Tensor, q_idx: Tensor, k_idx: Tensor ) -> Tensor: Where: - ``score``: A scalar tensor representing the attention score, with the same data type and device as the query, key, and value tensors. - ``batch``, ``head``, ``q_idx``, ``k_idx``: Scalar tensors indicating the batch index, query head index, query index, and key/value index, respectively. These should have the ``torch.int`` data type and be located on the same device as the score tensor. Args: query (Tensor): Query tensor; shape :math:`(B, Hq, L, E)`. For FP8 dtypes, should be in row-major memory layout for optimal performance. key (Tensor): Key tensor; shape :math:`(B, Hkv, S, E)`. For FP8 dtypes, should be in row-major memory layout for optimal performance. value (Tensor): Value tensor; shape :math:`(B, Hkv, S, Ev)`. For FP8 dtypes, should be in column-major memory layout for optimal performance. score_mod (Optional[Callable]): Function to modify attention scores. By default no score_mod is applied. block_mask (Optional[BlockMask]): BlockMask object that controls the blocksparsity pattern of the attention. scale (Optional[float]): Scaling factor applied prior to softmax. If none, the default value is set to :math:`\frac{1}{\sqrt{E}}`. enable_gqa (bool): If set to True, enables Grouped Query Attention (GQA) and broadcasts key/value heads to query heads. return_lse (bool): Whether to return the logsumexp of the attention scores. Default is False. **Deprecated**: Use ``return_aux=AuxRequest(lse=True)`` instead. kernel_options (Optional[FlexKernelOptions]): Options to control the behavior of the underlying Triton kernels. See :class:`FlexKernelOptions` for available options and usage examples. return_aux (Optional[AuxRequest]): Specifies which auxiliary outputs to compute and return. If None, only the attention output is returned. Use ``AuxRequest(lse=True, max_scores=True)`` to request both auxiliary outputs. Returns: output (Tensor): Attention output; shape :math:`(B, Hq, L, Ev)`. When ``return_aux`` is not None: aux (AuxOutput): Auxiliary outputs with requested fields populated. When ``return_aux`` is None (deprecated paths): lse (Tensor): Log-sum-exp of attention scores; shape :math:`(B, Hq, L)`. Only returned if ``return_lse=True``. Shape legend: - :math:`N: \text{Batch size} ... : \text{Any number of other batch dimensions (optional)}` - :math:`S: \text{Source sequence length}` - :math:`L: \text{Target sequence length}` - :math:`E: \text{Embedding dimension of the query and key}` - :math:`Ev: \text{Embedding dimension of the value}` .. warning:: `torch.nn.attention.flex_attention` is a prototype feature in PyTorch. Please look forward to a more stable implementation in a future version of PyTorch. Read more about feature classification at: https://pytorch.org/blog/pytorch-feature-classification-changes/#prototype T) allow_lowp_kvrvz-NYI: query, key, and value must be 4D tensorszGExpect query and key/value to have the same number of heads but got Hq=z and Hkv=z&. Try setting enable_gqa=True for GQA.rerzMExpect number of query heads to be a multiple of kv heads for GQA but got Hq=r=NzlExpect query and key/value to have the same batch size, or non-none block_mask, but got block_mask=None, Bq=z , and Bkv=zxExpect query and key/value to have the same batch size, or block_mask and query to have the same batch size, but got Bq=z, Bkv=z, B_block_mask=r<z+Cannot use mask_mod from a sliced BlockMaskrrz,block_mask was created for block_mask.shape=z but got q_len=z and kv_len=zz. As the block mask was created for a smaller length than you're using it for, you likely need to create a new block mask.ad. As the block mask was created for a larger length than you're using it for, you can either 1. create a new block mask with the correct length, or 2. 'adjust' the existing block mask to the correct length by calling block_mask._adjust(q_len, kv_len). This essentially 'crops' the block mask to the upper left corner, which does not work for all mask_mods!zquery.size(-2) (z) != block_mask_q_len (r,zkey.size(-2) (z) != block_mask_kv_len (g?z=Expect q/k/v and block_mask to be on the same device but got z and z|Cannot specify both return_lse and return_aux. return_lse is deprecated, please use return_aux=AuxRequest(lse=True) instead.deprecated_return_lsezjreturn_lse is deprecated and will be removed in v2.10. Please use return_aux=AuxRequest(lse=True) instead.r_ctjd}|xs|duxr |j}|duxr |j}|r|j dkDr||znd}|r|j dkDr||znd}||t ||fS|r||fS|S)zFNormalize stats and build return value (aux-aware, legacy-compatible).g@Nr)r^r_)mathlogr^r_rzr/) rr^r_r6r4ln2 return_max lse_scaled max_scaleds r+_finalize_outputsz)flex_attention.._finalize_outputsshhsmL:T#9#Ljnn t+E 0E0E #-#))+/S3Y !+ 0@0@0BQ0FJ T   ! %   ? " r-)r6r4flex_attention_performanceaflex_attention called without torch.compile() - this will use an unfused implementation that materializes the full scores matrix instead of generating a fused kernel. SOLUTION: Use torch.compile(flex_attention)(...) If you want to debug your score_mod/mask_mod, you can set: torch.nn.attention.flex_attention._FLEX_ATTENTION_DISABLE_COMPILE_DEBUG = True This will allow you to use print statements or breakpoints. Note: This doesn't work with the backwards pass and may produce incorrect results.)r(r)z&flex_attention requires dynamo supportct|i|Sr)flex_attention_hop)argsrs r+_flex_attention_hop_wrapperz3flex_attention.._flex_attention_hop_wrappers!42622r-)backend fullgraph)r6AuxRequest | Noner4rZ)#rr>rErZrr3rrrrr,rrrr:r)rr~rhsqrtrr, FutureWarningr;r"r#is_dynamo_compiling_dynamo mark_staticrqrK%_FLEX_ATTENTION_DISABLE_COMPILE_DEBUGis_dynamo_supportedrr")r*r+rrr r]r^r4r5r6HqHkvblock_mask_q_lenblock_mask_kv_lenrnrkrr^r_rsrtflex_fns r+rrsX^U$?sE*UC',UC?E3 yy{a3779>UYY[A-=!"QRR EJJrNchhrl:**R.)388B<.A3 4   ZZ]hhqk 8q= T3%q2  zz!} #  //4zz!}oZQR }TUW   # # ( ( +uzz!} <#jjm_F388A;-zOgOgOlOlmnOoNppqs   -eS9 z:t,0FFHII a $<<  ! !! $(@ @ %++B/&,,R0 ::b>, , ?P0P>z?O?O>PP_`e`j`jkm`n_oo{|}E}EFH}I|JJKK  JJrN- -#((2,BS2Sjjn 00SXXb\DU5U>z?O?O>PP_`e`j`jkm`n_oo{|}E}EFH}I|JJuu  ::b>- - "5::b>"22IJZI[[\]  88B<, ,  ".FGXFYYZ[  }dii 2// ||z//666||nE**B*B*I*I)J! M  j, \    * # B"  +    N &   : ~~))+e$ -A MM % %a , MM % %a , - 2        !   S*! jZJ   13a  == , , .CDD3   G 01Gmm+WG '        !   S* "  S*  #  s AV--V6)r(rr)rr*z type[Warning]rr-)rz?_score_mod_signature | _mask_mod_signature | Callable[..., Any]rrd)rVrF) rCallable[..., _R]rtuple[int | None, ...]rrrzint | list[int | None]rrZrr) rrrrrrrrrrrr) rrrrrrrrrr) rrrrrrrrrr)rrrrrr)rrrtuple[Tensor, Tensor])rrrrrr) rrrrrrrrrrr)rzset[int] | NonerzUtuple[tuple[BaseArgumentTypes, ...], TreeSpec, _StrippedClosure | Callable[..., Any]])rkrrrrr)rkrrrrr) rrrrrrrrZrtuple[Tensor, Tensor | None])rr;rr;)r rrrrrrr) r rr<rr3r2rrrrrr.)rz*_score_mod_signature | _mask_mod_signaturer int | NonerrrrrrrDeviceLikeType | Nonerr)r<r;rrrrrrrrrrr:rrr.)r*rr+rrr.)r*rr+rrrr4rZr5FlexKernelOptions | Noner6rvrrY)r*rr+rrrrr-)r*rr+rrrrz/tuple[torch.Tensor, torch.Tensor, torch.Tensor])......)r*rr+rrrr_score_mod_signature | Noner BlockMask | Noner] float | Noner^rZr4rr5rr6r-rr)r*rr+rrrrrr rr]rr^rZr4rr5rr6r-rr)r*rr+rrrrrr rr]rr^rZr4rZr5rr6r0rztuple[Tensor, AuxOutput])r*rr+rrrrrr rr]rr^rZr4rr5rr6r0rr)NNNFFN)r*rr+rrrrrr rr]rr^rZr4rZr5rr6rvrz9Tensor | tuple[Tensor, Tensor] | tuple[Tensor, AuxOutput])crT __future__rrtrorrhrvr r/r%collections.abcrenumrrrrr r r r typing_extensionsr rrrrr"r&torch._higher_order_ops.flex_attentionrrqtorch._higher_order_ops.utilsrtorch.nn.attention._utilsrtorch.utils._pytreerrrrr TYPE_CHECKINGtorch._prims_commonr torch.fx.noderr|rrrU UserWarningr,__all___score_mod_signaturer;r;r<r1rYr0r/rdrrrr6rrr)rrrrrrrrrrrrr.rrrr4r5r rr3r2r,r;r>rErZrxrVr-r+rsHP"  $OOOMM W?: 2/&).%E!>I((!(-:( (   H& PQ?GHHI)I T]C CL"35 % %   )G ) )N&('( ++ "+ #+% +  +  +\        J J JJ J  J     6!"<66,266        #2&q)!!!"F%%=v((=,"&K4K4K4\%P=-=-@d #d #N 5 33!& 2$ 2$2$2$ 2$ " 2$j   42     &33 ,(!     @%) 0! 60!0!0!  0!  0! " 0! 0!r%)(B R!RRR  R  R " R&RRj *%) Q Q Q Q Q - Q " Q!Qh  $8 88'-848v .1#&!$/2        +  !     -        3 .1#& #/2           +  !        -         .1#&/2 #  #  #  #+ # ! #  # # #- # # #  # .1#& #/2       +  !     -      &.2#'/3R%)R R R R+ R ! R  RRR-R"R?Rr-