[sssshhhhhh]

Currently always set to true but if someone wants to help test we can try enabling fa2 encoder self attention. Possible speedup from my rocm branch:

whisper encoder fa
0.07418210001196712s
 41.34%  58.30%  41.34% Gemm                    30.71ms
 16.96%  16.96%  58.30% BiasAdd                 12.60ms
 10.79%  10.79%  69.09% FlashAttention          8.01ms
  8.36%   8.36%  77.46% LayerNorm               6.21ms
  7.69%  54.03%  85.14% FlashMultiHeadAttention 5.71ms
  6.77%   6.77%  91.91% Add                     5.03ms
  3.56%   3.56%  95.48% Split                   2.65ms
  2.71%  97.69%  98.19% TransformerEncoderLayer 2.01ms
  0.67%   1.15%  98.86% Conv1D                  0.50ms
  0.23%  99.55%  99.09% WhisperEncoder          0.17ms
  0.21%   0.21%  99.31% GELU                    0.16ms
  0.13%   0.13%  99.44% Transpose               0.10ms
  0.12%  57.94%  99.55% Dense                   0.09ms
  0.11%  99.66%  99.66% WhisperReplica::encode  0.08ms

whisper encoder naive
0.15253070002654567s
 37.34%  37.34%  37.34% SoftMax                 56.99ms
 21.03%  29.76%  58.36% Gemm                    32.09ms
 13.20%  13.20%  71.56% MatMul                  20.15ms
  8.73%   8.73%  80.29% BiasAdd                 13.32ms
  4.43%   4.43%  84.72% LayerNorm               6.76ms
  3.96%  76.65%  88.68% MultiHeadAttention      6.04ms
  3.74%   3.74%  92.42% Transpose               5.71ms
  3.34%   3.34%  95.76% Add                     5.09ms
  1.82%   1.82%  97.58% Split                   2.78ms
  1.34%  98.77%  98.92% TransformerEncoderLayer 2.05ms
  0.36%   0.62%  99.28% Conv1D                  0.55ms
  0.19%  50.73%  99.47% dot_product_attention   0.29ms
  0.12%  99.74%  99.59% WhisperEncoder          0.18ms
  0.09%   0.09%  99.69% GELU                    0.14ms
  0.05%  99.79%  99.74% WhisperReplica::encode  0.08ms
  0.05%  29.55%  99.79% Dense                   0.08ms

[jordimas]

Thanks, makes sense