Antonio Sánchez
8588d8c74b
Correct pnegate for floating-point zero.
2022-11-15 18:07:23 +00:00
Charles Schlosser
82b152dbe7
Add signbit function
2022-11-04 00:31:20 +00:00
Rasmus Munk Larsen
c475228b28
Vectorize atan() for double.
2022-10-01 01:49:30 +00:00
Rasmus Munk Larsen
7b2901e2aa
Add vectorized integer division for int32 with AVX512, AVX or SSE.
2022-09-21 00:27:23 +00:00
Rasmus Munk Larsen
f913a40678
Revert "Add AVX int32_t pdiv"
...
This reverts commit ea84e7ad63
2022-09-16 22:48:08 +00:00
Charles Schlosser
ea84e7ad63
Add AVX int32_t pdiv
2022-09-16 17:06:29 +00:00
Rasmus Munk Larsen
bd393e15c3
Vectorize acos, asin, and atan for float.
2022-08-29 19:49:33 +00:00
Charles Schlosser
e5af9f87f2
Vectorize pow for integer base / exponent types
2022-08-29 19:23:54 +00:00
Rasmus Munk Larsen
7064ed1345
Specialize psign<Packet8i> for AVX2, don't vectorize psign<bool>.
2022-08-26 17:02:37 +00:00
Rasmus Munk Larsen
1a09defce7
Protect new pblend implementation with EIGEN_VECTORIZE_AVX2
2022-08-22 18:28:03 +00:00
Matthew Sterrett
7a3b667c43
Add support for AVX512-FP16 for vectorizing half precision math
2022-08-17 18:15:21 +00:00
Ilya Tokar
e618c4a5e9
Improve pblend AVX implementation
2022-07-29 18:45:33 +00:00
Shi, Brian
fc1d888415
Remove AVX512VL dependency in trsm
2022-04-14 12:44:24 -07:00
Antonio Sánchez
07db964bde
Restrict new AVX512 trsm to AVX512VL, rename files for consistency.
2022-04-14 16:58:32 +00:00
b-shi
518fc321cb
AVX512 Optimizations for Triangular Solve
2022-03-16 18:04:50 +00:00
Sean McBride
f1b9692d63
Removed EIGEN_UNUSED decorations from many functions that are in fact used
2022-03-03 20:19:33 +00:00
Antonio Sánchez
9c07e201ff
Modified sqrt/rsqrt for denormal handling.
2022-03-02 17:20:47 +00:00
Rasmus Munk Larsen
8b875dbef1
Changes to fast SQRT/RSQRT
2022-02-23 17:32:21 +00:00
Erik Schultheis
7197b577fb
Remove unused macros in AVX packetmath.
...
The following macros are removed:
* EIGEN_DECLARE_CONST_Packet8f
* EIGEN_DECLARE_CONST_Packet4d
* EIGEN_DECLARE_CONST_Packet8f_FROM_INT
* EIGEN_DECLARE_CONST_Packet8i
2022-02-14 10:34:23 +00:00
Rasmus Munk Larsen
979fdd58a4
Add generic fast psqrt and prsqrt impls and make them correct for 0, +Inf, NaN, and negative arguments.
2022-02-05 00:20:13 +00:00
Rasmus Munk Larsen
51311ec651
Remove inline assembly for FMA (AVX) and add remaining extensions as packet ops: pmsub, pnmadd, and pnmsub.
2022-01-26 04:25:41 +00:00
Rasmus Munk Larsen
ea2c02060c
Add reciprocal packet op and fast specializations for float with SSE, AVX, and AVX512.
2022-01-21 23:49:18 +00:00
Ilya Tokar
a0fc640c18
Add support for packets of int64 on x86
2022-01-21 19:55:23 +00:00
Kolja Brix
8d81a2339c
Reduce usage of reserved names
2022-01-10 20:53:29 +00:00
Rasmus Munk Larsen
96dc37a03b
Some fixes/cleanups for numeric_limits & fix for related bug in psqrt
2022-01-07 01:10:17 +00:00
Ilya Tokar
e1cb6369b0
Add AVX vector path to float2half/half2float
...
Makes e. g. matrix multiplication 2x faster:
name old cpu/op new cpu/op delta
BM_convers 181ms ± 1% 62ms ± 9% -65.82% (p=0.016 n=4+5)
Tested on all possible input values (not adding tests, since they
take a long time).
2021-10-28 13:59:01 -04:00
Antonio Sanchez
3c724c44cf
Fix strict aliasing bug causing product_small failure.
...
Packet loading is skipped due to aliasing violation, leading to nullopt matrix
multiplication.
Fixes #2327 .
2021-09-17 21:09:34 +00:00
Rasmus Munk Larsen
da027fa20a
Remove unused variable.
2021-09-16 20:02:42 +00:00
Rasmus Munk Larsen
d7d0bf832d
Issue an error in case of direct inclusion of internal headers.
2021-09-10 19:12:26 +00:00
Antonio Sanchez
7792b1e909
Fix AVX2 PacketMath.h.
...
There were a couple typos ps -> epi32, and an unaligned load issue.
2021-09-03 19:47:57 +00:00
Antonio Sanchez
def145547f
Add missing packet types in pset1 call.
...
Oops, introduced this when "fixing" integer packets.
2021-09-02 16:21:07 -07:00
Antonio Sanchez
3d4ba855e0
Fix AVX integer packet issues.
...
Most are instances of AVX2 functions not protected by
`EIGEN_VECTORIZE_AVX2`. There was also a missing semi-colon
for AVX512.
2021-09-01 14:14:43 -07:00
Jakub Lichman
dc5b1f7d75
AVX512 and AVX2 support for Packet16i and Packet8i added
2021-08-25 19:38:23 +00:00
Gauri Deshpande
e6a5a594a7
remove denormal flushing in fp32tobf16 for avx & avx512
2021-08-09 22:15:21 +00:00
Rasmus Munk Larsen
9312a5bf5c
Implement a generic vectorized version of Smith's algorithms for complex division.
2021-07-01 23:31:12 +00:00
Rasmus Munk Larsen
52a5f98212
Get rid of code duplication for conj_helper. For packets where LhsType=RhsType a single generic implementation suffices. For scalars, the generic implementation of pconj automatically forwards to numext::conj, so much of the existing specialization can be avoided. For mixed types we still need specializations.
2021-06-24 15:47:48 -07:00
Antonio Sanchez
0845df7f77
Fix uninitialized warning on AVX.
2021-02-17 13:13:39 -08:00
Antonio Sanchez
7ff0b7a980
Updated pfrexp implementation.
...
The original implementation fails for 0, denormals, inf, and NaN.
See #2150
2021-02-17 02:23:24 +00:00
Antonio Sanchez
4cb563a01e
Fix ldexp implementations.
...
The previous implementations produced garbage values if the exponent did
not fit within the exponent bits. See #2131 for a complete discussion,
and !375 for other possible implementations.
Here we implement the 4-factor version. See `pldexp_impl` in
`GenericPacketMathFunctions.h` for a full description.
The SSE `pcmp*` methods were moved down since `pcmp_le<Packet4i>`
requires `por`.
Left as a "TODO" is to delegate to a faster version if we know the
exponent does fit within the exponent bits.
Fixes #2131 .
2021-02-10 22:45:41 +00:00
Antonio Sanchez
b2126fd6b5
Fix pfrexp/pldexp for half.
...
The recent addition of vectorized pow (!330 ) relies on `pfrexp` and
`pldexp`. This was missing for `Eigen::half` and `Eigen::bfloat16`.
Adding tests for these packet ops also exposed an issue with handling
negative values in `pfrexp`, returning an incorrect exponent.
Added the missing implementations, corrected the exponent in `pfrexp1`,
and added `packetmath` tests.
2021-01-21 19:32:28 +00:00
Rasmus Munk Larsen
05754100fe
* Add iterative psqrt<double> for AVX and SSE when FMA is available. This provides a ~10% speedup.
...
* Write iterative sqrt explicitly in terms of pmadd. This gives up to 7% speedup for psqrt<float> with AVX & SSE with FMA.
* Remove iterative psqrt<double> for NEON, because the initial rsqrt apprimation is not accurate enough for convergence in 2 Newton-Raphson steps and with 3 steps, just calling the builtin sqrt insn is faster.
The following benchmarks were compiled with clang "-O2 -fast-math -mfma" and with and without -mavx.
AVX+FMA (float)
name old cpu/op new cpu/op delta
BM_eigen_sqrt_float/1 1.08ns ± 0% 1.09ns ± 1% ~
BM_eigen_sqrt_float/8 2.07ns ± 0% 2.08ns ± 1% ~
BM_eigen_sqrt_float/64 12.4ns ± 0% 12.4ns ± 1% ~
BM_eigen_sqrt_float/512 95.7ns ± 0% 95.5ns ± 0% ~
BM_eigen_sqrt_float/4k 776ns ± 0% 763ns ± 0% -1.67%
BM_eigen_sqrt_float/32k 6.57µs ± 1% 6.13µs ± 0% -6.69%
BM_eigen_sqrt_float/256k 83.7µs ± 3% 83.3µs ± 2% ~
BM_eigen_sqrt_float/1M 335µs ± 2% 332µs ± 2% ~
SSE+FMA (float)
name old cpu/op new cpu/op delta
BM_eigen_sqrt_float/1 1.08ns ± 0% 1.09ns ± 0% ~
BM_eigen_sqrt_float/8 2.07ns ± 0% 2.06ns ± 0% ~
BM_eigen_sqrt_float/64 12.4ns ± 0% 12.4ns ± 1% ~
BM_eigen_sqrt_float/512 95.7ns ± 0% 96.3ns ± 4% ~
BM_eigen_sqrt_float/4k 774ns ± 0% 763ns ± 0% -1.50%
BM_eigen_sqrt_float/32k 6.58µs ± 2% 6.11µs ± 0% -7.06%
BM_eigen_sqrt_float/256k 82.7µs ± 1% 82.6µs ± 1% ~
BM_eigen_sqrt_float/1M 330µs ± 1% 329µs ± 2% ~
SSE+FMA (double)
BM_eigen_sqrt_double/1 1.63ns ± 0% 1.63ns ± 0% ~
BM_eigen_sqrt_double/8 6.51ns ± 0% 6.08ns ± 0% -6.68%
BM_eigen_sqrt_double/64 52.1ns ± 0% 46.5ns ± 1% -10.65%
BM_eigen_sqrt_double/512 417ns ± 0% 374ns ± 1% -10.29%
BM_eigen_sqrt_double/4k 3.33µs ± 0% 2.97µs ± 1% -11.00%
BM_eigen_sqrt_double/32k 26.7µs ± 0% 23.7µs ± 0% -11.07%
BM_eigen_sqrt_double/256k 213µs ± 0% 206µs ± 1% -3.31%
BM_eigen_sqrt_double/1M 862µs ± 0% 870µs ± 2% +0.96%
AVX+FMA (double)
name old cpu/op new cpu/op delta
BM_eigen_sqrt_double/1 1.63ns ± 0% 1.63ns ± 0% ~
BM_eigen_sqrt_double/8 6.51ns ± 0% 6.06ns ± 0% -6.95%
BM_eigen_sqrt_double/64 52.1ns ± 0% 46.5ns ± 1% -10.80%
BM_eigen_sqrt_double/512 417ns ± 0% 373ns ± 1% -10.59%
BM_eigen_sqrt_double/4k 3.33µs ± 0% 2.97µs ± 1% -10.79%
BM_eigen_sqrt_double/32k 26.7µs ± 0% 23.8µs ± 0% -10.94%
BM_eigen_sqrt_double/256k 214µs ± 0% 208µs ± 2% -2.76%
BM_eigen_sqrt_double/1M 866µs ± 3% 923µs ± 7% ~
2020-12-16 18:16:11 +00:00
David Tellenbach
751f18f2c0
Remove comma at the end of enumeration list to silence C++03 warnings
2020-12-13 18:11:02 +01:00
Antonio Sanchez
8c9976d7f0
Fix more SSE/AVX packet conversions for peven.
...
MSVC doesn't like function-style casts and forces us to use intrinsics.
2020-12-11 15:46:42 -08:00
Rasmus Munk Larsen
125cc9a5df
Implement vectorized complex square root.
...
Closes #1905
Measured speedup for sqrt of `complex<float>` on Skylake:
SSE:
```
name old time/op new time/op delta
BM_eigen_sqrt_ctype/1 49.4ns ± 0% 54.3ns ± 0% +10.01%
BM_eigen_sqrt_ctype/8 332ns ± 0% 50ns ± 1% -84.97%
BM_eigen_sqrt_ctype/64 2.81µs ± 1% 0.38µs ± 0% -86.49%
BM_eigen_sqrt_ctype/512 23.8µs ± 0% 3.0µs ± 0% -87.32%
BM_eigen_sqrt_ctype/4k 202µs ± 0% 24µs ± 2% -88.03%
BM_eigen_sqrt_ctype/32k 1.63ms ± 0% 0.19ms ± 0% -88.18%
BM_eigen_sqrt_ctype/256k 13.0ms ± 0% 1.5ms ± 1% -88.20%
BM_eigen_sqrt_ctype/1M 52.1ms ± 0% 6.2ms ± 0% -88.18%
```
AVX2:
```
name old cpu/op new cpu/op delta
BM_eigen_sqrt_ctype/1 53.6ns ± 0% 55.6ns ± 0% +3.71%
BM_eigen_sqrt_ctype/8 334ns ± 0% 27ns ± 0% -91.86%
BM_eigen_sqrt_ctype/64 2.79µs ± 0% 0.22µs ± 2% -92.28%
BM_eigen_sqrt_ctype/512 23.8µs ± 1% 1.7µs ± 1% -92.81%
BM_eigen_sqrt_ctype/4k 201µs ± 0% 14µs ± 1% -93.24%
BM_eigen_sqrt_ctype/32k 1.62ms ± 0% 0.11ms ± 1% -93.29%
BM_eigen_sqrt_ctype/256k 13.0ms ± 0% 0.9ms ± 1% -93.31%
BM_eigen_sqrt_ctype/1M 52.0ms ± 0% 3.5ms ± 1% -93.31%
```
AVX512:
```
name old cpu/op new cpu/op delta
BM_eigen_sqrt_ctype/1 53.7ns ± 0% 56.2ns ± 1% +4.75%
BM_eigen_sqrt_ctype/8 334ns ± 0% 18ns ± 2% -94.63%
BM_eigen_sqrt_ctype/64 2.79µs ± 0% 0.12µs ± 1% -95.54%
BM_eigen_sqrt_ctype/512 23.9µs ± 1% 1.0µs ± 1% -95.89%
BM_eigen_sqrt_ctype/4k 202µs ± 0% 8µs ± 1% -96.13%
BM_eigen_sqrt_ctype/32k 1.63ms ± 0% 0.06ms ± 1% -96.15%
BM_eigen_sqrt_ctype/256k 13.0ms ± 0% 0.5ms ± 4% -96.11%
BM_eigen_sqrt_ctype/1M 52.1ms ± 0% 2.0ms ± 1% -96.13%
```
2020-12-08 18:13:35 -08:00
Rasmus Munk Larsen
f9fac1d5b0
Add log2() to Eigen.
2020-12-04 21:45:09 +00:00
Antonio Sanchez
e2f21465fe
Special function implementations for half/bfloat16 packets.
...
Current implementations fail to consider half-float packets, only
half-float scalars. Added specializations for packets on AVX, AVX512 and
NEON. Added tests to `special_packetmath`.
The current `special_functions` tests would fail for half and bfloat16 due to
lack of precision. The NEON tests also fail with precision issues and
due to different handling of `sqrt(inf)`, so special functions bessel, ndtri
have been disabled.
Tested with AVX, AVX512.
2020-12-04 10:16:29 -08:00
Rasmus Munk Larsen
f23dc5b971
Revert "Add log2() operator to Eigen"
...
This reverts commit 4d91519a9b
2020-12-03 14:32:45 -08:00
Rasmus Munk Larsen
4d91519a9b
Add log2() operator to Eigen
2020-12-03 22:31:44 +00:00
Rasmus Munk Larsen
e57281a741
Fix a few issues for AVX512. This change enables vectorized versions of log, exp, log1p, expm1 when AVX512DQ is not available.
2020-12-01 11:31:47 -08:00
Antonio Sanchez
89f90b585d
AVX512 missing ops.
...
This allows the `packetmath` tests to pass for AVX512 on skylake.
Made `half` and `bfloat16` consistent in terms of ops they support.
Note the `log` tests are currently disabled for `bfloat16` since
they fail due to poor precision (they were previously disabled for
`Packet8bf` via test function specialization -- I just removed that
specialization and disabled it in the generic test).
2020-11-30 16:28:57 +00:00
