Rasmus Munk Larsen
0488b708b4
Vectorize tensor.isnan() by using typed predicates.
2023-03-16 04:04:22 +00:00
Antonio Sánchez
394aabb0a3
Fix failing MSVC tests due to compiler bugs.
2023-03-10 22:36:57 +00:00
Rasmus Munk Larsen
1c0a6cf228
Get rid of EIGEN_HAS_AVX512_MATH workaround.
2023-02-23 23:16:41 +00:00
Rasmus Munk Larsen
ce62177b5b
Vectorize atanh & add a missing definition and unit test for atan.
2023-02-21 03:14:05 +00:00
Sean McBride
d70b4864d9
issue #2581 : review and cleanup of compiler version checks
2023-01-17 18:58:34 +00:00
Antonio Sánchez
8588d8c74b
Correct pnegate for floating-point zero.
2022-11-15 18:07:23 +00:00
Charles Schlosser
82b152dbe7
Add signbit function
2022-11-04 00:31:20 +00:00
Antonio Sánchez
e5794873cb
Replace assert with eigen_assert.
2022-10-04 17:11:23 +00:00
Rasmus Munk Larsen
c475228b28
Vectorize atan() for double.
2022-10-01 01:49:30 +00:00
Rasmus Munk Larsen
7b2901e2aa
Add vectorized integer division for int32 with AVX512, AVX or SSE.
2022-09-21 00:27:23 +00:00
Rasmus Munk Larsen
bd393e15c3
Vectorize acos, asin, and atan for float.
2022-08-29 19:49:33 +00:00
Charles Schlosser
e5af9f87f2
Vectorize pow for integer base / exponent types
2022-08-29 19:23:54 +00:00
Matthew Sterrett
7a3b667c43
Add support for AVX512-FP16 for vectorizing half precision math
2022-08-17 18:15:21 +00:00
Matthew Sterrett
39fcc89798
Removed unnecessary checks for FP16C
2022-08-16 18:14:41 +00:00
aaraujom
d49ede4dc4
Add AVX512 s/dgemm optimizations for compute kernel (2nd try)
2022-05-28 02:00:21 +00:00
Antonio Sánchez
9b9496ad98
Revert "Add AVX512 optimizations for matrix multiply"
...
This reverts commit 25db0b4a82
2022-05-13 18:50:33 +00:00
aaraujom
25db0b4a82
Add AVX512 optimizations for matrix multiply
2022-05-12 23:41:19 +00:00
Antonio Sánchez
07db964bde
Restrict new AVX512 trsm to AVX512VL, rename files for consistency.
2022-04-14 16:58:32 +00:00
Antonio Sánchez
9a14d91a99
Fix AVX512 builds with MSVC.
2022-03-18 16:04:53 +00:00
b-shi
518fc321cb
AVX512 Optimizations for Triangular Solve
2022-03-16 18:04:50 +00:00
Antonio Sánchez
e7f4a901ee
Define EIGEN_HAS_AVX512_MATH in PacketMath.
2022-02-04 22:25:52 +00:00
Antonio Sánchez
96da541cba
Fix AVX512 math function consistency, enable for ICC.
2022-02-04 19:35:18 +00:00
Rasmus Munk Larsen
51311ec651
Remove inline assembly for FMA (AVX) and add remaining extensions as packet ops: pmsub, pnmadd, and pnmsub.
2022-01-26 04:25:41 +00:00
Rasmus Munk Larsen
ea2c02060c
Add reciprocal packet op and fast specializations for float with SSE, AVX, and AVX512.
2022-01-21 23:49:18 +00:00
Ilya Tokar
a0fc640c18
Add support for packets of int64 on x86
2022-01-21 19:55:23 +00:00
Kolja Brix
afa616bc9e
Fix some typos found
2021-09-23 15:22:00 +00:00
Rasmus Munk Larsen
7b975acb1f
Remove unused variable.
2021-09-16 20:27:13 +00:00
Rasmus Munk Larsen
92849d814b
Remove unused variable.
2021-09-16 20:21:31 +00:00
Rasmus Munk Larsen
d7d0bf832d
Issue an error in case of direct inclusion of internal headers.
2021-09-10 19:12:26 +00:00
Antonio Sanchez
3d4ba855e0
Fix AVX integer packet issues.
...
Most are instances of AVX2 functions not protected by
`EIGEN_VECTORIZE_AVX2`. There was also a missing semi-colon
for AVX512.
2021-09-01 14:14:43 -07:00
Jakub Lichman
dc5b1f7d75
AVX512 and AVX2 support for Packet16i and Packet8i added
2021-08-25 19:38:23 +00:00
Gauri Deshpande
e6a5a594a7
remove denormal flushing in fp32tobf16 for avx & avx512
2021-08-09 22:15:21 +00:00
Jakub Lichman
d87648a6be
Tests added and AVX512 bug fixed for pcmp_lt_or_nan
2021-04-25 20:58:56 +00:00
Jakub Lichman
2b1dfd1ba0
HasExp added for AVX512 Packet8d
2021-04-20 19:07:58 +00:00
Antonio Sanchez
1d79c68ba0
Fix ldexp for AVX512 ( #2215 )
...
Wrong shuffle was used. Need to interleave low/high halves with a
`permute` instruction.
Fixes #2215 .
2021-04-20 16:25:22 +00:00
Christoph Hertzberg
69a4f70956
Revert "Uses _mm512_abs_pd for Packet8d pabs"
...
This reverts commit f019b97aca
2021-03-23 18:52:19 +00:00
Steve Bronder
f019b97aca
Uses _mm512_abs_pd for Packet8d pabs
2021-03-18 15:47:52 +00:00
Antonio Sanchez
7ff0b7a980
Updated pfrexp implementation.
...
The original implementation fails for 0, denormals, inf, and NaN.
See #2150
2021-02-17 02:23:24 +00:00
Antonio Sanchez
4cb563a01e
Fix ldexp implementations.
...
The previous implementations produced garbage values if the exponent did
not fit within the exponent bits. See #2131 for a complete discussion,
and !375 for other possible implementations.
Here we implement the 4-factor version. See `pldexp_impl` in
`GenericPacketMathFunctions.h` for a full description.
The SSE `pcmp*` methods were moved down since `pcmp_le<Packet4i>`
requires `por`.
Left as a "TODO" is to delegate to a faster version if we know the
exponent does fit within the exponent bits.
Fixes #2131 .
2021-02-10 22:45:41 +00:00
Rasmus Munk Larsen
cdd8fdc32e
Vectorize pow(x, y). This closes https://gitlab.com/libeigen/eigen/-/issues/2085 , which also contains a description of the algorithm.
...
I ran some testing (comparing to `std::pow(double(x), double(y)))` for `x` in the set of all (positive) floats in the interval `[std::sqrt(std::numeric_limits<float>::min()), std::sqrt(std::numeric_limits<float>::max())]`, and `y` in `{2, sqrt(2), -sqrt(2)}` I get the following error statistics:
```
max_rel_error = 8.34405e-07
rms_rel_error = 2.76654e-07
```
If I widen the range to all normal float I see lower accuracy for arguments where the result is subnormal, e.g. for `y = sqrt(2)`:
```
max_rel_error = 0.666667
rms = 6.8727e-05
count = 1335165689
argmax = 2.56049e-32, 2.10195e-45 != 1.4013e-45
```
which seems reasonable, since these results are subnormals with only couple of significant bits left.
2021-01-18 13:25:16 +00:00
Antonio Sanchez
839aa505c3
Fix typo in AVX512 packet math.
2020-12-11 21:35:44 -08:00
Antonio Sanchez
8c9976d7f0
Fix more SSE/AVX packet conversions for peven.
...
MSVC doesn't like function-style casts and forces us to use intrinsics.
2020-12-11 15:46:42 -08:00
Rasmus Munk Larsen
125cc9a5df
Implement vectorized complex square root.
...
Closes #1905
Measured speedup for sqrt of `complex<float>` on Skylake:
SSE:
```
name old time/op new time/op delta
BM_eigen_sqrt_ctype/1 49.4ns ± 0% 54.3ns ± 0% +10.01%
BM_eigen_sqrt_ctype/8 332ns ± 0% 50ns ± 1% -84.97%
BM_eigen_sqrt_ctype/64 2.81µs ± 1% 0.38µs ± 0% -86.49%
BM_eigen_sqrt_ctype/512 23.8µs ± 0% 3.0µs ± 0% -87.32%
BM_eigen_sqrt_ctype/4k 202µs ± 0% 24µs ± 2% -88.03%
BM_eigen_sqrt_ctype/32k 1.63ms ± 0% 0.19ms ± 0% -88.18%
BM_eigen_sqrt_ctype/256k 13.0ms ± 0% 1.5ms ± 1% -88.20%
BM_eigen_sqrt_ctype/1M 52.1ms ± 0% 6.2ms ± 0% -88.18%
```
AVX2:
```
name old cpu/op new cpu/op delta
BM_eigen_sqrt_ctype/1 53.6ns ± 0% 55.6ns ± 0% +3.71%
BM_eigen_sqrt_ctype/8 334ns ± 0% 27ns ± 0% -91.86%
BM_eigen_sqrt_ctype/64 2.79µs ± 0% 0.22µs ± 2% -92.28%
BM_eigen_sqrt_ctype/512 23.8µs ± 1% 1.7µs ± 1% -92.81%
BM_eigen_sqrt_ctype/4k 201µs ± 0% 14µs ± 1% -93.24%
BM_eigen_sqrt_ctype/32k 1.62ms ± 0% 0.11ms ± 1% -93.29%
BM_eigen_sqrt_ctype/256k 13.0ms ± 0% 0.9ms ± 1% -93.31%
BM_eigen_sqrt_ctype/1M 52.0ms ± 0% 3.5ms ± 1% -93.31%
```
AVX512:
```
name old cpu/op new cpu/op delta
BM_eigen_sqrt_ctype/1 53.7ns ± 0% 56.2ns ± 1% +4.75%
BM_eigen_sqrt_ctype/8 334ns ± 0% 18ns ± 2% -94.63%
BM_eigen_sqrt_ctype/64 2.79µs ± 0% 0.12µs ± 1% -95.54%
BM_eigen_sqrt_ctype/512 23.9µs ± 1% 1.0µs ± 1% -95.89%
BM_eigen_sqrt_ctype/4k 202µs ± 0% 8µs ± 1% -96.13%
BM_eigen_sqrt_ctype/32k 1.63ms ± 0% 0.06ms ± 1% -96.15%
BM_eigen_sqrt_ctype/256k 13.0ms ± 0% 0.5ms ± 4% -96.11%
BM_eigen_sqrt_ctype/1M 52.1ms ± 0% 2.0ms ± 1% -96.13%
```
2020-12-08 18:13:35 -08:00
Antonio Sanchez
e2f21465fe
Special function implementations for half/bfloat16 packets.
...
Current implementations fail to consider half-float packets, only
half-float scalars. Added specializations for packets on AVX, AVX512 and
NEON. Added tests to `special_packetmath`.
The current `special_functions` tests would fail for half and bfloat16 due to
lack of precision. The NEON tests also fail with precision issues and
due to different handling of `sqrt(inf)`, so special functions bessel, ndtri
have been disabled.
Tested with AVX, AVX512.
2020-12-04 10:16:29 -08:00
Rasmus Munk Larsen
e57281a741
Fix a few issues for AVX512. This change enables vectorized versions of log, exp, log1p, expm1 when AVX512DQ is not available.
2020-12-01 11:31:47 -08:00
Antonio Sanchez
89f90b585d
AVX512 missing ops.
...
This allows the `packetmath` tests to pass for AVX512 on skylake.
Made `half` and `bfloat16` consistent in terms of ops they support.
Note the `log` tests are currently disabled for `bfloat16` since
they fail due to poor precision (they were previously disabled for
`Packet8bf` via test function specialization -- I just removed that
specialization and disabled it in the generic test).
2020-11-30 16:28:57 +00:00
Guoqiang QI
4700713faf
Add AVX plog<Packet4d> and AVX512 plog<Packet8d> ops,also unified AVX512 plog<Packet16f> op with generic api
2020-10-15 00:54:45 +00:00
Rasmus Munk Larsen
af6f43d7ff
Add specializations for pmin/pmax with prescribed NaN propagation semantics for SSE/AVX/AVX512.
2020-10-14 23:11:24 +00:00
Rasmus Munk Larsen
4e4d3f32d1
Clean up packetmath tests and fix various bugs to make bfloat16 pass (almost) all packetmath tests with SSE, AVX, and AVX512.
2020-10-09 20:05:49 +00:00
Teng Lu
3ec4f0b641
Fix undefine BF16 union behavior in AVX512.
2020-07-29 02:20:21 +00:00
