devtools/eigen
1
0
Fork 0
mirror of https://gitlab.com/libeigen/eigen.git synced 2026-04-10 11:34:33 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,441 Commits 15 Branches 93 Tags
94e2250b36f22e821507b4e6e47ce36236bf65da
Commit Graph

55 Commits

Author SHA1 Message Date
Rasmus Munk Larsen
a235ddef39 Get rid of code duplication for conj_helper. For packets where LhsType=RhsType a single generic implementation suffices. For scalars, the generic implementation of pconj automatically forwards to numext::conj, so much of the existing specialization can be avoided. For mixed types we still need specializations. 
(cherry picked from commit 52a5f98212)
 2021-06-24 23:30:42 +00:00
Christoph Hertzberg
9e0dc8f09b Revert addition of unused paddsub<Packet2cf>. This fixes #2242 
(cherry picked from commit 722ca0b665)
 2021-05-07 16:23:03 +00:00
Guoqiang QI
38ae5353ab 1)provide a better generic paddsub op implementation 
2)make paddsub op support the Packet2cf/Packet4f/Packet2f in NEON
3)make paddsub op support the Packet2cf/Packet4f in SSE
 2021-01-13 22:54:03 +00:00
David Tellenbach
c7eb3a74cb Don't guard psqrt for std::complex<float> with EIGEN_ARCH_ARM64 2020-12-11 12:41:52 +01:00
David Tellenbach
00be0a7ff3 Fix vectorization of complex sqrt on NEON 2020-12-10 15:23:23 +00:00
Brad King
880fa43b2b Add support for CastXML on ARM aarch64 
CastXML simulates the preprocessors of other compilers, but actually
parses the translation unit with an internal Clang compiler.
Use the same `vld1q_u64` workaround that we do for Clang.

Fixes: #1979
 2020-09-16 13:40:23 -04:00
Rasmus Munk Larsen
225ab040e0 Remove unused packet op "palign". 
Clean up a compiler warning in c++03 mode in AVX512/Complex.h.
 2020-05-07 17:14:26 -07:00
Rasmus Munk Larsen
e80ec24357 Remove unused packet op "preduxp". 2020-04-23 18:17:14 +00:00
Joel Holdsworth
52d54278be Additional NEON packet-math operations 2020-03-26 20:18:19 +00:00
Joel Holdsworth
eacf97f727 NEON: Implemented half-size vectors 2020-03-10 22:46:19 +00:00
Joel Holdsworth
da5a7afed0 Improvements to the tidiness and completeness of the NEON implementation 2020-01-10 18:31:15 +00:00
Eugene Zhulenev
e9f0eb8a5e Add masked_store_available to unpacket_traits 2019-05-02 14:52:58 -07:00
Anuj Rawat
8c7a6feb8e Adding lowlevel APIs for optimized RHS packet load in TensorFlow 
SpatialConvolution

Low-level APIs are added in order to optimized packet load in gemm_pack_rhs
in TensorFlow SpatialConvolution. The optimization is for scenario when a
packet is split across 2 adjacent columns. In this case we read it as two
'partial' packets and then merge these into 1. Currently this only works for
Packet16f (AVX512) and Packet8f (AVX2). We plan to add this for other
packet types (such as Packet8d) also.

This optimization shows significant speedup in SpatialConvolution with
certain parameters. Some examples are below.

Benchmark parameters are specified as:
Batch size, Input dim, Depth, Num of filters, Filter dim

Speedup numbers are specified for number of threads 1, 2, 4, 8, 16.

AVX512:

Parameters                  | Speedup (Num of threads: 1, 2, 4, 8, 16)
----------------------------|------------------------------------------
128,   24x24,  3, 64,   5x5 |2.18X, 2.13X, 1.73X, 1.64X, 1.66X
128,   24x24,  1, 64,   8x8 |2.00X, 1.98X, 1.93X, 1.91X, 1.91X
 32,   24x24,  3, 64,   5x5 |2.26X, 2.14X, 2.17X, 2.22X, 2.33X
128,   24x24,  3, 64,   3x3 |1.51X, 1.45X, 1.45X, 1.67X, 1.57X
 32,   14x14, 24, 64,   5x5 |1.21X, 1.19X, 1.16X, 1.70X, 1.17X
128, 128x128,  3, 96, 11x11 |2.17X, 2.18X, 2.19X, 2.20X, 2.18X

AVX2:

Parameters                  | Speedup (Num of threads: 1, 2, 4, 8, 16)
----------------------------|------------------------------------------
128,   24x24,  3, 64,   5x5 | 1.66X, 1.65X, 1.61X, 1.56X, 1.49X
 32,   24x24,  3, 64,   5x5 | 1.71X, 1.63X, 1.77X, 1.58X, 1.68X
128,   24x24,  1, 64,   5x5 | 1.44X, 1.40X, 1.38X, 1.37X, 1.33X
128,   24x24,  3, 64,   3x3 | 1.68X, 1.63X, 1.58X, 1.56X, 1.62X
128, 128x128,  3, 96, 11x11 | 1.36X, 1.36X, 1.37X, 1.37X, 1.37X

In the higher level benchmark cifar10, we observe a runtime improvement
of around 6% for AVX512 on Intel Skylake server (8 cores).

On lower level PackRhs micro-benchmarks specified in TensorFlow
tensorflow/core/kernels/eigen_spatial_convolutions_test.cc, we observe
the following runtime numbers:

AVX512:

Parameters                                                     | Runtime without patch (ns) | Runtime with patch (ns) | Speedup
---------------------------------------------------------------|----------------------------|-------------------------|---------
BM_RHS_NAME(PackRhs, 128, 24, 24, 3, 64, 5, 5, 1, 1, 256, 56)  |  41350                     | 15073                   | 2.74X
BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 1, 1, 256, 56)  |   7277                     |  7341                   | 0.99X
BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 2, 2, 256, 56)  |   8675                     |  8681                   | 1.00X
BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 1, 1, 256, 56)  |  24155                     | 16079                   | 1.50X
BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 2, 2, 256, 56)  |  25052                     | 17152                   | 1.46X
BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 1, 1, 256, 56) |  18269                     | 18345                   | 1.00X
BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 2, 4, 256, 56) |  19468                     | 19872                   | 0.98X
BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 1, 1, 36, 432)   | 156060                     | 42432                   | 3.68X
BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 2, 2, 36, 432)   | 132701                     | 36944                   | 3.59X

AVX2:

Parameters                                                     | Runtime without patch (ns) | Runtime with patch (ns) | Speedup
---------------------------------------------------------------|----------------------------|-------------------------|---------
BM_RHS_NAME(PackRhs, 128, 24, 24, 3, 64, 5, 5, 1, 1, 256, 56)  | 26233                      | 12393                   | 2.12X
BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 1, 1, 256, 56)  |  6091                      |  6062                   | 1.00X
BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 2, 2, 256, 56)  |  7427                      |  7408                   | 1.00X
BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 1, 1, 256, 56)  | 23453                      | 20826                   | 1.13X
BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 2, 2, 256, 56)  | 23167                      | 22091                   | 1.09X
BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 1, 1, 256, 56) | 23422                      | 23682                   | 0.99X
BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 2, 4, 256, 56) | 23165                      | 23663                   | 0.98X
BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 1, 1, 36, 432)   | 72689                      | 44969                   | 1.62X
BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 2, 2, 36, 432)   | 61732                      | 39779                   | 1.55X

All benchmarks on Intel Skylake server with 8 cores.
 2019-04-20 06:46:43 +00:00
Gael Guennebaud
7e3084bb6f Fix compilation on ARM. 2019-02-22 14:56:12 +01:00
Rasmus Munk Larsen
4d7f317102 Add a few missing packet ops: cmp_eq for NEON. pfloor for GPU. 2019-02-21 13:32:13 -08:00
Gael Guennebaud
eb4c6bb22d Fix conflicts and merge 2019-01-30 15:57:08 +01:00
Gael Guennebaud
250dcd1fdb bug #1652: fix position of EIGEN_ALIGN16 attributes in Neon and Altivec 2019-01-14 21:45:56 +01:00
Gustavo Lima Chaves
e763fcd09e Introducing "vectorized" byte on unpacket_traits structs 
This is a preparation to a change on gebp_traits, where a new template
argument will be introduced to dictate the packet size, so it won't be
bound to the current/max packet size only anymore.

By having packet types defined early on gebp_traits, one has now to
act on packet types, not scalars anymore, for the enum values defined
on that class. One approach for reaching the vectorizable/size
properties one needs there could be getting the packet's scalar again
with unpacket_traits<>, then the size/Vectorizable enum entries from
packet_traits<>. It turns out guards like "#ifndef
EIGEN_VECTORIZE_AVX512" at AVX/PacketMath.h will hide smaller packet
variations of packet_traits<> for some types (and it makes sense to
keep that). In other words, one can't go back to the scalar and create
a new PacketType, as this will always lead to the maximum packet type
for the architecture.

The less costly/invasive solution for that, thus, is to add the
vectorizable info on every unpacket_traits struct as well.
 2018-12-19 14:24:44 -08:00
Gael Guennebaud
6e7118265d Fix compilation with NEON+MSVC 2018-04-26 10:50:41 +02:00
Gael Guennebaud
686fb57233 fix const cast in NEON 2018-04-18 18:46:34 +02:00
Gael Guennebaud
b240080e64 bug #1436: fix compilation of Jacobi rotations with ARM NEON, some specializations of internal::conj_helper were missing. 2017-06-15 10:16:30 +02:00
Gael Guennebaud
89e315152c bug #1325: fix compilation on NEON with clang 2016-10-12 16:55:47 +02:00
Konstantinos Margaritis
ef05463fcf Merged kmargar/eigen/tip into default, Altivec/VSX port should be working ok now. 2016-07-10 16:11:46 +03:00
Benoit Jacob
38fb606052 Avoid global variables with static constructors in NEON/Complex.h 2016-06-28 11:12:49 -04:00
Gael Guennebaud
0028049380 bug #1240: Remove any assumption on NEON vector types. 2016-06-09 23:08:11 +02:00
Sean Templeton
bd21243821 Fix compile errors initializing packets on ARM DS-5 5.20 
The ARM DS-5 5.20 compiler fails compiling with the following errors:

"src/Core/arch/NEON/PacketMath.h", line 113: Error:  #146: too many initializer values
    Packet4f countdown = EIGEN_INIT_NEON_PACKET4(0, 1, 2, 3);
                         ^
"src/Core/arch/NEON/PacketMath.h", line 118: Error:  #146: too many initializer values
    Packet4i countdown = EIGEN_INIT_NEON_PACKET4(0, 1, 2, 3);
                         ^
"src/Core/arch/NEON/Complex.h", line 30: Error:  #146: too many initializer values
  static uint32x4_t p4ui_CONJ_XOR = EIGEN_INIT_NEON_PACKET4(0x00000000, 0x80000000, 0x00000000, 0x80000000);
                                    ^
"src/Core/arch/NEON/Complex.h", line 31: Error:  #146: too many initializer values
  static uint32x2_t p2ui_CONJ_XOR = EIGEN_INIT_NEON_PACKET2(0x00000000, 0x80000000);
                                    ^

The vectors are implemented as two doubles, hence the too many initializer values error.
Changed the code to use intrinsic load functions which all compilers
implementing NEON should have.
 2016-06-03 10:51:35 -05:00
Konstantinos Margaritis
950158f6d1 add name to copyrights 2016-04-28 14:32:11 -03:00
Gael Guennebaud
7ad1aaec1d bug #1103: fix neon vectorization of pmul(Packet1cd,Packet1cd) 2015-12-10 16:06:33 +01:00
Gael Guennebaud
ce57dbd937 Let unpacket_traits<> exposes the required alignment and make use of it everywhere 2015-08-07 10:44:01 +02:00
Benoit Jacob
807793ec3b Fix uninitialized var warning. The compiler was clearing the register anyway, so this does not change resulting code 2015-05-15 11:15:53 -04:00
Benoit Jacob
ccc1277a42 must also disable complex<double> when disabling double vectorization 2015-03-03 10:17:05 -05:00
Gael Guennebaud
45cbb0bbb1 The usage of DenseIndex is deprecated, so let's replace DenseIndex by Index 2015-02-16 15:05:41 +01:00
Gael Guennebaud
ee06f78679 Introduce unified macros to identify compiler, OS, and architecture. They are all defined in util/Macros.h and prefixed with EIGEN_COMP_, EIGEN_OS_, and EIGEN_ARCH_ respectively. 2014-11-04 21:58:52 +01:00
Konstantinos Margaritis
fae4fd7a26 Added ARMv8 support 2014-10-22 07:39:49 +00:00
Gael Guennebaud
b47ef1431f Fix many long to int implicit conversions 2014-07-08 16:47:11 +02:00
Gael Guennebaud
4def7b1fa5 Fix ptranspose overload prototypes for NEON 2014-04-25 11:15:13 +02:00
Gael Guennebaud
3d8d0f6269 Enable vectorization of pack_rhs with a column-major RHS. 
Rename and generalize Kernel<*> to PacketBlock<*,N>.
 2014-04-25 10:56:18 +02:00
Benoit Steiner
ccb4dec719 Created a NEON version of the ptranspose packet primitives 2014-04-23 18:22:10 -07:00
Benoit Steiner
6d6df90c9a Implemented the pgather/pscatter packet primitives for the arm/NEON architecture 2014-04-17 12:28:01 -07:00
Gael Guennebaud
d5a795f673 New gebp kernel handling up to 3 packets x 4 register-level blocks. Huge speeup on Haswell. 
This changeset also introduce new vector functions: ploadquad and predux4.
 2014-04-16 17:05:11 +02:00
Gael Guennebaud
10aa14592a Add a mechanism to recursively access to half-size packet types 2014-03-28 10:18:04 +01:00
Gael Guennebaud
19521c83b8 bug #677: fix usage of pld instrinsics for ccomplexes 2013-11-02 12:10:48 +01:00
Simon Pilgrim
ca67c60150 Fix bug #591: minor optimization in NEON vectorization support 2013-06-10 15:59:03 +02:00
Benoit Jacob
69124cfca2 Automatic relicensing to MPL2 using Keirs script. Manual fixup follows. 2012-07-13 14:42:47 -04:00
Jitse Niesen
3c412183b2 Get rid of include directives inside namespace blocks (bug #339). 2012-04-15 11:06:28 +01:00
Marton Danoczy
f422668d39 Patches to support ARM NEON with Clang 3.0 and LLVM-GCC 2011-11-04 16:37:10 +01:00
Gael Guennebaud
97b6d26f5b fix compilation on ARM NEON (missing AlignedOnScalar) 2011-05-06 09:03:48 +02:00
Gael Guennebaud
c121e6f390 implement ploaddup for complex and SSE/NEON even though they are not used in practice 2011-02-23 16:31:42 +01:00
Gael Guennebaud
6e01780541 fix a couple of issues with pcplxflip 2011-02-23 17:51:40 +03:00
Gael Guennebaud
7dc18b20bb same for neon 2011-02-23 09:41:55 +01:00