View source for Multiply–accumulate operation

{{Short description|operation common in numerical signal processing}}
In [[computing]], especially [[digital signal processing]], the '''multiply–accumulate operation''' is a common step that computes the product of two numbers and adds that product to an [[accumulator (computing)|accumulator]].  The hardware unit that performs the operation is known as a '''multiplier–accumulator''' ('''MAC''', or '''MAC unit'''); the operation itself is also often called a MAC or a MAC operation.  The MAC operation modifies an accumulator ''a'':

:<math>\ a \leftarrow a + ( b \times c )</math>

When done with [[floating point]] numbers, it might be performed with two [[rounding]]s (typical in many [[digital signal processors|DSP]]s), or with a single rounding.  When performed with a single rounding, it is called a '''fused multiply–add''' ('''FMA''') or '''fused multiply–accumulate''' ('''FMAC''').

Modern computers may contain a dedicated MAC, consisting of a multiplier implemented in [[combinational logic]] followed by an [[adder (electronics)|adder]] and an accumulator register that stores the result. The output of the register is fed back to one input of the adder, so that on each clock cycle, the output of the multiplier is added to the register. Combinational multipliers require a large amount of logic, but can compute a product much more quickly than the [[peasant multiplication|method of shifting and adding]] typical of earlier computers. [[Percy Ludgate]] was the first to conceive a MAC in his Analytical Machine of 1909,<ref>{{cite web |url=https://www.fano.co.uk/ludgate/ |title=The Feasibility of Ludgate's Analytical Machine}}</ref> and the first to exploit a MAC for division (using multiplication seeded by reciprocal, via the convergent series {{math|(1+''x'')<sup>−1</sup>}}). The first modern processors to be equipped with MAC units were [[digital signal processor]]s, but the technique is now also common in general-purpose processors.

==In floating-point arithmetic==
When done with [[integer]]s, the operation is typically exact (computed [[modular arithmetic|modulo]] some [[power of two]]). However, [[floating-point]] numbers have only a certain amount of mathematical [[arithmetic precision|precision]]. That is, digital floating-point arithmetic is generally not [[associativity|associative]] or [[distributivity|distributive]]. (See {{slink|Floating point#Accuracy problems}}.)
Therefore, it makes a difference to the result whether the multiply–add is performed with two roundings, or in one operation with a single rounding (a fused multiply–add). [[IEEE 754-2008]] specifies that it must be performed with one rounding, yielding a more accurate result.<ref>{{cite web |url=https://developer.nvidia.com/sites/default/files/akamai/cuda/files/NVIDIA-CUDA-Floating-Point.pdf |title=Precision & Performance: Floating Point and IEEE 754 Compliance for NVIDIA GPUs |publisher=nvidia |first1=Nathan |last1=Whitehead |first2=Alex |last2=Fit-Florea |year=2011 |accessdate=2013-08-31}}</ref>

==Fused multiply–add==
A ''fused'' multiply–add (sometimes known as ''FMA'' or ''fmadd'')<ref>{{cite web|title=fmadd instrs|url=https://www.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.alangref/idalangref_fmadd_instrs.htm}}</ref>
is a floating-point multiply–add operation performed in one step, with a single rounding. That is, where an unfused multiply–add would compute the product ''b''×''c'', round it to ''N'' significant bits, add the result to ''a'', and round back to ''N'' significant bits, a fused multiply–add would compute the entire expression ''a''+''b''×''c'' to its full precision before rounding the final result down to ''N'' significant bits.

A fast FMA can speed up and improve the accuracy of many computations that involve the accumulation of products:

* [[Dot product]]
* [[Matrix multiplication]]
* [[Polynomial]] evaluation (e.g., with [[Horner's rule]])
* [[Newton's method]] for evaluating functions (from the inverse function)
* [[Convolutions]] and [[artificial neural networks]] 

Fused multiply–add can usually be relied on to give more accurate results. However, [[William Morton Kahan|William Kahan]] has pointed out that it can give problems if used unthinkingly.<ref>{{cite web |title=IEEE Standard 754 for Binary Floating-Point Arithmetic |author-first=William |author-last=Kahan |author-link=William Morton Kahan |url=http://www.cs.berkeley.edu/~wkahan/ieee754status/ieee754.ps |date=1996-05-31}}</ref> If {{math|''x''<sup>2</sup> − ''y''<sup>2</sup>}} is evaluated as {{math| ((''x''×''x'') − ''y''×''y'')}} using fused multiply–add, then the result may be negative even when {{math|''x'' {{=}} ''y''}} due to the first multiplication discarding low significance bits. This could then lead to an error if, for instance, the square root of the result is then evaluated.

When implemented inside a [[microprocessor]], an FMA can actually be faster than a multiply operation followed by an add.  However, standard industrial implementations based on the original IBM RS/6000 design require a 2''N''-bit adder to compute the sum properly.<ref name="bridged">{{cite web |url=http://users.ece.utexas.edu/~quinnell/Research/Bridged%20Floating-Point%20Fused%20Multiply-Add%20Design.pdf |title=Bridged Floating-Point Fused Multiply–Add Design |author-first=Eric |author-last=Quinnell<!-- |date=undated, circa 2006 --> |display-authors=etal}}{{Dead link|date=March 2011}}</ref><ref>{{cite thesis |url=http://repositories.lib.utexas.edu/bitstream/handle/2152/3082/quinnelle60861.pdf |date=May 2007 |title=Floating-Point Fused Multiply–Add Architectures |author-first=Eric |author-last=Quinnell |degree=PhD |access-date=2011-03-28}}</ref>

A useful benefit of including this instruction is that it allows an efficient software implementation of [[division (mathematics)|division]] (see [[division algorithm]]) and [[square root]] (see [[methods of computing square roots]]) operations, thus eliminating the need for dedicated hardware for those operations.<ref name="goldschmidt_algo">{{cite web |citeseerx=10.1.1.85.9648 |title=Software Division and Square Root Using Goldschmidt's Algorithms |author-first=Peter |author-last=Markstein |date=November 2004}}</ref>

===Dot product instruction===
Some machines combine multiple fused multiply add operations into a single step, e.g. performing a four-element dot-product on two 128-bit [[SIMD]] registers ''a0''×''b0''+''a1''×''b1''+''a2''×''b2''+''a3''×''b3'' with single cycle throughput.

===Support===
The FMA operation is included in [[IEEE 754-2008]].

The [[Digital Equipment Corporation|DEC]] [[VAX]]'s POLY instruction is used for evaluating polynomials with [[Horner's rule]] using a succession of multiply and add steps.  Instruction descriptions do not specify whether the multiply and add are performed using a single fma step.<ref>{{Cite web |url=http://uranium.vaxpower.org/~isildur/vax/week.html |title=VAX instruction of the week: POLY}}</ref> This instruction has been a part of the VAX instruction set since its original 11/780 implementation in 1977.

The [[C99|1999 standard]] of the [[C (programming language)|C programming language]] supports the FMA operation through the <code>fma</code> standard math library function, and standard pragmas controlling optimizations based on FMA.

The fused multiply–add operation was introduced as ''multiply–add fused'' in the IBM [[POWER1]] (1990) processor,<ref>{{cite journal |author-first1=R. K. |author-last1=Montoye |author-first2=E. |author-last2=Hokenek |author-first3=S. L. |author-last3=Runyon |url=http://domino.research.ibm.com/tchjr/journalindex.nsf/4ac37cf0bdc4dd6a85256547004d47e1/e3d1d5353695231c85256bfa0067fa31?OpenDocument |title=Design of the IBM RISC System/6000 floating-point execution unit |journal=IBM Journal of Research and Development |volume=34 |issue=1 |date=January 1990 |pages=59–70 |issn=0018-8646 |doi=10.1147/rd.341.0059}}</ref> but has been added to numerous other processors since then:

* [[Hewlett-Packard|HP]] [[PA-8000]] (1996) and above
* [[Hitachi, Ltd.|Hitachi]] [[SuperH#SH-4|SuperH SH-4]] (1998)
* [[Sony Computer Entertainment|SCE]]-[[Toshiba]] [[Emotion Engine]] (1999)
* Intel [[Itanium]] (2001)
* STI [[Cell (microprocessor)|Cell]] (2006)
* [[Fujitsu]] [[SPARC64 VI]] (2007) and above
* ([[MIPS architecture|MIPS]]-compatible) [[Loongson]]-2F (2008)<ref>{{cite web |url=http://www.mdronline.com/mpr/h/2008/1103/224401.html |title=Godson-3 Emulates x86: New MIPS-Compatible Chinese Processor Has Extensions for x86 Translation}}</ref>
* [[Elbrus-8S|Elbrus-8SV]] (2018)
* x86 processors with [[FMA instruction set|FMA3 and/or FMA4 instruction set]]
** AMD [[Bulldozer (processor)|Bulldozer]] (2011, FMA4 only)
** AMD [[Piledriver (microarchitecture)|Piledriver]] (2012, FMA3 and FMA4)<ref>https://pl.scribd.com/document/138572809/New-Bulldozer-and-Piledriver-Instructions</ref>
** AMD [[Steamroller (microarchitecture)|Steamroller]] (2014)
**AMD [[Excavator (microarchitecture)|Excavator]] (2015)
**AMD [[Zen (microarchitecture)|Zen]] (2017, FMA3 only)
**[[Intel Haswell]] (2013, FMA3 only)<ref>{{cite web |url=http://www.reghardware.co.uk/2008/08/19/idf_intel_architecture_roadmap/ |title=Intel adds 22nm octo-core 'Haswell' to CPU design roadmap |work=The Register}}</ref>
**Intel [[Skylake (microarchitecture)|Skylake]] (2015, FMA3 only)
* ARM processors with VFPv4 and/or NEONv2:
** [[ARM Cortex-M4F]] (2010)
** [[ARM Cortex-A5]] (2012)
** [[ARM Cortex-A7 MPCore|ARM Cortex-A7]] (2013)
** [[ARM Cortex-A15 MPCore|ARM Cortex-A15]] (2012)
** [[Krait (CPU)|Qualcomm Krait]] (2012)
** [[Apple A6]] (2012)
** All [[ARM architecture#ARMv8-A|ARMv8]] processors
* GPUs and GPGPU boards:
** [[Advanced Micro Devices]] GPUs (2009) and newer
*** [[TeraScale (microarchitecture)#TeraScale 2|TeraScale 2 "Evergreen"]]-series based
*** [[Graphics Core Next]]-based
** [[NVidia]] GPUs (2010) and newer
*** [[Fermi (microarchitecture)|Fermi]]-based (2010)
*** [[Kepler (microarchitecture)|Kepler]]-based (2012)
*** [[Maxwell (microarchitecture)|Maxwell]]-based (2014)
*** [[Pascal (microarchitecture)|Pascal]]-based (2016)
*** [[Volta (microarchitecture)|Volta]]-based (2017)
** Intel GPUs since [[Intel HD and Iris Graphics#Sandy Bridge|Sandy Bridge]]
** [[Intel MIC]] (2012)
** [[Mali (GPU)|ARM Mali T600 Series]] (2012) and above
* Vector Processors:
** [[NEC SX-Aurora TSUBASA]]

==References==
{{Reflist}}
{{Graphics Processing Unit}}

{{DEFAULTSORT:Multiply-accumulate operation}}
[[Category:Computer arithmetic]]
[[Category:Digital signal processing]]