[{"content":" Reference: Thomas L. Floyd, Digital Fundamentals, Pearson. Chapter 1, Introductory Concepts.\nThis course deals with circuits that process signals represented by the two values 0 and 1. The first half covers the rules of numbers and logic, such as number systems, logic gates and Boolean algebra, and the second half covers the smallest circuits that implement those rules, such as latches, flip-flops, registers and counters1. The material below corresponds to Sections 1.1 through 1.4 of the textbook and to lecture slides2 2 through 26. Chapter 1 defines only the inputs and outputs of each block, and later chapters build the circuits inside.\nAnalog and digital quantities An analog quantity takes continuous values. The temperature over a day passes through every intermediate value between any two instants, so it is an analog quantity. A digital quantity takes only distinct discrete values, and in a digital circuit those values are 0 and 13.\nTo bring an analog quantity into a digital circuit, its value is read at fixed time intervals. Reading the value is sampling, and forcing each reading onto one of a fixed set of levels is quantization. In the figure below the cyan curve is the continuously varying temperature and the magenta dots are the samples read once per hour.\nThe textbook contrasts the two kinds of quantity with three systems.\nSystem Signal processing Public address system (microphone, amplifier, speaker) Amplifies the audio waveform as it is. Analog from end to end CD player Reads digitally stored data and rebuilds an analog audio waveform with a DAC4 Mechatronics (robotic arm) Electronic controls decide digitally and an electromechanical interface moves the motors Transmitting a voice follows the same order. The analog voice is sampled into data, the data is sent, and the receiving side restores an analog waveform5. The converters that move between AC and DC among the charger, battery and motor of an electric vehicle6 are another case where both forms of signal are used in turn inside one device.\nLogic levels In a digital circuit 1 and 0 are defined as voltage ranges. HIGH (1) runs from $V_{H(min)}$ to $V_{H(max)}$, LOW (0) runs from $V_{L(min)}$ to $V_{L(max)}$, and the band between the two ranges is not interpreted as either value. When the voltage falls in that band the circuit does not guarantee a value.\nThe upper limit of HIGH is the supply voltage applied to the circuit. The 74-series ICs7 used in the laboratory run with 5 V on pin 14, $V_{CC}$.\nPulses A pulse is a single excursion from one level to the other and back. The instant of going from LOW to HIGH is the rising edge (leading edge) and the instant of going from HIGH to LOW is the falling edge (trailing edge). A pulse that starts from LOW is positive-going and a pulse that starts from HIGH is negative-going.\nAn ideal pulse changes voltage instantaneously at both edges. A real pulse takes time at each edge, and the textbook describes its characteristics in five terms.\nTerm Definition Rise time $t_r$ Time to go from 10 % to 90 % of the amplitude Fall time $t_f$ Time to go from 90 % to 10 % of the amplitude Pulse width $t_W$ Time between the 50 % points of the rising and falling edges Overshoot, ringing Passing the target level right after an edge and oscillating back to it Droop Gradual sag of the voltage during the HIGH interval When the period is on the order of 1 s, $t_r$ and $t_f$ are negligible compared with the period. When the clock reaches the GHz range and the period is on the order of ns, $t_r$ takes up a substantial part of the period and the waveform moves away from a square wave toward a triangle8. The theory in the textbook assumes ideal pulses.\nPeriodic waveforms, frequency and duty cycle A periodic waveform repeats the same shape at a fixed time interval, and that interval is the period $T$. A nonperiodic waveform has no fixed repetition interval. The frequency $f$ is the number of repetitions in one second and its unit is Hz.\n$$ f = \\frac{1}{T}, \\qquad T = \\frac{1}{f} $$With $T = 0.1\\ \\mathrm{s}$ the frequency is $f = 10\\ \\mathrm{Hz}$, and with $f = 1\\ \\mathrm{GHz}$ the period is $T = 1\\ \\mathrm{ns}$9. The duty cycle is the fraction of one period during which the waveform is HIGH, that is the fraction taken by $t_W$.\n$$ \\text{duty cycle} = \\frac{t_W}{T} \\times 100\\ \\% $$The left side of the figure below is a periodic waveform made of ideal pulses with $T$ and $t_W$ marked. The right side is a single real pulse with the 10 % and 90 % lines used to measure $t_r$ and $t_f$.\nThe clock and timing diagrams The clock is a pulse waveform with a constant period, and it sets the instants at which the other signals in a circuit may change value. One clock period is called the bit time. A data waveform represents one bit per bit time and does not change value within a bit time10. In the laboratory the clock comes from a function generator11 (an instrument that outputs a waveform of the frequency and amplitude entered on it).\nA timing diagram draws several digital signals side by side on one time axis to show the state of each signal and the relative timing of their transitions. The figure below shows the clock and three inputs A, B and C over clock periods 1 through 8, with period 7, where all three inputs are HIGH, shaded.\nReading (A, B, C) in each clock period gives the table below. With C as the most significant bit, periods 1 through 7 count upward in binary from 1 to 7.\nPeriod 1 2 3 4 5 6 7 8 A 1 0 1 0 1 0 1 0 B 0 1 1 0 0 1 1 0 C 0 0 0 1 1 1 1 0 Each column of this table is an input combination for a logic gate. A gate produces a fixed output for each column, so the output waveform is updated with the same period as the clock12.\nSerial and parallel transfer Serial transfer sends bits one per bit time over a single data line. Sending 8 bits takes 8 bit times. Parallel transfer uses one line per bit and sends all bits at once in a single bit time13.\nBasic logic functions A logic gate is a circuit that produces a fixed output bit for each combination of input bits. NOT, AND and OR are the three basic functions, and every digital operation in later chapters is a combination of these three. The inputs are written $A$ and $B$ and the output $X$.\nFunction Operation Truth table ($A\\,B \\to X$) NOT Inverts the input $0 \\to 1$, $1 \\to 0$ AND 1 only when every input is 1 $00 \\to 0$, $01 \\to 0$, $10 \\to 0$, $11 \\to 1$ OR 1 when at least one input is 1 $00 \\to 0$, $01 \\to 1$, $10 \\to 1$, $11 \\to 1$ A truth table lists every possible input combination with its output, and with $n$ inputs it has $2^n$ rows. The small circle at the end of a symbol means inversion14. AND and OR differ only when the two inputs differ, and the function that outputs 1 exactly in that case is XOR15 (exclusive OR).\nLogic function blocks Section 1.4 of the textbook introduces blocks that group several gates into one function, described only by their inputs and outputs16.\nBlock Inputs Outputs Comparator Binary numbers $A$, $B$ Three lines $A \u003e B$, $A = B$, $A \u003c B$; only the line whose condition holds is HIGH Adder Binary numbers $A$, $B$ and carry input $C_{in}$ Sum $\\Sigma$ and carry output $C_{out}$ Encoder One HIGH line among many input lines The binary code of that input\u0026rsquo;s number Decoder A binary code The output pattern assigned to that code With $A = 2$ and $B = 5$ only the $A \u003c B$ line of the comparator is HIGH. The adder\u0026rsquo;s $C_{out}$ is the 1 produced when the sum overflows one digit, and $C_{in}$ is the 1 carried up from the digit below17. An encoder turns the one key pressed on a calculator keypad into a binary code for storage, and a decoder takes that code and selects which bars of a 7-segment display18 to light. With 10 keys, $2^3 = 8 \u003c 10 \\le 16 = 2^4$, so the encoder needs 4 output lines.\nBlocks that operate on the clock Registers and counters are circuits whose stored value changes each time a clock pulse arrives. Unlike the blocks of the previous section, whose result appears as soon as the inputs are applied, time order is involved19.\nBlock Operation Multiplexer (MUX) Connects the one input line chosen by a select signal to a single output line Demultiplexer (DEMUX) Routes the received data to the one output line chosen by a select signal Serial shift register On each clock pulse, loads a new bit into the first cell and moves the stored bits one cell over Parallel register Stores several bits at once on a single clock pulse Counter Advances to the next binary number on each incoming pulse Joining a MUX and a DEMUX with one line lets the three data streams A, B and C pass in turn during the intervals $\\Delta t_1$, $\\Delta t_2$ and $\\Delta t_3$, and the receiving side separates them in the same order onto D, E and F. The figure below shows 0101 entering a 4-bit serial shift register. Loading 4 bits takes 4 clock pulses, whereas a parallel register takes the same 4 bits in a single pulse.\nAs pulses 1, 2, 3, 4 and 5 arrive at a counter, its output changes to the binary codes for 1, 2, 3, 4 and 5. After 6 pulses the output is 110.\nThe tablet bottling system Figure 1-28 of the textbook connects all of the blocks above into a system that fills each bottle with a preset number of tablets and accumulates the total20.\nThe preset count is keyed in, the encoder converts it to a binary code, and register A stores it. Each tablet that drops produces one pulse from the sensor, and the counter advances by 1. The comparator compares the preset count in register A with the current count in the counter. When they match, the $A = B$ output goes HIGH. That HIGH goes to two places. It closes the valve and advances the conveyor, and at the same time it tells register B to store the new sum. The adder adds the counter value to the running total in register B, and register B stores the result as the new total. The decoder shows it on the display and the MUX sends it to a computer. When the next bottle is in place, a reset pulse returns the counter to zero. The counter itself has no upper limit. It stops at the preset count because the closed valve cuts off the sensor pulses. Without the reset pulse the counter would run past the preset count on the second bottle, $A = B$ would never go HIGH again, and the valve would never close. The connection that returns the output of register B to the adder input is feedback21, and filling three bottles with a preset count of 8 updates register B to 8, 16 and 24.\nPeriod, cycle, pulse and bit time Name Meaning Length Period $T$ Time for the waveform to repeat once $T$ Clock cycle One repetition of the clock waveform $T$ Bit time Time during which one data bit is held $T$ Pulse The HIGH interval within a cycle $t_W$ Frequency $f$ Number of cycles in one second $1/T$, in Hz The first three are the same interval under different names. Each cycle contains exactly one pulse, so the counts agree while the lengths differ, with $t_W \\le T$. A question about time is answered in seconds, a question about repetitions per second in Hz, and a question about a ratio with the duty cycle.\nExercise For a system with a 2.5 GHz clock and a 25 % duty cycle, find the following.\n(a) The clock period and the pulse width $t_W$ (b) The time to transfer 8 bits serially with this clock (c) The outputs of $\\mathrm{AND}(A, C)$ and $\\mathrm{OR}(B, C)$ in period 5 of the timing diagram above (d) The minimum number of output lines needed to encode a 16-key keypad (e) In the bottling system, the value of register B right after three bottles are filled with a preset count of 8, and the minimum number of bits needed to hold it\nSolution. (a) $T = 1/(2.5 \\times 10^9) = 0.4\\ \\mathrm{ns} = 400\\ \\mathrm{ps}$ and $t_W = 0.25\\,T = 100\\ \\mathrm{ps}$. (b) The bit time is $T$, so the transfer takes $8T = 3.2\\ \\mathrm{ns}$. (c) In period 5, $(A, B, C) = (1, 0, 1)$, so $\\mathrm{AND}(A, C) = 1$ and $\\mathrm{OR}(B, C) = 1$. (d) $2^4 = 16$, so 4 lines. (e) $8 + 8 + 8 = 24$, and $2^4 = 16 \\le 24 \u003c 32 = 2^5$, so 5 bits.\nOrientation [19:44], [20:19]. The instructor split the keywords of the syllabus into \u0026ldquo;concepts of numbers, logic and formulas\u0026rdquo; and \u0026ldquo;the circuits that implement them\u0026rdquo;. The course covers Chapters 1 through 9, with one midterm, one final, and about four homework sets drawn mainly from the exercises.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSlide numbers are the printed numbers at the bottom of the lecture slides \u0026ldquo;Ch. 1. Introductory Concepts\u0026rdquo; (Kwangeun Kim, School of Electronics and Electrical Engineering, Hongik University). Timestamps [MM:SS] refer to the transcript of the 2026-09-02 lecture recording, and those marked \u0026ldquo;orientation\u0026rdquo; refer to the transcript of the first class on 2026-09-01.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLecture [27:07]. The instructor described digital as \u0026ldquo;values distinguished as 0 and 1\u0026rdquo; and analog as \u0026ldquo;continuous values\u0026rdquo;, using the temperature over time sampled into digital form as the example.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nDigital-to-Analog Converter, a circuit that turns a digital code into an analog voltage. In the CD player block diagram on slide 5, the digital data passes through the DAC on its way to the speaker.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLecture [28:52]. The signal processing and communications courses treat this process as sampling and modulation.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLecture [31:25]. The charger receives AC and the battery stores DC, so a converter sits between them, and an inverter turns the DC back into AC to drive the motor.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOrientation [29:54], lecture [07:06]. The chip handed out at the orientation was an SN74HC86N, with pin 14 as $V_{CC}$ and pin 7 as GND. The power supply keeps 5 V on that pin.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLecture [34:53], [35:54]. Once CPU clocks reach the GHz range the period is in ns and the rise time is a value below 1 ns, so it can no longer be ignored. The instructor put it as \u0026ldquo;a digital signal becomes analog when the time axis is viewed finely enough\u0026rdquo; and called it an item that matters in actual implementation.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLecture [37:08]. The instructor explained period and frequency as reciprocals with the examples 0.1 s and 10 Hz, and 1 GHz and 1 ns.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSlide 11 marks \u0026ldquo;Bit time\u0026rdquo; across the width of one clock period. This is the definition for the case of one bit per clock period, which is how Chapter 1 of this course always treats it.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLecture [09:39], [25:25]. A function generator produces sine and square waves, and for a square wave the period, the ON time and the duty cycle are entered. Because the clock has a constant period, it is produced by the function generator in the laboratory.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLecture [24:41], [26:03], [38:26]. Values are read only where the clock coincides, and the output follows the same period. A, B and C are data that vary independently over time, and the clock cuts out the combination of the three values at fixed intervals.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSlide 13. The transcript contains no matching remark.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOrientation [43:09]. The instructor pointed out that the circle on a gate symbol is the NOT concept.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOrientation [31:53], [36:40]. The 74HC86 handed out is an IC containing four 2-input XOR gates, and the logic circuit laboratory verifies truth tables directly with AND, OR and XOR chips.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLecture [40:58]. The instructor noted that the blocks come one slide each after the list of gates.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSlide 19 reads $3 + 9 = 12$ as a sum digit of 2 and a carry of 1. Adding directly in binary gives $0011 + 1001 = 1100$, with each digit producing one bit of $\\Sigma$ and one carry bit. The rules of binary addition are covered in Chapter 2.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA component that shows one digit with seven bars. At [41:53] the instructor explained that choosing which of the seven bars to light produces the digit.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOrientation [24:08]. Latches, flip-flops, registers and counters were set apart as \u0026ldquo;circuits that respond to data changing over time and thereby affect the result\u0026rdquo;.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLecture [42:47], [43:09]. Individual blocks are never used one at a time and are grouped for a purpose. The instructor went on to define engineering as \u0026ldquo;developing something out of the need for it to be used somewhere\u0026rdquo; and added that devices carrying people, such as electric vehicles and aircraft, have safety requirements on top.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA connection that returns an output to its own input. From Chapter 7 on, latches, flip-flops and counters hold their values with this structure.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://bias92.github.io/en/posts/digital-logic-circuits/","summary":"Lecture notes for Digital Logic Circuits based on Floyd: analog and digital quantities, logic levels, pulses and periodic waveforms, the clock and timing diagrams, NOT, AND and OR, the comparator, adder, encoder and decoder, registers and counters, and the tablet bottling system that ties the blocks together.","title":"Digital Logic Circuits"},{"content":" Reference: Samir S. Soliman, Mandyam D. Srinath, Continuous and Discrete Signals and Systems, 2nd ed., Prentice Hall, 1998. Chapter 1, Representing Signals.\nThis course covers continuous-time signals with a single independent variable, time, and leaves discrete-time signals to the DSP1 course. The material below corresponds to Sections 1.1 through 1.3 of the textbook and to lecture slides2 2 through 9.\nDefinition of a signal A signal is a detectable physical quantity that carries information. The definition has three parts.\nIt is a physical quantity or a variable that represents one. The amplitude of a voice, the brightness of a screen pixel, and the current in a circuit all qualify. It is detectable3. The receiving side must be able to measure the quantity. It carries information. A quantity that can be detected but carries no information is not treated as a signal. Mathematically a signal is written as a function of one or more independent variables (the inputs of the function). The number of independent variables4 depends on the signal.\nSignal Notation Independent variables Voice, circuit current $i(t)$, voltage $v(t)$ $A(t)$ Time $t$ only Image $p(x, y)$ Two coordinates $x, y$ Electric field $E(x, y, z, t)$ Three spatial coordinates and time This course deals only with signals $x(t)$ whose single independent variable is time $t$. An image $p(x, y)$ belongs to an image processing course, and a video is $p(x, y, t)$ with time added.\nContinuous-time and discrete-time signals A continuous-time signal is a signal whose independent variable is defined over an entire interval of real numbers. In the definition, the word continuous modifies the independent variable, not the function value5. For a radio signal $v(t)$ the independent variable is time $t$, and for atmospheric pressure $p(h)$ it is altitude $h$. Both independent variables sweep the whole real line, so both are continuous-time signals. An independent variable other than time, such as altitude, falls in the same class as long as it is continuous.\nA discrete-time signal is a signal whose independent variable takes only discrete values. Taking the values of a continuous-time signal $x(t)$ only at $t = kT_s$ gives $x(kT_s)$. $T_s$6 is a fixed positive real number and $k$ is an integer ($0, \\pm 1, \\pm 2, \\dots$). With $T_s = 2$ the signal is defined only at $t = 0, \\pm 2, \\pm 4, \\dots$. In the figure below the cyan curve is $x(t)$, which has a value at every real $t$, and the magenta dots are $x(kT_s)$, which has values only at $t = kT_s$. To tell the two apart, look at where the points sit on the $t$ axis rather than at the height of the function values.\nContinuity of a function A signal is a function, so the continuity of a signal follows the continuity of a function. $x(t)$ is continuous at $t = t_1$ when two conditions hold in order.\nThe left-hand limit $x(t_1^-)$ and the right-hand limit $x(t_1^+)$ are equal. The limit at $t_1$ then exists. That limit equals the function value $x(t_1)$. A signal continuous at every $t$ is a continuous signal. A signal with a finite number of discontinuities is piecewise continuous7. A signal with infinitely many discontinuities is not piecewise continuous.\nValue at a discontinuity Ordinary mathematics leaves the function value at a discontinuity undefined. This textbook defines the value at a discontinuity $t_1$ as the average8 of the left-hand and right-hand limits.\n$$ x(t_1) = \\frac{1}{2}\\left[x(t_1^+) + x(t_1^-)\\right] $$The unit step function $u(t)$ is 0 for $t \u003c 0$ and 1 for $t \u003e 0$. At $t = 0$ the left-hand limit 0 and the right-hand limit 1 differ, so the limit does not exist, and the signal is piecewise continuous rather than continuous. Applying the average definition gives $u(0) = \\tfrac{1}{2}$. In the left panel of the figure below, the two open circles are the left-hand limit $a$ and the right-hand limit $b$, and the filled dot is the value $\\tfrac{a + b}{2}$ defined by the textbook. Applying the same rule in the right panel with $a = 0$ and $b = 1$ gives $u(0) = \\tfrac{1}{2}$.\nAs an example, $x(t) = 2u(t - 1) - u(t - 3)$ is 0 for $t \u003c 1$, 2 for $1 \u003c t \u003c 3$, and 1 for $t \u003e 3$. The discontinuities are at $t = 1, 3$, with $x(1) = \\tfrac{0 + 2}{2} = 1$ and $x(3) = \\tfrac{2 + 1}{2} = \\tfrac{3}{2}$. With two discontinuities the signal is piecewise continuous.\nRectangular pulse and pulse train The rectangular pulse $\\mathrm{rect}(t/\\tau)$ is 1 for $|t| \u003c \\tfrac{\\tau}{2}$ and 0 for $|t| \u003e \\tfrac{\\tau}{2}$.\n$$ \\mathrm{rect}(t/\\tau) = \\begin{cases} 1, \u0026 |t| \u003c \\tau/2 \\\\ 0, \u0026 |t| \u003e \\tau/2 \\end{cases} $$The height is fixed at 1, and the parameter $\\tau$9 sets the width of the base. The discontinuities are the two points $t = \\pm\\tfrac{\\tau}{2}$, so the pulse is piecewise continuous, and since $t$ is defined over the whole real line it is a continuous-time signal.\nShifting $\\mathrm{rect}(t/\\tau)$ in time by a fixed spacing and adding the copies gives a pulse train10. A train of pulses of width 1 repeated every 2 units is written as follows, with $n$ ranging over all integers.\n$$ x(t) = \\sum_{n=-\\infty}^{\\infty} \\mathrm{rect}(t - 2n) $$The discontinuities at $t = 2n \\pm \\tfrac{1}{2}$ are infinite in number, so the train is neither continuous nor piecewise continuous. Since $t$ is defined over the whole real line, it is a continuous-time signal. Cutting the train to a finite number of pulses leaves a finite number of discontinuities, which makes it piecewise continuous.\nContinuous versus continuous-time The two terms describe properties on different axes.\nTerm Axis Criterion continuous / piecewise continuous Vertical axis (function value $x$) Number of points where the limit conditions fail continuous-time / discrete-time Horizontal axis (independent variable $t$) Whether $t$ is defined over a whole real interval or only at $t = kT_s$ Signal Function-value property Independent-variable property $\\sin t$ continuous continuous-time Rectangular pulse, $u(t)$ piecewise continuous continuous-time Infinite pulse train Infinitely many discontinuities, not piecewise continuous continuous-time $x(kT_s)$ Function-value property not considered discrete-time However many discontinuities there are, a signal whose $t$ axis is the whole real line is a continuous-time signal. Every signal in this course is continuous-time, and the only question within that class is whether it is continuous or piecewise continuous.\nExercise 1 For $x(t) = 3\\,\\mathrm{rect}\\!\\left(\\frac{t - 4}{2}\\right)$, find the following.\n(a) The interval of $t$ where the value is 3 (b) The discontinuities (c) The value at each discontinuity under the textbook definition (d) All integers $k$ for which $x(kT_s)$ is nonzero when sampled with $T_s = 1$\nSolution. From the definition of $\\mathrm{rect}(t/\\tau)$, $\\tau = 2$ and $t$ is replaced by $t - 4$, so the value is 3 where $|t - 4| \u003c 1$, that is, for $3 \u003c t \u003c 5$. The discontinuities are at $t = 3, 5$. At both points the left-hand and right-hand limits are 0 and 3, so $x(3) = x(5) = \\tfrac{0 + 3}{2} = \\tfrac{3}{2}$. With $T_s = 1$ we have $t = k$, and the values at $k = 3, 4, 5$ are $\\tfrac{3}{2}, 3, \\tfrac{3}{2}$ respectively, with 0 at every other $k$.\nPeriodic signals and the fundamental period A periodic signal is a continuous-time signal for which some positive $T$ satisfies the following at every $t$.\n$$ x(t) = x(t + nT), \\qquad n = 1, 2, 3, \\dots $$Because the condition must hold for every $t$, the signal is assumed to exist from $-\\infty$ to $+\\infty$11. If $T$ is a period, then $2T$, $3T$, and $4T$ are periods as well. The smallest positive $T$ that satisfies the condition is the fundamental period12, written $T_0$. A signal with fundamental period 2 also has 4 and 6 as periods. A signal with no such $T$ is aperiodic.\nSinusoids A real-valued sinusoid is fixed by three parameters.\n$$ x(t) = A\\sin(\\omega_0 t + \\phi) $$$A$ is the amplitude, and $x(t)$ stays between $-A$ and $A$. $\\omega_0$ is the radian frequency in rad/s. $\\phi$ is the initial phase13 in rad. The relation between radian frequency and frequency $f_0$ in Hz14, and the fundamental period, are as follows.\n$$ \\omega_0 = 2\\pi f_0, \\qquad T_0 = \\frac{1}{f_0} = \\frac{2\\pi}{\\omega_0} $$In the figure below, $A$ sets the vertical range, $T_0$ is the horizontal distance between two points of equal phase, and $\\phi$ sets the starting height $A\\sin\\phi$ at $t = 0$.\nHarmonics For a fundamental radian frequency $\\omega_0$, the $k$th harmonic15 is the sinusoid whose radian frequency is $k\\omega_0$. $k$ is the harmonic number and starts at 1.\n$$ \\omega_k = k\\,\\omega_0, \\qquad f_k = k f_0, \\qquad T_k = \\frac{2\\pi}{k\\,\\omega_0} = \\frac{T_0}{k} $$With $\\omega_0 = 2\\pi$ the first, second, and third harmonics have radian frequencies $2\\pi$, $4\\pi$, $6\\pi$ and periods $1$, $\\tfrac{1}{2}$, $\\tfrac{1}{3}$ s. In the figure below, a larger $k$ oscillates $k$ times within the same $T_0$, and all three curves return to their starting point together at $t = T_0$. This is why the fundamental period of any sum of harmonics is $T_0$.\nTo decide whether a sum of sinusoids is periodic and to find its fundamental period, look only at the radian frequencies. The fundamental radian frequency of the sum is the greatest common divisor16 of the radian frequencies of the terms, and the harmonic number of each term is its radian frequency divided by $\\omega_0$.\n$$ x(t) = \\cos(4\\pi t) + \\sin(6\\pi t): \\quad \\omega_0 = 2\\pi, \\quad T_0 = 1 \\text{ s}, \\quad k = 2, 3 $$If the ratio of the radian frequencies is irrational, no greatest common divisor exists and the sum is aperiodic.\nHarmonically related complex exponentials Writing the harmonics as complex exponentials17 instead of sinusoids gives the following set.\n$$ \\phi_k(t) = e^{\\,jk\\omega_0 t}, \\qquad k = 0, \\pm 1, \\pm 2, \\dots $$For $k \\neq 0$ the signal is periodic with radian frequency $|k|\\omega_0$ and fundamental period $\\tfrac{2\\pi}{|k|\\omega_0}$. The absolute value keeps the frequency and period positive when $k$ is negative, and for positive $k$ the expression matches the harmonic formulas. Every $\\phi_k(t)$ has $T_0 = \\tfrac{2\\pi}{\\omega_0}$ as a common period. For $k = 0$18, $\\phi_0(t) = 1$ is a constant and no period is defined.\nEuler\u0026rsquo;s formula splits the exponential into real and imaginary parts, which places it as a point on the complex plane (real part horizontal, imaginary part vertical).\n$$ e^{\\,j\\omega_0 t} = \\cos\\omega_0 t + j\\sin\\omega_0 t $$The point at time $t$ is $(\\cos\\omega_0 t, \\sin\\omega_0 t)$ with magnitude $\\sqrt{\\cos^2 + \\sin^2} = 1$. It starts at $(1, 0)$ when $t = 0$, moves counterclockwise as $t$ increases, and returns to the start when $\\omega_0 t = 2\\pi$. Since it lies on the unit circle of radius 1 at every $t$, in polar form it has magnitude 1 and angle $\\omega_0 t$.\nExercise 2 For $x(t) = 2\\cos(6\\pi t) + \\sin(9\\pi t)$, find the following.\n(a) The fundamental radian frequency $\\omega_0$ and fundamental period $T_0$ (b) The harmonic number of each term (c) The radian frequency $\\omega_1$ and period $T_1$ of the first harmonic\nSolution. The greatest common divisor of $6\\pi$ and $9\\pi$ is $3\\pi$, so $\\omega_0 = 3\\pi$ rad/s and $T_0 = \\tfrac{2\\pi}{3\\pi} = \\tfrac{2}{3}$ s. Since $\\tfrac{6\\pi}{3\\pi} = 2$ and $\\tfrac{9\\pi}{3\\pi} = 3$, $2\\cos(6\\pi t)$ is the second harmonic and $\\sin(9\\pi t)$ is the third. The first harmonic has $\\omega_1 = \\omega_0 = 3\\pi$ rad/s and $T_1 = T_0 = \\tfrac{2}{3}$ s.\nDigital Signal Processing, the follow-on course that covers sampling of discrete-time signals, the discrete Fourier transform, and the z-transform.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSlide numbers are the printed numbers in the top-right corner of the lecture slides \u0026ldquo;1. Representing Signals\u0026rdquo; (Seung-Chan Lim, School of Electronics and Electrical Engineering, Hongik University).\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nDetectable means that a measuring device or a sense organ can read the quantity. A voice is detected by ears and microphones, screen brightness by eyes and cameras, and current by an ammeter.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe number of independent variables is the dimension of the signal: one (time) for a voice, two coordinates for an image, two coordinates plus time for a video, and three spatial coordinates plus time for an electric field. This count is why electromagnetics requires vector calculus and several coordinate systems.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe criterion is the horizontal axis. No matter how often the function value breaks on the vertical axis, the signal is continuous-time when the domain is the whole real line and discrete-time when the domain is a set of isolated points such as $kT_s$.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n$T_s$ is the sampling period, and its reciprocal $f_s = \\tfrac{1}{T_s}$ is the sampling frequency, the number of samples per second. The textbook describes $T_s$ only as \u0026ldquo;a fixed positive real number\u0026rdquo;.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe textbook reads \u0026ldquo;piecewise continuous if it has only finite discontinuities\u0026rdquo;. This course reads finite as the number of discontinuities. In the mathematical literature the phrase is also used to mean that each jump is finite in size, and under that reading the infinite pulse train is piecewise continuous as well.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOrdinary analysis leaves the value at a discontinuity undefined. Fixing it as the average of the two one-sided limits matches the value to which a Fourier series converges at a jump discontinuity, so that in later chapters the series and the original signal agree at every point.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWith $\\tau = 10^8$ the base runs from $-5 \\times 10^7$ to $5 \\times 10^7$, with $\\tau = 10^{-7}$ it runs over $\\pm 5 \\times 10^{-8}$, and the height is 1 in both cases. Sending the width to zero while raising the height to $\\tfrac{1}{\\tau}$ so that the area stays 1 gives the unit impulse of slide 26.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe train on slide 5 repeats a pulse of width 1 every 2 units and is discontinuous at $t = 0, \\pm 1, \\pm 2, \\dots$. The drawing with six pulses has 12 discontinuities and is piecewise continuous, and the infinitely extended signal has infinitely many and is not.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA graph drawn over a finite interval cannot settle whether a signal is periodic. Only with the assumption that the same shape continues outside the graph can $x(t) = x(t + T)$ be stated for every $t$.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe reciprocal $f_0 = \\tfrac{1}{T_0}$ is the fundamental frequency and $\\omega_0 = \\tfrac{2\\pi}{T_0}$ is the fundamental radian frequency. The subscript 0 marks \u0026ldquo;fundamental\u0026rdquo;, not a harmonic number. The period $T_1$ of the first harmonic equals $T_0$.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n$\\phi$ shifts the curve along the time axis by $-\\tfrac{\\phi}{\\omega_0}$. For $\\phi \u003e 0$ the curve is shifted left, so at $t = 0$ it has already risen to $A\\sin\\phi$.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nHz is 1/s, the number of repetitions per second. The radian frequency $\\omega_0$ is the angle traversed per second in rad/s, so dividing by one full turn of $2\\pi$ rad gives the number of turns per second, $f_0$. The only difference between the two units is the factor $2\\pi$.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nInteger multiples are used because of the common period. $T_k = \\tfrac{T_0}{k}$ divides $T_0$, so any sum of harmonics has $T_0$ as a period. A non-integer multiple such as 2.5 does not share $T_0$ as a period. The Fourier series of Chapter 3 writes a periodic signal as a weighted sum of these harmonics.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWhen the radian frequencies carry a factor of $\\pi$, remove it, take the greatest common divisor of the integers, and put $\\pi$ back. For $4\\pi$ and $6\\pi$ the greatest common divisor of 4 and 6 is 2, giving $2\\pi$. Working with periods instead gives the least common multiple of the two periods, with the same result.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA sinusoid is real-valued and a complex exponential is complex-valued. One complex exponential carries both $\\cos$ and $\\sin$ as its real and imaginary parts, which is why the Fourier series of Chapter 3 uses $e^{jk\\omega_0 t}$ as its terms instead of $\\cos$ and $\\sin$. Negative $k$ is allowed so that $e^{jk\\omega_0 t}$ and $e^{-jk\\omega_0 t}$ can be added to form a real $\\cos$.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWith $k = 0$ the exponent is 0 and $e^0 = 1$. A constant satisfies $x(t) = x(t + T)$ for every $T$, so no smallest positive period can be chosen, and the textbook applies the periodicity statement only to $k \\neq 0$.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://bias92.github.io/en/posts/signals-and-systems/","summary":"Lecture notes for Signals and Systems based on Soliman \u0026amp; Srinath: the definition of a signal, continuity and the value at a discontinuity, the rectangular pulse, continuous-time versus discrete-time, periodic signals and sinusoids, harmonics, and harmonically related complex exponentials.","title":"Signals and Systems"},{"content":"LICM (loop-invariant code motion) is an LLVM optimization pass that moves a loop-invariant instruction out of its loop. A loop-invariant instruction produces the same result on every iteration. LLVM applies LICM only when it can prove that program behavior remains unchanged.\nMoving an instruction to a point before its loop is called hoisting. In the following transform function, the multiplication and addition that produce factor are hoisting candidates.\nThe Same Computation on Every Iteration unsigned is a C integer type that represents nonnegative values. input and output each point to count elements of that type, and count is nonnegative. When count is positive, the loop variable i ranges from 0 through count - 1. scale and offset are function arguments, and factor is the result of scale * scale + offset.\n// licm.c void transform(const unsigned *input, unsigned *output, int count, unsigned scale, unsigned offset) { for (int i = 0; i \u0026lt; count; ++i) { unsigned factor = scale * scale + offset; output[i] = input[i] * factor; } } An operand is a value consumed by an instruction. Neither scale nor offset changes inside the loop. The expressions scale * scale and scale * scale + offset are therefore loop invariant.\ni increases on every iteration. The addresses of input[i] and output[i] change with it. In the unoptimized IR, the factor computation sits in the loop body and repeats once for every executed iteration.\nIR Immediately Before LICM clang is the frontend that translates C to LLVM IR, and opt is the optimizer that runs passes on IR. Both tools come from Homebrew LLVM 22.1.81. In the commands, LLVM_BIN is the directory containing those LLVM executables. -fno-discard-value-names preserves C identifiers in IR names, and -Xclang passes the following option to clang\u0026rsquo;s internal frontend. -disable-O0-optnone suppresses the optnone attribute.\nLocal names in LLVM IR follow SSA (static single assignment) form. SSA defines each virtual register once. A phi instruction selects a value according to the preceding basic block from which control arrived. A basic block is a sequence of IR instructions that executes from top to bottom with no branch in the middle.\nThe header is the first basic block reached when control enters a loop from outside. The preheader is the single entry block outside the loop that leads only to the header. A backedge is control flow from inside the loop back to its header, and the latch is the block from which the backedge starts. An exit block is reached after control leaves the loop.\nBefore LICM, the IR passes through mem2reg, loop-simplify, and lcssa.\nPass Role mem2reg promotes eligible local variables created by alloca into SSA value flow loop-simplify ensures Loop Simplify Form, with a preheader, one backedge, and dedicated exit blocks reached only from inside the loop lcssa (loop-closed SSA) routes values defined inside the loop and used outside it through phi nodes in exit blocks A loop pass processes one loop at a time. LLVM automatically runs loop-simplify and lcssa before a loop pass. raw.ll is the IR emitted by clang, and before.ll is the IR after the three preparation passes. The preparation makes before.ll readable and leaves only LICM\u0026rsquo;s changes in the later comparison.\nLLVM_BIN=/opt/homebrew/opt/llvm/bin \u0026#34;$LLVM_BIN/clang\u0026#34; -O0 -Xclang -disable-O0-optnone \\ -fno-discard-value-names \\ -S -emit-llvm licm.c -o raw.ll \u0026#34;$LLVM_BIN/opt\u0026#34; -S \\ -passes=\u0026#39;mem2reg,loop-simplify,lcssa\u0026#39; \\ raw.ll -o before.ll transform uses no loop-defined result outside the loop, so lcssa adds no new phi node to before.ll.\nWith file-level settings, parameter and function attributes, and metadata removed, transform reads as follows.\nThe target triple in before.ll is arm64-apple-macosx15.0.0. On arm64-apple-macosx15.0.0, pointers are 64 bits and both int and unsigned are 32 bits.\ndefine void @transform(ptr %input, ptr %output, i32 %count, i32 %scale, i32 %offset) { entry: br label %for.cond for.cond: %i.0 = phi i32 [ 0, %entry ], [ %inc, %for.inc ] %cmp = icmp slt i32 %i.0, %count br i1 %cmp, label %for.body, label %for.end for.body: %mul = mul i32 %scale, %scale %add = add i32 %mul, %offset %idxprom = sext i32 %i.0 to i64 %arrayidx = getelementptr inbounds i32, ptr %input, i64 %idxprom %0 = load i32, ptr %arrayidx, align 4 %mul1 = mul i32 %0, %add %idxprom2 = sext i32 %i.0 to i64 %arrayidx3 = getelementptr inbounds i32, ptr %output, i64 %idxprom2 store i32 %mul1, ptr %arrayidx3, align 4 br label %for.inc for.inc: %inc = add nsw i32 %i.0, 1 br label %for.cond for.end: ret void } The five basic blocks have the following roles.\nBasic block Role entry function entry and preheader, the only block outside the loop that enters for.cond for.cond header, checks i \u0026lt; count and branches to for.body or for.end for.body loads an array element, multiplies it by factor, and stores the result for.inc latch, increments i and follows the backedge to for.cond for.end exit block reached after control leaves the loop The IR instructions split the C loop as follows.\nIR Meaning ptr, i1, i32, i64 pointer, one-bit integer, 32-bit integer, and 64-bit integer types, respectively br label, br i1 branch unconditionally to another basic block or branch according to an i1 condition %i.0 = phi ... select 0 on the first iteration and %inc on later iterations %cmp = icmp slt ... compute i \u0026lt; count with a signed less-than comparison %mul, %add, %mul1 compute scale * scale, %mul + offset, and input[i] * factor, respectively %idxprom, %idxprom2 sign-extend i from i32 to i64 with sext for the input and output addresses, respectively %arrayidx, %arrayidx3 compute the addresses of input[i] and output[i] with getelementptr inbounds; inbounds promises that each address stays within the permitted range of the same allocated object %0 = load, store, align 4 load input[i] from %arrayidx into %0 and store %mul1 through %arrayidx3; the addresses of 32-bit unsigned elements are aligned to a four-byte boundary %inc = add nsw ... compute i + 1; nsw promises no signed overflow, which follows from nonnegative count and i \u0026lt; count Running LICM -passes='licm' selects LICM as the transforming pass. LLVM 22 opt automatically prepares Loop Simplify Form, LCSSA, and MemorySSA2. MemorySSA records use-def relations from each memory use to its reaching definition and which writes may change a loaded value.\nafter.ll is the result of running LICM on before.ll. diff -u compares two text files line by line. -I '^; ModuleID' excludes the ModuleID comment, which contains only the input filename.\n\u0026#34;$LLVM_BIN/opt\u0026#34; -S \\ -passes=\u0026#39;licm\u0026#39; \\ before.ll -o after.ll diff -u -I \u0026#39;^; ModuleID\u0026#39; before.ll after.ll The function body changes by four lines. A - marks a line removed from before.ll, and a + marks a line added to after.ll.\nentry: + %mul = mul i32 %scale, %scale + %add = add i32 %mul, %offset br label %for.cond for.body: - %mul = mul i32 %scale, %scale - %add = add i32 %mul, %offset %idxprom = sext i32 %i.0 to i64 %mul and %add move from for.body to entry. Every path from function entry to for.body passes through entry. This relation is called dominance: entry dominates for.body. The moved definition of %add therefore executes before %mul1 uses %add.\nWhen count is zero, for.body does not execute, but the hoisted %mul and %add execute once in entry. Both instructions keep the low 32 bits of their results and neither reads nor writes memory. LLVM IR defines their result for every scale and offset, so speculative execution (running them before they are known to be needed) is safe.\nThis LICM path checks three conditions before moving an instruction unchanged into the preheader.\nEvery operand of the instruction must be loop invariant. Memory reads, memory writes, calls, and other externally visible behavior must remain unchanged. Speculative execution on paths that did not execute the original instruction must be safe. %mul and %add satisfy all three conditions.\nWhy an Invariant Division Stays in the Loop transform_div multiplies each array element by numerator / denominator. numerator is the dividend, and denominator is the divisor. input and output each point to count unsigned elements, and count is nonnegative. When count is positive, the range of i is 0 through count - 1.\n// licm_div.c void transform_div(const unsigned *input, unsigned *output, int count, unsigned numerator, unsigned denominator) { for (int i = 0; i \u0026lt; count; ++i) { unsigned factor = numerator / denominator; output[i] = input[i] * factor; } } transform_div goes through mem2reg, loop-simplify, lcssa, and licm in that order. %div remains in for.body after LICM. udiv is the IR instruction that divides two unsigned integers and defines the quotient as %div.\nfor.body: %div = udiv i32 %numerator, %denominator ; instructions that load input[i], multiply by %div, and store to output[i] When count is zero, for.body does not execute. The original function performs no division on that path even when denominator is zero. Moving %div into entry could introduce a division by zero. LLVM IR defines division by zero as undefined behavior, an execution for which LLVM specifies no result or subsequent behavior. Both operands are invariant, but speculative execution is unsafe, so LICM leaves %div in place.\nMemory instructions need proofs beyond the address itself. Aliasing is the possibility that different pointers refer to the same memory location. Even a load from a loop-invariant address can change when a store or call in the loop modifies an aliased location. LLVM uses alias analysis and MemorySSA to find conflicting writes, then separately checks whether reading the address is safe on a path where the loop executes zero times.\nA volatile access is a memory access that the compiler may not omit or merge. It cannot be hoisted because its execution count is externally observable. For an ordinary call, call semantics, attributes, and analyses must first prove that its memory effects and other side effects permit movement. LICM then checks the remaining conditions.\nReferences LLVM\u0026rsquo;s Analysis and Transform Passes: LICM: LICM behavior and conditions for moving memory instructions. LLVM Loop Terminology: definitions of headers, preheaders, latches, exits, and Loop Simplify Form. Using the New Pass Manager: opt -passes syntax and automatically selected pass scope. MemorySSA: memory use-def relations and conflict queries using alias analysis. LLVM Language Reference: udiv: unsigned division and the semantics of a zero divisor. LLVM 22.1.0 LICM implementation: the implemented hoisting checks and use of MemorySSA. The commands and IR use Homebrew LLVM 22.1.8 from /opt/homebrew/opt/llvm/bin. IR spelling and pass-manager structure can change in other LLVM versions.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOn LLVM 22.1.8, -passes='licm' -print-pipeline-passes prints function(loop-mssa(licm\u0026lt;allowspeculation\u0026gt;)),verify. loop-mssa provides MemorySSA to LICM, and function(...) applies the loop pass to each function in a module. A module is the contents of one IR file. allowspeculation permits speculative execution after the safety checks pass, while verify checks the structure of the transformed IR.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://bias92.github.io/en/posts/llvm-1-licm/","summary":"LLVM LICM moves an invariant multiplication and addition from a loop body into its preheader. A C-to-IR diff shows how invariant operands, memory effects, and the zero-trip path determine whether hoisting is legal.","title":"01 Loop-Invariant Code Motion: From Loop Body to Preheader"},{"content":"A compiler is a program that translates the source code humans read and write into machine code so the CPU can run it. A CPU can natively execute only machine code, so code written in a language like C must go through this translation once before it runs.\nDepending on when the translation happens, the ways to run a program split broadly into three.\nMethod When it translates What persists between runs Examples1 Interpreter never; re-interprets the source on every run nothing CPython JIT (just-in-time) during execution, only for hot code machine code kept in memory Java JVM, V8 in browsers AOT (ahead-of-time) everything, before the program runs an executable file clang, GCC The difference between the three methods shows up as time once the same function is called repeatedly. The graph below shows per-call times for the same integer loop, called twelve times each as CPython (interpreter), numba2 (JIT), and a C library compiled ahead of time with clang -O2 (AOT).\nThe interpreter repeats the same interpretation on every call and pays the same cost each time. The JIT pays its compilation cost on the first call (warmup3) and reuses the translated machine code afterwards, while the AOT build pays that cost before the program runs and is fast from the first call.\nLLVM4 is an AOT compiler for native languages (languages that compile to machine code the CPU executes directly, with no virtual machine5) like C and C++.\nThe structure of stepping down through an intermediate language instead of translating source straight to machine code already appeared with nvcc in CUDA C Basics. nvcc, NVIDIA\u0026rsquo;s CUDA compiler, splits a .cu file into host code that runs on the CPU and device code that runs on the GPU, lowering the device code to PTX6, an intermediate instruction set, and then to SASS, the GPU\u0026rsquo;s machine code. CPU-side compilers step down through intermediate stages the same way, and LLVM is the representative one. In fact cicc, the device code compiler inside nvcc, is built on LLVM, so the two stacks correspond layer by layer.\nThe Pipeline Frontend (clang): Test.c → Test.ll. Parsing7 and type checking. Middle-end (opt): Test.ll → Test.ll'. Passes rewriting the IR, which is optimization. Backend (llc, as, ld): Test.ll' → Test.s → Test.o → a.out. Code generation and assembly. A single IR (intermediate representation, the language between source and machine code) travels through all three stages: frontend, middle-end, and backend. Other compilers hold their IR only as in-memory objects, so it cannot be pulled out into a file. LLVM IR is text with a published grammar, so you can save it to a file, edit it by hand, and feed it back to the compiler.\nFrom C to IR The example is a two-function C file.\n// Test.c int func1(void) { int a = 4; return a; } int main(void) { return 0; } clang -emit-llvm -S Test.c # → Test.ll -emit-llvm -S tells clang to stop at human-readable IR instead of going all the way to machine code. func1 in the resulting Test.ll reads as follows.\ndefine i32 @func1() #0 { %1 = alloca i32, align 4 store i32 4, ptr %1, align 4 %2 = load i32, ptr %1, align 4 ret i32 %2 } IR Meaning C counterpart define i32 @func1() #0 define function func1 returning i32 (32-bit int); #0 refers to an attribute group int func1(void) %1 = alloca i32, align 4 reserve one i32 slot on the stack8; its address is named %19 the slot for int a store i32 4, ptr %1 store 4 at that address a = 4 %2 = load i32, ptr %1 load from that address into %2 reading a in return a ret i32 %2 return %2 return These are the 4 symbols that appear in the body.\nSymbol Meaning i32 32-bit integer type. i1, i8, i64 also exist %name local name. A virtual register10, so there is no limit on how many @name global name. Functions live here ; comment The target triple11 at the top, the attributes12 at the bottom, and the ! metadata are environment configuration, not needed for decoding the body.13\nFrom IR to Assembly llc Test.ll -o Test.s llc is the backend tool that lowers IR to the assembly14 of the CPU named by the target triple11, so the output assembly differs per target CPU: targeting x86-64 produces x86-64 assembly, ARM64 produces ARM64 assembly, RISC-V produces RISC-V assembly, and an option such as llc -mtriple=x86_64-pc-linux-gnu Test.ll switches the target. One IR fanning out to per-target backends is exactly the role of IR described in the pipeline section.\nThe following table maps func1 in the ARM64-target output.15\nTest.ll (virtual) Test.s (ARM, physical) What happens %1 = alloca i32 sub sp, sp, #16 reserve the stack frame; %1 becomes the slot at sp+12 store i32 4, ptr %1 mov w8, #4 → str w8, [sp, #12] put 4 in a register, store it to the stack %2 = load i32, ptr %1 ldr w0, [sp, #12] load from the stack into w0; %2 becomes w0 ret i32 %2 add sp, sp, #16 → ret release the frame and return; w0 carries the return value The backend assigns virtual names (%N) to physical places (registers, stack slots). An assembly file has 3 kinds of lines.\nLine shape What it is When reading starts with . assembler directive16 skip name: label17 position marker indented CPU instruction what to read Change store i32 4 to store i32 9 in Test.ll, run llc again, and the output shows mov w8, #9. The IR text itself is the compiler input, so editing the IR alone changes the program without going through the frontend.\nFrom Assembly to an Executable Test.s is a text file holding instructions like mov w8, #4 as characters. A CPU, however, is a circuit that acts only when the fixed bit pattern of an instruction arrives, so it cannot execute character data like \u0026rsquo;m\u0026rsquo;, \u0026lsquo;o\u0026rsquo;, \u0026lsquo;v\u0026rsquo; as instructions. The assembler (as) therefore turns the text notation into instruction bit patterns, and the result is the object file Test.o.\nA program is usually built from several source files, each becoming one object file, plus libraries (bundles of precompiled object files holding common functions such as printf). An object file is machine code but not runnable on its own, because the addresses of functions it calls in other object files or libraries are still unresolved. The linker (ld) collects those object files and libraries, fills in the missing addresses, and joins them into an executable. This process is called linking.\nclang -c Test.s -o Test.o # assemble clang Test.o -o a.out # link Both steps can be invoked through the clang command, which calls as and ld internally.\nPasses and Optimization Levels A pass is a small program built into LLVM that sweeps over the entire IR once, performing one predetermined analysis or transformation. Each pass owns exactly one transformation.\npass The one thing it does mem2reg replaces the memory round trip of local variables (alloca, store, load) with direct value flow instcombine rewrites instruction combinations into shorter ones with the same result simplifycfg deletes unreachable blocks18 and simplifies branches dce deletes instructions that do not affect the result (dead code elimination) licm moves computations whose value is the same on every iteration out of the loop (loop invariant code motion) Running passes splits into two ways. With opt you pick and run a single pass of your choosing, and with clang\u0026rsquo;s -O1 -O2 -O3 options a predefined list of passes runs in order. And the three options\u0026rsquo; lists nest inside one another.19\n$$ O_0(0) \\subset O_1(98) \\subset O_2(115) \\subset O_3(118) $$The price of a higher level is compile time, and the passes -O3 adds trade code size for speed, which is why release builds usually stop at -O2.\nThe following command prints the actual contents of a list. Every pass in the table above is inside it.\nopt -passes=\u0026#39;default\u0026lt;O1\u0026gt;\u0026#39; -print-pipeline-passes Test.ll -S -o /dev/null Watching Optimization as a Diff The payoff of readable IR shows up when comparing before and after optimization.\nclang -O1 -emit-llvm -S Test.c -o Test_O1.ll Comparing Test.ll emitted at -O0 (the default, left) with Test_O1.ll emitted at -O1 (right) in the VS Code diff editor20 gives the following.\nMapping the body of func1 line by line:\nLeft, -O0 (before) Right, -O1 (after) Change %1 = alloca i32 gone the stack slot for variable a is removed store i32 4, ptr %1 gone the instruction writing 4 to memory is removed %2 = load i32, ptr %1 gone the instruction reading it back from memory is removed ret i32 %2 ret i32 4 the constant 4 is returned directly instead of a value read from the variable The optimization passes proved that this function stores 4 to memory and immediately loads it back, so the answer is always 4, and erased the variable\u0026rsquo;s existence. The C code did not change, the program went from four lines to one, and the whole event is captured in a text diff.\noptnone optnone is a do-not-optimize marker that clang attaches to the attributes12 of every function when it emits IR at -O0. The marker shows itself as follows when experimenting with passes.\nApply mem2reg from the previous section on its own with opt.\nopt -passes=mem2reg Test.ll -S -o Test_m2r.ll The output comes back identical to the input. Before processing a function, opt reads its attributes and skips the function when it sees optnone. Attributes control whether passes run at all.\nEmitting without the marker restores normal behavior.21\nclang -Xclang -disable-O0-optnone -emit-llvm -S Test.c -o Test_noopt.ll opt -passes=mem2reg Test_noopt.ll -S func1 folds to a single ret i32 4. This option is the standard way to emit IR for pass experiments.\nReferences LLVM for Grad Students: the What is LLVM? / The Pieces / Understanding LLVM IR chapters. LLVM Language Reference: the official definition of IR syntax. The Architecture of Open Source Applications: LLVM: Chris Lattner on the design background of LLVM. CPython is the standard Python interpreter, the Java JVM runs Java, V8 is the JavaScript engine in Chrome, and GCC is another C/C++ compiler in the same role as clang.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA JIT library that compiles Python functions to machine code during execution.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe name for the slow early calls in which a JIT pays its translation cost.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLLVM began as an acronym for Low Level Virtual Machine, but the project has long outgrown virtual machines and the name now stands on its own.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA program that executes intermediate code instead of machine code; the Java JVM is the canonical example.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nShort for Parallel Thread Execution.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBreaking source characters down by grammar rules into a structured tree.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe stack is the memory region where a function\u0026rsquo;s local data accumulates; it grows on function entry and shrinks on return.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nalign 4 tells the compiler to place the address at a multiple of four bytes.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA register is one of a CPU\u0026rsquo;s few fast internal storage slots, fixed in number. A virtual register is a name the IR can mint without limit; the backend assigns it to a physical register.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe target triple is the target specification written at the top of the IR file, made of three parts as the name says (CPU-vendor-OS). Example: arm64-apple-macosx15.0.0.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nattributes is the list collecting the properties applied to a function: the attributes #0 = {...} line near the bottom of the IR file, which the #0 on a function definition points to.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOlder LLVM (before 15) wrote typed pointers like i32* instead of ptr. LLVM 15 unified this as opaque pointers (a notation that does not spell out the pointed-to type); different generation of syntax, same meaning.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAssembly is a notation that writes CPU instructions as human-readable text. Its relation to machine-code bits is covered in the next section.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIn the table, sp is the register holding the current top address of the stack (the stack pointer); w8 and w0 are ARM general-purpose registers, and w0 carries the return value.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA directive is an instruction to the assembler, not a CPU instruction.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA label is a name attached to a position in the code; branches and calls refer to positions by these names.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA block (basic block) is a run of instructions executed strictly top to bottom with no branches inside.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe numbers in parentheses count the passes in the printed pipeline of LLVM 22 and vary by version.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA view showing the differences between two files side by side: deleted lines in red on the left, added lines in green on the right.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIn the command, -Xclang makes clang pass the following option straight to its internal frontend, and -disable-O0-optnone is the option that suppresses the marker.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://bias92.github.io/en/posts/llvm-0-ir-pipeline/","summary":"Starting from the difference between interpreters, JITs, and AOT compilers, emit LLVM IR from C with clang, decode it, and follow the compilation pipeline down through assembly to an executable. Observe how optimization passes rewrite the IR via an -O0 vs -O1 diff, and record the optnone attribute that makes opt silently do nothing.","title":"00 LLVM IR and the Compilation Pipeline: From C Code to Machine Code"},{"content":" Source: 07 Concurrency\nA CUDA program uses a CPU and a GPU together. The CPU side is called the host and the GPU side the device; host memory is the system RAM the CPU uses, and device memory is the memory attached to the GPU. The data to compute on starts out in host memory, so for the GPU to process it the data has to be moved into device memory. The basic flow of a CUDA program is therefore three steps: copy the input from host memory to device memory, run the computation on the GPU, and bring the result back to host memory. The total time differs between finishing these three steps one after another and running steps that belong to different data in the same time window, and the mechanism that decides that arrangement is the stream.\nHost Memory and Device Memory Allocation is the act of reserving a memory region for the program to use and receiving its starting address as a pointer. A pointer is a variable that holds a memory address. Host memory is allocated with malloc and released with free. Device memory is then allocated with cudaMalloc and released with cudaFree, and the pointer returned points to a region the GPU accesses. Because the two pointers point to different memories, a value the CPU wrote into the malloc region has to be copied before the GPU can read it. In the code below, N is the number of float elements, bytes is their total size in bytes, and size_t is an integer type used for memory sizes. h_x is the input the CPU fills and h_y is the host memory that receives the result, while d_x and d_y are device memory of the same size. These four pointers keep the same meaning throughout this post.\nconst size_t N = 1000; const size_t bytes = N * sizeof(float); float *h_x = (float *)malloc(bytes); // host input float *h_y = (float *)malloc(bytes); // host output float *d_x = nullptr; float *d_y = nullptr; cudaMalloc(\u0026amp;d_x, bytes); // device input cudaMalloc(\u0026amp;d_y, bytes); // device output H2D Copy, Kernel Launch, D2H Copy A copy from host memory to device memory is an H2D (Host to Device) copy, and the opposite direction is a D2H (Device to Host) copy. cudaMemcpy is the function that performs this copy, with the direction given in its last argument. Next, the function to run on the GPU, the kernel, is launched with the \u0026lt;\u0026lt;\u0026lt;grid, block\u0026gt;\u0026gt;\u0026gt; syntax, and this call is the kernel launch. Here a thread is the GPU\u0026rsquo;s unit of work that executes the kernel, a block is a bundle of threads placed together, and grid is the number of blocks.\nThis post uses a single kernel throughout. transform multiplies each element of the input x by two and writes it to the same index of the output y.\n__global__ void transform(const float *x, float *y, size_t count) { const size_t i = blockIdx.x * blockDim.x + threadIdx.x; if (i \u0026lt; count) { y[i] = x[i] * 2.0f; } } __global__ marks the function as a kernel that runs on the GPU. blockIdx.x is this block\u0026rsquo;s index in the grid, blockDim.x is the number of threads in one block, and threadIdx.x is this thread\u0026rsquo;s index inside the block. The i combined from those three is the element index this thread handles, and indices at or beyond count are skipped.\nThe block passed to a kernel launch is the number of threads in one block and grid is how many such blocks are needed. Threads execute in groups of 32 on the GPU, so the block size is a multiple of 32; this post uses 256. One thread handles one element, so processing N elements needs N / 256 blocks, rounded up when the division is not exact. Once the kernel has written its result into device memory, a D2H copy brings the result back to host memory.\nconst int block = 256; const int grid = (N + block - 1) / block; cudaMemcpy(d_x, h_x, bytes, cudaMemcpyHostToDevice); // H2D copy transform\u0026lt;\u0026lt;\u0026lt;grid, block\u0026gt;\u0026gt;\u0026gt;(d_x, d_y, N); // kernel launch cudaMemcpy(h_y, d_y, bytes, cudaMemcpyDeviceToHost); // D2H copy These three lines have a fixed order. The kernel must run after the input has arrived in device memory, and the D2H copy must start after the kernel has finished writing the result. So when one piece of data is processed whole, the H2D copy, kernel, and D2H copy line up one after another, and the total time is the sum of the H2D copy time $T_H$, the kernel time $T_K$, and the D2H copy time $T_D$.\n$$ T_{\\text{serial}} = T_H + T_K + T_D $$All three steps are necessary, but steps that belong to different data can overlap. For example, while a kernel processes the first input, the copy engine can perform the H2D copy of the second input, keeping the compute unit and copy unit busy at the same time. The copy engine must continue reading host memory after the CPU has moved on to subsequent code, so first we need to see how that memory is managed by the operating system.\nPage and Pageable Memory The operating system manages host memory in fixed-size units called pages. A common page size is 4KB. Both the address space the program sees and the actual RAM are divided into this size, and the operating system keeps a table of which RAM location each page of the program is placed in. When malloc is called, only the pages of the program\u0026rsquo;s address space are reserved at first, and RAM is attached when that address is first read or written. Host memory whose connection to RAM is made when needed, and may later change, is called pageable memory.\nPage Fault When the program reads or writes a page that is not yet in RAM, a page fault occurs. A page fault is the signal for the operating system to step in and attach RAM to that page. If RAM is short at that point, the operating system writes the contents of a page that has not been used for a while out to disk and frees that spot; the disk area that holds these evicted pages is called swap or the page file. Reading an evicted page again raises another page fault, and the operating system brings it back from disk into RAM. Because of this, pageable memory can handle data larger than RAM, but which pages are in RAM changes from moment to moment, and the operating system has to step in every time one is missing.\nPinned Memory The GPU has hardware dedicated to copies between host memory and device memory, called a copy engine. In this section, it executes DMA (Direct Memory Access). DMA means that dedicated hardware moves data between memories instead of a CPU core moving each byte. When the CPU calls cudaMemcpyAsync, the CUDA runtime passes the request to the CUDA driver. The driver is software that submits work to the GPU, and sends the source address, destination address, and size as a copy command. The copy engine executes that command while the CPU runs subsequent code.\nA discrete GPU is a separate card connected to the host system through PCIe. Data read from system RAM passes through the CPU\u0026rsquo;s I/O path and the PCIe root complex, the host hardware that connects PCIe devices. After crossing PCIe, the data enters the GPU through PCIe I/O and its internal data path. The GPU memory subsystem contains the L2 cache, which temporarily keeps recently used data, and memory controllers, which handle reads and writes to GPU memory. The copy engine is inside the GPU and executes this H2D transfer. Exact internal placement differs by GPU architecture, so the diagram below shows only the public logical connections. The SM clusters in the diagram are groups of GPU compute units; they are separate from the copy engine.\nThe copy engine must keep reading the same RAM pages until the copy finishes. The operating system must therefore not move those pages to different page frames or write them out to disk during the transfer. Pageable memory gives no such guarantee. CUDA asks the operating system to leave those page frames in RAM, and host memory fixed this way is called pinned memory. Since its pages do not go out to disk, pinned memory is also called non-pageable memory.\nPinned memory is allocated with cudaHostAlloc and released with cudaFreeHost. cudaHostAlloc is an allocation function that returns host memory just like malloc, differing only in that the returned pointer points to pinned memory. It is not a function that copies data, and it does not replace cudaMalloc, which creates device memory. So even after switching host memory to pinned memory, device memory is still created separately with cudaMalloc.\nFunction Region created malloc / free pageable host memory cudaHostAlloc / cudaFreeHost pinned host memory cudaMalloc / cudaFree device memory const int N = 1000; const size_t bytes = N * sizeof(float); float *h_x = nullptr; float *h_y = nullptr; // pageable version from the earlier section // float *h_x = (float *)malloc(bytes); // float *h_y = (float *)malloc(bytes); cudaHostAlloc(\u0026amp;h_x, bytes, cudaHostAllocDefault); // pinned input cudaHostAlloc(\u0026amp;h_y, bytes, cudaHostAllocDefault); // pinned output float *d_x = nullptr; float *d_y = nullptr; cudaMalloc(\u0026amp;d_x, bytes); // device input cudaMalloc(\u0026amp;d_y, bytes); // device output // ... H2D copy, kernel, D2H copy ... // pageable version from the earlier section // free(h_x); // free(h_y); cudaFreeHost(h_x); cudaFreeHost(h_y); cudaFree(d_x); cudaFree(d_y); To pin a region that was already created with malloc, use cudaHostRegister; to unpin it while keeping the region, use cudaHostUnregister.\nPinned memory occupies that much real RAM, so no more of it can be created than the RAM size, and past that limit cudaHostAlloc returns an out-of-memory error. And pinning a large share of RAM leaves the operating system with less RAM to work with and slows down the host side, so only the regions that exchange data with the GPU are made pinned.\nAsynchronous Calls and cudaMemcpyAsync An asynchronous call is a call where the CPU moves on to the next line without waiting for the GPU work to complete. A kernel launch is asynchronous to begin with, so the CPU runs the next code before the kernel finishes. To request a copy in the same way, use cudaMemcpyAsync. Its arguments are the same as cudaMemcpy with one stream appended at the end. To overlap CPU execution with an H2D or D2H copy as in this post, the host-side pointer must refer to pinned memory. With that condition satisfied, the CPU returns from the call before the copy finishes while the copy engine continues the transfer.\nFor example, if a D2H copy is requested asynchronously to bring the result in d_y back to h_y, the CPU can run other code before the copy finishes. That does not mean the result in h_y is ready yet.\nCPU: request D2H copy → CPU code unrelated to the copy → wait for stream → use h_y GPU: D2H copy in progress cudaStreamSynchronize(stream) makes the CPU wait until all work in that stream has finished. Therefore, h_y is read only after this wait completes.\nAn asynchronous call only means the CPU does not wait; it does not mean two GPU operations actually run at the same time. Even when the call returns early, the two operations may run one after another inside the GPU. Which operations run in which order is decided by the stream.\nStream A stream groups GPU operations whose submission order must be preserved.\nRule 1) Operations in the same stream keep their submission order. If an H2D copy, kernel, and D2H copy are placed in one stream, the kernel runs after the H2D copy finishes, and the D2H copy starts after the kernel finishes.\nRule 2) There is no prescribed order between different streams. CUDA does not guarantee which operation starts first, so either one may run first, at the same time, or later. Operations must be placed in different streams to run in the same time window. Even then, if the GPU cannot run the copy and computation together, the copy and the computation run one after another.\nA stream is declared as a variable of type cudaStream_t and created with cudaStreamCreate. The created stream goes into the last argument of cudaMemcpyAsync and the fourth argument of the kernel launch\u0026rsquo;s \u0026lt;\u0026lt;\u0026lt;\u0026gt;\u0026gt;\u0026gt;. In \u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream\u0026gt;\u0026gt;\u0026gt;, the third value is the number of bytes of shared memory, the small memory inside the GPU that the threads of a block use together, to reserve additionally at run time, and 0 means no extra space.\ncudaStream_t stream; cudaStreamCreate(\u0026amp;stream); cudaMemcpyAsync(d_x, h_x, bytes, cudaMemcpyHostToDevice, stream); transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream\u0026gt;\u0026gt;\u0026gt;(d_x, d_y, N); cudaMemcpyAsync(h_y, d_y, bytes, cudaMemcpyDeviceToHost, stream); cudaStreamSynchronize(stream); cudaStreamDestroy(stream); None of the three calls holds the CPU, but because they enter the same stream, the kernel runs after the H2D copy finishes and the D2H copy starts after the kernel finishes. So the H2D copy → kernel → D2H copy order for the same data is kept by the stream. cudaStreamSynchronize is the function that makes the CPU wait until all work in that stream has finished, and cudaStreamQuery only reports whether the stream is empty without waiting. A stream that is no longer needed is removed with cudaStreamDestroy.\nChunk When a large array is processed in one piece, the kernel starts only after the entire input has been copied from host to device, and the result is copied back only after the entire kernel has finished. The array is divided into several ranges to reduce this waiting. One such piece of data is called a chunk.\nFor example, consider y[i] = x[i] * 2 on an array of 8 elements, where i runs from 0 to 7. Dividing that array into two chunks of 4 elements makes x[0] through x[3] chunk 0 and x[4] through x[7] chunk 1. Each of y[0] through y[3] needs only the x value at the same index, so it can be computed without any value from chunk 1. The two chunks are therefore processed without waiting for each other.\nCall the H2D copy, kernel, and D2H copy for chunk 0 H0, K0, and D0, and place all three in stream 0. Place H1, K1, and D1 for chunk 1 in stream 1. Each stream preserves H0 → K0 → D0 and H1 → K1 → D1. There is no prescribed order between the two streams, so when the GPU can run a copy and a kernel at the same time, H1 can be copied while K0 runs and D0 can be copied while K1 runs.\nThe loop submits all three operations of one chunk before moving to the next, and chunk % streamCount sends chunk 0 to stream 0, chunk 1 to stream 1, chunk 2 to stream 2, and chunk 3 to stream 3. The top row of the figure below is the order in which the CPU submits, and the four rows beneath it are the streams each operation went into.\nBoth rows above use the same horizontal scale, and the dashed lines inside each serial bar mark where that bar divides into four chunks. One dashed division has the same width as one chunk below it, so both arrangements perform the same amount of work. What changes is only where that work is placed on the time axis. The times on the right of the figure were measured on an NVIDIA A100.1\nTo put this structure in code, several streams are created and rotated across chunks. Device memory is allocated once at the full array size, and only the start of each chunk is moved with offset. offset is the element index at which the current chunk begins, and d_x + offset is the address of the element offset positions after the one d_x points to. The loop increases offset by chunkElements2 each time, so every chunk points at a different range of the same array.\nconstexpr int streamCount = 4; constexpr size_t N = 1ULL \u0026lt;\u0026lt; 24; // 16,777,216 elements constexpr size_t chunkElements = 1 \u0026lt;\u0026lt; 20; // 1,048,576 elements constexpr size_t bytes = N * sizeof(float); float *h_x = nullptr; float *h_y = nullptr; float *d_x = nullptr; float *d_y = nullptr; // pageable version from before pinned memory // float *h_x = (float *)malloc(bytes); // float *h_y = (float *)malloc(bytes); cudaHostAlloc(\u0026amp;h_x, bytes, cudaHostAllocDefault); cudaHostAlloc(\u0026amp;h_y, bytes, cudaHostAllocDefault); cudaMalloc(\u0026amp;d_x, bytes); // device memory is the same in both versions cudaMalloc(\u0026amp;d_y, bytes); for (size_t i = 0; i \u0026lt; N; ++i) { // the host fills in the input values h_x[i] = static_cast\u0026lt;float\u0026gt;(i); } cudaStream_t streams[streamCount]; for (int i = 0; i \u0026lt; streamCount; ++i) { cudaStreamCreate(\u0026amp;streams[i]); } constexpr size_t chunkBytes = chunkElements * sizeof(float); constexpr int block = 256; constexpr int grid = chunkElements / block; // 4096 for (size_t chunk = 0, offset = 0; offset \u0026lt; N; ++chunk, offset += chunkElements) { cudaStream_t stream = streams[chunk % streamCount]; // the synchronous version has no stream argument // cudaMemcpy(d_x + offset, h_x + offset, chunkBytes, // cudaMemcpyHostToDevice); cudaMemcpyAsync(d_x + offset, h_x + offset, chunkBytes, cudaMemcpyHostToDevice, stream); transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream\u0026gt;\u0026gt;\u0026gt;( d_x + offset, d_y + offset, chunkElements); cudaMemcpyAsync(h_y + offset, d_y + offset, chunkBytes, cudaMemcpyDeviceToHost, stream); } cudaDeviceSynchronize(); for (int i = 0; i \u0026lt; streamCount; ++i) { cudaStreamDestroy(streams[i]); } // pageable version from before pinned memory // free(h_x); // free(h_y); cudaFreeHost(h_x); cudaFreeHost(h_y); cudaFree(d_x); cudaFree(d_y); transform computes the y[i] = x[i] * 2 operation used above. Both h_x and h_y are pinned memory because they are used for asynchronous H2D and D2H copies, while d_x and d_y are device memory. Each trip through the loop handles one chunk and submits its three operations to the same stream. After all chunks have been submitted, cudaDeviceSynchronize3 waits for all work on the device before the streams and memory are released.\nSubmitting all three operations for one chunk before moving on to the next chunk is called depth-first4 submission order.\nWith four streams, chunk 4 goes back into streams[0], the stream chunk 0 used. By rule 1, the H2D copy of chunk 4 starts in streams[0] only after the D2H copy of chunk 0 has finished. Memory allocation and stream creation are setup steps that do not need to be repeated for every chunk, so they are completed once before the loop. The loop contains only the H2D copy, kernel launch, and D2H copy, and keeps using the memory and streams created in advance.\nThe actual shape of the overlap depends on how much data each chunk copies and how long the kernel runs. If the kernel is very short, overlapping a copy with the kernel saves little time. On a GPU whose copy-engine configuration can handle H2D and D2H at the same time, the overlap between the next chunk\u0026rsquo;s H2D copy and the previous chunk\u0026rsquo;s D2H copy can provide a larger gain.\nDefault Stream A kernel launch or cudaMemcpy with no stream specified goes into the default stream. The default stream in the standard configuration is called the legacy default stream. When it is used together with streams created by cudaStreamCreate above, a default-stream operation starts only after every operation submitted earlier to the other streams has finished, and an operation submitted to those streams afterwards starts only once the default-stream operation has finished.\nBelow, three chunks from the previous section are submitted, but the middle line is missing its stream argument. c is the number of elements in one chunk, and the three launches handle chunks 0, 1, and 2.\nconst size_t c = chunkElements; transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, streams[0]\u0026gt;\u0026gt;\u0026gt;(d_x, d_y, c); // A: chunk 0 transform\u0026lt;\u0026lt;\u0026lt;grid, block\u0026gt;\u0026gt;\u0026gt; (d_x + c, d_y + c, c); // B: stream argument missing transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, streams[1]\u0026gt;\u0026gt;\u0026gt;(d_x + 2 * c, d_y + 2 * c, c); // C: chunk 2 B has no stream argument, so it goes into the legacy default stream. B therefore starts after A finishes and C starts after B finishes, and A and C, which were in different streams and could have overlapped, no longer do. For this reason, every copy and kernel launch in a region intended to overlap names a stream explicitly.\nWith the compiler option nvcc --default-stream per-thread, each CPU thread gets its own default stream, and B above does not automatically block between A and C. This option is used when code written to use the default stream is run together with explicitly created streams.\nPutting a Host Function into a Stream cudaLaunchHostFunc inserts a function that runs on the CPU as an operation in a stream. The stream below is one created with cudaStreamCreate. If a CPU function named process must read the result of transform, the kernel, D2H copy, and host function are placed in the same stream in that order. CUDART_CB marks the function form CUDA uses to call this CPU function.\nvoid CUDART_CB process(void *data) { float *result = static_cast\u0026lt;float *\u0026gt;(data); // Process result on the CPU. Do not call a CUDA API here. } transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream\u0026gt;\u0026gt;\u0026gt;(d_x, d_y, N); cudaMemcpyAsync(h_y, d_y, bytes, cudaMemcpyDeviceToHost, stream); cudaLaunchHostFunc(stream, process, h_y); process is called after the D2H copy has finished, so it can read the completed h_y. The stream does not advance to its next operation until process returns, and no CUDA API calls such as kernel launches or cudaMalloc are made inside process.\nCUDA Event A CUDA event marks a position in a stream. Calling cudaEventRecord inserts the event into the stream, and the event completes when all preceding operations have finished and execution reaches that position. To measure a kernel\u0026rsquo;s execution time, a start event, the kernel, and a stop event are placed in the same stream in that order.\ncudaEvent_t start; cudaEvent_t stop; cudaEventCreate(\u0026amp;start); cudaEventCreate(\u0026amp;stop); cudaEventRecord(start, stream); transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream\u0026gt;\u0026gt;\u0026gt;(d_x, d_y, N); cudaEventRecord(stop, stream); cudaEventSynchronize(stop); float milliseconds = 0.0f; cudaEventElapsedTime(\u0026amp;milliseconds, start, stop); cudaEventDestroy(start); cudaEventDestroy(stop); cudaEventSynchronize(stop) makes the CPU wait until the stop event completes. After that, cudaEventElapsedTime writes the GPU time between start and stop into milliseconds.\nEvents are also used to create an order between two streams. Below, d_z is device memory allocated with cudaMalloc at the same size as d_x and d_y, and stream0 and stream1 are streams created with cudaStreamCreate. The transform in stream 0 writes its result into d_y, and the transform in stream 1 reads that d_y as input and writes d_z. Because the two kernels are in different streams, rule 2 leaves their order undefined, so the kernel in stream 1 could start first. A ready event is therefore recorded after the kernel in stream 0, and stream 1 waits for that event before its kernel.\ncudaEvent_t ready; cudaEventCreate(\u0026amp;ready); transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream0\u0026gt;\u0026gt;\u0026gt;(d_x, d_y, N); cudaEventRecord(ready, stream0); cudaStreamWaitEvent(stream1, ready, 0); transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream1\u0026gt;\u0026gt;\u0026gt;(d_y, d_z, N); cudaStreamSynchronize(stream1); cudaEventDestroy(ready); In the code above, cudaStreamWaitEvent makes only the later work in stream 1 wait; it does not make the CPU wait. The final argument 0 specifies no additional behavior. cudaStreamWaitEvent creates an order only between the kernel in stream 0 and the kernel in stream 1, without stopping the whole device.\nRunning Several Kernels at Once Two kernels that process different arrays do not need to wait for each other\u0026rsquo;s results. All four pointers below are device memory allocated with cudaMalloc at bytes each: d_x0 and d_y0 are the input and output of the first computation, and d_x1 and d_y1 are the input and output of the second. Placing the two kernels in different streams creates the possibility that they run at the same time on the same GPU.\ntransform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream0\u0026gt;\u0026gt;\u0026gt;(d_x0, d_y0, N); transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream1\u0026gt;\u0026gt;\u0026gt;(d_x1, d_y1, N); An SM (Streaming Multiprocessor) is a GPU compute unit on which a kernel\u0026rsquo;s blocks are actually placed. If the first kernel\u0026rsquo;s blocks occupy every execution slot on every SM, the second kernel waits for a slot to open even though it is in another stream. If the first kernel uses only some of the slots, blocks from the second kernel can enter the remaining slots and execute in the same time window.\nIf a single kernel can fill the GPU, that one kernel is the fastest. Running several kernels at once is meaningful when work arrives in small units that are hard to merge into one kernel.\nStream priority is the priority the GPU consults when deciding which stream\u0026rsquo;s kernel to take the next block from. For example, a long-running background kernel can be placed in a low-priority stream, while a short kernel that needs to start promptly can be placed in a high-priority stream. High priority does not interrupt a block that is already running. It only makes the GPU choose the next block from the high-priority stream first when a slot opens on an SM. Priority streams are created with cudaStreamCreateWithPriority, and the available priority range is read with cudaDeviceGetStreamPriorityRange.\nStreams on Multiple GPUs The same stream rules carry over when there are several GPUs. cudaGetDeviceCount reads the number of GPUs, and cudaSetDevice selects the GPU that subsequent CUDA calls will target. The selected GPU is the current device. Device memory and streams are bound to the current device at the time they are created. In the example below, d0_x, d0_y, and stream0 belong to GPU 0, while d1_x, d1_y, and stream1 belong to GPU 1, and each GPU runs the transform defined earlier on its own arrays.\nfloat *d0_x = nullptr, *d0_y = nullptr; float *d1_x = nullptr, *d1_y = nullptr; cudaStream_t stream0; cudaStream_t stream1; cudaSetDevice(0); cudaMalloc(\u0026amp;d0_x, bytes); cudaMalloc(\u0026amp;d0_y, bytes); cudaStreamCreate(\u0026amp;stream0); // stream bound to GPU 0 transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream0\u0026gt;\u0026gt;\u0026gt;(d0_x, d0_y, N); cudaSetDevice(1); cudaMalloc(\u0026amp;d1_x, bytes); cudaMalloc(\u0026amp;d1_y, bytes); cudaStreamCreate(\u0026amp;stream1); // stream bound to GPU 1 transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream1\u0026gt;\u0026gt;\u0026gt;(d1_x, d1_y, N); cudaSetDevice(0); cudaStreamSynchronize(stream0); cudaStreamDestroy(stream0); cudaFree(d0_x); cudaFree(d0_y); cudaSetDevice(1); cudaStreamSynchronize(stream1); cudaStreamDestroy(stream1); cudaFree(d1_x); cudaFree(d1_y); Because kernel launches do not make the CPU wait, the CPU can submit transform to GPU 0 and then immediately submit it to GPU 1. At the end, each GPU is selected again and the CPU waits for its stream to finish. To move data between GPUs, peer access can be used. Peer access is the ability of one GPU to read and write another GPU\u0026rsquo;s memory directly, and it requires the two GPUs to be on the same interconnect such as PCIe or NVLink. cudaDeviceCanAccessPeer checks whether it is supported; if copies go in both directions, cudaDeviceEnablePeerAccess is called on both sides, and then cudaMemcpyPeerAsync performs the copy. The data then moves straight from one GPU\u0026rsquo;s memory to the other\u0026rsquo;s without passing through host memory.\nUnified Memory and Prefetch The stream rules are the same with Unified Memory. cudaMemPrefetchAsync moves a region created with Unified Memory toward the CPU or a GPU in advance. In the code below, x and y are Unified Memory pointers allocated with cudaMallocManaged that both the CPU and the GPU access through the same pointer, and device is the number of the GPU that will run the kernel. cudaCpuDeviceId is the CUDA constant that names the CPU as the destination.\nconst int device = 0; cudaSetDevice(device); float *x = nullptr; float *y = nullptr; cudaMallocManaged(\u0026amp;x, bytes); cudaMallocManaged(\u0026amp;y, bytes); cudaStream_t stream; cudaStreamCreate(\u0026amp;stream); cudaMemPrefetchAsync(x, bytes, device, stream); // move input to GPU transform\u0026lt;\u0026lt;\u0026lt;grid, block, 0, stream\u0026gt;\u0026gt;\u0026gt;(x, y, N); cudaMemPrefetchAsync(y, bytes, cudaCpuDeviceId, stream); // move result to CPU cudaStreamSynchronize(stream); cudaStreamDestroy(stream); cudaFree(x); cudaFree(y); Because all operations are in the same stream, the kernel starts after the prefetch toward the GPU finishes, and the prefetch toward the CPU starts after the kernel finishes. Once the final wait completes, the CPU can read y. This movement happens page by page and also updates page records on both the CPU and the GPU, which can leave empty gaps on the execution timeline.\nChecking in Nsight Systems Whether concurrent execution actually happened is checked on the GPU execution timeline. Nsight Systems is a tool that records the CPU\u0026rsquo;s CUDA calls and the GPU\u0026rsquo;s copies and kernel executions on the same timeline while the program runs.\nCompiling the chunk code above with nvcc produces an executable; calling it overlap, it is run as follows.\nnsys profile --stats=true ./overlap This command saves the run as a report file and prints a summary of CUDA calls, kernels, and copies. Opening the report file in the Nsight Systems window shows the CPU-side calls on top and the GPU-side copies and kernels below. In the serial code the H2D copy, kernel, and D2H copy appear in a single line; in the code with several streams the rows split per stream and the kernel of one chunk appears in the same time range as the copy of another chunk.\nIn the end, concurrency is not a technique for removing dependencies. The H2D copy → kernel → D2H copy order of the same data is kept with the same stream, and only independent chunks are split across different streams. Whether the times actually overlap is decided by the copy-engine configuration and by open execution slots on the SMs.\nReferences OLCF CUDA Training Series: CUDA Concurrency CUDA Concurrency slides OLCF CUDA Training Series: HW7 CUDA Programming Guide: Asynchronous Execution CUDA C++ Best Practices Guide: Asynchronous and Overlapping Transfers with Computation CUDA Runtime API: API Synchronization Behavior Nsight Systems User Guide The device was an NVIDIA A100-SXM4-80GB in a RunPod container with CUDA 12.4 and driver 580.159.04, built with nvcc -O3 -arch=sm_80. The run used N of 16,777,216 elements (64MB), 16 chunks, and 4 streams, on a GPU whose asyncEngineCount is 3. Timing came from cudaEvent over 30 repetitions after 5 warm-up runs, reported as the median: 5.230 ms serial (min 5.204, max 7.976) and 3.384 ms streamed (min 3.340, max 3.681). The measurement code is in overlap_bench.cu.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nchunkElements is the number of elements placed in one chunk. The code below sets it to 1 \u0026lt;\u0026lt; 20, that is 1,048,576 elements, and with N at 16,777,216 elements that yields 16 chunks. A larger value produces fewer chunks and therefore fewer opportunities to overlap, while a smaller value raises the share of kernel launch and copy requests per chunk.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWhen the loop ends the CPU has only submitted the work and the GPU is still running it. Without this line the CPU goes straight on to cudaFreeHost and cudaFree, releasing memory that the GPU is still copying from or reading. For the same reason any code that reads the results out of h_y has to come after this line. The synchronous version that uses cudaMemcpy does not need it, because the copy is already complete when that function returns.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe opposite order submits the same kind of operation for every chunk first, and is called breadth-first.\n// depth-first for (chunk 0..15) { H2D; kernel; D2H; } // breadth-first for (chunk 0..15) { H2D; } for (chunk 0..15) { kernel; } for (chunk 0..15) { D2H; } Both orders execute H → K → D within a stream.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://bias92.github.io/en/posts/cuda-5-concurrency/","summary":"How data copies and kernel execution are placed in the same time window, explained in the order host and device memory, pinned memory, cudaMemcpyAsync, stream, and chunk.","title":"05 CUDA Concurrency: Streams, Async Copies, and Overlap"},{"content":"A CUDA program uses two different processors together, a CPU and a GPU. The CPU is called the Host and the GPU the Device. The two processors differ in how they execute instructions and how they access memory. A structure like this is called a heterogeneous system.\nIn Host-Device Data Flow, a CPU-side h_data and a GPU-side d_data were created separately, and the data between the two memories was copied with cudaMemcpy. That is explicit memory management, where the location and the moment of each move are visible in the code. As data structures grow more complex, the developer keeps the two memory regions, the copy directions, and their lifetimes all in step.\nWith Unified Memory, cudaMallocManaged creates a memory region that the CPU and GPU use together. A region that CUDA manages in this way is called a managed allocation. This post explains virtual addresses, data placement, synchronization, and cache coherence in turn, and then applies them to the actual device attributes of Jetson AGX Orin.\nCPU Memory and GPU Memory Allocation is the act of securing a memory region for the program to use. An allocation API sets aside a region of the requested size and returns its starting address as a pointer.\nmalloc creates an allocation for CPU code to use. cudaMalloc creates a device allocation that CUDA manages for the GPU to access. The pointers the two APIs return may point to different memory regions.\nIn a typical PC with a discrete GPU, the CPU\u0026rsquo;s main memory, system DRAM, and the GPU\u0026rsquo;s dedicated memory, VRAM, are physically separate. The two memories exchange data through a connection called PCIe. When the CPU\u0026rsquo;s malloc allocation and the GPU\u0026rsquo;s cudaMalloc allocation are used separately as in the previous post, an H2D (Host to Device) copy is done before the computation and a D2H (Device to Host) copy before the CPU reads the GPU result.\nWith an integrated GPU, the CPU and GPU use the same system DRAM. The two processing units each use their own address translation unit and their own cache, the store that keeps frequently used data close by. So even with the same DRAM, each processing unit needs its own address connection, and an access order between the CPU and GPU is needed.\nThe program sets the access order, for example by having the CPU read after the GPU work has finished. The CUDA Runtime is the library that provides the APIs the program calls, and the CUDA driver is the system software that controls GPU execution and address connections. The runtime, the driver, and the hardware share the management of address connections, data placement, and cache state.\nVirtual Address and Physical Memory Address Translation A process is the OS unit that refers to one running program, a different word from processor (CPU or GPU). A pointer in a CUDA process holds a virtual address. The MMU (Memory Management Unit) is the device that translates this address into a physical address in DRAM or VRAM.\nThe virtual address space is the full range of virtual addresses one process can use. This range is usually divided into units of a fixed size called pages. Physical memory is divided into frames (page frames) of the same size. The page table records which physical frame each virtual page is connected to and which accesses are allowed. This connection is called a mapping, and it is the address connection mentioned above. Some virtual pages are not yet connected to a physical frame.\nWhen the CPU or GPU reads or writes through a pointer, that processing unit\u0026rsquo;s MMU translates the virtual address into a physical address. A virtual address splits into a virtual page number and an offset that gives the position inside the page. Address translation replaces the virtual page number with a physical frame number and keeps the offset as it is.\nThe MMU first looks in the TLB (Translation Lookaside Buffer) for a recent virtual page to physical frame translation. A cache is a store that keeps a small copy of frequently used items close by and checks there first. The TLB is a cache that holds address translation results.\nThe translated physical address points to the location in system DRAM or VRAM where the data actually is. The NVIDIA CUDA Programming Guide explains that CUDA manages the placement and movement of data among these several physical memories.\nPlacement and Migration A managed allocation is a memory region created with cudaMallocManaged. The CUDA Runtime and driver manage where this region is stored, when it moves, and the mapping for each processing unit. cudaMalloc creates a device allocation, and the program requests cudaMemcpy when it exchanges data with the host.\nThe CUDA documentation keeps these three apart. Below is one case on a discrete GPU where the data that x points to moves from system DRAM to VRAM. The pointer x holds the virtual address V, and the value of *x is assumed to be 41.\nTerm What it refers to Example following x mapping The address relation connecting a virtual page to a physical frame Before the move, V is connected to frame A in system DRAM. After the move, on the GPU the same V is connected to frame B in VRAM. The pointer holds the same V before and after the move. placement Which physical memory the data is currently stored in If 41 is in frame A of system DRAM, the placement is system DRAM. After the move, if it is in frame B, the placement is VRAM. migration Moving data to a different physical memory, changing its placement The page holding 41 is copied from frame A in system DRAM to frame B in VRAM, and the GPU\u0026rsquo;s address connection is changed to frame B. This figure is the discrete GPU path where CPU DRAM and VRAM are separated by PCIe. Placement is decided per page of a managed allocation, and the possible locations depend on the hardware structure.\nWhere managed data can be placed Moving to a separate VRAM discrete GPU system DRAM or VRAM yes integrated GPU shared system DRAM accessed within the shared DRAM The value that must be read at an address is the result of the last write completed in the order set by synchronization. That result may be in a CPU or GPU cache, so before the next processing unit reads the same address, the access order and the cache state are aligned.\nUVA and Unified Memory CUDA\u0026rsquo;s UVA (Unified Virtual Addressing) places the CPU memory and each GPU memory of a process in one virtual address space. The CPU and GPU each use the mappings that are valid for them. UVA provides the address scheme that tells the memories apart. The accessor of a cudaMalloc allocation is the GPU. Unified Memory manages the access and placement of managed allocations and lets the next processing unit read the result of a write whose order was set by CUDA synchronization.\nUnified Memory and Managed Allocation Unified Memory provides managed allocations that both CPU and GPU code can use. cudaMallocManaged is the basic Runtime API that creates this allocation.\nint *x = nullptr; cudaMallocManaged(\u0026amp;x, sizeof(*x)); It secures space of sizeof(*x) bytes and records the starting address in the pointer variable x. \u0026amp;x passes the address of the pointer variable x itself to the function, so that the function can write the starting address into x. This allocation is released with cudaFree(x).\nThe explicit-copy style keeps a CPU-side h_data, a GPU-side d_data, H2D, and D2H in the code. The managed style has the CPU and GPU use the single x, and the data movement is handled by the Runtime, driver, and hardware according to what the current system supports.\nThe GPU Modifying a Value the CPU Wrote The code below is the basic form in which the CPU and GPU use the same managed allocation in turn. A kernel is a function that runs on the GPU, and a thread is the unit of work that runs that function. This code makes 42 because one GPU thread adds 1 to 41 once.\n#include \u0026lt;cstdio\u0026gt; #include \u0026lt;cuda_runtime.h\u0026gt; __global__ void add_one(int *x) { *x += 1; } int main() { int *x = nullptr; cudaMallocManaged(\u0026amp;x, sizeof(*x)); *x = 41; std::printf(\u0026#34;before kernel: %d\\n\u0026#34;, *x); add_one\u0026lt;\u0026lt;\u0026lt;1, 1\u0026gt;\u0026gt;\u0026gt;(x); cudaDeviceSynchronize(); std::printf(\u0026#34;after kernel: %d\\n\u0026#34;, *x); cudaFree(x); } __global__ declares a kernel. A block is a bundle of GPU threads placed together. \u0026lt;\u0026lt;\u0026lt;1, 1\u0026gt;\u0026gt;\u0026gt; places one thread in one block. Right after the kernel launch the CPU keeps executing the next code, so cudaDeviceSynchronize(), which waits until the GPU work has finished, is placed between the GPU write and the CPU read.\nSynchronization and Cache Coherence The example above has the CPU write 41, the GPU change it to 42, and then the CPU read that value. Two things are needed here. The CPU must read after the GPU work has finished, and when it does it must read 42, the result of the GPU write.\nThe first is synchronization. cudaDeviceSynchronize() makes the CPU thread wait until the GPU work submitted earlier has finished. So the CPU read starts after the GPU write has finished.\nThe second is cache coherence. To reduce DRAM accesses, the CPU and GPU keep recent data in their own caches. A cache brings data in units of cache lines, which are bundles of consecutive bytes. After the GPU writes 42 into its cache, the cache state is aligned so that the CPU\u0026rsquo;s next read obtains 42.\nThere are two ways to achieve this cache coherence. When hardware aligns the cache state between processors directly, it is hardware coherence. When the driver adjusts the address connections, data movement, and cache state at access and synchronization boundaries so that the next processing unit reads the result of the earlier one\u0026rsquo;s write, it is software coherence. Among these, the work of making a cache\u0026rsquo;s changed values visible to another processing unit, or of discarding an older cached copy, is called cache maintenance.\nSynchronization decides when the CPU reads, and cache coherence decides which value is visible at that moment. Placement names the physical memory where the data lies. When several processing units modify the same location, synchronization sets the access order.\nUnified Memory Support Models CUDA divides the way managed allocations are accessed into a Full model and a Limited model. The criteria are when the GPU prepares the managed allocation and whether CPU access is allowed while the GPU is running.\nFull model concurrentManagedAccess is the device attribute that indicates whether the CPU and GPU can use a managed allocation at the same time. If this value is 1, it is the Full model. When the GPU accesses a virtual page, CUDA sets up the GPU mapping and, if needed, moves that data into a physical frame in GPU memory. The CPU and GPU can use different addresses of the same managed allocation at the same time.\nLimited model If concurrentManagedAccess=0, it is the Limited model. CUDA makes the managed memory usable by the GPU at the kernel launch boundary and reopens CPU access after synchronization.\nItem Full model Limited model cudaMallocManaged available available When the GPU gains access to the data When the GPU accesses a virtual page, CUDA handles the mapping or migration At the kernel launch boundary, CUDA puts the managed allocation into a GPU-accessible state CPU access to managed memory while the GPU runs Different addresses can be accessed The CPU accesses after synchronizing the GPU work Managed allocation larger than the physical memory the GPU can use Allocations larger than GPU memory are usable Used within the physical memory capacity available to the GPU Jetson AGX Orin uses shared DRAM together with the Limited model. There are also systems where a discrete GPU with separate CPU DRAM and GPU VRAM operates in the Full model.\nIn the Full model, the data belonging to each virtual page of a managed allocation is usually placed in the memory of the processing unit that first read or wrote that virtual page. The CUDA documentation calls this First touch. The program can tell the driver a preferred location with cudaMemAdvise. This information is called a hint, and the driver uses it in later placement decisions.\nThe current support model is decided by the combination of the operating system and its core, the OS kernel (a different word from the GPU kernel above), the CUDA driver, the GPU, and the CPU–GPU connection structure. So the support values of the current environment are checked with cudaDeviceGetAttribute.\nmanagedMemory tells whether managed allocations created by explicit request, such as cudaMallocManaged, are supported. The next three attributes are read in the order below.\nIf concurrentManagedAccess is 0, it is the Limited model. If that value is 1, it is the Full model, and when pageableMemoryAccess is 0 only managed allocations created explicitly through the CUDA API use this model. If both values are 1, system allocations such as malloc, new, and mmap also fall within Unified Memory. Only then is pageableMemoryAccessUsesHostPageTables read. 0 is software coherence, the method in which the driver manages mappings and migration to achieve the cache coherence above. 1 is hardware coherence, in which the CPU and GPU use the same host page table and hardware aligns the cache state directly. Page Fault and Migration on a Discrete GPU The following is the case of handling a GPU page fault by migration in a software-coherent Full model where CPU DRAM and GPU memory are separate. Software coherence is the method in which the driver manages the address connections and data movement of the CPU and GPU.\nIn the initial state, the managed data is in a physical frame of CPU memory, and the CPU mapping points to that frame. When the GPU first reads the same virtual address, a page fault occurs and the memory access stops. The page fault is the signal to prepare the GPU mapping to be used for that virtual page.\nThere are two ways to handle the fault: migration and remote mapping. The figure below is the migration path. The operating-system memory manager, which manages page tables and physical frames, works with the CUDA driver to prepare a physical frame in GPU memory, move the data, and install the GPU mapping. Once the mapping is ready, the stopped GPU instruction resumes.\nIn the remote mapping path, the data is left in the physical frame of CPU memory and the GPU mapping is connected to that frame. Migration changes the data placement; remote mapping keeps it.\nWhen the CPU and GPU modify the same pages in turn, page ping-pong can occur, where migration in both directions repeats. cudaMemPrefetchAsync moves the data of a given range in advance and brings the placement forward. CUDA synchronization sets the execution order of the CPU and GPU.\nHMM HMM (Heterogeneous Memory Management) is the Linux kernel subsystem that connects CPU page table changes, GPU faults, and page migration. In a Full model that uses HMM, system allocations made with malloc, new, and mmap can also be used by the GPU. Device attributes classify this support range, and running nvidia-smi, the command-line tool shipped with the NVIDIA driver, with -q shows whether HMM is currently in use in the Addressing Mode item.\nJetson AGX Orin: Shared DRAM and the Limited model The concepts above were applied to an actual device. The environment is the Jetson AGX Orin Developer Kit, L4T (Linux for Tegra) R36.5.0, the Jetson Linux distribution, JetPack 6.2.2, which bundles the CUDA development tools, and CUDA 12.6. Tegra is the name of NVIDIA\u0026rsquo;s SoC product family that Orin belongs to. Orin is an SoC (System on Chip) that puts the CPU and GPU on one chip. Device 0 was an integrated GPU with compute capability 8.7. Compute capability indicates the generation of CUDA hardware features a GPU supports.\ndevice=0 name=Orin cc=8.7 integrated=1 managedMemory=1 concurrentManagedAccess=0 pageableMemoryAccess=0 Verdict managedMemory=1 means explicit managed allocations are supported, and concurrentManagedAccess=0 means the Limited model. pageableMemoryAccess=0 limits the scope of Unified Memory to explicit managed allocations such as cudaMallocManaged.\nThe device output above determines the Limited model. The NVIDIA Tegra memory model describes the shared SoC DRAM and cache behavior.\nShared DRAM and Cache According to the Tegra documentation, the CPU and integrated GPU of Tegra share the SoC DRAM, and device memory, host memory, and unified memory are all allocated from the same physical SoC DRAM. The integrated=1 in the actual output also confirms the integrated structure of the Orin GPU.\nOrin\u0026rsquo;s managed allocations are placed in the shared SoC DRAM. On top of the shared DRAM, the CPU and GPU can each store managed data in their own caches. The process of aligning the cache state so that the next processing unit reads the result of the earlier one\u0026rsquo;s write is cache coherence.\nOrin\u0026rsquo;s one-way I/O coherency lets the GPU read values the CPU wrote into its cache. In the direction where the CPU reads values the GPU wrote, the CUDA driver manages the GPU cache state at synchronization boundaries.\nThe Tegra documentation explains that in an environment with concurrentManagedAccess=0, cache maintenance work is added to kernel launches and synchronization, and that this work can increase execution latency.\nBuilding the example code above (managed_add.cu) for sm_87, the compile target for compute capability 8.7, and running it gave the following result.\nbefore kernel: 41 after kernel: 42 The 41 → 42 in the output is the CPU reading 42, the result of the GPU write, after the GPU work finished. The device attributes recorded alongside show the Unified Memory support range of this Orin.\nThe full runnable code is in managed_add.cu, the attribute query code in orin_um_probe.cu, and the actual output in Orin observation.\nReferences CUDA Programming Guide: Unified and System Memory: UVA, Unified Memory support models, device attributes, prefetch, HMM. CUDA Programming Guide: Unified Memory: detailed description of page faults, migration, coherence, and performance behavior. CUDA for Tegra: Memory Management: Tegra\u0026rsquo;s shared SoC DRAM, cache coherence, and Limited model guidance. ","permalink":"https://bias92.github.io/en/posts/cuda-4-unified-memory/","summary":"How the CPU and GPU share one memory allocation, explained through virtual addresses, data placement and migration, synchronization, and cache coherence, and applied to the actual device attributes of Jetson AGX Orin.","title":"04 CUDA Unified Memory: Virtual Address, Placement, and Coherence"},{"content":" Source: MIT 6.5930/1 Spring 2026, L02 - Overview of Deep Neural Network Components\nL01 established that data movement is expensive. L02 turns that statement into actual numbers. It takes one matrix-vector multiplication and calculates the operation count, minimum memory traffic, and traffic generated by a particular loop order. The best-case compute intensity is 0.99, while a simple implementation reaches only 0.33. Both compute the same expression, but one moves nearly three times as much data.\nThe second half of the lecture covers CNNs and fully connected layers. This is not the usual deep learning introduction about model accuracy or training methods. Each layer is written as a tensor expression, labeled with ranks and shapes, and eventually lowered to a seven-deep loop nest and matrix multiplication. Accelerator mapping needs the workload in this form before it can begin.\nThe PDF has 102 slides, many of which are animation frames that advance a dot or partial sum one step at a time. I consolidated each such sequence into its completed frame. The notes still cover the full conceptual range from L02-1 through L02-102.\nAccelerator Design Methodology TeAAL divides the design process into five stages.\nArchitecture description Workload development Workload evaluation Implementation comparison Design optimization The sequence looks ordinary, but most of this course fits between stages 2 and 3. Saying that an accelerator \u0026ldquo;handles matrix multiplication\u0026rdquo; is not enough to calculate either traffic or throughput. Hardware behavior appears only after deciding the tensor traversal order, where values are stored, and which PE owns each iteration.\nThe lecture deliberately starts with a very small architecture.\nOne PE with an ALU and local register, plus DRAM as global storage.\nThe PE has a multiplier, an adder, and a register, with DRAM outside it. There is no cache, global buffer, or NoC. This is less a realistic accelerator than a minimal model that exposes the effect of one mapping on traffic.\nOnce the architecture is fixed, the workload receives four kinds of specifications.\nCascade of Einsums: which tensor operations run and how they depend on one another Mapping: in what order the iteration space is traversed and how it is tiled and parallelized Format: how tensors are represented, such as dense, CSR, or COO Binding: where computation and data are assigned among the physical PEs, registers, buffers, and networks Descriptions are shorter near the top and accumulate implementation decisions toward the bottom. An Einsum defines only the computation. It says nothing yet about loop order or dataflow.\nThe evaluation stage derives compute count, memory traffic, and compute intensity. Comparisons between implementations must hold hardware conditions such as PE count, storage capacity, and bit width constant. Only the specification responsible for a bottleneck is then changed before evaluating again. This iterative process is what TeAAL targets. Instead of comparing complete accelerator diagrams as indivisible objects, it isolates whether a difference came from computation, mapping, format, or binding.\nTensor Terminology A tensor is a multidimensional array. A scalar is zero-dimensional, a vector is one-dimensional, a matrix is two-dimensional, and a cube is three-dimensional.\nThis course calls each dimension a rank. This use is different from matrix rank in linear algebra.\nNumber of ranks: the number of dimensions Rank shape: the number of elements along each dimension Tensor shape: the ordered list of rank shapes Tensor size: the product of all rank shapes, or the total number of elements For example, \\(B[N,K]\\) is a rank-2 tensor. Its rank names are \\(N,K\\), its shape is \\([N,K]\\), and its size is \\(NK\\). Rank names carry more meaning than numeric positions alone. Within a workload, \\(N\\) can identify the batch rank and \\(C\\) the input-channel rank.\nThe lecture\u0026rsquo;s matrix multiplication diagram labels the tensor shapes as \\(A[M,K]\\), \\(B[N,K]\\), and \\(Z[M,N]\\). The following Einsum places \\(k\\), the reduction rank, first and writes the operation as\n$$ Z_{m,n} = \\sum_k A_{k,m} B_{k,n}. $$\\(K\\) is the rank shared and reduced by both inputs. \\(M,N\\) remain in the output. The order of subscripts in an Einsum should not be read directly as the physical memory layout. Matching rank names define the contraction in an Einsum; the physical rank order and storage representation are separate format and mapping decisions.\nEinsum and Operational Definition Einstein summation notation implies reductions from the indices on each side.\n$$ Z_{m,n} = A_{k,m} B_{k,n} $$The \\(k\\) that appears only on the right is summed. The expression means the same thing without writing \\(\\sum_k\\) explicitly. Reducing it to matrix-vector multiplication makes the notation simpler.\n$$ Z_m = A_{k,m} B_k $$This one line specifies the input tensors \\(A,B\\), the output tensor \\(Z\\), and the multiplication and reduction performed at each point. One thing is missing: the execution order.\nTeAAL\u0026rsquo;s Operational Definition of an Einsum reads the expression in three parts.\nThe input and output tensors and their ranks The iteration space, defined as the Cartesian product of every legal coordinate The operation performed at each iteration point The iteration space of the matrix-vector multiplication above is \\(K \\times M\\). With \\(K=8\\) and \\(M=6\\), it contains 48 points. Point \\((4,2)\\) represents one operation: multiply \\(A_{4,2}\\) by \\(B_4\\) and add the result to \\(Z_2\\).\nThe work at each point is fixed.\nSelect \\(A_{k,m}\\) and \\(B_k\\) Multiply the two values Update \\(Z_m\\) Reduce by addition because multiple \\(k\\) values contribute to the same \\(m\\) Every point in the iteration space must be visited. The expression does not say whether to traverse \\(k\\) or \\(m\\) first, how many points to group into a tile, or which rank to distribute across PEs.\nKeeping the distinction between the Einsum as the algorithm and the mapping as its execution order makes the later dataflows easier to follow.\nWorkload Analysis Operation Count Each of the \\(K \\times M\\) points performs one multiplication, so\n$$ N_{\\text{mul}} = KM. $$Each output \\(Z_m\\) is the sum of \\(K\\) products. The exact number of additions is\n$$ N_{\\text{add}} = (K-1)M. $$The first product initializes an empty partial sum, while the remaining \\(K-1\\) products require additions.\nThese counts do not depend on processing order. Unless an algorithmic optimization such as zero skipping is applied to dense inputs, no mapping can avoid the \\(KM\\) useful multiplications.\nBest-case Memory Traffic The lecture defines compute intensity in units of multiplications/value:\n$$ \\text{CI} = \\frac{\\text{number of multiplications}} {\\text{number of values moved}}. $$The conventional Roofline model uses FLOPs/byte. The lecture\u0026rsquo;s definition temporarily removes differences caused by whether a MAC counts as one or two operations and whether each value is FP32 or INT8. This makes workload reuse easier to isolate.\nIf every tensor element moves from DRAM only the minimum number of times required, the traffic is\n\\(A\\): \\(KM\\) value loads \\(B\\): \\(K\\) value loads \\(Z\\): \\(M\\) value stores $$ T_{\\text{best}} = KM + K + M $$and\n$$ \\text{CI}_{\\text{best}} = \\frac{KM}{KM+K+M}. $$Each \\(A_{k,m}\\) is unique to one iteration point, so it has no reuse in this expression. Each \\(B_k\\) is reused \\(M\\) times across all \\(m\\). Each \\(Z_m\\) has to remain in local storage until the reduction over all \\(k\\) is complete.\nThe best case assumes that all of this reuse can be retained. It is a workload-level upper bound that does not yet account for register count or processing order.\nMapping and Data Reuse The iteration space can be traversed in several directions. The most direct loop nest places \\(k\\) outside.\nfor k in range(K): for m in range(M): Z[m] += A[k, m] * B[k] The simple architecture has only one register. Making its loads and stores explicit gives\nfor k in range(K): b_reg = B[k] for m in range(M): a_reg = A[k, m] z_reg = Z[m] z_reg += a_reg * b_reg Z[m] = z_reg \\(B_k\\) is loaded once when the outer loop advances, then used \\(M\\) times by the inner \\(m\\) loop. \\(B\\) is stationary. In contrast, \\(Z_m\\) returns to DRAM after the update for the current \\(k\\). A single register cannot carry all \\(M\\) partial sums forward to the next \\(k\\).\nThe traffic for this mapping is\n$$ \\begin{aligned} A\\text{ loads} \u0026= KM \\\\ B\\text{ loads} \u0026= K \\\\ Z\\text{ loads} \u0026= (K-1)M \\\\ Z\\text{ stores} \u0026= KM. \\end{aligned} $$For the first \\(k\\), \\(Z\\) starts at zero and does not need to be loaded. This is why the number of \\(Z\\) loads is \\((K-1)M\\), not \\(KM\\).\n$$ T_{\\text{achieved}} = 3KM-M+K $$and\n$$ \\text{CI}_{\\text{achieved}} = \\frac{KM}{3KM-M+K}. $$\nFor \\(K=250\\) and \\(M=100\\),\n$$ \\text{CI}_{\\text{best}} = \\frac{250 \\times 100} {250 \\times 100 + 250 + 100} \\approx 0.99, $$while\n$$ \\text{CI}_{\\text{achieved}} = \\frac{250 \\times 100} {3(250 \\times 100)-100+250} \\approx 0.33. $$Both perform 25,000 multiplications. The difference comes from repeatedly writing the \\(Z\\) partial sums to DRAM and loading them back.\nReversing the loops and placing \\(m\\) outside keeps \\(Z_m\\) in a register.\nfor m in range(M): z_reg = 0 for k in range(K): z_reg += A[k, m] * B[k] Z[m] = z_reg This mapping is close to output-stationary. Without a cache or separate buffer, however, it reads \\(B_k\\) again for every \\(m\\). One small register cannot preserve reuse of both \\(B\\) and \\(Z\\) at the same time.\nPreserving both requires more storage. Several \\(Z\\) tiles can reside in a local buffer, a \\(B\\) tile can be multicast to several PEs, or \\(K\\) and \\(M\\) can be blocked to create a region where the two kinds of reuse overlap. Mapping is not merely loop reordering. It selects the reuse that the architecture\u0026rsquo;s storage and network can support.\nRoofline Model Roofline expresses the throughput ceiling as the smaller of two terms:\n$$ \\text{Throughput} \\le \\min(P_{\\text{peak}}, BW \\times \\text{CI}). $$ \\(P_{\\text{peak}}\\): maximum throughput of the compute hardware \\(BW\\): memory bandwidth \\(\\text{CI}\\): multiplications performed per value moved At low CI, \\(BW \\times \\text{CI}\\) is the limiting term. This is the sloped, memory-bound region of the graph. Once CI is high enough to reach the horizontal line at \\(P_{\\text{peak}}\\), the implementation becomes compute-bound.\nThe compute roof in the slide\u0026rsquo;s example is 8 MACs/cycle. Increasing the lane count from 8 to 16 does not improve the current throughput of a memory-bound point; it only raises the horizontal roof. Increasing reuse so that CI moves from 0.33 to 0.99, on the other hand, raises throughput at the same bandwidth.\nRoofline answers three questions.\nIs the current implementation limited by compute or memory? Should parallelism or bandwidth be increased? How far is the current point from the applicable roof? A point well below the roof indicates losses beyond peak compute and memory bandwidth. Possible causes include pipeline stalls, instruction overhead, mapping limitations, and load imbalance.\nThis completes the design loop in the first half of the lecture: calculate the workload\u0026rsquo;s best-case CI, calculate the mapping\u0026rsquo;s achieved CI, and locate the bottleneck on the Roofline chart. Change the architecture or mapping, then repeat the same analysis.\nCNN Workload Overview CNNs are used not only for image classification but also for speech spectrograms, medical imaging, and game play. Although their inputs differ, they share the structure of scanning for local patterns with filters and building a hierarchy of features.\nEarly convolution layers detect low-level features such as edges and textures. In deeper layers, multiple pixels and features from preceding layers combine into representations closer to object parts or classes. Modern CNNs range from tens to hundreds of layers, with some approaching 1,000 layers.\nThe basic block applies an activation after convolution. The activation is a nonlinear function such as ReLU. A fully connected layer similarly applies an activation after a linear operation. Normalization and pooling may appear between these blocks.\nConvolution: a weighted sum over a local receptive field Activation: an element-wise nonlinearity Normalization: adjustment of the activation distribution or scales across channels Pooling: spatial downsampling and local aggregation Fully connected: dense connections between every input activation and output neuron In classical deep CNNs, convolution often accounts for more than 90% of all operations.\nPooling and activation still have to run, but CONV dominates multiplication count, runtime, and energy. This is why L02 spends much more time on convolution than on the other layers.\n2D Convolution Element-wise Product and Partial Sum In the smallest case, there is one input feature map and one filter. The input is \\(H \\times W\\), and the filter is \\(R \\times S\\).\nPlace the filter over one input position and multiply corresponding elements. Summing the \\(RS\\) products produces one output activation. The intermediate value being accumulated is the partial sum, usually written as psum.\nMoving the filter window horizontally and vertically fills the output feature map. The \\(R \\times S\\) region covered by the filter is that output activation\u0026rsquo;s receptive field.\nThe slide example uses a 5×5 input and a 3×3 filter. With stride 1 and padding 0, the output is 3×3.\nThe output shape is\n$$ P = \\left\\lfloor \\frac{H-R}{U} \\right\\rfloor + 1, \\qquad Q = \\left\\lfloor \\frac{W-S}{U} \\right\\rfloor + 1, $$where \\(U\\) is the stride.\nEach output point uses \\(RS=9\\) multiplications, and there are \\(PQ=9\\) output points. The total number of multiplications is therefore\n$$ PQRS = 3 \\times 3 \\times 3 \\times 3 = 81. $$Zeros in the filter still count as multiplications unless the hardware supports sparse zero skipping.\nSlides L02-58 through L02-63 animate the window moving one position at a time to fill the 3×3 output. The completed L02-64 image above is the result of those six frames.\nStride Stride is the number of positions by which the filter window moves at each step.\n\\(U=1\\): 3×3 output, 81 multiplications \\(U=2\\): 2×2 output, 36 multiplications \\(U=3\\): 1×1 output, 9 multiplications The stride-2 and stride-3 results are equivalent to sampling the stride-1 output every two and three positions, respectively. Slides L02-65 through L02-70 animate these window movements.\nA larger stride reduces the number of output activations and the amount of computation, but it also samples spatial information more coarsely. From the hardware perspective, the smaller \\(P,Q\\) change both the iteration space and the input reuse pattern.\nZero Padding Without padding, each convolution reduces the spatial dimensions.\nAdding \\(D\\) positions of zero padding around the input gives the output shape\n$$ P = \\left\\lfloor \\frac{H+2D-R}{U} \\right\\rfloor + 1, \\qquad Q = \\left\\lfloor \\frac{W+2D-S}{U} \\right\\rfloor + 1. $$With a 3×3 filter, stride 1, and \\(D=1\\), the input and output have the same \\(H,W\\). PyTorch\u0026rsquo;s Conv2d defaults to padding 0. An integer applies the same padding on all sides, while a tuple specifies the height and width directions separately.\nA dense implementation may read and compute padded zeros just like ordinary input values. Handling the boundary separately or generating zeros implicitly can avoid that memory traffic at the cost of more complex control.\nReceptive Field As the network grows deeper, each output activation depends on a larger region of the original input. With only 3×3 filters, stride 1, and dilation 1, the receptive-field width grows by two per layer:\n$$ r_L = 1 + 2L. $$Layer 1 sees a 3×3 region, layer 2 sees 5×5, and layer 3 sees 7×7. With stride or dilation, the increment must also account for the sampling jump inherited from preceding layers.\nThis is the spatial basis for the usual explanation that CNNs progress from low-level to high-level features. For an accelerator, it also creates opportunities to reuse one input activation across neighboring outputs and multiple layers.\nOperations called convolution in deep learning libraries are usually cross-correlations that do not flip the filter. The slide\u0026rsquo;s index expression and naive loop follow this convention. A learned filter makes the distinction irrelevant to model behavior, but it matters when comparing the code with the signal-processing definition of convolution.\nMulti-channel Convolution Tensor Shapes A real CNN input is not a single feature map. Even an RGB image has three channels, and intermediate layers can have tens to thousands of channels.\nTo produce one output channel, a filter spans all \\(C\\) input channels. If there are \\(M\\) output channels, there are \\(M\\) such filters.\nAdding a batch applies the same filter set to \\(N\\) input feature maps.\nThe lecture uses the following symbols.\nSymbol Rank shape \\(N\\) Batch size \\(C\\) Input channels \\(H,W\\) Input height, width \\(R,S\\) Filter height, width \\(M\\) Output channels, number of filters \\(P,Q\\) Output height, width \\(U\\) Stride Grouped into tensor shapes, these are\n$$ I[N,C,H,W], $$$$ F[M,C,R,S], $$$$ O[N,M,P,Q], $$and\n$$ B[M]. $$\\(I\\) contains the input activations, \\(F\\) the filter weights, \\(O\\) the output activations, and \\(B\\) one bias value for each output channel.\nThe weight size is \\(MCRS\\), the input-activation size is \\(NCHW\\), and the output-activation size is \\(NMPQ\\). In addition to computation, these three quantities determine the capacity required at each memory level.\nConvolution Einsum The complete convolution can be written in one expression:\n$$ O_{n,m,p,q} = B_m + I_{n,c,Up+r,Uq+s} F_{m,c,r,s}. $$The \\(n,m,p,q\\) ranks remain in the output, while \\(c,r,s\\) are reduced. Stride and the sliding window appear in the input\u0026rsquo;s spatial coordinates, \\(Up+r,Uq+s\\), rather than simply \\(p,q\\).\nThe multiplication count equals the size of the iteration space:\n$$ N_{\\text{mul}} = NMPQCRS. $$Each output reduces \\(CRS\\) products, and there are \\(NMPQ\\) outputs.\nThe reuse direction of each tensor can also be read from the expression.\nFilter \\(F_{m,c,r,s}\\): reused along \\(n,p,q\\) Input \\(I_{n,c,Up+r,Uq+s}\\): reused across multiple \\(m\\) values and overlapping windows Output \\(O_{n,m,p,q}\\): reused as a partial sum during the \\(c,r,s\\) reduction Bias \\(B_m\\): reused across all \\(n,p,q\\) The dataflow determines which kinds of reuse are preserved in local registers and buffers.\nSeven-loop Implementation for n in range(N): for m in range(M): for q in range(Q): for p in range(P): O[n, m, p, q] = B[m] for c in range(C): for r in range(R): for s in range(S): O[n, m, p, q] += ( I[n, c, U*p+r, U*q+s] * F[m, c, r, s] ) O[n, m, p, q] = activation(O[n, m, p, q]) This loop enforces the order \\(s \\rightarrow r \\rightarrow c \\rightarrow p \\rightarrow q \\rightarrow m \\rightarrow n\\). \\(O\\) remains stationary through the inner \\(c,r,s\\) loops, which makes it easy to reduce partial-sum traffic. Without a cache or explicit tiling, however, it misses the longer-range reuse of filters and inputs.\nThe Einsum imposes none of this order. The mapping stage can apply loop interchange, tiling, and spatial unrolling. This is where weight-stationary, output-stationary, and row-stationary dataflows diverge from the same CONV expression.\nFully Connected Layer Connectivity In a fully connected layer, a weight connects every input neuron to every output neuron. With \\(K\\) inputs and \\(M\\) outputs, the layer has \\(MK\\) weights.\nThe sparsely connected variant in the figure retains only some of the edges. A pruned FC layer can take this form. Its actual benefit must include the indexing and control costs of storing sparse weights and skipping zeros.\nFC as Convolution From the CONV perspective, FC is the case in which a filter covers the entire input feature map.\n$$ R=H,\\qquad S=W,\\qquad P=Q=1. $$For a batch size of one, the expression becomes\n$$ O_m = I_{c,h,w} F_{m,c,h,w}. $$All of \\(c,h,w\\) are reduction ranks. Each output \\(m\\) is the dot product of the entire input feature map and its corresponding filter.\nFlattening The three ranks \\(C,H,W\\) can be flattened into one \\(CHW\\) rank.\nFor a row-major layout, the coordinate transformation is\n$$ chw = H W c + W h + w. $$Thus\n$$ I_{c,h,w} \\rightarrow I_{chw} $$and\n$$ F_{m,c,h,w} \\rightarrow F_{m,chw}. $$The FC Einsum becomes\n$$ O_m = I_{chw} F_{m,chw}. $$\nFlattening does not change the operation count. It reindexes three nested reduction loops as one linear loop. If the memory layout matches this flattening order, accesses are contiguous; otherwise a transpose or strided accesses are required.\nGEMV and GEMM An FC layer with a batch size of one is a matrix-vector multiplication:\n$$ \\underbrace{F[M,CHW]}_{\\text{matrix}} \\times \\underbrace{I[CHW]}_{\\text{vector}} = \\underbrace{O[M]}_{\\text{vector}}. $$Slides L02-92 through L02-97 animate the partial sum as \\(chw\\) advances, then change \\(m\\) to compute the next output.\nWith a batch of \\(N\\), an \\(n\\) rank is added to the input and output:\n$$ O_{n,m} = I_{n,chw} F_{m,chw}. $$\nThis is now matrix-matrix multiplication:\n$$ F[M,K] \\times I[K,N] = O[M,N], $$where \\(K=CHW\\).\nThe lecture writes the FC expression as \\(O_{n,m}\\), while conventional matrix multiplication writes \\(C_{m,n}=A_{m,k}B_{k,n}\\), so the rank orders look different. In an Einsum, the computational relationship is unchanged as long as matching rank names connect and the reduction rank agrees. Which of \\(N\\) and \\(M\\) is contiguous in the physical memory layout is a separate question.\nBatch size also affects hardware efficiency. GEMV reads a weight matrix, applies it to one vector, and finishes, so it has little weight reuse and is often memory-bound. GEMM can reuse the same weight tile across \\(N\\) inputs and therefore reaches higher compute intensity.\nCONV can likewise be transformed into GEMM by expanding its input windows with im2col. Materializing a large im2col matrix increases the memory footprint because activations are duplicated. This is why high-performance libraries use implicit GEMM or dedicated convolution kernels.\nSlide Coverage Slides Content L02-1 ~ 3 Lecture scope and workload-to-hardware framing L02-4 ~ 8 Design methodology, architecture/workload separation, TeAAL concerns L02-9 ~ 14 Tensor rank, shape, size, matrix multiplication, and Einsum L02-15 ~ 20 Matrix-vector ODE, iteration space, and reduction L02-21 ~ 26 Operation count, best-case traffic, and CI L02-27 ~ 41 Loop traversal, stationarity, achieved traffic, and CI L02-42 ~ 44 Roofline, implementation comparison, and optimization loop L02-45 ~ 52 CNN applications, depth, and CONV/FC/NORM/POOL L02-53 ~ 64 2D convolution and stride-1 animation L02-65 ~ 71 Stride-2/3 animation and downsampling L02-72 ~ 75 Zero padding, PyTorch convention, and receptive field L02-76 ~ 83 Channel and batch tensors, decoder ring, CONV Einsum, and seven-loop nest L02-84 ~ 91 FC connectivity, CONV interpretation, and flattening L02-92 ~ 99 GEMV partial-sum animation and flattened FC Einsum L02-100 ~ 102 Batched FC, GEMM, and conventional matmul notation References MIT 6.5930/1 Spring 2026 L02 slides TeAAL: A Declarative Framework for Modeling Sparse Tensor Accelerators, MICRO 2023 TeAAL and HiFiber tutorial Roofline: An Insightful Visual Performance Model for Multicore Architectures Efficient Processing of Deep Neural Networks, Sze et al. ","permalink":"https://bias92.github.io/en/posts/mit-6.5930-l02/","summary":"Complete L02 notes: accelerator design methodology, tensors and Einsum, iteration spaces, memory traffic, compute intensity, Roofline analysis, CNN convolution, and mapping fully connected layers to GEMV/GEMM.","title":"6.5930 L02 - From Einsum to DNN Workloads"},{"content":"A function that runs on the GPU is called a kernel, and the unit of work that executes a kernel is a thread. Threads are grouped into blocks, and one block runs from start to finish on a single SM (Streaming Multiprocessor), the execution unit inside the GPU. The threads of a block read and write two kinds of memory. Global memory is the large memory attached to the GPU that every thread can reach, and shared memory is a small memory inside the SM that only the threads of the same block use together. Shared memory is far faster than global memory, but the kernel code itself decides what to load into it and when to discard it, which makes it different from a cache that hardware fills on its own.\nShared memory lets a block reuse data that was read from global memory once. In exchange, because the threads of a block use the same memory together, three new problems appear: a barrier that orders writes and reads, a bank conflict that arises when several threads touch the same storage unit at the same time, and an occupancy problem where a larger block reduces the number of threads an SM can hold. Matrix multiplication, transpose, and reduction expose these three in turn. The starting point is how the data that will be loaded into shared memory is read from global memory.\nGlobal Memory and Coalescing The GPU does not run threads one at a time. It bundles 32 of them and issues the same instruction to all of them at once; this bundle of 32 is a warp, and each thread\u0026rsquo;s slot inside the warp is a lane. When a warp executes a global memory load, the 32 lanes each present their own address. Global memory does not sell single bytes: it moves data only in 32-byte chunks called sectors, whose boundaries are fixed at 32-byte intervals from address 0, so even if only one byte inside a sector is needed the whole sector is transferred. Merging the accesses of many lanes into a small number of sector transfers is called coalescing. So the cost of a global memory access is set not by the number of lanes but by the number of distinct sectors actually touched.\nA float is 4 bytes, so 32 lanes reading 32 consecutive floats cover a 128-byte range, and if the starting address falls on a sector boundary (a multiple of 32) that takes 4 sectors. If the starting address is shifted by one float, the same 128 bytes straddle one more sector boundary and take 5, and if the gap between lanes grows to 32 bytes or more, each lane lands in a different sector and the count grows to 32. In all three cases the useful data is the same 128 bytes, but the amount actually transferred is 128, 160, and 1024 bytes.\nThe pointer returned by cudaMalloc is sufficiently aligned, but the start of a sub-range made by adding an offset to it may not be. And the number 4 comes from the condition \u0026ldquo;32 lanes, one float each\u0026rdquo;, so when the lane count or the data width changes, the minimum sector count changes with it.\nShared Memory and Tiling In the matrix product $C = A \\times B$, one element of $C$ needs one row of $A$ and one column of $B$. The simplest kernel gives each thread one element of $C$ and reads every value it needs from global memory each time.\n__global__ void matmul_naive(const float* A, const float* B, float* C, int N) { int row = blockIdx.y * blockDim.y + threadIdx.y; int col = blockIdx.x * blockDim.x + threadIdx.x; float acc = 0.0f; for (int k = 0; k \u0026lt; N; k++) acc += A[row * N + k] * B[k * N + col]; C[row * N + col] = acc; } Here threadIdx is the thread\u0026rsquo;s index inside its block, blockIdx is the block\u0026rsquo;s index inside the grid, and blockDim is the number of threads along one side of the block. Each iteration of the inner loop reads 8 bytes and does one multiply and one add. But the same row of $A$ is needed again for all $N$ elements in that row of $C$, so this kernel reads the same values from global memory many times.\nTiling manages that reuse directly in shared memory. A tile is a $T \\times T$ square piece of a matrix. The threads of a block together copy one tile of $A$ and one tile of $B$ into shared memory, accumulate every partial product that tile pair allows, and then move on to the next tile pair. In this way each element of a tile is read from global memory once and reused $T$ times from shared memory.\n#define T 32 __global__ void matmul_tiled(const float* A, const float* B, float* C, int N) { __shared__ float As[T][T]; __shared__ float Bs[T][T]; int row = blockIdx.y * T + threadIdx.y; int col = blockIdx.x * T + threadIdx.x; float acc = 0.0f; for (int t = 0; t \u0026lt; N / T; t++) { As[threadIdx.y][threadIdx.x] = A[row * N + (t * T + threadIdx.x)]; Bs[threadIdx.y][threadIdx.x] = B[(t * T + threadIdx.y) * N + col]; __syncthreads(); for (int k = 0; k \u0026lt; T; k++) acc += As[threadIdx.y][k] * Bs[k][threadIdx.x]; __syncthreads(); } C[row * N + col] = acc; } __shared__ declares that a variable lives in shared memory. __syncthreads() is a barrier: threads that arrive at this line wait until every thread of the block has arrived. The first barrier keeps a thread from reading a tile that other threads have not finished filling, and the second keeps the next iteration from overwriting a tile that is still being read. The barrier comes with a rule. If some threads of a block return early while the rest reach __syncthreads(), the behavior is undefined. The code above has every thread follow the same path because $N$ is a multiple of $T$; code that accepts arbitrary sizes keeps out-of-range threads in the loop and the barrier instead of returning them, fills their loads with 0, and guards only the final store to C with a range check.\nThe reuse shows up as arithmetic intensity, the number of operations performed per byte read from global memory. One block copies $N/T$ tile pairs, reading $8NT$ bytes, and its $T^2$ threads each perform $2N$ operations, so\n$$ I_{\\text{tiled}} = \\frac{2NT^2}{8NT} = \\frac{T}{4}\\ \\text{FLOP/B} $$With $T = 32$ that is 8 FLOP/B, 32 times the 0.25 FLOP/B of reading every value without a tile. The tile copies read consecutive addresses along a row, so they also satisfy the coalescing condition from the previous section.\nBank Conflict Shared memory is divided into 32 banks. A bank is an independent storage unit that makes up shared memory, and consecutive 4-byte words are assigned to banks 0, 1, \u0026hellip;, 31, 0, 1, \u0026hellip; in rotation. When the 32 lanes of a warp touch different banks, they are served at once; when they touch different addresses in the same bank, they are served one after another. This serialization of $n$ accesses to the same bank is an $n$-way bank conflict. Several lanes reading the same address is a broadcast that hands one value to all of them, so it is not a conflict.\nConflicts typically appear when a two-dimensional tile is read column-wise, and a transpose through shared memory is the example. Transpose swaps the rows and columns of a matrix; by placing a tile read row-wise from global memory into shared memory and taking it out column-wise, both the read and the write can be coalesced.\n__shared__ float tile[32][32]; tile[threadIdx.y][threadIdx.x] = in[...]; // row-wise write: banks spread __syncthreads(); out[...] = tile[threadIdx.x][threadIdx.y]; // column-wise read: 32-way conflict In the column-wise read, the lanes read tile[0][c], tile[1][c], ..., which are 32 words apart, so all of them land in bank c. There are two ways to fix this.\nThe first is padding. Making the row length 33 shifts the bank number by one per row as the access walks down a column.\n__shared__ float tile[32][33]; In general, reading with stride $S$ gives a conflict degree of $\\gcd(S, 32)$. Stride 32 is 32-way, 33 is conflict-free, and 2 and 4 are 2-way and 4-way. Padding is therefore the act of making the common divisor of the stride and the bank count equal to 1, at a cost of 4 wasted bytes per row.\nThe second is swizzle. XOR-ing the column index with the row number at store time gives the same effect without using more memory.\n__shared__ float tile[32][32]; tile[threadIdx.y][threadIdx.x ^ threadIdx.y] = in[...]; // write: banks spread __syncthreads(); out[...] = tile[threadIdx.x][threadIdx.y ^ threadIdx.x]; // read also spread Once the columns inside a row are rearranged by XOR, both row-wise and column-wise accesses land on each of the 32 banks exactly once. The cost is one index computation and the requirement that the tile width be a power of two.\nIt is also possible to flip the mapping between lanes and rows and columns so that the shared memory access becomes consecutive, but then the global memory access becomes strided again. That moves the conflict on one side into a coalescing failure on the other, so it is not a fix. The tiled matrix multiply in the previous section does not have this problem. Bs[k][threadIdx.x] runs along a row, so the banks spread, and As[threadIdx.y][k] is a broadcast because the lanes of a warp read the same address.\nOccupancy and Block Size Occupancy is the number of warps actually resident on an SM divided by the maximum number of warps the SM can hold at once. It matters because of how the GPU hides memory latency. While one warp waits for a global memory response, the SM runs another warp that is resident on the same SM, so the more warps are resident, the more of the waiting is covered. The number of blocks an SM can hold is set by whichever of four limits runs out first: threads, block slots, registers, and shared memory. A register is the fastest storage inside the SM, where a thread keeps the values it is computing with.\nIn the tiled matrix multiply, $T = 32$ makes one block 1024 threads. Compute capability is the number that identifies the generation of CUDA hardware features a GPU supports, and a GPU with compute capability 8.9 has a resident-thread limit of 1536 per SM, so only one 1024-thread block fits on an SM. That fills 1024 of the 1536 thread slots, and occupancy becomes 66.7%. With $T = 16$ the block is 256 threads, six of them fit, and all 1536 slots are filled.\nEnlarging the tile increases reuse but also enlarges the block and reduces the parallelism an SM can hold. This is why the way to stack more reuse is not a bigger block but register tiling, where one thread keeps several elements of $C$ in registers and accumulates them. And occupancy is only an upper bound on the number of warps the SM can choose from, not a fraction of execution time, so a kernel with low occupancy can still be fast if each warp has plenty to do.\nWarp Divergence A warp issues the same instruction to its 32 lanes at once. When the condition of an if differs from lane to lane, some lanes of the warp must take the true path and the rest the false path. This situation is warp divergence. With lane number $\\ell \\in \\{0,\\ldots,31\\}$ and branch condition $p_\\ell$, the set $A$ of true lanes and the set $B$ of false lanes are\n$$ A = \\{\\ell \\mid p_\\ell = 1\\}, \\qquad B = \\{\\ell \\mid p_\\ell = 0\\} $$and divergence occurs when both sets are non-empty.\nint lane = threadIdx.x \u0026amp; 31; if (lane \u0026lt; 16) A(); else B(); In this code, lanes 0–15 of every warp choose A and lanes 16–31 choose B. A warp cannot run both paths at once, so it first runs the A path with only lanes 0–15 enabled, and then the B path with only lanes 16–31 enabled. The 32-bit value that records which lanes write the result of the current instruction is the active mask.\nSuppose the A and B paths compile to $n_A$ and $n_B$ warp instructions. If only one path is chosen, that region issues $n_A$ instructions; if both are chosen, it issues $n_A + n_B$. The lane count does not multiply the issue count, so whether the split is 16:16 or 31:1, the number of instructions issued is the same if the two paths are the same length. With $\\eta$ as the fraction of issued lane slots that are actually enabled,\n$$ \\eta = \\frac{|A|n_A + |B|n_B} {32(n_A+n_B)} $$and when the two paths are the same length, $n_A = n_B$, this gives $\\eta = 1/2$.\nTo avoid divergence, align the condition with warp boundaries.\nint warp = threadIdx.x \u0026gt;\u0026gt; 5; if ((warp \u0026amp; 1) == 0) A(); else B(); Even-numbered warps choose A with all 32 lanes and odd-numbered warps choose B with all 32, so the condition is uniform inside each warp. Different warps running different code is not divergence.\nA diverging if in the source does not always produce a real branch either. When the body is short, the compiler removes the branch and turns it into a predicated instruction. Predication issues the instruction once to all lanes and lets only the lanes whose condition is true write the result.\nfloat y = x; if (lane \u0026lt; 16) y = 2.0f * x; In SASS, the GPU machine code, this conceptually becomes the form below. The exact instruction names and registers vary with architecture and compiler version.\nISETP.LT ... P0, lane, 16 @P0 FMUL y, x, 2.0 In this case the warp does not split into two paths. FMUL is issued once and only the lanes with predicate P0 true write the result. So there is no branch divergence, but the disabled lanes do no useful work on that instruction. A divergent branch and predication can both lower the fraction of enabled lanes, but they are not the same phenomenon.\nReduction Reduction turns an array of $N$ values into a single value. Sums, maxima, and means belong here, and they keep appearing inside ML kernels as the maximum and denominator sum of softmax and the mean and variance of layernorm.\nFolding as a tree, half of the threads combine two values at each step and the number of steps is $\\log_2 N$. The total number of additions is still $N - 1$, so parallelization reduces depth, not the amount of work. In return, every step needs a guarantee that the previous writes have finished, and how far that synchronization cost is reduced is the difference between the four versions below.\nThe common structure is multi-pass. Each block forms its share of the partial sum in shared memory, and the same kernel is run again on the array of partial sums until one value remains. For example, with $2^{24}$ inputs and a block of 256, the kernel runs three times: 65,536 → 256 → 1.\nVersion 0 is the literal tree. tid is the thread\u0026rsquo;s index inside the block, and buf is the input the block has loaded into shared memory.\nfor (int s = 1; s \u0026lt; blockDim.x; s *= 2) { if (tid % (2 * s) == 0) buf[tid] += buf[tid + s]; __syncthreads(); } Two things make it slow. The first is scattered active lanes. The lanes that satisfy the condition thin out to the even-numbered ones at s = 1 and multiples of four at s = 2, and although the number of working lanes drops to 16, 8, 4, \u0026hellip;, the warp\u0026rsquo;s instructions are still issued. The second is the % operation. The divisor 2 * s changes every iteration, so the compiler cannot turn it into a bit operation and a cluster of division instructions remains.\nVersion 1 is sequential addressing. It gathers the working threads at the front of the block.\nfor (int s = blockDim.x / 2; s \u0026gt; 0; s \u0026gt;\u0026gt;= 1) { if (tid \u0026lt; s) buf[tid] += buf[tid + s]; __syncthreads(); } Both versions reduce 256 elements in eight steps, and the number of threads doing an addition at step $j \\in \\{0,\\ldots,7\\}$ is the same, $a_j = 256 / 2^{j+1}$. The difference is how many warps those $a_j$ threads are spread across. In version 0 the active threads are scattered across the whole block, so the number of warps with at least one true condition is $\\min(8, a_j)$, which sums to 47 over the eight steps. Version 1 packs the active threads at the front, giving $\\lceil a_j / 32 \\rceil$, which sums to 12. So at s = 128 the first four warps work as whole warps, at s = 64 two, and at s = 32 one, and in these three steps the condition is uniform per warp. From s = 16 on, the condition splits inside the first warp, but the active lanes are contiguous from the front. buf[tid] and buf[tid + s] are also consecutive addresses, so there is no bank conflict, and the % is gone.\nVersion 2 is warp shuffle. From s = 16 on, only the first warp works, so from that boundary the fold happens in registers without shared memory or __syncthreads().\nif (tid \u0026lt; 32) { float x = buf[tid] + buf[tid + 32]; for (int off = 16; off \u0026gt; 0; off \u0026gt;\u0026gt;= 1) x += __shfl_down_sync(0xffffffffu, x, off); if (tid == 0) out[blockIdx.x] = x; } __shfl_down_sync passes a register value directly between lanes inside a warp; its first argument is a mask of the participating lanes and its third is how many lanes below to take the value from. This removes the shared memory round trips and block barriers of the last six steps. The mask 0xffffffffu is valid because the whole first warp satisfies tid \u0026lt; 32; in code where only some lanes participate, every participating lane must execute the same function with the same mask.\nVersion 3 is one atomic per block. An atomic is an operation that guarantees updates to the same address from many threads are applied one at a time even when they arrive together. Instead of multi-pass, lane 0 of each block executes atomicAdd(out, x) once to add the block\u0026rsquo;s partial sum straight into the result. An atomic per element would touch the same address $N$ times, but after the block reduction it is touched only as many times as there are blocks. This version needs a cudaMemset to zero the result variable before it runs, and float atomics add in an order that changes from run to run, so they do not guarantee bit-identical results.\nA reduction with one operation per element is purely bound by memory bandwidth, so the ceiling for a well-written reduction is the speed of a memcpy of the same size. DeviceReduce::Sum from CUB, the library shipped with CUDA, reaches that range for arbitrary types and sizes, and version 3 above shows the same structure fixed to a single float array.\nSource Code The full sources of the three kernels are in gemm_bench.cu, transpose_bench.cu, and reduce_bench.cu.\nnvcc -O3 -arch=sm_89 -o gemm_bench gemm_bench.cu nvcc -O3 -arch=sm_89 -o transpose_bench transpose_bench.cu nvcc -O3 -arch=sm_89 -std=c++17 -o reduce_bench reduce_bench.cu References CUDA C++ Best Practices Guide: the reference document for coalescing, shared memory, bank conflicts, occupancy, and branch predication CUDA C++ Programming Guide: the exact meaning of SIMT divergence, synchronization, atomics, and warp intrinsics Mark Harris, An Efficient Matrix Transpose in CUDA C/C++: the standard example showing coalescing, shared tiles, and padding in one place Andreas Holt, Shared-Memory Tiled Matrix Multiplication: tiled GEMM with figures and boundary handling Lei Mao, CUDA Shared Memory Bank: details of bank address mapping Lei Mao, CUDA Shared Memory Swizzling: details of swizzle address mapping Fabian Schütze, Visualizing Bank Conflicts: supplementary notes on bank behavior in modern architectures Mark Harris, Optimizing Parallel Reduction in CUDA: the classic seven-step reduction; it is old, so do not copy its warp-synchronous code as is Faster Parallel Reductions on Kepler: shuffle and hierarchical atomics; read the code with the modern __shfl_down_sync() Lei Mao, CUDA Reduction: a write-up centered on a batched reduction implementation CUTLASS: Efficient GEMM in CUDA: the higher-level reference on threadblock/warp/thread tiling, register reuse, and double buffering Simon Boehm, How to Optimize a CUDA Matmul Kernel: a worklog from register tiling up to warptiling CUB: the production implementation layered as WarpReduce → BlockReduce → DeviceReduce ","permalink":"https://bias92.github.io/en/posts/cuda-3-shared-memory/","summary":"Global memory coalescing, shared memory tiling, bank conflicts with padding and swizzle, occupancy, warp divergence and predication, and four stages of reduction, explained by how the code works.","title":"03 CUDA Shared Memory: Tiling, Bank Conflicts, and Reduction"},{"content":" Source: 01 CUDA C Basics\nThe CUDA Stack CUDA (Compute Unified Device Architecture) is NVIDIA\u0026rsquo;s parallel computing platform, and it is more than one thing. It bundles a programming model, the driver and runtime APIs, a compiler toolchain (nvcc, lowering to PTX and then SASS), and a library stack (cuBLAS, cuDNN, and the rest). CUDA C++ is the specific layer that extends C++ with device code; it is one part of the platform, not the whole of it. Since its 2007 release CUDA has become the de facto standard for deep-learning infrastructure, opening up the GPU as a general-purpose compute device (GPGPU) rather than a graphics-only one.\nBut when someone says \u0026ldquo;I use CUDA,\u0026rdquo; what exactly are they using? Running a model in PyTorch is CUDA. Writing a __global__ kernel by hand is also CUDA. The confusion comes from the fact that CUDA is a stack, not a single layer.\nWhat people usually call \u0026ldquo;CUDA\u0026rdquo; spans layers 3-5 (CUDA C/C++, PTX, SASS).\nLayer Role CUDA C/C++ The programming model you write directly. __global__, threadIdx, the Grid/Block/Thread abstraction CUDA Runtime API cudaMalloc, cudaMemcpy, kernel launches, etc. nvcc NVIDIA\u0026rsquo;s compiler that lowers the above to PTX / SASS PTX A virtual ISA. Handles forward compatibility across generations SASS The actual machine code, compiled for a specific GPU generation. Architecture-specific GPGPU GPGPU (General-Purpose computing on GPU) means exactly what it says: using the GPU for compute beyond graphics. Before deep learning took off, the GPU was mostly a device for drawing polygons, but today any large-scale parallel numerical workload gets offloaded to it.\nYou\u0026rsquo;ve probably seen a CUDA - GPUs option in the settings of a video editor or an ML framework. That\u0026rsquo;s a setting for which GPU runs a program\u0026rsquo;s CUDA work: not for games, but for GPGPU workloads like video editing and machine learning.\nThe workloads where GPGPU shines share one trait: they repeat the same operation over huge amounts of data, independently. That structure maps cleanly onto the GPU\u0026rsquo;s SIMT (Single Instruction, Multiple Threads) execution model.\nWorkload Underlying operation Video encoding / filters Parallel numeric ops over a pixel matrix DL training / inference Tensor (multi-dim matrix) MatMul Crypto mining Massively parallel hash execution Scientific simulation Parallel updates of grid / particle systems 3D rendering (Blender Cycles, etc.) Per-ray parallel computation As an aside: one of the earliest cases of using a GPU to train a neural network was a 2004 paper by Korean researchers (Oh \u0026amp; Jung, \u0026ldquo;GPU implementation of neural networks,\u0026rdquo; Pattern Recognition). There was no CUDA back then, so they ran the network on shaders.\nHeterogeneous Computing Heterogeneous computing means different architectures (CPU and GPU) cooperating within a single system. The key idea in CUDA is not \u0026ldquo;run everything on the GPU.\u0026rdquo; Control flow and light logic stay on the CPU (Host); only the heavy compute (matrix and tensor ops) is offloaded to the GPU (Device).\nWhy the split? It comes down to two design philosophies. A CPU puts a few cores behind large caches and aggressive branch prediction to minimize latency on sequential code. A GPU does the opposite: it strips control logic and cache down and packs thousands of small cores to maximize throughput. So branchy, sequential code belongs on the CPU, while code that repeats the same operation thousands of times belongs on the GPU.\nThe CPU optimizes for latency with a few large cores; the GPU optimizes for throughput with thousands of small ones\nHost-Device Data Flow In the explicit-copy model this post uses, the CPU (Host) and GPU (Device) have separate physical memories: a variable allocated on the CPU is not visible to a kernel, so you move the data yourself. (cudaMallocManaged and Unified Memory, and integrated GPUs can share physical memory, so their placement rules differ; this post first establishes the explicit model.) Under it, every CUDA program goes through these three stages to bridge the memory gap.\nHost to Device (cudaMemcpy) cudaMemcpy(d_data, h_data, size, cudaMemcpyHostToDevice); Copy the source data from CPU memory to GPU memory over the host-device interconnect, usually PCIe on a discrete GPU (NVLink only in topology-specific systems such as GH200).\nExecute Kernel (\u0026lt;\u0026lt;\u0026lt;...\u0026gt;\u0026gt;\u0026gt;) kernel\u0026lt;\u0026lt;\u0026lt;gridDim, blockDim\u0026gt;\u0026gt;\u0026gt;(d_data); Run the parallel kernel on the GPU to do the actual work.\nDevice to Host (cudaMemcpy) cudaMemcpy(h_result, d_result, size, cudaMemcpyDeviceToHost); Bring the finished result back from GPU memory to CPU memory.\nWhy is this the single biggest optimization bottleneck? The bandwidth numbers make it obvious, as long as you keep the paths straight. The default host-to-device path for a discrete GPU is PCIe: PCIe Gen4 x16 is ~32 GB/s per direction (theoretical), Gen5 x16 ~64 GB/s. NVLink is far faster but topology-dependent; an H100\u0026rsquo;s NVLink is ~900 GB/s aggregate (bidirectional), and it applies to GPU-to-GPU over NVLink/NVSwitch or Grace-to-Hopper over the on-package NVLink-C2C link on GH200, not to a plain cudaMemcpy from system RAM, which still crosses PCIe. On-device HBM, meanwhile, is ~2.0 TB/s on an A100 SXM (HBM2e) and ~3.35 TB/s on an H100 SXM (HBM3). So on-device memory outruns the PCIe host link by roughly 30-100×, depending on the PCIe generation and the GPU.\nThat means each host-device round trip is expensive, which is why much of real-world optimization comes down to minimizing transfers. Pinned (page-locked) memory raises PCIe transfer bandwidth, cudaMemcpyAsync overlaps transfer with compute, and kernel fusion cuts the intermediate global-memory traffic and per-launch overhead between kernels; it removes a host round trip only in a pipeline that was staging intermediates back to the host. That\u0026rsquo;s the subject of the next post.\nCUDA C Syntax and Kernels What kind of \u0026ldquo;heavy computation\u0026rdquo; is worth paying that communication bottleneck? The example that most clearly shows the payoff in intro CUDA is vector addition. In c[i] = a[i] + b[i], each index is independent, so one thread per element is all you need. A textbook embarrassingly parallel problem.\nTo run this on the GPU, you first have to declare where each function runs and where it\u0026rsquo;s called from. CUDA C adds function qualifiers to C/C++ for exactly this.\nQualifier Runs on Called from Notes __global__ Device (GPU) Host (CPU) The kernel that runs on the GPU. Must return void because the launch is asynchronous __device__ Device (GPU) Device (GPU) A helper callable only from within the GPU __host__ Host (CPU) Host (CPU) An ordinary C/C++ function (the default, can be omitted). Any unqualified function is __host__ The reason a __global__ kernel only returns void is the execution model. A kernel launch is asynchronous, so the CPU that called kernel\u0026lt;\u0026lt;\u0026lt;...\u0026gt;\u0026gt;\u0026gt;() does not wait for it to finish; it moves straight to the next line. There is no synchronous path to return a value on. If you need the result, you wait for completion with cudaMemcpy (an implicit sync) or cudaDeviceSynchronize, then read it back from device memory.\nThe nvcc Compilation Pipeline A single .cu file can freely mix CPU code (main) and GPU code (__global__). NVIDIA\u0026rsquo;s compiler nvcc scans the source and splits the two apart.\nDigging into the pipeline: host code is handed to a system C++ compiler (GCC, MSVC, etc.) as-is. Device code is lowered in two steps. First cicc (NVVM/LLVM-based) turns C++ into PTX, then ptxas compiles that PTX into SASS for a specific architecture. The final binary (a fatbin) usually embeds SASS for a few named architectures plus one forward-compatible PTX version. SASS is binary-compatible forward within a single major compute capability (sm_80 code also runs on sm_86 and sm_89, all major 8) but never across a major bump (sm_80 will not run on sm_90). So a next-major-generation GPU runs your binary only if it carries the forward-compatible PTX, which the driver JIT-compiles to SASS at load time. Build SASS alone with no PTX (as -arch=native does, since it emits SASS only), and the first next-major GPU it meets fails to load it. This is why PTX is called a \u0026ldquo;virtual ISA,\u0026rdquo; and why the -gencode arch=...,code=... flag controls both which architectures get pre-built SASS and whether PTX is carried along.\nRather than describe it, open it up. Dumping the add kernel with nvcc -arch=sm_80 -ptx vector_add.cu gives PTX whose core is:\nmad.lo.s32 %r1, %r3, %r4, %r5; // thread index i setp.ge.s32 %p1, %r1, %r2; // i \u0026gt;= N ? @%p1 bra $L__BB0_2; // out of range -\u0026gt; skip ... ld.global.f32 %f1, [%rd8]; // b[i] ld.global.f32 %f2, [%rd6]; // a[i] add.f32 %f3, %f2, %f1; // a[i] + b[i] st.global.f32 [%rd10], %f3; // c[i] = ... You can see exactly how the single C line (c[i] = a[i] + b[i]) lowers. The index math folds into one mad.lo.s32, a 32-bit integer multiply-add for the address calculation, not an FP32 fused-multiply-add, if (i \u0026lt; N) becomes setp plus a predicated branch (@%p1 bra), and the actual work is two global loads, one FP32 add, and one global store. The \u0026ldquo;12 bytes and 1 FLOP per element\u0026rdquo; from the roofline analysis later is exactly these four lines. From here ptxas lowers this PTX to architecture-specific SASS, which you can inspect with cuobjdump -sass.\nThread and Block Limits Threads per block max out at 1024. You\u0026rsquo;re free to split across dimensions (dim3), but if the product exceeds 1024 the launch fails with cudaErrorInvalidConfiguration. dim3(32, 32, 1) (=1024) passes; dim3(32, 32, 2) (=2048) dies. There\u0026rsquo;s also a separate cap of 64 on the z-axis that\u0026rsquo;s easy to forget. Grids are far more generous: 2³¹-1 on the x-axis and 65535 each on y/z. You won\u0026rsquo;t hit these with any reasonable dataset. Shared memory has three numbers people conflate. The static per-block limit is 48 KB (a compatibility default, the same on every architecture). The opt-in dynamic per-block maximum is higher and requested with cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, bytes): ~163 KB on A100 (cc 8.0), ~227 KB on H100 (cc 9.0). Both are carved out of the SM\u0026rsquo;s combined L1/shared-memory capacity, 192 KB on A100, 256 KB on H100, which hardware partitions between L1 cache and shared memory. These numbers aren\u0026rsquo;t arbitrary; they come from the hardware. A block runs to completion on exactly one SM (Streaming Multiprocessor) and is never split across SMs, and the SM schedules that block in units of warps. So the 1024-thread cap per block is 32 warps. On top of that, an SM has a finite register file: recent architectures have 65,536 32-bit registers per SM, shared among every thread resident on that SM. If a thread uses a lot of registers, fewer threads can be resident at once. That is exactly the occupancy story in the next section.\nWarps and SIMT Execution The GPU executes threads in groups called warps. A warp is 32 threads, a number fixed across every generation of NVIDIA GPU that developers cannot change.\nWhy does the warp matter for performance? The warp is the unit a scheduler issues: on a given cycle a warp scheduler issues one instruction for the active mask of its 32 lanes (SIMT). When lanes take different sides of an if/else (warp divergence), those paths serialize, the hardware runs one path with the other lanes masked off, then the other. Since Volta (cc 7.0), independent thread scheduling gives every thread its own program counter, so divergent lanes may interleave and there is no guaranteed immediate reconvergence at the branch\u0026rsquo;s post-dominator; call __syncwarp() when you need the lanes stepping together again. Either way, a partly-filled final warp still occupies a full 32-lane issue slot, which is why a thread count that is not a multiple of 32 wastes lanes.\nFor example, a block of 100 threads makes the GPU reserve scheduling slots for 4 warps (128 threads) while only 100 actually work. The remaining 28 slots sit idle, so utilization drops to 100/128 ≈ 78%. You throw away about 22% before you even start.\nThat is why block sizes are usually 128, 256, or 512, but the real decision criterion is occupancy: active warps resident on an SM divided by the SM\u0026rsquo;s maximum, which is itself architecture-dependent: 64 warps on A100 (cc 8.0) and H100 (cc 9.0), 48 on consumer Ampere (cc 8.6) and Ada (cc 8.9). The other per-SM resources are fixed the same way. On cc 8.0: up to 2048 resident threads, 32 resident blocks, 65,536 registers, and up to 164 KB of shared memory (228 KB on H100). Whichever runs out first sets the occupancy.\nThe reason occupancy matters is that the GPU\u0026rsquo;s performance model comes down to latency hiding. A global memory access takes hundreds of cycles (roughly 400 to 800), while an FP operation finishes in 4 to 6. When a warp stalls on a global load, the SM\u0026rsquo;s warp scheduler switches in the same cycle to another warp that is ready to run. That switch is free because the registers of every warp resident on the SM live in the register file at once, so there is no context to save and restore as there is on a CPU. The more warps are resident, the higher the chance that some warp is ready while others wait on memory. Raising occupancy is not about keeping cores busy; it is about hiding memory latency.\nThe number of blocks that can be resident on one SM is the minimum over four independent limits, threads, hardware block slots, registers, and shared memory, and occupancy follows from it:\n$$ B_{\\text{res}} = \\min\\!\\left( \\left\\lfloor \\tfrac{T_{\\text{SM}}}{T_{\\text{block}}} \\right\\rfloor,\\; B_{\\text{SM}}^{\\max},\\; \\left\\lfloor \\tfrac{R_{\\text{SM}}}{R_{\\text{thread}}\\, T_{\\text{block}}} \\right\\rfloor,\\; \\left\\lfloor \\tfrac{S_{\\text{SM}}}{S_{\\text{block}}} \\right\\rfloor \\right) $$$$ \\text{active warps} = B_{\\text{res}} \\left\\lceil \\tfrac{T_{\\text{block}}}{32} \\right\\rceil, \\qquad \\text{occupancy} = \\frac{\\text{active warps}}{W_{\\text{SM}}^{\\max}} $$These per-SM limits are set by compute capability. On cc 8.0 (A100): $T_{\\text{SM}}=2048$, $B_{\\text{SM}}^{\\max}=32$, $R_{\\text{SM}}=65536$, $W_{\\text{SM}}^{\\max}=64$. Take $T_{\\text{block}}=256$: the thread limit gives $\\lfloor 2048/256 \\rfloor = 8$ blocks, and registers bound it too, 8 resident blocks need $8 \\cdot 256 \\cdot R_{\\text{thread}} \\le 65536$, i.e. $R_{\\text{thread}} \\le 32$. Above 32 registers per thread, fewer blocks fit and occupancy falls. (Hardware allocates registers per warp at a fixed granularity, so the real cutoff is a little coarser than this bound.) One edge case the formula needs: a kernel that allocates no shared memory has $S_{\\text{block}} = 0$, so the shared-memory term is dropped (read it as $+\\infty$) and never binds. Vector add is exactly that case.\nHigh occupancy is not the goal in itself; it is one lever for hiding memory latency. Once latency is already hidden, more occupancy buys nothing and can even hurt by shrinking the per-thread register budget. Instruction-level parallelism, achieved DRAM bandwidth, and cache behavior all matter alongside it. Measure rather than assume: sm__warps_active.avg.pct_of_peak_sustained_active in Nsight Compute reports achieved occupancy, which is what actually counts.\nMemory Coalescing How you read global memory (HBM) inside the kernel matters as much as the host-device transfer. The GPU coalesces the global memory requests issued by a warp\u0026rsquo;s 32 threads into hardware transactions.\nOn compute capability 6.0 and later (Pascal onward, including A100 and H100), the global memory transaction unit is a 32-byte sector. A warp reading 32 contiguous 4-byte words touches a 128-byte span, which is exactly four sectors. When a warp issues a global load, the hardware maps the 32 lane addresses to the sectors that cover them and moves those whole sectors. Let $S$ be the number of distinct 32-byte sectors a warp touches on one load. Bus efficiency is the bytes requested over the bytes actually moved:\n$$ S = \\bigl|\\{\\, \\lfloor \\text{addr}_{\\text{lane}}/32 \\rfloor \\,\\}\\bigr|, \\qquad \\eta = \\frac{\\text{requested bytes}}{32\\,S} $$Work the two ends for a warp loading 32 floats (requested $= 32 \\times 4 = 128$ B):\nContiguous and aligned: the 128 B spans exactly four sectors, $S = 4$, so $\\eta = 128 / (32 \\cdot 4) = 1$. The right description is four 32-byte sectors (a 128-byte contiguous span), fully used, not \u0026ldquo;one transaction.\u0026rdquo; Fully scattered: each lane lands in its own sector, $S = 32$, so $32 \\cdot 32 = 1024$ B move for 128 B requested and $\\eta = 128/1024 = 1/8$. For 4-byte elements the floor is $1/8$, not $1/32$; the old \u0026ldquo;$1/32$\u0026rdquo; assumes 128-byte transactions, which is not how sectored access works. This is why vector addition is fast: i = blockIdx.x * blockDim.x + threadIdx.x makes adjacent lanes read adjacent addresses (a[0], a[1], a[2] …), so a warp hits four contiguous sectors and $\\eta = 1$. Read it strided, like a[i * stride], and $\\eta$ falls toward $1/8$ while the kernel slows in step. Nsight Compute measures it directly: l1tex__t_sectors_pipe_lsu_mem_global_op_ld.sum counts sectors moved and l1tex__t_requests_pipe_lsu_mem_global_op_ld.sum counts the load requests, so their ratio is the average sectors-per-request (ideal 4 for a full-warp 32-bit load, worst 32), which is the coalescing quality. The rule \u0026ldquo;map threads to the fastest-varying dimension (x)\u0026rdquo; is just keeping that ratio at its floor.\nBlock Independence Threads within the same block share shared memory and synchronize with __syncthreads(), because they physically sit on the same SM.\nDifferent blocks are different. What CUDA guarantees is that thread blocks in an ordinary kernel must be able to run independently in any order, with no general in-kernel barrier or ordering between arbitrary blocks, Block 7 may finish before Block 0. Blocks can still communicate indirectly through global memory and atomics, just with no ordering or visibility guarantee unless you impose one. When you truly need a device-wide barrier, the sanctioned options are a second kernel launch, a cooperative launch with Cooperative Groups grid.sync(), or (on Hopper) thread block clusters with distributed shared memory.\nThe reason is that the block-to-SM mapping is a hardware constraint. Threads in the same block share shared memory because it physically lives in that SM\u0026rsquo;s SRAM; blocks can\u0026rsquo;t talk to each other because they\u0026rsquo;re on different SMs with separate SRAM.\nBut this constraint is actually CUDA\u0026rsquo;s most important design decision. Because blocks are independent, the runtime can hand them out to whatever SMs are free, in any order. Whether a GPU has 20 SMs (a laptop chip) or 132 (an H100), the same kernel binary parallelizes across however many SMs are present. You change nothing, and a bigger GPU just runs faster. NVIDIA calls this transparent scalability. Giving up inter-block communication is the price paid for it, and it is why the Grid/Block/Thread software hierarchy lines up with the SM/warp/lane hardware, though, as the diagram below simplifies, that correspondence is one of scheduling, not a fixed one-to-one binding. A block is assigned to an SM and stays there; the SM issues that block\u0026rsquo;s threads 32 at a time as warps; and a \u0026ldquo;CUDA core\u0026rdquo; is a scalar execution unit (an ALU/FP lane) that runs one lane\u0026rsquo;s arithmetic on the cycle its warp issues. A thread is a software execution context, not a physical core it owns.\nGrid/Block/Thread (software) scheduled onto GPU/SM/warp-scheduler/execution-units (hardware). A block is placed on an SM, threads issue 32 at a time as warps, and a CUDA core executes one lane\u0026rsquo;s arithmetic, a scheduling correspondence, not a fixed thread-to-core binding.\nHere\u0026rsquo;s that hierarchy laid out by dimension:\nLayout by dimension. 1D kernel\u0026lt;\u0026lt;\u0026lt;4, 8\u0026gt;\u0026gt;\u0026gt; (32 threads), 2D kernel\u0026lt;\u0026lt;\u0026lt;dim3(2,2), dim3(4,4)\u0026gt;\u0026gt;\u0026gt;, 3D kernel\u0026lt;\u0026lt;\u0026lt;dim3(2,2,2), dim3(2,2,2)\u0026gt;\u0026gt;\u0026gt; (64 threads). The global index is blockIdx.x * blockDim.x + threadIdx.x\nExecution Configuration: \u0026lt;\u0026lt;\u0026lt;\u0026gt;\u0026gt;\u0026gt; Calling a __global__ function like a normal function is a compile error. You must use the triple chevron syntax.\nmykernel\u0026lt;\u0026lt;\u0026lt;gridDim, blockDim\u0026gt;\u0026gt;\u0026gt;(args); // ^^^^^^^ ^^^^^^^^ // number of blocks, threads per block gridDim: how many blocks are in the grid blockDim: how many threads are in each block Total threads = gridDim × blockDim The simplest example:\nmykernel\u0026lt;\u0026lt;\u0026lt;1, 1\u0026gt;\u0026gt;\u0026gt;(); // 1 block, 1 thread → effectively sequential To process N elements like in vector addition, you need N threads. The original video simplifies this to \u0026lt;\u0026lt;\u0026lt;N, 1\u0026gt;\u0026gt;\u0026gt;, but because of the warp efficiency above, 128 to 512 threads per block is more efficient in practice. Just size the grid by dividing N by the block size, rounding up.\nint N = 10000; int blockSize = 256; int gridSize = (N + blockSize - 1) / blockSize; // ceiling division add\u0026lt;\u0026lt;\u0026lt;gridSize, blockSize\u0026gt;\u0026gt;\u0026gt;(a, b, c, N); One more common idiom: instead of sizing the grid exactly to the data, you can fix the grid and have each thread process several elements, a grid-stride loop. Each thread steps by blockDim.x * gridDim.x, which decouples the launch config from the data size and stays safe when N exceeds the grid\u0026rsquo;s capacity.\n__global__ void add(float* a, float* b, float* c, int N) { int stride = blockDim.x * gridDim.x; int i = blockIdx.x * blockDim.x + threadIdx.x; for (; i \u0026lt; N; i += stride) c[i] = a[i] + b[i]; } Worked Example: Vector Add Putting the pieces together (the 3-stage transfer, the qualifiers, the launch config) gives this. Save it as vector_add.cu and it compiles and runs as-is.\n#include \u0026lt;cstdio\u0026gt; #include \u0026lt;cstdlib\u0026gt; #include \u0026lt;cmath\u0026gt; #include \u0026lt;cuda_runtime.h\u0026gt; __global__ void add(const float* a, const float* b, float* c, int N) { int i = blockIdx.x * blockDim.x + threadIdx.x; if (i \u0026lt; N) c[i] = a[i] + b[i]; // skip out-of-range threads } int main() { const int N = 1 \u0026lt;\u0026lt; 20; // ~1M elements const size_t bytes = N * sizeof(float); // 1) Host allocation + init float *h_a = (float*)malloc(bytes); float *h_b = (float*)malloc(bytes); float *h_c = (float*)malloc(bytes); for (int i = 0; i \u0026lt; N; i++) { h_a[i] = 1.0f; h_b[i] = 2.0f; } // 2) Device allocation float *d_a, *d_b, *d_c; cudaMalloc(\u0026amp;d_a, bytes); cudaMalloc(\u0026amp;d_b, bytes); cudaMalloc(\u0026amp;d_c, bytes); // 3) Host -\u0026gt; Device cudaMemcpy(d_a, h_a, bytes, cudaMemcpyHostToDevice); cudaMemcpy(d_b, h_b, bytes, cudaMemcpyHostToDevice); // 4) Launch (256 threads/block, grid by ceiling division) int blockSize = 256; int gridSize = (N + blockSize - 1) / blockSize; add\u0026lt;\u0026lt;\u0026lt;gridSize, blockSize\u0026gt;\u0026gt;\u0026gt;(d_a, d_b, d_c, N); // 5) Device -\u0026gt; Host (cudaMemcpy implicitly waits for the kernel) cudaMemcpy(h_c, d_c, bytes, cudaMemcpyDeviceToHost); // 6) Verify: 1.0 + 2.0 should be 3.0 float maxErr = 0.0f; for (int i = 0; i \u0026lt; N; i++) maxErr = fmaxf(maxErr, fabsf(h_c[i] - 3.0f)); printf(\u0026#34;max error: %f\\n\u0026#34;, maxErr); // 7) Cleanup cudaFree(d_a); cudaFree(d_b); cudaFree(d_c); free(h_a); free(h_b); free(h_c); return 0; } Compile and run:\n$ nvcc vector_add.cu -o vector_add $ ./vector_add max error: 0.000000 This single file contains everything from the earlier sections: the qualifier (__global__), the three transfers (cudaMemcpy three times), the launch config (\u0026lt;\u0026lt;\u0026lt;gridSize, blockSize\u0026gt;\u0026gt;\u0026gt;), and the bounds check (if (i \u0026lt; N)). One caveat: this code skips error handling for brevity. In real code you check the return value of every CUDA call and call cudaGetLastError() right after the launch, because a failed kernel launch fails silently.\nRoofline: Bandwidth-Bound Vector addition gets used as the example of \u0026ldquo;compute the GPU speeds up,\u0026rdquo; but an expert reads it the opposite way. This kernel barely does any FLOPs. Per element it moves 12 bytes (two loads for a and b, one store for c) and performs exactly one addition. That ratio is the arithmetic intensity.\n$$ \\begin{aligned} Q \u0026= 2 \\times 4\\,\\text{B} \\;+\\; 1 \\times 4\\,\\text{B} = 12\\,\\text{B} \\quad(\\text{load } a,b\\text{; store } c) \\\\ I \u0026= \\frac{W}{Q} = \\frac{1\\ \\text{FLOP}}{12\\ \\text{B}} \\approx 0.083\\ \\text{FLOP/B} \\end{aligned} $$This value decides whether a kernel is compute-bound or memory-bound. The framework is the roofline model: the performance a kernel can reach is capped by the lower of its compute ceiling and its bandwidth ceiling.\n$$P = \\min\\!\\bigl(P_{\\text{peak}},\\ I \\cdot \\beta\\bigr)$$The two ceilings meet at the ridge point $I^{*} = P_{\\text{peak}} / \\beta$. If $I$ sits left of it the kernel is memory-bound, right of it compute-bound. For an A100 (FP32 peak $P_{\\text{peak}} \\approx 19.5$ TFLOP/s, HBM $\\beta \\approx 2.0$ TB/s) the ridge point is\n$$I^{*} = \\frac{19.5 \\times 10^{12}}{2.0 \\times 10^{12}} \\approx 9.75 \\ \\text{FLOP/byte}$$Vector addition\u0026rsquo;s $I = 0.083$ sits more than 100× to the left of that. It is entirely memory-bound, and its performance ceiling is\n$$ \\begin{aligned} P_{\\text{vadd}} = I \\cdot \\beta \u0026= \\frac{1\\ \\text{FLOP}}{12\\ \\text{B}} \\times 2.0\\times10^{12}\\ \\text{B/s} \\\\ \u0026= 1.67\\times10^{11}\\ \\text{FLOP/s} \\\\ \u0026\\approx 166\\ \\text{GFLOP/s} \\quad(0.85\\%\\text{ of }P_{\\text{peak}}) \\end{aligned} $$less than 1% of the A100\u0026rsquo;s FP32 peak. So vector addition demonstrates bandwidth and parallel indexing, not the GPU\u0026rsquo;s compute power. The operations that actually win on a GPU have high arithmetic intensity, like matrix multiplication, which reads data once and reuses it many times. Much of deep-learning kernel optimization is exactly this: raising data reuse to push $I$ to the right of the ridge point.\nYou can confirm the memory-bound verdict empirically instead of trusting the algebra. Time the kernel with cudaEvents and divide bytes moved by elapsed time for effective bandwidth; in Nsight Compute, dram__bytes.sum.per_second (or DRAM Throughput %) reports how close you got to $\\beta$. A well-tuned vector add can approach a high fraction of peak HBM bandwidth (measure your own GPU for the number) and stays nowhere near peak FLOP/s, exactly what the roofline predicts.\nThe numbers inside \u0026lt;\u0026lt;\u0026lt; \u0026gt;\u0026gt;\u0026gt; map directly onto the GPU\u0026rsquo;s physical structure (SM, warp, block). Unlike other languages that hide the hardware, CUDA exposes it and puts performance in the developer\u0026rsquo;s hands. That\u0026rsquo;s both why you learn CUDA and why it\u0026rsquo;s hard.\nReferences CUDA C++ Programming Guide: the primary source for the programming model, occupancy, and the memory hierarchy CUDA Compiler Driver NVCC: the compilation pipeline and -gencode Nsight Compute: profiling occupancy and limiting resources ","permalink":"https://bias92.github.io/en/posts/cuda-c-basics/","summary":"The CUDA C execution model from the ground up: the compilation pipeline, why the 3-stage memory transfer is the bottleneck (with bandwidth numbers), how thread/block/grid schedule onto SM/warp/lane, occupancy, and kernel launch syntax.","title":"02 CUDA C Basics"},{"content":" The primary sources for this post are NVIDIA\u0026rsquo;s architecture whitepapers and official product pages. Narrative and microarchitecture interpretation draw on Fabien Sanglard, Chips and Cheese, and SemiAnalysis. Full sources are listed in the references at the end.\nOverview From Tesla in 2006 to Rubin in 2026, NVIDIA has shipped over a dozen GPU architectures in twenty years. Listed by name, that looks like a lot to memorize. In practice, the whole lineage fits one simple frame, and the goal of this post is to build it: take any NVIDIA GPU, place it on a timeline, and state in one sentence what it changed and why.\nThis post uses the 2006 Tesla chip — covered in the previous post — as its anchor, and follows that anchor forward.\nThe family tree: a shared trunk through Pascal, then a fork at Volta into a datacenter line (top) and a graphics line (bottom).\nThe frame is this: there is a foundation that almost never changes, and every other change is a response to one of two pressures.\nThe unchanging foundation is the execution model and the memory hierarchy. The SIMT (Single Instruction, Multiple Threads) model, in which one instruction is executed simultaneously by a bundle of 32 threads (a warp); the thread block that runs to completion on a single SM (Streaming Multiprocessor — the basic building block of a GPU, bundling execution units, schedulers, and shared memory); and the memory hierarchy of registers → shared memory → global DRAM. Learn this once in the CUDA C post and it applies unchanged from G80 to Rubin. It is why CUDA code written twenty years ago still compiles for today\u0026rsquo;s GPUs.\nThe two pressures driving change are these. First, the workload moved. As the GPU\u0026rsquo;s main customer shifted from graphics to AI, the SM kept its general-purpose cores and kept bolting specialized units around them: Tensor Cores first, then RT Cores, then the Transformer Engine. Second, the scale pressure. When one die stopped being enough, the unit of design grew from a chip to two dies to an entire rack. In short, the pattern of the genealogy is: the general-purpose SM stays, accelerators accrete around it, and the package keeps growing.\nThis post follows the SM for a simple reason: the SM is the unit onto which CUDA programs are scheduled. The rest of the GPU (L2 cache, memory controllers, ROPs, copy engines, host interface, fabric) matters for performance and system design, but the changes a programmer actually feels — warp execution, registers, shared memory, Tensor Cores, RT Cores, TMA/TMEM — show up at the SM.\nEach architecture section below pairs the SM diagram with that generation\u0026rsquo;s SM spec table. In the tables, bold values mark what changed from the previous generation of the same line. Figures are from NVIDIA whitepapers.\nA GPU die is an array of SMs wrapped by L2, memory controllers, DRAM, graphics fixed-function, and host/fabric interfaces. Every per-generation diagram below zooms into one SM.\nTesla (2006, G80) The starting point of the lineage. GPUs before it had fixed pipelines with separate hardware for vertex and pixel processing. Tesla replaced both with a single unified array of programmable cores. That one decision turned the GPU from a graphics-only device into a general-purpose compute engine, and made the CUDA programming model possible. The configuration looks modest by today\u0026rsquo;s standards: 8 scalar processors (SPs) per SM, one warp scheduler, 90nm process.\nTesla SM (G80): 8 scalar SPs, 1 scheduler, 16 KB shared memory. The origin of everything.\nChip Partitions FP32/SM FP64/SM Tensor/SM Schedulers/SM Shared+L1 Register file G80 monolithic 8 (SP) — — 1 16 KB (shared only, no L1) 32 KB Fermi (2010, GF100) The generation that deliberately turned a graphics chip into a compute chip. Going from \u0026ldquo;a GPU can compute\u0026rdquo; to \u0026ldquo;a GPU is a serious programming target\u0026rdquo; required specific things, and Fermi added them: a real L1 data cache and an L2, ECC memory, fused multiply-add (FMA), fully IEEE-compliant double precision (FP64), and C++ support. The SM itself grew to 32 CUDA cores with 2 warp schedulers, and the texture units moved inside the SM. If Tesla proved GPU compute was possible, Fermi made it something you could build numerical libraries on.\nFermi SM (GF100): 32 cores, 2 schedulers, the first L1 data cache on a GPU.\nChip Partitions FP32/SM FP64/SM Tensor/SM Schedulers/SM Shared+L1 Register file GF100 monolithic 32 16 FMA/clk¹ — 2 64 KB (shared/L1 combined, 48+16 split) 128 KB ¹ GF100\u0026rsquo;s FP64 is listed as per-clock FMA throughput because that is how NVIDIA disclosed it, rather than as a unit count.\nKepler (2012, GK110) A bet on throughput. Kepler widened the SM dramatically — renamed SMX, with 192 CUDA cores — while moving much of instruction scheduling from hardware to the compiler to save power. The bet was that many cores plus a simple scheduler, run at lower clocks, would win on performance per watt. It half worked. Kepler was efficient in aggregate but hard to keep fed, and per-core utilization suffered. It remains the generation people point to when they say \u0026ldquo;a wider SM is not automatically a faster SM.\u0026rdquo;\nKepler SMX (GK110): 192 cores, 4 schedulers, compiler-driven scheduling.\nChip Partitions FP32/SM FP64/SM Tensor/SM Schedulers/SM Shared+L1 Register file GK110 monolithic 192 64 — 4 64 KB combined + 48 KB read-only 256 KB Maxwell (2014, GM200) Maxwell corrected Kepler\u0026rsquo;s overreach. It narrowed the SM back to 128 cores and split it into 4 processing blocks of 32 cores each, each block with its own scheduler and register file. Since 32 is exactly the warp size, this partitioning mapped the hardware cleanly back onto warps, 1:1. With no new process node — just a cleaner design — Maxwell delivered one of NVIDIA\u0026rsquo;s biggest efficiency jumps ever. It is the standard example that tidy architecture can beat brute width. The \u0026ldquo;SM = 4 warp-sized partitions\u0026rdquo; structure that settled here carries forward through every generation since.\nMaxwell and Pascal: the SM split into 4 warp-sized partitions, a structure that persists to this day.\nChip Partitions FP32/SM FP64/SM Tensor/SM Schedulers/SM Shared+L1 Register file GM200 4 × 32 (first split) 128 4 — 4 96 KB (dedicated shared, separate L1) 256 KB Pascal (2016, GP100 / GP102) This is where the fork becomes visible. The consumer part (GP102, GTX 1080 Ti) was essentially Maxwell moved to 16nm with GDDR5X attached — a process-and-bandwidth generation. The datacenter part (GP100, P100) was a different machine: only 64 FP32 lanes per SM, but serious FP64 hardware, plus the first appearance of NVLink (high-speed GPU-to-GPU interconnect) and HBM2 (high-bandwidth stacked memory). Pascal is the point where consumer and datacenter stopped being \u0026ldquo;the same chip, different bin\u0026rdquo; (binning: grading the same die by yield and performance and selling the grades as different products).\nChip Partitions FP32/SM FP64/SM Tensor/SM Schedulers/SM Shared+L1 Register file GP100 (datacenter) 2 × 32 64 32 — 2 64 KB (dedicated shared) 256 KB GP102 (consumer) 4 × 32 128 4 — 4 96 KB 256 KB The GP102 row carries no bold because it essentially keeps the Maxwell configuration.\nVolta (2017, GV100) The hinge of the entire genealogy. This is where the first Tensor Core appears: a dedicated unit that performs a small matrix multiply-accumulate (MMA, hereafter) as a single instruction. The reason it was needed: when you do matrix multiplication with ordinary FP instructions, most of the energy goes not into arithmetic but into instruction fetch/decode/schedule overhead. Batch the work into matrix-sized instructions and that overhead disappears.\nVolta\u0026rsquo;s second legacy is independent thread scheduling. From this generation on, each thread in a warp has its own program counter. The lockstep assumption — that every thread in a warp executes the same instruction on the same beat — broke here, which is exactly why the CUDA C post has to qualify warp lockstep and introduce __syncwarp(). Volta shipped as datacenter-only, with no consumer part. Everything in modern AI hardware starts here.\nVolta SM (GV100): the first Tensor Core joins the CUDA cores.\nChip Partitions FP32/SM INT32/SM FP64/SM Tensor/SM Schedulers/SM Shared+L1 Register file GV100 4 × 16 64 64 (separate datapath) 32 8 (1st gen, FP16) 4 128 KB (unified shared+L1) 256 KB The INT32 column starts here for a reason: before Volta, integer work shared the FP32 cores\u0026rsquo; datapath.\nTuring (2018, TU102) Volta\u0026rsquo;s ideas reach the graphics line. Turing put a 2nd-generation Tensor Core and a brand-new RT Core (dedicated ray-tracing hardware) into consumer GPUs, and split the datapath so the SM could issue FP32 and INT32 instructions concurrently — a real win, since address arithmetic and other integer work is constantly interleaved with FP math in practice. This is the moment the graphics line stopped being purely graphics and started carrying AI and ray-tracing accelerators, and it is what makes DLSS (rendering at low resolution and upscaling frames with a neural network to buy performance) possible.\nTuring and Ada: RT Cores and graphics-facing Tensor Cores enter the SM.\nChip Partitions FP32/SM INT32/SM FP64/SM Tensor/SM RT/SM Shared+L1 Register file TU102 4 × 16 64 64 (concurrent issue) 2 8 (2nd gen, INT8/4) 1 (1st gen) 96 KB unified 256 KB As the first entry of the graphics line, this row is marked relative to Volta (GV100).\nAmpere (2020, GA100 / GA102) Scale and formats. The 3rd-generation Tensor Core added TF32 (FP32\u0026rsquo;s exponent range with a shortened mantissa — a drop-in for training code) and BF16, and claimed 2× throughput with structured sparsity (zeroing half the weights in a fixed pattern, then skipping those zeros at compute time). Just as important is the cp.async instruction: previously, moving data from global to shared memory had to route through registers; cp.async performs the copy without touching them, relieving the register pressure that chronically limits Tensor Core kernels. Ampere also introduced MIG (Multi-Instance GPU), which partitions one A100 into fully isolated GPU instances.\nThe Ampere name exists on both lines, and as the table shows, the SMs genuinely differ: the datacenter A100 has 64 FP32 lanes per SM, consumer RTX 30 parts have 128.\nAmpere SM (GA100): 3rd-gen Tensor Cores, and cp.async feeding shared memory directly.\nChip Partitions FP32/SM INT32/SM FP64/SM Tensor/SM RT/SM Shared+L1 Register file GA100 (datacenter) 4 × 16 64 64 32 **4 (3rd gen, TF32/BF16)**² — 192 KB unified (shared up to 164 KB) 256 KB GA102 (consumer) 4 × 32 128 (64 dedicated + 64 shared w/ INT) 64 2 4 (3rd gen) 1 (2nd gen) 128 KB unified 256 KB ² The drop from 8 to 4 Tensor Cores is not a regression — each unit\u0026rsquo;s tile got larger. See The Evolution of the Tensor Core below. The GA100 row is marked against GV100, the GA102 row against TU102.\nHopper (2022, GH100) The Transformer Engine generation. The Transformer Engine is a hardware-plus-software mechanism that automatically picks the right precision (FP8 vs FP16) per layer — taking low precision\u0026rsquo;s speed while guarding against accuracy collapse. The 4th-generation Tensor Core added FP8 (E4M3, E5M2), and Hopper wrapped it in machinery aimed squarely at LLMs: asynchronous matrix instructions issued at warpgroup granularity (wgmma, where a warpgroup is 4 warps), TMA (Tensor Memory Accelerator — a bulk asynchronous copy engine that a single thread kicks off and hardware completes), and thread block clusters with distributed shared memory, letting SMs exchange data directly.\nIn SemiAnalysis\u0026rsquo;s framing, the motivating problem is that \u0026ldquo;Tensor Core throughput doubles every generation, but global memory latency does not improve.\u0026rdquo; So Hopper spent its budget not on raw FLOPs but on hiding latency and feeding the units. The flagship is H100, with HBM3 and 900 GB/s NVLink 4.\nHopper SM (GH100): FP8 Tensor Cores, TMA, wgmma, thread block clusters.\nChip Partitions FP32/SM INT32/SM FP64/SM Tensor/SM RT/SM Shared+L1 Register file GH100 4 × 32 128 64 64 4 (4th gen, FP8, wgmma) — 256 KB unified (shared up to 228 KB) 256 KB Ada (2022, AD102) The graphics-line counterpart to Hopper, launched the same year. It carries 4th-generation Tensor Cores and 3rd-generation RT Cores, adds Shader Execution Reordering (SER — hardware that regroups divergent ray-tracing threads to recover efficiency), and the DLSS 3 frame-generation stack. The flagship is the RTX 4090 on TSMC 4nm.\nChip Partitions FP32/SM INT32/SM FP64/SM Tensor/SM RT/SM Shared+L1 Register file AD102 4 × 32 128 64 2 4 (4th gen, FP8) 1 (3rd gen) 128 KB unified 256 KB Blackwell (2024, B200 / GB202) Two chips, one name. The datacenter part (B200) is where the GPU stopped being a single die: two dies, each grown to the reticle limit (the largest area a lithography machine can expose at once), are fused with a 10 TB/s link and presented to software as one GPU — 208 billion transistors combined, with HBM3e. The consumer part (GB202 family, RTX 5090) takes the opposite approach: a single die near 750mm² with GDDR7. The full GB202 die is laid out for 192 SMs; the RTX 5090 ships with 170 of them enabled (21,760 CUDA cores). Chips and Cheese reads this design as \u0026ldquo;scale over specialization\u0026rdquo;: a 64-bank L2 (~8.7 TB/s) that chooses bandwidth over latency, winning through sheer core density rather than per-core cleverness.\nBoth parts share the 5th-generation Tensor Core. Its additions: FP4 (the NVFP4 and microscaling MXFP formats), a dedicated Tensor Memory (TMEM) that holds matrix operands outside the register file, and CTA-pair MMA, where two SMs cooperate on one matrix operation (CTA, cooperative thread array, is the hardware-side name for a thread block). At the system level, GB200 pairs two datacenter Blackwell GPUs with a Grace CPU, and GB200 NVL72 links 72 such GPUs into an NVLink domain that behaves like a single rack-scale GPU.\nBlackwell SM (B200): FP4 Tensor Cores, dedicated TMEM, CTA-pair MMA.\nChip Partitions FP32/SM INT32/SM FP64/SM Tensor/SM RT/SM Shared+L1 Register file B200 (datacenter)³ 4 × 32 128 128 64 4 (5th gen, FP4, tcgen05) — 256 KB 256 KB + TMEM 256 KB GB202 (consumer) 4 × 32 128 (all cores unified FP32/INT32) 128 2 4 (5th gen, FP4) 1 (4th gen) 128 KB unified 256 KB ³ At the time of writing, no SM-level whitepaper for datacenter Blackwell has been published, so the B200 row uses figures from NVIDIA\u0026rsquo;s technical blog and public educational material. The B200 row is marked against GH100, the GB202 row against AD102.\nRubin (2026) The current generation, and the point where the shift of the design unit from chip to rack completes. NVIDIA\u0026rsquo;s public material presents the Rubin GPU at 336 billion transistors, 288GB of HBM4 at 22 TB/s, and NVLink 6 at 3.6 TB/s. NVIDIA marks these public specs as preliminary, and SM details are not public — which is why this section has no spec table.\nMore important is how the name is used. \u0026ldquo;Blackwell\u0026rdquo; is still the name of a GPU you can point at; \u0026ldquo;Rubin\u0026rdquo; is mostly the name of a system. NVIDIA\u0026rsquo;s own description of Vera Rubin is a rack — the NVL72: 72 Rubin GPUs and 36 Vera CPUs in a single liquid-cooled NVLink-6 domain, delivering roughly 3.6 EFLOPS of FP4 inference and 20.7 TB of HBM4. The Vera CPU is its own chip, with 88 custom Olympus Arm cores. NVIDIA\u0026rsquo;s current Vera Rubin page describes the platform as a seven-chip platform spanning compute, networking, storage, and switching. This is why trying to summarize Rubin with a single SM diagram is already the wrong level of abstraction.\nRubin is a platform: GPU, Vera CPU, NVLink switches, DPU, and Ethernet in one rack.\nSo keep the two names separate. The Rubin GPU is a microarchitecture; compare it to GB100. Vera Rubin is a co-designed rack-scale computer; compare it to GB200 NVL72. The endpoint of the genealogy is not a faster chip — it is the admission that the interesting unit is now the rack.\nArchitecture snapshots: the shared trunk moves from graphics to compute; the upper branch is datacenter AI, the lower branch RTX graphics.\nThe Evolution of the Tensor Core Among all the threads in this genealogy, the one that determined where transistors and R\u0026amp;D actually went is the Tensor Core. It is a matrix multiply-accumulate unit, and its five generations from Volta to Blackwell move along two axes: precision and asynchrony.\nPrecision dropped every generation: FP16 (Volta) → INT8/INT4 (Turing) → TF32 and BF16 (Ampere) → FP8 (Hopper) → FP4 (Blackwell). The move is possible because AI workloads tolerate low precision, and every halving of precision doubles the arithmetic per transistor and per byte moved. The Transformer Engine introduced in the Hopper section is exactly the safety mechanism for this move.\nThe easily missed half of the story is that Tensor Cores grew by tile size, not by count. A matrix multiply performs roughly $N^3$ operations while moving roughly $N^2$ data, so arithmetic intensity — operations per byte moved — rises with the tile\u0026rsquo;s edge length:\n$$I \\sim \\frac{N^3}{N^2} = N$$Bigger tiles amortize data movement better. So instead of stamping out more small units, NVIDIA made each instruction compute a bigger matrix every generation (4×4×4, then 8×8×4, then 16×8×16 and beyond). This is why the per-section spec tables show the Tensor Core count per SM actually falling from 8 in Volta to 4 from Ampere onward — each unit got much larger, and throughput still doubled per generation.\nThe execution model evolved for the same reason. Tensor throughput keeps doubling while memory latency does not improve, so the ability to overlap compute with data movement becomes the whole game: synchronous warp-level MMA (Volta) → asynchronous warpgroup MMA (Hopper\u0026rsquo;s wgmma) → fully asynchronous single-thread MMA with operands resident in dedicated Tensor Memory (Blackwell). The thesis that runs the entire length of the lineage: the bottleneck is not the math — it is feeding the math.\nTwo Lines: Consumer and Datacenter From Volta onward, the family runs as two branches that share DNA but optimize for different things.\nThe datacenter line (GV100 → GA100 → GH100 → B200 → Rubin) maximizes AI throughput and interconnect: fewer FP32 lanes per SM but more INT32/FP64/Tensor hardware, HBM instead of GDDR, NVLink growing into full rack-scale fabric (the network that ties chips and nodes into one domain), and datacenter-only features like MIG and thread block clusters.\nThe graphics line (TU102 → GA102 → AD102 → GB202) keeps enough Tensor Cores for DLSS, adds RT Cores and rendering features, and uses GDDR memory.\nOne caveat worth internalizing: the names Ampere and Blackwell exist on both branches. \u0026ldquo;An Ampere GPU\u0026rdquo; can mean an A100 or an RTX 3090, and their SMs differ substantially (64 vs 128 FP32 lanes). The generation name alone is not enough — you have to say which line.\nThe Family Tree Generation Year SM / codename Defining change Process Flagship Tesla 2006 SM, 8 SP (G80) unified shaders, SIMT, CUDA 90 nm 8800 GTX Fermi 2010 SM, 32 (GF100) L1 data cache, FMA, FP64, C++ 40 nm GTX 480 Kepler 2012 SMX, 192 (GK110) compiler scheduling, wide SM 28 nm K20 Maxwell 2014 SMM, 128 (GM200) efficiency, 4x32 partitions 28 nm GTX 980 Ti Pascal 2016 GP100 / GP102 NVLink, HBM2 (GP100), 16nm 16 nm P100 Volta 2017 GV100, 64 FP32 1st Tensor Core, independent thread scheduling 12 nm V100 Turing 2018 TU102, 64 FP32 RT Core + 2nd Tensor to graphics 12 nm RTX 2080 Ti Ampere 2020 GA100, 64 FP32 3rd Tensor (TF32/sparsity), MIG 7 nm A100 Ada 2022 AD102, 128 FP32 4th Tensor, 3rd RT, SER 4 nm RTX 4090 Hopper 2022 GH100, 128 FP32 Transformer Engine (FP8), TMA, clusters 4 nm H100 Blackwell 2024 2 dies, 208B FP4, TMEM, 5th NVLink, scale-first TSMC 4NP B200 / GB200 Rubin 2026 2 dies, 336B HBM4, NVLink 6; Vera Rubin = rack platform preliminary per NVIDIA Rubin / Vera Rubin NVL72 Synthesis: Three Trajectories Three trajectories run the full length of the lineage. First, specialization increases — the move away from doing every computation on one general-purpose core, toward dedicated hardware for specific operations. The SM keeps its general-purpose cores while stacking Tensor Cores, RT Cores, the Transformer Engine, and dedicated Tensor Memory around them. Second, precision decreases — from FP32 to FP4 — because AI can pay for throughput in bits. Third, the unit of design grows: a chip, then two dies, then a rack.\nThe one thing that never changes is where the bottleneck sits. From memory coalescing in the CUDA C post to Hopper\u0026rsquo;s TMA and Blackwell\u0026rsquo;s TMEM, every generation spends most of its new hardware budget not on raw FLOPs but on moving data and hiding latency. The per-section spec tables tell the same story: the register file has not moved from 256 KB since Kepler, while shared memory grew from 16 KB to 228 KB and Blackwell added a whole new storage pool in TMEM. Compute has been cheap for a decade; feeding it has not. The single thread running through the whole family tree is the memory wall.\nReferences Fabien Sanglard, A history of NVidia Stream Multiprocessor: the narrative and SM design changes from Tesla through Turing. SemiAnalysis, NVIDIA Tensor Core Evolution: Volta to Blackwell: the precision, asynchrony, and tile-size argument. Chips and Cheese, Blackwell: NVIDIA\u0026rsquo;s Massive GPU: the scale-over-specialization microarchitecture reading. NVIDIA primary architecture documents: Fermi, Kepler GK110, Maxwell tuning, Pascal GP100, Volta GV100, Turing, Ampere A100, Ampere GA102, Ada, Hopper, H100 whitepaper, GeForce RTX Blackwell. NVIDIA Blackwell Architecture and Vera Rubin Platform: primary figures for the latest generations. NVIDIA Vera Rubin NVL72 and the NVIDIA Rubin platform technical blog: Rubin GPU, NVLink 6, NVL72, and the preliminary-spec caveat. Cornell Virtual Workshop, B200 SM: B200 SM configuration figures. ","permalink":"https://bias92.github.io/en/posts/cuda-2-gpu-genealogy/","summary":"One line from Tesla (2006) to Rubin (2026): how the NVIDIA SM stayed the same (SIMT, warp = 32, block per SM) while accreting specialized accelerators, how the Tensor Core evolved across five generations, why the consumer and datacenter lines split, and why \u0026lsquo;Rubin\u0026rsquo; is a platform, not just a GPU.","title":"01 NVIDIA GPU Architecture Genealogy: Tesla to Rubin"},{"content":" Source: Lindholm, Nickolls, Oberman, Montrym, \u0026ldquo;NVIDIA Tesla: A Unified Graphics and Computing Architecture,\u0026rdquo; IEEE Micro 28(2), 2008.\nMotivation: The Hardware Under CUDA The CUDA C post talks about blocks, warps, SMs, occupancy, and memory coalescing as if they were language features. They are not. They are the software-visible names of a hardware architecture NVIDIA shipped in November 2006 as the G80 (GeForce 8800 GTX) and refined in 2008 as the GT200 (GTX 280). NVIDIA calls that architecture Tesla, and the reference is the IEEE Micro paper above.\nIf you learn CUDA C without this layer, warp = 32 and \u0026ldquo;coalesce your accesses\u0026rdquo; are rules to memorize. If you see the hardware first, they are consequences. This post walks the Tesla data path once, top to bottom, so that the CUDA abstractions in the CUDA C post land on something physical.\nOne caveat up front: Tesla (G80/GT200) is the historical anchor here, not a snapshot of a modern GPU. The SM has been rebuilt many times since 2006. FP32 lanes per SM went from 8 to 128, one warp scheduler became four, shared memory grew from 16 KB to over 200 KB, and units that did not exist then (a general-purpose L1 data cache, tensor cores, async copy) were added. What survives is the naming lineage and the mental model: SPA to TPC to SM to SP is the genealogy behind CUDA\u0026rsquo;s blocks, warps, and SM scheduling. Read the numbers below as a 2008 snapshot, not today\u0026rsquo;s spec sheet, and take the concepts as the part that lasts.\nThe Unified Shader Architecture A pre-Tesla GPU had a fixed pipeline of specialized processors: vertex shaders, then rasterization, then pixel (fragment) shaders, each a different unit with its own instruction set and its own silicon. The ratio between vertex and pixel work is fixed at design time, so a vertex-heavy or pixel-heavy frame leaves half the chip idle.\nTesla threw that out. It replaced the separate shader stages with one array of identical programmable processors that every shader type time-shares. Vertices, geometry, and pixels all run on the same cores; the hardware just re-points the array at whatever stage has work. NVIDIA calls that array the SPA (Streaming Processor Array).\nThat single decision is what created GPGPU. Once you have a general, programmable processor array with its own memory system, aiming it at non-graphics compute is a matter of exposing it, which is exactly what CUDA does. CUDA C is not bolted onto a graphics chip; it is a second front-end to the same unified array.\nG80 shown: 8 TPC × 2 SM × 8 SP = 128 SP, 6 DRAM partitions. The SPA is the array of 8 TPCs; the compute work distribution path (magenta) is the one CUDA drives.\nThe Data Path Follow the work from memory, through the chip, and back to memory. In graphics mode the path is:\nInput Assembler. Reads vertex indices and attributes from DRAM and assembles them into primitives (points, lines, triangles). This is the front door. Work distribution. Tesla has separate distributors for vertex, pixel, and compute work. Each one hands batches of work to the SPA and load-balances them across the processors. In compute mode, the compute work distribution unit is the piece that hands thread blocks to SMs, one block at a time as SMs free up. That load-balancer is the physical origin of CUDA\u0026rsquo;s transparent scalability. SPA. The processor array executes the shader (or the kernel). For graphics it runs vertex shading, then later pixel shading; for CUDA it runs the kernel. It is the compute engine and everything else feeds it or drains it. Setup / raster (graphics only). Between vertex and pixel work, fixed-function units clip, set up triangles, and rasterize them into fragments, which the pixel distributor then feeds back into the SPA. ROP (Raster Operations Processor). After pixel shading, the ROP does the fixed-function back end: depth and stencil test, color blend, antialiasing, and the final write to the framebuffer. Each ROP is tied to a DRAM partition. DRAM. The memory partitions, where everything starts and ends. For CUDA the graphics-specific stages (setup, raster, ROP) sit idle, and the path collapses to: host launches a grid, the compute work distributor spreads blocks over SMs, the SPA runs the kernel, and loads/stores move data to and from DRAM. Same silicon, fewer stages lit up.\nThe Compute Hierarchy: SPA to TPC to SM to SP The SPA is not a flat pool of cores. It is a three-level hierarchy, and each level maps to a CUDA concept.\nTPC (Texture / Processor Cluster). The SPA is divided into TPCs. A TPC bundles a texture unit with a small number of SMs that share it. G80 has 8 TPCs of 2 SMs each; GT200 has 10 TPCs of 3 SMs each. SM (Streaming Multiprocessor). The unit that actually runs threads. Each SM holds 8 SPs, 2 SFUs, a multithreaded instruction fetch/issue unit, a register file, and 16 KB of shared memory. The SM is where a CUDA thread block lands and stays. SP (Streaming Processor). A scalar ALU that executes one thread\u0026rsquo;s floating-point and integer work, primarily a MAD (multiply-add). Eight per SM. This is the unit later marketing renamed the \u0026ldquo;CUDA core.\u0026rdquo; The totals fall straight out of the hierarchy:\n$$ \\text{G80: } 8\\ \\text{TPC} \\times 2\\ \\text{SM} \\times 8\\ \\text{SP} = 128\\ \\text{SP} \\qquad \\text{GT200: } 10 \\times 3 \\times 8 = 240\\ \\text{SP} $$Two more execution units per SM matter for CUDA:\nSFU (Special Function Unit). Two per SM. It computes transcendentals (reciprocal, reciprocal-sqrt, sin, cos, log, exp) and, in graphics, interpolates pixel attributes. When a CUDA kernel calls __sinf or rsqrtf, this is the unit. LSU (Load/Store Unit). The path that issues loads and stores to global and local memory through the memory pipeline. Its behavior under a warp is the whole subject of coalescing in the CUDA C post. SIMT and the Warp Here is where the CUDA abstraction is literally defined by the hardware. The SM\u0026rsquo;s instruction unit does not track threads one at a time. It creates, manages, schedules, and executes them in groups of 32 called a warp. The Tesla paper coined the term SIMT (Single-Instruction, Multiple-Thread) for this: the SM issues one instruction to a warp, and all 32 threads execute it, each on its own data and its own registers.\nWhy 32, and why does one warp instruction take more than one clock? Count the lanes. An SM has 8 SPs, but a warp is 32 threads, so the SM streams the warp over the 8 SPs across four fast clocks:\n$$ \\frac{32\\ \\text{threads/warp}}{8\\ \\text{SP}} = 4\\ \\text{shader clocks per warp instruction} $$So the physical SIMD width is 8, but the architectural width the programmer sees is 32. NVIDIA fixed the warp at 32 here and has never changed it, which is why CUDA code a generation later still assumes 32. When a warp branches (divergence), the threads that do not take the branch are masked for those clocks, which is the cost the CUDA post describes.\nThreads are not free-floating either. Each SM has a fixed register file and 16 KB of shared memory, and every resident warp carves its registers and shared memory out of those pools. A G80 SM holds up to 24 warps (768 threads) at once; how many actually fit depends on how greedy each thread is. That trade, resident warps versus per-thread resources, is exactly occupancy, and it exists because these pools are finite hardware on the SM.\nClock Domains A Tesla GPU does not run at one frequency. It has separate clock domains, and mixing them up wrecks any performance estimate.\nThe core (graphics) clock drives the front end, setup, raster, and ROP. The shader clock drives the SPs, and it runs much faster than the core clock. On the 8800 GTX the core is 575 MHz while the shaders run at 1.35 GHz, roughly 2.35x. The memory clock is separate again, driving the GDDR3 interface. The shaders run hot on purpose: throughput comes from the SP array, so NVIDIA clocks it as high as the process allows and lets the rest of the chip stay slower and cooler. This is why a FLOP estimate uses the shader clock, not the core clock. For the 8800 GTX, counting one MAD (2 FLOP) per SP per shader clock:\n$$ 128\\ \\text{SP} \\times 1.35\\ \\text{GHz} \\times 2\\ \\text{FLOP} \\approx 346\\ \\text{GFLOP/s} $$NVIDIA quoted a higher figure (518 GFLOP/s) by also counting a MUL that the SFU can co-issue in the same clock, which real kernels rarely sustain. That gap between the marketing peak and the achievable peak is a habit worth keeping: always ask which clock and which instruction mix a number assumes.\nThe Memory Subsystem: DRAM Partitions and Coalescing Tesla\u0026rsquo;s DRAM is not one monolithic memory. It is split into independent partitions, each with its own memory controller and its own ROP. G80 has 6 partitions of 64 bits each (a 384-bit aggregate bus); GT200 has 8 (512-bit). Addresses are interleaved across partitions so that sequential memory marches across all controllers in parallel, and total bandwidth is the sum of the partitions.\nFor the 8800 GTX, with a 384-bit bus and GDDR3 at 900 MHz (double data rate, so 1.8 Gb/s per pin):\n$$ \\frac{384\\ \\text{bit}}{8} \\times 1.8 \\times 10^{9}\\ \\text{s}^{-1} = 48\\ \\text{B} \\times 1.8\\ \\text{GT/s} \\approx 86.4\\ \\text{GB/s} $$This partitioned, interleaved, wide memory is the reason coalescing exists as a rule in CUDA. When a warp\u0026rsquo;s 32 lanes issue loads, the LSU turns them into memory transactions that the partitions service. If the 32 addresses are contiguous, they fall into a few wide transactions spread across the controllers and the bus runs full. If they are scattered, each address drags its own transaction and most of every fetched line is thrown away. The CUDA-level advice \u0026ldquo;make warp accesses contiguous\u0026rdquo; is just \u0026ldquo;feed the partitioned memory system the wide, aligned transactions it was built for.\u0026rdquo;\nContinuity and Change Everything above is a 2008 chip. Open an A100 (2020) or H100 (2022) SM and the numbers are unrecognizable, but the skeleton is the same. That split, stable concepts and shifting magnitudes, is the whole reason Tesla is worth reading.\nG80 (2006) A100 (2020) H100 (2022) FP32 lanes per SM 8 64 128 Warp schedulers per SM 1 4 4 Warp size 32 32 32 Shared memory per SM 16 KB up to 164 KB up to 228 KB 32-bit registers per SM 8,192 65,536 65,536 Added since baseline L1 data cache, tensor cores, ITS, async copy + TMA, thread block clusters What did not move: the warp is still 32 threads, a block still runs on one SM, the SM still schedules warps and hides latency by switching between them, shared memory still lives on the SM, and global memory is still partitioned and wants coalesced access. Those are the invariants CUDA is built on. What did move: the SM got wider (more lanes, more schedulers), deeper (a general-purpose L1 data cache, a larger register file), and specialized (tensor cores, TMA). So use Tesla for the model and the vocabulary, and use the current architecture whitepaper for any number you plan to optimize against.\nTesla Hardware to CUDA Software Every abstraction in the CUDA C post is a name for something on this chip:\nTesla hardware CUDA C concept Compute work distributor how a grid\u0026rsquo;s blocks spread over SMs (transparent scalability) SM where one thread block runs, start to finish Warp (32 threads, 4 clocks over 8 SP) the SIMT execution and scheduling unit SP (scalar MAD ALU) the \u0026ldquo;CUDA core\u0026rdquo; SFU __sinf, rsqrtf, and other intrinsics Register file + 16 KB shared memory per SM the resources occupancy trades against Shared memory __shared__ LSU + interleaved DRAM partitions why coalescing and bandwidth matter Read the CUDA C post after this and the abstractions stop being arbitrary. warp = 32 is the SM streaming 32 threads over 8 SPs. Occupancy is the SM\u0026rsquo;s finite register file and shared memory. Coalescing is the DRAM partitions wanting wide transactions. CUDA did not invent these; it exposed them.\nReferences Lindholm, Nickolls, Oberman, Montrym, \u0026ldquo;NVIDIA Tesla: A Unified Graphics and Computing Architecture\u0026rdquo;: the primary source for the SPA/TPC/SM/SP hierarchy, SIMT, and the graphics data path. NVIDIA GeForce 8800 GPU Architecture Technical Brief: G80 clocks, partitions, and register/shared-memory sizes. CUDA C++ Programming Guide: how the software model maps onto this hardware, on current architectures. NVIDIA Hopper (H100) Architecture: the modern SM (128 FP32 lanes, 4th-gen tensor cores, TMA) that the Tesla lineage grew into. CUDA C++ Best Practices Guide: coalescing, occupancy, pinned memory, and host-device transfer applied to today\u0026rsquo;s hardware. ","permalink":"https://bias92.github.io/en/posts/cuda-0-gpu-architecture/","summary":"Before CUDA C: the Tesla unified architecture (G80/GT200) from the 2008 IEEE Micro paper. The graphics-to-compute data path (Input Assembler, work distribution, SPA, TPC, SM/SP, SFU/LSU, ROP, DRAM), how the warp and SIMT were born in hardware, clock domains, and how every structure becomes a CUDA term.","title":"00 GPU Architecture Primer: The Tesla Foundation"},{"content":"Problem Flashing JetPack 6.2.2 (L4T R36.5.0) onto a Jetson AGX Orin 64GB Developer Kit fails at the tegrarcm_v2 stage with ERROR: might be timeout in USB write on an AMD B650-based host PC. This happens regardless of the host OS environment (Windows WSL2, Ubuntu Live Boot, Ubuntu native install).\n[ 0.2574 ] Sending bct_br [ 0.2576 ] ERROR: might be timeout in USB write. Error: Return value 3 Command tegrarcm_v2 --new_session --chip 0x23 ... Background The target setup was a real-time voice chat demo (LlamaSpeak) running on-device. The board arrived pre-flashed with JetPack 5.1.3 (CUDA 11.4), and text-based LLM inference (Llama 3 8B via MLC) was already working. However, every available ASR path required JetPack 6.\nASR Option Failure Reason on JP5.1.3 Riva 2.14.0 Models removed from NGC (HTTP 403) Riva 2.19.0 Requires CUDA 12, cudaError_t 35 Riva 2.13.1 riva_init.sh internal NGC CLI ignores host API keys whisper_trt JP6-only module, ImportError This made a JetPack 6 reflash mandatory.\nEnvironment Host PC (AMD, fails) AMD B650 AORUS ELITE V2 RTX 4060 Ti USB 3.x ports (AMD USB controller) Host PC (Intel, works) Samsung NT930SBE laptop Intel Core i5-8265U 8GB RAM, USB-C only (3 ports) Target Jetson AGX Orin 64GB Developer Kit eMMC + 1TB NVMe Original USB-A to C data cable Root Cause: AMD USB Controller Incompatibility NVIDIA\u0026rsquo;s tegrarcm_v2 tool communicates with the Jetson in APX recovery mode over USB. On the AMD B650 USB controller, this communication fails at the very first stage (Sending bct_br) with a write timeout.\nThe following environments were tested on the AMD host, and all produced the identical error.\nHost Environment Result Windows SDK Manager (WSL2) USB timeout Ubuntu 22.04 Live Boot (USB) USB timeout Ubuntu 22.04 Native (External SSD) USB timeout Windows SDK Manager (Native WSL) USB timeout Mitigations attempted (all ineffective):\nDisabling USB autosuspend: echo -1 \u0026gt; /sys/module/usbcore/parameters/autosuspend Different USB ports (front/rear) Zadig driver replacement (VBoxUSB → WinUSB) Different host OS environments The common denominator was the AMD USB controller. Switching to an Intel-based host eliminated the USB timeout entirely, and tegrarcm_v2 completed with zero errors on the first attempt.\nPartial Flash Consequences Before the root cause was identified, one flash.sh run on the AMD host partially completed. The result:\nCLI boot worked (tty2 login functional) Internet connectivity intact nvidia kernel modules loaded dpkg showed L4T 36.5.0 packages installed GUI completely broken: black screen after the NVIDIA logo /dev/nvhost* missing: zero GPU device nodes Xorg: eglInitialize() failed, no screens found Weston: NvRmMemInitNvmap failed with Permission denied The root cause was an incomplete Device Tree Blob (DTB) due to a corrupted USB transfer. This is not recoverable through software: no amount of xorg.conf editing, package reinstallation, or ldconfig configuration will fix missing device tree entries. The only fix is a clean reflash with reliable USB transfer.\nflash.sh reporting success does not guarantee a clean flash. USB transmission errors can silently corrupt firmware/DTB. Always verify /dev/nvhost* presence after flashing.\nSolution: Intel Host + chroot The Intel laptop had only 8GB RAM and could not boot from the external SSD via BIOS. This introduced two additional problems that had to be solved.\nRAM exhaustion: flash.sh builds a ~6GB system.img in memory. With the Ubuntu GUI running in Live Boot, 8GB RAM is insufficient, so a segfault occurs during sed operations on the rootfs. SQUASHFS corruption: Ubuntu Live Boot loads system binaries from a SQUASHFS filesystem on the USB stick. Under heavy I/O, the USB stick degraded, producing read errors, and a segfault occurred when loading sed, bash, and so on. Failed Intermediate Approaches Approach Failure Live Boot + GUI Segfault, RAM exhaustion Live Boot + GUI + 4GB swap on SSD Segfault, GUI still consuming too much RAM Live Boot + CLI (systemd.unit=multi-user.target) Segfault, SQUASHFS read errors on dying USB stick Working Approach: CLI Boot + chroot The solution was to use the USB stick only as a minimal bootstrap, then chroot into a full Ubuntu installation on the external SSD. Inside the chroot, all binaries load from the SSD, completely bypassing the USB stick\u0026rsquo;s SQUASHFS.\nStep 1: Boot into CLI mode At the GRUB menu, press e to edit the boot entry. Append systemd.unit=multi-user.target to the end of the linux line.\nlinux /casper/vmlinuz file=/cdrom/preseed/ubuntu.seed maybe-ubiquity quiet splash --- systemd.unit=multi-user.target Press F10 to boot. Log in with ubuntu (no password).\nStep 2: Mount SSD and enter chroot sudo -i mount -o rw /dev/sda1 /mnt mount --bind /dev /mnt/dev mount --bind /proc /mnt/proc mount --bind /sys /mnt/sys mount --bind /run /mnt/run chroot /mnt /bin/bash export PATH=/usr/bin:/usr/sbin:/bin:/sbin Step 3: Configure networking WiFi must be configured outside the chroot (NetworkManager runs on the host).\n# Exit chroot exit # Connect WiFi nmcli dev wifi connect \u0026lt;BSSID\u0026gt; password \u0026#34;\u0026lt;password\u0026gt;\u0026#34; # Re-enter chroot chroot /mnt /bin/bash export PATH=/usr/bin:/usr/sbin:/bin:/sbin echo \u0026#34;nameserver 8.8.8.8\u0026#34; \u0026gt; /etc/resolv.conf Step 4: Install dependencies apt-get update \u0026amp;\u0026amp; apt-get install -y libxml2-utils binutils Both are required by flash.sh: xmllint for XML validation and strings for binary inspection.\nStep 5: Flash Put the Jetson in recovery mode (unplug power → hold the middle Force Recovery button → plug power → wait 10s → release). Connect the USB cable, then run:\ncd /home/\u0026lt;user\u0026gt;/nvidia/nvidia_sdk/JetPack_6.2.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/ ./flash.sh jetson-agx-orin-devkit internal The internal flag targets eMMC, preserving any data on NVMe.\nResult [ 913.6337 ] Flashing completed [ 913.6338 ] Coldbooting the device *** The target generic has been flashed successfully. *** All 60 partitions written at 100%. After rebooting the Jetson:\nUbuntu GUI: functional /dev/nvhost*: 15+ device nodes present nvidia-smi inside Docker: Driver 540.5.0, CUDA 12.6 Post-Flash Setup # Install JetPack SDK components sudo apt update \u0026amp;\u0026amp; sudo apt install -y nvidia-jetpack # Verify CUDA export PATH=/usr/local/cuda/bin:$PATH nvcc --version # Should show CUDA 12.6 # Mount NVMe (if applicable) sudo mkdir -p /mnt/nvme sudo mount /dev/nvme0n1p1 /mnt/nvme echo \u0026#34;/dev/nvme0n1p1 /mnt/nvme ext4 defaults 0 2\u0026#34; | sudo tee -a /etc/fstab # Docker setup with NVMe storage sudo apt install -y docker.io nvidia-container-toolkit sudo tee /etc/docker/daemon.json \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; { \u0026#34;data-root\u0026#34;: \u0026#34;/mnt/nvme/docker\u0026#34;, \u0026#34;default-runtime\u0026#34;: \u0026#34;nvidia\u0026#34;, \u0026#34;runtimes\u0026#34;: { \u0026#34;nvidia\u0026#34;: { \u0026#34;path\u0026#34;: \u0026#34;nvidia-container-runtime\u0026#34;, \u0026#34;runtimeArgs\u0026#34;: [] } } } EOF sudo systemctl restart docker # Verify Docker + GPU sudo docker run --rm --runtime nvidia nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi Summary Factor Detail Root cause AMD USB controller incompatible with tegrarcm_v2 Solution Use an Intel USB host RAM constraint chroot into SSD-based Ubuntu to avoid the Live Boot SQUASHFS dependency CLI boot Required to keep RAM usage under 8GB Flash target internal (eMMC) to preserve NVMe data Verification Check /dev/nvhost*, not just the flash.sh exit status References NVIDIA SDK Manager Install Guide JetPack Installation Guide LlamaSpeak Tutorial NVIDIA Developer Forum, Jetson AGX Orin Forum Post: USB Timeout on Flash ","permalink":"https://bias92.github.io/en/posts/jetson-flash-troubleshooting/","summary":"Documenting the root cause of the tegrarcm_v2 USB write timeout on AMD hosts when flashing a Jetson AGX Orin, and a chroot-based workaround using an Intel laptop with limited RAM.","title":"JetPack 6.2.2 Flash Troubleshooting: AMD USB Incompatibility and the chroot Solution"},{"content":"The Growth of DNNs and the Need for Efficiency On ImageNet, AlexNet brought the Top-5 error down to 16.4% in 2012, then GoogLeNet 6.7%, ResNet 3.6%, and SENet 2.3%, passing the human error rate (5.1%) from 2015. Model size and compute grew along the way.\nModel size went from Transformer (0.05B, 2017) → GPT-2 (1.5B) → GPT-3 (175B) → MT-NLG (530B, 2022), roughly 10,000x in 5 years. Over the same period GPU memory grew about 5x, from TPUv2 16GB to A100 80GB. Because of this gap, models stop fitting on a single GPU and distributed techniques like tensor parallelism become necessary. ChatGPT showing \u0026ldquo;at capacity\u0026rdquo; or limiting users to 50 messages per 3 hours was also a serving cost problem.\nRedundancy DNNs contain a lot of redundancy. Many weights contribute little to the output or overlap with other weights in what they do. This redundancy is what makes pruning, quantization, and distillation work: unnecessary weights can be removed, important information survives lower precision, and the core knowledge can be moved into a smaller model. Large models also tolerate quantization better because they have more redundancy, so the remaining weights absorb the precision loss of the others.\nModel Compression Model Compression is the umbrella category, with pruning (removing unimportant weights, neurons, heads, or tokens), quantization (FP32 → FP16 → INT8 → INT4), knowledge distillation, low-rank factorization, and NAS underneath. These techniques are combined rather than used alone. In the Lite Transformer example, applying quantization and pruning on top of the architecture change went from 176MB to 9.7MB (18.2x) while BLEU only dropped from 39.9 to 39.5.\nInteger vs Floating Point Quantization maps the continuous weight distribution of FP16 onto discrete values like INT8 (-128 to 127) or INT4 (-8 to 7) with a scale factor. FP16 takes 2 bytes per weight, INT8 takes 1, and INT4 takes 0.5, so memory shrinks, and integer arithmetic is faster and uses less power than floating point in hardware. The cost is a possible drop in accuracy.\nIn practice the formats are mixed by use case: FP32 or BF16 for training, FP16 or W8A8 for server inference, W4A16 or W4A8 for edge inference, and FP16 for attention because of softmax precision. Starting with Blackwell (B100/B200), FP4 (NVFP4) has hardware support. It is 4-bit but keeps floating point semantics, so its dynamic range is wider than INT4, and NVIDIA claims less than 1% accuracy drop versus FP8.\nQuantization Techniques RTN (Round-To-Nearest) rounds each weight to the nearest integer. It holds up on large models (70B+) or at INT8, but quality breaks down at INT4 on a 7B model. The lecture shows an example where the model is given the Mona Lisa and fails to answer \u0026ldquo;Leonardo da Vinci\u0026rdquo;.\nAWQ (Activation-aware Weight Quantization) is from Song Han\u0026rsquo;s lab. It analyzes activation distributions and protects the important weight channels while quantizing. It has become the de facto standard for weight-only (W4A16) quantization. At the same INT4, the quality gap versus RTN is large.\nSmoothQuant deals with the activation outlier problem. Activations are hard to quantize because of outliers while weights are evenly distributed and easy, so the transformation Y = XW = (X/s)(sW) migrates quantization difficulty from activations to weights. Both sides then become easier to quantize, which makes W8A8 workable.\nTinyChatEngine is a lightweight LLM inference engine written from scratch in C/C++ by Han\u0026rsquo;s lab. It serves AWQ-quantized models on edge devices. There is a demo running LLaMA-2-7B W4A16 at 30 tokens/s on a Jetson Orin.\nSparsity and Pruning Pruning sets unimportant weights to zero, which makes the matrix sparse. Multiplication by zero always gives zero, so those operations can be skipped. A100\u0026rsquo;s 2:4 structured sparsity uses a pattern where 2 out of every 4 values must be zero, which the Tensor Cores recognize in hardware and skip, for a theoretical 2x speedup.\nThe jump from Q8000 (Turing) 261 TOPS to A100 1248 TOPS in the lecture graph should not be read as a sparsity effect alone. A100\u0026rsquo;s INT8 dense performance is 624 TOPS, so 261 → 624 comes from the generation change (3rd-gen Tensor Cores), and 624 → 1248 corresponds to the theoretical 2x from 2:4 sparsity.\nSpAtten SpAtten prunes unimportant tokens and heads at runtime in attention. In the sentiment classification example on \u0026ldquo;As a visual treat, the film is almost perfect\u0026rdquo;, layer 1 computes all 11 tokens and 12 heads, layer 2 drops to 5 tokens and 10 heads (38%), layer 3 to 2 tokens and 8 heads (12%), and at the end it classifies Positive with only \u0026ldquo;film perfect\u0026rdquo; left.\nToken importance is computed by accumulating attention probability vertically. Function words like \u0026ldquo;I\u0026rdquo;, \u0026ldquo;a\u0026rdquo;, \u0026ldquo;the\u0026rdquo; have low cumulative importance and get cut first. The distinguishing point is that this is token/head-level dynamic pruning whose outcome changes per input, not weight pruning. Where FlashAttention reduces attention\u0026rsquo;s memory access, SpAtten reduces the amount of computation itself.\nOn-Device / Edge AI When data cannot leave the device for privacy, offline, or personalization reasons, both inference and training have to happen on the edge device. Training is the harder one. Inference is a single forward pass, but training stores all activations from the forward pass, computes gradients in the backward pass, and keeps optimizer state on top.\nMCUNet V3 The MCUNet V3 paper reports cutting training memory from 303MB (PyTorch) down to 141KB, a 2300x reduction. Against the 652MB TensorFlow (cloud) baseline it is about 4600x. Three techniques applied in order:\nQuantization-Aware Scaling: correct gradients by the inverse square of the weight scale. 5.7MB → 2.9MB (2.0x) Sparse Layer Update: run backward only on important layers. 2.9MB → 335KB (8.8x) Sparse Tensor Update: update only the important parts within a tensor. 335KB → 141KB (2.4x) VLM and VILA ViT is a vision encoder that splits an image into patches and feeds them to a Transformer. A VLM attaches an LLM to that and understands both images and text; LLaVA and GPT-4V belong here. LLaVA is a CLIP ViT + projection layer + LLaMA structure. VILA is a VLM where NVIDIA and Song Han\u0026rsquo;s lab improved on LLaVA, and it can be quantized with AWQ and deployed on a Jetson. VLM (image → text, understanding) and Stable Diffusion (text → image, generation) go in opposite directions; what they share is the use of vision-language alignment models like CLIP.\nHardware Trend Looking at 42 years of microprocessor data, transistor counts keep rising but clock frequency and single-thread performance have stalled since around 2005 (the end of Dennard scaling). The direction since then has been dedicated compute units like Tensor Cores rather than more general-purpose cores.\nThe \u0026ldquo;317x in 8 years\u0026rdquo; figure in the lecture (K20X 3.94 → A100 1248) needs careful reading. The K20X number is FP32 TFLOPS while the A100 number is INT8 with sparsity included, so it is not a fixed-precision performance gain. It is a trendline that folds together precision reduction, Tensor Cores, and sparsity.\nMemory Wall Compute (TOPS) grows 2 to 3x per generation while HBM bandwidth grows around 2x. As the gap accumulates, the bottleneck becomes memory access rather than compute. This is why FlashAttention increases computation through recomputation and still wins by reducing HBM reads/writes.\nCloud vs Edge The lecture slides have a table comparing B100 (1,750 TOPS, 700W) against Qualcomm and Apple mobile chips in TOPS/W, with the edge chips ahead in efficiency. Edge chips are designed around performance per watt because of battery and thermal constraints. The numbers themselves are vendor marketing figures though, and do not compare directly: the precision behind each TOPS figure differs by vendor and is mostly undisclosed, and the power side mixes GPU module TDP with whole-SoC estimates. What to take away is roughly this: with a power budget around 10W, computation has to be cut in software through quantization and pruning.\nCourse Overview The following lectures cover pruning/sparsity, quantization, NAS, knowledge distillation, on-device training, efficient attention (FlashAttention and others), and distributed training/inference one by one. Software (6.5940) is best read alongside Hardware (6.5930).\n","permalink":"https://bias92.github.io/en/posts/6.5940-lec01-introduction/","summary":"Notes on MIT 6.5940 (Song Han) Lecture 1. From why DNN efficiency matters through Model Compression, Quantization (AWQ/SmoothQuant/RTN), Sparsity, Edge AI, VLM/VILA, and Hardware Trend.","title":"6.5940 L01: Introduction and Overview"},{"content":"A Word from Ilya Sutskever (L01-3) Early in the lecture there is a quote from Ilya Sutskever.\n\u0026ldquo;Compute has been the oxygen of deep learning.\u0026rdquo;\nHe said it at the 50th anniversary event for the ACM Turing Award in 2017. Good algorithms alone run nothing. Compute has to back them up. I read this sentence as the shortest answer to why MLSys is needed.\nModels keep changing. CNNs gave way to Transformers, and structures like MoE and Mamba are common now. Each shift shakes the model-side knowledge, but the problem of running models efficiently on real hardware does not go away. \u0026ldquo;AI-resistant career\u0026rdquo; sounds a bit grand, but I agree that MLSys is a field with staying power.\nAI Ingredients: The Three Elements (L01-2) The lecture groups the ingredients of today\u0026rsquo;s AI into Big Data, GPU Acceleration, and New ML Techniques.\nBig Data. Facebook receives 350 million images a day, YouTube accumulates 300 hours of video per minute, and Walmart processes 2.5PB of data per hour. Large-scale training became possible because this much data piled up for models to consume.\nGPU Acceleration. GPUs like the Tesla T4 process matrix multiplications in massive parallel. They brought training workloads that CPUs could not handle into realistic time budgets.\nNew ML Techniques. AlexNet opened the CNN era and the Transformer created today\u0026rsquo;s LLMs. When the algorithm changes, the conditions for well-matched hardware change with it.\nRemove any one of the three and AI at today\u0026rsquo;s scale would be hard to reach. This course digs into the compute side.\nHow GPUs Evolved for DNNs (L01-4) Up through Pascal, matrix multiplication was mostly handled by general-purpose CUDA cores. In 2017 the V100 added Tensor Cores. Instead of repeating scalar FMAs on CUDA cores, small-matrix multiply-accumulate runs on a dedicated unit. That raised DNN throughput per area substantially, and FP16 support added to the gain.\nWith the A100 in 2020, Tensor Cores began supporting 2:4 structured sparsity. If the pattern of two zeros out of every four consecutive elements is met, the zero terms are skipped and only the valid values are computed.\nNot every sparse matrix gets faster here. The pruning result has to fit the 2:4 pattern the hardware understands. This is one place where it becomes clear that algorithms and hardware cannot be designed separately.\nSoftware Companies Building Their Own HW (L01-5) Google and Amazon buy NVIDIA GPUs in volume and also build their own chips. The reasons are cost and optimization.\nA GPU is a general-purpose device that has to serve everything from graphics to scientific computing. If the datacenter workload narrows down to DNN inference and training, a chip built for just those operations can win on power and area.\nGoogle\u0026rsquo;s TPU v1 came out for inference and used a 256×256 systolic array. From v2 it supported training as well. Amazon built Inferentia and Trainium to lower inference and training costs on AWS. In Korea, FuriosaAI and Rebellions are attacking the same problem in their own ways.\nCerebras WSE (L01-6) Cerebras WSE is the case that pushes this direction to the limit. Normal chips are small dies cut from a wafer; Cerebras uses the entire wafer as one chip. That gives it 18GB of on-chip SRAM. Considering that A100 on-chip SRAM is around 20MB and TPUs stay under 32MB, the scale is different.\nIt is not free, of course. Yield, cooling, and price problems follow. It is less a widely used answer than a design that shows how bold you can get for a specific workload.\nMobile SoCs for DNNs (L01-7) DNN-specific units are not only in datacenters. They are already in phones and laptops.\nApple\u0026rsquo;s A11 first shipped a 2-core ANE in 2017, mainly for on-device inference like Face ID and Animoji. The M2\u0026rsquo;s ANE in 2022 has 16 cores. The 26x performance gain over A11 includes both the core count increase and per-generation architecture improvements.\nOn mobile, model size, integer quantization, and prunability matter in particular, because power and memory cannot be spent as freely as on a server.\nRising Energy Consumption of Computing (L01-8) The slide cites a 2024 Goldman Sachs projection that US datacenters\u0026rsquo; share of electricity could grow from 3% in 2022 to 8% by 2030, along with a forecast that ICT as a whole could account for 20.9% of global electricity.\nModels keep growing and power is not unlimited. That is how finishing the same computation with less energy became a systems problem.\nWhat matters here is that data movement costs more than the computation itself. In the slide\u0026rsquo;s 65nm numbers, one DRAM access consumes about 200x the energy of an ALU operation. Making the arithmetic units faster is not enough. The number of long-distance data trips has to come down first.\nThat is why Cerebras made its on-chip SRAM huge. Ordinary accelerators follow the same principle with a memory hierarchy of register file, local buffer, global buffer, and DRAM, reusing data from the near end as much as possible.\nComputing Cost of ChatGPT (L01-12) GPT-3 has 96 layers and 175B parameters. Training is put at a total of 3.14×10²³ FLOPs.\nBy the slide\u0026rsquo;s arithmetic, that takes 355 years on a single V100 and costs about $4.6M in cloud compute, roughly 6 billion KRW. GPT-4's training cost is estimated at over $100M.\nThe exact range matters less than the scale. Each generation of model growth pushed the compute cost up at a pace that is hard to absorb.\nChanging Trends: DeepSeek (L01-13) The slide puts GPT-4\u0026rsquo;s estimated training cost next to DeepSeek\u0026rsquo;s published figure: over 130 billion KRW for GPT-4 versus about 8 billion KRW for DeepSeek.\nThe two are not the same line item, though. The DeepSeek number is close to the GPU rental cost of the final training run, while the GPT-4 number is a much broader total cost estimate. Dividing one by the other and calling it several times cheaper breaks the comparison.\nWhat stands out in DeepSeek\u0026rsquo;s architecture is MoE. Total parameters are 671B, but only 37B activate per token. Instead of running every expert every time, it picks the necessary subset and cuts the computation.\nTraining vs Inference (L01-15) Training costs a lot per run, but per model it runs few times. Inference costs relatively little per call and keeps getting called for as long as the service is alive.\nSending one question to a chatbot and getting an answer is one inference. When such requests pile up to trillions per day, the cumulative cost becomes a bigger problem than training. This is also why LLM inference optimization exists as its own job category.\nWhy On-Device (L01-18) There are three reasons to run inference on the device instead of the cloud.\nCommunication. Where the network is absent or unstable, a cloud model cannot be called.\nPrivacy. Data that should not leave, like medical records or defense data, is safer processed on the device.\nLatency. In systems that must react immediately, like autonomous driving, tens to hundreds of milliseconds of round trip can be fatal.\nSelf-Driving Cars (L01-19) Autonomous driving is an example where the demands of edge inference show up all at once.\nCameras and radar generate about 6GB of data every 30 seconds. Assume one car runs 10 DNNs at 60Hz over 10 cameras: that is 21.6M inferences per hour. With a million vehicles, 21.6 trillion per hour.\nWhen a prototype compute rig draws 2,500W, air cooling barely holds. Sending all the sensor data to the cloud is not an option either; the traffic and latency turn directly into safety problems.\nThe Slowing of Moore\u0026rsquo;s Law and the Need for Domain-Specific HW (L01-22) Behind MLSys becoming its own field are the slowdowns of Moore\u0026rsquo;s Law and Dennard Scaling.\nTransistor counts still grow, but not at the old rate. Transistors per dollar are stagnating too. Dennard Scaling, where power shrank along with transistor size, broke around 2005. That is why clock speed and TDP no longer climb much in the slide.\nSince general-purpose processor performance and efficiency no longer improve automatically with a new process, hardware matched to specific operations became necessary. Tensor Cores focus on matrix multiplication, Google TPUs on DNN training and inference, Apple ANE on on-device inference, and FuriosaAI NPUs on AI inference.\nL01-23 to L01-25: The Evolution of the CPU Pipeline This part shows how much control logic a general-purpose CPU carries to achieve high utilization. For regular matrix computations like DNNs, that complexity does not turn into gains.\nSimple In-Order Pipeline (L01-23) The pipeline on the slide is a 6-stage version of the IF, ID, EX, MEM, WB structure, with ID split into Decode and Reg Read. Doing less per stage leaves room to raise the clock. It handles multiple instructions per cycle, so it is also superscalar.\nFetch reads the instruction at the PC\u0026rsquo;s address from the I-cache. The next PC is chosen by the branch predictor.\nDecode interprets the opcode, rs, rd, and immediate. Reg Read pulls the needed operands from the register file.\nExecute is where the ALU does the actual work. Three red chevrons in the figure mean three functional units usable at once. Think of a combination like an integer ALU, an FP or multiply unit, and a branch unit.\nThe D-cache and store buffer handle loads and stores, and Reg Write puts results back into the register file.\nBasic Out-of-Order Pipeline (L01-24) An in-order superscalar stalls everything behind an instruction that hits a cache miss, even if later instructions are ready. Three functional units sit idle if work does not arrive.\nAn out-of-order design adds an issue queue and sends whichever instructions have their operands ready. While an earlier load waits on memory, independent later instructions fill the empty slots.\nSMT: Simultaneous Multi-Threading (L01-25) SMT here is a different concept from the GPU\u0026rsquo;s SIMT.\nThe red, yellow, and green in the figure are different threads. The pipeline is not replicated per thread; multiple threads share one pipeline. Intel famously worked this to great effect under the name Hyper-Threading.\nWhen one thread stalls, ready instructions from another thread fill the empty slots and raise throughput.\nStructure Characteristics Scalar 5-stage, width=1, one instruction per cycle Superscalar (In-Order) width\u0026gt;1, several per cycle, in order OoO + Superscalar width\u0026gt;1, ready instructions go first OoO + SMT multiple threads share one pipeline Branch prediction, register renaming, the issue queue, retire, thread choosing: all of it is control logic added for generality. For the repetitive MatMuls of DNNs this complexity is overkill. That is why accelerators lead with a PE array and a memory hierarchy instead of a complex CPU pipeline.\nAccelerator Diversity (L01-26) Not all DNN accelerators look alike. When the target workload changes, PE placement, memory hierarchy, and dataflow change too.\nEyeriss is a spatial array for CNN inference with a local scratchpad in each PE.\nEyeriss V2 added a more flexible NoC.\nSCNN focuses on skipping zeros in sparse CNNs.\nExTensor and Gamma target sparse tensor and sparse matrix computation respectively.\nspZip handles compression and decompression of sparse data.\nCPU pipelines are fairly standardized in their broad structure. Accelerators diverge from the design stage depending on dense versus sparse, CNN versus Transformer. Analyzing these differences in a common language is the main body of the course.\nPE stands for Processing Element. It is a small compute unit close to a CPU\u0026rsquo;s ALU, usually a multiplier, an adder, and a small register file bundled to handle MACs.\nTeAAL Pyramid of Concerns \u0026amp; FuseMax (L01-28 to L01-34) The vocabulary suddenly thickens here. In the first lecture, rather than memorizing every name, it is enough to pick up the sense of splitting things into layers.\nTeAAL Pyramid of Concerns (L01-28) TeAAL splits accelerator design into four layers. Decisions get finer toward the top, and the architecture constrains what each layer can do.\nLayer Meaning Course link Compute What operation runs. MatMul, Attention, etc. Lab 1: Einsum Mapping How the operation is placed on hardware. Tiling, dataflow, etc. Lab 2, 3 Format How data is stored. Dense, sparse, etc. Lab 4: Sparsity Binding How operations and data are assigned to actual PEs and buffers Lab 2, 3 Most of the design problems later in the course can be placed back into these four boxes.\nFuseMax: The Pyramid Applied (L01-29) FuseMax is a design that accelerates Transformer attention. Reworking compute, mapping, and binding in turn, it lifts a low PE utilization to about 90%.\nStep Layer Change Result Cascade Compute Fuse MatMul and Softmax to cut intermediate data movement small utilization gain Architecture change Architecture and Mapping Change the structure so a new mapping becomes possible further gain Binding improvement Binding Distribute work more evenly across PEs about 90% utilization Unfused runs Q×K, Softmax, and ×V separately, writing intermediate results to memory at each step. FLAT is the existing partial fusion approach. Cascade binds the stages more tightly so intermediates never leave for memory.\nNo single step delivers the performance alone. How the operations are fused, the hardware structure, and the actual resource assignment have to line up together.\nPE Utilization Across the Steps (L01-30 to L01-34) In the L01-30 baseline, BERT, TrXL, T5, and XLM all sit below 0.25 PE utilization from sequence length 1K to 1M. Hundreds of PEs are built and mostly idle.\nThe x-axis is sequence length, the y-axis PE utilization. Red Unfused and orange FLAT both stay low. Applying Cascade, Architecture, and Binding in turn from L01-31 to L01-33 raises utilization to about 90%.\nThe Basic Structure of a DNN Accelerator (L01-43) A typical DNN accelerator has a PE array and a memory hierarchy instead of a complex CPU pipeline.\nData comes from DRAM into a Global Buffer and on to the PE array. Each PE holds a Reg File smaller than 1kB, an ALU, and simple control. PE counts run from hundreds to thousands, and the Global Buffer is roughly 100kB to 500kB.\nEach PE concentrates on simple, repetitive operations like MAC. With no area spent on branch prediction or register renaming, more compute units fit in the same chip area.\nThe slide\u0026rsquo;s normalized energy cost at 65nm:\nWhere the data comes from Energy cost The ALU operation itself 1× Reg File to ALU 1× Inter-PE NoC to ALU 2× Global Buffer to ALU 6× DRAM to ALU 200× One DRAM access costs 200x the actual computation. The same table explains why Cerebras put in 18GB of SRAM.\n\u0026ldquo;Farther and larger memories consume more power.\u0026rdquo;\nKeep this sentence in mind and the dataflow and tiling discussion later in the course reads much more easily.\nAccelerator Design Decisions (L01-44) The items a designer has to choose connect to the course Labs.\nDesign decision Details Lab PE array Number of PEs and how they connect Lab 2, 3 Memory hierarchy Number of levels, capacity, data placement Lab 2, 3 Scheduling Mapping, dataflow, tiling, parallelism, fusion Lab 2, 3 Sparsity handling Gating, skipping, compression formats Lab 4 Implementation technology RRAM, optical, superconductors, etc. Lab 5 TeAAL\u0026rsquo;s Compute, Mapping, Format, and Binding come down to actual design variables here.\nRoofline-Based Inefficiency Analysis (L01-45) The Roofline Model is a frame for seeing whether performance is blocked by computation or by memory bandwidth. The x-axis is compute intensity, the y-axis performance. A larger x value means more operations per piece of data fetched. The sloped line is the limit set by memory bandwidth; the flat line on the right is the limit set by compute peak.\nStuck to the left slope means memory-bound; stuck to the right plateau means compute-bound.\nSteps 1 through 7 on the slide show how much the theoretical peak gets shaved as it meets real constraints.\nStep Constraint Lab Step 1 Maximum parallelism in the workload itself Lab 1 Step 2 Maximum parallelism the dataflow allows Lab 2, 3 PE count The hardware\u0026rsquo;s theoretical peak none Step 3 Finite PE array size Lab 2, 3 Step 4 PE array dimension constraints Lab 2, 3 Step 5 Finite storage capacity Lab 2, 3 Step 6 Insufficient average bandwidth Lab 2, 3 Step 7 Insufficient instantaneous bandwidth none Each constraint added tightens the roofline downward. Instead of stamping actual performance as a single number from the start, you can separate out where and how much got shaved.\n","permalink":"https://bias92.github.io/en/posts/mit-6.5930-l01/","summary":"\u003ch2 id=\"a-word-from-ilya-sutskever-l01-3\"\u003eA Word from Ilya Sutskever (L01-3)\u003c/h2\u003e\n\u003cp\u003e\u003cimg alt=\"Ilya Sutskever Quote\" loading=\"lazy\" src=\"/posts/mit-6.5930-l01/images/L01-3-ilya-quote.png\"\u003e\u003c/p\u003e\n\u003cp\u003eEarly in the lecture there is a quote from Ilya Sutskever.\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;Compute has been the oxygen of deep learning.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eHe said it at the 50th anniversary event for the ACM Turing Award in 2017. Good algorithms alone run nothing. Compute has to back them up. I read this sentence as the shortest answer to why MLSys is needed.\u003c/p\u003e\n\u003cp\u003eModels keep changing. CNNs gave way to Transformers, and structures like MoE and Mamba are common now. Each shift shakes the model-side knowledge, but the problem of running models efficiently on real hardware does not go away. \u0026ldquo;AI-resistant career\u0026rdquo; sounds a bit grand, but I agree that MLSys is a field with staying power.\u003c/p\u003e","title":"6.5930 L01 - Introduction and Applications"},{"content":" MIT EECS 6.5930, Prof. Vivienne Sze and Prof. Joel Emer\nCourse: https://csg.csail.mit.edu/6.5930/\nDraft in progress.\n","permalink":"https://bias92.github.io/en/posts/mit-6.5930-hardware-for-deep-learning/","summary":"MIT EECS 6.5930 by Prof. Vivienne Sze and Prof. Joel Emer, Hardware Architecture for Deep Learning","title":"MIT 6.5930: Hardware Architecture for Deep Learning"},{"content":"Interests GPU Systems \u0026amp; CUDA Kernel Optimization MLSys (vLLM, FlashInfer, DeepSpeed) TinyML and Efficient ML AWQ, GPTQ Megatron-LM (DGX Scale) ","permalink":"https://bias92.github.io/en/posts/hello-world/","summary":"GPU Systems \u0026amp; Deep Learning Hardware","title":"Opening Note"}]