Compiler

The compiler is written in Go. Pipeline:

• The low command lives in cmd/low. The pipeline it shares with hot reload lives in internal/compiler.
• low ir prints the SSA form and low asm the generated assembly.
• Compilation is deterministic: the same source and target always produce the same SSA form, assembly and binary. No pass depends on map iteration order; where order matters the compiler walks slices or stacks.

Modules:

• internal/compiler finds the root, loads the entry module, then loads each imported directory once, breadth first, and rejects import cycles.
• Every file records its module path and every source position its file, so errors name the file they come from.
• The checker keeps a scope of types and functions per module. The prelude belongs to the entry module, and its __ names cannot clash with anything else.
• A function in module ui/text becomes the symbol fn_ui.text.shout. Types display with their module path, such as geometry.Circle.

The runtime, internal/amd64/sched.asm and internal/amd64/runtime.asm, is emitted with every program and talks to the kernel through raw syscalls. It ends the process with exit_group, so every thread stops.

Scheduler

• Green threads run on OS threads, one per CPU in the process's affinity mask. Threads start as work appears, with clone in a static program and pthread_create in a dynamic one. A thread whose green thread is in a C call does not count against the limit, so another starts while C blocks.
• Run queues:
  • Each OS thread keeps a 256-entry run queue. A full queue spills to a global queue, and spawn queues on the global queue when its own thread already holds 16.
  • A thread takes work in this order: expired timers; the global queue every 61st schedule; its own runnext and queue; a batch from the global queue (its length divided by the thread count, plus one, at most 128); ready descriptors; stealing.
  • A thief takes half of the victim's queue into its own. It tries the victim's runnext only in the last of 4 rounds, and takes it at once when the victim is in a C call or stopped (asleep on a runtime lock or held by a collection), or when the running victim's runnext sequence has stayed unchanged for 2,000 timestamp-counter ticks.
  • At most half of the running threads spin looking for work. With nothing to do, a thread sleeps on a futex or becomes the single poller that blocks in epoll_wait until the next timer.
  • Every 32nd schedule on an OS thread moves runnext to the tail of its run queue and takes the queue's head, so green threads that keep readying each other cannot starve the rest of the queue.
• A green thread reserves 1 MB of stack with a guard page below it. Pages are committed as they are touched, and finished green threads are reused. Its context is its saved registers on its own stack. A fault in the guard page panics with stack overflow; any other fault panics with invalid memory access.
• Per-thread state lives in a 24-byte .tbss block directly below the thread pointer: the thread record at fs:-24, the running green thread at fs:-16 and the preemption flag at fs:-8. A static program sets fs with arch_prctl and CLONE_SETTLS; a dynamic program uses the block glibc allocates for every thread. Generated code uses the fixed offsets, so hot reload patches need no TLS relocations.
• Preemption:
  • Every loop back-edge tests the preemption flag.
  • A monitor thread sets the flag every 10 ms on a thread whose green thread has run since the last check, and the collector sets it everywhere.
  • A set flag calls rt_poll, which saves every general register and xmm2 to xmm15 and yields or stops for the collector.
  • Code that holds a channel lock does not yield. Recursion without loops is not preempted.
• park and ready hand a green thread off with a per-thread lock and a wake token, so a wakeup that arrives before the green thread has finished parking is not lost.
• Wakeups:
  • ready puts the green thread in its OS thread's runnext slot, so it runs as soon as the current one parks, and bumps a per-thread runnext sequence. A green thread pushed out of that slot goes to the stealable run queue.
  • ready then calls wakep, which acts only when its OS thread has not woken anyone in the last 100,000 timestamp-counter ticks or has not scheduled since it last did. A pair of green threads that hand work to each other therefore stays on one OS thread, while a sender that keeps computing after ready always has an idle thread woken to steal its runnext within the 2,000-tick delay.
  • wakep does nothing while an OS thread is already spinning. Otherwise it wakes one idle thread, or starts a new one below the CPU limit, and the woken thread counts as spinning until it finds work or gives up. A spinner that finds work while more is queued wakes a successor.
  • spawn always wakes a thread without the interval.
  • The monitor thread moves the runnext and wakes a thread for the queue of an OS thread that sits in a C call or a blocking system call.
• In-call threads:
  • An OS thread in a C call, or in a system call that can block, is marked in-call, which the collector treats as stopped. rt_world_stop does not wait for it, and on return from the call the thread parks in rt_m_running while a collection is in progress.
  • The code generator emits __syscall inline only when its number is a constant on a short list of calls that never block on external events: close, lseek, memory mapping, signals, pipe, dup, socket creation and configuration, process creation and exit, time, identity, limits, epoll_ctl, eventfd and getrandom.
  • Every other number, and every non-constant number, calls rt_sysblock. It saves the argument registers on the green thread's scanned stack, marks the thread in-call, makes the call and leaves the in-call state through the same path as a C call.
  • The monitor counts in-call threads that have not scheduled since its previous visit, and wakep does not count them against the CPU limit, so work queued behind one finds a running thread.
  • A green thread blocked in read on a pipe therefore does not delay a collection started by another.
• Timers are a locked array of deadlines. sleep and select's after arms add to it, and the poller wakes through an eventfd when an earlier timer arrives. Descriptors are registered with EPOLLONESHOT.
• Channels and select are generic prelude code over __Chan[T]: a lock word, a ring buffer and ring queues of waiters that keep their capacity.
  • A waiter is claimed with compare-and-swap before it is woken, so a select waiting on several channels and timers fires exactly once. Queued entries whose waiter has been claimed are skipped and dropped lazily.
  • An operation that completes at once allocates nothing. A buffered send or receive that needs no waiter runs inline in __chan_send and __chan_recv.
  • An operation that must wait allocates the waiter and value box under the lock it already holds, queues them, releases the lock and parks, so each blocking send or receive takes the lock once. Green threads to wake are readied after the lock is released.
  • select locks its channels in address order and tries each arm with a probe that returns the green thread to wake, using a stack slot as the receive box. A ready arm allocates nothing.
  • When no arm is ready and there is no else, select allocates one waiter, queues it on every channel with the stack slots as boxes, releases the locks, adds the after timers and parks. When woken it deletes the timers and clears the slots.
• Channel lock word: 0 when free, 1 when held, 2 when held with sleepers.
  • rt_lock takes a free lock with one compare-and-swap.
  • A contender that finds the lock held makes 4 polling rounds. Each round is a pause loop that waits 8,000 timestamp-counter ticks, doubling every round, before it reads the word, so contenders do not bounce the cache line that holds the channel's fields. The contender checks the stop flag each round and takes the lock with a compare-and-swap when it reads 0.
  • After the rounds it yields its OS thread once, then exchanges 2 into the word. Reading 0 means it holds the lock; otherwise it waits on a futex until woken and exchanges again. While it sleeps its OS thread counts as stopped, so a collection can proceed.
  • rt_unlock exchanges 0 into the word and wakes one sleeper only when the old value was 2.
• C calls run on the OS thread's system stack (8 MB in a static program). Around a C call the generated code pushes the callee-saved registers on the green thread's stack, records where the green thread stopped, marks the thread as in C and switches stacks. On return it marks the thread running again, waiting if a collection is under way.
• A C callback runs on the green thread's stack. Its thunk marks the thread running, switches back to the green thread's stack below the C call's saved registers, and locks the green thread to the OS thread until it returns, because the C frames live on that OS thread's system stack: if the callback waits, the OS thread waits with it. Callbacks must arrive on the thread that made the C call.

Linking

• A static program is linked with ld -static and starts at the generated _start. _start initialises the heap and the scheduler, starts main on the first green thread and enters the scheduler on the process stack.
• A dynamic program is linked with ld -dynamic-linker /lib64/ld-linux-x86-64.so.2 against glibc's crt1.o, crti.o and crtn.o, then -l<name> for each library and -lc.
  • The start files are looked for in /usr/lib, /usr/lib64, /usr/lib/x86_64-linux-gnu and /lib/x86_64-linux-gnu.
  • glibc's start-up code calls the generated C main, which does the same work as _start.
  • When Low's main returns, the program calls exit(0), letting libc run its exit handlers. An error from main prints and exits with status 1.

Prelude:

• internal/compiler/prelude.low is Low source compiled ahead of every program. It holds:
  • UTF-8 decoding.
  • The string builder behind interpolation and concatenation.
  • The generic map table.
  • Integer formatting.
  • Float formatting by unrounded scaling against a table of 128-bit powers of ten from 1e-348 to 1e347, which finds the shortest digits that read back to the same value without allocating.
• cmd/tablegen pow10 generates the table twice: as the __pow10_hi and __pow10_lo functions the prelude holds, and as std/text/pow10.bin, which std/text embeds for float parsing because standard library files cannot call prelude functions.
• Prelude names start with __, which user code cannot use. Four intrinsics are available only in the prelude:
  • __bits gives a float's bit pattern.
  • __str_view turns a []u8 into a str without copying.
  • __hash hashes any Hash value.
  • __zero is the all-zero value of its expected type.
• A function's IR is built only when main reaches it through a call or a function address, so a program that formats no floats neither builds nor carries any of this code. Reachable functions are built in call order and emitted in declaration order.

Standard library:

• It lives in internal/compiler/std and is embedded in low with go:embed.
• An import whose path is std or starts with std/ is read from there, never from the project root.
• Its files are lexed like the prelude, so they may use __ names and intrinsics, and they are left out of the sources low dev watches.
• embed in a standard library file reads from the embedded tree and writes the bytes once to lowlang/embed/ under the user cache directory, named by their SHA-256, for the assembler's .incbin.

Standard library intrinsics:

• __argc(), __argv(i) and __envv(i) read the argument count, argument i and environment entry i (null past the end) as NUL-terminated ptr[u8]s. The start code saves them from the initial stack in a static program and from main's arguments in a dynamic one.
• __syscall(n, args...) makes Linux syscall n with up to six integer, bool or ptr arguments and returns rax as i64, a negative errno on failure.
  • It lowers to a runtime-call instruction: syscall for a constant number on the non-blocking list and sysblock otherwise (see the Scheduler section).
  • The code generator emits syscall inline: it moves the arguments into rax, rdi, rsi, rdx, r10, r8 and r9 with the parallel move and executes syscall.
  • Being a call, it keeps no value in a caller-saved register across it, which covers the rcx and r11 the kernel clobbers.
• __slice_ptr(x) gives the data address of a slice or str as a ptr[T].
• __mulhi(a, b) is the high 64 bits of the 128-bit product of two u64. It is an IR instruction that becomes one mul.
• __hash_seed(i) is word i, a literal 0 or 1, of the process's 128-bit hash seed. It is an IR instruction that becomes one load from rt_hash_seed.
• __extend_by(mut b, n) grows a []u8 by n bytes with at most one allocation and returns the index of the first new byte, whose contents the caller overwrites.
• __make(n) is a zeroed slice of n elements of the expected slice type, allocated through the runtime's grow routine.
• __memmove(dst, src) copies min(dst.len, src.len) elements with the runtime's overlap-safe rt_memmove and returns the count. It takes a str source for a []u8 destination. rt_memmove loads up to 64 bytes into registers before storing them and uses rep movsb beyond that.
• __clock_nanos(id) returns clock_gettime(id) in nanoseconds, or 0 on failure, through rt_clock_nanos. rt_clock_nanos calls __vdso_clock_gettime when start-up found it from AT_SYSINFO_EHDR and the vDSO's dynamic symbol table (DT_HASH or DT_GNU_HASH), and makes the syscall otherwise.
• __context() reads the current green thread's context slot as the expected pointer type, __set_context(p) replaces it and __has_context() tells whether it is set. spawn copies the parent's slot into the child, and the collector marks every live green thread's slot.
• __serialize, __deserialize and __schema are the derivations described below, allowed only in std/serde.

The garbage collector is a non-moving mark-sweep collector in the runtime:

• The heap is a 16 GB reservation made with mmap and MAP_NORESERVE. Pages are committed as they are touched.
• Object layout:
  • Each object has a one-word header. Bits 3 and up hold the object's size in words, header included, so the word is the size in bytes with its low three bits used as flags: bit 0 is always zero, bit 1 marks a free chunk and bit 2 marks an object with no pointers to scan.
  • A bitmap records the first word of every object and free chunk, which resolves pointers into the middle of an object, such as a mut parameter aimed at a heap field.
  • Two summary bitmaps above it hold one bit per non-empty bitmap word and one per non-empty summary word. Finding the start behind an interior pointer skips empty spans 32 KB and 2 MB at a time instead of reading every bitmap word between the pointer and its object's header.
  • A second bitmap of the same shape holds the mark bit of each object. It is zero outside a collection.
  • The start bitmap, the mark bitmap, the first summary bitmap and the mark stack are touched sparsely or in small steps, so they are excluded from transparent huge pages on Linux; one touched byte would otherwise make a 2 MB page resident.
• Allocation buffers:
  • Each OS thread allocates objects of up to 255 words from its own buffer of up to 4096 words without taking the heap lock: it bumps a pointer, writes the header and sets the object's start bit.
  • Only the start-bitmap words at the buffer's unaligned edges are set atomically, with lock or; the other words are updated with a plain or. The summary bits are tested first and set atomically only when clear, and each thread remembers the last bitmap word whose summary bits it set, so allocation skips the summary update while it stays in that word. The memory is reset whenever the thread takes a new buffer.
  • A thread whose buffer is full takes the heap lock and refills it from, in order: the smallest free chunk large enough in the size-class bins below, the exact-size free lists from the largest down, then the end of the heap. It zeroes the buffer's dirty pages in one pass after releasing the lock.
  • A collection turns every thread's unused buffer tail into a free chunk before marking, and an allocation that would leave a single word in a buffer takes that word too, so the sweep finds a header at every object boundary.
• Free chunks:
  • Free chunks of 64 words or more sit in 256 size-class bins, four per power of two, and a 256-bit bitmap marks the non-empty ones. Bins 0 to 127 hold chunks whose pages may be non-zero (dirty) and bins 128 to 255 hold chunks known to be zero (clean).
  • An allocation scans up to eight chunks of its own bin for the smallest that fits, then takes the first chunk of the lowest non-empty bin above it, preferring a dirty bin to a clean one of the same class, and splits off the remainder.
  • Chunks under 64 words sit in exact-size free lists that only buffer refills read.
  • Objects over 255 words take the heap lock and allocate from the bins, then from the end of the heap, and are zeroed after the lock is released.
• Dirty pages:
  • A bitmap with one bit per 4096-byte heap page records which pages may hold non-zero words. Pages never handed out are zero from the operating system and are clean.
  • An allocation zeroes only the dirty pages of its range and then marks the whole range dirty, so a large zeroed object that is never written costs no resident memory.
  • Every free chunk of 1 MB or more whose interior pages are all clean is filed in a clean bin.
• Returning memory:
  • Before the heap end moves for an allocation, every dirty free chunk of at least 1 MB returns its whole interior pages to the operating system on Linux with madvise(MADV_DONTNEED), clears their dirty bits and moves to a clean bin.
  • The walk starts at the first bin that can hold such a chunk and visits dirty bins only, so a chunk is advised once and not again until it is reused and freed.
  • Resident memory holds the live objects and the new one rather than also the chunks that were too small.
  • On Windows the pages stay committed and chunks stay dirty.
• Collection runs when the words allocated since the last one exceed the words that survived it, with a 4 MB floor. A thread's allocation buffer counts by the size of the chunk it receives, so refilling from small free chunks after a sweep counts no more than the memory handed out.
• LOW_GCTRACE=1 prints one line per collection to stderr: gc N pause_us= mark_us= sweep_us= live_kb= heap_kb= slow_allocs= refills=, from timestamps around the stop, mark, sweep and restart and from per-thread counters of buffer refills.
• LOW_GC=off never collects. LOW_GC=stress collects after every 8192 words allocated.
• An allocation whose type holds no pointers is noscan: new of a pointer-free value, and the backing arrays of slices whose elements hold none, such as []u8, []i64 and []f32. A noscan object is marked but its contents are never read. Other objects are scanned word by word.
• Allocation and pinning take a heap lock. A collection stops the world: it sets a stop flag and every thread's preemption flag, then waits until each other thread is stopped at a safepoint, idle in the scheduler or in a C call, and holds those threads there until it finishes. Threads waiting for a lock stop for the collector while they wait.
• Roots come from stack maps.
  • At every call the code generator records the frame words that may hold a heap pointer at that point: the saved callee-saved registers, spill slots of values that may be pointers and are live across the call, and every frame slot. The maps form a table sorted by return address.
  • The words of rt_cells, one per hoisted literal, are scanned as roots in every collection.
• Green-thread stacks:
  • Every green thread records where it stopped: the return address, stack pointer and frame pointer of the Low code that entered the runtime, and the lowest stack word holding registers it saved.
  • The collector scans each live green thread's saved registers conservatively, then follows its rbp chain, looking up each frame's return address and scanning only its mapped words, and scans the green thread's first frame conservatively.
  • Frames without a map, such as code patched in by hot reload or a loop interrupted by a preemption check, are scanned whole. A green thread that has not started is scanned whole.
  • Values in callee-saved registers are found in the save slots of whichever later frame saved them, or among the registers saved when the green thread stopped.
• A callback thunk crosses C frames by segment. Its frame record holds the Low caller's frame pointer and return address, so the rbp chain leads from the thunk straight back to the caller. Its frame keeps the bounds of the C frames on the system stack and of the registers the C call saved. The thunk's call site has a map of count -1, which tells the collector to scan both regions conservatively before following the chain.
• Every candidate word, mapped or not, is checked against the heap before it is followed, so a map that names a word holding an integer is harmless.
• A ptr counts as a possible heap pointer, so a cstr copy or a pinned object stays alive while Low holds its address. pin records the address in a table of 65536 roots that every collection scans; unpin removes it.
• Marking resolves a candidate word first by testing the start bit of the word before it, which is the header when the candidate is an object's first field address, and falls back to the summary walk otherwise. Candidates are prefetched through a ring of pending headers before they are marked, so cache misses overlap. Marking sets the object's bit in the mark bitmap and adds its size to the live word count. Newly marked noscan objects are not pushed on the mark stack.
• Sweeping walks the start bitmap, jumping over empty words with the summary bitmaps, so its cost follows the number of objects and not the address span of the heap.
  • It rebuilds the free lists and bins from empty.
  • Within a word, the start bits without a mark bit are dead and the rest are live. Sweeping jumps between the two kinds of run to coalesce adjacent dead objects into free chunks.
  • A chunk keeps only its first start bit, and the word's mark bits are cleared once it is done.
  • A trailing free run returns to the end of the heap.

The derivations are generated as source:

• When the checker meets __serialize(v, w), __deserialize[T](r) or __schema[T]() with a concrete type, it writes a function for that type in Low text, binding the type and each type it mentions to names like __X0.
• It parses the text with the prelude's privileges, marks it trusted so it may read and build private fields, and checks it like a generic instance.
• Calls inside it to the same intrinsics derive the types they reach. Each derived function is cached by type.
• Bound names resolve in type positions, struct literals and variant selectors.
• A field reference T.field is checked as serde.Field[T, F] and lowered to the field's name and index as constants.

IR

• Generic templates never reach the IR.
  • The checker keeps an untouched copy of each generic function's syntax tree, clones it for each distinct set of type arguments and checks the clone with those types bound. The result is an ordinary function named like max[i64]; generic structs and enums become ordinary types named like Stack[str].
  • Instances are made on demand, checked from a work list, and limited to 1000 per template.
  • Symbols escape every character outside A-Za-z0-9_. as . and its hex code, so max[i64] is fn_max.5bi64.5d.
• SSA is built directly from the checked tree with Braun et al.'s algorithm: variables are read and written per block, phis are created lazily and sealed once a block's predecessors are known, and trivial phis are removed.
• Every value is one 64-bit word. A struct, enum or result of n words is n SSA values, so aggregates live in registers with no frame temporaries.
• Narrow integers are kept sign-extended (i32) or zero-extended (u32, u16, u8) in their word. Floats are kept as their bit pattern, f32 zero-extended. Arithmetic, conversion and comparison values carry their type.
• Frame slots and SSA variables:
  • A variable passed as mut to a function, captured by a closure, or of a C or vector type lives in a frame slot (or a heap box when captured) and is accessed with loads and stores. Every other variable is pure SSA.
  • push, __extend, __extend_by and delete take their mut operand in place. On an SSA variable they read and write its words (a slice's pointer, length and capacity) as SSA values, so a slice that is only pushed to, indexed and passed by value stays in registers.
  • Lengths and capacities these builtins compute are typed integers and never count as pointers.
• Integer division carries its zero and overflow checks inside the instruction. A constant divisor other than 0 and -1 needs neither check.
• A call to a C function becomes a ccall carrying its System V AMD64 plan. Each argument word is classified:
  • Integers, bool and pointers take rdi, rsi, rdx, rcx, r8 and r9.
  • Floats take xmm0 to xmm7.
  • An extern struct of at most 16 bytes is split into eightbytes that go to vector registers when every field in them is a float and to integer registers otherwise.
  • A larger struct, or one that does not fit in the registers still free, goes on the stack.
  • Variadic f32 arguments are widened to f64, and al holds the number of vector registers used.
  • A struct result of at most 16 bytes comes back in rax, rdx, xmm0 and xmm1 by class. A larger one is written to a buffer whose address goes in rdi.
  • Narrow integer, bool and f32 results are re-extended, and so are narrow places passed as mut once the call returns.
• Every frame keeps rsp 16-byte aligned at its calls, as C requires: frames are rounded to 16 bytes, and _start rounds the space it reserves for main's result.
• An extern struct value is its C bytes, one word per eightbyte. Its locals and parameters live in frame slots. A field is loaded or stored at its own width, and a struct literal or a field of a computed value goes through a temporary slot. A struct whose size is not a multiple of 8 is stored and loaded up to its last byte, with 4-, 2- and 1-byte accesses for the final partial word, so writing a nested field or an element of a view never touches the bytes after it.
• defer keeps a stack of deferred statements per block and lowers them again at every exit from that block.
• Drop and using:
  • A struct's drop becomes a function T..drop whose parameters are the struct's fields.
  • Every droppable type gets a generated T..release that calls T..drop and then the releases of its droppable fields.
  • A using binding adds a deferred if flag { T..release(x) } to its block. A move reads the value and clears the flag.
  • Moves are found by a flow-sensitive pass over each checked function body, which joins moved states across branches and runs a loop body a second time when an iteration moves a binding.
• == on a struct is the conjunction of its fields' comparisons. On an enum it compares tags, then branches on the tag to compare only that variant's fields, since the other payload words are undefined. An extern struct is stored in two frame slots and compared field by field at each field's offset and width, so padding is ignored.
• Function values and closures:
  • A Low function value is two words: a code address and an environment pointer. Code reached through a function value takes the environment as parameter word 0.
  • A closure is checked in place and becomes a function named after its parent, such as counter..c1. Its environment is a heap array, or 0 when it captures none. Environment words follow the order of the closure's captures.
  • A captured variable that is assigned after its binding, passed as mut, a C struct or a vector lives in a heap box allocated when it is bound. Every access, in its own function and in closures, loads and stores through the box, and the environment holds the box pointer in one word.
  • Any other captured variable stays an SSA variable of its function and the environment holds its words inline. The closure reads them into SSA variables on entry.
  • spawn binds its callee and arguments to such variables, so a spawn allocates the environment and nothing else.
  • A named function used as a value goes through an adapter, name..value, which drops the environment and calls it.
• A call through a function value moves the code address to r10 and calls it, passing the environment and the arguments like any call. A call through an extern fn value is a ccall with the same System V plan as a named C function.
• A C callback is a thunk, cb_name, emitted once per program for every function used as one. It saves the C argument registers, re-extends narrow integers and bool from them and from the C stack, calls the function with Low's convention on the green thread's stack and moves a float result to xmm0. On Windows it also saves and restores xmm6 to xmm15, which C treats as callee-saved.
• Every loop's back-edge block is marked. The code generator ends it with cmp byte ptr fs:[-8], 0 and a branch to an out-of-line call rt_poll.
• spawn f(a, b) becomes a block that binds the callee and the arguments to fresh locals and passes a closure calling f with them to rt_spawn.
• Sends, receives, close and channel loops call the prelude's instances for the element type. select calls the prelude's per-arm helpers and the runtime's rt_lock_all, rt_timer_add and rt_park.
• funcaddr gives a function's code address, its thunk, or an extern function's address through the GOT. Functions reached only through funcaddr are kept.
• In dev builds every C function gets a global trampoline, cfn_name: jmp name, and calls go through it, so a patch can link against the running binary.
• An embed becomes an address and a length. The data is emitted with .incbin under a label named by the file's content hash, so a changed file changes the code of every function that embeds it.
• Every value records whether it may hold a heap pointer.
  • Loads, parameters and call results take this from their type's word layout, where an enum word counts as a pointer if any variant puts one there.
  • Constants, typed arithmetic, string literals and frame addresses never hold pointers.
  • Untyped address arithmetic and anything unclassified count as pointers.
  • A phi holds a pointer if any input may.
• A str is two words and a slice three. An element access computes pointer + index * stride after a check instruction, which compares unsigned and jumps to the index panic. Packed elements load with movzx, movsxd or a 32-bit mov and store at their width.
• Maps:
  • A map [K: V] is a pointer to the prelude's generic __Map[K, V], whose entries slice holds __Entry[K, V] records of a live flag, the key and the value.
  • An open-addressing index of entry numbers with linear probing finds them. Deleted entries leave tombstones until the next rebuild, which compacts the entries in order and resizes the index to keep it at most three quarters full.
  • Integer, bool and pointer keys hash their word and str keys their bytes, both with the seeded multiply of the Maps section.
  • Every concrete map type instantiates __map_new, __map_get, __map_set, __map_delete, __map_len, __map_next, __map_key and __map_value for its key and value types. Map literals, lookups, assignments, delete, .len and loops become calls to them.
  • A compound assignment such as m[k] += v is a get, the operation on the value or zero, and a set.
• push and the formatting helpers call the runtime's grow routine and the prelude. The buffer behind an interpolation lives in a frame slot passed as a mut argument. A push that finds the slice full grows it to twice the capacity plus 4.
• Counted push loops:
  • A counted loop for i in a..b whose whole body is push(mut xs, v) becomes one reservation instead of a push per iteration. xs must be an SSA variable. v must be made only of literals, other variables, field reads, conversions, unary operators and non-trapping arithmetic or comparison on numbers.
  • The trip count is b - a as an unsigned word when a < b; the loop is skipped otherwise. A trip count above 2^40 bytes of elements calls the grow routine with an impossible size so the program stops with the out-of-memory panic.
  • The slice grows once to exactly its length plus the trip count, or not at all when the capacity already suffices, and its length becomes that sum. When xs is known to be empty the growth is unconditional.
  • A value that is zero in every word (a zero constant or a conversion of one) needs no stores in a freshly grown buffer, since the runtime returns zeroed memory. Otherwise, and in the no-growth case for a zero value, the elements are written with a counted loop.
  • A value that calls a function, can trap or reads the slice itself keeps the push loop.
• String building:
  • The prelude appends a whole string to a []u8 with __extend, which grows the buffer at most once and copies the bytes with the runtime's rt_memmove, so concatenation, interpolation, bytes and str(b) copy in bulk.
  • An integer is formatted by counting its digits against powers of ten, growing the buffer once with __extend_by and writing the digits from the end.
  • A concatenation chain and an interpolation, including interpolations and chains nested inside either, are flattened into one list of pieces. Adjacent literal pieces merge.
  • Every piece is evaluated first, in order. The result buffer is allocated once with the grow routine at the sum of the literal bytes, the lengths of the str pieces and the widest rendering of each number (20 for an integer, 25 for a float, 5 for a bool), which no rendering exceeds: the longest float is a negative value with exponent -6 and 17 significant digits, -0.0000012345678901234567.
  • A chain with no number pieces copies each piece with rt_memmove at a running offset. With number pieces the buffer lives in a frame slot, str pieces are copied at its length and numbers go through the prelude's __append_*.
  • String ordering calls the prelude's __str_cmp.
• Unreachable blocks are pruned after construction.

Frame-slot promotion:

• A frame slot is promoted to SSA values by the optimiser when its address is used only by word-sized loads and stores at word offsets inside it, and is never passed to a call, stored, compared with another address or reached through arithmetic with a variable. Its stores disappear, each load becomes the value stored on every path that reaches it, and phis join the paths. A slot read before any store reads an undefined value.
• A slot of a C or vector type, a select box, a lock array and a slot with more than one address value stay in memory.
• After inlining, a mut argument whose callee is inlined therefore lives in registers.
• Loads and stores carry the flags raw (a narrow or tail access of a C struct, or load and store) and trap (load, which is never discarded).

Optimiser:

• An optimiser runs over each function's SSA before phis are classified.
  • It keeps each value's type, float width and may-hold-a-pointer flag. A value that replaces another holds a pointer if either did.
  • A pass leaves the control-flow graph alone, so phi inputs and predecessor lists stay valid while it runs.
  • Between passes a branch on a constant becomes a jump to the taken successor, blocks left unreachable are pruned together with their predecessor entries and phi inputs, and trivial phis are removed. The pass runs again until no branch folds.
• The optimiser builds a dominator tree with the Cooper-Harvey-Kennedy algorithm and walks it from the entry, so each value sees only facts established by its dominators. The tree is generic over its node type and the code generator builds it over machine blocks. Predecessors that the entry cannot reach are ignored.
• Constant folding and simplification:
  • Copies are propagated, and integer operations, comparisons and not on constants are folded at their own type: i32 results are sign-extended and u8, u16 and u32 results zero-extended exactly as the generated code would leave them.
  • Division and remainder of constants by zero or by -1 are never folded, so the runtime panic and the overflow case stay in the instruction.
  • Adding, subtracting, or-ing, xor-ing or shifting by zero, multiplying by one, and multiplying or and-ing by zero reduce to an operand or a constant. Multiplication by a power of two becomes a shift.
• Division and remainder by a constant other than 0 and -1 become arithmetic on the 64-bit word, which is exact for narrow operands because they are kept extended.
  • A divisor of 1 is the operand or zero.
  • Unsigned powers of two are a shift or a mask.
  • Signed powers of two add (x >> 63) >>> (64 - k) before the arithmetic shift so the result truncates toward zero. A negative divisor negates the quotient.
  • Any other divisor multiplies by a magic number with the high-multiply instruction, signed (imul) or unsigned (mul), adds the operand back or halves and re-adds it when the magic number needs 65 bits, shifts, and for signed values adds the dividend's sign bit.
  • A remainder is the dividend minus the quotient times the divisor.
  • Signed division by the minimum 64-bit value stays an instruction with no checks.
• Common subexpressions are removed within the dominator tree for add, subtract, multiply, and, or, xor, shifts, conversions, function addresses and check. Comparisons, constants and loads are not merged: a load may observe a store, and a comparison that stays single-use can fuse into its branch.
• Bounds-check elimination:
  • A check [i, n] is removed when an identical check dominates it, or when i < n is known on the path and i is known to be non-negative.
  • A branch on a single-predecessor block's comparison of integers establishes i < n on the true edge of i < n, on the true edge of n > i, on the false edge of i >= n and on the false edge of n <= i.
  • i is non-negative when it is one of:
    • a non-negative constant;
    • an unsigned value;
    • a value a dominating signed comparison bounds below by a constant of at least 0 (or by -1 strictly);
    • a product of non-negative values that cannot overflow;
    • an induction variable that starts non-negative and only grows by a non-negative step without overflowing.
  • A variable stepping by one cannot overflow below a bound it is compared against. Any other step cannot overflow when the upper bounds of the variable and the step, added, stay within the type's range.
  • An upper bound is derived for a value from a constant, from the width of a narrow unsigned type, from a dominating signed comparison against a value that has an upper bound (strict comparisons subtract one), and for a sum or product of two non-negative values from their upper bounds when the result stays within the type's range. Only comparisons whose block dominates the use contribute.
  • A check [i, n] with a constant n is removed when i is non-negative and its upper bound is below n.
  • Branch facts also record i <= n on the false edge of i < n and the true edge of i >= n. These never remove a check and never prove a step cannot overflow.
  • A length that is a phi satisfies n >= L when every input satisfies it on its incoming edge, by being L, a constant at least L, a phi that does, or a value that a fact on the input's dominators bounds below by L. When i < L is known and n >= L holds, i non-negative removes the check, and otherwise the check compares against L instead of n, which keeps the same panics.
  • A slice filled by a counted push loop from 0 to L has length L when L > 0 and 0 otherwise, so indexing it by a counter below L needs no check.
• Dead-value removal:
  • Values nothing reads are removed, together with groups of values that only read each other, such as a phi and the add feeding it.
  • Only these operations are removed: constants, strings, embeds, undefined values, phis, add, subtract, multiply, and, or, xor, shifts, negation, not, complement, conversions, sqrt, floor, comparisons, function addresses, high multiplies, trailing-zero counts, loads, float division, and integer division or remainder by a constant other than 0 and -1.
  • Every other instruction stays, including calls, stores, checks, vector instructions, frame slots and divisions that can panic.
  • An unread load through a reference, a slice element or a frame slot is removed, since *T values are never null and an index is checked before its element is loaded. A load through a ptr[T] stays, so a null raw pointer traps even when the value is unused.
• Float arithmetic, negation, sqrt, floor and conversions between integers, f32 and f64 are folded on constants with the result the SSE instruction produces.
  • f32 operations round to single precision.
  • A float to integer conversion of a NaN or an out-of-range value gives the minimum 64-bit value.
  • Unsigned 64-bit conversions are left to the instruction.
  • A result that is NaN is never folded, so the sign and payload of a NaN come from the hardware.
• Loop-invariant code motion:
  • It works on natural loops found from back edges whose header dominates the latch, innermost first, in loops with a single outside predecessor that ends in a plain jump. That predecessor is the preheader.
  • Only values whose operands are defined outside the loop or already hoisted move: multiply, and, or, xor, shifts, negation, not, complement, conversions, sqrt, floor, function addresses, high multiplies, trailing-zero counts, float division, and add, subtract or shift that is not the constant-offset address arithmetic an addressing mode absorbs.
  • Loads, checks, calls, stores and integer division stay where they are, so a loop that runs zero times executes nothing it would not have executed.
  • A constant a hoisted value reads is copied into the preheader, because the code generator defines a constant that is not an immediate at its first use.
• Strength reduction replaces i * c by an add induction variable when i is a basic induction variable of a loop with a constant step, and replaces i * c + d derived from it the same way. The new variable starts at the start value multiplied by c in the preheader and steps by step * c on the back edge, with 64-bit wrapping arithmetic, so the result equals the product for every start, step and trip count, including overflow and negative steps.
• Devirtualisation and specialisation:
  • A call through a function value whose code address is a function address becomes a direct call that keeps the environment argument.
  • Before inlining, a function loses its function-address and environment parameter words when it is never called through a function value, is not main or a callback, and receives the same function address and the same constant environment from every direct call site. The parameters become the constants, the arguments leave every call site, and the function and its callers are simplified again. Calls from the function to itself that pass its own parameters unchanged count as agreeing. This runs for up to four rounds, so a function value handed down through several calls reaches the call that uses it.
  • A function of at most 150 values that calls a function-value parameter, or hands it to another call, is cloned per function address when its direct call sites pass two or more different function addresses for that parameter, or one function address with different environments.
    • Each clone takes the call sites of one function address and is named name..withN.target, where N is the first parameter word.
    • The clone has that word as the constant, drops the environment word too when every one of those call sites passes the same constant, and is simplified, so its call through the function value becomes direct.
    • Calls from the clone to the original that pass the parameters unchanged call the clone. The original stays for the remaining call sites.
    • A function gets at most 4 clones, and clones add at most 4000 values to the program.
  • The same conditions let an integer parameter that receives one and the same constant from every direct call site become that constant, but only when it pays: the function is simplified as a copy with the constants substituted and as a copy without, and the substitution is kept when the copy with it has fewer checks, blocks or values besides constants and parameters.
• Inlining:
  • Release builds inline direct calls bottom-up over the strongly connected components of the call graph.
  • A callee is eligible when it:
    • is not part of a cycle;
    • is not a callback;
    • has at most 48 values besides constants, parameters and results;
    • has at most one loop and an entry block no jump returns to;
    • uses no vector instruction.
  • Frame slots, ccalls and runtime calls are cloned with the body, and the callee's slots are appended to the caller's.
  • A callee of at most 6 values is free. Any other site is charged to the caller, which grows by at most 240 values, and to the program, whose budget is a fifth of its value count before inlining, at least 6000 and at most 40000. The last quarter of the program budget is kept for call sites inside loops.
  • A caller inlines up to four rounds. Each round takes its sites inside the deepest loops first, then the smaller callees, then the earlier values; splits the call's block; clones the callee's blocks with parameters bound to the arguments; forwards a single return's values or joins several returns in phis; and simplifies the caller. Calls that an inlined body or a devirtualised call exposes are inlined in the next round.
  • Loop back edges keep their marks, and checks and divisions keep their panics.
  • Dev builds do not inline, specialise or hoist literals, so hot reload patches see every function and every literal's data.
• Literal hoisting:
  • A release build hoists a non-empty slice or map literal whose elements are all constants: integer, float, bool and str literals, negated numbers, none, enum variants without payload and non-C struct literals of constants.
  • The checker marks a literal read-only when it is the operand of an index, a for or .len, or is bound by a let whose every use is such an operand. An assignment, a mut argument or any other use anywhere in the target path of an element or field makes the binding non-read-only.
  • Each hoisted literal owns a word of rt_cells. The code loads the word, and when it is 0 builds the literal as usual and stores the pointer, so the first evaluation builds it and a racing thread at most builds a second equal copy.
  • A read-only literal uses the cell's pointer, as a slice of its length for a slice.
  • Any other use copies it: a slice with one grow call that moves the template's bytes, a map with the prelude's __map_clone, which copies the entry array and the index. A literal below the copy thresholds of the Slices and Maps sections is built each time in that case.
• Block merging and jump threading:
  • A block that ends in a jump to a block with no other predecessor and no phis is merged into it.
  • A block that holds a phi and at most one more value and branches on that phi, or on its negation, is skipped by a predecessor that passes a constant: the predecessor jumps to the successor the constant selects.

Code generation

• Phis and layout:
  • Critical edges are split and each phi becomes a parallel copy at the end of every predecessor, emitted with cycles broken through r11.
  • An argument whose live range ends at that copy prefers the phi's register, and the phi prefers the registers of its arguments, so a loop-carried value usually needs no move.
• Block layout:
  • Blocks are laid out in reverse post-order, visiting the false successor first so the true successor follows its branch.
  • Inside a loop, the true arm of an integer == comparison against a constant is treated as rare when it holds a loop, a call, a return or an unreachable end. The blocks between the branch and its false successor move after the last block of the innermost loop, so the common path falls through to the back edge. A light arm stays in line, because the extra taken branches would cost more than the arm when it is the common path.
  • A jump or branch to an empty block goes to that block's final target, a jump to the next block in layout is omitted, and a conditional branch whose true target is next is inverted. A block holding phi copies is never empty.
• Loop latches:
  • A loop's latch whose header holds only a fused compare and branch ends with that compare and a conditional branch to the body, after the poll check when the block has one, instead of a jump to the header.
  • A header with up to 12 instructions, calls included, is copied after the latch instead, so the back edge tests the condition without a jump and the first test still runs at the original header.
• A comparison used only by the branch that ends its block is fused into a cmp and conditional jump.
• Addressing:
  • A load or store addresses [base + index*scale + disp] at every width.
  • A 64-bit add read only by addresses folds into them, as does a shl or mul by 1, 2, 4 or 8 feeding the index. A constant operand becomes the displacement. Other strides keep one multiply and fold the add and the displacement.
  • The folded base and index are uses of the load or store, so they stay live to it, across blocks where the definition dominates. A base or index in a spilled slot is reloaded into r10 or r11 immediately before the instruction. An instruction with a memory operand takes any other temporary from rax.
• Load folding:
  • A load is folded into its only consumer as a memory operand when both are in one block and no store, call, runtime call, vector operation or poll lies between them.
  • Integer arithmetic, logic, bounds checks and compares fold a 64-bit load.
  • A narrow load is never folded into a 64-bit operation. It folds only into a compare of the same type against a constant the type holds, as cmp of a sized memory operand with an immediate.
  • Scalar SSE arithmetic and compares fold a load of the operation's own width, f64 or f32.
• A float multiplication by 2.0 is an addition of the operand to itself, which is exact for every input including NaN, infinities and signed zeros. Other float constants keep their multiply.
• Boolean branches:
  • A branch on not x swaps its successors instead of computing the complement, which is exact for float comparisons as well.
  • A comparison read only by not is emitted with the inverted condition.
  • A block that only branches on a boolean phi is threaded: a predecessor that passes a constant jumps to the successor the constant selects, a predecessor that passes the condition of its own branch jumps to the matching successor, and any other predecessor branches on the value it passes.
  • Evaluation order and short-circuiting of && and || operands are unchanged.
  • A branch on the result of a not or a comparison defined in its own block uses that operand directly, so the not and the materialised boolean are not emitted.
• Spilling:
  • A value's spill weight is the sum over its uses and definitions of 8 raised to the block's loop depth, capped at depth 4.
  • When no register is free the allocator spills the conflicting value with the lowest spill cost, breaking ties by the later end of its live range.
  • The allocator runs three times per function, with the spill cost set to the weight, to the weight divided by the length of the live range and to the weight divided by its square root. It keeps the assignment whose spilled values have the lowest total weight plus 2 for each callee-saved register it uses. The weight alone wins ties.
• Constants that fit in 32 bits are used as immediates and take no register. An undefined value used as an operand or stored to memory is the immediate 0, so a variant's unused payload words never hold stale pointers.
• Narrow integer results:
  • Results of narrow integer operations that can leave the type's range are re-extended with movsxd, movzx or a 32-bit mov.
  • A u32 add, subtract, multiply, negation, complement and constant left shift uses the 32-bit form of the instruction, whose write zero-extends and needs no re-extension.
  • A conversion to an integer type that holds every value of the source type is a register copy. A right shift of an extended value needs no re-extension.
  • Unsigned comparisons and branches use the unsigned condition codes, and unsigned division uses div.
  • A division by a constant other than 0 and -1 that reaches the code generator, which is signed division by the minimum 64-bit value, emits no zero or -1 test.
• Float registers:
  • Float values live in xmm2 to xmm15; xmm0 and xmm1 are scratch.
  • A value is a float register value when it is produced or consumed by float arithmetic, a float comparison, a conversion, sqrt or floor. A float that only passes through moves, calls, stores and phis stays in a general register.
  • Scalar SSE instructions (addsd, subss, sqrtsd, roundsd, cvttsd2si and so on) work on registers directly, with a spilled operand as a memory operand.
  • An f32 in an xmm register keeps bits 32 to 63 zero, so movq moves it to a general register or the frame unchanged. movq is used only where a float meets an integer word: call, return and parameter words, stores and loads, bit casts, and C argument registers.
• A float constant that does not fit in 32 bits is a .rodata quadword, shared by value and used as a memory operand. Zero is xorps.
• A float comparison used only by the branch that ends its block is fused into ucomisd or ucomiss and a conditional jump. < and <= swap their operands and use ja and jae, so an unordered result is false. == and != also test the parity flag.
• Shifts and rotates:
  • Variable shift counts go through cl, with rcx saved in r11 around the shift. Counts above 63 take a separate path.
  • The pair (x << k) | (x >> (w - k)) is one rol when it meets all of these: both shifts act on one u32 or u64 value x of width w, k is a constant between 1 and w - 1, the shifts are read only by the or, and they appear in either order. No other combination is a rotate.
• exp is a runtime routine: range reduction by ln 2 split into high and low parts, a degree-13 polynomial, and scaling by two powers of two so subnormal results stay in range.
• A comparison of a str against a literal of up to 16 bytes is inline: a length compare, then one compare of the bytes at the pointer against an immediate of 1, 2, 4 or 8 bytes, or two when the length is none of these, ending in sete. The literal's address and length are never materialised and the comparison is not a call. Any other str equality moves the arguments into the argument registers, returns 0 without calling rt_str_eq when the lengths differ and calls it otherwise.
• Register allocation:
  • Registers are assigned by linear scan over each value's live segments, so a value does not occupy a register where it is dead and a register's holes hold other values.
  • rcx, rsi, rdi, r8 and r9 hold values that are not live across a call. rbx and r12 to r15 hold values that are, and are saved in the function's frame when used. Values that find no register are spilled to the frame. rax, rdx, r10 and r11 are scratch.
  • A growth call, rt_grow_scan or rt_grow_noscan according to whether the new buffer may hold pointers, takes the old pointer, the used bytes and the new size in rax, rdx and r10 and returns the new pointer in rax. It preserves every other general register, so a value that cannot hold a pointer stays in rcx, rsi, rdi, r8 or r9 across it. A value that may hold a pointer is treated as live across a call: the routine saves the preserved registers below the scanned part of the stack, where the collector does not read them. The routine records the caller's return address and frame pointer like any runtime entry, and the old buffer stays reachable through the words it keeps in the scanned region.
  • Float register values are allocated from xmm2 to xmm15 by the same scan. No xmm register survives a call, so a float live across a call is spilled, and spilled floats are not in the stack maps. A function that uses vectors gets no float registers, so its float operations go through xmm0 and xmm1.
• Register preferences:
  • A value used as a call argument prefers that argument's register, a parameter prefers the register it arrives in, and a copy and its source prefer each other's registers.
  • The result of a two-operand add, subtract, multiply, and, or, xor, shift or float divide prefers the register of its first operand, and for a commutative operation of its second, so the instruction can work in place. Where it cannot share, it avoids the register of the operand it still has to read.
• Wrapped saves:
  • A function whose calls all lie below one block, the nearest common dominator of its call blocks, saves the callee-saved registers it uses at the head of that block instead of in the prologue and restores them only on returns reached through it.
  • The paths that skip the block use no callee-saved register, have no calls or polls and are at most 8 instructions, copies included, where they rejoin the shared tail, which is duplicated for them.
  • A value defined on a skipping path and live into the block is copied at the end of that path into a fresh value, and the block's head copies it into the original. The original lives only below the block, so a loop counter initialised on the light path stays in a caller-saved register.
  • A value defined in the region below the block and live into it that crosses a call is given a fresh value there.
  • Save slots keep their frame positions, and the stack maps name every save slot at every safepoint.
  • A vector function is not wrapped. The block and the tail are found with the same dominator tree that the optimiser uses.
• Frame modes: a function has one of three.
  • A full frame is built in the prologue.
  • A function with a wrapped region builds its frame, push rbp, mov rbp, rsp and sub rsp, at the head of the block that holds the saves, and the paths that skip the block return with a bare ret. This needs every skipping block to be free of anything that reads or writes the frame: no spilled value, frame address, slot access, parameter beyond the sixth, poll, panic or return wider than two words.
  • A function with no calls, polls, panicking checks, frame slots or spilled values and no callee-saved register has no frame at all and contains no safepoint.
• A frame slot's address is not held in a register: a load or store through it addresses [rbp+offset] directly and any other use computes the address with lea into a temporary where it is needed. A spilled value copied from another value whose live range it does not overlap shares that value's slot, so splitting a value at the wrapped block adds no slot-to-slot copy.
• A call result in rax or rdx that is read once, by the instruction directly after the call, stays there and is not moved. The immediate consumers are integer arithmetic and logic, a store of the value, call arguments, the return, a branch condition and a phi copy. A result word nothing reads is not moved at all.
• A block that only returns, with at most four predecessors that end in a jump, is copied into them, so each returns without a jump.
• A return restores with leave, or pop rbp in a frame of size 0. The last instruction of a returning block writes rax directly when it defines the first result and nothing else reads it. Up to two results are placed with a parallel move.
• Instruction selection:
  • An add or subtract of a constant or a register into a different register is lea.
  • A multiply by a constant is the three-operand imul.
  • A commutative operation takes whichever operand already sits in the destination.
  • A compare with 0 is test. A compare of a memory operand with an immediate reads the memory directly.
  • The constant 0 is xor r32, r32 where the flags are dead.
  • A 32-bit add or subtract into a different register is a lea with a 32-bit destination and 64-bit address registers.
  • A move that undoes the move before it is dropped.
• The poll check is cmp byte ptr fs:[-8], 0 and jne to an out-of-line stub that calls rt_poll and jumps back to a label placed directly after the jne.
• Alignment:
  • Every function and every C callback thunk starts on a 32-byte boundary.
  • low build and low run on Linux assemble with -mbranches-within-32B-boundaries, which pads so that no jump, and no fused compare and jump, crosses or ends at a 32-byte boundary. Hot patches are assembled without it.
  • A loop that is a single back-edge block, ends in its own conditional jump and has at most four instructions and phi copies besides the poll, if any, and the compare starts on a 32-byte boundary, so the whole loop sits in one window and needs no padding inside.

Calling convention:

• The first six argument words go in rdi, rsi, rdx, rcx, r8 and r9. The rest go in an outgoing area at the bottom of the caller's frame, so rsp does not move inside a function.
• Up to two result words return in rax and rdx. Wider results are written to the outgoing area after the stack arguments.
• Argument and parameter registers, general and xmm, are filled with a parallel move that breaks cycles through r11.

Vectors

• A vector local lives in a frame slot, like a variable passed as mut, and a vector expression evaluates to the address of a slot holding its result.
• A vector operation is one IR instruction, vector, taking the destination and operand addresses, or vectorscalar for mask.
• At calls, returns, fields and other aggregates a vector is its words, loaded from or stored to a slot. A let of a computed vector adopts the slot of the result instead of copying it.
• A function containing vector instructions is compiled twice. Its symbol is a stub that jumps to the AVX2 version when the CPU and the operating system support AVX2, and to the SSE2 version otherwise. rt_cpu_init checks cpuid and xgetbv at startup.
• LOW_SIMD=sse2 in the environment selects the SSE2 versions, which the fixture runner uses to test both.
• The AVX2 version uses three-operand VEX instructions on ymm registers. A slot written once and read only by vector instructions later in the same block, with no call in between, is kept in ymm8 to ymm15 instead of memory. vzeroupper runs before every call and return of such a function.
• The SSE2 version splits 256-bit operations into two 128-bit halves and keeps every vector in memory.
• An operation without an instruction in the version being generated runs as a scalar loop over the lanes in a scratch area of the frame. These operations are pshufb on SSE2, 64-bit multiplication, minimum, maximum and comparison where unavailable, byte shifts and arithmetic 64-bit shifts. Unsigned comparisons flip the sign bit of both operands and compare signed.