Just-In-Time Compilation: Techniques, Tiers, and Modern JITs

Compilers Compilers 7 min read 1464 words Beginner ExcellentWiki Editorial Team

Just-in-time (JIT) compilation is an execution strategy that compiles code to native machine code at runtime, rather than ahead of time. JIT systems sit at the intersection of interpretation (which offers fast startup and rich debugging) and ahead-of-time (AOT) compilation (which offers peak performance). By profiling the running program and compiling only the hot code paths, JIT compilers achieve near-native performance while preserving the portability and interactive qualities of interpreted languages. This article covers the fundamental JIT architectures — method-based JIT, tracing JIT, and tiered compilation — and surveys the JIT compilers in Java HotSpot, V8, and LuaJIT.

How JIT Compilation Works

A JIT compiler operates as part of a language runtime. The typical JIT pipeline has four stages:

Execution — the program starts executing, typically via interpretation or bytecode VM
Profiling — the runtime monitors execution, collecting metrics such as method invocation counts, loop iteration counts, and type feedback
Compilation decision — when profiling data crosses a threshold, the runtime queues the hot code for compilation
Compilation and replacement — a background compilation thread translates the code to native machine code; once ready, the runtime patches the call site to use the compiled version

The compiled code includes the profiling infrastructure itself. For example, the compiled version may include instrumentation to detect when an assumption (like the dynamic type of a variable) becomes invalid. If the assumption breaks, the code can be deoptimized — reverting to interpretation with the current program state preserved.

Method-Based JIT

Method-based JIT compiles entire methods or functions. The triggering event is typically a method invocation count exceeding a threshold. This approach is used by Java HotSpot, .NET RyuJIT, and most traditional JIT compilers.

Compilation Triggers

A method is queued for compilation when its invocation counter exceeds a threshold (default 10,000 in HotSpot for C1 compilation). The counter is periodically decayed using a background thread so that rarely-called methods are eventually considered cold again. This prevents methods that are called many times only during startup from triggering compilation.

On-Stack Replacement (OSR)

When a long-running loop is identified as hot, waiting for the method to return before compiling is impractical — the method may never return during the current invocation. OSR transitions from interpreted execution of a loop to compiled execution mid-method.

OSR works by:

Mapping the interpreter state (local variables, operand stack) to the compiled code’s register allocation
Jumping from the interpreter loop into the compiled loop body
Executing the compiled version until the loop exits, then returning to the compiled method’s normal epilogue

OSR is critical for compute-intensive workloads that spend most of their time in loops. Without OSR, long-running loops would never benefit from optimization.

Tracing JIT

Tracing JIT (or trace-based JIT) compiles hot paths through loops rather than whole methods. The compiler records a linear sequence of instructions executed during a loop iteration — the trace — and compiles it as a single unit. Traces can span multiple methods, capturing inlined call chains.

LuaJIT is the most famous tracing JIT. It profiles running Lua code and identifies frequently executed loop traces. Each trace is compiled using a lightweight compiler that generates efficient machine code.

Tracing JIT advantages:

Automatic inlining — traces naturally cross function boundaries without explicit inlining analysis
Type specialization — the trace records the actual types observed at runtime, enabling monomorphic compiled code even for dynamic languages
Lower compilation overhead — traces are shorter than full methods, compiling faster

Disadvantages:

Trace selection overhead — deciding which paths to trace and when to stop tracing is complex
Trace linking — traces must be linked together, with guard instructions checking that the runtime matches the traced assumptions
Side exit handling — when a guard fails, execution must fall back to the interpreter or a different trace

Tiered Compilation

Modern JIT systems use multiple compilation tiers to balance startup speed and peak performance:

Tier 0: Interpreter

The program starts in the interpreter. No compilation cost, but slow execution.

Tier 1: Baseline Compiler (Trivial JIT)

A very simple compiler that produces straightforward machine code with minimal optimization. In V8, Sparkplug produces code that is 2×–3× faster than interpretation but still 5×–10× slower than optimized code. The baseline compiler’s key advantage is that it avoids the profiling overhead of the interpreter while still being fast enough for most code.

Tier 2: Optimizing Compiler

An aggressive compiler that applies inlining, loop optimization, vectorization, and register allocation. In HotSpot, C2 is the optimizing tier. In V8, TurboFan is the optimizing tier. These compilers are slower (compilation takes 10×–100× longer than baseline) but produce much faster code.

Profile-Guided Optimization

Profiling data flows between tiers. During interpretation, the runtime collects type feedback — which virtual methods are called, what types variables hold, which branches are taken. When compiled at Tier 1, this data is used to guide inlining and devirtualization. When the code reaches Tier 2, the optimizer has rich profile data to drive optimization decisions.

Case Study: Java HotSpot

HotSpot has two JIT compilers, selectable via flags:

Client Compiler (C1) — designed for fast compilation and moderate optimization. Used for startup. Produces code comparable to GCC -O1.
Server Compiler (C2) — designed for peak performance. Uses aggressive optimizations: inlining, loop unrolling, vectorization, lock coarsening, escape analysis. Produces code comparable to GCC -O2.

By default, HotSpot uses tiered compilation: methods first run in the interpreter, then C1 compiles them, then C2 recompiles the hottest methods. The thresholds are adjusted based on platform (C2 compilation is slower on ARM than x86, so thresholds are higher).

C2 implements the following techniques:

Sea-of-nodes IR — a graph representation where data and control dependencies are explicit
Ideal Graph Visualization (IGV) — a tool for visualizing C2’s optimizer state
Iterative compilation — C2 can recompile with different settings based on performance feedback

Case Study: V8 (TurboFan)

V8’s modern compilation pipeline:

Ignition (interpreter) — bytecode interpreter with low overhead
Sparkplug (baseline) — fast compiler that emits straightforward x86/ARM code
TurboFan (optimizing) — SSA-based optimizing JIT with advanced techniques

TurboFan uses a sea-of-nodes IR similar to HotSpot’s C2. It performs:

Inlining — aggressive inlining based on type feedback
Object allocation sinking — avoiding allocation of objects that do not escape the method
Number typing — tracking integer vs. floating-point types for optimization
Constant propagation — leveraging type feedback to constant-fold polymorphic operations

JIT Compilation Challenges

Compilation Overhead

Compilation consumes CPU cycles and memory. A balance must be struck: compile too eagerly (long startup), too lazily (poor peak performance), or compile too many methods (memory bloat). Tiered compilation addresses this by using a fast baseline compiler for most methods and saving the optimizing compiler for the hottest fraction.

Code Cache Management

JIT compilers store compiled code in a code cache. When the cache is full, older or unused compiled code is evicted and execution falls back to interpretation or baseline code. Managing cache size and eviction policy is critical for long-running applications.

Deoptimization

When assumptions made during compilation (e.g., “this variable is always an integer”) break at runtime, the JIT must deoptimize. Execution transfers from compiled code back to the interpreter with the same program state. This requires detailed mapping from compiled code registers to interpreter state — a significant engineering challenge.

Security

JIT compilation introduces writable-and-executable (W+X) memory pages, which are a security concern. Modern JITs mitigate this by using W^X memory protection (write XOR execute) and moving the compiled code from writeable to executable pages after generation.

FAQ

What is the difference between method-based and tracing JIT? Method-based JIT compiles entire methods when they are hot. Tracing JIT compiles hot loop paths (traces) that may cross method boundaries. Tracing JIT is better for dynamic languages with unstable types; method-based JIT is better for statically typed languages.

How does JIT handle recursive functions? Recursive methods are compiled like any other method. The hotness threshold counts invocations; deep recursion may trigger OSR to compile the looping path. In tracing JIT, recursive calls can be unrolled in the trace.

Do JIT compilers support garbage collection? Yes. JIT compilers generate GC maps — tables that describe which registers and stack slots contain object references at each safepoint. When GC triggers, the compiled code checks the safepoint at the current PC, reads the GC map, and reports live references to the collector.

Can I write my own JIT compiler? Yes. LLVM provides a JIT framework (OrcJIT) that handles code generation, relocation, and memory management. Cranelift is a lightweight code generator designed for JIT use. You write the profile and compilation decision logic, and these frameworks handle the machine code generation.