The Solana eBPF Virtual Machine

Perhaps you’ve been seeing all of the recent hype about the Solana Virtual Machine (SVM) and all of the exciting projects being built on it. Maybe you even read our previous blog post about the SVM API.

In either case, you’re probably looking for more information about the virtual machines that power Solana. This guide will take you through the Agave validator’s use of the rBPF virtual machine, explaining what it is, how it works, and how the validator uses it to execute Solana programs.

The rBPF VM

The rBPF Virtual Machine is a Rust implementation of an Extended Berkeley Packet Filter (eBPF) virtual machine created by Quentin Monnet. In Solana’s early days, the rBPF project was forked under Solana Labs and modified slightly to support custom Solana-specific functionality, which will be covered in later sections. Today, the rBPF fork is maintained by Anza engineers.

As mentioned in the repository’s README file, the rBPF VM is designed to run in user-space rather than on the kernel. This makes rBPF an ideal candidate for the virtual machine Solana validators use to execute programs, since the validator’s runtime operates in the node’s user-space.

The popular term “SVM” is actually a bit of a misnomer. Across the ecosystem, when Solana developers refer the the Solana Virtual Machine (SVM), they are often referring to the entire transaction processing pipeline within the Solana runtime, or the execution layer. However, the actual virtual machine responsible for executing Solana programs is an eBPF VM with constraints imposed by the Solana Virtual Machine Instruction Set Architecture (SVM ISA).

Solana rBPF is the Rust virtual machine that implements the SVM ISA and is used by the Agave validator. Firedancer, for example, has a completely reimplemented version of a virtual machine that adheres to the SVM ISA.

The virtual machine itself also has access to a set of system calls (”sycalls”) defined by the Solana protocol, which will be covered in later sections. These syscalls are also part of the constraints imposed on the lower-level Solana virtual machine environment.

Berkeley Packet Filter

Solana programs are compiled to Berkeley Packet Filter (BPF) format. BPF was originally designed for Berkeley Software Distribution (BSD) Unix systems to filter network packets in the operating system’s kernel. The format leverages qualifiers, similar to discriminators, allowing for efficient filtering of packets without the need to copy data.

BPF programs use these qualifiers to define conditions under which packets are captured or dropped. They define a set of instructions (op codes) that operate on registers, memory, or packet data.

BPF eventually evolved into Extended Berkeley Packet Filter (eBPF) format, which is what Solana’s LLVM compiles to today. eBPF allows programs to configure a restricted instruction set and constraints specifically designed for safe kernel execution. This is useful for Solana programs, since it prevents the validator from crashing and creates a consistent environment for all programs.

Solana programs are typically written in Rust, which are then compiled to eBPF by the Solana platform tools. However, programs can also be written in Zig, C or Assembly. The platform tools ensure the program is compiled to the proper eBPF format, which respects the eBPF restrictions imposed by the Solana VM, described in the next section.

Solana rBPF ISA

As mentioned in the previous section, eBPF allows platforms — such as virtual machines — to impose a tight instruction set and constraints for eBPF programs. The Instruction Set Architecture (ISA) of the Solana rBPF repository defines exactly that. All Solana virtual machines must adhere to this ISA in order to be compliant with the Solana protocol.

The first section covers the registries supported by the rBPF VM. They are 64 bits wide, meaning they can hold 64-bit integers or addresses.

|  name | feature set | kind            | Solana ABI 
|-------:|:-----------|:----------------|:----------
|  `r0` | all         | GPR             | Return value
|  `r1` | all         | GPR             | Argument 0 
|  `r2` | all         | GPR             | Argument 1 
|  `r3` | all         | GPR             | Argument 2 
|  `r4` | all         | GPR             | Argument 3 
|  `r5` | all         | GPR             | Argument 4 <br/>

Registers are small memory locations that store data currently being operated on. The ISA defines ten General-Purpose Registers (GPRs).

• r0 holds the function’s return data.

• r1 through r5 store function arguments, and r5 can actually store “spillover” data, which is represented by a pointer to some stack data.

• r6 through r9 are call-preserved registers, which means that their values are preserved across function calls.

Aside from the GPRs, there is a frame pointer (r10) that references the current stack frame in memory, a stack pointer (r11) that keeps track of where the top of the stack is, and the program counter (pc) that holds the address of the current instruction being executed.

The next section covers the instruction layout. As mentioned in the documentation, bytecode is encoded in 64-bit slots, and instructions can occupy one or two slots, indicated by the op code of the first slot.

The instruction layout covers exactly how instructions are encoded in the VM and what each bit means.

• Instruction class: Identifies the type of instruction (arithmetic, memory access, etc.).

• Op code: The specific operation itself.

• Destination register: The register where the result of the operation will be stored.

• Source register: The register where the operation input data is sourced.

• Offset: Used for memory access or jump offsets.

• Immediate: Constant values.

The next section covers all of the rBPF VM’s supported op codes. The table provided in the documentation gives a detailed breakdown of every single op code supported by the VM, where the row labels are the upper four bits and the column labels are the lower four bits of the op code.

After the op codes, the ISA defines an “Instructions by Class” section, which defines the particulars about specific operations and their constraints. For example, it covers both 32-bit and 64-bit arithmetic, multiplication, division, remainders, memory access, and control flow. For each of the sections, specific information is provided about expected panics. These are the eBPF constraints mentioned earlier, and they are explicitly defined in the SVM ISA.

Note that the presence of 32-bit and 64-bit arithmetic definitions in the ISA does not mean the virtual machine can operate on both 32-bit and 64-bit architectures. These sections specifically define arithmetic operations, which may use 32 bits for memory optimization or 64 bits where necessary.

The panics defined in the ISA are fairly straightforward. For division, it defines division by zero and negative overflow as panic cases. For memory access, it references out of bounds or access violations (ie. writing to read-only sections). Finally, for control flow, jumps out of bounds, references to unregistered functions, and stack overflows are mentioned.

Finally, the verification section defines the rules for verifying an eBPF program, which pertains to static analysis of the program binary. All together, this makes up the entire eBPF VM ISA definition for the Solana Virtual Machine.

Solana VM Builtin Programs (Loaders)

The functions within a compiled eBPF program are read into what’s called a function registry when the binary is loaded by the eBPF VM. However, the rBPF VM supports something called “builtin programs”, which also have their own function registries.

You may be familiar with this term from the Solana runtime, which uses builtin (sometimes called “native”) programs. The terminology is designed to be the same, since the two share some of the same behavior. Solana native programs provide to the runtime what rBPF virtual machine builtin programs provide to executing BPF programs: access to functions built into the execution environment.

When the Solana runtime encounters an instruction for a builtin program — such as a System transfer — it does not load and execute some compiled BPF program, but instead simply calls a function that is built into the runtime to perform the transfer. This builtin function is the System program, and its code actually ships with the Solana runtime and is an integral part of its environment. These programs do not exist on-chain, but instead have on-chain placeholders at their corresponding addresses.

Similarly, in the rBPF virtual machine environment, the executing program can actually have instructions defined that invoke builtin functions. These functions — like the runtime’s builtin programs — are built into the VM.

// <https://github.com/solana-labs/rbpf/blob/9d1a9a0c394e65a322be2826144b64f00fbce1a4/src/vm.rs#L365>
impl<'a, C: ContextObject> EbpfVm<'a, C> {
	 /* ... */
	 pub fn execute_program(
   &mut self,
   executable: &Executable<C>,
   interpreted: bool,
 ) -> (u64, ProgramResult)
}

// <https://github.com/solana-labs/rbpf/blob/7364447cba1319e8b63d54b521776439181853a7/src/elf.rs#L249>
pub struct Executable<C: ContextObject> {
 /* ... */
 function_registry: FunctionRegistry<usize>,
 loader: Arc<BuiltinProgram<C>>,
}

// <https://github.com/solana-labs/rbpf/blob/7364447cba1319e8b63d54b521776439181853a7/src/program.rs#L214>
pub struct BuiltinProgram<C: ContextObject> {
    /* ... */
    functions: FunctionRegistry<BuiltinFunction<C>>,
}

The VM-level builtin program — which gives the executable access to the set of builtin functions — is called a loader. There are many types of VM builtin functions, but the primary functions provided by a VM loader are system calls (or “syscalls”).

Solana syscalls allow executing eBPF programs to call functions outside their compiled bytecode, built into the virtual machine, to do many things, such as:

• Print log messages

• Invoke other Solana programs (CPI)

• Perform cryptographic arithmetic operations

Similar to the virtual machine’s ISA, all Solana syscalls are part of the Solana protocol and have well-defined interfaces. Changes to these interfaces, as well as the introduction of new syscalls, are governed by the Solana Improvement Documents (SIMD) process.

The Agave validator implements all Solana syscalls on the BPF Loader, which is the loader mechanism provided to the VM. There have been several versions of the BPF Loader, including the currently in-development Loader v4.

Solana BPF Loaders are also runtime builtin programs — similar to the System program — that are invoked by the runtime. In fact, when an on-chain eBPF program is invoked by an instruction, the runtime will actually invoke the BPF Loader program that owns it in order to execute it. More on this later.

Program Execution

The rBPF VM library can execute eBPF programs in two ways: via interpreter or using JIT compilation to x86_64 machine code.

The interpreter simply walks each instruction one-by-one, interpreting each one and executing it at runtime. This can add a slight bit of runtime overhead, since the interpreter must determine at runtime what each instruction does before executing it, but the benefit is a massive reduction in load time.

Conversely, Just-in-Time (JIT) compilation of the program to x86_64 machine code makes the execution of the program much faster, but at the cost of a much longer load time as a result of the initial compilation.

Agave currently uses JIT compilation for several reasons. First and foremost, syscalls are currently dynamically registered, which means they cannot be processed by the verification step during static analysis, and are instead marked as “unknown” external function calls. It is during the JIT compilation step that syscall function references in the program binary are linked to their registered builtin functions.

Tour de Agave

With all of the general context about how the rBPF virtual machine works, it’s time to walk through the Agave validator and see exactly how the rBPF VM is used to execute Solana programs when users send transactions containing instructions for on-chain programs.

Program Deployment

Before getting started on the tour through Agave’s instruction processing pipeline, it’s important to understand what happens when a developer deploys a Solana program.

Program deployments are done by invoking the BPF Loader program, which, as mentioned previously, is a builtin program. Its status as a builtin allows the program to access additional computational resources, which enables a few key steps that are necessary for verifying a program that is requesting to be deployed.

When you run the CLI command solana program deploy for example, the CLI will send a set of transactions that will first allocate a buffer account and write your program’s ELF file into it. ELFs are quite large, so this occurs over several transactions in which the ELF is chunked. After the buffer contains the entire ELF, the program can then be “deployed” (the final CLI instruction).

When the BPF Loader program’s “deploy” instruction is invoked, it will attempt to verify the ELF stored in the provided buffer account and, if successful, move the ELF into a program account and mark it as executable. Only after this successful verification can a program be invoked by a Solana transaction instruction.

Verification of program ELFs is well-contained in the BPF Loader’s deploy_program! macro. The steps are as follows:

1. Load the program as an eBPF executable with a “strict” runtime environment. The purpose of a “strict” environment in this step is to prevent deployment of programs with deprecated ELF headers or syscalls. This uses the load method from rBPF, which validates the ELF file structure and performs relocation of instructions.

2. Verify the loaded executor bytes against the ISA. This uses the verify method from rBPF.

3. Reload the program with the current runtime environment.

ELF verification is a very important step in program deployment, since it directly relates to the expectations set by the virtual machine’s ISA. The BPF Loader program will actually use the eBPF verification tooling provided by the rBPF library to verify the program binary, ensuring it does not violate any constraints.

This means that only valid Solana eBPF program binaries can become active Solana programs. From a performance perspective, this allows the runtime to quickly discard invalid Solana program binaries by simply checking whether it is an executable program, since it can only become executable after passing verification on deployment.

Transaction Pipeline

As mentioned previously, the runtime will only encounter an executable BPF program once it has been successfully deployed and thus verified. With this assumption in mind, the lifecycle of a transaction instruction for a valid on-chain BPF program can be traced through the transaction pipeline within Agave.

Transactions are scheduled for processing by the scheduler and eventually processed via a Bank instance. A Bank processes transactions with the SVM API’s transaction batch processor, specifically the load_and_execute_sanitized_transactions method.

Given a batch of transactions, the processor will first asses any necessary accounts for the ability to pay transaction fees. Then, it will filter any executable program accounts, to be loaded by the program JIT cache.

The program JIT cache is merely a cache of programs that have already undergone JIT compilation to x86_64 machine code, as detailed earlier, and are ready to be executed. The biggest responsibility of the program cache is actually to load the correct version of a program across forks, considering potential conflicting versions as a result of a deployment or closure.

Shortly after, all of the necessary accounts are loaded to process the transactions. Then, the transaction is processed, and if it’s a valid transaction, executed. There are a lot of small adjacent steps involved in executing a transaction, but one can focus mainly on the path for an instruction’s execution against a loaded BPF program for this exercise.

The first thing required is an InvokeContext instance. This is the Agave-specific context object, which contains many Solana protocol-specific context configurations required by the rBPF VM. In fact, the rBPF VM itself is generic over some context object.

/// Main pipeline from runtime to program execution.
pub struct InvokeContext<'a> {
    /// Information about the currently executing transaction.
    pub transaction_context: &'a mut TransactionContext,
    /// The local program cache for the transaction batch.
    pub program_cache_for_tx_batch: &'a mut ProgramCacheForTxBatch,
    /// Runtime configurations used to provision the invocation environment.
    pub environment_config: EnvironmentConfig<'a>,
    /// The compute budget for the current invocation.
    compute_budget: ComputeBudget,
    /// Instruction compute meter, for tracking compute units consumed against
    /// the designated compute budget during program execution.
    compute_meter: RefCell<u64>,
    log_collector: Option<Rc<RefCell<LogCollector>>>,
    /// Latest measurement not yet accumulated in [ExecuteDetailsTimings::execute_us]
    pub execute_time: Option<Measure>,
    pub timings: ExecuteDetailsTimings,
    pub syscall_context: Vec<Option<SyscallContext>>,
    traces: Vec<Vec<[u64; 12]>>,
}

With this invoke context, the transaction’s message is processed, during which each instruction is executed one by one. For each instruction, the target program is invoked using an eBPF VM, either directly or indirectly. The relationship between invocation styles is covered in the next section.

Invoking a BPF Program

The process of invoking a BPF program at runtime is quite complex. However, this section will break down the process to shed light on the various nuances that can be found in the Agave source.

First and foremost, it’s important to look again to Solana builtin programs. These programs, as we mentioned, are built into the runtime, therefore they don’t need an eBPF virtual machine to be executed. However, one is used anyway.

The use of an eBPF VM to execute runtime builtin programs is primarily a means to enforce consistent interfaces between builtin and BPF programs. This common interface is known as the program entrypoint.

// Psuedo-code Rust interface
fn rust(
    vm: &mut ContextObject,
    arg_a: u64,
    arg_b: u64,
    arg_c: u64,
    arg_d: u64,
    arg_e: u64,
    memory_mapping: &mut MemoryMapping,
) ->

Returning to the tour through Agave’s transaction pipeline, we left off at the InvokeContext. All instructions are processed by the InvokeContext within the process_executable_chain method. Inside this method, only builtin programs are directly invoked.

First, the runtime determines which loader owns the target program. If it’s the native loader, the target program is a builtin. If it’s one of the BPF loaders (which all BPF programs are owned by), that particular BPF Loader builtin program is invoked to actually invoke the target BPF program. This step simply acquires the proper loader ID to use.

let builtin_id = {
    let borrowed_root_account = instruction_context
        .try_borrow_program_account(self.transaction_context, 0)
        .map_err(|_| InstructionError::UnsupportedProgramId)?;
    let owner_id = borrowed_root_account.get_owner();
    if native_loader::check_id(owner_id) {
        *borrowed_root_account.get_key()
    } else {
        *owner_id
    }
};

Next, a reference is taken to the loader’s entrypoint function (the interface covered earlier), taken from its function registry. This will be used to invoke the loader builtin.

// The Murmur3 hash value (used by RBPF) of the string "entrypoint"
const ENTRYPOINT_KEY: u32 = 0x71E3CF81;
let entry = self
    .program_cache_for_tx_batch
    .find(&builtin_id)
    .ok_or(InstructionError::UnsupportedProgramId)?;
let function = match &entry.program {
    ProgramCacheEntryType::Builtin(program) => program
        .get_function_registry()
        .lookup_by_key(ENTRYPOINT_KEY)
        .map(|(_name, function)| function),
    _ => None,
}
.ok_or(InstructionError::UnsupportedProgramId)?;

Just a few more lines down is where the eBPF VM is finally created. However, this VM is merely a mockup. The use of a mocked-up VM enforces interface adherence and also allows the runtime to call into the builtin program as an rBPF builtin function (or syscall).

let mock_config = Config::default();
let empty_memory_mapping =
    MemoryMapping::new(Vec::new(), &mock_config, &SBPFVersion::V1).unwrap();
let mut vm = EbpfVm::new(
    self.program_cache_for_tx_batch
        .environments
        .program_runtime_v2
        .clone(),
    &SBPFVersion::V1,
    // Removes lifetime tracking
    unsafe { std::mem::transmute::<

Inside of rBPF, the invoke_function method for the EbpfVm merely calls into the Rust interface, without knowing anything about the entity it’s calling into. In this case, it’s a Solana builtin program.

/// Invokes a built-in function
pub fn invoke_function(&mut self, function: BuiltinFunction<C>) {
    function(
        unsafe {
            std::ptr::addr_of_mut!(*self)
                .cast::<u64>()
                .offset(get_runtime_environment_key() as isize)
                .cast::<Self>()
        },
        self.registers[1],
        self.registers[2],
        self.registers[3],
        self.registers[4],
        self.registers[5],
    );
}

You might be wondering at this point: if only builtins are invoked by the runtime, using a mocked-up eBPF VM, where does my instruction’s actual target BPF program get invoked? The answer is not immediately clear from the runtime code, because the mechanism is actually part of the BPF Loader program’s processor.

As mentioned above, when a BPF program is targeted by an instruction, the runtime is going to invoke its owner, which is one of the BPF Loader programs. The BPF Loader program’s processor is going to determine which type of instruction it has received. This can either be a program account management instruction (ie. upgrade, close) or the invocation of a BPF program.

If the BPF Loader program’s account is present in the instruction context, the processor infers the instruction to be for the BPF Loader program. Conversely, if the target program is inferred to be a BPF program, then the BPF Loader’s execute function is called.

pub fn process_instruction_inner(
    invoke_context: &mut InvokeContext,
) -> Result<u64, Box<dyn std::error::Error>> {
    /* ... */
    let program_account =
        instruction_context.try_borrow_last_program_account(transaction_context)?;

    // Program Management Instruction
    if native_loader::check_id(program_account.get_owner()) {
        /* ... */
        return {
         /* more logic ... */
         process_loader_upgradeable_instruction(invoke_context)
        }
    }
    
    // If the program account is not the BPF Loader program,
    // execute the BPF program.
}

The BPF Loader’s execute function contains all of the setup steps for the real eBPF VM, which will execute the target BPF program.

Executing a BPF Program

As shown in the previous section, when an eBPF VM is used to invoke a Solana builtin program, the invoke_function method is called, which blindly calls into a builtin function. However, when a BPF program is executed, the runtime will actually call the VM’s execute_program method. This is where proper setup of the VM becomes paramount.

The tour through Agave’s runtime left off in the last section at the BPF Loader’s execute function. Within this function, a proper eBPF VM is provisioned and used to execute a BPF program. There are four important steps involved in setting up the VM.

1. Parameter serialization

2. Stack and heap provision

3. Memory mapping configuration

4. Syscall context configuration

Parameter serialization is where the canonical program parameters are serialized into the VM’s memory regions (program ID, account infos, instruction data). During this step, all accounts, then the instruction data, then the program ID are serialized, where they will eventually be deserialized by the SDK’s entrypoint! macro known to most Solana developers.

let (parameter_bytes, regions, accounts_metadata) = serialization::serialize_parameters(
    invoke_context.transaction_context,
    instruction_context,
    !direct_mapping,
)?;

Next up, the stack and heap for the program’s memory are provisioned. Developers can request more heap space using the Compute Budget program.

macro_rules! create_vm {
	/* ... */
	let stack_size = $program.get_config().stack_size();
	let heap_size = invoke_context.get_compute_budget().heap_size;
	let heap_cost_result = invoke_context.consume_checked($crate::calculate_heap_cost(
	    heap_size,
	    invoke_context.get_compute_budget().heap_cost,
	));
	/* ... */
}

Now that the parameters have been serialized into memory regions and the stack and heap have been provisioned, all of these regions can be used to build a memory mapping of host memory to VM memory. This will combine these newly provisioned regions with the regions in the program’s ELF to make a complete map, to be used by the VM.

Finally, the syscall context in the invoke context is set, which is used to store the memory addresses of account fields in order to provide better error stack tracing.

pub struct SyscallContext {
    pub allocator: BpfAllocator,
    pub accounts_metadata: Vec<SerializedAccountMetadata>,
    pub trace_log: Vec<[u64; 12]>,
}

pub struct SerializedAccountMetadata {
    pub original_data_len: usize,
    pub vm_data_addr: u64,
    pub vm_key_addr: u64,
    pub vm_lamports_addr: u64,
    pub vm_owner_addr: u64,
}

impl<'a> ContextObject for InvokeContext<'a> {
    fn trace(&mut self, state: [u64; 12]) {
        self.syscall_context
            .last_mut()
            .unwrap()
            .as_mut()
            .unwrap()
            .trace_log
            .push(state);
    }

    fn consume(&mut self, amount: u64) {
        // 1 to 1 instruction to compute unit mapping
        // ignore overflow, Ebpf will bail if exceeded
        let mut compute_meter = self.compute_meter.borrow_mut();
        *compute_meter = compute_meter.saturating_sub(amount);
    }

    fn get_remaining(&self) -> u64 {
        *self.compute_meter.borrow()
    }
}

The implementation of rBPF’s ContextObject trait on the InvokeContext struct (alluded to earlier) shows where the syscall context is used to provide a trace. The InvokeContext is also responsible for metering compute units (CUs) for the entire transaction. The ContextObject method consume is called at the end of each program’s execution, reducing the CU meter for the rest of the transaction.

The VM has checks for CU meter overflows at every instruction, so once the maximum CU budget is reached, the next instruction will be aborted with Error::ExceededMaxInstructions. This prevents any long-running processes or infinite loops from performing Denial-of-Service attacks on a Solana validator through malicious program processes. It also will return an error, causing the transaction to fail when CUs are exceeded.

After completion of those four essential steps, the eBPF VM can be properly provisioned and the program can be executed.

Ok(EbpfVm::new(
    program.get_loader().clone(),
    program.get_sbpf_version(),
    invoke_context,
    memory_mapping,
    stack_size,
))

let (compute_units_consumed, result) = vm.execute_program(executable, !use_jit);

The execution of the program within rBPF is simply a process of executing each op code in the program binary until the program has finished executing. As mentioned in the earlier section, this can be done through interpretation or execution of the JIT-compiled binary. Upon completion, the VM will return a u64 code, where zero represents program success.

Panics within the VM are handled by the EbpfError in the rBPF library. On the Agave runtime side, these errors are converted into runtime InstructionError types by the InvokeContext. For example, if a syscall throws an InstructionError, that error is passed through the VM via EbpfError::SyscallError(..), then re-cast into its proper InstructionError in the runtime. For most other EbfError variants, the infamous InstructionError::ProgramFailedToComplete error is thrown.

Finally, the return code from the virtual machine is propagated through the runtime, ultimately yielding the program result of the transaction instruction. This results in the transaction result common to many Solana developers, and thus concludes this tour of Agave’s runtime and virtual machine!

As we continue to evolve and optimize the runtime, Anza remains intensely focused on increased performance, optimization of compute units, and extended functionality. Get involved in the discussions by lending your opinion in the Solana Improvement Documents process!