最近在做Linux Kernel Kexec的工作，Windows 10的HyperV VM里装了个Fedora 29，惊讶的发现Kexec竟然不能工作。
调用Kexec会直接蹦出这个Panic：
[ 19.966336] kexec_core: Starting new kernel
[ 19.994408] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[ 19.995152] PGD 8000000057995067 P4D 8000000057995067 PUD 57990067 PMD 0
[ 19.995152] Oops: 0002 [#1] SMP PTI
[ 19.995152] CPU: 0 PID: 1016 Comm: kexec Not tainted 4.18.16-300.fc29.x86_64 #1
[ 19.995152] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v3.0 03/02/2018
[ 19.995152] RIP: 0010:0xffffc9000001d000
[ 19.995152] Code: Bad RIP value.
[ 19.995152] RSP: 0018:ffffc9000495bcf0 EFLAGS: 00010046
[ 19.995152] RAX: 0000000000000000 RBX: ffffc9000001d000 RCX: 0000000000020015
[ 19.995152] RDX: 000000007f553000 RSI: 0000000000000000 RDI: ffffc9000495bd28
[ 19.995152] RBP: 0000000000000002 R08: 0000000000000000 R09: ffffffff8238aaf8
[ 19.995152] R10: ffffffff8238aae0 R11: 0000000000000000 R12: ffff88007f553008
[ 19.995152] R13: 0000000000000001 R14: ffff8800ff553000 R15: 0000000000000000
[ 19.995152] FS: 00007ff5c0e67b80(0000) GS:ffff880078e00000(0000) knlGS:0000000000000000
[ 19.995152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 19.995152] CR2: ffffc9000001cfd6 CR3: 000000004f678006 CR4: 00000000003606f0
[ 19.995152] Call Trace:
[ 19.995152] ? __send_ipi_mask+0x1c6/0x2d0
[ 19.995152] ? hv_send_ipi_mask_allbutself+0x6d/0xb0
[ 19.995152] ? mp_save_irq+0x70/0x70
[ 19.995152] ? __ioapic_read_entry+0x32/0x50
[ 19.995152] ? ioapic_read_entry+0x39/0x50
[ 19.995152] ? clear_IO_APIC_pin+0xb8/0x110
[ 19.995152] ? native_stop_other_cpus+0x6e/0x170
[ 19.995152] ? native_machine_shutdown+0x22/0x40
[ 19.995152] ? kernel_kexec+0x136/0x156
[ 19.995152] ? __do_sys_reboot+0x1be/0x210
[ 19.995152] ? kmem_cache_free+0x1b1/0x1e0
[ 19.995152] ? __dentry_kill+0x10b/0x160
[ 19.995152] ? _cond_resched+0x15/0x30
[ 19.995152] ? dentry_kill+0x47/0x170
[ 19.995152] ? dput.part.34+0xc6/0x100
[ 19.995152] ? __fput+0x147/0x220
[ 19.995152] ? _cond_resched+0x15/0x30
[ 19.995152] ? task_work_run+0x38/0xa0
[ 19.995152] ? do_syscall_64+0x5b/0x160
[ 19.995152] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 19.995152] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables sunrpc vfat fat crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_rapl_perf hv_balloon joydev xfs libcrc32c hv_storvsc serio_raw scsi_transport_fc hv_netvsc hyperv_keyboard hyperv_fb hid_hyperv crc32c_intel hv_vmbus
跟着Trackback看了下代码，大概是Kexec尝试关闭其他CPU的时候，Send IPI发生了Panic。
# gdb vmlinux
(gdb) list *(__send_ipi_mask+0x1c6)
0xffffffff81024166 is in __send_ipi_mask (./arch/x86/include/asm/mshyperv.h:138).
133
134 #ifdef CONFIG_X86_64
135 if (!hv_hypercall_pg)
136 return U64_MAX;
137
138 __asm__ __volatile__("mov %4, %%r8\n"
139 CALL_NOSPEC
140 : "=a" (hv_status), ASM_CALL_CONSTRAINT,
141 "+c" (control), "+d" (input_address)
142 : "r" (output_address),Panic就发生在那个Inline ASM里面。这个RIP: 0010:0xffffc9000001d000以及Code: Bad RIP value，又是怎么回事呢？
根据汇编代码分析一下：
x = y + ((x > y) ? phys_base : (__START_KERNEL_map - PAGE_OFFSET));
ffffffff8102413b: 48 c7 c2 00 00 00 80 mov $0xffffffff80000000,%rdx
ffffffff81024142: 48 2b 15 97 39 16 01 sub 0x1163997(%rip),%rdx # ffffffff82187ae0 <page_offset_base>
ffffffff81024149: 48 8b 1d a0 62 94 01 mov 0x19462a0(%rip),%rbx # ffffffff8296a3f0 <hv_hypercall_pg>
ffffffff81024150: 4c 01 f2 add %r14,%rdx
ffffffff81024153: 48 85 db test %rbx,%rbx
ffffffff81024156: 0f 84 59 ff ff ff je ffffffff810240b5 <__send_ipi_mask+0x115>
__asm__ __volatile__("mov %4, %%r8\n"
ffffffff8102415c: 31 c0 xor %eax,%eax
ffffffff8102415e: 49 89 c0 mov %rax,%r8
! ===> ffffffff81024161: ff d3 callq *%rbx # ! <=== rbx is 0xffffc9000001d000, value of <hv_hypercall_pg>
ffffffff81024163: 90 nop
ffffffff81024164: 90 nop
ffffffff81024165: 90 nop
ffffffff81024166: 48 89 c3 mov %rax,%rbx
if ((status & HV_HYPERCALL_RESULT_MASK) != HV_STATUS_SUCCESS)
ffffffff81024169: 66 85 c0 test %ax,%ax
ffffffff8102416c: 75 0a jne ffffffff81024178 <__send_ipi_mask+0x1d8>注意0xffffffff81024166之前，有一条mov 0x19462a0(%rip),%rbx，此处把hv_hypercall_pg赋值给了rbx。0xffffc9000001d000应该是此时hv_hypercall_pg的值。
可以看出是hv_hypercall_pg被使用的时候发生了Page Fault，初步估计是什么东西错误地提前把它释放了。
先做一个Bisect吧。
经过10多次recompile，发现commit 引入了这个问题，内容很清晰，用Hypercall去做IPI。
之后又仔细梳理了一下Kexec的过程，发现Panic的地方正好对的上，分别是在和APIC通讯和Send REBOOT_VECTOR这个IPI。分别在这两个步骤之前直接返回失败，kexec便会Fail，不会Panic。
发现HyperV重写了