Skip to content

Instantly share code, notes, and snippets.

@srenatus
Created January 14, 2014 10:32
Show Gist options
  • Save srenatus/8416347 to your computer and use it in GitHub Desktop.
Save srenatus/8416347 to your computer and use it in GitHub Desktop.
openvswitch kernel panic; openvswitch srcversion 216E480D4E0B314F8FCD8AC from openvswitch-datapath-lts-raring-dkms; Linux 3.8.0-35-generic
Jan 14 10:30:00 node15 kernel: [ 1130.796116] IPv6: ADDRCONF(NETDEV_UP): qvbd9c941da-54: link is not ready
Jan 14 10:30:00 node15 kernel: [ 1130.855830] device qvbd9c941da-54 entered promiscuous mode
Jan 14 10:30:00 node15 kernel: [ 1130.914772] IPv6: ADDRCONF(NETDEV_CHANGE): qvbd9c941da-54: link becomes ready
Jan 14 10:30:00 node15 kernel: [ 1130.967049] device qvod9c941da-54 entered promiscuous mode
Jan 14 10:30:00 node15 kernel: [ 1131.073473] qbrd9c941da-54: port 1(qvbd9c941da-54) entered forwarding state
Jan 14 10:30:00 node15 kernel: [ 1131.073478] qbrd9c941da-54: port 1(qvbd9c941da-54) entered forwarding state
Jan 14 10:30:00 node15 ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl -- --may-exist add-port br-int qvod9c941da-54 -- set Interface qvod9c941da-54 external-ids:iface-id=
d9c941da-542f-4b4c-b472-83714d577961 external-ids:iface-status=active external-ids:attached-mac=fa:16:3e:f2:57:3b external-ids:vm-uuid=f2ba1a61-d39d-4284-9d23-dc55b1ac859e
Jan 14 10:30:00 node15 kernel: [ 1131.573655] type=1400 audit(1389695400.973:34): apparmor="STATUS" operation="profile_load" name="libvirt-f2ba1a61-d39d-4284-9d23-dc55b1ac859e" pid
=5008 comm="apparmor_parser"
Jan 14 10:30:01 node15 kernel: [ 1131.649534] device tapd9c941da-54 entered promiscuous mode
Jan 14 10:30:01 node15 kernel: [ 1131.653559] qbrd9c941da-54: port 2(tapd9c941da-54) entered forwarding state
Jan 14 10:30:01 node15 kernel: [ 1131.653568] qbrd9c941da-54: port 2(tapd9c941da-54) entered forwarding state
Jan 14 10:30:02 node15 ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 set Port qvod9c941da-54 tag=1
Jan 14 10:30:04 node15 kernel: [ 1134.139162] BUG: unable to handle kernel NULL pointer dereference at 0000000000000006
Jan 14 10:30:04 node15 kernel: [ 1134.139676] IP: [<ffffffffa058bb16>] ovs_tnl_send+0x566/0xb50 [openvswitch]
Jan 14 10:30:04 node15 kernel: [ 1134.140131] PGD 1fd29dd067 PUD 1fcbbd9067 PMD 0
Jan 14 10:30:04 node15 kernel: [ 1134.140456] Oops: 0000 [#1] SMP
Jan 14 10:30:04 node15 kernel: [ 1134.140677] Modules linked in: nf_conntrack_ipv6(F) nf_defrag_ipv6(F) xt_mac(F) xt_physdev(F) veth(F) ib_iser(F) rdma_cm(F) ib_cm(F) iw_cm(F) ib_s
a(F) ib_mad(F) ib_core(F) ib_addr(F) iscsi_tcp(F) libiscsi_tcp(F) libiscsi(F) scsi_transport_iscsi(F) nfsd(F) nfs_acl(F) auth_rpcgss(F) nfs(F) fscache(F) lockd(F) sunrpc(F) openvsw
itch(OF) ip6table_filter(F) ip6_tables(F) ebtable_nat(F) ebtables(F) ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_sta[34/127]
_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) iptable_filter(F) ip_tables(F) x_tables(F) bridge(F) stp(F) llc(F) btrfs(F) zlib_deflate(F) ufs(F) qnx4(F)
hfsplus(F) hfs(F) minix(F) ntfs(F) msdos(F) jfs(F) xfs(F) libcrc32c(F) reiserfs(F) vesafb(F) coretemp(F) kvm_intel(F) kvm(F) ghash_clmulni_intel(F) aesni_intel(F) ablk_helper(F) c
ryptd(F) lrw(F) aes_x86_64(F) xts(F) gf128mul(F) gpio_ich(F) microcode(F) joydev(F) mei(F) ioatdma(F) lpc_ich(F) wmi(F) mac_hid
Jan 14 10:30:04 node15 kernel: (F) lp(F) parport(F) ext2(F) hid_generic(F) usbhid(F) hid(F) isci(F) ahci(F) libsas(F) libahci(F) scsi_transport_sas(F) ixgbe(F) mdio(F) igb(F) dca(F
) ptp(F) pps_core(F)
Jan 14 10:30:04 node15 kernel: [ 1134.148161] CPU 16
Jan 14 10:30:04 node15 kernel: [ 1134.148274] Pid: 35169, comm: ovs-vswitchd Tainted: GF O 3.8.0-35-generic #50~precise1-Ubuntu Supermicro X9DRT-HF+/X9DRT-HF+
Jan 14 10:30:04 node15 kernel: [ 1134.148974] RIP: 0010:[<ffffffffa058bb16>] [<ffffffffa058bb16>] ovs_tnl_send+0x566/0xb50 [openvswitch]
Jan 14 10:30:04 node15 kernel: [ 1134.149579] RSP: 0018:ffff881fd005d788 EFLAGS: 00010246
Jan 14 10:30:04 node15 kernel: [ 1134.149917] RAX: ffff883fd0da5428 RBX: 000000000000001c RCX: 0000000000000040
Jan 14 10:30:04 node15 kernel: [ 1134.162398] RDX: 0000000000000040 RSI: 000000000000001c RDI: 0000000000000000
Jan 14 10:30:04 node15 kernel: [ 1134.174814] RBP: ffff881fd005d848 R08: 0000000000000000 R09: ffff883fd0da5400
Jan 14 10:30:04 node15 kernel: [ 1134.186846] R10: ffff883fce7c5440 R11: 0000000000000000 R12: ffff883fd02c3700
Jan 14 10:30:04 node15 kernel: [ 1134.198812] R13: ffff883fce412a00 R14: ffff883fd2a6c480 R15: 0000000000000000
Jan 14 10:30:04 node15 kernel: [ 1134.210783] FS: 00007f02c482e700(0000) GS:ffff883fff180000(0000) knlGS:0000000000000000
Jan 14 10:30:04 node15 kernel: [ 1134.233196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 14 10:30:04 node15 kernel: [ 1134.246277] CR2: 0000000000000006 CR3: 0000001fae1f5000 CR4: 00000000001407e0
Jan 14 10:30:04 node15 kernel: [ 1134.257930] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 14 10:30:04 node15 kernel: [ 1134.269214] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 14 10:30:04 node15 kernel: [ 1134.280673] Process ovs-vswitchd (pid: 35169, threadinfo ffff881fd005c000, task ffff881fccff45c0)
Jan 14 10:30:04 node15 kernel: [ 1134.301819] Stack:
Jan 14 10:30:04 node15 kernel: [ 1134.312699] 0000000000000282 ffff881fd005d7a8 0000000000000282 ffffffff81cbe3f0
Jan 14 10:30:04 node15 kernel: [ 1134.333455] ffff881fd005d7e8 40ffffff8108a4b3 ffff881fd005d700 0000000000000000
Jan 14 10:30:04 node15 kernel: [ 1134.354349] 0000000000000040 00000000000000d0 ffff883fcfdb37e0 0000000000000000
Jan 14 10:30:04 node15 kernel: [ 1134.375306] Call Trace: [9/127]
Jan 14 10:30:04 node15 kernel: [ 1134.385162] [<ffffffff8160ca8d>] ? netlink_broadcast_filtered+0x12d/0x190
Jan 14 10:30:04 node15 kernel: [ 1134.395250] [<ffffffffa058cc6e>] ovs_vport_send+0x1e/0x50 [openvswitch]
Jan 14 10:30:04 node15 kernel: [ 1134.405204] [<ffffffffa058218e>] do_output+0x2e/0x50 [openvswitch]
Jan 14 10:30:04 node15 kernel: [ 1134.415101] [<ffffffffa05826e6>] do_execute_actions+0x386/0x960 [openvswitch]
Jan 14 10:30:04 node15 kernel: [ 1134.435114] [<ffffffff815d779b>] ? __kmalloc_reserve+0x3b/0xa0
Jan 14 10:30:04 node15 kernel: [ 1134.444905] [<ffffffff811840ec>] ? ksize+0x1c/0xc0
Jan 14 10:30:04 node15 kernel: [ 1134.454221] [<ffffffffa0582ef3>] ? reserve_sfa_size+0x33/0xe0 [openvswitch]
Jan 14 10:30:04 node15 kernel: [ 1134.464005] [<ffffffffa0582d2c>] ovs_execute_actions+0x6c/0x110 [openvswitch]
Jan 14 10:30:04 node15 kernel: [ 1134.484002] [<ffffffffa0585410>] ovs_packet_cmd_execute+0x210/0x240 [openvswitch]
Jan 14 10:30:04 node15 kernel: [ 1134.508786] [<ffffffff8160f67d>] genl_rcv_msg+0x22d/0x2b0
Jan 14 10:30:04 node15 kernel: [ 1134.521880] [<ffffffff815d830b>] ? __alloc_skb+0x8b/0x2a0
Jan 14 10:30:04 node15 kernel: [ 1134.532319] [<ffffffff8160f450>] ? genl_rcv+0x40/0x40
Jan 14 10:30:04 node15 kernel: [ 1134.542574] [<ffffffff8160ee29>] netlink_rcv_skb+0xa9/0xd0
Jan 14 10:30:04 node15 kernel: [ 1134.552248] [<ffffffff8160f435>] genl_rcv+0x25/0x40
Jan 14 10:30:04 node15 kernel: [ 1134.561578] [<ffffffff8160e771>] netlink_unicast+0x1b1/0x230
Jan 14 10:30:04 node15 kernel: [ 1134.571097] [<ffffffff8160eaee>] netlink_sendmsg+0x2fe/0x3b0
Jan 14 10:30:04 node15 kernel: [ 1134.581002] [<ffffffff815cb282>] sock_sendmsg+0xd2/0xf0
Jan 14 10:30:04 node15 kernel: [ 1134.590436] [<ffffffff815cb8e7>] ? sock_aio_read.part.10+0x57/0x80
Jan 14 10:30:04 node15 kernel: [ 1134.599667] [<ffffffff815da406>] ? verify_iovec+0x56/0xd0
Jan 14 10:30:04 node15 kernel: [ 1134.609190] [<ffffffff815cd09c>] ___sys_sendmsg+0x3ac/0x3c0
Jan 14 10:30:04 node15 kernel: [ 1134.618139] [<ffffffff81045cd9>] ? default_spin_lock_flags+0x9/0x10
Jan 14 10:30:04 node15 kernel: [ 1134.627608] [<ffffffff811e3294>] ? ep_scan_ready_list.isra.8+0x1b4/0x1c0
Jan 14 10:30:04 node15 kernel: [ 1134.636730] [<ffffffff811e339b>] ? ep_poll+0xdb/0x380
Jan 14 10:30:04 node15 kernel: [ 1134.645589] [<ffffffff815ceba9>] __sys_sendmsg+0x49/0x90
Jan 14 10:30:04 node15 kernel: [ 1134.653963] [<ffffffff815cec09>] sys_sendmsg+0x19/0x20
Jan 14 10:30:04 node15 kernel: [ 1134.662144] [<ffffffff816fdf1d>] system_call_fastpath+0x1a/0x1f
Jan 14 10:30:04 node15 kernel: [ 1134.670179] Code: 00 00 00 0f b6 52 04 88 50 09 8b 55 a0 89 50 10 8b 55 cc 88 48 01 0f b7 4d 80 89 50 0c 0f b6 95 6f ff ff ff 66 89 48 06 88 50 08
<f6> 47 06 40 0f 84 d8 01 00 00 f6 40 7c 01 0f 85 ce 01 00 00 66
Jan 14 10:30:04 node15 kernel: [ 1134.696694] RIP [<ffffffffa058bb16>] ovs_tnl_send+0x566/0xb50 [openvswitch]
Jan 14 10:30:04 node15 kernel: [ 1134.705449] RSP <ffff881fd005d788>
Jan 14 10:30:04 node15 kernel: [ 1134.713950] CR2: 0000000000000006
Jan 14 10:30:04 node15 kernel: [ 1134.735591] ---[ end trace b4e9674cfcdf4178 ]---
@shraddha-pandhe
Copy link

Hi srenatus,

I am seeing the same crash on my setup with openvswitch 1.9.0. Can you tell me which version this crash was observed? Any idea if this is fixed in ovs? If yes, which version of ovs has the fix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment