nominatim servers keep overflowing kernel log #29
Description
The linux kernel on the nominatim machines keeps flooding the kernel log with the following message:
Apr 8 04:55:37 pummelzacken kernel: [403057.479427] ------------[ cut here ]------------
Apr 8 04:55:37 pummelzacken kernel: [403057.479436] WARNING: CPU: 2 PID: 17095 at /build/buildd/linux-3.13.0/net/core/dst.c:285 dst_release+0x45/0x60()
Apr 8 04:55:37 pummelzacken kernel: [403057.479437] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs cpuid act_police cls_basic cls_flow cls_fw cls_u32 sch_fq_codel sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_LOG xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_nat ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_defrag_ipv6 xt_time xt_TCPMSS xt_tcpmss xt_sctp xt_policy xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 gpio_ich joydev lrw gf128mul glue_helper serio_raw ablk_helper cryptd ioatdma wmi mei_me sb_edac edac_core lpc_ich mei ipmi_si shpchp mac_hid ipmi_devintf xfs lp parport libcrc32c raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq hid_generic igb ixgbe raid1 i2c_algo_bit usbhid raid0 dca hid multipath ahci ptp linear psmouse libahci aacraid mdio pps_core
Apr 8 04:55:37 pummelzacken kernel: [403057.479529] CPU: 2 PID: 17095 Comm: postgres Not tainted 3.13.0-48-generic #80-Ubuntu
Apr 8 04:55:37 pummelzacken kernel: [403057.479530] Hardware name: Supermicro X9DRW/X9DRW, BIOS 3.0c 03/24/2014
Apr 8 04:55:37 pummelzacken kernel: [403057.479531] 0000000000000009 ffff8810270bbb18 ffffffff81721506 0000000000000000
Apr 8 04:55:37 pummelzacken kernel: [403057.479535] ffff8810270bbb50 ffffffff810677dd ffff882025b2c600 00000000ffffffff
Apr 8 04:55:37 pummelzacken kernel: [403057.479538] ffff8820262d0000 0000000000000000 0000000000000000 ffff8810270bbb60
Apr 8 04:55:37 pummelzacken kernel: [403057.479540] Call Trace:
Apr 8 04:55:37 pummelzacken kernel: [403057.479545] [<ffffffff81721506>] dump_stack+0x45/0x56
Apr 8 04:55:37 pummelzacken kernel: [403057.479548] [<ffffffff810677dd>] warn_slowpath_common+0x7d/0xa0
Apr 8 04:55:37 pummelzacken kernel: [403057.479551] [<ffffffff810678ba>] warn_slowpath_null+0x1a/0x20
Apr 8 04:55:37 pummelzacken kernel: [403057.479552] [<ffffffff8162dea5>] dst_release+0x45/0x60
Apr 8 04:55:37 pummelzacken kernel: [403057.479555] [<ffffffff8160eed9>] sk_dst_check+0xb9/0xe0
Apr 8 04:55:37 pummelzacken kernel: [403057.479559] [<ffffffff816c18df>] ip6_sk_dst_lookup_flow+0x2f/0x1b0
Apr 8 04:55:37 pummelzacken kernel: [403057.479563] [<ffffffff816dc7ce>] udpv6_sendmsg+0x61e/0xb10
Apr 8 04:55:37 pummelzacken kernel: [403057.479566] [<ffffffff81613e37>] ? kfree_skbmem+0x37/0xa0
Apr 8 04:55:37 pummelzacken kernel: [403057.479569] [<ffffffff81616e64>] ? consume_skb+0x34/0x80
Apr 8 04:55:37 pummelzacken kernel: [403057.479573] [<ffffffff8136929e>] ? radix_tree_lookup_slot+0xe/0x10
Apr 8 04:55:37 pummelzacken kernel: [403057.479577] [<ffffffff8114f0be>] ? find_get_page+0x1e/0xa0
Apr 8 04:55:37 pummelzacken kernel: [403057.479580] [<ffffffff81693d84>] inet_sendmsg+0x64/0xb0
Apr 8 04:55:37 pummelzacken kernel: [403057.479584] [<ffffffff81313d37>] ? apparmor_socket_sendmsg+0x17/0x20
Apr 8 04:55:37 pummelzacken kernel: [403057.479586] [<ffffffff8160ca2b>] sock_sendmsg+0x8b/0xc0
Apr 8 04:55:37 pummelzacken kernel: [403057.479590] [<ffffffff8172d5b4>] ? __do_page_fault+0x204/0x560
Apr 8 04:55:37 pummelzacken kernel: [403057.479592] [<ffffffff8160cf71>] SYSC_sendto+0x121/0x1c0
Apr 8 04:55:37 pummelzacken kernel: [403057.479595] [<ffffffff811c31af>] ? SYSC_newstat+0x2f/0x40
Apr 8 04:55:37 pummelzacken kernel: [403057.479597] [<ffffffff8172d92a>] ? do_page_fault+0x1a/0x70
Apr 8 04:55:37 pummelzacken kernel: [403057.479599] [<ffffffff8160d95e>] SyS_sendto+0xe/0x10
Apr 8 04:55:37 pummelzacken kernel: [403057.479602] [<ffffffff81731fbd>] system_call_fastpath+0x1a/0x1f
Apr 8 04:55:37 pummelzacken kernel: [403057.479603] ---[ end trace 435c86112e482386 ]---
Apr 8 04:56:06 pummelzacken kernel: [403086.544482] ------------[ cut here ]------------
It starts with a message every few minutes and keeps increasing until it floods the log and the machine needs to be restarted which stops the messages for a while. Observed on pummelzacken at UCL and poldi at UCL and IC.