Skip to content

gnrc_sixlowpan_frag: release packet rbuf_add error cases#10680

Merged
gschorcht merged 2 commits intoRIOT-OS:masterfrom
miri64:gnrc_sixlowpan_frag/fix/rbuf-full-release
Dec 30, 2018
Merged

gnrc_sixlowpan_frag: release packet rbuf_add error cases#10680
gschorcht merged 2 commits intoRIOT-OS:masterfrom
miri64:gnrc_sixlowpan_frag/fix/rbuf-full-release

Conversation

@miri64
Copy link
Copy Markdown
Member

@miri64 miri64 commented Dec 30, 2018

Contribution description

Otherwise, there will be leaks ;-).

Testing procedure

Together with #10679, run steps to reproduce outlined in #10672. The packet buffer should now still be full at some point, but after ping finishes it should be empty again (give the reassembly buffer 3 secs to garbage collect itself before you check).

Issues/PRs references

Together with #10679 this is a fix for #10672.

@miri64
gnrc_sixlowpan_frag: release packet when rbuf is full
a409603 
Otherwise, there will be leaks ;-).
@miri64 miri64 added Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) Area: network Area: Networking CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Dec 30, 2018
@miri64 miri64 changed the title gnrc_sixlowpan_frag: release packet when rbuf is full gnrc_sixlowpan_frag: release packet rbuf_add error cases Dec 30, 2018
@miri64
Copy link
Copy Markdown
Member Author

miri64 commented Dec 30, 2018

Found another fix for that function (unrelated to the reassembly buffers state but still an error case in that function) so I fixed the title.

@TimoRoth
Copy link
Copy Markdown
Contributor

TimoRoth commented Dec 30, 2018 •
edited
Loading

This fixes the indefinitely overfilled buffer for me. pings still fail very frequently, but it recovers cleanly when the buffer runs full.

Edit: And as I speak, with only one of my two boards still pinging, it runs into it again.
The pktbuf looks different this time, but it's full and not being cleaned up either way:

pktbuf 

2018-12-30 01:18:56,057 - INFO #  pktbuf
2018-12-30 01:18:56,065 - INFO # packet buffer: first byte: 0x3ffb8da8, last byte: 0x3ffba5a8 (size: 6144)
2018-12-30 01:18:56,069 - INFO #   position of last byte used: 6136
2018-12-30 01:18:56,074 - INFO # =========== chunk   0 (0x3ffb8da8 size: 1624) ===========
2018-12-30 01:18:56,080 - INFO # 00000000  20  8F  FB  3F  A8  99  FB  3F  F6  00  00  00  01  00  00  00
2018-12-30 01:18:56,086 - INFO # 00000010  00  00  00  00  00  00  00  00  48  94  FB  3F  A0  9A  FB  3F
2018-12-30 01:18:56,093 - INFO # 00000020  F6  00  00  00  01  00  00  00  00  00  00  00  53  53  53  53
2018-12-30 01:18:56,099 - INFO # 00000030  00  00  00  00  90  8E  FB  3F  D8  04  00  00  01  00  00  00
2018-12-30 01:18:56,105 - INFO # 00000040  03  00  00  00  18  00  00  00  00  00  00  00  70  94  FB  3F
2018-12-30 01:18:56,112 - INFO # 00000050  D8  04  00  00  01  00  00  00  03  00  00  00  53  53  53  53
2018-12-30 01:18:56,119 - INFO # 00000060  F0  8D  FB  3F  68  93  FB  3F  28  00  00  00  01  00  00  00
2018-12-30 01:18:56,125 - INFO # 00000070  02  00  00  00  53  53  53  53  00  00  00  00  98  9B  FB  3F
2018-12-30 01:18:56,131 - INFO # 00000080  D8  04  00  00  01  00  00  00  03  00  00  00  00  00  00  00
2018-12-30 01:18:56,137 - INFO # 00000090  D8  8D  FB  3F  50  8E  FB  3F  28  00  00  00  01  00  00  00
2018-12-30 01:18:56,143 - INFO # 000000A0  02  00  00  00  00  00  00  00  60  00  00  00  28  00  FF  00
2018-12-30 01:18:56,151 - INFO # 000000B0  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
2018-12-30 01:18:56,157 - INFO # 000000C0  FD  19  FF  FF  00  00  00  00  32  AE  A4  FF  FE  1B  C0  E9
2018-12-30 01:18:56,163 - INFO # 000000D0  00  94  FB  3F  18  94  FB  3F  56  00  00  00  01  00  00  00
2018-12-30 01:18:56,170 - INFO # 000000E0  00  00  00  00  18  00  00  00  80  00  00  00  00  53  25  85
2018-12-30 01:18:56,176 - INFO # 000000F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,181 - INFO # 00000100  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,189 - INFO # 00000110  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,195 - INFO # 00000120  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,201 - INFO # 00000130  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,208 - INFO # 00000140  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,214 - INFO # 00000150  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,220 - INFO # 00000160  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,227 - INFO # 00000170  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,233 - INFO # 00000180  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,239 - INFO # 00000190  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,245 - INFO # 000001A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,251 - INFO # 000001B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,257 - INFO # 000001C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,265 - INFO # 000001D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,271 - INFO # 000001E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,277 - INFO # 000001F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,284 - INFO # 00000200  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,289 - INFO # 00000210  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,295 - INFO # 00000220  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,302 - INFO # 00000230  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,309 - INFO # 00000240  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,315 - INFO # 00000250  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,321 - INFO # 00000260  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,327 - INFO # 00000270  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,333 - INFO # 00000280  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,341 - INFO # 00000290  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,347 - INFO # 000002A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,353 - INFO # 000002B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,360 - INFO # 000002C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,365 - INFO # 000002D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,371 - INFO # 000002E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,379 - INFO # 000002F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,384 - INFO # 00000300  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,391 - INFO # 00000310  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,398 - INFO # 00000320  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,404 - INFO # 00000330  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,410 - INFO # 00000340  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,416 - INFO # 00000350  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,422 - INFO # 00000360  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,428 - INFO # 00000370  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,436 - INFO # 00000380  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,442 - INFO # 00000390  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,447 - INFO # 000003A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,455 - INFO # 000003B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,461 - INFO # 000003C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,467 - INFO # 000003D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,474 - INFO # 000003E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,480 - INFO # 000003F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,486 - INFO # 00000400  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,492 - INFO # 00000410  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,499 - INFO # 00000420  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,505 - INFO # 00000430  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,511 - INFO # 00000440  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,517 - INFO # 00000450  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,525 - INFO # 00000460  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,530 - INFO # 00000470  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,536 - INFO # 00000480  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,544 - INFO # 00000490  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,550 - INFO # 000004A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,556 - INFO # 000004B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,562 - INFO # 000004C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,568 - INFO # 000004D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,574 - INFO # 000004E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,581 - INFO # 000004F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,587 - INFO # 00000500  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,594 - INFO # 00000510  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,600 - INFO # 00000520  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,606 - INFO # 00000530  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,612 - INFO # 00000540  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,620 - INFO # 00000550  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,626 - INFO # 00000560  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,632 - INFO # 00000570  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,638 - INFO # 00000580  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,644 - INFO # 00000590  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,650 - INFO # 000005A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,658 - INFO # 000005B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,664 - INFO # 000005C0  60  00  00  00  B0  00  FF  00  00  00  00  00  00  00  00  00
2018-12-30 01:18:56,670 - INFO # 000005D0  00  00  00  00  00  00  00  00  FD  19  FF  FF  00  00  00  00
2018-12-30 01:18:56,676 - INFO # 000005E0  32  AE  A4  FF  FE  1B  C0  E9  20  8E  FB  3F  A8  93  FB  3F
2018-12-30 01:18:56,682 - INFO # 000005F0  28  00  00  00  01  00  00  00  02  00  00  00  00  00  00  00
2018-12-30 01:18:56,689 - INFO # 00000600  60  00  00  00  70  00  FF  00  00  00  00  00  00  00  00  00
2018-12-30 01:18:56,695 - INFO # 00000610  00  00  00  00  00  00  00  00  FD  19  FF  FF  00  00  00  00
2018-12-30 01:18:56,701 - INFO # 00000620  32  AE  A4  FF  FE  1B  C0  E9  00  00  00  00  70  A0  FB  3F
2018-12-30 01:18:56,707 - INFO # 00000630  D8  04  00  00  01  00  00  00  03  00  00  00  00  00  00  00
2018-12-30 01:18:56,715 - INFO # 00000640  D0  93  FB  3F  48  99  FB  3F  28  00  00  00  01  00  00  00
2018-12-30 01:18:56,717 - INFO # 00000650  02  00  00  00  00  00  00  00
2018-12-30 01:18:56,723 - INFO # ~ unused: 0x3ffb9400 (next: 0x3ffb9970, size:   24) ~
2018-12-30 01:18:56,727 - INFO # =========== chunk   1 (0x3ffb9418 size: 1368) ===========
2018-12-30 01:18:56,733 - INFO # 00000000  01  E5  00  00  2C  96  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,741 - INFO # 00000010  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,747 - INFO # 00000020  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,752 - INFO # 00000030  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,760 - INFO # 00000040  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,765 - INFO # 00000050  53  53  53  53  53  53  53  53  80  00  00  00  00  53  27  2E
2018-12-30 01:18:56,772 - INFO # 00000060  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,778 - INFO # 00000070  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,784 - INFO # 00000080  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,790 - INFO # 00000090  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,797 - INFO # 000000A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,803 - INFO # 000000B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,810 - INFO # 000000C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,816 - INFO # 000000D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,822 - INFO # 000000E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,829 - INFO # 000000F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,835 - INFO # 00000100  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,841 - INFO # 00000110  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,847 - INFO # 00000120  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,855 - INFO # 00000130  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,861 - INFO # 00000140  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,867 - INFO # 00000150  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,873 - INFO # 00000160  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,879 - INFO # 00000170  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,886 - INFO # 00000180  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,893 - INFO # 00000190  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,898 - INFO # 000001A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,905 - INFO # 000001B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,913 - INFO # 000001C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,917 - INFO # 000001D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,923 - INFO # 000001E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,931 - INFO # 000001F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,936 - INFO # 00000200  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,942 - INFO # 00000210  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,950 - INFO # 00000220  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,956 - INFO # 00000230  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,962 - INFO # 00000240  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,968 - INFO # 00000250  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,974 - INFO # 00000260  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,983 - INFO # 00000270  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,988 - INFO # 00000280  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:56,994 - INFO # 00000290  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,000 - INFO # 000002A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,006 - INFO # 000002B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,012 - INFO # 000002C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,019 - INFO # 000002D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,025 - INFO # 000002E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,031 - INFO # 000002F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,039 - INFO # 00000300  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,045 - INFO # 00000310  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,051 - INFO # 00000320  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,057 - INFO # 00000330  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,063 - INFO # 00000340  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,069 - INFO # 00000350  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,077 - INFO # 00000360  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,083 - INFO # 00000370  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,089 - INFO # 00000380  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,095 - INFO # 00000390  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,101 - INFO # 000003A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,107 - INFO # 000003B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,114 - INFO # 000003C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,120 - INFO # 000003D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,127 - INFO # 000003E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,134 - INFO # 000003F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,140 - INFO # 00000400  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,146 - INFO # 00000410  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,152 - INFO # 00000420  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,158 - INFO # 00000430  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,165 - INFO # 00000440  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,171 - INFO # 00000450  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,177 - INFO # 00000460  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,183 - INFO # 00000470  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,190 - INFO # 00000480  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,196 - INFO # 00000490  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,203 - INFO # 000004A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,209 - INFO # 000004B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,215 - INFO # 000004C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,221 - INFO # 000004D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,229 - INFO # 000004E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,235 - INFO # 000004F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,241 - INFO # 00000500  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,248 - INFO # 00000510  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,254 - INFO # 00000520  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,260 - INFO # 00000530  60  00  00  00  60  00  FF  00  00  00  00  00  00  00  00  00
2018-12-30 01:18:57,266 - INFO # 00000540  00  00  00  00  00  00  00  00  FD  19  FF  FF  00  00  00  00
2018-12-30 01:18:57,269 - INFO # 00000550  32  AE  A4  FF  FE  1B  C0  E9
2018-12-30 01:18:57,275 - INFO # ~ unused: 0x3ffb9970 (next: 0x3ffba548, size:   56) ~
2018-12-30 01:18:57,280 - INFO # =========== chunk   2 (0x3ffb99a8 size: 2976) ===========
2018-12-30 01:18:57,286 - INFO # 00000000  01  C5  00  00  21  41  60  00  00  00  04  D8  3A  40  FD  19
2018-12-30 01:18:57,293 - INFO # 00000010  FF  FF  00  00  00  00  32  AE  A4  FF  FE  1B  C0  E9  FD  19
2018-12-30 01:18:57,299 - INFO # 00000020  FF  FF  00  00  00  00  3E  71  BF  FF  FE  83  D5  19  80  00
2018-12-30 01:18:57,305 - INFO # 00000030  97  13  00  53  00  10  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,311 - INFO # 00000040  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,317 - INFO # 00000050  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,323 - INFO # 00000060  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,331 - INFO # 00000070  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,337 - INFO # 00000080  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,343 - INFO # 00000090  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,349 - INFO # 000000A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,355 - INFO # 000000B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,361 - INFO # 000000C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,369 - INFO # 000000D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,375 - INFO # 000000E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,381 - INFO # 000000F0  53  53  53  53  53  53  53  53  01  C5  00  00  2B  41  60  00
2018-12-30 01:18:57,387 - INFO # 00000100  00  00  04  D8  3A  40  FD  19  FF  FF  00  00  00  00  32  AE
2018-12-30 01:18:57,393 - INFO # 00000110  A4  FF  FE  1B  C0  E9  FD  19  FF  FF  00  00  00  00  3E  71
2018-12-30 01:18:57,400 - INFO # 00000120  BF  FF  FE  83  D5  19  80  00  97  0C  00  53  00  17  53  53
2018-12-30 01:18:57,407 - INFO # 00000130  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,413 - INFO # 00000140  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,419 - INFO # 00000150  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,426 - INFO # 00000160  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,432 - INFO # 00000170  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,438 - INFO # 00000180  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,444 - INFO # 00000190  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,450 - INFO # 000001A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,458 - INFO # 000001B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,464 - INFO # 000001C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,469 - INFO # 000001D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,475 - INFO # 000001E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,482 - INFO # 000001F0  80  00  00  00  00  53  2B  E1  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,488 - INFO # 00000200  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,494 - INFO # 00000210  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,502 - INFO # 00000220  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,508 - INFO # 00000230  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,514 - INFO # 00000240  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,521 - INFO # 00000250  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,527 - INFO # 00000260  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,532 - INFO # 00000270  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,540 - INFO # 00000280  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,546 - INFO # 00000290  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,552 - INFO # 000002A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,558 - INFO # 000002B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,564 - INFO # 000002C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,572 - INFO # 000002D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,578 - INFO # 000002E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,584 - INFO # 000002F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,591 - INFO # 00000300  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,597 - INFO # 00000310  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,603 - INFO # 00000320  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,610 - INFO # 00000330  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,616 - INFO # 00000340  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,622 - INFO # 00000350  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,628 - INFO # 00000360  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,634 - INFO # 00000370  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,641 - INFO # 00000380  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,648 - INFO # 00000390  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,654 - INFO # 000003A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,660 - INFO # 000003B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,667 - INFO # 000003C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,673 - INFO # 000003D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,679 - INFO # 000003E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,686 - INFO # 000003F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,692 - INFO # 00000400  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,698 - INFO # 00000410  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,705 - INFO # 00000420  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,711 - INFO # 00000430  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,717 - INFO # 00000440  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,724 - INFO # 00000450  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,730 - INFO # 00000460  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,736 - INFO # 00000470  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,743 - INFO # 00000480  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,750 - INFO # 00000490  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,756 - INFO # 000004A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,762 - INFO # 000004B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,767 - INFO # 000004C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,775 - INFO # 000004D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,781 - INFO # 000004E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,787 - INFO # 000004F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,793 - INFO # 00000500  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,801 - INFO # 00000510  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,806 - INFO # 00000520  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,812 - INFO # 00000530  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,820 - INFO # 00000540  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,826 - INFO # 00000550  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,831 - INFO # 00000560  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,839 - INFO # 00000570  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,844 - INFO # 00000580  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,850 - INFO # 00000590  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,856 - INFO # 000005A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,863 - INFO # 000005B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,869 - INFO # 000005C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,876 - INFO # 000005D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,882 - INFO # 000005E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,888 - INFO # 000005F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,895 - INFO # 00000600  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,902 - INFO # 00000610  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,908 - INFO # 00000620  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,914 - INFO # 00000630  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,920 - INFO # 00000640  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,926 - INFO # 00000650  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,933 - INFO # 00000660  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,939 - INFO # 00000670  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,945 - INFO # 00000680  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,952 - INFO # 00000690  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,959 - INFO # 000006A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,965 - INFO # 000006B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,971 - INFO # 000006C0  53  53  53  53  53  53  53  53  80  00  00  00  00  53  2B  E2
2018-12-30 01:18:57,977 - INFO # 000006D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,983 - INFO # 000006E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,990 - INFO # 000006F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:57,996 - INFO # 00000700  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,002 - INFO # 00000710  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,010 - INFO # 00000720  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,016 - INFO # 00000730  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,022 - INFO # 00000740  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,028 - INFO # 00000750  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,034 - INFO # 00000760  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,040 - INFO # 00000770  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,047 - INFO # 00000780  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,053 - INFO # 00000790  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,059 - INFO # 000007A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,066 - INFO # 000007B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,073 - INFO # 000007C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,079 - INFO # 000007D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,086 - INFO # 000007E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,092 - INFO # 000007F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,098 - INFO # 00000800  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,104 - INFO # 00000810  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,110 - INFO # 00000820  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,116 - INFO # 00000830  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,123 - INFO # 00000840  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,130 - INFO # 00000850  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,136 - INFO # 00000860  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,143 - INFO # 00000870  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,149 - INFO # 00000880  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,155 - INFO # 00000890  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,161 - INFO # 000008A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,167 - INFO # 000008B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,173 - INFO # 000008C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,181 - INFO # 000008D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,186 - INFO # 000008E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,192 - INFO # 000008F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,199 - INFO # 00000900  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,205 - INFO # 00000910  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,211 - INFO # 00000920  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,218 - INFO # 00000930  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,224 - INFO # 00000940  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,230 - INFO # 00000950  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,238 - INFO # 00000960  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,244 - INFO # 00000970  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,250 - INFO # 00000980  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,256 - INFO # 00000990  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,262 - INFO # 000009A0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,268 - INFO # 000009B0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,276 - INFO # 000009C0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,282 - INFO # 000009D0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,288 - INFO # 000009E0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,294 - INFO # 000009F0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,300 - INFO # 00000A00  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,306 - INFO # 00000A10  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,314 - INFO # 00000A20  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,319 - INFO # 00000A30  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,325 - INFO # 00000A40  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,333 - INFO # 00000A50  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,338 - INFO # 00000A60  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,345 - INFO # 00000A70  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,352 - INFO # 00000A80  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,358 - INFO # 00000A90  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,365 - INFO # 00000AA0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,371 - INFO # 00000AB0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,377 - INFO # 00000AC0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,384 - INFO # 00000AD0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,390 - INFO # 00000AE0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,396 - INFO # 00000AF0  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,402 - INFO # 00000B00  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,408 - INFO # 00000B10  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,414 - INFO # 00000B20  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,421 - INFO # 00000B30  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,427 - INFO # 00000B40  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,433 - INFO # 00000B50  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,440 - INFO # 00000B60  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,446 - INFO # 00000B70  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,452 - INFO # 00000B80  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,459 - INFO # 00000B90  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53  53
2018-12-30 01:18:58,463 - INFO # ~ unused: 0x3ffba548 (next: 0x0, size:   96) ~

So I guess there must be some missing free somewhere still. It's interesting how those packets are all larger than the MTU.

@miri64
Copy link
Copy Markdown
Member Author

miri64 commented Dec 30, 2018

Even together with #10679?

@miri64
Copy link
Copy Markdown
Member Author

miri64 commented Dec 30, 2018

So I guess there must be some missing free somewhere still. It's interesting how those packets are all larger than the MTU.

Are you sure? The chunks are not packets. They can be multiple. The packet buffer just "sees" the empty spaces between the allocated space, so the chunks are the inverse of that. Here is the beginning of the illustration I was preparing

Green are packet snips marking 6Lo frames, cyan are packet snips marking gnrc_netif_hdr_ts, bright green are 6Lo frames, bright cyan gnrc_netif_hdr_ts. Red arrows are next pointers, blue arrows are data pointers of the snips.

@miri64 miri64 mentioned this pull request Dec 30, 2018
Merged
@miri64
Copy link
Copy Markdown
Member Author

miri64 commented Dec 30, 2018

@TimoRoth your dump looks an aweful lot like my dump when I don't merge #10679 ;-)

@miri64
Copy link
Copy Markdown
Member Author

miri64 commented Dec 30, 2018

The pointers point to weird positions though :-/. However, with #10679 I can't reproduce #10672 on native anymore :-/

@TimoRoth
Copy link
Copy Markdown
Contributor

TimoRoth commented Dec 30, 2018

The test was done with this and #10679 applied.

@miri64 miri64 mentioned this pull request Dec 30, 2018
Merged
@miri64
Copy link
Copy Markdown
Member Author

miri64 commented Dec 30, 2018

The test was done with this and #10679 applied.

Did you wait until no pinging is happening anymore and gave the GC time to clean? Sorry, I'm asking. But I really can't reproduce this anymore, so this seems to be something ESP-now specific if anything :-/.

@TimoRoth
Copy link
Copy Markdown
Contributor

TimoRoth commented Dec 30, 2018

Yes, when this happened, one of the two boards was already done sending pings entirely. The one where the error showed up had those things in the buffer for way longer than 3 seconds. Even after waiting for several minutes pktbuf still looked like that, and any attempt to ping throws the usual error.

@gschorcht
Copy link
Copy Markdown
Contributor

gschorcht commented Dec 30, 2018 •
edited
Loading

A first test provided some strange results. Test environment were two ESP32 nodes, one is pinging, the other is only answering:

ping6 1000 fe80::32ae:a4ff:fe18:7a3d 1232 0

I have added the following printf's to avoid tampering the test with debug messages:
#### in case of pktbuf: no space left in packet buffer in _pktbus_alloc
~~~~ in case of 6lo rbuf in rbuf_add

At the moment, I'm only able to describe what I can observe:

  • Sometimes, the test can be executed multiple times without any ping timeout when nodes are fresh from reset.
  • Sometimes, only some single ping timeouts occur quite early after the test is started.
  • Once there are ping timeouts, the number of ping timeouts increases with time with single successful pings in between. I can't observe neither pktbuf full nor rbuf full errors, neither on pinging nor on answering node. 
    ping timeout
1240 bytes from fe80::32ae:a4ff:fe18:7a3d: id=84 seq=758 hop limit=64 time = 48.595 ms
ping timeout
ping timeout
ping timeout
1240 bytes from fe80::32ae:a4ff:fe18:7a3d: id=84 seq=762 hop limit=64 time = 42.417 ms
ping timeout
ping timeout
  • Later, I'm getting only ping timeouts either with only pktbuf full or with pktbuf full followed by rbuf full. These errors can occur either on pinging node or on asnwering node. 
    ping timeout
####
ping timeout
ping timeout
ping timeout
ping timeout
ping timeout
####
ping timeout
####
ping timeout
####
ping timeout
ping timeout
ping timeout
  • Sometimes, error: packet buffer full errors occur.
  • Sometimes, single successful pings occur when the problems above happened.
  • During the problems above, I can observe the packets on air interface.
  • When a test is executed after a test which had the problems above, the new test starts directly with these problems from the beginning. Resetting the answering node doesn't help.
  • Important:
    • I always get ping timeout before I get pktbuf full.
    • I always get pktbuf full before I get rbuf full.
  • Mostly, the packet buffer is flushed completely after the tests but sometimes I can observe hanging 6Lo fragments which are definitely not freed after more than 10 seconds, for example on pinging node:
packet buffer: first byte: 0x3ffb84fc, last byte: 0x3ffb9cfc (size: 6144)
  position of last byte used: 6144
~ unused: 0x3ffb84fc (next: 0x3ffb8aa4, size: 1424) ~
=========== chunk   0 (0x3ffb8a8c size:   24) ===========
00000000  D4  90  FB  3F  F4  90  FB  3F  F6  00  00  00  01  00  00  00
00000010  00  00  00  00  08  00  00  00
~ unused: 0x3ffb8aa4 (next: 0x3ffb8fec, size: 1328) ~
=========== chunk   1 (0x3ffb8fd4 size:   24) ===========
00000000  EC  9B  FB  3F  04  9C  FB  3F  F6  00  00  00  01  00  00  00
00000010  00  00  00  00  18  00  00  00
~ unused: 0x3ffb8fec (next: 0x3ffb91ec, size:  264) ~
=========== chunk   2 (0x3ffb90f4 size:  248) ===========
00000000  01  E5  00  04  AF  1E  54  54  54  54  54  54  54  54  54  54
00000010  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000020  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000030  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000040  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000050  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000060  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000070  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000080  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000090  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000A0  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000B0  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000C0  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000D0  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000E0  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000F0  54  54  54  54  54  54  AE  A4
~ unused: 0x3ffb91ec (next: 0x0, size: 2584) ~
=========== chunk   3 (0x3ffb9c04 size:  248) ===========
00000000  01  E5  00  05  20  1E  54  54  54  54  54  54  54  54  54  54
00000010  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000020  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000030  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000040  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000050  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000060  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000070  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000080  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
00000090  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000A0  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000B0  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000C0  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000D0  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000E0  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54  54
000000F0  54  54  54  54  54  54  00  00

@gschorcht
Copy link
Copy Markdown
Contributor

gschorcht commented Dec 30, 2018

In addition to my comment #10680 (comment) an example trace on pinging nodes that shows the order of errors quite good. After the test, I had again two hanging 6Lo fragments.

1240 bytes from fe80::32ae:a4ff:fe18:7a3d: id=83 seq=709 hop limit=64 time = 59.951 ms
1240 bytes from fe80::32ae:a4ff:fe18:7a3d: id=83 seq=710 hop limit=64 time = 54.603 ms
1240 bytes from fe80::32ae:a4ff:fe18:7a3d: id=83 seq=711 hop limit=64 time = 52.192 ms
1240 bytes from fe80::32ae:a4ff:fe18:7a3d: id=83 seq=712 hop limit=64 time = 54.630 ms
1240 bytes from fe80::32ae:a4ff:fe18:7a3d: id=83 seq=713 hop limit=64 time = 61.679 ms
ping timeout
ping timeout
1240 bytes from fe80::32ae:a4ff:fe18:7a3d: id=83 seq=714 hop limit=64 time = 32.615 ms
ping timeout
ping timeout
####
####
~~~~
ping timeout
####
ping timeout
####
ping timeout
####
ping timeout
ping timeout
ping timeout
ping timeout
####
~~~~
ping timeout
####
...
####
~~~~
####
~~~~
ping timeout
ping timeout
1240 bytes from fe80::32ae:a4ff:fe18:7a3d: id=83 seq=900 hop limit=64 time = 43.578 ms
ping timeout
1240 bytes from fe80::32ae:a4ff:fe18:7a3d: id=83 seq=902 hop limit=64 time = 44.196 ms
ping timeout
####
ping timeout
####

@gschorcht
Copy link
Copy Markdown
Contributor

gschorcht commented Dec 30, 2018

In addition to #10680 (comment): I could also observe one hanging 6Lo fragment on answering node

@gschorcht
Copy link
Copy Markdown
Contributor

gschorcht commented Dec 30, 2018 •
edited
Loading

I have investigated a bit more the answering node side when ping timeouts happen on pinging node with some simple printf`s :

  • S - esp_now_send is called in _send
  • + - esp_now_send returned sucessfully
  • R - esp_now_recv_cb callback esp_now_send` is called by WiFi interface
  • - - esp_now_recv_cb callback esp_now_send` is left successful
  • X - esp_now_recv_cb callback esp_now_send` returns with ringbuffer full error

This is how it looks on answering node on a successful ping (six fragments received successful and six fragment sent successful):

R-R-R-R-R-R-S+S+S+S+S+S+

When the ping timeout occurs it looks like the following:

RXRXRXRXRXRXR-R-R-R-R-R-S+S+S+S+S+S+

That is, on each receiption of a fragment, esp_now_recv_cb returns with ringubffer full error.

@miri64
Copy link
Copy Markdown
Member Author

miri64 commented Dec 30, 2018

Yes, when this happened, one of the two boards was already done sending pings entirely. The one where the error showed up had those things in the buffer for way longer than 3 seconds. Even after waiting for several minutes pktbuf still looked like that, and any attempt to ping throws the usual error.

but when the other node is still pinging, the packets you see might just be the packets from the other node still in transit. I'm also seeing those. but when the other node is also done, both packet buffers are empty.

@gschorcht
Copy link
Copy Markdown
Contributor

gschorcht commented Dec 30, 2018

but when the other node is still pinging, the packets you see might just be the packets from the other node still in transit. I'm also seeing those. but when the other node is also done, both packet buffers are empty.

I have observed the same after the test has been finished for at least 10 seconds.

@gschorcht
Copy link
Copy Markdown
Contributor

gschorcht commented Dec 30, 2018 •
edited
Loading

Even more strange, after a test with all different errors (ping timeout, pktbuf full, rbuf full), a single ping with only 52 bytes of data also gives a timeout although packet buffers on both sides are empty. The behaviour on answering node is

R-F-F-

where F stands for fetch with _recv from esp_now_netdev. That is, the echo request is received and fetched but the node doesn't react with echo reply 😕

Maybe, there is something wrong with rbuf.

@miri64
Copy link
Copy Markdown
Member Author

miri64 commented Dec 30, 2018

@gschorcht ping timeouts and full packet buffers may still occur. Remember that we are in a high load scenario, so if the node just doesn't have the recources, it drops the packet. The important thing is, that no memory leaks are formed. I read your analysis and observations later, when I find the time, but here is what I've seen on native: at some point the reassembly buffer (due to lost in transit fragments etc.) runs just full. Since the packets are so huge this fills the packet buffer (packet buffer full, ping timeout on the other node). Due to the high load it never really is able to free those until the ping is done. Only after ping is done the buffers get some time to breath and finally clean after a while.

@gschorcht
Copy link
Copy Markdown
Contributor

gschorcht commented Dec 30, 2018 •
edited
Loading

@miri64

ping timeouts and full packet buffers may still occur.

That's clear, I didn't expect not to get the ping timeouts or full packet buffers. The traffic produced with the ping command is too much.

I just wanted to share what I observed.

The real problems I see are:

Concerning the second problem, I would expect that a single small ping should be possible without any problems when the test with heavy traffic has been finished for more than 3 seconds. But this is obvious not the case.

@miri64
Copy link
Copy Markdown
Member Author

miri64 commented Dec 30, 2018

(sorry I didn't see your answers between my two answers, it only became clear to me that you still observe leaks with those)

Ok. But since I can't reproduce native, this seems to be a different problem from what I am trying to fix here. As I said, I'll read what you wrote, when I find the time, but I rather would go for a new issue, since the steps to reproduce in #10672 are fixed. Also we should merge this, as I suppose it still makes things better, right?

@gschorcht
Copy link
Copy Markdown
Contributor

gschorcht commented Dec 30, 2018 •
edited
Loading

As I said, I'll read what you wrote, when I find the time, but I rather would go for a new issue, since the steps to reproduce in #10672 are fixed.

Yes, of course. I am very grateful for your support. I will try to investigate it further.

@gschorcht gschorcht merged commit 934c422 into RIOT-OS:master Dec 30, 2018
@miri64 miri64 deleted the gnrc_sixlowpan_frag/fix/rbuf-full-release branch December 30, 2018 15:03
@gschorcht
Copy link
Copy Markdown
Contributor

gschorcht commented Dec 30, 2018 •
edited
Loading

Just to share my observation. Precondition:

  • Test with heavy traffic errors executed: ping6 20 fe80::32ae:a4ff:fe18:7a3d 1232 0
  • Packet buffer on both sides is empty!

Send one single small ping:

ping6 1 fe80::32ae:a4ff:fe18:7a3d

On air, I can dump the following WiFi frame

0040   ** ** ** ** dd 3b 18 fe 34 04 01 01 41 60 00 00
0050   00 00 0c 3a 40 fe 80 00 00 00 00 00 00 32 ae a4
0060   ff fe 41 60 f9 fe 80 00 00 00 00 00 00 32 ae a4
0070   ff fe 18 7a 3d 80 00 52 cc 00 54 00 01 54 54 54
0080   54

where dd 3b 18 fe 34 04 01 01 is the ESP-NOW header including the one byte esp_now_header_t.

On answering node this leads to

6lo rfrag: entry 0x3ffbc53c (30:AE:A4:41:60:F9, 30:AE:A4:18:7A:3D, 1280, 18) created
6lo rfrag: add interval (960, 1199) to entry (30:AE:A4:41:60:F9, 30:AE:A4:18:7A:3D, 1280, 18)

Ups? Obviously, old information are used by 6Lo. This might be the reason for our problem. Seems like an rbuf inconsistency.

@miri64
Copy link
Copy Markdown
Member Author

miri64 commented Dec 30, 2018

@gschorcht we should discuss this in a separate issue, otherwise this might get lost.

@gschorcht
Copy link
Copy Markdown
Contributor

gschorcht commented Dec 30, 2018

yes.

This was referenced Jan 3, 2019
Merged
Closed
@aabadie aabadie added this to the Release 2019.01 milestone Jan 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gschorcht gschorcht gschorcht approved these changes

@cgundogan cgundogan Awaiting requested review from cgundogan

Assignees

No one assigned

Labels

Area: network Area: Networking CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors)

Projects

None yet

Milestone

Release 2019.01

Development

Successfully merging this pull request may close these issues.

4 participants

@miri64 @TimoRoth @gschorcht @aabadie