Pcapdnet: monitor mode + full support without dnet + cleanup by gpotter2 · Pull Request #1350 · secdev/scapy

gpotter2 · 2018-04-17T10:45:45Z

This PR:

Removes tons of duplicated code, which is used while receiving packets. To do so, we introduce a recv_raw function, which receives the packet, calculates its TimeStamp and returns the info required to build it. It will then be built in the recv function, which is now common to all SuperSockets.
Updates API calls for pypcap, libpcap & pcapy
Adds monitor mode support on pypcap and libpcap
Makes conf.use_pcap useable without conf.use_dnet
Because the pcap handlers can now send packet, remove the now useless "PCAP with DNET" mode. DNET is deprecated, but can still be used as a way to get system data

codecov-io · 2018-04-17T11:51:10Z

Codecov Report

Merging #1350 into master will increase coverage by 0.06%.
The diff coverage is 53.72%.

@@            Coverage Diff             @@
##           master    #1350      +/-   ##
==========================================
+ Coverage   85.07%   85.14%   +0.06%     
==========================================
  Files         174      174              
  Lines       40299    40102     -197     
==========================================
- Hits        34285    34144     -141     
+ Misses       6014     5958      -56

Impacted Files	Coverage Δ
scapy/sendrecv.py	`78.57% <100%> (+0.42%)`	⬆️
scapy/arch/pcapdnet.py	`66.05% <47.29%> (-3.67%)`	⬇️
scapy/supersocket.py	`72.65% <53.33%> (-0.95%)`	⬇️
scapy/arch/linux.py	`77.58% <89.47%> (+2.74%)`	⬆️
scapy/layers/ntp.py	`91.51% <0%> (-0.54%)`	⬇️
scapy/plist.py	`82.05% <0%> (+0.32%)`	⬆️

guedou

Nice PR. Please see my comments.

guedou · 2018-04-18T07:56:33Z

scapy/arch/linux.py

Please add a docstring.

guedou · 2018-04-18T07:56:44Z

scapy/arch/linux.py

Please add a docstring.

guedou · 2018-04-18T07:56:48Z

scapy/arch/linux.py

Please add a docstring.

guedou · 2018-04-18T07:57:02Z

scapy/arch/linux.py

Are these comments needed?

I have no idea what the "direction" represents, so as the PR removes this feature (which is only available and used in this function), I left the comments.

Otherwise, we could simply remove this "recv" function, as it only calls its super one.

I means that the packet was sent locally, see http://man7.org/linux/man-pages/man7/packet.7.html We don't use at al.

guedou · 2018-04-18T07:57:32Z

scapy/arch/pcapdnet.py

Could you make this line shorter?

guedou · 2018-04-18T07:57:43Z

scapy/arch/pcapdnet.py

Please add a docstring.

guedou · 2018-04-18T08:00:11Z

Given the state of current libpcap Python bindings, I think that we can clean the code even further and only keep pcapy support.

I would like to have #1315 merged before applying this PR.

gpotter2 · 2018-04-18T15:17:58Z

@guedou Problem with pcapy is that it does not support monitor mode :/
python-pypcap have a PR for it
python-libpcap already supports it (e.g.) + is maintained by Adam Karpierz (what a small world)

On the other hand, dnet and dumbnet do not seem to have a real main repository or any official mainteners. I've seen tons of forks, nothing official and everything seems outdated. If we had to delete something, I would definatly chose those

hellais · 2018-04-18T15:45:48Z

On the other hand, dnet and dumbnet do not seem to have a real main repository or any official mainteners. I've seen tons of forks, nothing official and everything seems outdated. If we had to delete something, I would definatly chose those

Yeah the current situation of libdnet bindings (or even dnet itself for that matter) is quite suboptimal as well as a lot of the other python based low level network toolkits.

I have been trying to maintain it here: https://github.com/pynetwork/pydumbnet, but that has in practice really meant just unbreaking pip installation in some cases and nothing much else.

pypcap is in a similar situation and it would be great if we could get some other people to help out with that.
The main struggle is just reviewing all the various PRs and ensuring they all work and don't break things.

gpotter2 · 2018-04-18T16:15:52Z

This PR "theoretically" adds support for monitor mode using python-pypcap and python-libpcap.
Please note that pypcap calls are based on the, currently un-merged, https://github.com/pynetwork/pypcap/pull/67/files

If someone has a computer that supports monitor mode (which I currently don't), it would require for that very nice person to:

install the pypcap version from https://github.com/pynetwork/pypcap/pull/67/files
test this PR of scapy using tshark(monitor=True) for instance

guedou · 2018-04-21T10:09:32Z

scapy/arch/pcapdnet.py

Can you specify the required version?

The fact is, the version supporting that is not merged yet.
https://github.com/pynetwork/pypcap/pull/67/files
They are awaiting our confirmation that it works to merge it, but I can’t test it as I don’t have access anymore to a machine supporting monitor mode

Understood.

guedou · 2018-04-21T10:10:06Z

scapy/arch/pcapdnet.py

Same here, could you specify the version?

Done (they published 1.2.1)

guedou · 2018-04-21T10:11:13Z

Awesome. See my two new comments. After rebasing, this PR can be merged quickly.

guedou · 2018-04-21T14:34:34Z

A file is conflicting.

gpotter2 · 2018-04-21T14:36:33Z

@guedou @p-l- @speakinghedge Sorry for bothering.. Has one of you access to a computer with the WiFi card supporting monitor mode ? If so, would it be possible for you to test this PR using:
https://github.com/pynetwork/pypcap/pull/67/files

I will try to find a computer with monitor mode supported, but it might take a while

guedou · 2018-04-22T08:20:56Z

@gpotter2 I can't test it either:

# tcpdump -i wlan0 --monitor-mode
tcpdump: wlan0: That device doesn't support monitor mode

speakinghedge · 2018-04-22T18:44:10Z

Will do.

speakinghedge · 2018-04-23T20:26:21Z

Just to make sure that I'm on the right path: I set conf.use_pcap = True so L2pcapListenSocket is used...

If the above statement is true -> __init__ of L2pcapListenSocket in pcapdnet.py (line 498) is missing the monitor attribute - same applies to open_pcap call in line 508.

After adding the monitor flag I got the expected output:

...
   36	RadioTap / 802.11 Management 8 90:5c:44:a1:ad:f8 > ff:ff:ff:ff:ff:ff / Dot11Beacon / SSID='Pouerhausen' / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt
   37	RadioTap / 802.11 Control 9 84:38:38:e5:a0:4b > 90:5c:44:a1:ad:f8 / Raw
   38	RadioTap / 802.11 Management 8 b0:48:7a:xx:xx:xx > ff:ff:ff:ff:ff:ff / Dot11Beacon / SSID='slartibart' / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt
   39	RadioTap / 802.11 Management 8 b0:48:7a::xx:xx:xx > ff:ff:ff:ff:ff:ff / Dot11Beacon / SSID='slartibart' / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt
   40	RadioTap / 802.11 Management 8 b0:48:7a::xx:xx:xx > ff:ff:ff:ff:ff:ff / Dot11Beacon / SSID='slartibart' / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt
   41	RadioTap / 802.11 Data 4 84:38:38:e5:a0:4b > 90:5c:44:a1:ad:f8
   42	RadioTap / 802.11 Data 4 84:38:38:e5:a0:4b > 90:5c:44:a1:ad:f8
   43	RadioTap / 802.11 Data 4 84:38:38:e5:a0:4b > 90:5c:44:a1:ad:f8
   44	RadioTap / 802.11 Data 4 84:38:38:e5:a0:4b > 90:5c:44:a1:ad:f8
   45	RadioTap / 802.11 Management 8 b0:48:7a:xx:xx:xx > ff:ff:ff:ff:ff:ff / Dot11Beacon / SSID='slartibart' / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt / Dot11Elt
...

gpotter2 · 2018-04-23T22:14:47Z

@speakinghedge Thank you very much !!!

gpotter2 · 2018-04-23T22:27:24Z

@guedou I won't remove DNET on this PR, as its deprecation needs to be warned during at least a version, but it really is the next thing to do

guedou · 2018-04-24T08:21:08Z

It looks that some tests are failing because they hang.

guedou · 2018-04-27T17:39:46Z

I restarted the failing tests but the hangs looks related to this PR.

gpotter2 · 2018-04-27T17:53:41Z

>>> p = PipeEngine()
>>> 
>>> s = SniffSource()
>>> d1 = Drain(name="d1")
>>> c = QueueSink(name="c")
>>> s > d1 > c
<c [d1>#]>
>>> 
>>> p.add(s)
>>> p.start()
Exception RuntimeError: RuntimeError('maximum recursion depth exceeded in cmp',) in Exception RuntimeError: RuntimeError('maximum recursion depth exceeded in cmp',) in Exception RuntimeError: RuntimeError('maximum recursion depth exceeded in cmp',) in Exception RuntimeError: RuntimeError('maximum recursion depth exceeded in cmp',) in Exception RuntimeError: RuntimeError('maximum recursion depth exceeded in cmp',) in Exception RuntimeError: RuntimeError('maximum

Will have a look

gpotter2 · 2018-04-28T16:15:55Z

@guedou Hi, news about this PR:

I have added the setnonblock and send functions support for pcapy, pypcap and libpcap (it used to only support reading). This makes pcap useable without dnet, and allow us to remove the "dnet with pcap mode".
Updated first post of this PR (description)

The next PR of that kind (will wait a bit), will fully remove DNET

gpotter2 · 2018-04-28T18:07:33Z

(please ignore the failing test, passing on my repo: https://travis-ci.org/gpotter2/scapy/builds/372462884)

guedou · 2018-05-02T12:59:59Z

scapy/arch/pcapdnet.py

+# DEPRECATED

 if conf.use_dnet:
+    warning("Dnet usage with scapy is deprecated, and will be removed in a future version.")


The correct name is dnet.

guedou · 2018-05-02T13:00:38Z

@p-l- could you also have a look?

p-l- · 2018-05-06T13:58:24Z

@gpotter2 this is great work! Thanks!

guedou requested changes Apr 18, 2018

View reviewed changes

gpotter2 mentioned this pull request Apr 18, 2018

Support monitor mode helpsystems/pcapy#45

Closed

guedou reviewed Apr 21, 2018

View reviewed changes

guedou approved these changes Apr 21, 2018

View reviewed changes

gpotter2 mentioned this pull request Apr 23, 2018

Rfmon support rebased pynetwork/pypcap#67

Merged

gpotter2 closed this Apr 25, 2018

gpotter2 reopened this Apr 25, 2018

gpotter2 added 3 commits April 27, 2018 18:24

Introduce recv_raw: remove duplicated code

9834817

Flake8 fixes

77f52ca

Docstrings

c979d94

gpotter2 changed the title ~~Clean Pcapdnet & Linux : remove duplicated code~~ Pcapdnet: full support without dnet + cleanup Apr 28, 2018

Update pypcap & libpcap API calls

a0b9d02

Deprecate DNET + remove DNET&PCAP mode

7238607

gpotter2 changed the title ~~Pcapdnet: full support without dnet + cleanup~~ Pcapdnet: monitor mode + full support without dnet + cleanup Apr 28, 2018

guedou reviewed May 2, 2018

View reviewed changes

guedou approved these changes May 2, 2018

View reviewed changes

Typo

116c032

gpotter2 mentioned this pull request May 4, 2018

Implementation of ISO-TP sockets [ready for merge] #1363

Merged

p-l- merged commit 9a2c191 into secdev:master May 6, 2018

gpotter2 deleted the pcapdnetremaster branch May 6, 2018 14:41

gpotter2 mentioned this pull request Jun 6, 2019

Drop pcapy-like integrations in favor of winpcapy.py (no acitivity) #2057

Merged

Uh oh!

Conversation

gpotter2 commented Apr 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-io commented Apr 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

guedou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guedou commented Apr 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpotter2 commented Apr 18, 2018

Uh oh!

hellais commented Apr 18, 2018

Uh oh!

gpotter2 commented Apr 18, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guedou commented Apr 21, 2018

Uh oh!

guedou commented Apr 21, 2018

Uh oh!

gpotter2 commented Apr 21, 2018

Uh oh!

guedou commented Apr 22, 2018

Uh oh!

speakinghedge commented Apr 22, 2018

Uh oh!

speakinghedge commented Apr 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpotter2 commented Apr 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpotter2 commented Apr 23, 2018

Uh oh!

guedou commented Apr 24, 2018

Uh oh!

guedou commented Apr 27, 2018

Uh oh!

gpotter2 commented Apr 27, 2018

Uh oh!

gpotter2 commented Apr 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

gpotter2 commented Apr 17, 2018 •

edited

Loading

codecov-io commented Apr 17, 2018 •

edited

Loading

guedou commented Apr 18, 2018 •

edited

Loading

speakinghedge commented Apr 23, 2018 •

edited

Loading

gpotter2 commented Apr 23, 2018 •

edited

Loading

gpotter2 commented Apr 28, 2018 •

edited

Loading