Skip to content

Conversation

@fjahr
Copy link
Collaborator

@fjahr fjahr commented Jan 7, 2024

This is a demo PR adding the asmap.dat result for the coordinated launch from #4 from the epoch 1704463200. We had a match from 4 out of 5 participants in the coordinated launch.

This particular case required an additional cleanup step to account for the current configuration/limitation of the compression algorithm. It allows ASNs up to 33521664, even though all 4 byte numbers are technically allowed (aside from bogon ranges). The result contained a single offending line: 2606:9dc0:6666::/48 AS4199999899. This line is removed as shown below. Going forward I will add some flag to kartograf to do this by default since it's not the first time we have seen this issue and each time it looked the entry was not an actual route but a leaked internal mapping/test.

For the following to work you woull also need to check out: bitcoin/bitcoin#28793

$ shasum -a 256 1704463200_final_result.txt
4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b  1704463200_final_result.txt

$ wc -l 1704463200_final_result.txt
 1256210 1704463200_final_result.txt

$ sed '/2606:9dc0:6666::\/48 AS4199999899/d' 1704463200_final_result.txt > 1704463200_final_result_edit.txt

$ wc -l 1704463200_final_result_edit.txt
 1256209 1704463200_final_result_edit.txt

$ python3 contrib/asmap/asmap-tool.py encode 1704463200_final_result_edit.txt 1704463200_asmap.dat

$ shasum -a 256 1704463200_asmap.dat
150a8a4c14b81da331394d9a481646470d6802c3b113b345923d1e6e9b305b4b  1704463200_asmap.dat

@Emzy
Copy link

Emzy commented Jan 8, 2024

Only difference seems to be the final asmap.dat file.

$ shasum -a 256 1704463200/final_result.txt
4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b  1704463200/final_result.txt
$ wc -l 1704463200/final_result.txt
1256210 1704463200/final_result.txt
$ sed '/2606:9dc0:6666::\/48 AS4199999899/d' 1704463200/final_result.txt > 1704463200/final_result_edit.txt
$ wc -l 1704463200/final_result_edit.txt
1256209 1704463200/final_result_edit.txt
$ python3 ~/bitcoin/contrib/asmap/asmap-tool.py encode 1704463200/final_result_edit.txt 1704463200/asmap.dat
$ shasum -a 256 1704463200/asmap.dat
7779317bbd97833e807d453a7fc9bbd73df1d6a9de7dd5c467e573b7718cf1c1  1704463200/asmap.dat

$ ls -l 1704463200/asmap.dat
-rw-r--r-- 1 emzy emzy 1678365 Jan  8 13:55 1704463200/asmap.dat

@0xB10C
Copy link

0xB10C commented Jan 8, 2024

Seems like the asmap.py tool doesn't create a deterministic asmap.dat.

$ shasum -a 256 final_result_0xb10c.txt
4aba172195b8dc375c2bb1b39c3508f48c212cedc02983a7dc794a373820653b  final_result_0xb10c.txt
$ sed '/2606:9dc0:6666::\/48 AS4199999899/d' final_result_0xb10c.txt > final_result_edit.txt
$ wc -l final_result_edit.txt
1256209 final_result_edit.txt
$ python3 asmap-tool.py encode final_result_edit.txt asmap.dat
$ shasum -a 256 asmap.dat
8706a208e84339b39f42fb9407e6a3260391b339cbfa203b657f10c5dfcd8d6b  asmap.dat

@0xB10C
Copy link

0xB10C commented Jan 8, 2024

Hm, running it twice on my machine produces the same output asmap.dat.

Can @Emzy or @fjahr, can you upload your file? (you'll have to zip it for github to accept it)

Here's my asmap.dat:
asmap.dat.gz

@Emzy
Copy link

Emzy commented Jan 8, 2024

Hm, running it twice on my machine produces the same output asmap.dat.

I also get the same after a rerun.

Here my asmap.dat:
asmap.dat.gz

@0xB10C
Copy link

0xB10C commented Jan 8, 2024

The decoded files (asmap-tool.py decode asmap.dat) match again.

@Sjors
Copy link

Sjors commented Jan 9, 2024

Mmm, I got 328c7700ce980e5114047c0757a0159532c5ca51fd46fc62ec9da20e3d951177 on Ubuntu (tried twice), e6b96412647f2f34380e99ed03f8c3ef9eb2f094dcda466db0a1c29f261772c9 on Intel macOS 14.2.1 (tried twice).

Hash of 1704463200_final_result_edit.txt: cedeac924c949895b11ea0245aaf3cfa08ca3838fa8d96580788c382d65c66ea (both on Linux and Mac, so the sed command is probably not the issue)

Sounds like it's putting something machine specific in?

@fjahr
Copy link
Collaborator Author

fjahr commented Jan 13, 2024

The issue with asmap-tool should be fixed now, please check out the latest version of the branch at bitcoin/bitcoin#28793 and see if we get the same result. I was able to replicate it by using different python versions but didn't test it on multiple platforms yet.

The hash of my file is now 45b87814c47b5ea056c2e47fcc08a75afbb535c8bfb80d734bb364cddb432e73.

@Emzy
Copy link

Emzy commented Jan 13, 2024

Works. Now I got the same asmap.dat file.

~/kartograf/out$ python3 ~/bitcoin/contrib/asmap/asmap-tool.py encode 1704463200/final_result_edit.txt 1704463200/asmap.dat
~/kartograf/out$ shasum -a 256 1704463200/asmap.dat
45b87814c47b5ea056c2e47fcc08a75afbb535c8bfb80d734bb364cddb432e73  1704463200/asmap.dat

@Sjors
Copy link

Sjors commented Jan 18, 2024

I get the same hash now, hooray!

@fjahr
Copy link
Collaborator Author

fjahr commented Jan 24, 2024

Alright, for the first run two ACKs are good enough for me :)

Please join the next run in ~24h if possible, which hopefully should run a lot cleaner so we can use it as a demo: #7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants