While reading through contrib/seeds/makeseeds.py I noticed parseline() can crash on a malformed line instead of skipping it.
The function checks that a line has enough columns:
if len(sline) < 11:
# line too short to be valid, skip it.
return None
but further down it does:
agent = sline[11][1:-1]
sline[11] is the 12th column, so a valid line actually needs 12 fields, not 11. A line with exactly 11 fields gets past the check and then throws IndexError. 11 is the highest index the function uses, so the check should be < 12.
Quick way to see it:
cd contrib/seeds
python -c "import makeseeds; makeseeds.parseline('1.2.3.4:8333 1 1700000000 100.00% 100.00% 100.00% 100.00% 100.00% 910001 000000000000040d 70016')"
9 and 10 fields return None like they should, 12 fields parse fine, only 11 crashes.
It takes down the whole run, because main() calls it in a list comprehension with no try/except:
Loading asmap database "asmap.dat"...Done.
Loading and parsing DNS seeds...Traceback (most recent call last):
File "contrib/seeds/makeseeds.py", line 270, in <module>
main()
File "contrib/seeds/makeseeds.py", line 219, in main
ips = [parseline(line) for line in lines]
^^^^^^^^^^^^^^^
File "contrib/seeds/makeseeds.py", line 118, in parseline
agent = sline[11][1:-1]
~~~~~^^^^
IndexError: list index out of range
Annoying because it happens after the asmap database is already loaded, so that work is wasted too.
Expected: an 11 field line should return None and get skipped, same as 9 and 10.
This isn't only theoretical. README.md builds seeds_main.txt by appending one crawler's output onto another's:
curl https://21.ninja/seeds.txt.gz | gzip -dc > seeds_main.txt
curl https://mainnet.achownodes.xyz/seeds.txt.gz | gzip -dc >> seeds_main.txt
so the file is a mix of two independently maintained formats, and a truncated download would leave a short last line as well. That's the case the check is there for.
There's no test covering makeseeds, so this isn't caught anywhere. Happy to open a PR with the fix and a test for it.
Seen on master@dc0395c5.
