The crawler itself is now totally debugged (had one until the last minute which I kept missing which wouldn’t actually get the individual talent trees for each arena member).
I gather all the data I want to for this project, but have yet to code the statistics part.
My next issue is the execution time of the whole thing – hitting the armory with 410 requests in short progression, with a fopen() and fclose for every single one is horribly inefficient. I raised my script execution time to 5 minutes and I cannot even get all 400 members of a 2v2 ladder that way.
Looks like I have to investigate more into getting cURL to work instead of fopen() to get the armory data, as it is supposed to be a lot more efficient, so a recode is due shortly.
In the next post I’ll dump the data I have now, a lot less than expected but a lot more than what I did manually.
Related posts:
