User talk:Amaurice

From the RuneScape Wiki, the wiki for all things RuneScape
Jump to: navigation, search
2007   2008   2009
2010   2011   2012
2013   2014

Hiscore counts

Could you take a look at how your bot updates the Module:Hiscore counts, specifically the "Overall" count for count_120s? According to this, the number is incorrect. IP83.101.44.209 (talk) 07:39, January 15, 2015 (UTC)

Unfortunately, the Runescape "overall" highscore pages only give enough information get the number of players maxed, not the number of players with everything to 120. It is not possible for the bot to calculate the number by comparing the list of players with 120 in each of the other skills because it would require several thousand highscore page requests to get the lists, and the Jagex DoS protection software limits the average number of highscore page requests to less than 1 per minute per IP address. Amaurice talk 10:06, January 15, 2015 (UTC)

Your bot and the overall counts of 120s and 200mxp's

I see your bot adds the total amount of players who are maxed (16k) to the values for the total amount of players with all 120s and all 200m xp's which are 13 and 11 respectively. Farming-icon.png Salix of Prifddinas (Talk) Prifddinas lodestone icon.png 11:21, February 4, 2015 (UTC)

Oh I see IP83 already brought it to your attention in the post above. Farming-icon.png Salix of Prifddinas (Talk) Prifddinas lodestone icon.png 11:23, February 4, 2015 (UTC)
The amount of players with 200m experience in all skills should be determinable from the hiscores, should it not? It should be the number of players with maximum total level and maximum experience possible; and those should be the first X players of the total-hiscore ranking. Good to have proper numbers, for now, but are we intent on keeping that value up-to-date manually? IP83.101.44.209 (talk) 11:56, February 4, 2015 (UTC)
Atm is the possible to keep track manually, because of the lower numbers atm, but I see this will increase over time, so something automatically would be helpfull. So I'm trying to make something to get those numbers automatically. Farming-icon.png Salix of Prifddinas (Talk) Prifddinas lodestone icon.png 12:30, February 4, 2015 (UTC)


Oh my. I did not realize that was required for the bot. I removed it because I was looking to make the inclusion shorter and wanted the default to always be "<Monster param>/Charm log". If it's possible, I'd like the bot to recognize that, otherwise I'll add the parameter back right away. I had actually given up on converting the data to modules. One reason is, as you said, you no longer play, and have thus been less involved in the wiki. The other is that I saw less benefit than what was happening with Exchange pages. TL;DR is that the modulization of charm logs is actually finished, as I only planned to convert Template:Charm data and leave everything else. Anyways, just tell me if you can update the bot to default to "<Monster param>/Charm log" if no log param is set, otherwise I'll just add them all back in (it won't be hard). MolMan 13:34, February 7, 2015 (UTC)

AmauriceBot tasks

Hi, I'd be willing to have a go at taking over AmauriceBot's tasks, but I had a few questions regarding the current technical implementations:

  • What error/vandalism/validation checks are performed before merging data into charm logs?
  • How are monsters such as Vorago handled (I seem to remember it's multiple charm drops caused some issues)?
  • I saw mention of rate limits on query pages on jagex's servers. What are they specifically in terms of time?
  • What APIs are used for highscores (for the level99s task), or are they pulled from html?
  • Are there any problems/catches to be aware of when running these tasks?
  • What are the switches on hiscore data pages used for (if they're used at all)?

I'm also wondering if we can scrap some of the tasks altogether. The highscores could potentially be pulled from the API using RSHiscores (when some new code gets reviewed and merged), the adventurer's logs could utilise an RSS feed extension, e.g. User:Cqm/Completion, although it's returning some odd errors at the moment. Depending on the charm log checks used, it might be possible to do it all with js given the current error checking already in place. cqm 11:17, 8 Feb 2015 (UTC) (UTC)

A list of users currently using the adventurer's log and highscores templates would be great :)
The charm log stuff could potentially be better implemented as an extension of the existing js form and some abusefilters to tag suspicious log entries possibly using the edit summary as a hook. I'll see if I can get hold of Joey to discuss it further. cqm 21:52, 9 Feb 2015 (UTC) (UTC)
The alog changes should relatively good to go already, although they might need sound additional formatting to bring them in line with current tabular usage. The source code can be found here which I'll copy and alter the callback for. There's some issues with error caching I just sent a bug report in for too.
The highscores however is very much dependent on code review by Wikia. I believe the person who needs to review it is currently working on a number of high priority security tickets so I have no idea how long it might take to get sorted. In the mean time I'll look at how the highscores are currently used to see if the historical portion can be removed.
A question I hit whilst making a highscores scraper: what are the category_type, date and time_filter URL parameters used for/are they required? cqm 16:01, 10 Feb 2015 (UTC) (UTC)
Also, how are the overall counts calculated, especially f2p? I'm struggling to think of a way to extract those with minimal http requests. cqm 00:22, 11 Feb 2015 (UTC) (UTC)
I think the vast majority of the code is written now, just the formatting to be added back into the page is left really.
Going back to the overall counts, I had some thoughts:
  • 120 overall requires checking all players with >2,711,102,342 xp (120 lvl xp * 26). At time of writing this is 169
    • This can be reduced by working out the xp players must have achieved to always have all 120s (5b - 200m + 120 lvl xp). At time of writing this is 11, leaving us with 158 players to lookup
  • 99 overall requires checking all players with >2574 total level and >338,895,206 xp (I didn't check exactly how many this was, but it was 25k-30k iirc)
    • This can be reduced in the same way in a similar way as above by subtracting all the players with >2595 total lvl, bringing it down to around 15k
    • F2P hiscores requires a level between 1655 and 1594 which is ~30k players.
If I did it with html scraping it would take ~1400 requests, which is just shy of 24 hours of non-stop requests (assuming one request/minute) if the rate limit was turned on. As it happens, it seems to be deactivated at the moment. Have you any ideas on how to reduce the number of requests further? cqm 02:21, 15 Feb 2015 (UTC) (UTC)
It's more or less written, but I felt it was a better idea to write something more extensible that I could add future tasks to. You can see it so far here. However, my main concern is getting the bot hosted somewhere, unfortunately I don't have the option of leaving my computer on for large lengths of time at home. I seem to remember TyBot used EC2, which is free for the first year so I'll likely try something with that. cqm 11:10, 21 Feb 2015 (UTC) (UTC)
I thought I'd let you know that the bot had it's first full trial run today and was almost completely working aside from a minor error I overlooked. I'm still working on error handling, specifically logging tracebacks and emailing unrecoverable errors to me if/when they occur, and automation. cqm 23:29, 2 Mar 2015 (UTC) (UTC)
I've started a second run this morning, which seems to be going much faster than the 3 hours it took yesterday. How long did the hiscores task normally take you? I've also started an approval thread on the YG to formalise the changeover :) cqm 08:39, 3 Mar 2015 (UTC) (UTC)

Charm Log

Hi there, I'm having an issue submitting a charm log for Legio Quartus both ways. I was doing an ascension task and decided to do some Legiones and brought 100 Quartus keys. 

From these keys I got the amount of charms:

  • 96 Gold Charms
  • 72 Green Charms
  • 282 Crimson Charms
  • 114 Blue Charms

When I tried to submit it, the wiki said it was an "attempted" vandalism attempt and refused to upload it. I was wondering if you are able to fix this issue for me, as some people redirected me to you since your bot seems to be handling all of the charm log work. Funds (talk) 02:52, February 13, 2015 (UTC)

Can you check Black Dragon's charm log? Some normal entries got marked as vandalism. It's not impossible to get 0 Crimson charms from 61 kills, and the 2000-kill entry looks perfectly achievable to me. Undamaged (talk) 09:31, April 17, 2015 (UTC)

The charm submission scripts (including vandalism submission checks) are maintained by Joeytje50 (talk).


I'm pretty sure Cam is working on this atm. I thought you of all people would know :p MolMan 15:15, April 17, 2015 (UTC)

The script isn't massively far off, I've done a few more tweaks tonight. Major things left to do are abusefilters, add an API so I can quickly test submissions without actually submitting anything, sort out how to handle Vorago and Araxxor (Waterfiends should be handled already, although it needs testing to be sure) and some user interaction stuff (such as confirm the submission).
I looked over your notes on the current vandalism checks to see how they'd best fit into the script and realised you're working on percentages, while the script works on standard deviations. There's some extra tweaks I'm looking to make regarding sensitivity and kill counts (AmauriceBot uses lower kill counts), but have you any thoughts on if percentages are inferior/superior to standard deviations? cqm 23:33, 3 May 2015 (UTC) (UTC)
I think it's finished except for testing. All I need to do is test various submissions to see if they pass it fail and make sure the filters are doing their job. Unfortunately, I'm away the rest of the week, but I should be able to get that done next week.
Sorry for all this taking quite so long, and thanks for all the help you've given in the changeover :) cqm 08:14, 6 May 2015 (UTC) (UTC)