Forum:Automatic Hiscore Update

From the RuneScape Wiki, the wiki for all things RuneScape
Jump to: navigation, search
Forums: Yew Grove > Automatic Hiscore Update
This page or section is an archive.
Please do not edit the contents of this page.
This thread was archived on 7 August 2009 by Azaz129.

Hello. I was wondering whether we have a bot that can perform automatic updates of a user's hiscore data page, used for Template:ParseHiscoreData. If not, I propose to create a bot to automatically perform the updates for any user who wishes to use it. My idea is as follows:

  1. The user places a template on their page and provides the following parameters to it:
    1. player - The name of the RuneScape player account.
    2. page - The subpage where the hiscores are to be stored, such as User:Quarenon/HiscoreData
    3. interval - The update interval, say in hours, at which the bot should try to update the page. The bot will use a fixed interval of 24 hours (minimum).
  2. The bot occasionally checks for users with this template (a category will be used for this), and fetches the hiscores for those users, at the rate specified in the interval parameter, and then posts an update to the given page.

The template can be placed anywhere in the user's subpage area. However, the bot will enforce that the user specified in the page parameter be the same as the user who placed the template, preventing a user from making the bot update a different user's subpages. The interval parameter is also done on a best-effort basis, it can update slower if needed to throttle the rate at which it edits.

I will be working on the code for this, which will be made available under GPL, if approved. --Quarenon  Talk 21:05, 18 June 2009 (UTC)


Support as nom. --Quarenon  Talk 22:50, 18 June 2009 (UTC)

Awesome. I don't think we already have a bot to do this, but I always thought there was (or could be) a need/want for it. A quick question: I use a #switch: function in mine, would it be able to cope with that? If not, don't worry, I'll create another subpage for it. Quest.png Gaz Lloyd 7:^]Events!99s 21:14, 18 June 2009 (UTC)

I think the simplest/universal way would be to have the hiscore data alone in a separate subpage, which should work for your case as well as most cases I can think of. If anyone has an objection to this though let me know and I can see if there's a better way to do it. --Quarenon  Talk 22:37, 18 June 2009 (UTC)
I thought as much. No matter though. I support, obviously. Quest.png Gaz Lloyd 7:^]Events!99s 13:19, 19 June 2009 (UTC)

Support - this would be very useful for anyone that wants to keep their levels up to date. Andrew talk 22:42, 18 June 2009 (UTC)

Support - I've been wanting to suggest this for a LONG time, but didn't because I don't know how to make it, so why bother?

InstantWinstonDragon 2h sword old.pngold edits | new edits

23:07, 18 June 2009 (UTC)

Support - That would be great if it can be done..then I can watch from work to see when my strength rank drops to 1.5M! Could a template also be created that could compare different users that use the template, similar to what's on the official RS website? Comparing could be done similar to how we practice coding in a sandbox, with no saves made. Vadanea 04:16, 19 June 2009 (UTC)

Comment - Sure, with the existance of Template:ParseHiscoreData, I don't see why it wouldn't be possible. Someone just has to take the time to code it! --Quarenon  Talk 11:41, 19 June 2009 (UTC)
I couldn't resist. Basic code is in my second sandbox, feel free to test it out - that is, put {{User:Gaz Lloyd/Sandbox 2|username1|username2|data for username1|data for username2}} anywhere (if you wish to save it, use your userpage). The data should either be the string of numbers from hiscores lite, or (more preferably, as this is what the bot is for...) a transclusion of the subpages where the data is held (e.g. if Example has his data saved in User:Example/stats he would put {{User:Example/stats}} for username1's data).
Probably could be improved and moved to the Template: namespace, but hey, its a draft. Quest.png Gaz Lloyd 7:^]Events!99s 21:45, 19 June 2009 (UTC)
Neat! I tried it at my Sandbox, it looks good except the unranked stats have weird effects on the table (using Google Chrome). Very cool regardless! --Quarenon  Talk 02:28, 20 June 2009 (UTC)
Ah, yeah. The code output new rows randomly, probably due to the -1s... fixed anyway, it now says "Not in Hiscores" for unranked skills. Thanks for that, I tested wth users which had all ranked skills. Quest.png Gaz Lloyd 7:^]Events!99s 20:58, 20 June 2009 (UTC)
Looks awesome now, thanks! --Quarenon  Talk 21:03, 20 June 2009 (UTC)

Support - If you're willing to script it, I'm all for the implementation of the bot. It would be rather helpful. Dragon medium helm! Whaddaya know?Chiafriend12Better than rune!I have 12 friends. 04:52, 19 June 2009 (UTC)

Support - I can't create something that complicated myself! Zig 06:22, 19 June 2009 (UTC)

Support - Per all. Prayer.png Jedi Talk HS Log Tracker Summoning.png 08:12, 19 June 2009 (UTC)

Comment - While I do support this idea as it can be a real pain sometimes to update my stats I thought that all calls to the RS website had to be done on demand or limited to the number of auto calls a day. This means you couldnt really have it so it updated every few hours, you would only be able to do one auto update a day. Perhaps there could be a button or something to click which runs the bot on your scores to update it seperate to the auto updating? --Gold ore.png Mercifull UK serv.svg (Talk) 08:36, 19 June 2009 (UTC)

Comment - I looked on the RuneScape website to find a specific mention of the number of times the hiscores may be accessed, but I only found the [ instructions] on where to find it. If Jagex made a statement about that and someone can produce a link to it I would certainly honor it, otherwise I intend to ensure that the overall hiscore update rate stays reasonable. As mentioned above, the interval parameter given in the template is done only on a best-effort basis. The bot will throttle itself if it is trying to make too many requests to Jagex or too many edits to Wikia at the same time, which means the actual interval might be higher than what is requested in the template. I was also careful in choosing the units of hours for that value, in addition to making sure that the value is an integer whole number, to make sure someone doesn't try to get their stats updated unreasonably fast. --Quarenon  Talk 11:53, 19 June 2009 (UTC)
I found where it says it can't be done. "It must not automatically make repeated page/content requests from our website or otherwise make excessively bandwidth-intensive requests from our servers (each page/content request should only be in direct response to a user request each time)." There's really no way around this, so this cannot pass. Christine 20:23, 22 June 2009 (UTC)
I think we need to carefully look at this statement and ensure we know the reasons Jagex put it in place. First, I believe that the intention of this rule is to prevent such things as Denial-of-Service attacks, where people knowingly make a high number of repeated requests to a page to try and slow or stop RuneScape's web servers. In support of this, note the fact that Jagex has taken no known action against web crawlers from the likes of Google and Yahoo which make automated and repeated requests to a vast number of RuneScape's pages. This is fine because a) the crawlers provide a useful function of bringing users to RuneScape and b) the crawlers generally run in a way that is not excessive in bandwidth or requests. Secondly, I call upon the established precedence on this wiki, which supports this interpretation of the rule, and that is of the existence of User:SmackBot and User:GEBot, which clearly perform automated, regular requests to RuneScape to retrieve GE price information. I am all against excessive bandwidth consumption and requests because denial-of-service attacks are the bane of anyone who runs a web-facing server, and that is why I have embedded a variety of throttling mechanisms and load control in the bot. And if any admin believes that the bot is operating outside of acceptable levels, I have conveniently placed a block link on the bot's user page. --Quarenon  Talk 22:20, 22 June 2009 (UTC)
Just because we've done it before, doesn't mean it's right. It specifically says we can't do that. Christine 03:39, 23 June 2009 (UTC)
Are you saying that this violates the clause of "excessive bandwidth-intensive requests" or that "each page/content request should only be in direct response to a user request each time"? If the latter, would you then support a link/button that a user must click on to perform an update to the hiscores page (this would then be a direct response to a user request)? Are you also going to support someone opening a YG topic calling into question the legitimacy of User:SmackBot and User:GEBot? I have totally suspended operation of the bot for the moment as a result of your original comment, as I would want to ensure that we comply with Jagex rules. --Quarenon  Talk 04:22, 23 June 2009 (UTC)
"Each page/content request should only be in direct response to a user request each time." I think that it needs at least a button, and I also think that 6 hours is excessive. 24 at the minimum imo... Anyways, I was under the impression that the GE bot responded to a human running it, I know that I read that somewhere in the original forum, and I have no idea wtf SmackBot is. Christine 11:48, 23 June 2009 (UTC)
I can see if we can adjust things to require a button press, and also the update time is being discussed below, so it seems like raising the minimum up to around 24 hours is likely; although, if we require a button press, I'm not sure if it will be as easy to limit the update rate. The only issue I raise with this Jagex rule is that it's not enforced/enforceable by Jagex in its literal form (forbidding any and all automated requests), otherwise you would see Jagex raising legal issue with automatic web crawlers and browser page prefetchers, and that's definitely not happening any time soon. Therefore I am going to seek Jagex's clarification on this. --Quarenon  Talk 13:26, 23 June 2009 (UTC)
The only comment I can come up with for that is that web crawlers/search engines are making Jagex more money, what we're doing would not :P Christine 16:40, 23 June 2009 (UTC)
You are saying that if I am able to pay Jagex some amount of money or provide them with enough additional traffic, I get a free pass to violate a rule. Surely there is a fallacy in the rule as written, or it should be interpreted in the way I've given above, which I believe is sensible. --Quarenon  Talk 17:05, 23 June 2009 (UTC)
Surely this is allowed on a basis of at leasst once every 24 hours. Dynamic signatures are available on most if not all of the Jagex approved fansites. These update userstats automaticly too. As do sites like RScript and Runemonkey. These are all allowed by Jagex. Perhaps the bot could be changed to update every 24 hours at a user specified time in the template with the option to also update on command via a button or link. --Gold ore.png Mercifull UK serv.svg (Talk) 12:50, 26 June 2009 (UTC)

I have found this on the official Runescape website.

We have a feature on the Hiscores that enables fansites to access data from the Hiscores much more quickly. This is most useful for use as a 'signature', so you can show the world how your skills are going.
To access a 'light' version of your character's high scores simply go to the following address:
Simply replace 'YourUsername' with your character's name and this will provide you with a string of numbers organised into groups. The numbers at the front are grouped in threes, and are your rank, level and experience in each skill, in the same order that they appear on the Hiscores. The remaining numbers are grouped in twos, and are your rank and score in each minigame, again in the same order as on the Hiscores. If you are unranked in a skill or minigame, your rank will be marked -1.
If you run a fansite and wish to enable players to publish their hiscores, it should be a simple matter to format this string of numbers, as there is no HTML code attached to the output. In the script of the output, each rank, level and experience group is separated by a new line, and each field is separated by a comma (without spaces).
Note that if you do not specify a user name, the user has no rankings or the user does not exist, you will receive a standard 404 error.

Nothing in this passage of text states that there is a limit to the number of queries you can do and even suggest dynamic signatures as an example of usage which work by updating automaticly (like a bot) not via a person clicking an update link. I see no reason now why the bot cannot update at least once every 24 hours based on this post. --Gold ore.png Mercifull UK serv.svg (Talk) 13:44, 29 June 2009 (UTC)

I agree that there is some lack of clarity on this rule. I made a RuneScape forum post seeking some assistance. This might sound like RS:AEAE#RuneScape_authorities but I feel that there is a wording flaw in the rules quotes in this thread. I will keep the bot inactive for now until an official statement comes in that allows automated requests; preferably in the form of an actual change in the rules, and then if necessary I can call for a new discussion on the bot proposal. --Quarenon  Talk 03:16, 30 June 2009 (UTC)

Support per all bad_fetustalk 13:09, 19 June 2009 (UTC)

Support That would make everything so much easier :O FredeTalk 13:50, 19 June 2009 (UTC)

Support - Same as Frede. Makes things a hole lot better. Runecrafting.gif Mo 55 55 Talk|Sign 14:20, 19 June 2009 (UTC)

Comment - As a status update, the code for this is largely complete and should be ready to roll in a day or so. I would like the admins to approve the creation of a bot account QBot for performing the updates, a template Template:Hiscore autoupdate for providing the data to the bot, and a category Category:Hiscore autoupdate to organize the pages containing the template. --Quarenon  Talk 14:39, 19 June 2009 (UTC)

The source code is now available. The bot has been tested (not thoroughly) at a test wiki I have set up. The files are hiscores.php (main script) and wikibot.php (the bot framework). This bot requires PHP 5 (CLI) with cURL support; in Linux, the bot can be launched using: $ php hiscores.php >hiscores.log & --Quarenon  Talk 03:10, 20 June 2009 (UTC)

Comment - Would anyone like to participate in the testing? The template to place is Template:Hiscore autoupdate. The template is being used on my user page if you would like an example. --Quarenon  Talk 23:45, 21 June 2009 (UTC)

Strong Support - I'd love to have some script that could automatically update my high scores for me. It's so annoying to have to update it myself every few days. Don't those and runehq calculators have a function that looks up your skill experience and level if you're ranked? We need something similar in our userpages. What a great idea. Now if only you could do something about the tracks unlocked and quest points... Dragon claw.png Zezzima Talk Edit # SkillsFile:Turmoil.gif|Yay turmoil! 04:45, 22 June 2009 (UTC)

Comment - Yep, the data will contain your level, experience, and rank if you are ranked for a given skill. You just have to make appropriate use of Template:ParseHiscoreData or a similar template that can format it in the way you like. As for music/QP, unfortunately I don't foresee it happening in the near future just because I've seen it suggested many times on the RS forums but Jagex has not really done anything about it. --Quarenon  Talk 22:28, 22 June 2009 (UTC)

I am participating now.

InstantWinstonDragon 2h sword old.pngold edits | new edits

00:47, 22 June 2009 (UTC)

Support! That would be great! It would help players save a lot of time and they can spend that time on other things. I am going to add it to my page in a moment. Bandos godsword.pngC00l dud4Torag's platebody.png 09:32, 22 June 2009 (UTC)


I have just added the template to my userpage now. I srt the integer for 1 for the first update as a test and then I will probably change it to 5 or 6 when its completed the first run. One thing that you might want to think about adding however is a date field of some sort. Its all very nice having an auto update for the stats but I still have to manually go in an edit in a date. Is there a pre existing wikicode for showing the date the last edit was made on an article or does it have to be built into the bot somehow? --Gold ore.png Mercifull UK serv.svg (Talk) 08:59, 22 June 2009 (UTC)

{{#time: j F Y|{{REVISIONTIMESTAMP}}}} should work. (outputs "26 December 2020" for this page). All possible ways to customise the output are listed here. Quest.png Gaz Lloyd 7:^]Events!99s 10:57, 22 June 2009 (UTC)
I know how to get the revision timestamp on the current page but cant seem to find a way to show this information somewhere else. For example: How could I display the revision date/time from User:Mercifull/Hiscores on User:Mercifull? --Gold ore.png Mercifull UK serv.svg (Talk) 12:56, 22 June 2009 (UTC)
If all else fails with using {{REVISIONTIMESTAMP}} or some other tag, then I may be able to add the timestamp directly into the edit in a way that doesn't break Template:ParseHiscoreData. If I can get this to work then I will go ahead and make the change to the bot, make a template for it, and let you know. --Quarenon  Talk 22:24, 22 June 2009 (UTC)
Yeah, I played around with {{REVISIONTIMESTAMP}} for a bit, and browsed around a bunch of help pages for magic words and parser functions, but (short of asking staff) it seems you can't get the timestamp of another article without the aid of a parser. The following code would do the trick:
{{#switch: {{lc:{{{1|}}}}}
| #default=<hiscores lite data>
{{User:<name>/<subpage>|time}} would give the time, and {{User:<name>/<subpage>}} would still give the HS data (so it wouldn't affect existing usage). Qbot would have to update whatever comes between "#default" and "}}" only. Quest.png Gaz Lloyd 7:^]Events!99s 19:56, 23 June 2009 (UTC)
This looks like a good solution, thanks! --Quarenon  Talk 19:28, 24 June 2009 (UTC)
For some reason any templates on my userpage do not update until I edit it. Is this a problem with anyone else? 16:35, 24 June 2009 (UTC)
Quarenon has ceased the bot's functions until we have discerned if we are allowed to use it, according to Jagex's ToS (specifically here; discussion above). Quest.png Gaz Lloyd 7:^]Events!99s 17:51, 24 June 2009 (UTC)
In general, if you edit a template that is transcluded in another page, that page may not get updated immediately (pages needing an update are added to the job queue). But you can force it to update by appending ?action=purge to the URL, e.g.: --Quarenon  Talk 19:28, 24 June 2009 (UTC)

Time between intervals

The minimum 1 hour is a tad bit too short, in my opinion. We should be using intervals of days, not hours. Even GEMW price updates are not done every day.   az talk   16:42, 22 June 2009 (UTC)

Every 1 or 2 days? Ancient talisman.png Oil4 Talk 16:45, 22 June 2009 (UTC)
Can we agree on 6 hours as the absolute minimum time between updates? Users can certainly feel free to choose a higher interval like 24 hours or greater, but I would like to give people some flexibility too, and 6 hours doesn't seem to be overkill. We can also raise this minimum as needed if the bot becomes used more often. --Quarenon  Talk 22:35, 22 June 2009 (UTC)
No, 6 hours is just too short. Players don't usually level every 6 hours. I'd prefer a minimum of 2-3 days between updates, similar to our price bots. No less. If users want updates more frequently, they should not use the bot. Having a bot update a bunch of user pages every 6 hours would severely lag RuneScape Wiki.   az talk   23:28, 22 June 2009 (UTC)
To be honest, I don't see why 6 hours is all that bad, and the bot places about as much load on the servers to update 10 users as it would to update 1,000,000 users; of course, in the later case, each user would be waiting significantly longer to get their update. The point is the bot maintains a constant, appropriate edit rate and doesn't make an effort to edit faster in the face of a long queue of users needing an update. In addition, the market bots understandably update in the interval of days because that's how often the price is adjusted by Jagex. However, ranking and and (sometimes) experience fluctuate constantly, and for new players, levels do too. I'm not wanting everyone to use the fastest rate possible. I would support encouraging 24 hours or greater on the template page, and use good faith in people choosing the right rate associated with their amount of playing and their level. Having said that, I'm open to changing the interval unit to days or keeping it in hours but enforcing a higher limit of 24 hours or more, if that's what the admins or community-at-large prefers. Can you propose a specific minimum time (not a range) and the best unit (hours or days) for that minimum? Thanks! --Quarenon  Talk 01:42, 23 June 2009 (UTC)

Lol,Az beat me to it of course. Once a day is most likely more than sufficient. And lets do it at a dead time to not clog recent changes, I am sure in 2 weeks there will be 200 people using it (well, maybe)--Degenret01 17:02, 22 June 2009 (UTC)

Surely its better to stagger it rather than do them all en masse at a specific time to stop server lag etc. I have set mine to 5 hours for the testing period but would be willing to change to 12 hours (twice a day) when it goes out of beta, i think this is reasonable. --Gold ore.png Mercifull UK serv.svg (Talk) 17:18, 22 June 2009 (UTC)
The current behavior is for the updates not to occur at any specific time. If the bot is flagged as a bot by a bureaucrat, then its edits should be automatically hidden from Recent Changes (note the "Show bots" link in the Recent Changes options). Also, during peak times, the bot uses a feature of MediaWiki that causes the bot to perform an exponential backoff to allow the server load to decrease before the bot begins editing again. --Quarenon  Talk 02:30, 23 June 2009 (UTC)
What MediaWiki feature specifically? Please elaborate.   az talk   05:31, 23 June 2009 (UTC)
See mw:Maxlag parameter. The bot only completes requests if the database servers are able to keep up with the master by 5 seconds or less. If that's exceeded, which happens under high load conditions, then the bot defers the request by exponential time (15, 30, 60 seconds and so on up to a maximum of 5 minutes). So this is largely a Quality of Service feature that prioritizes humans over bots in times of high load, because MediaWiki generally doesn't defer human editors when database lag goes up. I also plan (e.g. it's not doing this currently) to perform the same type of exponential backoff whenever any generic 500 server error occurs or if a network error occurs between my bot and the Wikia servers. --Quarenon  Talk 13:13, 23 June 2009 (UTC)
Let me clarify also that the current behavior for a 500 server error, network timeout or other abnormal response is for the bot to throw an unhandled exception and quit which means I have to manually restart the bot again. As mentioned above I want to change the code so the bot retries these requests and performs exponential backoff for each failure. --Quarenon  Talk 17:13, 23 June 2009 (UTC)

Interim Solution

Since the bot has been stopped due to a concern over Jagex's rules about page requests, I've come up with an interim solution that most people should like since all requests are initiated manually. I call it "EZ Hiscores," a user script you install that creates a reminder box at intervals you specify, and provides a one-click link that will fetch the hiscores for you and edit your data page. It also implements Gaz's solution for providing the last-update timestamp by using {{User:USERNAME/PAGENAME|time}}. A guide and screenshot is at User:Quarenon/EzHiscoresGuide, and I would greatly appreciate any feedback.

I would still like to pursue a bot-based solution but this should suffice for now. --Quarenon  Talk 14:17, 27 June 2009 (UTC)

I for one like it. Works well, haven't found any problems with it yet. Quest.png Gaz Lloyd 7:^]Events!99s 18:34, 27 June 2009 (UTC)
I agree with what Quarenon said. A bot would be better for people who don't go on the wiki often. And yes, I do know that Quarenon made the bot and script. ShinyUnown T | C | E 20:44, 27 June 2009 (UTC)
What about Monobook skin users? Monaco How do you install it for Monobook? 22:54, 27 June 2009 (UTC)
You will need to edit Special:MyPage/monobook.js instead of Special:MyPage/monaco.js. Also, I thought I had already fixed it for Monobook users but apparently the fix was not working. Try again now; you may need to force-refresh your browser. --Quarenon  Talk 23:16, 27 June 2009 (UTC)

I actually prefer this to the bot. It updates it only when I need it to.

InstantWinstonDragon 2h sword old.pngold edits | new edits

16:49, 28 June 2009 (UTC)

I think it would be hard to do this with a bot unless you edited teh page (template), set a parameter to request an update the next time the bot is by, and then had the bot change the request back to no request after its done the update. At that point, a link would just be a lot faster. Unless there is a manual call for the bot to update, I don't see how you could do this within Jagex rules. Unless of course Jagex's highscores lite is still around, because I think you are allowed to request those as much as you like. There just text strings so the bandwidth is minimal. Randox 18:45, 28 June 2009 (UTC)

Hiscores lite is 'still around'; it never left, I thought. ParseHiscoreData depends on it. Quest.png Gaz Lloyd 7:^]Events!99s 20:01, 28 June 2009 (UTC)
I believe there should be a bot, and, if you don't want it automatic, a script for Special:MyPage/monaco.js which creates a toolbox option called "Update hiscores." Just my two cents. ShinyUnown T | C | E 23:33, 28 June 2009 (UTC)
Comment - The original bot proposal is not linked in any technical way to the manual update script provided in this section. Also, the original bot did use the page. --Quarenon  Talk 01:53, 29 June 2009 (UTC)
Comment Anyone got a quote from Jagex about requesting the 'lite' version of the hiscores "as much as you like". Because if its true then this basicly gives the green light to the original bot. --Gold ore.png Mercifull UK serv.svg (Talk) 08:01, 29 June 2009 (UTC)
Comment - May be someone could ask in the Jagex Lite forum. - TehKittyCatTalk Wikian-Book 04:19, 30 June 2009 (UTC)
Comment - I've already started a topic in the Website Feedback forum. --Quarenon  Talk 02:50, 1 July 2009 (UTC)
Why don't you get the bot to access hiscores_lite with a custom useragent? Thats what RuneScript does so that if Jagex feel you are making too many requestsd they can block you and it won't affect anyone else. RScript must make a thousand times more requests to the hiscores than this bot ever will. -Gold ore.png Mercifull UK serv.svg (Talk) 07:54, 1 July 2009 (UTC)
Comment - I can change the user agent, but I don't see anywhere on the Jagex site saying that the user agent affects whether automated requests are allowed and/or how often they can be made. --Quarenon  Talk 14:47, 1 July 2009 (UTC)
No, but it gives Jagex more flexability to block and/or throttle queries at their end. It also gives them more information in server logs on the bot usage etc. There is currently nowhere on the Runescape website or forums which state explicitly how many automated requests are allowed and/or how often they can be made. The dynamic signature scripts in use by all the fansites listed in the official fansite directory employ automated updates at least once every 24 hours. To me this is sufficient evidence to employ a bot to run this often. I think it is still wise to put a bit more pressure on Jagex via the forums to release an official statement but see no reason why we cannot go ahead with the bot you have created with a fixed update timer. The reasons given by people on this thread for disabling the bot just don't hold water. The post by Jagex about limiting access is meant to be for 3rd party clients accessing whole pages of information direct from the Runescape manual rather than pulling a few numbers off static page containing only a string of numbers. p.s this talk page is a MESS to navigate around. --Gold ore.png Mercifull UK serv.svg (Talk) 15:25, 1 July 2009 (UTC)
I can certainly set up a custom user agent but I still can't find an official statement on RuneScape about this. IP addresses in general are a much better way to block or throttle. Page accesses can be made with any user agent; it can even spoof a regular web browser like IE or Firefox and change on a request-by-request basis, so making a block based on user agent alone when someone is trying to overload a page on purpose wouldn't work well. --Quarenon  Talk 21:49, 1 July 2009 (UTC)
I say simply create a connection that the bot will use that Jagex can block, and if they do, delete the bot. ShinyUnown T | C | E 23:19, 14 July 2009 (UTC)
The bot runs from my IP, and if they block it by IP, they could easily end up blocking me from playing or mark my account. Maybe they would block me from just the hiscores page or only blocking a custom user-agent used by the bot, but that has problems as I mentioned before. So, doing that can be very risky unless Jagex outlines a better policy for these types of situations. --Quarenon  Talk 05:18, 17 July 2009 (UTC)
Well, this is going absolutely nowhere. Jagex hasn't responded, and everyone says to listen to them. What are we to do? ShinyUnown T | C | E 18:34, 17 July 2009 (UTC)

Against the rules

From Rules of RuneScape > Macroing, and use of bots or third-party software for Third-party software that displays pages/content from Jagex's websites: "It must not automatically make repeated page/content requests from our website or otherwise make excessively bandwidth-intensive requests from our servers (each page/content request should only be in direct response to a user request each time)." So, I would presume this would include the hiscores update bot which makes it against the rules to use. The JavaScript version since it does not make repeated requests automatically and only operates by a user request should be fine. - TehKittyCatTalk Wikian-Book 18:41, 29 July 2009 (UTC)

Its a grey area because Jagex created the "lite" version of the high score precisely FOR sites to scrape the data for dynamic sigs etc. That rule is in place mainly to stop 3rd party clients form loading whole pages of the Runescape website. Unfortunately it has been difficult to get Jagex to make an official statement saying its either against the rules or not. If it does prove to be against the rules then there are LOT of runescape fan sites breaking it which I think is very unlikely. --Gold ore.png Mercifull UK serv.svg (Talk) 18:51, 29 July 2009 (UTC)
TehKittyCat, there has not been any change on this since I disabled the bot weeks ago for that reason and tried raising an issue about it on the Jagex forums. The last response I got was:
Hi guys,

I don't deal with this kind of issue myself, but I am in the position to get a response for you from someone who may know. Hold tight and I'll pass it on for their attention.


Mod Howes

--Quarenon  Talk 19:06, 29 July 2009 (UTC)
Well, we have 1 gemw bot(we had 2, but GEBot and Nightgunner5 seem to be gone) which make repeated accesses to the RuneScape site everyday. They haven't been banned, so it might be ok otherwise they would have been blocked. How about use the user comparison to get data on 2 users at once and split the updates over a few days to reduce the load on us and Jagex. Also, only update every few days or once a week. - TehKittyCatTalk Wikian-Book 23:02, 29 July 2009 (UTC)
I would be willing to put money on all the dynamic sig sites (tipit, draynor etc) and tracking sites (runescript, runemonkey etc) updating at least every 24 hours. --Gold ore.png Mercifull UK serv.svg (Talk) 07:45, 30 July 2009 (UTC)
And we have a lot less data to retrieve than those sites, so would use less load. As far as I know Draynor(and likely the ohers) is 24 hours. - TehKittyCatTalk Wikian-Book 10:10, 30 July 2009 (UTC)

I asked Mod Ianh (Head of Web Systems) about this. He has some concerns about automation. I'll paraphrase rather than quoting.

Short version: Once every 24 hours is OK, but no more, please.

Long version: Bear in mind that the Hiscore module is already under a fairly significant load just from current 'normal' usage.

Let's take a rough estimate of about four million players. That's not an unreasonable number for us (in fact it might even be conservative) in terms of players active over any particular period.

If all of those 4m players were to request a Lite hiscore even once every 24 hours, it would add an average of about 3000 hits per minute to the Hiscore server load. And that's assuming they were all completely evenly spaced, so in practice you'd see transients that would be much, much bigger (and thus harder on the back end).

Obviously it's not entirely likely that 4m players would start using the Lite Hiscores feed regularly. But it's not impossible, and it should provide a decent illustration of the potential (and potentially unsustainable) loads that are possible. 50 hits a second, sustained, would be a big additional strain. And as I said, that's just from each player making a single query at 24-hour intervals.

Hope this makes sense. Sorry to keep you waiting for so long for the info - not sure where the communication breakdown happened, but we've got there in the end, I think.


Once every 24 hours... Hmm. ShinyUnown T | C | E 21:35, 31 July 2009 (UTC)

Also, allow me to emphasize something we may have learned about the hiscores:
Bear in mind that the Hiscore module is already under a fairly significant load just from current 'normal' usage.
This implies that the normal hiscores pages and hiscores lite use the same server. ShinyUnown T | C | E 21:47, 31 July 2009 (UTC)
Of course they use the same server. The data is pulled from the database the same it's just displayed as plain text rather than surrounded by loads of html code and images. So from this post can we safely assume that once every 24 hours is ok. The bot can come back (with limits) --Gold ore.png Mercifull UK serv.svg (Talk) 22:59, 31 July 2009 (UTC)

Modified Proposal

Per comments here:

Once every 24 hours is OK, but no more, please.

Mod Jon H

Based on this reply, it seems that Jagex is fine with a 24 hour interval per player. I've modified the proposal so that the bot will now use a fixed 24-hour interval for updates, beginning at the time when the bot first notices the template. In other words, there will be no specific time when the updates are made, to help even out the load. I suppose we need new consensus based on this new information so post your new thoughts now.

Note: Per Azaz, consensus on this will be enough to get QBot the bot flag.

Support - As nominator. --Quarenon  Talk 04:40, 1 August 2009 (UTC)

Support - With it following wiki and Jagex's rules I see no problem with it. You've proved you can run a bot and most of the above discussions should have worked out the kinks and possible issues. One note, make sure it handles Wikia's common issues with blank/slow loads well, and does not edit more than once about every 12 seconds. - TehKittyCatTalk Wikian-Book 05:27, 1 August 2009 (UTC)

Comment - I forgot to address this--the initial edit rate will be set to around 2 edits per minute. It's unlikely that it needs to go any faster unless literally thousands of users start using the bot. As far as Wikia server issues, the bot handles this very well. Any abnormal conditions (e.g. connection timeouts, non-200 status codes, malformed/empty response, or high db lag) results in a retry with an exponentially increasing delay. The goal of that is to minimize load as much as possible when Wikia experiences those occasional server hiccups. --Quarenon  Talk 09:15, 5 August 2009 (UTC)

Support - Per Catcrewser. ShinyUnown T | C | E 12:17, 1 August 2009 (UTC)

I think it would be a good idea to have the bot download the logs and upload them on wiki userpages (per here). ShinyUnown T | C | E 15:27, 2 August 2009 (UTC)

Comment - To be polite and respectful of other fansites and of Jagex load I think it would be respectful to increase the request to every 48 hours. I know they said 24 but that is a rough approx. so perhaps we can respect their load and offer some leeway.

Bonziiznob Talk

15:35, 2 August 2009 (UTC)

I don't think I'd like to have my hiscores updated every 2 days... ShinyUnown T | C | E 15:42, 2 August 2009 (UTC)
There's nothing stopping you updating your HS data yourself, or not using the bot at all (using the EZ Hiscores script maybe).
Support every 48hrs seems fine to me. Quest.png Gaz Lloyd 7:^]Events!99s 20:22, 2 August 2009 (UTC)

Support - Im good with 24 hour. -White partyhat.png Chasingu 03:29, 3 August 2009 (UTC)

Support new bot How about as a compromise we have a template (bear with the hypothetical-ness) thats soemthing like {{hiscoresbot}} which causes the bot to update your scores every 2 days. However it also has the ability to specify the number of days to update {{hiscoresbot|1}} for every 24 hours {{hiscoresbot|7}} for a week etc. That way by default (for most users) it will only update every 2 days but for those that level up quite often they get the ability to specify a shorter time (daily). This would also enable people to change it to weekly 2 weeks if they are going on holiday or a break. --Gold ore.png Mercifull UK serv.svg (Talk) 08:08, 3 August 2009 (UTC)

Support - I support bot idea, especially if we include the suggestions from Mercifull. Strength-icon.png Hammer2092 Time |KOTA Bandos godsword.png 14:07, 3 August 2009 (UTC)

Fyi, we already have the template. Also, who would like the bot to download the logs and upload them on wiki userpages? Quarenon says if enough people like it he'll do it. ShinyUnown T | C | E 14:14, 3 August 2009 (UTC)
The template is already being used for this, see Hiscore autoupdate. I can change interval to be in units of days and default to 2. Would this satisfy most people? --Quarenon  Talk 21:10, 3 August 2009 (UTC)
I'm going to support Quarenon's solution. ShinyUnown T | C | E 21:13, 3 August 2009 (UTC)
Support :) --Gold ore.png Mercifull UK serv.svg (Talk) 21:41, 3 August 2009 (UTC)
Support - I support the 2 day template change. 15:44, 5 August 2009 (UTC)

Closed - QBot will now receive the bot flag.--

Helm of neitiznot (charged).png Azaz129 Crystal shield.png Talk Edits Contribs

16:28, 7 August 2009 (UTC)