Forum:Scraping hiscore data for calculators

From the RuneScape Wiki, the wiki for all things RuneScape
Jump to: navigation, search
Forums: Yew Grove > Scraping hiscore data for calculators
This page or section is an archive.
Please do not edit the contents of this page.
This thread was archived on 22 September 2014 by Cqm.

I recently received the suggestion to implement hiscore scraping for JS calculators, like these ones, and I've finished the script; you can try it by pasting


here, and then loading any js calculator. It adds a text field and a button to the calculator interface, and when the button is clicked, the script fetches the level (or XP, depending on the selection) of the account whose name was entered in the field from the hiscores, and inserts that number into the corresponding field.

I think it would be a nice addition to the wiki. If you think it's too disruptive, I can replace it with a small checkbox that will summon it only if the user wants to. I've also considered logging the last username the user entered with a cookie, as Joey's QC script does, but, of course, that would need community consensus.

9xFavoY.pngI wish I was Scuzzy Betahib8CAd.pngHe's a lot cooler and also smells like flowers I hearOil4 I made this 16:19, July 25, 2014 (UTC)


Big caek support 9xFavoY.pngI wish I was Scuzzy Betahib8CAd.pngHe's a lot cooler and also smells like flowers I hearOil4 I made this 16:19, July 25, 2014 (UTC)

Support - And I'd like the caekie cookie too Ancient talisman.png Oil4 Talk 19:51, July 25, 2014 (UTC)

Comment - Looks pretty good, however I would suggest having this feature be an opt-in for calculators by having a class on the form, rather than adding it to all calculators. ʞooɔ 13:48, July 26, 2014 (UTC)

If all the calculators that you want to have this are modified as such, then this is a very easy modification. Alternatively, we could just have a .js file where we whitelist all the calculators that need this. 9xFavoY.pngI wish I was Scuzzy Betahib8CAd.pngHe's a lot cooler and also smells like flowers I hearOil4 I made this 14:03, July 26, 2014 (UTC)
Whitelisting the calculators is dependent on an admin being around who's confident in doing something like that. Adding a class or a way of autodetecting where it should load is easier to maintain imo. cqm 19:40, 27 Jul 2014 (UTC) (UTC)

Comment - A few notes on your code:

  • Use $( func ) over addOnloadHook( func ); The latter was deprecated years ago.
  • Would it be possible to not load getHSData.js in your script unless it's not already loaded? We can add it to the existing imports array in the Common.js and import it using importArticles taking advantage of ResourceLoader which minifies it and reduces http requests. We'd need to import getHSData.js before importStats, but RL honours the order of the imports iirc.
  • Maybe stick the ES6 polyfill for String.prototype.contains in a conditional for those browsers that do support it and have have a marginally better way of doing it.
  • Why did you add jQuery into the closure, but then use jQuery over $ in when you extend jQuery itself?
  • Why did you elect to extend jQuery? Surely there's a easier to follow method, such as looking for a class. Maybe altering Q's calc script for something you can hook off?
  • You should be defining your functions before using them ideally, which you've followed for those in the local scope, but not for those that end up in the global scope.

This imported script looks good apart from all the globals which you've avoided in the proposed script. You've also extended the String prototype to avoid doing someString.replace( / /g, '\n' ); which saves all of one character in a method you use once... cqm 19:40, 27 Jul 2014 (UTC) (UTC)

  1. OK, will do.
  2. Er, OK. As long as it's loaded wherever importStats is loaded, that's fine.
  3. What?
  4. Sorry, that was dumb. I meant to just isolate the code in a closure, and I most of the times I've done that was to extend jQuery. Reflex action, I guess.
  5. ^that.
  6. All the functions in importStats.js are limited to that closure, so are not global. The only global functions are in getHSData, which is imported at the *top* of importStats.js. Anyway, it doesn't matter where I define them if I use the function keyword... right?
  7. I'm just used to extending prototypes whenever I code something, if I need to. Anyway, replaceAll is quite a commonly used extension, so there shouldn't be any conflicts.
9xFavoY.pngI wish I was Scuzzy Betahib8CAd.pngHe's a lot cooler and also smells like flowers I hearOil4 I made this 09:40, July 28, 2014 (UTC)
This is what I mean by putting the polyfill in a conditional. I originally thought it was a custom string method until I researched it, but apparently it's part of the EMCAScript 6 proposal. So far it's just Firefox that supports it. cqm 09:54, 28 Jul 2014 (UTC) (UTC)
Also if you extend a global in a closure, you extend that global. It only works like you think if you assign a global to a local variable and then extend that local variable:
// this extends jQuery
( function ( $ ) {

    $.foo = function () {
        console.log( 'foo' );


}( jQuery ) );

// this won't
( function ( $ ) {

    var $local = $;

    $ = function () {
        console.log( 'foo' );


}( jQuery ) );
I also made a quick edit to your script here to show you what I meant about the closure thing. Undo my undo if you want to keep it :) cqm 10:04, 28 Jul 2014 (UTC) (UTC)
Ah, I see now. The reason I used jQuery and not $ was because I initially wrote the extension outside of a closure, and it was when I was putting it into one that I carelessly copypasted it in. Now that I've 'accepted' your edit, I see no problem with extending jQuery in this way. 9xFavoY.pngI wish I was Scuzzy Betahib8CAd.pngHe's a lot cooler and also smells like flowers I hearOil4 I made this 10:11, July 28, 2014 (UTC)
This is not true! Both extends jQuery. Try running the last one in your browsers console and after that run ``. Prayer-icon.pngTyiloTalkQuest.png 22:54, August 16, 2014 (UTC)
So it does, my bad. Learn something new every day. cqm 23:22, 16 Aug 2014 (UTC) (UTC)

Comment When I click the link to paste the calculator it tells me the page does not exist. Can somebody explain what I am doing wrong? This is exactly what I think the calculators have been missing. Looks like a good idea. --Yayepicyay (talk) 21:18, August 8, 2014 (UTC)

If Special:MyPage/wikia.js doesn't exist, it means you need to create it. MolMan 21:22, August 8, 2014 (UTC)
Think I must be doing something wrong because the change bluefire mentioned did not appear to have happened. Do I need to do anything special on Special:MyPage/wikia.js to get it to work? Thank you, --Yayepicyay (talk) 21:33, August 8, 2014 (UTC)

Comment - Nothing interesting happens. (Aside from skills not defined error being thrown when importscript is being bad.) I even went and copypasta'd the entire script and still nothing. Quest.png Gaz Lloyd 7:^]Events!99s 00:49, August 9, 2014 (UTC)

Great, it doesn't work now. It worked when I made this thread. I'll take a look at it... 9xFavoY.pngI wish I was Scuzzy Betahib8CAd.pngHe's a lot cooler and also smells like flowers I hearOil4 I made this 07:59, August 9, 2014 (UTC)

Support - It works for me now and it's looking excellent, we should have done this years ago. I like to cookie idea too, just remember to account for display names changing since when the cookie was last set. cqm 12:46, 17 Aug 2014 (UTC) (UTC)

Support - It works now for me as well. What did you change from last version Bluefire? Yayepicyay (talk) 12:58, August 17, 2014 (UTC)

Comment - Doesn't work me for me, I just get 2 alerts with the text "Unable to load data. Please try again later." I can see that my browser tries to load which redirects to which returns "jQuery18208853682687040418_1408293345423({"contents":"alarm\n"})" Prayer-icon.pngTyiloTalkQuest.png 16:38, August 17, 2014 (UTC)

It's working for me Ancient talisman.png Oil4 Talk 19:22, August 17, 2014 (UTC)
It works for me too now. I blame Prayer-icon.pngTyiloTalkQuest.png 19:24, August 17, 2014 (UTC)
I've had some problems with myself too when implementing HS fetching in S:C. It happened regularly that the server just replied with a single word "alarm" too, instead of replying with the data. There didn't seem to be any logic behind this reply, and it sometimes even consistently didn't work for one username, but it did work for another username at the same time. Weird stuff. JOEYTJE50TALKpull my finger 11:40, August 20, 2014 (UTC)

Support - It's working for me. Temujin 00:21, August 18, 2014 (UTC)

Problem - It has stopped working for me... When I try using, get error: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at This can be fixed by moving the resource to the same domain or enabling CORS. "alarm " "parsing the data" "executing the callback function"

I have a screenshot of console output... Why is this being inconsistent? It worked before I went to bed... Yayepicyay (talk) 13:22, August 18, 2014 (UTC)

That error is nothing to do with the script, that's a font file loaded by the wikia servers (which interestingly isn't working atm). The number of issues however might warrant some alternative to anyorigin be explored. Are there any plans for that (@Bluefire)? cqm 14:25, 18 Aug 2014 (UTC) (UTC)
I think the best solution would be to use the wiki's server (maybe using a php script?), to do the same as anyorigin. Prayer-icon.pngTyiloTalkQuest.png 14:51, August 18, 2014 (UTC)
Could we use action=parse of {{#hs:USERNAME}} instead? cqm 23:35, 18 Aug 2014 (UTC) (UTC)
Correct me if I'm wrong as I couldn't find the documentation for {{#hs:USERNAME}}, but it isn't currently possible to extract more than one skill level/xp/rank in one request. Some calculators uses more than one skill, so it should be modified so that it could return all the data and the javascript script could then extract the xp for the relevant skills. Once that is done, the script can simply do something like:
var username = 'Tyilo';
var wikitext = '{{#hs:' + escape(username) + '}}';

$.get('' + encodeURIComponent(wikitext),
	function(data) {
		var totalLevel = Number($(data.parse.text['*'])[0].textContent);
		console.log(username + '\'s total level: ' + totalLevel);
Prayer-icon.pngTyiloTalkQuest.png 00:15, August 19, 2014 (UTC)
Here's the docs although it doesn't appear to support divination (I don't know about minigames either). cqm 08:54, 19 Aug 2014 (UTC) (UTC)
Seeing as it's just pulling data from jagex's api, I think all we need to do to update it for more recent additions is tweak the templates it's used in and update the comments in the source code. Everything's updated on the docs, I just need to get the actual code updated, but it looks like it's limited to 2 calls per page. Is that ok, or do we need to get it increased? cqm 09:17, 19 Aug 2014 (UTC) (UTC)
I don't think the limit applies to the api call (just tested it). Inserting this code:
if($skill == 'data') {
    return $data;
just after the lines
if( $status == 200 ) {
	$data = $wgRSHiscoreCache[$player]
should allow us to call {{#hs:Tyilo|data}} to get the raw data from Jagex's hiscores. Prayer-icon.pngTyiloTalkQuest.png 13:35, August 19, 2014 (UTC)
I've submitted a pull request to get the comments for which numbers are what updated, so hopefully I can do this extra thing without much trouble. I just need to get my devbox booted up to test any code changes before submitting them, so it might be a little longer to get that bit updated. cqm 23:12, 19 Aug 2014 (UTC) (UTC)

Comment - is buggy and has been for quite some time, and so the data failing to fetch is an issue outside my control. I've made a script that does the same thing on my website, and after some testing I'll put it into this (Joey's QC script already uses it, so it should be fine). Hopefully this will fix the issue. 9xFavoY.pngI wish I was Scuzzy Betahib8CAd.pngHe's a lot cooler and also smells like flowers I hearOil4 I made this 16:27, August 18, 2014 (UTC)

Question - Is there any way we could make the users computer call the runescape server/have the wiki do it? This using an outside server seems kinda sketchy... Yayepicyay (talk) 20:55, August 18, 2014 (UTC)

Sadly, no. 9xFavoY.pngI wish I was Scuzzy Betahib8CAd.pngHe's a lot cooler and also smells like flowers I hearOil4 I made this 12:39, August 19, 2014 (UTC)
We can make the wiki do it, see the discussion between me and Cqm above. Prayer-icon.pngTyiloTalkQuest.png 13:38, August 19, 2014 (UTC)
Correct me if I'm wrong, but that's an extension coded in php, which is processed before JS. This would only work if I could HTTP call a template and specify a parameter... could I? 9xFavoY.pngI wish I was Scuzzy Betahib8CAd.pngHe's a lot cooler and also smells like flowers I hearOil4 I made this 23:07, August 20, 2014 (UTC)
Yes. But you have to believe. MolMan 23:11, August 20, 2014 (UTC)
See the first of Tyilo's code snippets above. You can use mw:API:Parsing wikitext#parse to get the highscores data, although I seem to remember there's a bug with the preprocessor report showing up where it shouldn't. I can't remember what exactly it affects though so you'll need to do some experimenting. cqm 23:51, 20 Aug 2014 (UTC) (UTC)

Comment - I think the script should always input the experience and change the select dropdown box to "Experience". The only reason the Level is there in the first place is if you're near a level in xp and is too lazy to enter your experience. (This doesn't apply to the goal level.) Prayer-icon.pngTyiloTalkQuest.png 18:46, August 23, 2014 (UTC)

+1 Temujin 09:14, August 25, 2014 (UTC)

Question - Have any changes been made to the hs lookup call that avoids anyorigin (either through your own server or through the #hs parser function mentioned above)? The discussion has more or less finished, so I'd like to try and get this sorted before it loses momentum and we forget about it. cqm 23:01, 1 Sep 2014 (UTC) (UTC)

Closed - Whilst there is consensus for this to be added, the use of anyorigin causes it to be unreliable. I haven't installed the script wiki-wide but when said reliability issues have been addressed, the author or the script may do so informally. cqm 07:55, 22 Sep 2014 (UTC) (UTC)