Forum:GE Price "Live Feed"

From the RuneScape Wiki, the wiki for all things RuneScape
Jump to: navigation, search
Forums: Yew Grove > GE Price "Live Feed"
This page or section is an archive.
Please do not edit the contents of this page.
This thread was archived on 26 July 2008 by Azaz129.

I am interested in programming a web application that would query information from the official Runescape GE Price database, so that all prices could be current and accurate. No other fansites have this, although it has been done with the hiscores. I haven't looked into it too much yet but I believe it is possible. Is anyone interested in having this as a feature of the wiki? (if not I'll sell the patent to =P). I could also implement the type of thing Reddo suggested way up at the top. Quote: "now that the GE database has been released couldn't someone create a bot to pull the prices for each item off of there once a day or something?" Excalo 01:32, 19 June 2008 (UTC)

Having a bot to edit those pages would be much more convenient and efficent, it would be great if this was done in my opinion. --

Helm of neitiznot (charged).png Azaz129 Crystal shield.png Talk Edits Contribs

03:32, 19 June 2008 (UTC)

I think Pointy said something about this a while back. Let me try to find the link, it might still be on this page actually. Skill 05:11, 19 June 2008 (UTC)
Yep, it's under the first two headings at the top of the page. Skill 05:20, 19 June 2008 (UTC)
I've not really had time lately to make any progress on a price-scraping bot, and it looks like it wouldn't be allowed to run anyway based on the bot talk page. I'm still running the chart updates once a week or so, but thats about all I've got time for at the moment I'm afraid. Pointy 15:51, 19 June 2008 (UTC)
Did your bot get to the point where it could retrieve the prices from the database? Excalo 18:00, 20 June 2008 (UTC)

From our discussions here, Jagex rules state that we can't send an automatic repeating request for updated information and so if a bot gets created to update the prices, a human would have to be responsible for sending the request.--Diberville 14:14, 19 June 2008 (UTC)

Ah, I was wondering if there was a rule like that. Perhaps a button next to each price, named "refresh price". Only when it is pressed would the application pull the info from the database. Excalo 14:48, 19 June 2008 (UTC)
I also posted this above but what about a bot that only grabbed the prices once every 24 hours? I believe that this falls within the rules that jagex has stated. Reddo 02:46, 20 June 2008 (UTC)
"It must not automatically make repeated page/content requests from our website or otherwise make excessively bandwidth intensive requests from our servers (each page/content request should only be in direct response to a user request each time)."
The way they worded it suggests that they don't want us doing even that. However, it would probably be feasible to have a refresh price button on every page and hope that people click on them often. In fact, there could probably be links that would allow simultaneous updating of groups of pages (for example, headings on the GEMW pages) without violating this rule, as long as we don't go overboard. The rule is a bit ambiguous as to what constitutes "overboard", though. Skill 04:04, 20 June 2008 (UTC)
The whole "direct response to a user request each time" kind of puts an end to a bot getting them for us. I like the idea of having a price refresh button on each page but I don't know how we would create something like this (maybe a bot that is activated by clicking on the button). In the response to the "hope that people click on them often" we could add a parser function like this to all of the exchange pages
What if, for each item, it asked "Do you want to update ITEM-NAME?". Technically each request would be in direct response to a user then. Also, what about Wikia's policy on bots? Do they have one? immibis 08:55, 1 July 2008 (UTC)
{{#ifexpr: {{DATEDIFF|{{REVISIONYEAR}}|{{REVISIONMONTH}}|{{REVISIONDAY}}|{{CURRENTYEAR}}|{{CURRENTMONTH}}|{{CURRENTDAY}}}} > 1| [[Category:Needs price update]] | }}
This would put all exchange articles that haven't been updated in the last 2 days into a new category, (GE articles needing price updates) this could help us keep a little more update with the current prices without having to assign particular articles to people to update each day and without having a bot to do the whole thing for us. Reddo 05:14, 20 June 2008 (UTC)
Within the guidelines they layed out, saying that requests have to be activated by a user, we could allow users to update all the prices at once. However, I don't think that this would be a very good option. Querying 4,000 of their pages will take a while. Perhaps in the Category:Needs price update, an option to update all articles needing price updates. Excalo 18:00, 20 June 2008 (UTC)
I added the parser function, the list of old prices can be found here. Reddo 16:56, 23 June 2008 (UTC)

(reindent) I'm not too great at JS myself, but would there be any way to read the content directly from the RS servers using AJAX (or similar) and then automatically edit the page based on said data? I know the JS editing should work, as they have scripts for it on Wikipedia, but I'm not too sure the AJAX would go too well... especially since the page might not be well-formed XML (Jagex uses HTML 4). I have no way of checking whether it is at the moment, as the GE database went down about 40 minutes ago. :\ Skill 09:22, 1 July 2008 (UTC)

It's back up now, it must have been down when they applied the website updates. It's under Game Guide -> Grand Exchange on the new website, BTW. Also I was thinking of scraping it from an external program, ie it runs on some user's desktop every day, because I don't think Wikia allows JavaScript on a wiki page. 03:53, 2 July 2008 (UTC)
Oops, I thought I was logged in when I posted that last one -- immibis 04:13, 2 July 2008 (UTC)
The bot is done (that is to say, it works. I've got a few more changes in mind like a configuration screen so you can change how it runs. It has successfully made one edit (to Exchange:Ibis_pouch, see diff here. As per the bot policy I ask people to vote on this.
The bot works by downloading the prices for every tradable item from the RuneScape website, then for each item it makes the necessary changes to each price, moving the Price and Date fields to Last and LastDate, and updating the current price and date. It ignores pages that don't exist or are missing one or more fields. However, it is accessing pages without human intervention, which Jagex doesn't want to happen. immibis 07:34, 4 July 2008 (UTC)
The whole point of the discussion was that according to [ rule 7] (point number 7 in the section I linked), every request to a page on the RS site has to be a direct response to a user request. In other words, we can't have a bot to automatically request every item page listed. There might be some ways to work around this, which is what we were trying to get to, but a fully automated bot won't work. Sorry. :\ Skill 08:27, 4 July 2008 (UTC)
Oh. What if it asked "do you want to download and scrape" for each page? Would that work? immibis 08:31, 4 July 2008 (UTC)
But how do we implement that for each and every query that is made by the bot? Skill 11:02, 4 July 2008 (UTC)

(reindent) Simple. You know when you close Word and, if you haven't saved, whether you want to save? Like that, only for each Grand Exchange page that is downloaded. Also, would it be a good idea to discuss this in User_talk:Immibot as this section is getting a bit long? immibis 23:06, 4 July 2008 (UTC)

And you're willing to press "yes" manually for each of the 1500 pages every day? If that's what you're trying to tell me, then go ahead... Skill 00:03, 5 July 2008 (UTC)
Actually, it can get the prices from the page with a list of prices (eg when you click on A on the website it shows the first N items starting with A) so it shouldn't be more than about 200. And if the button appears in the same place each time, you can just keep clicking on that spot, and it wouldn't need to ask this for updating the wiki, as Wikia doesn't have that rule.immibis 00:09, 5 July 2008 (UTC)

This conversation is continued at User talk:Immibot