Forum:Proposing AmauriceBot for GE updates

From the RuneScape Wiki, the wiki for all things RuneScape
Jump to: navigation, search
Forums: Yew Grove > Proposing AmauriceBot for GE updates
Archive
This page or section is an archive.
Please do not edit the contents of this page.
This thread was archived on 11 December 2009 by Calebchiam.

Having seen the existing SmackBot has stopped updating the Exchange prices, I have looked into building a Bot to do the job. I believe that Catcrewser might also be looking at building a Bot to update the prices too. I don't think having 2 Bots updating the prices would be a bad thing, because if one fails to run, the other can continue.

Specification

Primary Task:

The primary purpose of my Bot is to update the prices under the Exchange: namespace. This involves going through each page and performing the following steps:

  1. Get the existing price (and date) from the Exchange: article
  2. Get the market price (and minimim price and maximum price) from the Grand Exchange
  3. If the price is different (see note below), then update Price and Date
  4. If the old price is within 50% of new price (i.e. not vandal price), then update Last and LastDate

Note, the price is considered to be the same if within the precision of the Grand Exchange web site. For example, if the Grand Exchange web site has a price that says 12.2m and the price in the Exchange page is 12214923 (difference of 14923), then this will not get over-written by the Bot as the price could be a more accurate version entered by someone checking the in game Grand Exchange.

The actual update of price will occur under the following conditions:

  1. If last updated less than 12 hours ago, then only update if price wrong by more than 50% (in case of vandals)
  2. If last updated more than 12 hours ago, but less than 7 days, then only update if price is different (by more than the Grand Exchange precision factor)
  3. If last updated more than 7 days ago, then always update, even if the price is the same

Secondary Tasks:

In addition to the primary tasks, the Bot will also do the following:

  1. Write a log to a sub-page under User:AmauriceBot (including logging of vandalised prices)
  2. Update Template:Natprice that is used to determine the profitable high alch items).
  3. Update Template:JunkItems that lists all junk items where the GE price has hit hard lower limit.
  4. Update Template:AntiJunkItems that lists all anti-junk items where the GE price has hit hard upper limit.

Implementation:

The implementation is in five Java classes (Main, GEChecker, WikiSession, GESession and Utils), and I plan to run daily (as a scheduled task at around 5am GMT) 2 or 3 times a week. If the Bot gets the go-ahead, then I will make the source available under GPL. Should the need arise in the future for someone else to take over the Bot, then it can be compiled using the Java SDK from http://java.sun.com.

Any thoughts / comments? Amaurice talk 16:23, November 3, 2009 (UTC)

Discussion

Support - I hate how, since the other bots broke down (because of all the url change and stuff), everything is outdated. I hope this (these?) bot(s) will work soon. Ancient talisman.png Oil4 Talk 16:29, November 3, 2009 (UTC)

Support - Feels like a large chunk of the wiki has been missing since SmackBot stopped. Quest.png Gaz Lloyd 7:^]Events!99s 17:47, November 3, 2009 (UTC)

Support - But instead of "junk" and "anti-junk" how about "dargonflies" and "chickens"? Or if that's not good, how about "Items which have reached a price floor" and "Items which have reached a price ceiling"? I think using player slang might not be the best choice for something like this. kitty.pngPsycho Robot talkSilver bar.png 17:58, November 3, 2009 (UTC)

"Dargonflies"? --Iiii I I I 18:05, November 3, 2009 (UTC)
Comment - The template names are generally short and not seen by the average reader. The names or descriptions on the pages including the templates can be more descriptive for the end readers. Amaurice talk 11:29, November 5, 2009 (UTC)

Support - Per the people before me. --Iiii I I I 18:05, November 3, 2009 (UTC)

Support - Per Above. Magic-icon.pngStelercusIlluminated Book of Balance.png 19:36, November 3, 2009 (UTC)

Support - Sounds helpful. Air rune.png Tollerach hates SoF Fire rune.png 20:26, November 3, 2009 (UTC)

Comment - The code for the primary task is complete, and has been tested updating a few items. The processing has been tested on 500 items so far, but not written back into the wiki to avoid filling the recent history. I will test on the remaining items (without writing back to the wiki). Once AmauriceBot has Bot status then I will run it through writing back (as Bot changes do not show up in the recent log unless asked for). Amaurice talk 21:12, November 3, 2009 (UTC)

Support/Comment - The wiki depends on it and we have lost the other bots(although I think SmackBot is supposed to return). I am going to work on my own bot following similar tasks as above, but I have not yet started. I think redundancy is very important for the continued success of the GEMW. - TehKittyCatTalk Wikian-Book 21:22, November 3, 2009 (UTC)

Support - I had no idea that SmackBot had stopped running and GEBot kind of disappeared as well..we need active bots and I trust you to make one. Andrew talk 21:23, November 3, 2009 (UTC)

Support - Lol Didn't realise someone was working on a bot as a backup SmackBot. I recently added Task 7 to AzBot to do exactly that. In fact, AzBot has been running for a few hours now. It is definitely slower than SmackBot with only 3-4 edits/min, so I don't intend to run this task often... only when SmackBot breaks down unintentionally.

Concerns: For the primary tasks, running it daily is unnecessary. I think the current frequency of SmackBot is adequate, i.e. 2-3 days once. This bot should only complement SmackBot, not run concurrently.
Question: How does the bot determine these "hard limits" you mention?   az talk   23:43, November 3, 2009 (UTC)
Answer: If the minimum price is the same as the average price, then it means its hit a price floor. If the maximum price is the same as teh average price, then it means its hit a price cieling. kitty.pngPsycho Robot talkSilver bar.png 23:54, November 3, 2009 (UTC)

Support And then Azbot task7 could be its backup, although I also think every 2 or 3 days is enough for most items.--Degenret01 08:38, November 4, 2009 (UTC)

Comment - I agree that daily is probably more than needed and will run 2 or 3 times a week. It needs to be about 2 or 3 times a week because prices on manipulated items can be way out after a few days. The Bot can process about 1 item a second, so the 3151 items under the Exchange namespace takes about 50 minutes to process. If another Exchange Bot is running regularly, then I will reduce mine to just once a week (just in case the other Bot starts failing, like with the SmackBot situation). Amaurice talk 08:55, November 4, 2009 (UTC)
That's pretty fast. How do you scrape the prices from the website? I usually scrape the prices one-by-one as I update each item, but it would be faster to scrape multiple prices off the pages like these. And, do you scrape all the prices first before updating? FYI, there are currently 3,145 items in the Grand Exchange.   az talk   09:47, November 4, 2009 (UTC)
The items are scraped one by one from the Grand Exchange as it goes. Each update only involves 3 web accesses. One to get the wiki page (via the API), one to get the GE page and one to update the wiki (if appropriate).
The 3151 is the number of non-redirect articles in the Exchange: namespace. Not all of these are proper Exchange: template pages (such as Exchange:Robert Horning/Sandbox5) and some just contain text like 'Not sold'. I have removed 3 or 4 rogue Exchange: articles. Amaurice talk 10:25, November 4, 2009 (UTC)

Question - The SmackBot used to run about once a day (sometimes 2 or 3 times in one day, and sometimes 2 or 3 days between runs), but about every 2 weeks a full update was run that updated all Exchange artible prices and not just those that have changed price. Is it useful to do a full update occasionally? Amaurice talk 09:05, November 4, 2009 (UTC)

Wouldn't we need it that way to show prices that have been steady? On the GE graphs I mean.--Degenret01 09:32, November 4, 2009 (UTC)
(Edit conflict) Yes, indeed. Performing full updates every once in a while makes the GEMH graphs smoother and nicer.   az talk   09:47, November 4, 2009 (UTC)
I was thinking that might be the case. The question is then how often would a full update be wanted? Amaurice talk 09:38, November 4, 2009 (UTC)
A full update every week is adequate.   az talk   09:47, November 4, 2009 (UTC)

Support - although there are a couple of minor items items I'd love to see added to the exchange data scrapings: the min price, the max price, and (as applicable) the trade volume (aka the number of this item that has been sold). The min and max prices are being suggested here to try and detect trends to price to see if their price caps have been hit, although the [[Grand Exchange Market Watch/Max Price|Max Price]] and [[Grand Exchange Market Watch/Min Price|Min Price]] pages already are displaying this information in a more general fashion. The trade volume might be more useful... especially for commonly traded items. Until now, we haven't been keeping track of this number, but it would be useful to try and come up with some statistics that Jagex isn't providing such as the market caps of various items, and to let players know on item pages how many items are being traded (if the number is known). Unfortunately, the trade volume is given only for the 100 most commonly traded items. I would like to get more information, but Jagex isn't providing it. Perhaps this needs to be a separate bot request and something I come up with, but it would provide some useful information that could be interesting to provide on the individual item pages and may have some applications for other side projects on this wiki as well. The trade volume page on the GE Database is right here at the moment. --Robert Horning 15:39, November 6, 2009 (UTC)

Comment - The scraper already pulls the min and max (for working out the items that have hit the limit). I could extend it to add "Min=" and "Max=" properties to the Exchange: files if it is of use. For the Volume, as you say, only the top 100 are listed. For these items it should be possible top add "Volume=", for items not in the top 100 volume, I assume it's best to add the Volume but leave it blank (or clear it again if the item drops out the top 100). I will look at making these enhancements once AmauriceBot has Bot status so it can be running fully without flooding the Recent Changes. Amaurice talk 17:05, November 6, 2009 (UTC)
Question? - Any thoughts on the order of the new template parameters in the Exchange: files? The order of the changing parameters is generally Price, Last, Date and then LastDate. Where is it best to put Min, Max and Volume in the list? It does not matter for automatic update, but for manual updates that people do, what what will be the best order position for the new ones? We could put them after the other four and assume people won't be updating, or we could expect people to update, in which case the edit tutorial will need updating too. As it is, people don't necessarily update the existing four. Expecting these to be updated might mean they are less bothered to update the extra ones (other than the Price). Any comments? Amaurice talk 17:52, November 6, 2009 (UTC)
Order is irrelevant to actually using the information, especially if it is a named parameter (as this is). If you were adding these parameters, I would put it immediately after the "LastDate" parameter, but that is a style question rather than one that is important. This isn't a "critical" update issue, but it would be useful. --Robert Horning 04:24, November 7, 2009 (UTC)
Amaurice, could you leave a message at my talk page whenever you make this change? I need to update AzBot to reflect any changes to the Exchange pages. Smile   az talk   17:40, November 9, 2009 (UTC)

Is it botted yet? Can we close this?

Closed - AmauriceBot has been botted. C.ChiamTalk 15:17, December 11, 2009 (UTC)