User:Gaz Lloyd/using gemw

From the RuneScape Wiki, the wiki for all things RuneScape
Jump to: navigation, search

GE Price Usage

This guide applies to both the RSW and OSRSW.

Grand Exchange prices are a very common thing to want to use, both within articles and externally. This is a guide to using them in many contexts.

This guide covers the four main areas of using GE prices: on wiki articles via templates, within a lua module, within wiki javascript, and externally. There is also a section on understanding the data structure of the price data on the wiki.

Templates[edit | edit source]

Using these templates doesn't necessarily require any understanding of the data structure, which is why that is the next section.

Basic[edit | edit source]

The most common usage on the wiki is in an article directly, which is easily done via templates. We have two templates for simply using a GE price:

Template:GEP
{{GEP|item}}

Outputs the price of item.

{{GEP|item|number}}

Outputs the price of item times number - handy syntactic sugar to skip a normal multiplication. Number can be fractional or negative!

Template:GEPrice
{{GEPrice|item}}

Outputs the price of item with thousands separators if necessary (which will be commas as this is an English wiki).

Additional[edit | edit source]

We have a few extra templates to simplify some operations with GE prices, us them if you wish.

Template:GETotal
{{GETotal|item1|item2|item3|...}}

Outputs the sum of the prices of one of each of item1, item2, .... No limit on number of items.

Template:GETotalqty
{{GETotalqty|name1=item1|qty1=#1|name2=item2|qty2=#2|...|name20=item20|qty20=#20}}

Outputs the sum of #X times itemX for X from 1 to 20. i.e., its GETotal but you can specify the quantity of each item individually.

Supporting[edit | edit source]

These are not strictly GE price fetchers, but are useful for working with prices.

#expr
{{#expr: expression}}

The basic parser function that will calculate a mathematical expression. Will throw an error if something is not right with the expression.

eg {{#expr:{{GEP|Shortbow}} - {{GEP|Logs}} - {{GEP|Flax}}}}

formatnum
{{formatnum: number}}

Formats the number with thousands separators.

Template:fe
{{fe|expression}}

Syntactic sugar to combine #expr with formatnum to calculate and expression then format the result with thousands separators.

Template:Coins
{{Coins|expression}}

Calculates expression (with #expr) then formats the result with an image of coins and a colour representing profit/loss. Generally should be used in tables and not prose.

Template:NoCoins
{{NoCoins|expression}}

Like Template:Coins but without the image. As the image can distrupt the layout of prose, this template is preferred to Coins.

{{NoCoins|expression|c}}

Adds the word 'coins' after the result, which is also coloured.n

Understanding[edit | edit source]

Understanding the data structure is important for more technical uses of the data. There are three main pages types involved with GE prices. Every item has one of each.

Exchange pages
Exchange:Bones - osrsw:Exchange:Bones

Mostly a front end view of the data now. Historically the home of the data before the move to lua.

Information module
Module:Exchange/Bones - osrsw:Module:Exchange/Bones

Contains all the 'current' information about the item. When a price is requested for an item, it will usually end up loading this module.

This is a lua module which returns a table, a data structure fairly similar to JSON (and easily parsed into it).

Data module
Module:Exchange/Bones/Data - osrsw:Module:Exchange/Bones/Data

This contains historical information on the price - used to generate the graphs - in the format 'UNIXtimestamp:price' or 'UNIXtimestamp:price:volume' if volume is available for the item at that timestamp (volume information is only available for the top 100 most traded items (for OSRS, volume is available for almost every item).

This is also a lua module, returning a table array.

Bulk pages

The bulk pages are several pages that contain information for every item in GEMW - they are:

RSW OSRSW Purpose
Module:GEPrices/data osrsw:Module:GEPrices/data Maps of item names to current GE price
Module:GEPricesByIDs/data osrsw:Module:GEPricesByIDs/data Maps item ID to current item price, as an additional shorthand
Module:LastPrices/data osrsw:Module:LastPrices/data Maps item name to the previous GE price
Module:GELimits/data osrsw:Module:GELimits/data Maps item name to GE buying limit
Module:GEValues/data osrsw:Module:GEValues/data Maps item name to item value (which determines alchemy price - every item in the Grand Exchange only)
Module:GEHighAlchs/data osrsw:Module:GEHighAlchs/data Maps item name to item high-level alchemy
Module:GELowAlchs/data osrsw:Module:GELowAlchs/data Maps item name to item low-level alchemy
Module:GEIDs/data osrsw:Module:GEIDs/data Maps item name to item ID (GE items only)
Module:GEVolumes/data osrsw:Module:GEVolumes/data Maps item name to item volume (only top 100 for RSW)

These are also lua tables. For raw text processing, they're in the format ["item"] = value,, with any " in the item name escaped as \" (I don't think there are any examples of this at time of writing, but just in case).

To parse these you can use the regular expression: \["(.*?)"\] = (\d+), where group 1 is the item name and group 2 is the item value or price. You should be able to just iterate the matches. (For GEPricesByIDs you need \[(\d+)\] = (\d+),.) Some of the modules will have the value being 'nil', which is the lua equivalent to null or undefined. This regex will ignore those, but you may need to account for that if you use a split-by-linebreak-and-parse method.

Module[edit | edit source]

Modules are lua code that runs as a layer above wikitext. Modules are generally easier to program complex things in, as lua is an actual programming language with proper variables, datatypes, loops, etc, instead of wikitext.

This guide assumes basic understanding of lua and how to use that on the wiki - see guide for more information.

Loading prices[edit | edit source]

As prices are stored in lua, there are a few ways to load them in to a module.

Recommended - Module:ExchangeLite

ExchangeLite is the recommended way to obtain GE prices. Usage is simple.

  1. First load in the module local exg = require('Module:ExchangeLite')
  2. When a price is required, it can be loaded using the exg.price(item) function, e.g. exg.price('Bones'). (This is syntactic sugar for #3).
  3. For any other field from the information module, use exg.load({args = { item, field }}), e.g. exg.load({args = { 'Bones', 'limit' }})

The primary downsides of the module are that there are no error checks and no title redirects, so you will have to implement those yourself if that is necessary. See example for error checking.

Sometimes useful - Module:Exchange

Exchange is the full module for loaded GE prices. As it is larger, it is not recommended for larger, more complex modules that use a lot of GE prices as it can cause additional computation time and memory usage over ExchangeLite.

The important functions are (given local exg = require('Module:Exchange')):

  • exg._price(item, multiplier, format, round) - item to get the price of, multiplier for the price, format the result with commas, round the result to X decimal places
  • exg._limit(item) - GE limit for item
  • exg._value(item) - internal value for item
  • exg._diff(item) - price difference for item
  • exg._exists(item) - does the exchange module for the page exist

The Exchange module does have error protection and title redirects, unlike ExchangeLite.

Risky - loading Module:Exchange/Item directly

Since the data pages are modules, it is possible to simply load them directly. This should be done with local bones_ge = mw.loadData('Module:Exchange/Bones') (see guide for require vs loadData).

This works just fine, but is not safe against future changes to the exchange page structure - or a complete rework/move away from the lua. It also has no error protection or redirects. This should be reserved only for modules that can hit the memory or computation limit, as it is very slightly faster than using ExchangeLite.

Example module[edit | edit source]

This module prints the name, price, and limit of an item supplied to it.

local p = {}
local gep = require('Module:ExchangeLite')

-- helper function for fetching data with error checking
local function get_data(item)
    local ret = {'', ''}
    local noErr, data = pcall(gep.price, item)
    if noErr then
        ret[1] = data
    end
    noErr, data = pcall(gep.load, {args={item, 'limit'}})
    if noErr then
        ret[2] = data
    end
    return ret
end

-- template entry point
p.main = function(frame)
    p._main(frame:getParent().args[1])
end

-- module/console entry point
p._main = function(item)
    local data = get_data(item)
    if data[1] == '' then
        return 'Could not find price for '..item
    end
    local l = 'Unknown'
    if data[2] ~= '' then
        l = data[2]
    end
    return string.format('%s: price = %s, limit = %s', item, data[1], l)
end

Wiki javascript[edit | edit source]

Wiki javascript usage of prices is essentially a subset of external usage, but with some additional advantages of being directly on the wiki. This covers methods that can only be used by being on the wiki.

Making the wiki do the loading[edit | edit source]

The easiest way to load prices with wiki javascript is to not load them with wiki javascript at all! It is simpler to, on the page the script is running on, have the wiki load the price and put it in a containing tag that can be found easily.

e.g. one could use <span class="example-calc-info" data-name="Bones" data-price="{{GEP|Bones}}">{{Coins|{{GEP|Bones}}}}</span>. This can easily be selected via $('span.example-calc-info').attr('data-price').

This can apply for both scripts that run on one page and scripts that run on large numbers of pages.

Javascript loading[edit | edit source]

If you need the javascript to load the information and can't put it into a wiki page, then there's a few ways to load the data. All the methods from #External usage also apply.

While using JS on the wiki, you have access to mw.Api which is generally slightly preferred over using $.ajax or similar, but is still fairly hefty and is probably only really worth using when more than just 'current status of item' is needed. The API can easily handle requests for multiple revisions, edit requests, etc.

As an additional note, once you have the content of the page by whatever means, the rs.parseExchangeModule() function can easily parse the module for you to get the price.

Two example usages, to get and parse current data:

// method one - via api
var api = new mw.Api();

api.get({
    action: 'query',
    titles: 'Module:Exchange/Bones',
    prop: 'revisions',
    rvprop: 'content',
    rvlimit: 1
}).done(function(res){
    var pages, content, data;
    pages = res.query.pages;
    if (pages[-1]) {
        //no page found
    }

    // raw wikitext of the page
    content = pages[Object.keys(pages)[0]].revisions[0]['*'];

    // object version of above
    data = rs.parseExchangeModule(content);
});

// method 2 - raw via $.get
$.get('https://runescape.wiki/w/Module:Exchange/Bones?action=raw').done(function(res, jqxhr){
    // res is the raw wikitext so we just have to parse it
    var data = rs.parseExchangeModule(res);
});


External usage[edit | edit source]

Again, there are several ways to fetch a price, many of which are shared by the on-wiki javascript section. This should work in any language (you'll need some sort of requests library, as well as regular expressions or JSON/XML parsing), but I'll be giving examples in javascript.

Google Sheets

While not particularly efficient, a quick and easy way to use prices in a Google sheet is the simple formula =IMPORTXML("https://runescape.wiki/w/Exchange:Bones","//*[@id='GEPrice']")

In an ideal world you'd write some custom script for the sheet in the script editor and use one of the below methods to fetch and parse the prices. But this is probably fine for the time being.

action=raw

The simplest way to query prices is to just send a get request to the action=raw version of the page, i.e. https://runescape.wiki/w/Module:Exchange/Bones?action=raw. You can then parse that from a lua module into whatever data structure.

API - revisions

Much like the API helper in on-wiki scripting, you can access stuff via the API directly. You can find full documentation on the API at https://runescape.wiki/api.php or at https://www.mediawiki.org/wiki/API:Main_page.

In short you'll want along the lines of https://runescape.wiki/api.php?action=query&titles=Module:Exchange/Bones&prop=revisions&rvprop=content&rvlimit=1&format=json - the key parts being prop=revisions and rvprop=content; titles=Module:Exchange/Bones is what page(s) you're fetching (you can bundle multiple together with | - remove rvlimit if you do that), and rvlimit=1 reduces output if you're querying just one page since you probably won't need more than the current revision. You can get the results in json or xml format, whichever you prefer.

Check out the returned data in your browser - the actual wikitext of the module is under result.query.pages[PAGEID].revisions[0]['*' ] for JSON (you'll need something like Object.keys(pages) to get the page IDs if you don't do another query ahead of time to fetch them). In XML its under api > query > pages > page[pageid=..][title=.. ] > revisions > rev[0]

Once you have the raw wikitext you can parse that like from action=raw into whatever data structure you like.

action=parse - code fragment

The parse action asks the server to parse the wikitext you pass it. So, we can use the methods of the templates section above in an external request.

An example of such a request is https://runescape.wiki/api.php?action=parse&disablelimitreport=1&prop=text&text={{GEP%7CBones}}. This asks the wiki to parse the wikitext {{GEP|Bones}} (which should return 554), and prop=text makes it only return the parsed text (the default includes some other stuff which is unnecessary). It also uses disablelimitreport =1 to disable the preprocessor limit report - this gives information on the limits of a parse request, which is a useful tool for debugging but not always necessary (eg see one with it enabled). (This limit report is present on every page - use inspect element to find it!) Like before, this is returned in json or xml format.

The returned parsed text in json is at result.parse.text['*'] and api > parse > text in xml. The parsed text is the escaped HTML of the main content of a wikipage (selector div#mw-content-text). This generally means that it will be surrouned by an escaped <p> tag. Thus, to actually use a price, you'll want to unescape and/or remove this to get the price within.

However, a potentially easier way is to apply some sort of marker characters and then just regex that. e.g. https://runescape.wiki/api.php?action=parse&disablelimitreport=1&format=json&prop=text&[email protected]:%20{{GEP%7CBones}}@ - you can use this regex to grab the price - /@Bones: (\d+)@/.

Indeed, it is easy to have many GE prices in one request this way - just add on to the wikitext. With https://runescape.wiki/api.php?action=parse&format=json&disablelimitreport=1&prop=text&[email protected]:%20{{GEP%7CBones}}@Fire%20rune:{{GEP%7CFire%20rune}}@Logs:%20{{GEP%7CLogs}}@ you can loop over /@(.+?): (\d+)@/g and for each store price of $1 as $2.

This leads me on to the next section...

action=parse - entire page

Instead of using text=... in a parse request, we can use page=PAGE NAME to parse an entire page.

I've set up an example page at User:Gaz Lloyd/parseme. It contains the same wikitext as the previous query, just one per line with a marker character at each end. We can make the API parse this with https://runescape.wiki/api.php?action=parse&format=json&prop=text&disablelimitreport=1&page=User:Gaz%20Lloyd/parseme. The output has an additional div tag, but is otherwise the same as before.

Thus, you can set up a page in your userspace listing all of the items you need data for with separators and GEP, and then just ask the wiki to parse that whenever you want to update all your prices.

If you want to collect mass data on every item, your best bet is to use DPL to generate the list for you (see [[@TODO|DPL guide]] for more information). I have made a short example at User:Gaz Lloyd/parseme2 - this uses DPL to fetch the item prices and names, and since DPL has a limit of 500 results per usage, there is a second usage with an offset of 500. You can add more to this, but I would expect you to hit the processing limit before getting every item done. Three ways around this: make extra pages with increased offsets, templatify the page and use the parse fragment repeatedly, or parse fragment the DPL directly and repeatedly change the offset there.

For example https://runescape.wiki/api.php?action=parse&format=json&prop=text&text=%7B%7B%23dpl%3Auses%3DTemplate%3AExchangeItem%7Cnamespace%3DExchange%7Ccategory%3DGrand%2520Exchange%7Cordermethod%3Dtitle%7Cformat%3D%2C%40%25TITLE%25%3A%2520%C2%B2%7BGEP%C2%A6%25TITLE%25%7D%C2%B2%40%0A%2C%2C%7Ccount%3D500%7Coffset%3D0%7D%7D (this URL requires encoding).