Forum:My database code is finally ready

From the RuneScape Wiki, the wiki for all things RuneScape
Jump to: navigation, search
Forums: Yew Grove > My database code is finally ready
This page or section is an archive.
Please do not edit the contents of this page.
This thread was archived on 28 April 2014 by Cqm.

I've been working on a general client-side searchable database for the wiki (mainly for stuff like bestiaries, but it's highly versatile), and it's finally done. Thanks to everyone who helped with technical stuff. At the moment, a test case can be found at User:Yitzi/test, drawing from a 10,000-element database I made for stress testing. (It takes a few seconds to load due to the sheer size of that table, but once loaded isn't slowed down noticeably unless you generate and show a large table of results.)

Regarding how to make a database and query form:

  • The syntax for the interface page (where you go to query the database) is:
database  = {insert title of database page here}
form      = {ID of the location where the query form should appear}
result    = {Same for the location where the result tables should appear.}
  • The database is determined by the table contents (or, more precisely, the contents of tr objects) on the database page. The page can include as much information (for instructions for use, explanations, etc.) as you want, without affecting the database itself, so long as it isn't inside a tr object.
  • The tr objects should ideally be put inside a table to make the database easier to see directly, but this is not actually necessary for functionality. I'll assume from here on that they are, and that it is a single table.
  • The table MUST be made as an HTML table, using tr, th, and td markup, rather than a wiki table.
  • The first row of the table should be made of all th elements; these will not only form the header for the tables returned by the queries, but will also be the categories by which the database searches. All other rows must be made up of all td elements.
  • The database can handle internal links using wiki markup, and of course the HTML table itself, but other than that it does not accept markup: HTML markup is removed from queries, and wiki markup is presented as plaintext.

Regarding how to use the query form (this should be fairly obvious for the most part, but it's better to have it clear just in case):

  • Upon load, the query form will display a number of lines (one for each column of the table), with the name of the property associated with that column and three options:
    1. Restrict by this property is just what it sounds like: If this is not checked, the query ignores that column when determining which lines to return. Running the query with none of these checked will return the entire database table.
    2. The next is a drop-down menu, to determine whether to allow only those lines that DO contain one of the values entered, or to allow only those lines that contain a value OTHER THAN the values entered.
    3. The third is the actual place where the allowed (or disallowed) values are to be entered. Both in the database and the query, it ignores case and trims spaces; a dash between two numbers is taken as including all numbers between them, and a comma is taken as separating different values. An empty string is considered a value, so you can search for empty entries by checking the "restrict" box, leaving the dropdown menu at "allowed values", and leaving the value box blank.
  • At the bottom, you can assign a name for the table of results. Every table MUST have a unique name; if you re-use an existing name, it'll tell you that the name is already in use and not do the query. (The empty string is a valid name, so you can do one query with no name entered, but for more you'll need names.) Next to the name box is the button to actually perform the query.
  • The query will return only those rows that match all the given conditions.
  • Once the query is performed, it creates a sortable table consisting of only the rows that match the query, but by default only the headers are shown. (This allows multiple tables to be created and manipulated without having to actually render, or scroll past, what might be a fairly large table.) Below the table are two buttons: One to show the table (or hide it if it is currently being shown), and one to permanently delete it (freeing up its name.)
  • Below that are modification options. This is for if you want, for example, to show all entries that fit one of two different queries. This has two lines: The first determines how to modify the table (it can be copied without change, added to another table, subtract another table from it, or take the intersection with another table), and the name of the table to modify by. The second line determines the name of the new table, and has two buttons: One to replace the current table and another to make a new table. As before, names must be unique, so you can't use an already-existing name unless you replace a table with another one that has the same name.

The code can currently be found at User:Yitzi/wikia.js; unfortunately, when I tried to post it here it thought it was a disallowed link and wouldn't let me post it. But the wiki administrators can go to that page and get it, look it over, and include it.  —The preceding unsigned comment was added by Yitzi (talk) on 20:42, February 6, 2014‎.

Link to script - User:Yitzi/wikia.js.


Comment - A proper usage example would be really helpful, I'm not 100% sure how to go about interacting with it as a user, let alone use it in a page. I'm guessing js isn't what you're used to looking at it's structure, but you avoided some common pitfalls like making every function global. Some improvements that could be made:

  • Write human readable code. This means whitespace between things like var foo=bar or if (foo<=bar) { and ideally indenting a little more than 2 spaces. MediaWiki comes with a minifier that strips all whitespace and comments so you don't have to worry about using too many spaces. Try to be consistent about how you use whitespace as well so someone can copy it when altering it (if necessary).
  • Comments - they're not for you, they're for the people who will eventually have to fix/change your code. I don't want to spend all day taking code apart piece by piece with a dozen console.log() call to figure out how everything works and then fix the bug. Document your functions with what they do, what you pass to them (parameter by parameter) and what they return. I realise at this point in time this isn't an issue, but as someone who went through almost all the wiki js and css to remove things that don't do anything anymore, comments are really helpful sometimes.
  • Don't iterate through arrays with loops, see here for an explanation. That link also show the correct way to iterate through arrays in javascript.
  • Be careful with != null. It's not a strict check and can return false positives.
  • You don't need to use the domain when building links for the same domain. /wiki/PAGENAME is fine.
  • It's a good idea to use wrap anything that interacts with the DOM in $(function () {...});. This makes sure the DOM is fully loaded before you interact with it. It's not really necessary given it's not going to be imported until the import in MediaWiki:Common.js is run (which won't happen until after the DOM is ready) but it's something to bear in mind for the future.

I don't expect you to use MediaWiki coding conventions, or even follow what's in [[MediaWiki:Common.js]] exactly, just something a little more user friendly in term of structure would be a step in the right direction Wink

I'm a little curious as to why it doesn't accept wikitable syntax. It's not really hard to extract the content from one.

Other than that, it looks good =D cqm 00:55, 7 Feb 2014 (UTC) (UTC)

Also, I wouldn't worry too much about using $.trim(). I'm guessing you're using it over String.prototype.trim() due to lack of IE8 support. cqm 01:07, 7 Feb 2014 (UTC) (UTC)
-I'll try to get to making a usage example, via a sample bestiary (I'll only add a few monsters, though.) Then you can look at the result and let me know what, if anything, remains unclear.
-I generally prefer two spaces because I find it makes the code readable while minimizing the danger that it will have to go to another line (which decreases readability); still, as you say it's for other people rather than for me, so how many spaces indent for each loop/condition/function would you suggest?
-As for consistency, I'm pretty sure I was consistent: Two spaces indent for each loop, condition, or function, with the curly brackets either on their own line or together with only parentheses/semicolons.
-Regarding comments: I'll try to add more: What each argument does and what it returns should be doable.
-I'll fix the thing.
-In what situation can a !=null return a false positive for a regex? What would you use instead?
-I'll fix the domain thing.
-As for why it doesn't extract wikitable syntax, it's because I'm not just extracting content. I'm taking the row as-is and using it in the result table, and that doesn't work with wikitable syntax (or at least I don't know how to make it work). I could convert it, of course, but that'd be more work and I figure there are probably more important projects.
-Actually, the reason I used $.trim is because that's what I found in another function and decided to copy it...but the lack of IE8 support is a good reason to use it anyway. (Clearly I need something, so that "value1, value2" will match "value2" and not just " value2".)
Yitzi (talk) 01:16, February 7, 2014 (UTC)
  • I think four spaces is the generally accepted indentation across Wikia. 2 spaces is something I've only ever come across on enwiki js. Tabs is another alternative, but bear in mind most people don't have access to them without a script or copy/pasting them every time they need an indent. Or they can use a proper script editor (again, not something many people will have).
  • There's some small discrepancies:
// near the top of the script
// whitespace around operators/assingments
var database = '';
// then further down
var queryMins=[];

// whitespace around parens
// near the top again
for (var i in lines)
// then near the bottom
for ( var i in pages ) 
  • !=null is fine for regex (as it returns null or an array), but it will also return true if it's undefined. I couldn't see any strict comparisons in your code so wasn't sure if you were aware there could be any problems from it or knew enough to not bother.
  • You'll find Array.prototype.indexOf used in [[MediaWiki:Common.js]] which replaced $.inArray() not long after Wikia dropped support for IE8. I can't remember if they've dropped support for IE9 yet, but it's not far off if it hasn't happened already. Anyway, I'd be surprised if IE8 would get as far as importing your script after encountering .indexOf.
  • One other minor thing, the parens in return(x) aren't necessary. Most js uses return x instead. I'm told it's used in C/C++, but it's a little confusing to make a return statement look like a method in js. cqm 13:24, 8 Feb 2014 (UTC) (UTC)
Ok, all the code changes are done; next step is to create a somewhat more convenient usage example (though for a simple usage example that also demonstrates performance, you can look at User:Yitzi/test, and its database at User:Yitzi/testtable). Yitzi (talk) 17:38, February 9, 2014 (UTC)
And a usage example, using the free-to-play bestiary, is up at I tried to make things self-explanatory; if they aren't, just let me know what needs to be made clearer. Yitzi (talk) 19:46, February 11, 2014 (UTC)

Comment - Are there any plans to implement this database anywhere? I'm happy adding it to the common.js, but if it's not going to be used there's not a lot of point.

A quick note on the UI - is it possible to move the buttons for each result table to the top of the table rather than the bottom. It seems a bit odd to show a table and have to scroll all the way down just to collapse it. The existing collapsible tables have their trigger at the top, it seems logical to follow that precedent. cqm 22:52, 6 Mar 2014 (UTC) (UTC)

Also, should this be added it would be nice to have a proper documentation page. The script should be moved out of your /wikia.js too, even if it's just to something like /database.js. cqm 22:54, 6 Mar 2014 (UTC) (UTC)
1. I can write up a corresponding database for the bestiary, though it may take a bit of time. So that's my next project.
2. I can definitely move the buttons to the top above the table header, but I suspect you meant the top below the table header...which should still be doable, though it'd be trickier.
3. I can definitely move the script to /database.js, and can do a documentation page; where would you like the documentation page? Yitzi (talk) 12:26, March 7, 2014 (UTC)
So at the moment it's something like this on the example f2p bestiary:
-> table of data

-> form for generating results

// when a table is generated
-> buttons for table

// when a table is uncollapsed
-> table
-> buttons for table
Simply swap the order of the last 2 :)
Assuming your code is on /database.js, /database would be a logical place for the docs. Make sure there's a note of where the documentation page is in the code too, I'd put it up with the opening nowiki and pre tags so it's easy to find. cqm 15:56, 7 Mar 2014 (UTC) (UTC)
Ok. The tricky part, though, is that it's not quite like what you showed. What it really is is:
-> table of data

-> form for generating results

// when a table is generated
-> table caption
-> buttons for table

// when a table is uncollapsed
-> table with caption
-> buttons for table
So the tricky part would be having the buttons after the caption (which is the name for the table) but before the table itself. It could be done, it just isn't quite as simple. 18:24, March 7, 2014 (UTC)

Closed - User:Yitzi/database.js is now lazy loaded where required. Good luck in converting over the bestiary :) cqm 17:40, 28 Apr 2014 (UTC) (UTC)