Norganna's AddOns

We have a JIRA issue to do so for the incremental scans we do already: http://jira.norganna.org/browse/ADV-505

But we don't have one to disable a full category or getall scan, yet. Given the 15 minute lockout, It's a little weird to not take advantage of a GetAll, regardless of which addon initiated it, though I suppose that weirdness should take a back seat to issues people have with game crashes due to Blizzard bugs. So I've created http://jira.norganna.org/browse/ADV-594

Appreciate it.

A minor point of correction: Auctioneer is not crashing the game. A bug in the game client is crashing the game.

I'm not an expert on the inner workings of Auctioneer but my addon just checks if the number of auctions in a getall scan is equal to 42554 and if so it knows the bug occurred and can just exit safely so I wonder if auctioneer could do this too? I assume auctioneer uses the 2nd return value of GetNumAuctionItems("list") which is a very high and random number which is what's causing the problem...where for a getall the first number will be correct even if the bug occurs.

I added a JIRA for better management of the 42554 bug as well.

I did a little digging through Auctioneer's scanning code and found that you hook a bunch of the AH related API functions (I'm mostly concerned with QueryAuctionItems) and that the original function is stored globally. Any reason I couldn't do the following (atleast until a proper API is put in)?

local QueryAuctionItems = AucAdvanced and AucAdvanced.Scan.Private.Hook.QueryAuctionItems or QueryAuctionItems

Thanks

The correct solution (I talked about this in the IRC channel ages ago) would be to write a small library that does the very basics of AH talking. And get AH addons to use it.

The hooking race can only end messily.

Basically what such a library would do is accept queries and do callouts to interested parties (including the caller) that "hey, this query was just run, here's the results". Probably with some sort of lockout mechanism so that you can tell the library if you're finished with the page or not (think having to click for buyouts).

Quick design suggestion would be something along the lines of:

lib:QueryAuctionItems(...)
lib.RegisterCallback(context,"Results"/"Resolved"/"Unlocked",callbackFunc)
lib:CanSendAuctionQuery()
lib:IsResolved(items,sellers) - check items/sellers are resolved in the resultset
lib:ResolveAuctions(items,sellers) - do the whole item/seller resolving
lib.Lock(context)
lib.Unlock(context)
lib:IsLocked()

^ notice . vs : above. This is for embedding by addons using that.

"callbackFunc" receives the following parameters:
(context, ...) "..." comes from QueryAuctionItems

(Hint this is the exact syntax used by the CallbackHandler lib.)

Basically I expect the following to happen:
1. Someone calls QueryAuctionItems()
- lib:Lock() <- notice taking a lock on itself<br />2. Results are returned from server
3. Library does "Results" callouts to N interested addons
- One or more addons may call :ResolveAuctions().
- We only remember if any wanted "items" and/or "sellers".
- Callouts finish
(steps 4--6 obviously optional)
4. Check if someone wanted resolving. Perform it.
5. Resolving finishes
6. Library does "Resolved" callouts.
7. lib:Unlock() <- unlocking itself<br />
Locking works by tracking locks per context.
If an Unlock clears the table of locks, an "Unlocked" callback fires.
QueryAuctionItems refuses to work while the library is locked.

:QueryAuctionItems
----------------
The library should work by hooksecurefunc()ing the original _G.QueryAuctionItems global and triggering its logic from there on.

The builtin :QueryAuctionItems() would mostly be about checking if a query can run now due to locks/CanSendAuctionQuery and returning "no" / calling _G.QueryAuctionItems() and returning "yes".

:CanSendAuctionQuery
------------------
1. Get the 2 results of _G.CanSendAuctionQuery
2. If lib:IsLocked(), set the first return to false
3. return the values

(Yeah this is mostly convenciance)


I see one problem here and that is multiple addons waiting for "Unlocked". If CallbackHandler is used, all addons will get it, even if the first addon re-locks the lib. Hrm, probably shouldn't use CallbackHandler for that callback eh.



obtw: i'm not offering to write this :D

Interesting idea. I was actually thinking of doing something similar a while back but never ended up doing it and although it certainly wouldn't be that hard to do, I wouldn't offer to write something like this atm either as I don't really have time....

The real reason for disabling auctioneer is not to prevent it from getting data (which is a side effect of preventing it from scanning and would be solved by the library idea), but to prevent it from doing its processing after each item query and thus slowing down the other addon's scan, or even crashing wow as is the case with the getall scan bug (or "was the case" if it's been fixed).

Auctioneer does not crash wow. Wow crashes wow.

I was mistaken that Auctioneer crashes wow because of the getall scan (although Auctioneer's processing after a getall scan does slow me down to 7fps from 120). Regardless, I still eagerly await a method for preventing Auctioneer from doing its processing during another addon's scan and slowing down said other addon's scanning.

Thanks

On second thought, I'm considering writing the library Mikk discussed. Mikk maybe we could talk in more detail about it over IRC? I'm "Sapu" on freenode and quakenet.

mikk is mikk on freenode.

http://jira.norganna.org/browse/ADV-598

Question. Are there any addons besides Auctioneer that do real-time scanning? By that I mean scanning queries besides the addon's own queries.

Sapu,

Not sure, off hand but would recommend checking in with Zirco, Udorn, and MiralKilrogg of Auctionator, AuctionMaster, and AuctionLite, respectively, to see about their interest/need. The only other is perhaps LilSparky, of LilSparky'sWorkshop and GnomeWorks, who does a lot with pricing, and may hook something, too.

All are available via Curse's PM system and/or WoWAce and are very cool, responsive types.

Nikk B-)

Still haven't heard from Zirco but I'm pretty sure the answer to my question is no.

So, don't mean to bump an old thread but...today I was informed (and upon further inspection noticed) that the 5.11 preview version of Auctioneer makes all the original functions (I'm mostly worried about QueryAuctionItems) local; preventing any other addon from using anything but Auctioneer's replacement functions. This seems like a very deliberate move to prevent any possible way of making an addon that does AH queries without Auctioneer intruding and doing it's own scanning and processing at the same time.

Let me explain why I never went through with the scanning library explained by Mikk in this thread. The main reason was simply it only made the problem worse. It seemed like the purpose was to block addons even more from interfering with auctioneer scanning every last auction that was queried. Put simply, it allowed an addon the ability to completely lock all other addons out from querying the auction house which, considering Auctioneer is the only addon I could find that scans / processes queries other than their own, seemed pretty self-serving.

Now I don't mean to downgrade the work that has gone into Auctioneer as I don't think anybody will argue that it has certainly been one of the most revolutionary and widely used addons in the game and I'm not trying to take away from that. But, I don't understand the reasoning behind not making sure that other addons can be used side by side with Auctioneer. It would be a real shame if somebody had to pick between Auctioneer and another addon that does AH queries rather than have them both work side-by-side.

A simple solution to this would be to just implement an API (or even setup a global variable) that other addons could use to tell auctioneer to pause it's scanning / processing until the other addon is finished with its queries. I still haven't seen (or read from a dev) any downside to doing this. A compromise might even be to do some light scanning of the auctions. Meaning don't do any processing or queries at all while the other addon is doing its scan, but still scan all the pages and just store them for later processing (when the other addon is done you could even have a popup asking the user if they want Auctioneer to process that data or not).

Thanks,
Sapu

PS: oops guess I have two accounts here...Sapu and Sapu94

Reasoning? Quite simple: Reduce server stress from constant auction scanning, while maintaining high data availability to the end user.
To put it mildly, very mildly... WoW code base is ancient, was never intended to handle more than a handful of players per server, and weren't rewritten to be more efficient when it became apparent that the game popularity is rising. It seems just like Blizzard following the very common trend of MMO development: "Never ever rewrite your game code, milk it to the bottom then throw away. Patch only ridiculous bugs. Nobody care."
Everything Norganna team (and some other people) do is working around limitations in game protocol and language (I mean Lua) specifics.
Solution? Be nice and use Auctioneer API, when it present.

This has nothing to do with server stress, limitations of lua, or any bugs with the game at all. In fact, if you want to put it in those terms, this is Auctioneer limiting how effectively other addons can use the auction APIs that blizzard provides. Where is the harm in allowing a user to decide when they want to do a quick 10 second getall scan to quickly update min buyout prices with another addon or when they want to wait a few minutes for Auctioneer to do its processing after the scan is complete?

If an addon is negatively effecting another addon, isn't it pretty straightforward that if the end goal is to be able to use both addons simultaneously, the amount of negative interaction should be decreased as much as possible? Or is that not the end goal? Is the end goal of Auctioneer to better itself over anything else? Over the wishes of the user?

"Solution? Be nice and use Auctioneer API, when it present. "

I asked for such an API 3 and a half months ago. I realize as a programmer myself that some things just don't get done for numerous reasons, so in the mean time I was happy with just getting at the original function (AucAdvanced.Scan.Private.Hook.QueryAuctionItems) while I waited for a proper API. Of course than that the ability to do that was removed and here we are. As far as the "being nice", no comment other than I wish Auctioneer would be nice to other addons.

Users who don't wish Auctioneer to collect data except when a full scan is initiated already have that option, it's a setting accessible via the in-game config gui.

The only place I see this not working is GetAll scans, and I believe our logic with getall scans is that, since they can only be done every 15 minutes, we don't want to miss one, but I'm not one of the developers who spends much time with the scanning core.

Because of the extremely poor implementation on Blizzard's side of the getall code, we absolutely do not want to try to collect data on a scan that does not go through our scanning engine, as every other scanning engine I've looked at is guaranteed to be missing huge amounts of data from the scan (they just don't actually check that the data is complete, or notify the user when it isn't). We have a lot of code to minimize the missing data, and without that, it makes the statistics extremely tricky.

The only real way to allow collection of data on all full scans, which is currently a priority of the addon, is to hook the functions universally, as we have done.

I do wonder, however (and I'm just shooting from the hip here) if it might be feasible to add a terminal flag to our calling of QueryAuctionItems that is a boolean that would allow the calling code to request that Auctioneer ignore the request and pass it straight through to QueryAuctionItems and otherwise ignore the scan. I would also, however, suggest that if we did that, we also allow the user the option of whether or not to honor that flag, via a toggle in our configuration. Just as frustrating as an Auctioneer that refuses to rest is to you would be another addon having the ability to tell Auctioneer to take a break without me being able to say, "No, I don't think I will, I really want that data."

Like I said, I don't work on the scanning engine much, so I can't speak to the reasons for the changes there, except that I expect that the hooking race may have been contributing to corruption in our scan data tables, but that's simply conjecture on my part, as is most of my post here.

If nothing else, I'll see if I can ping one of the devs who does work on the scanning engine to the thread for more discussion.

Where is the option to have Auctioneer not collect data except on a full-scan? By full-scan do you mean an auctioneer scan? I have never seen such an option and if it exists that would definitely solve the issue.

A terminal flag for QueryAuctionItems would also solve the issue.

EDIT: Found the option....testing it out now.

By full scan, I mean a full scan of the AH (rather than a scan for a specific item, or of a specific sub-category). It doesn't directly address your concern, but may improve life for you in the immediate future while any discussion about this continues to happen.

Ok, so I disabled the "Search while browsing" option which is an improvement for non-getall scans.

So, the issue is really just with getall scans. A possible solution would be to have a terminal flag to the QueryAuctionItems as you suggested and then have an option in Auctioneer to override that terminal flag (turned off by default). Then, when the flag is set and Auctioneer is not set to override it, Auctioneer could print out a message or put up a popup that says "Hey some addon is preventing auctioneer from getting data on this scan. If this isn't what you want go to the options and check the override button."

Like I said, I don't work on that portion of the addon, but I've pinged those that do to come check out this thread when they get the chance.

I've chatted with the other devs for a bit, and we've discussed allowing the user to set, via the in-game config, if he wants Auc to watch while other addons interact with the AH. This will default to the current behavior.

We'll likely provide a simple API for other AH addons to call if they detect Auctioneer present to present the user with a notice instructing them how to disable the behavior if they feel Auctioneer is interfering with their other AH addons.

This way, if we relocate the option or change its functionality at all, other authors won't have to worry about keeping up with our changes, we can just change the message presented to the user as we change our on code.

That sounds great. Will that also apply to getall scans? Or perhaps a sub-option for getall scans so a user could have Auctioneer only interact on getall scans of other addons if they wish?

It will be all-or-nothing. Either Auctioneer will watch all searches, or it will watch only its own, depending on the user's choice. No one is stepping up to actually do the coding, so it may end up being me doing it, and I'm in the middle of moving across the country right now, so this may not happen for a month or more, but if no one has picked it up when I'm done moving I'll start tinkering around.

I could pick this up. I had thoughts about doing something about http://jira.norganna.org/browse/ADV-505 which sounds to be the same thing, but never got round to it.

I think I see a quick way of doing it, though it would be nice to streamline the whole process in the long run.

sapu: In the Preview Download now is the first stage, simply an option for the user to disable scanning on non-auctioneer searches.

As yet I've not done any interface functions for other AddOns - I'm still troubleshooting the basic change. If we don't find any basic problems I'll be looking at what we can do for that :)

Sounds great. I'll grab it tonight and test out the option. What sort of function are you thinking about for other addons to call? Some way to tell if that option is checked or not would be great. Or possibly an API for another addon to turn on that option with a confirmation box from Auctioneer asking if the user wants to permanently turn it on, turn it on just this session, or keep it turned off. Just some ideas. Thanks for your work on this.

The option is worded so that when it is "On" it scans all AH searches; you turn it "Off" to only scan Auctioneer-initiated searches.

I was planning an API at least so that another AddOn can check the status of this setting, but not sure exactly what might be best to do beyond that.
However I was thinking of making it a generalized function, the idea being that if we implement any more compatibility tweaks in future they would all be handled by the same function.

"However I was thinking of making it a generalized function, the idea being that if we implement any more compatibility tweaks in future they would all be handled by the same function."

Unless you can foresee today all classes of compatibility tweaks that might be added in the future.... I'd just say "don't do it" tbfh =)

sapu: Have you tested this yet? Does it actually fix the conflict with your own AddOn?

Sorry I've been pretty busy lately. I just tested it and it seems to be working as best as I can tell. Thanks :)