Possible solution to "/auc getall" disconnecting people
-
Please give http://norganna.org/up/show.php/ENED0z/CoreScan.lua a test (just drop it into your Auc-Advanced folder). It may be enough to solve the disconnect problems that some people see with "/auc getall".
edit: There's a newer version posted.
For the interested:
- I've basically nuked :HasAllData(), which was traversing all records in the AH at once, querying for owner names
- Instead, StorePageFunction() will now re-try (with a yield()) when it finds records with nil owners
- This should result in a more even stream of owner GUID<>name queries to the server, and may be enough to keep people from disconnecting. If not, I'll try with more intelligent rate throttling.
It should also solve APPR-95 (auction owner names sometimes going blank when you hit refresh in appraiser)
Oh yeah, it will also barf debug output spectacularily into your chatframe. You probably don't want to use it live :)
See my post below for info on how to test! -
Replacing the CoreScan.lua from aadv r2887 with the version you linked here does not fix disconnects for me when using /aadv getall at the Horde auction house on Mal'Ganis. If size matters, this AH typically has more than 400 pages on weekends and I've never seen it drop below 350 pages.
I don't see any debug messages printed in ChatFrame1 except for the initial "StorePageFunction!" and the usual "BottomScanner: Switching to Mal'Ganis-Horde config zone" messages when I open the AH window.
After typing /aadv getall, the auction window list is replaced with the text "Scanning..." and a status bar is shown with the text "AucAdv: Scanning". The status bar never progresses visually. No additional debug messages appear at any time. I am disconnected after approximately 10 seconds.
Edit: "Normal" page-by-page scans have always worked fine as long as I log out immediately after scanning to save the data. -
To chip in on this, I haven't had the opportunity to test this fix yet, however:
I HAVE noticed that when I get a disconnect using the getall feature, if I log back in, do a normal scan, then do a getall scan after a few minutes, I no longer disconnect. It seems entirely possible that your thoughts about it needing to wait for owner names could be the source of the problem have a lot of merit, as when I scan the AH normally, that's what hangs up each page. I'll see a page displayed, and it will remain displayed until all the names have shown up on that page. Sometimes there's quite a wait for that to happen. I imagine if it were trying to wait for all the names to populate for the whole AH it just might cause a timeout, and thus a disconnect. I'll test the fix to see if it helps me, I suspect it will because of the behavior I've noted above.
Thanks for all the work on this! -
so far i have not had any disconnects using this fix from mikk. i think your on the right track with the names being a problem as i have noticed as well that it hangs until the names have all been updated
(on a side note the refresh on the appraiser tab doesn't get the names updated unless i do a refresh 2 times in a row first time it will update the auctions then the second time it will get all the sellers names.) -
To test properly:
1. Log out
2. Log in
3. Talk to auctioneer without doing anything else
4. Did you get a "StorePageFunction!" debug output? If not, you haven't installed my patch properly.
4. Run "/aadv getall"
Take note of what debug output you get. Are you getting a "StorePageFunction!"? Are you getting any "Yield #(number)" Keep looking at the output. If you get disconnected, report if you saw "StorePageFunction!" or not, and the highest "Yield" number you see before a disconnect.
For a proper run, you should be getting PAGES of "Yield #(number)", often at 500-number intervals, while your "Scanning" progress bar is running.
When "Processing" has begun, all server communications are completed, and if you get disconnected HERE, something very strange is going on.
Game restarts should be no different from relogging. Reloading is not enough.
If you do get disconnects, try running a full normal (page-by-page) scan, and then "/aadv getall" after that (WITHOUT relogging). This SHOULD NOT disconnect you under any circumstances. If it does, I'm dead wrong about the cause for the disconnects. Please do include if you get disconnected from a normal scan + aadv getall or not.
Of course, there's a chance of false positives here. If you're up to it, please reinstall the regular CoreScan.lua, log out, log in, and run /aadv getall and see if that disconnects you.
On another note: If you're still getting disconnects with my version of CoreScan.lua and you're on an EU realm, please let me know which one it is (and which faction) so I can get some testing done there myself.
Thanks! -
Also note that BtmScanner does its own page scanning, so running it concurrently with a getAll scan will only result in erroneous results.
-
To test properly:Posted By: Mikk
1. Log out
- done
2. Log in
- done
3. Talk to auctioneer without doing anything else
- done
4. Did you get a "StorePageFunction!" debug output? If not, you haven't installed my patch properly.
- yes
4. Run "/aadv getall"
- no additional debug output; disconnected from server after approxmiately 10 seconds
Take note of what debug output you get. Are you getting a "StorePageFunction!"? Are you getting any "Yield #(number)" Keep looking at the output. If you get disconnected, report if you saw "StorePageFunction!" or not, and the highest "Yield" number you see before a disconnect.
- I see no StorePageFunction! at all other than the first one when logging in
For a proper run, you should be getting PAGES of "Yield #(number)", often at 500-number intervals, while your "Scanning" progress bar is running.
- I don't get any Yield messages, presumably because scanning fails immediately
When "Processing" has begun, all server communications are completed, and if you get disconnected HERE, something very strange is going on.
- unfortunately, I never get past "AucAdv: Scanning"
Game restarts should be no different from relogging. Reloading is not enough.
- I did log out and log back in
If you do get disconnects, try running a full normal (page-by-page) scan, and then "/aadv getall" after that (WITHOUT relogging). This SHOULD NOT disconnect you under any circumstances. If it does, I'm dead wrong about the cause for the disconnects. Please do include if you get disconnected from a normal scan + aadv getall or not.
- While doing a normal scan, I do see several Yield messages on each page. Once I'm done with the normal scan (took about 15 mins to scan and then process 433 pages tonight), running /aadv getall without even closing the AH window produces the same immediate disconnect.
Of course, there's a chance of false positives here. If you're up to it, please reinstall the regular CoreScan.lua, log out, log in, and run /aadv getall and see if that disconnects you.
- The regular CoreScan.lua from r2877 produces identical behavior, except of course I do not see the initial "StorePageFunction!" debug message when opening the AH window.
On another note: If you're still getting disconnects with my version of CoreScan.lua and you're on an EU realm, please let me know which one it is (and which faction) so I can get some testing done there myself.
- Unfortunately I'm on the US-Mal'Ganis realm (Horde AH). I can't help but wonder if the size of our AH could be contributing to this. I will try running /aadv getall at our neutral AH (which usually has only 3-4 pages) and see what happens.
Thanks! -
Okay here are the results of running /aadv getall at our Neutral AH on US-Mal'Ganis (I didn't even have to run a normal page-by-page scan first):
StorePageFunction! <-- the initial one when opening the AH window
StorePageFunction! <-- after typing /aadv getall
Yield at <a href="/index.php?p=/search&amp;Search=%231&amp;Mode=like">#1</a>
Yield at <a href="/index.php?p=/search&amp;Search=%231&amp;Mode=like">#1</a>
Yield at <a href="/index.php?p=/search&amp;Search=%232&amp;Mode=like">#2</a>
Auctioneer Advanced finished scanning 63 auctions:
0 items in DB at start (0 matched query)
0 unchanged items
62 new items
0 updated items
0 items removed
It seems that maybe size really does matter. Have you tested /aadv getall successfully at an AH with over 20,000 auctions? If so, and it worked, I need more debugging information about what's going on immediately at the beginning of the scan, since on the Horde AH I get disconnected immediately. -
Okay, I just tried /aadv getall on a more moderately sized realm (US-Darrowmere-Horde) with only about 10,000 concurrent auctions. And all I have to say is "HOLY CRAP!" it only took 37 seconds to scan and process 10,000 records!
(... lots of these)
Yield at <a href="/index.php?p=/search&amp;Search=%239458&amp;Mode=like">#9458</a>
Yield at <a href="/index.php?p=/search&amp;Search=%239458&amp;Mode=like">#9458</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2310050&amp;Mode=like">#10050</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2310050&amp;Mode=like">#10050</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2310050&amp;Mode=like">#10050</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2310050&amp;Mode=like">#10050</a>
StorePageFunction!
Auctioneer Advanced finished scanning 10232 auctions:
50 items in DB at start (50 matched query)
50 unchanged items
10085 new items
0 updated items
0 items removed
97 filtered items
0 missed items
10135 items in DB at end
37 Secs Spent Scanning Auction House
I really, really, REALLY want to get this working for the Mal'Ganis AH data. Since I started out with an empty image on this new server, I'll try purging my Mal'Ganis data and starting from scratch there. -
Okay, I backed up then deleted Auc-ScanData.lua from my SVs and tried doing /aadv getall again on US-Mal'Ganis Horde. I didn't run a page-by-page scan first.
(lots of these)
Yield at <a href="/index.php?p=/search&amp;Search=%2318986&amp;Mode=like">#18986</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2318986&amp;Mode=like">#18986</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2319512&amp;Mode=like">#19512</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2319512&amp;Mode=like">#19512</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2319707&amp;Mode=like">#19707</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2320233&amp;Mode=like">#20233</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2320233&amp;Mode=like">#20233</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2320248&amp;Mode=like">#20248</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2320840&amp;Mode=like">#20840</a>
Yield at <a href="/index.php?p=/search&amp;Search=%2320840&amp;Mode=like">#20840</a>
StorePageFunction!
Auctioneer Advanced finished scanning 21307 auctions:
0 items in DB at start (0 matched query)
0 unchanged items
20704 new items
0 updated items
0 items removed
603 filtered items
0 missed items
20704 items in DB at end
1 Mins 32 Secs Spent Scanning Auction House
It worked! At the very beginning, it looked like I was about to be disconnected. Nothing moved for about 8 seconds. It then magically did its thing. I'm going to log off for 15 minutes, then log on and try it again and see what happens.
Edit: After logging off for 15 minutes, when I log back in and try /aadv getall, I get disconnected again. :( I've tested 3 times now, and I'm successful on Mal'Ganis only after I've deleted Auc-ScanData.lua from my SVs. Doing a page-by-page scan before a getall scan doesn't help, so I'm not sure if asking for owners is causing the problem for me. For some reason, having data for 20,000 items already in the saved scan image is causing issues.
On Darrowmere, I am able to do a new /aadv getall without deleting the SVs first. Its scan image contains data for approximately 10,000 items.
Is there any preprocessing of the saved scan image going on that could be blocking my client from acknowledging the receipt of the query results with getall? Is there a way I could combine getall with a category ID to try it with a smaller subset of items? (It seems the API should allow this, but I'm not sure there's an /aadv command that supports such a scenario.)
Edit yet again: I'm pretty sure now it's not the size of the saved scan image that's causing problems. I think it's just the size of the query results being sent back. I'm occasionally able to use getall now with 20,000+ items already in the saved scan image. Sometimes it disconnects, sometimes it doesn't. As long as I see the first "SavePageFunction!" within about 45 seconds of typing /aadv getall, things seem to progress with no problems after that point. I still think it would be useful to be able to combine catid/subcatid and other query parameters with getall (assuming the API allows it) for further testing. -
I still think it would be useful to be able to combine catid/subcatid and other query parameters with getall (assuming the API allows it) for further testing.Posted By: Nandini
The API does allow it (it allows anything allowed on a regular search query, as getall is merely adding a TRUE flag as an additional argument) Please add a JIRA for this.
Edit: now that I think about it, we probably don't want to do getall on the categories, due to the 15min delay between getalls. If you feel differently, please discuss.
BTW, Have been testing on Greymane-Alliance (typically 16-20k itmes), and have been disconnected once out of about 10 getalls. It was about halfway through scanning. -
Hey guys,
Computer science student at North Texas and long time user of auctioneer/auctioneer advanced. I've been having disconnect issues with the getall function since its release and decided to test out the yielding function to see if it would solve it. I'm on Norgannon-alliance testing with an approximate 350 page AH on peak time 3:00 cst sunday.
Mac OS X Leopard 10.5.2
MacBook Pro 2.33 GHz Core 2 Duo
2GB 67 MHz DDR2 SDRAM
1.5 mb down/ 300 kb up connection on PPPoE DSL provided by AT&T SBC Yahoo
Azureus BitTorrent client running in the background capped at 100kb/s - 90ms ping in game, very playable even during raids.
Testing done:
Starting from my large dataset compiled over 6 months of using AADV with all stat modules active, over 48 hours since last full scan. I attempted /auc getall with the packaged version of CoreScan.lua 5 times, disconnected each attempt before any scan or processing progress showed on bar.
Switched the CoreScan.lua with your test version and attempted to getall 5 more times, disconnected every attempt with the only debug info showing as "StorePageFunction!" once when opening the AH.
Deleted Auc-ScanData.lua from WTF/Account/ACCOUNTNAME/SavedVariables
Same results except an additional debug message of "Warning: replacing auc-scandata with auctioneer advanced scan data" or something like that, disconnects followed promptly.
Deleted ALL Auctioneer saved variables files from WTF/Account/ACCOUNTNAME/SavedVariables and tried again.
Scanned 17048 auctions successfully using a blank dataset. Logged out for 15 minutes, logged back in and reattempted, Disconnected on 5 successive attempts.
HUGE NOTE
I exited Azureus at this point, leaving no background programs or connections running.
All successive /auc getall calls have been successful with a 17500-17900 item AH.
From only End User testing without poring over the code it seems there may be an issue with overwriting or recalling some portion of the existing dataset while having background downloads or programs running. Does this have to do with some Blizzard dynamic network throttling, or is it something in AADV?
Just wanted to throw this in for dev consideration, love Auctioneer and I figured this might be useful in making it even better. -
Very interesting, Genshoku. That would seem to indicate that the disconnect is because of fighting over network bandwidth.
Not sure if there's anything we can do, as addons can't do any network handling, but the Blizz devs might be able to track down what's going on from their side. -
After further testing on several different realms with AH sizes of 8000-21000 items it seems that this version of the CoreScan appropriately addresses the disconnects due to excessive server requests and personally confirms that I need to stop pirating things before i scan the AH. Could be that Azureus is utilizing the same port as the AH download, not confirmed though. Would this be ready for a clean up and preview release on the next build?
Some observations:
The overall scan process, although about twice as long as the preview version release of the CoreScan, is much smoother and causes less FPS problems for me. I assume this is because the calculations are being throttled over a slightly longer time frame, given that data will be yielded every few hundred auctions scanned.
I still have a 5-10 second lag spike when doing a getall, after which the game does its speed up act and everything resumes as normal. Is this a pause in everything to ensure the quickest download of AH data or something else?
Starting from scratch (new server/faction/deleted variables) gives a much faster processing time. -
I did add a suggestion on Jira (http://jira.norganna.org/browse/ADV-191) for combining at least some query parameters with GetAll. As I mentioned, it would be useful for testing. From additional feedback here, it appears that available bandwidth and network latency may be contributing to server disconnects with /aadv getall.
Restricting the query to a particular category might help establish a threshhold for the number of auctions returned that will work for users with a particular combination of bandwidth and latency. Also, you mentioned the 15 minute limit blackout on using GetAll; if you log out of your character and back in, this timer appears to be reset to zero. It wouldn't present an additional barrier to testing, especially if one had already become disconnected. :)
One thing that seems to be increasing my odds of success with GetAll is using a different character stationed at a little used AH in a lightly traveled area, and leaving all chat channels. I did this to try to minimize or eliminate other events or data the server may be trying to send my client (and expects an acknowledgement for) while the huge transmission of query results is being sent to my client. For Horde players, Silvermoon City seems to be an excellent place to do this. -
As for using the same ports, you'd have to look at your firewall logs to determine that.
The scan progress isn't actually taking longer, though the period during which the progress bar moves is longer. What's happening is that the pausing for names is during the scan, instead of what was happening in the build version, which was the 20-30s wait at the beginning during which the progress bar didn't move at all. (so the progress bar is actually a better indication now of how far along it is)
the 5-10 second lag spike at the beginning of the getall is the client (and Blizz's UI code) processing all the information coming in at once. It happens even without any addons loaded. -
For myself... The old version used to disconnect me almost every time. This seems to work almost everytime.
If I relog and run it it disconnects me more often compared to my normal behaviour of checking my mailbox before scanning. I believe this is because checking my auction mail populates many of the common names in the AH. -
It sounds to me like I might be on the right track then. But to actually solve it all, it looks like I'll have to make an actual rate limiter on how many queries for unknown items we send per second. It's going to slow down the scan some more but .. hey, rather that than a disconnect.
-
Okay, here's a new version that takes into account query bandwidth (14 bytes per query btw) and applies the same limits that ChatThrottleLib does (except it doesn't actually use CTL - so chat spam + this can still disconnect you).
Unfortunately for most of you this is a lot slower than it COULD be. My own server supports upwards of 3000 cps even at peak time. But there's several that don't even support 1000. CTL, and this test, caps the bandwidth at 800 cps.
http://www.norganna.org/up/show.php/pAFwsl/CoreScan.lua
To test: relog, run /aadv getall. If I'm right, this should never disconnect anyone.
This takes about a minute for me to complete with 15k entries on the AH, vs ~15 seconds earlier. It's quite possible that the finished product should have a "safety" slider where people can choose to speed the process up if their connection supports it. -
anyone using this version and still getting discoed: do you have wowecon installed? it appears that wowecon will merrily cycle through the whole auction list doing something, which completely bypasses the safety checks we're trying to use here to avoid disconnects.
if you are getting discoed, try logging out (NOT reloading), disabling wowecon, then log back in and try it again. -
I am still getting disconnected when doing a /auc getall it seems to happen if i haven't scanned in a while(like 4 hours+) it must have something to do with the names or something as if you don't scan for awhile the auctions get canceled and new ones get posted so auctioneer doesn't know who posted them. just my input. hope there is a fix as i love the getall feature.
-
Been using this successfully since my last post. I notice that it always yields twice at auction 1 and then once more sometime before auction 10, followed by yields every 200-400, down from yields every 500-700. If it cannot retrieve a name from the server for some abnormality in the AH data it spams "Yield at 4859" about 50 times before saying it can't retrieve the name and moving on, i assume this is intended or just a result of the message output placement in the code. Any progress on a slider or something to adjust the bandwidth limit?
-
Yes. It works occasionally, but generally even if I do a full normal scan first it will _still_ disconnect me doing getall (on alliance side only, horde works 80% of the time).
I have between 300 and 3000 latency. I also get disconnected any time I logout or just /rl (again, with or without wowecon).
Messing about with my network card settings hasn't helped (though it is likely they are related network issues). -
I've been Using the version of CoreScan.lua Mikk posted on March 4th, and haven't been disconnected once during scanning or processing. The few times I have been disconnected were before the actual scanning began. I think this is when the server is sending the very large result list back to my client, and has only happened during times of high latency or bandwidth usage. I can avoid this always by scanning on my unguilded alt who is positioned near an unpopular AH (Silvermoon City) and has left all chat channels. The timing of this occasional initial disconnect may indicate that Blizzard is being too aggressive in sending the search results to my client. I'm not sure if it's possible for them to throttle this communication to allow clients to receive and acknowledge other incoming events, but perhaps someone could suggest it to them.
-
There are a few things that need looking at with getall because getall can not be paused but many of the other parts of auctioneer assume that the scan can be paused. (For example hitting the refresh button on the appraiser page.)
Mikk: Last night with WowEcon running on my main character I managed to getall successfully after emptying my mailbox. 25000 auctions at 700ms. -
Thats an excellent idea! Mikk, ping!
-
Just a question. I am NOT using mikk's fix. i did a normal scan on one of my characters.logged out,logged into another char on the same account/same server, went to the ah and tried to do a /auc getall and disconnected. if the names were the issue i should already have all the names of the sellers as i just scanned less then 2 minutes before the getall command was tried. Is this normal?
Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Categories
- All Discussions12,551
- ↳ Announcements41
- ↳ Auctioneer::News105
- ↳ Gatherer::News50
- ↳ Support::Auctioneer6,116
- ↳ Support::Gatherer778
- ↳ Support::Enchantrix458
- ↳ Support::BeanCounter195
- ↳ Support::Informant164
- ↳ Support::Other365
- ↳ Auctioneer::Discussion1,529
- ↳ Auctioneer::Documentation128
- ↳ Auctioneer::Technical717
- ↳ Gatherer::Discussion204
- ↳ Gatherer::Development34
- ↳ Gatherer::Documentation9
- ↳ Gatherer::Technical115
- ↳ BeanCounter::Discussion32
- ↳ Enchantrix::Discussion4
- ↳ Enchantrix::Technical9
- ↳ SearchUI::Discussion405
- ↳ SearchUI::Technical173
- ↳ Glypher::Discussion62
- ↳ Development::Applications118
- ↳ Development::Design47
- ↳ Development::Staff Goodies7
- ↳ Development::Testing399
- ↳ Development::Translations35
- ↳ Development::Wiki25
- ↳ General::Chit Chat88
- ↳ Mutty::Support0
- ↳ Mutty::Discussion1
- ↳ AuctioneerDB::Support58
- ↳ AuctioneerDB::Discussion60
- ↳ Watcher::Discussion20
Norganna's AddOns Network ·
World Of Minecraft ·
WoM Realms ·
Auctioneer Addon ·
Gatherer Addon ·
Addon Forums ·
RDRCT