mGalaxy forum

General Discussions => Bugs Reports => Topic started by: vonblubba on January 22, 2019, 08:51:40 AM

Title: Scraper issue
Post by: vonblubba on January 22, 2019, 08:51:40 AM
Ok, I'm re-posting this on the bugs section, since I got no answer whatsoever in the "need help" section. Maybe it will be picked up by devs sooner or later, who knows.

I'm trying to setup a mame-only system. I think I've done everything right, but during step 2 of db update (image/media scraping) I keep getting an "invalid parameter" error, followed by a "The database update has been stopped etc...". If I resume the update, it keeps running for a while, but then the errors show up again. It seems fairly random, I can't pinpoint any possible causes  :-\

Any idea what the cause may be, or where I can find some kind of log file?

My system is a Windows 10, running mGalaxy 7.0.0.0. I'm trying to setup mame 0.193 with full romset. The rom files are on a network partition on my NAS, mounted as a network drive in windows 10.
 
Title: Re: Scraper issue
Post by: mgalaxy on January 22, 2019, 09:02:04 AM
 Which option are you using to scrape in Runway: thegamesdb or screenscraper ?
Title: Re: Scraper issue
Post by: vonblubba on January 22, 2019, 09:15:22 AM
Tried both. Same issue.
Title: Re: Scraper issue
Post by: jmd on January 23, 2019, 05:03:14 AM
Quote
My system is a Windows 10, running mGalaxy 7.0.0.0. I'm trying to setup mame 0.193 with full romset. The rom files are on a network partition on my NAS, mounted as a network drive in windows 10.
Scraping MAME file works perfectly well with me (I know it won't help you ;) )...have you tried to run a test with local files (instead of network drive) to see how it gets and help the developer help you?
Title: Re: Scraper issue
Post by: vonblubba on January 23, 2019, 07:56:57 AM
Thanks for the suggestion, I'll try that as soon as possible.
Personally I think that instead of trying things more or less at random it would be more useful to know how to enable/access the application log, so to at least have an idea about where to look for the issue.
Title: Re: Scraper issue
Post by: vonblubba on January 24, 2019, 05:01:05 AM
Ok, unfortunately I've waited as long as I could to find a solution to the issue. Im' experiencing other minor problems, but there's no point in mentioning them, I suppose. Moving back to the "attract mode" frontend.
I consider the money I spent on mGalaxy as a kind of support to the arcade community as a whole, so that's ok. But add my name to the "unhappy customers" list for mGalaxy frontend (not because of the issues, those are expected in this kind of environment, but because my pleas for help went basically unanswered, and in my experience that's really unusual in a community as lively as the arcade one).

Thanks again to @jmd for the suggestions and see you around!
Title: Re: Scraper issue
Post by: jmd on January 24, 2019, 09:22:36 AM
What a pity...you've waited 2 days and are ready to leave us (and the ease of use, great themes..) !? ;) Have you tested with local files, it's easy to test!?
Title: Re: Scraper issue
Post by: mgalaxy on January 25, 2019, 11:35:32 AM
Hi there,
Just coming back from a short holiday time and I discover your thread.
You have to know that we do our very best to address all questions asked there. I simply was away for a few days and counted on some brave people here to help in the meantime.
So, if you're still willing to receive some help...it'll be my pleasure :)
I would first exactly recommend what was proposed by jmd!
Scraping media is achieved first by checking CRC of every file (to be sure that proper results are returned later). In the case of MAME, with thousand roms (of sometimes heavy weight) that could take some time on some system...but it will be worst if done by the network (even if we did never receive any complain on the subject)
May I please ask you to create a temp MAME folder with just a few (local) roms...and check how it's going?
Title: Re: Scraper issue
Post by: vonblubba on January 26, 2019, 12:43:16 PM
Hi,

sure, I'll take all the help I can get :-)

I've already tried what JMD suggested, but to no avail. Error is still there.
I don't think the error arises during CRC calculation, since some media files do get downloaded. Sometimes it keeps downloading for 10/15 minutes before stopping. Other times it hangs right away.
I thought about network issues, but I checked every setting on my router and everything seems ok. I also have no other netrwork issue whatsoever with other programs/devices. Moreover, other scrapers from other frontends work just fine.

I really don't know what else to try here.

Thanks in advance for any assistance.
Title: Re: Scraper issue
Post by: jmd on January 27, 2019, 09:43:09 AM
Quote
I don't think the error arises during CRC calculation, since some media files do get downloaded. Sometimes it keeps downloading for 10/15 minutes before stopping. Other times it hangs right away.
Oh! Ok, so you're able to scrape some media then it stops after a while (from the first message I thought that it didn't work at all!)
Just curious to know: in another post you're speaking of another frontend that you're using and that works "well"...what is this frontend and what is the source it scrapes from?
Title: Re: Scraper issue
Post by: mgalaxy on January 27, 2019, 09:49:07 AM
VONBLUBBA,
I'm extensively checking for your problem...but can't reproduce it here.
I am writing a special version to debug this particular problem and will propose it for you to download and test if you agree!?
Title: Re: Scraper issue
Post by: mgalaxy on January 27, 2019, 10:08:45 AM
OK, I got it...seems to be a chance in the way the info is sent from the database server...I check
Title: Re: Scraper issue
Post by: vonblubba on January 27, 2019, 12:04:00 PM
@JMD scraping works well with launchobx and attract mode, so I don't think it's a network issue.

@mgalaxy of course I'm available to try a debug build :-)
Title: Re: Scraper issue
Post by: SegaDoesWhatNintenDont on January 28, 2019, 01:28:39 PM
I would like to contribute to this thread because I'm having constant crashes with mGalaxy_Runway 7.0 on Windows 10 Home. Here's what I did:

1. fresh install of mGalaxy 7.0
2. Setup paths to respective folders in Runway
3. clicked update using the GamesDB server (notification: submitted games 781)
4. clicked update with Screenscraper selected...

Runway quickly goes through all the rom names, then says updating info for file: "whatever game starting with A..", goes through two or three names, lingers on the last one for 10 seconds or so and then just closes without any alert.

5. Trying again. This time Runway stops at the first game and closes.
6. Trying again. This time Runway stops at the second game and closes....

Now I can play the game of restarting Runway and hitting update to have one more game updated with scraped info each time I launch the app...
Title: Re: Scraper issue
Post by: SegaDoesWhatNintenDont on January 28, 2019, 04:32:53 PM
One thing that makes Runway crash are comments in the zip files, apparently. This still doesn't change anything about the above mentioned behavior, but in case one zip contained the additional comment property Runway would crash on the same zip every time... since I'm not getting any comments.. should I open a new thread for this?
Title: Re: Scraper issue
Post by: vonblubba on January 29, 2019, 01:14:15 AM
@SegaDoesWhatNintenDont so in your case it just crashes? No error messages whatsoever? It's probably a different issue then, I suppose. Does it manage to download any media file at all?

Good catch about the zip comments, though. I didn't think about that.
Title: Re: Scraper issue
Post by: SegaDoesWhatNintenDont on January 29, 2019, 09:44:30 AM
Yes, the window just closes. No message.

I tested mGalaxy on a different PC now, using a fake rom folder with about 10 zipped text files. All of them showed up in the games list and had screenshot, videos and logos scraped... I know now that all the features CAN work and I'm capable of setting everything up, but it doesn't solve my problem of course..

My PC at home is quite fast, while the wifi is OK-ish. I will do a test tonight, unzipping all the roms and rezipping them, thus getting rid of any comment properties or whatever.

If that doesn't work I'll test the fake rom approach to see if my install of windows 10 is giving mGalaxy trouble.
One thing is for sure, though: I tested the same roms with Launchbox, which worked like a charm without any crashes every time. So even with possible workarounds it is still mGalaxy that would benefit from being more compatible or forgiving with whatever is the issue here...
Title: Re: Scraper issue
Post by: mgalaxy on January 29, 2019, 09:51:13 AM
One thing that makes Runway crash are comments in the zip files, apparently. This still doesn't change anything about the above mentioned behavior, but in case one zip contained the additional comment property Runway would crash on the same zip every time... since I'm not getting any comments.. should I open a new thread for this?
could you please send me such a zip file?
Title: Re: Scraper issue
Post by: mgalaxy on January 29, 2019, 09:54:30 AM
Still trying to isolate the 'bug' while scraping, all of a sudden, without modifying anything, everything is working properly (tested during the last 6 hours)...the problem might come from a change in the way the screenscraper database is serving files...I continue to check
Title: Re: Scraper issue
Post by: SegaDoesWhatNintenDont on January 29, 2019, 03:33:27 PM
Hi, thanks for looking into it!

Here's one of the files that contain a comment. Unfortunately that alone does not seem to be the issue. I created a separate folder with  30 roms only and made sure to delete those with comments.. The crashes still occur.

As mentioned before it is not always the same rom. Usually the next time a rom will be scraped succesfully and the rom after that crashes the software. It's almost as if scraping all components at once is too much so that Runway needs several attempts per rom?
But it makes no difference if I select all components to be scraped or just the screenshots, for example.

Oh, the rom is 1,75 mb. Is there a way to send it? (Attachment  limit seems to be 1200 kb)
Title: Re: Scraper issue
Post by: SegaDoesWhatNintenDont on January 30, 2019, 07:07:25 AM
Quick update: I checked my roms on a different machine (running Win 7). I got only minor issues and one crash while I was able to scrape 700+ roms. So apparently mGalaxy doesn't play nice with my Win 10 setup for whatever reason...
Title: Re: Scraper issue
Post by: mgalaxy on February 01, 2019, 04:03:01 AM
Please find this beta version that should fix all inconsistent stops while scraping from Screenscraper.
Simply extract over the mGalaxy folder and replace all existing files.
Please report here how successful or not it then gets! ;)
Title: Re: Scraper issue
Post by: vonblubba on February 02, 2019, 11:17:43 AM
I started a new scraping process for the entire romset. It's been running for a couple of ours (a few hundreds roms scraped) without an issue. So far so good! I'll keep you posted on the final result.

Thanks for the effort, however this goes!
Title: Re: Scraper issue
Post by: vonblubba on February 07, 2019, 03:45:24 AM
I can confirm that scraping of the whole romset finished without any issues.

Thanks again for the effort!
Title: Re: Scraper issue
Post by: mgalaxy on February 07, 2019, 04:18:04 AM
I can confirm that scraping of the whole romset finished without any issues.

Thanks again for the effort!
Thanks for reporting, we are always committed to the satisfaction of our users!