[Ohrrpgce] Crash reporting

Sun Sep 9 19:59:38 PDT 2018

Ah yes. You need to install any version of Visual Studio, cv2pdb is
dependent on a .dll from it, unfortunately, because as I said PDB isn't
officially documented, so that's how cv2pdb solves the problem of creating
a .pdb file.
Actually I thought you already had VS installed on the nightly build
machine, to build gfx_directx.dll.
I tried manually copying the .dll next to cv2pdb.exe, but it didn't run.
Probably because the dll is using LoadLibrary to open other dlls - VS has
hundreds of them - although it doesn't statically require other dlls, so I
couldn't easily figure out what it required. Also, the dll isn't
distributable anyway.
I'm still using Visual Studio Express 2010.
Alternatively, since recently Microsoft now distribute "Visual Studio Build
Tools" which is their compiler toolchain without the IDE, so you could
install that instead.

On 10 September 2018 at 05:19, James Paige <Bob at hamsterrepublic.com> wrote:

> Okay, Progress! A little, anyway.
>
> The Windows VM boots, attempts to build, and shuts down correctly.
>
> The actually builds are still failing though, this time because it says a
> PDB DLL file is missing.
>
> ---
> James Paige
>
>
> On Sun, Sep 9, 2018 at 1:18 AM, Ralph Versteegen <teeemcee at gmail.com>
> wrote:
>
>> Oh, and it occurs to me that you're going to need to manually do an 'svn
>> update', because the script dies before it gets that far.
>> Also, manually updating svn will also make it skip building, because
>> it'll think there are no new commits. That's would be fine, although it'll
>> prevent you from actually seeing whether it's working or not.
>> Also, I forgot to ask you to delete the old nightly build .zips, for
>> builds which are no longer done or have been renamed.
>>
>> On Sun, 9 Sep 2018 at 17:07, Ralph Versteegen <teeemcee at gmail.com> wrote:
>>
>>> OK, that was enough to figure out what was going on.
>>>
>>> Quoting in batch scripts is really a nightmare!
>>> If you write
>>>  SET ISCC=C:\Program Files (x86)\Inno Setup 5\iscc.exe
>>> then the brackets cause a parsing error.
>>> If you write
>>>  SET ISCC="C:\Program Files (x86)\Inno Setup 5\iscc.exe"
>>> and then do
>>>   ECHO %ISCC%
>>> it prints the filename with quotes. But if you do
>>>   IF NOT EXISTS %ISCC% ( ...
>>> you get an error due to the unquoted (x86) in the filename, although it
>>> is quoted!
>>> So if you try instead
>>>   IF NOT EXISTS "%ISCC%" ( ...
>>> that expands to ""C:\Program Files (x86)\Inno Setup 5\iscc.exe"" and now
>>> cmd.exe DOES process the quotes, the double quotes apparently cancelling
>>> out, and cmd.exe complains about 'Files' being unexpected!! Talk about
>>> inconsistent!
>>> It turns out that the only possible way to get everything to work is to
>>> write
>>>   SET "ISCC=C:\Program Files (x86)\Inno Setup 5\iscc.exe"
>>> which does protect the (x86) but doesn't put the quotes in the variable
>>> itself, so that yuo can then write "%ISCC%" without causing double quoting.
>>>
>>> I take back every single nice thing I said about batch scripting!
>>>
>>> This is another reason why I want to convert most of our distrib scripts
>>> into python!
>>>
>>> On Sun, 9 Sep 2018 at 10:04, James Paige <Bob at hamsterrepublic.com>
>>> wrote:
>>>
>>>> So far I can see that the VM starts up, runs distrib-nightly-win.bat
>>>> inside c:\nightly\ohrrpgce\ and then immediately quits.
>>>>
>>>> The only output in c:\nightly\distrib-out.txt says:
>>>>
>>>> Searching for support programs...
>>>> \Inno was unexpected at this time.
>>>>
>>>> On Sat, Sep 8, 2018 at 6:35 AM, James Paige <Bob at hamsterrepublic.com>
>>>> wrote:
>>>>
>>>>> If I have time this weekend, I'll boot up that VM in interactive mode
>>>>> and investigate :)
>>>>>
>>>>>
>>>>> On Saturday, September 8, 2018, Ralph Versteegen <teeemcee at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Looks like I managed to break Windows nightly builds somehow. It's
>>>>>> not even sending email with the build log anymore!
>>>>>>
>>>>>> On Fri, 7 Sep 2018 at 01:03, Ralph Versteegen <teeemcee at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I see that the last nightly build failed.
>>>>>>>
>>>>>>> Firstly, there's this error:
>>>>>>>   'nightly\ohrrpgce' is not recognized as an internal or external
>>>>>>> command,
>>>>>>>    operable program or batch file.
>>>>>>> I saw this sort of crazy thing a lot while testing. "svn update"
>>>>>>> must have modified the copy of distrib-nightly-win.bat that was being
>>>>>>> executed! cmd.exe kept the same position in the file, but its contents
>>>>>>> changed, so it was suddenly in the middle of a line.
>>>>>>> I had thought that a separate copy of the source code was used to
>>>>>>> provide the nightly build scripts and that you had to manually update them
>>>>>>> if they changed, but I guess you changed that at some point. (I thought
>>>>>>> that comment at the top was obsolete!)
>>>>>>> I guess this isn't worth fixing, but just keep it mind it might
>>>>>>> cause more (temporary) build failures in future.
>>>>>>>
>>>>>>> All other errors are also caused by the same thing. The first few
>>>>>>> lines of the old .bat file and the rest of the new .bat were executed, so
>>>>>>> the necessary environmental variables didn't get set. The next nightly
>>>>>>> build should hopefully work correctly.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 6 Sep 2018 at 13:14, James Paige <Bob at hamsterrepublic.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Awesome! I'll take a look and see if I need to change anything on
>>>>>>>> the build VMs or on the dreamhost account when I get a chance. This has
>>>>>>>> been a very busy week, but I have been reading all your commit messages
>>>>>>>> with a smile on my face :D
>>>>>>>>
>>>>>>>> ---
>>>>>>>> James
>>>>>>>>
>>>>>>>> On Wednesday, September 5, 2018, Ralph Versteegen <
>>>>>>>> teeemcee at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> OK, I'm finally done with changes to the distrib scripts. I don't
>>>>>>>>> know how I managed to spend so many commits on cleaning them up...
>>>>>>>>> James, set the value of $SCPSYMBOLS, the location to upload the
>>>>>>>>> symbols archives to, to your liking. It's set in three different files.
>>>>>>>>> Although I renamed some scripts, that shouldn't affect anything.
>>>>>>>>>
>>>>>>>>> Hmm, I should make the windows nightly build script skip the build
>>>>>>>>> if there have been no svn commits, like the mac nightlies already do.
>>>>>>>>> (I'm not sure but I don't think we can easily reuse a .pdb file
>>>>>>>>> for two builds made from identical source code, because the .exe hash won't
>>>>>>>>> necessarily match)
>>>>>>>>> Skipping unnecessarily rebuilds would allow rebuilding the builds
>>>>>>>>> more than once per 24 hours (manually, or by cron).
>>>>>>>>> (I tried to let you still manually force additional early builds
>>>>>>>>> as long as there's been an svn commit, by putting the commit number in the
>>>>>>>>> archive name. But I just realised that won't work if you do a  manual build
>>>>>>>>> and then the scheduled build happens after, without any commits between. )
>>>>>>>>>
>>>>>>>>> On Tue, 4 Sep 2018 at 15:48, Ralph Versteegen <teeemcee at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> OK, great!
>>>>>>>>>> I think it would definitely be a good idea to get the php script
>>>>>>>>>> to forward reports to ohrrpgce-crash at hamsterrepublic.com so they
>>>>>>>>>> don't get forgotten (assuming there isn't a flood of them). Unless they are
>>>>>>>>>> too large, I guess - someone could try to include a large .rpg file in the
>>>>>>>>>> report. Is your email hosted by dreamhost? Then it seems that I can just
>>>>>>>>>> send email from another hamsterrepublic.com address to
>>>>>>>>>> ohrrpgce-crash using php's mail() function, according to the dreamhost docs.
>>>>>>>>>> And, I'll put the script in the web/ directory in svn.
>>>>>>>>>>
>>>>>>>>>> What about uploading the exe/pdb archives, where should they be
>>>>>>>>>> put? I guess I'll name them something like ohrrpgce-symbols-win-$RELEASE-$BACKENDS-$DATE-$SVNREV.7z
>>>>>>>>>> (I'll put 7z.exe in support/) and they can all go in one directory, and old
>>>>>>>>>> files can be deleted manually when we do a release, or automated later.
>>>>>>>>>> Incidentally, I notice that only mac nghtly builds skip building
>>>>>>>>>> if there are no changes to svn, linux and win builds build regardless.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, 4 Sep 2018 at 03:07, James Paige <Bob at hamsterrepublic.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Okay, you can now configure crashrpt to submit its reports to
>>>>>>>>>>> https://rpg.hamsterrepublic.com/crashrpt/
>>>>>>>>>>>
>>>>>>>>>>> They get saved in /home/james_paige/crash_reports/ which is not
>>>>>>>>>>> web accessible.
>>>>>>>>>>>
>>>>>>>>>>> TMC, if you send me an ssh public key, I can give you ssh/sftp
>>>>>>>>>>> access to that.
>>>>>>>>>>>
>>>>>>>>>>> Feel free to make whatever changes you like to the nightly build
>>>>>>>>>>> scripts, and let me know what I need to change, either on the VMs or on the
>>>>>>>>>>> dreamhost account
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 3, 2018 at 8:34 AM, James Paige <
>>>>>>>>>>> Bob at hamsterrepublic.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Anything that gives better crash reports sounds great to me.
>>>>>>>>>>>>
>>>>>>>>>>>> Increasing the download size a bit is not a big problem. I
>>>>>>>>>>>> think our downloads are already comparatively tiny.
>>>>>>>>>>>>
>>>>>>>>>>>> I also don't expect any problem with storing a few gigabytes of
>>>>>>>>>>>> exes and crash related data. My dreamhost account might be stingy about
>>>>>>>>>>>> processor resources, but it is very generous about storage space.
>>>>>>>>>>>>
>>>>>>>>>>>> I'll see what I can do about setting up that php script. We can
>>>>>>>>>>>> use it over either http or https since I have LetsEncrypt certs for all my
>>>>>>>>>>>> domains
>>>>>>>>>>>>
>>>>>>>>>>>> This is going to be a busy week, so if I don't get to it today
>>>>>>>>>>>> it will have to be next weekend. (I am making myself a reminder)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sunday, September 2, 2018, Ralph Versteegen <
>>>>>>>>>>>> teeemcee at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I've heard of quite a few reports of various kinds of crashes
>>>>>>>>>>>>> which I don't have enough information to do anything about, so am forced to
>>>>>>>>>>>>> ignore. (Mostly they've been reported on the SS Discord server which even
>>>>>>>>>>>>> has a dedicated channel for reporting bugs!).
>>>>>>>>>>>>>
>>>>>>>>>>>>> My previous attempt to generate stack traces on Windows
>>>>>>>>>>>>> required somewhat large extra dlls to parse the DWARF debug info in
>>>>>>>>>>>>> game/custom.exe, and they can't provide line numbers, which would require
>>>>>>>>>>>>> huge totally-unstripped builds. Line numbers would help hugely.
>>>>>>>>>>>>> So the new plan is to instead generate minidumps which get
>>>>>>>>>>>>> sent back for analysis. Win XP and later have a function to generate
>>>>>>>>>>>>> minidumps (and the dll can be downloaded for older Windows versions if
>>>>>>>>>>>>> needed) The minidump includes the stack of each thread, so you can use it
>>>>>>>>>>>>> to generate traces with line numbers and inspect variables. Optionally you
>>>>>>>>>>>>> can include more parts of the process memory like global variables, but
>>>>>>>>>>>>> that makes it large.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've been trying out an open-source reporter, which sends
>>>>>>>>>>>>> reports over the internet with minimal user will needed:
>>>>>>>>>>>>> http://crashrpt.sourceforge.net/
>>>>>>>>>>>>> There are a couple other such reporters, like BreakPad
>>>>>>>>>>>>> (originally from Mozilla, and used by Firefox and Chrome) but they seem
>>>>>>>>>>>>> pretty heavy-weight and more difficult to use.
>>>>>>>>>>>>>
>>>>>>>>>>>>> CrashRpt requires a separate .dll (I doubt I would be able to
>>>>>>>>>>>>> statically link) plus CrashSender.exe which does the actual work, and an
>>>>>>>>>>>>> ini file.
>>>>>>>>>>>>> Zipped, these are 300KB in total (I compiled a custom build to
>>>>>>>>>>>>> strip out the extravagant ability to record ogg theora videos and jpgs!).
>>>>>>>>>>>>> So that's fine for including with Custom in support/. But we wouldn't want
>>>>>>>>>>>>> these extra files when distributing games. However, if you have a full
>>>>>>>>>>>>> distribution of the OHRRPGCE, then any copy of game.exe can reuse the
>>>>>>>>>>>>> CrashRpt files from support/. If CrashRpt isn't available, game.exe can
>>>>>>>>>>>>> still easily save a minidump itself and ask the user to email it manually.
>>>>>>>>>>>>>
>>>>>>>>>>>>> == How this will work ==
>>>>>>>>>>>>>
>>>>>>>>>>>>> CrashRpt does not generate stack traces. It only collects data
>>>>>>>>>>>>> and sends it off. You can tell it which files to attach, and it includes
>>>>>>>>>>>>> some metadata and a minidump and can include a screenshot.
>>>>>>>>>>>>>
>>>>>>>>>>>>> fbc -gen gas generates STABS debug symbols, gcc generates
>>>>>>>>>>>>> DWARF symbols, and Visual C++ generates PDB symbols, which live in a
>>>>>>>>>>>>> separate .pdb file. PDB is undocumented, but recently LLVM folks have been
>>>>>>>>>>>>> creating documentation by examining code Microsoft open-sourced. So GCC and
>>>>>>>>>>>>> GDB don't support PDB or minidumps.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We can use the cv2pdb tool to convert DWARF to .pdb files,
>>>>>>>>>>>>> which we keep. The executables themselves get stripped, and also need to be
>>>>>>>>>>>>> kept; a .pdb alone is useless. (Currently we partially strip exes, leaving
>>>>>>>>>>>>> in function symbols so gcc can create a backtrace. I don't think that's
>>>>>>>>>>>>> worth keeping anymore, except in the debug build. By fully stripping, we
>>>>>>>>>>>>> can reduce each exe by another 300KB.) cv2pdb doesn't convert FB strings
>>>>>>>>>>>>> correctly, but those are on the heap so not in the minidump anyway.
>>>>>>>>>>>>>
>>>>>>>>>>>>> To examine the minidumps, you can use Visual Studio (the free
>>>>>>>>>>>>> versions will do) or windbg (CLI), or use a commandline tool or server-side
>>>>>>>>>>>>> to generate a stracktrace and other info from it. CrashRpt has such a tool,
>>>>>>>>>>>>> and scripts for handling large numbers of crash reports and sorting them.
>>>>>>>>>>>>> BreakPad also has tools for it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> == What we need to do ==
>>>>>>>>>>>>>
>>>>>>>>>>>>> Firstly the .pdb files should get uploaded somewhere. We will
>>>>>>>>>>>>> want to keep the .pdb files and .exes for every nightly build - not just
>>>>>>>>>>>>> the weekly archived nightlies! .pdb files alone are unfortunately useless.
>>>>>>>>>>>>> I'm going to get rid of most of most of our windows nightly
>>>>>>>>>>>>> builds, they're redundant now that you can switch graphics backend. So
>>>>>>>>>>>>> leaving just music_sdl, music_native, music_native2 and music_sdl-debug.
>>>>>>>>>>>>> (In fact, music_native and music_native2 appear broken and I couldn't find
>>>>>>>>>>>>> anyone who uses them or even remembers using them.)
>>>>>>>>>>>>> We should keep the files for at least every default
>>>>>>>>>>>>> (music_sdl) build since at least the last stable release, probably longer
>>>>>>>>>>>>> since not everyone upgrades promptly. Using xz, the game+custom exe+pdb
>>>>>>>>>>>>> files compress to 2.4MB. So 4 builds, 3 months of backups is ~1000MB. **
>>>>>>>>>>>>> An idea to save space: upload just the music_sdl files since
>>>>>>>>>>>>> the last stable release, (~250MB) and keep all the other files on the build
>>>>>>>>>>>>> machine in the unlikely event they're needed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> (** however I just remembered that exes from the nightly build
>>>>>>>>>>>>> machine are significantly larger than ones built on my machine)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Secondly we need to collect reports - CrashRpt can send via
>>>>>>>>>>>>> HTTP, HTTPS, SMTP and by opening an email client. I assume dreamhost will
>>>>>>>>>>>>> block emails from random IPs, so we will want HTTP.
>>>>>>>>>>>>> There is an example php script to accept reports and save them.
>>>>>>>>>>>>> http://crashrpt.sourceforge.net/docs/html/sending_error_repo
>>>>>>>>>>>>> rts.html#httpsend
>>>>>>>>>>>>> I guess it would be easiest if we modify this script to email
>>>>>>>>>>>>> us the zip file. And keep a copy on the server in case something goes
>>>>>>>>>>>>> wrong? Can we put us up at hamsterrepublic.com?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Ohrrpgce mailing list
>>>>>>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherh
>>>>>>>>>>> amster.org
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>> Ohrrpgce mailing list
>>>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherh
>>>>>>>> amster.org
>>>>>>>>
>>>>>>>
>>>> _______________________________________________
>>>> Ohrrpgce mailing list
>>>> ohrrpgce at lists.motherhamster.org
>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>>
>>>
>> _______________________________________________
>> Ohrrpgce mailing list
>> ohrrpgce at lists.motherhamster.org
>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>
>>
>
> _______________________________________________
> Ohrrpgce mailing list
> ohrrpgce at lists.motherhamster.org
> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.motherhamster.org/pipermail/ohrrpgce-motherhamster.org/attachments/20180910/951228dd/attachment-0001.html>