[Ohrrpgce] Crash reporting

Sat Sep 8 15:04:25 PDT 2018

So far I can see that the VM starts up, runs distrib-nightly-win.bat inside
c:\nightly\ohrrpgce\ and then immediately quits.

The only output in c:\nightly\distrib-out.txt says:

Searching for support programs...
\Inno was unexpected at this time.

On Sat, Sep 8, 2018 at 6:35 AM, James Paige <Bob at hamsterrepublic.com> wrote:

> If I have time this weekend, I'll boot up that VM in interactive mode and
> investigate :)
>
>
> On Saturday, September 8, 2018, Ralph Versteegen <teeemcee at gmail.com>
> wrote:
>
>> Looks like I managed to break Windows nightly builds somehow. It's not
>> even sending email with the build log anymore!
>>
>> On Fri, 7 Sep 2018 at 01:03, Ralph Versteegen <teeemcee at gmail.com> wrote:
>>
>>> I see that the last nightly build failed.
>>>
>>> Firstly, there's this error:
>>>   'nightly\ohrrpgce' is not recognized as an internal or external
>>> command,
>>>    operable program or batch file.
>>> I saw this sort of crazy thing a lot while testing. "svn update" must
>>> have modified the copy of distrib-nightly-win.bat that was being executed!
>>> cmd.exe kept the same position in the file, but its contents changed, so it
>>> was suddenly in the middle of a line.
>>> I had thought that a separate copy of the source code was used to
>>> provide the nightly build scripts and that you had to manually update them
>>> if they changed, but I guess you changed that at some point. (I thought
>>> that comment at the top was obsolete!)
>>> I guess this isn't worth fixing, but just keep it mind it might cause
>>> more (temporary) build failures in future.
>>>
>>> All other errors are also caused by the same thing. The first few lines
>>> of the old .bat file and the rest of the new .bat were executed, so the
>>> necessary environmental variables didn't get set. The next nightly build
>>> should hopefully work correctly.
>>>
>>>
>>>
>>> On Thu, 6 Sep 2018 at 13:14, James Paige <Bob at hamsterrepublic.com>
>>> wrote:
>>>
>>>> Awesome! I'll take a look and see if I need to change anything on the
>>>> build VMs or on the dreamhost account when I get a chance. This has been a
>>>> very busy week, but I have been reading all your commit messages with a
>>>> smile on my face :D
>>>>
>>>> ---
>>>> James
>>>>
>>>> On Wednesday, September 5, 2018, Ralph Versteegen <teeemcee at gmail.com>
>>>> wrote:
>>>>
>>>>> OK, I'm finally done with changes to the distrib scripts. I don't know
>>>>> how I managed to spend so many commits on cleaning them up...
>>>>> James, set the value of $SCPSYMBOLS, the location to upload the
>>>>> symbols archives to, to your liking. It's set in three different files.
>>>>> Although I renamed some scripts, that shouldn't affect anything.
>>>>>
>>>>> Hmm, I should make the windows nightly build script skip the build if
>>>>> there have been no svn commits, like the mac nightlies already do.
>>>>> (I'm not sure but I don't think we can easily reuse a .pdb file for
>>>>> two builds made from identical source code, because the .exe hash won't
>>>>> necessarily match)
>>>>> Skipping unnecessarily rebuilds would allow rebuilding the builds more
>>>>> than once per 24 hours (manually, or by cron).
>>>>> (I tried to let you still manually force additional early builds as
>>>>> long as there's been an svn commit, by putting the commit number in the
>>>>> archive name. But I just realised that won't work if you do a  manual build
>>>>> and then the scheduled build happens after, without any commits between. )
>>>>>
>>>>> On Tue, 4 Sep 2018 at 15:48, Ralph Versteegen <teeemcee at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> OK, great!
>>>>>> I think it would definitely be a good idea to get the php script to
>>>>>> forward reports to ohrrpgce-crash at hamsterrepublic.com so they don't
>>>>>> get forgotten (assuming there isn't a flood of them). Unless they are too
>>>>>> large, I guess - someone could try to include a large .rpg file in the
>>>>>> report. Is your email hosted by dreamhost? Then it seems that I can just
>>>>>> send email from another hamsterrepublic.com address to
>>>>>> ohrrpgce-crash using php's mail() function, according to the dreamhost docs.
>>>>>> And, I'll put the script in the web/ directory in svn.
>>>>>>
>>>>>> What about uploading the exe/pdb archives, where should they be put?
>>>>>> I guess I'll name them something like ohrrpgce-symbols-win-$RELEASE-$BACKENDS-$DATE-$SVNREV.7z
>>>>>> (I'll put 7z.exe in support/) and they can all go in one directory, and old
>>>>>> files can be deleted manually when we do a release, or automated later.
>>>>>> Incidentally, I notice that only mac nghtly builds skip building if
>>>>>> there are no changes to svn, linux and win builds build regardless.
>>>>>>
>>>>>>
>>>>>> On Tue, 4 Sep 2018 at 03:07, James Paige <Bob at hamsterrepublic.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Okay, you can now configure crashrpt to submit its reports to
>>>>>>> https://rpg.hamsterrepublic.com/crashrpt/
>>>>>>>
>>>>>>> They get saved in /home/james_paige/crash_reports/ which is not web
>>>>>>> accessible.
>>>>>>>
>>>>>>> TMC, if you send me an ssh public key, I can give you ssh/sftp
>>>>>>> access to that.
>>>>>>>
>>>>>>> Feel free to make whatever changes you like to the nightly build
>>>>>>> scripts, and let me know what I need to change, either on the VMs or on the
>>>>>>> dreamhost account
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 3, 2018 at 8:34 AM, James Paige <Bob at hamsterrepublic.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Anything that gives better crash reports sounds great to me.
>>>>>>>>
>>>>>>>> Increasing the download size a bit is not a big problem. I think
>>>>>>>> our downloads are already comparatively tiny.
>>>>>>>>
>>>>>>>> I also don't expect any problem with storing a few gigabytes of
>>>>>>>> exes and crash related data. My dreamhost account might be stingy about
>>>>>>>> processor resources, but it is very generous about storage space.
>>>>>>>>
>>>>>>>> I'll see what I can do about setting up that php script. We can use
>>>>>>>> it over either http or https since I have LetsEncrypt certs for all my
>>>>>>>> domains
>>>>>>>>
>>>>>>>> This is going to be a busy week, so if I don't get to it today it
>>>>>>>> will have to be next weekend. (I am making myself a reminder)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sunday, September 2, 2018, Ralph Versteegen <teeemcee at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I've heard of quite a few reports of various kinds of crashes
>>>>>>>>> which I don't have enough information to do anything about, so am forced to
>>>>>>>>> ignore. (Mostly they've been reported on the SS Discord server which even
>>>>>>>>> has a dedicated channel for reporting bugs!).
>>>>>>>>>
>>>>>>>>> My previous attempt to generate stack traces on Windows required
>>>>>>>>> somewhat large extra dlls to parse the DWARF debug info in game/custom.exe,
>>>>>>>>> and they can't provide line numbers, which would require huge
>>>>>>>>> totally-unstripped builds. Line numbers would help hugely.
>>>>>>>>> So the new plan is to instead generate minidumps which get sent
>>>>>>>>> back for analysis. Win XP and later have a function to generate minidumps
>>>>>>>>> (and the dll can be downloaded for older Windows versions if needed) The
>>>>>>>>> minidump includes the stack of each thread, so you can use it to generate
>>>>>>>>> traces with line numbers and inspect variables. Optionally you can include
>>>>>>>>> more parts of the process memory like global variables, but that makes it
>>>>>>>>> large.
>>>>>>>>>
>>>>>>>>> I've been trying out an open-source reporter, which sends reports
>>>>>>>>> over the internet with minimal user will needed:
>>>>>>>>> http://crashrpt.sourceforge.net/
>>>>>>>>> There are a couple other such reporters, like BreakPad (originally
>>>>>>>>> from Mozilla, and used by Firefox and Chrome) but they seem pretty
>>>>>>>>> heavy-weight and more difficult to use.
>>>>>>>>>
>>>>>>>>> CrashRpt requires a separate .dll (I doubt I would be able to
>>>>>>>>> statically link) plus CrashSender.exe which does the actual work, and an
>>>>>>>>> ini file.
>>>>>>>>> Zipped, these are 300KB in total (I compiled a custom build to
>>>>>>>>> strip out the extravagant ability to record ogg theora videos and jpgs!).
>>>>>>>>> So that's fine for including with Custom in support/. But we wouldn't want
>>>>>>>>> these extra files when distributing games. However, if you have a full
>>>>>>>>> distribution of the OHRRPGCE, then any copy of game.exe can reuse the
>>>>>>>>> CrashRpt files from support/. If CrashRpt isn't available, game.exe can
>>>>>>>>> still easily save a minidump itself and ask the user to email it manually.
>>>>>>>>>
>>>>>>>>> == How this will work ==
>>>>>>>>>
>>>>>>>>> CrashRpt does not generate stack traces. It only collects data and
>>>>>>>>> sends it off. You can tell it which files to attach, and it includes some
>>>>>>>>> metadata and a minidump and can include a screenshot.
>>>>>>>>>
>>>>>>>>> fbc -gen gas generates STABS debug symbols, gcc generates DWARF
>>>>>>>>> symbols, and Visual C++ generates PDB symbols, which live in a separate
>>>>>>>>> .pdb file. PDB is undocumented, but recently LLVM folks have been creating
>>>>>>>>> documentation by examining code Microsoft open-sourced. So GCC and GDB
>>>>>>>>> don't support PDB or minidumps.
>>>>>>>>>
>>>>>>>>> We can use the cv2pdb tool to convert DWARF to .pdb files, which
>>>>>>>>> we keep. The executables themselves get stripped, and also need to be kept;
>>>>>>>>> a .pdb alone is useless. (Currently we partially strip exes, leaving in
>>>>>>>>> function symbols so gcc can create a backtrace. I don't think that's worth
>>>>>>>>> keeping anymore, except in the debug build. By fully stripping, we can
>>>>>>>>> reduce each exe by another 300KB.) cv2pdb doesn't convert FB strings
>>>>>>>>> correctly, but those are on the heap so not in the minidump anyway.
>>>>>>>>>
>>>>>>>>> To examine the minidumps, you can use Visual Studio (the free
>>>>>>>>> versions will do) or windbg (CLI), or use a commandline tool or server-side
>>>>>>>>> to generate a stracktrace and other info from it. CrashRpt has such a tool,
>>>>>>>>> and scripts for handling large numbers of crash reports and sorting them.
>>>>>>>>> BreakPad also has tools for it.
>>>>>>>>>
>>>>>>>>> == What we need to do ==
>>>>>>>>>
>>>>>>>>> Firstly the .pdb files should get uploaded somewhere. We will want
>>>>>>>>> to keep the .pdb files and .exes for every nightly build - not just the
>>>>>>>>> weekly archived nightlies! .pdb files alone are unfortunately useless.
>>>>>>>>> I'm going to get rid of most of most of our windows nightly
>>>>>>>>> builds, they're redundant now that you can switch graphics backend. So
>>>>>>>>> leaving just music_sdl, music_native, music_native2 and music_sdl-debug.
>>>>>>>>> (In fact, music_native and music_native2 appear broken and I couldn't find
>>>>>>>>> anyone who uses them or even remembers using them.)
>>>>>>>>> We should keep the files for at least every default (music_sdl)
>>>>>>>>> build since at least the last stable release, probably longer since not
>>>>>>>>> everyone upgrades promptly. Using xz, the game+custom exe+pdb files
>>>>>>>>> compress to 2.4MB. So 4 builds, 3 months of backups is ~1000MB. **
>>>>>>>>> An idea to save space: upload just the music_sdl files since the
>>>>>>>>> last stable release, (~250MB) and keep all the other files on the build
>>>>>>>>> machine in the unlikely event they're needed.
>>>>>>>>>
>>>>>>>>> (** however I just remembered that exes from the nightly build
>>>>>>>>> machine are significantly larger than ones built on my machine)
>>>>>>>>>
>>>>>>>>> Secondly we need to collect reports - CrashRpt can send via HTTP,
>>>>>>>>> HTTPS, SMTP and by opening an email client. I assume dreamhost will block
>>>>>>>>> emails from random IPs, so we will want HTTP.
>>>>>>>>> There is an example php script to accept reports and save them.
>>>>>>>>> http://crashrpt.sourceforge.net/docs/html/sending_error_repo
>>>>>>>>> rts.html#httpsend
>>>>>>>>> I guess it would be easiest if we modify this script to email us
>>>>>>>>> the zip file. And keep a copy on the server in case something goes wrong?
>>>>>>>>> Can we put us up at hamsterrepublic.com?
>>>>>>>>>
>>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Ohrrpgce mailing list
>>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherh
>>>>>>> amster.org
>>>>>>>
>>>>>> _______________________________________________
>>>> Ohrrpgce mailing list
>>>> ohrrpgce at lists.motherhamster.org
>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.motherhamster.org/pipermail/ohrrpgce-motherhamster.org/attachments/20180908/6b02ab57/attachment-0001.html>