[Ohrrpgce] Crash reporting

James Paige Bob at hamsterrepublic.com
Sun Sep 9 10:19:12 PDT 2018


Okay, Progress! A little, anyway.

The Windows VM boots, attempts to build, and shuts down correctly.

The actually builds are still failing though, this time because it says a
PDB DLL file is missing.

---
James Paige


On Sun, Sep 9, 2018 at 1:18 AM, Ralph Versteegen <teeemcee at gmail.com> wrote:

> Oh, and it occurs to me that you're going to need to manually do an 'svn
> update', because the script dies before it gets that far.
> Also, manually updating svn will also make it skip building, because it'll
> think there are no new commits. That's would be fine, although it'll
> prevent you from actually seeing whether it's working or not.
> Also, I forgot to ask you to delete the old nightly build .zips, for
> builds which are no longer done or have been renamed.
>
> On Sun, 9 Sep 2018 at 17:07, Ralph Versteegen <teeemcee at gmail.com> wrote:
>
>> OK, that was enough to figure out what was going on.
>>
>> Quoting in batch scripts is really a nightmare!
>> If you write
>>  SET ISCC=C:\Program Files (x86)\Inno Setup 5\iscc.exe
>> then the brackets cause a parsing error.
>> If you write
>>  SET ISCC="C:\Program Files (x86)\Inno Setup 5\iscc.exe"
>> and then do
>>   ECHO %ISCC%
>> it prints the filename with quotes. But if you do
>>   IF NOT EXISTS %ISCC% ( ...
>> you get an error due to the unquoted (x86) in the filename, although it
>> is quoted!
>> So if you try instead
>>   IF NOT EXISTS "%ISCC%" ( ...
>> that expands to ""C:\Program Files (x86)\Inno Setup 5\iscc.exe"" and now
>> cmd.exe DOES process the quotes, the double quotes apparently cancelling
>> out, and cmd.exe complains about 'Files' being unexpected!! Talk about
>> inconsistent!
>> It turns out that the only possible way to get everything to work is to
>> write
>>   SET "ISCC=C:\Program Files (x86)\Inno Setup 5\iscc.exe"
>> which does protect the (x86) but doesn't put the quotes in the variable
>> itself, so that yuo can then write "%ISCC%" without causing double quoting.
>>
>> I take back every single nice thing I said about batch scripting!
>>
>> This is another reason why I want to convert most of our distrib scripts
>> into python!
>>
>> On Sun, 9 Sep 2018 at 10:04, James Paige <Bob at hamsterrepublic.com> wrote:
>>
>>> So far I can see that the VM starts up, runs distrib-nightly-win.bat
>>> inside c:\nightly\ohrrpgce\ and then immediately quits.
>>>
>>> The only output in c:\nightly\distrib-out.txt says:
>>>
>>> Searching for support programs...
>>> \Inno was unexpected at this time.
>>>
>>> On Sat, Sep 8, 2018 at 6:35 AM, James Paige <Bob at hamsterrepublic.com>
>>> wrote:
>>>
>>>> If I have time this weekend, I'll boot up that VM in interactive mode
>>>> and investigate :)
>>>>
>>>>
>>>> On Saturday, September 8, 2018, Ralph Versteegen <teeemcee at gmail.com>
>>>> wrote:
>>>>
>>>>> Looks like I managed to break Windows nightly builds somehow. It's not
>>>>> even sending email with the build log anymore!
>>>>>
>>>>> On Fri, 7 Sep 2018 at 01:03, Ralph Versteegen <teeemcee at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I see that the last nightly build failed.
>>>>>>
>>>>>> Firstly, there's this error:
>>>>>>   'nightly\ohrrpgce' is not recognized as an internal or external
>>>>>> command,
>>>>>>    operable program or batch file.
>>>>>> I saw this sort of crazy thing a lot while testing. "svn update" must
>>>>>> have modified the copy of distrib-nightly-win.bat that was being executed!
>>>>>> cmd.exe kept the same position in the file, but its contents changed, so it
>>>>>> was suddenly in the middle of a line.
>>>>>> I had thought that a separate copy of the source code was used to
>>>>>> provide the nightly build scripts and that you had to manually update them
>>>>>> if they changed, but I guess you changed that at some point. (I thought
>>>>>> that comment at the top was obsolete!)
>>>>>> I guess this isn't worth fixing, but just keep it mind it might cause
>>>>>> more (temporary) build failures in future.
>>>>>>
>>>>>> All other errors are also caused by the same thing. The first few
>>>>>> lines of the old .bat file and the rest of the new .bat were executed, so
>>>>>> the necessary environmental variables didn't get set. The next nightly
>>>>>> build should hopefully work correctly.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 6 Sep 2018 at 13:14, James Paige <Bob at hamsterrepublic.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Awesome! I'll take a look and see if I need to change anything on
>>>>>>> the build VMs or on the dreamhost account when I get a chance. This has
>>>>>>> been a very busy week, but I have been reading all your commit messages
>>>>>>> with a smile on my face :D
>>>>>>>
>>>>>>> ---
>>>>>>> James
>>>>>>>
>>>>>>> On Wednesday, September 5, 2018, Ralph Versteegen <
>>>>>>> teeemcee at gmail.com> wrote:
>>>>>>>
>>>>>>>> OK, I'm finally done with changes to the distrib scripts. I don't
>>>>>>>> know how I managed to spend so many commits on cleaning them up...
>>>>>>>> James, set the value of $SCPSYMBOLS, the location to upload the
>>>>>>>> symbols archives to, to your liking. It's set in three different files.
>>>>>>>> Although I renamed some scripts, that shouldn't affect anything.
>>>>>>>>
>>>>>>>> Hmm, I should make the windows nightly build script skip the build
>>>>>>>> if there have been no svn commits, like the mac nightlies already do.
>>>>>>>> (I'm not sure but I don't think we can easily reuse a .pdb file for
>>>>>>>> two builds made from identical source code, because the .exe hash won't
>>>>>>>> necessarily match)
>>>>>>>> Skipping unnecessarily rebuilds would allow rebuilding the builds
>>>>>>>> more than once per 24 hours (manually, or by cron).
>>>>>>>> (I tried to let you still manually force additional early builds as
>>>>>>>> long as there's been an svn commit, by putting the commit number in the
>>>>>>>> archive name. But I just realised that won't work if you do a  manual build
>>>>>>>> and then the scheduled build happens after, without any commits between. )
>>>>>>>>
>>>>>>>> On Tue, 4 Sep 2018 at 15:48, Ralph Versteegen <teeemcee at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> OK, great!
>>>>>>>>> I think it would definitely be a good idea to get the php script
>>>>>>>>> to forward reports to ohrrpgce-crash at hamsterrepublic.com so they
>>>>>>>>> don't get forgotten (assuming there isn't a flood of them). Unless they are
>>>>>>>>> too large, I guess - someone could try to include a large .rpg file in the
>>>>>>>>> report. Is your email hosted by dreamhost? Then it seems that I can just
>>>>>>>>> send email from another hamsterrepublic.com address to
>>>>>>>>> ohrrpgce-crash using php's mail() function, according to the dreamhost docs.
>>>>>>>>> And, I'll put the script in the web/ directory in svn.
>>>>>>>>>
>>>>>>>>> What about uploading the exe/pdb archives, where should they be
>>>>>>>>> put? I guess I'll name them something like ohrrpgce-symbols-win-$RELEASE-$BACKENDS-$DATE-$SVNREV.7z
>>>>>>>>> (I'll put 7z.exe in support/) and they can all go in one directory, and old
>>>>>>>>> files can be deleted manually when we do a release, or automated later.
>>>>>>>>> Incidentally, I notice that only mac nghtly builds skip building
>>>>>>>>> if there are no changes to svn, linux and win builds build regardless.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, 4 Sep 2018 at 03:07, James Paige <Bob at hamsterrepublic.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Okay, you can now configure crashrpt to submit its reports to
>>>>>>>>>> https://rpg.hamsterrepublic.com/crashrpt/
>>>>>>>>>>
>>>>>>>>>> They get saved in /home/james_paige/crash_reports/ which is not
>>>>>>>>>> web accessible.
>>>>>>>>>>
>>>>>>>>>> TMC, if you send me an ssh public key, I can give you ssh/sftp
>>>>>>>>>> access to that.
>>>>>>>>>>
>>>>>>>>>> Feel free to make whatever changes you like to the nightly build
>>>>>>>>>> scripts, and let me know what I need to change, either on the VMs or on the
>>>>>>>>>> dreamhost account
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 3, 2018 at 8:34 AM, James Paige <
>>>>>>>>>> Bob at hamsterrepublic.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Anything that gives better crash reports sounds great to me.
>>>>>>>>>>>
>>>>>>>>>>> Increasing the download size a bit is not a big problem. I think
>>>>>>>>>>> our downloads are already comparatively tiny.
>>>>>>>>>>>
>>>>>>>>>>> I also don't expect any problem with storing a few gigabytes of
>>>>>>>>>>> exes and crash related data. My dreamhost account might be stingy about
>>>>>>>>>>> processor resources, but it is very generous about storage space.
>>>>>>>>>>>
>>>>>>>>>>> I'll see what I can do about setting up that php script. We can
>>>>>>>>>>> use it over either http or https since I have LetsEncrypt certs for all my
>>>>>>>>>>> domains
>>>>>>>>>>>
>>>>>>>>>>> This is going to be a busy week, so if I don't get to it today
>>>>>>>>>>> it will have to be next weekend. (I am making myself a reminder)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sunday, September 2, 2018, Ralph Versteegen <
>>>>>>>>>>> teeemcee at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I've heard of quite a few reports of various kinds of crashes
>>>>>>>>>>>> which I don't have enough information to do anything about, so am forced to
>>>>>>>>>>>> ignore. (Mostly they've been reported on the SS Discord server which even
>>>>>>>>>>>> has a dedicated channel for reporting bugs!).
>>>>>>>>>>>>
>>>>>>>>>>>> My previous attempt to generate stack traces on Windows
>>>>>>>>>>>> required somewhat large extra dlls to parse the DWARF debug info in
>>>>>>>>>>>> game/custom.exe, and they can't provide line numbers, which would require
>>>>>>>>>>>> huge totally-unstripped builds. Line numbers would help hugely.
>>>>>>>>>>>> So the new plan is to instead generate minidumps which get sent
>>>>>>>>>>>> back for analysis. Win XP and later have a function to generate minidumps
>>>>>>>>>>>> (and the dll can be downloaded for older Windows versions if needed) The
>>>>>>>>>>>> minidump includes the stack of each thread, so you can use it to generate
>>>>>>>>>>>> traces with line numbers and inspect variables. Optionally you can include
>>>>>>>>>>>> more parts of the process memory like global variables, but that makes it
>>>>>>>>>>>> large.
>>>>>>>>>>>>
>>>>>>>>>>>> I've been trying out an open-source reporter, which sends
>>>>>>>>>>>> reports over the internet with minimal user will needed:
>>>>>>>>>>>> http://crashrpt.sourceforge.net/
>>>>>>>>>>>> There are a couple other such reporters, like BreakPad
>>>>>>>>>>>> (originally from Mozilla, and used by Firefox and Chrome) but they seem
>>>>>>>>>>>> pretty heavy-weight and more difficult to use.
>>>>>>>>>>>>
>>>>>>>>>>>> CrashRpt requires a separate .dll (I doubt I would be able to
>>>>>>>>>>>> statically link) plus CrashSender.exe which does the actual work, and an
>>>>>>>>>>>> ini file.
>>>>>>>>>>>> Zipped, these are 300KB in total (I compiled a custom build to
>>>>>>>>>>>> strip out the extravagant ability to record ogg theora videos and jpgs!).
>>>>>>>>>>>> So that's fine for including with Custom in support/. But we wouldn't want
>>>>>>>>>>>> these extra files when distributing games. However, if you have a full
>>>>>>>>>>>> distribution of the OHRRPGCE, then any copy of game.exe can reuse the
>>>>>>>>>>>> CrashRpt files from support/. If CrashRpt isn't available, game.exe can
>>>>>>>>>>>> still easily save a minidump itself and ask the user to email it manually.
>>>>>>>>>>>>
>>>>>>>>>>>> == How this will work ==
>>>>>>>>>>>>
>>>>>>>>>>>> CrashRpt does not generate stack traces. It only collects data
>>>>>>>>>>>> and sends it off. You can tell it which files to attach, and it includes
>>>>>>>>>>>> some metadata and a minidump and can include a screenshot.
>>>>>>>>>>>>
>>>>>>>>>>>> fbc -gen gas generates STABS debug symbols, gcc generates DWARF
>>>>>>>>>>>> symbols, and Visual C++ generates PDB symbols, which live in a separate
>>>>>>>>>>>> .pdb file. PDB is undocumented, but recently LLVM folks have been creating
>>>>>>>>>>>> documentation by examining code Microsoft open-sourced. So GCC and GDB
>>>>>>>>>>>> don't support PDB or minidumps.
>>>>>>>>>>>>
>>>>>>>>>>>> We can use the cv2pdb tool to convert DWARF to .pdb files,
>>>>>>>>>>>> which we keep. The executables themselves get stripped, and also need to be
>>>>>>>>>>>> kept; a .pdb alone is useless. (Currently we partially strip exes, leaving
>>>>>>>>>>>> in function symbols so gcc can create a backtrace. I don't think that's
>>>>>>>>>>>> worth keeping anymore, except in the debug build. By fully stripping, we
>>>>>>>>>>>> can reduce each exe by another 300KB.) cv2pdb doesn't convert FB strings
>>>>>>>>>>>> correctly, but those are on the heap so not in the minidump anyway.
>>>>>>>>>>>>
>>>>>>>>>>>> To examine the minidumps, you can use Visual Studio (the free
>>>>>>>>>>>> versions will do) or windbg (CLI), or use a commandline tool or server-side
>>>>>>>>>>>> to generate a stracktrace and other info from it. CrashRpt has such a tool,
>>>>>>>>>>>> and scripts for handling large numbers of crash reports and sorting them.
>>>>>>>>>>>> BreakPad also has tools for it.
>>>>>>>>>>>>
>>>>>>>>>>>> == What we need to do ==
>>>>>>>>>>>>
>>>>>>>>>>>> Firstly the .pdb files should get uploaded somewhere. We will
>>>>>>>>>>>> want to keep the .pdb files and .exes for every nightly build - not just
>>>>>>>>>>>> the weekly archived nightlies! .pdb files alone are unfortunately useless.
>>>>>>>>>>>> I'm going to get rid of most of most of our windows nightly
>>>>>>>>>>>> builds, they're redundant now that you can switch graphics backend. So
>>>>>>>>>>>> leaving just music_sdl, music_native, music_native2 and music_sdl-debug.
>>>>>>>>>>>> (In fact, music_native and music_native2 appear broken and I couldn't find
>>>>>>>>>>>> anyone who uses them or even remembers using them.)
>>>>>>>>>>>> We should keep the files for at least every default (music_sdl)
>>>>>>>>>>>> build since at least the last stable release, probably longer since not
>>>>>>>>>>>> everyone upgrades promptly. Using xz, the game+custom exe+pdb files
>>>>>>>>>>>> compress to 2.4MB. So 4 builds, 3 months of backups is ~1000MB. **
>>>>>>>>>>>> An idea to save space: upload just the music_sdl files since
>>>>>>>>>>>> the last stable release, (~250MB) and keep all the other files on the build
>>>>>>>>>>>> machine in the unlikely event they're needed.
>>>>>>>>>>>>
>>>>>>>>>>>> (** however I just remembered that exes from the nightly build
>>>>>>>>>>>> machine are significantly larger than ones built on my machine)
>>>>>>>>>>>>
>>>>>>>>>>>> Secondly we need to collect reports - CrashRpt can send via
>>>>>>>>>>>> HTTP, HTTPS, SMTP and by opening an email client. I assume dreamhost will
>>>>>>>>>>>> block emails from random IPs, so we will want HTTP.
>>>>>>>>>>>> There is an example php script to accept reports and save them.
>>>>>>>>>>>> http://crashrpt.sourceforge.net/docs/html/sending_error_
>>>>>>>>>>>> reports.html#httpsend
>>>>>>>>>>>> I guess it would be easiest if we modify this script to email
>>>>>>>>>>>> us the zip file. And keep a copy on the server in case something goes
>>>>>>>>>>>> wrong? Can we put us up at hamsterrepublic.com?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Ohrrpgce mailing list
>>>>>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-
>>>>>>>>>> motherhamster.org
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>> Ohrrpgce mailing list
>>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-
>>>>>>> motherhamster.org
>>>>>>>
>>>>>>
>>> _______________________________________________
>>> Ohrrpgce mailing list
>>> ohrrpgce at lists.motherhamster.org
>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>
>>
> _______________________________________________
> Ohrrpgce mailing list
> ohrrpgce at lists.motherhamster.org
> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.motherhamster.org/pipermail/ohrrpgce-motherhamster.org/attachments/20180909/d8e9c78b/attachment-0001.html>


More information about the Ohrrpgce mailing list