[Ohrrpgce] Crash reporting

[Ralph Versteegen]

OK, great!
I think it would definitely be a good idea to get the php script to
forward reports to ohrrpgce-crash at hamsterrepublic.com so they don't get
forgotten (assuming there isn't a flood of them). Unless they are too
large, I guess - someone could try to include a large .rpg file in the
report. Is your email hosted by dreamhost? Then it seems that I can just
send email from another hamsterrepublic.com address to ohrrpgce-crash using
php's mail() function, according to the dreamhost docs.
And, I'll put the script in the web/ directory in svn.

What about uploading the exe/pdb archives, where should they be put? I
guess I'll name them something like
ohrrpgce-symbols-win-$RELEASE-$BACKENDS-$DATE-$SVNREV.7z (I'll put 7z.exe
in support/) and they can all go in one directory, and old files can be
deleted manually when we do a release, or automated later.
Incidentally, I notice that only mac nghtly builds skip building if there
are no changes to svn, linux and win builds build regardless.


On Tue, 4 Sep 2018 at 03:07, James Paige <Bob at hamsterrepublic.com> wrote:

> Okay, you can now configure crashrpt to submit its reports to
> https://rpg.hamsterrepublic.com/crashrpt/
>
> They get saved in /home/james_paige/crash_reports/ which is not web
> accessible.
>
> TMC, if you send me an ssh public key, I can give you ssh/sftp access to
> that.
>
> Feel free to make whatever changes you like to the nightly build scripts,
> and let me know what I need to change, either on the VMs or on the
> dreamhost account
>
>
> On Mon, Sep 3, 2018 at 8:34 AM, James Paige <Bob at hamsterrepublic.com>
> wrote:
>
>> Anything that gives better crash reports sounds great to me.
>>
>> Increasing the download size a bit is not a big problem. I think our
>> downloads are already comparatively tiny.
>>
>> I also don't expect any problem with storing a few gigabytes of exes and
>> crash related data. My dreamhost account might be stingy about processor
>> resources, but it is very generous about storage space.
>>
>> I'll see what I can do about setting up that php script. We can use it
>> over either http or https since I have LetsEncrypt certs for all my domains
>>
>> This is going to be a busy week, so if I don't get to it today it will
>> have to be next weekend. (I am making myself a reminder)
>>
>>
>> On Sunday, September 2, 2018, Ralph Versteegen <teeemcee at gmail.com>
>> wrote:
>>
>>> I've heard of quite a few reports of various kinds of crashes which I
>>> don't have enough information to do anything about, so am forced to ignore.
>>> (Mostly they've been reported on the SS Discord server which even has a
>>> dedicated channel for reporting bugs!).
>>>
>>> My previous attempt to generate stack traces on Windows required
>>> somewhat large extra dlls to parse the DWARF debug info in game/custom.exe,
>>> and they can't provide line numbers, which would require huge
>>> totally-unstripped builds. Line numbers would help hugely.
>>> So the new plan is to instead generate minidumps which get sent back for
>>> analysis. Win XP and later have a function to generate minidumps (and the
>>> dll can be downloaded for older Windows versions if needed) The minidump
>>> includes the stack of each thread, so you can use it to generate traces
>>> with line numbers and inspect variables. Optionally you can include more
>>> parts of the process memory like global variables, but that makes it large.
>>>
>>> I've been trying out an open-source reporter, which sends reports over
>>> the internet with minimal user will needed:
>>> http://crashrpt.sourceforge.net/
>>> There are a couple other such reporters, like BreakPad (originally from
>>> Mozilla, and used by Firefox and Chrome) but they seem pretty heavy-weight
>>> and more difficult to use.
>>>
>>> CrashRpt requires a separate .dll (I doubt I would be able to statically
>>> link) plus CrashSender.exe which does the actual work, and an ini file.
>>> Zipped, these are 300KB in total (I compiled a custom build to strip out
>>> the extravagant ability to record ogg theora videos and jpgs!). So that's
>>> fine for including with Custom in support/. But we wouldn't want these
>>> extra files when distributing games. However, if you have a full
>>> distribution of the OHRRPGCE, then any copy of game.exe can reuse the
>>> CrashRpt files from support/. If CrashRpt isn't available, game.exe can
>>> still easily save a minidump itself and ask the user to email it manually.
>>>
>>> == How this will work ==
>>>
>>> CrashRpt does not generate stack traces. It only collects data and sends
>>> it off. You can tell it which files to attach, and it includes some
>>> metadata and a minidump and can include a screenshot.
>>>
>>> fbc -gen gas generates STABS debug symbols, gcc generates DWARF symbols,
>>> and Visual C++ generates PDB symbols, which live in a separate .pdb file.
>>> PDB is undocumented, but recently LLVM folks have been creating
>>> documentation by examining code Microsoft open-sourced. So GCC and GDB
>>> don't support PDB or minidumps.
>>>
>>> We can use the cv2pdb tool to convert DWARF to .pdb files, which we
>>> keep. The executables themselves get stripped, and also need to be kept; a
>>> .pdb alone is useless. (Currently we partially strip exes, leaving in
>>> function symbols so gcc can create a backtrace. I don't think that's worth
>>> keeping anymore, except in the debug build. By fully stripping, we can
>>> reduce each exe by another 300KB.) cv2pdb doesn't convert FB strings
>>> correctly, but those are on the heap so not in the minidump anyway.
>>>
>>> To examine the minidumps, you can use Visual Studio (the free versions
>>> will do) or windbg (CLI), or use a commandline tool or server-side to
>>> generate a stracktrace and other info from it. CrashRpt has such a tool,
>>> and scripts for handling large numbers of crash reports and sorting them.
>>> BreakPad also has tools for it.
>>>
>>> == What we need to do ==
>>>
>>> Firstly the .pdb files should get uploaded somewhere. We will want to
>>> keep the .pdb files and .exes for every nightly build - not just the weekly
>>> archived nightlies! .pdb files alone are unfortunately useless.
>>> I'm going to get rid of most of most of our windows nightly builds,
>>> they're redundant now that you can switch graphics backend. So leaving just
>>> music_sdl, music_native, music_native2 and music_sdl-debug. (In fact,
>>> music_native and music_native2 appear broken and I couldn't find anyone who
>>> uses them or even remembers using them.)
>>> We should keep the files for at least every default (music_sdl) build
>>> since at least the last stable release, probably longer since not everyone
>>> upgrades promptly. Using xz, the game+custom exe+pdb files compress to
>>> 2.4MB. So 4 builds, 3 months of backups is ~1000MB. **
>>> An idea to save space: upload just the music_sdl files since the last
>>> stable release, (~250MB) and keep all the other files on the build machine
>>> in the unlikely event they're needed.
>>>
>>> (** however I just remembered that exes from the nightly build machine
>>> are significantly larger than ones built on my machine)
>>>
>>> Secondly we need to collect reports - CrashRpt can send via HTTP, HTTPS,
>>> SMTP and by opening an email client. I assume dreamhost will block emails
>>> from random IPs, so we will want HTTP.
>>> There is an example php script to accept reports and save them.
>>>
>>> http://crashrpt.sourceforge.net/docs/html/sending_error_reports.html#httpsend
>>> I guess it would be easiest if we modify this script to email us the zip
>>> file. And keep a copy on the server in case something goes wrong? Can we
>>> put us up at hamsterrepublic.com?
>>>
>>>
> _______________________________________________
> Ohrrpgce mailing list
> ohrrpgce at lists.motherhamster.org
> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.motherhamster.org/pipermail/ohrrpgce-motherhamster.org/attachments/20180904/74315916/attachment.html>