[Ohrrpgce] Crash reporting

Ralph Versteegen teeemcee at gmail.com
Sun Sep 2 19:29:22 PDT 2018


I've heard of quite a few reports of various kinds of crashes which I don't
have enough information to do anything about, so am forced to ignore.
(Mostly they've been reported on the SS Discord server which even has a
dedicated channel for reporting bugs!).

My previous attempt to generate stack traces on Windows required somewhat
large extra dlls to parse the DWARF debug info in game/custom.exe, and they
can't provide line numbers, which would require huge totally-unstripped
builds. Line numbers would help hugely.
So the new plan is to instead generate minidumps which get sent back for
analysis. Win XP and later have a function to generate minidumps (and the
dll can be downloaded for older Windows versions if needed) The minidump
includes the stack of each thread, so you can use it to generate traces
with line numbers and inspect variables. Optionally you can include more
parts of the process memory like global variables, but that makes it large.

I've been trying out an open-source reporter, which sends reports over the
internet with minimal user will needed:
http://crashrpt.sourceforge.net/
There are a couple other such reporters, like BreakPad (originally from
Mozilla, and used by Firefox and Chrome) but they seem pretty heavy-weight
and more difficult to use.

CrashRpt requires a separate .dll (I doubt I would be able to statically
link) plus CrashSender.exe which does the actual work, and an ini file.
Zipped, these are 300KB in total (I compiled a custom build to strip out
the extravagant ability to record ogg theora videos and jpgs!). So that's
fine for including with Custom in support/. But we wouldn't want these
extra files when distributing games. However, if you have a full
distribution of the OHRRPGCE, then any copy of game.exe can reuse the
CrashRpt files from support/. If CrashRpt isn't available, game.exe can
still easily save a minidump itself and ask the user to email it manually.

== How this will work ==

CrashRpt does not generate stack traces. It only collects data and sends it
off. You can tell it which files to attach, and it includes some metadata
and a minidump and can include a screenshot.

fbc -gen gas generates STABS debug symbols, gcc generates DWARF symbols,
and Visual C++ generates PDB symbols, which live in a separate .pdb file.
PDB is undocumented, but recently LLVM folks have been creating
documentation by examining code Microsoft open-sourced. So GCC and GDB
don't support PDB or minidumps.

We can use the cv2pdb tool to convert DWARF to .pdb files, which we keep.
The executables themselves get stripped, and also need to be kept; a .pdb
alone is useless. (Currently we partially strip exes, leaving in function
symbols so gcc can create a backtrace. I don't think that's worth keeping
anymore, except in the debug build. By fully stripping, we can reduce each
exe by another 300KB.) cv2pdb doesn't convert FB strings correctly, but
those are on the heap so not in the minidump anyway.

To examine the minidumps, you can use Visual Studio (the free versions will
do) or windbg (CLI), or use a commandline tool or server-side to generate a
stracktrace and other info from it. CrashRpt has such a tool, and scripts
for handling large numbers of crash reports and sorting them. BreakPad also
has tools for it.

== What we need to do ==

Firstly the .pdb files should get uploaded somewhere. We will want to keep
the .pdb files and .exes for every nightly build - not just the weekly
archived nightlies! .pdb files alone are unfortunately useless.
I'm going to get rid of most of most of our windows nightly builds, they're
redundant now that you can switch graphics backend. So leaving just
music_sdl, music_native, music_native2 and music_sdl-debug. (In fact,
music_native and music_native2 appear broken and I couldn't find anyone who
uses them or even remembers using them.)
We should keep the files for at least every default (music_sdl) build since
at least the last stable release, probably longer since not everyone
upgrades promptly. Using xz, the game+custom exe+pdb files compress to
2.4MB. So 4 builds, 3 months of backups is ~1000MB. **
An idea to save space: upload just the music_sdl files since the last
stable release, (~250MB) and keep all the other files on the build machine
in the unlikely event they're needed.

(** however I just remembered that exes from the nightly build machine are
significantly larger than ones built on my machine)

Secondly we need to collect reports - CrashRpt can send via HTTP, HTTPS,
SMTP and by opening an email client. I assume dreamhost will block emails
from random IPs, so we will want HTTP.
There is an example php script to accept reports and save them.
http://crashrpt.sourceforge.net/docs/html/sending_error_reports.html#httpsend
I guess it would be easiest if we modify this script to email us the zip
file. And keep a copy on the server in case something goes wrong? Can we
put us up at hamsterrepublic.com?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.motherhamster.org/pipermail/ohrrpgce-motherhamster.org/attachments/20180903/763e7ef7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: crashrpt.png
Type: image/png
Size: 39399 bytes
Desc: not available
URL: <http://lists.motherhamster.org/pipermail/ohrrpgce-motherhamster.org/attachments/20180903/763e7ef7/attachment-0001.png>


More information about the Ohrrpgce mailing list