[Ohrrpgce] Crash reporting

James Paige Bob at hamsterrepublic.com
Mon Sep 3 05:34:13 PDT 2018


Anything that gives better crash reports sounds great to me.

Increasing the download size a bit is not a big problem. I think our
downloads are already comparatively tiny.

I also don't expect any problem with storing a few gigabytes of exes and
crash related data. My dreamhost account might be stingy about processor
resources, but it is very generous about storage space.

I'll see what I can do about setting up that php script. We can use it over
either http or https since I have LetsEncrypt certs for all my domains

This is going to be a busy week, so if I don't get to it today it will have
to be next weekend. (I am making myself a reminder)

On Sunday, September 2, 2018, Ralph Versteegen <teeemcee at gmail.com> wrote:

> I've heard of quite a few reports of various kinds of crashes which I
> don't have enough information to do anything about, so am forced to ignore.
> (Mostly they've been reported on the SS Discord server which even has a
> dedicated channel for reporting bugs!).
>
> My previous attempt to generate stack traces on Windows required somewhat
> large extra dlls to parse the DWARF debug info in game/custom.exe, and they
> can't provide line numbers, which would require huge totally-unstripped
> builds. Line numbers would help hugely.
> So the new plan is to instead generate minidumps which get sent back for
> analysis. Win XP and later have a function to generate minidumps (and the
> dll can be downloaded for older Windows versions if needed) The minidump
> includes the stack of each thread, so you can use it to generate traces
> with line numbers and inspect variables. Optionally you can include more
> parts of the process memory like global variables, but that makes it large.
>
> I've been trying out an open-source reporter, which sends reports over the
> internet with minimal user will needed:
> http://crashrpt.sourceforge.net/
> There are a couple other such reporters, like BreakPad (originally from
> Mozilla, and used by Firefox and Chrome) but they seem pretty heavy-weight
> and more difficult to use.
>
> CrashRpt requires a separate .dll (I doubt I would be able to statically
> link) plus CrashSender.exe which does the actual work, and an ini file.
> Zipped, these are 300KB in total (I compiled a custom build to strip out
> the extravagant ability to record ogg theora videos and jpgs!). So that's
> fine for including with Custom in support/. But we wouldn't want these
> extra files when distributing games. However, if you have a full
> distribution of the OHRRPGCE, then any copy of game.exe can reuse the
> CrashRpt files from support/. If CrashRpt isn't available, game.exe can
> still easily save a minidump itself and ask the user to email it manually.
>
> == How this will work ==
>
> CrashRpt does not generate stack traces. It only collects data and sends
> it off. You can tell it which files to attach, and it includes some
> metadata and a minidump and can include a screenshot.
>
> fbc -gen gas generates STABS debug symbols, gcc generates DWARF symbols,
> and Visual C++ generates PDB symbols, which live in a separate .pdb file.
> PDB is undocumented, but recently LLVM folks have been creating
> documentation by examining code Microsoft open-sourced. So GCC and GDB
> don't support PDB or minidumps.
>
> We can use the cv2pdb tool to convert DWARF to .pdb files, which we keep.
> The executables themselves get stripped, and also need to be kept; a .pdb
> alone is useless. (Currently we partially strip exes, leaving in function
> symbols so gcc can create a backtrace. I don't think that's worth keeping
> anymore, except in the debug build. By fully stripping, we can reduce each
> exe by another 300KB.) cv2pdb doesn't convert FB strings correctly, but
> those are on the heap so not in the minidump anyway.
>
> To examine the minidumps, you can use Visual Studio (the free versions
> will do) or windbg (CLI), or use a commandline tool or server-side to
> generate a stracktrace and other info from it. CrashRpt has such a tool,
> and scripts for handling large numbers of crash reports and sorting them.
> BreakPad also has tools for it.
>
> == What we need to do ==
>
> Firstly the .pdb files should get uploaded somewhere. We will want to keep
> the .pdb files and .exes for every nightly build - not just the weekly
> archived nightlies! .pdb files alone are unfortunately useless.
> I'm going to get rid of most of most of our windows nightly builds,
> they're redundant now that you can switch graphics backend. So leaving just
> music_sdl, music_native, music_native2 and music_sdl-debug. (In fact,
> music_native and music_native2 appear broken and I couldn't find anyone who
> uses them or even remembers using them.)
> We should keep the files for at least every default (music_sdl) build
> since at least the last stable release, probably longer since not everyone
> upgrades promptly. Using xz, the game+custom exe+pdb files compress to
> 2.4MB. So 4 builds, 3 months of backups is ~1000MB. **
> An idea to save space: upload just the music_sdl files since the last
> stable release, (~250MB) and keep all the other files on the build machine
> in the unlikely event they're needed.
>
> (** however I just remembered that exes from the nightly build machine are
> significantly larger than ones built on my machine)
>
> Secondly we need to collect reports - CrashRpt can send via HTTP, HTTPS,
> SMTP and by opening an email client. I assume dreamhost will block emails
> from random IPs, so we will want HTTP.
> There is an example php script to accept reports and save them.
> http://crashrpt.sourceforge.net/docs/html/sending_error_
> reports.html#httpsend
> I guess it would be easiest if we modify this script to email us the zip
> file. And keep a copy on the server in case something goes wrong? Can we
> put us up at hamsterrepublic.com?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.motherhamster.org/pipermail/ohrrpgce-motherhamster.org/attachments/20180903/24840554/attachment.html>


More information about the Ohrrpgce mailing list