[Ohrrpgce] Crash reporting

Wed Sep 5 16:04:38 PDT 2018

OK, I'm finally done with changes to the distrib scripts. I don't know how
I managed to spend so many commits on cleaning them up...
James, set the value of $SCPSYMBOLS, the location to upload the symbols
archives to, to your liking. It's set in three different files.
Although I renamed some scripts, that shouldn't affect anything.

Hmm, I should make the windows nightly build script skip the build if there
have been no svn commits, like the mac nightlies already do.
(I'm not sure but I don't think we can easily reuse a .pdb file for two
builds made from identical source code, because the .exe hash won't
necessarily match)
Skipping unnecessarily rebuilds would allow rebuilding the builds more than
once per 24 hours (manually, or by cron).
(I tried to let you still manually force additional early builds as long as
there's been an svn commit, by putting the commit number in the archive
name. But I just realised that won't work if you do a  manual build and
then the scheduled build happens after, without any commits between. )

On Tue, 4 Sep 2018 at 15:48, Ralph Versteegen <teeemcee at gmail.com> wrote:

> OK, great!
> I think it would definitely be a good idea to get the php script to
> forward reports to ohrrpgce-crash at hamsterrepublic.com so they don't get
> forgotten (assuming there isn't a flood of them). Unless they are too
> large, I guess - someone could try to include a large .rpg file in the
> report. Is your email hosted by dreamhost? Then it seems that I can just
> send email from another hamsterrepublic.com address to ohrrpgce-crash
> using php's mail() function, according to the dreamhost docs.
> And, I'll put the script in the web/ directory in svn.
>
> What about uploading the exe/pdb archives, where should they be put? I
> guess I'll name them something like
> ohrrpgce-symbols-win-$RELEASE-$BACKENDS-$DATE-$SVNREV.7z (I'll put 7z.exe
> in support/) and they can all go in one directory, and old files can be
> deleted manually when we do a release, or automated later.
> Incidentally, I notice that only mac nghtly builds skip building if there
> are no changes to svn, linux and win builds build regardless.
>
>
> On Tue, 4 Sep 2018 at 03:07, James Paige <Bob at hamsterrepublic.com> wrote:
>
>> Okay, you can now configure crashrpt to submit its reports to
>> https://rpg.hamsterrepublic.com/crashrpt/
>>
>> They get saved in /home/james_paige/crash_reports/ which is not web
>> accessible.
>>
>> TMC, if you send me an ssh public key, I can give you ssh/sftp access to
>> that.
>>
>> Feel free to make whatever changes you like to the nightly build scripts,
>> and let me know what I need to change, either on the VMs or on the
>> dreamhost account
>>
>>
>> On Mon, Sep 3, 2018 at 8:34 AM, James Paige <Bob at hamsterrepublic.com>
>> wrote:
>>
>>> Anything that gives better crash reports sounds great to me.
>>>
>>> Increasing the download size a bit is not a big problem. I think our
>>> downloads are already comparatively tiny.
>>>
>>> I also don't expect any problem with storing a few gigabytes of exes and
>>> crash related data. My dreamhost account might be stingy about processor
>>> resources, but it is very generous about storage space.
>>>
>>> I'll see what I can do about setting up that php script. We can use it
>>> over either http or https since I have LetsEncrypt certs for all my domains
>>>
>>> This is going to be a busy week, so if I don't get to it today it will
>>> have to be next weekend. (I am making myself a reminder)
>>>
>>>
>>> On Sunday, September 2, 2018, Ralph Versteegen <teeemcee at gmail.com>
>>> wrote:
>>>
>>>> I've heard of quite a few reports of various kinds of crashes which I
>>>> don't have enough information to do anything about, so am forced to ignore.
>>>> (Mostly they've been reported on the SS Discord server which even has a
>>>> dedicated channel for reporting bugs!).
>>>>
>>>> My previous attempt to generate stack traces on Windows required
>>>> somewhat large extra dlls to parse the DWARF debug info in game/custom.exe,
>>>> and they can't provide line numbers, which would require huge
>>>> totally-unstripped builds. Line numbers would help hugely.
>>>> So the new plan is to instead generate minidumps which get sent back
>>>> for analysis. Win XP and later have a function to generate minidumps (and
>>>> the dll can be downloaded for older Windows versions if needed) The
>>>> minidump includes the stack of each thread, so you can use it to generate
>>>> traces with line numbers and inspect variables. Optionally you can include
>>>> more parts of the process memory like global variables, but that makes it
>>>> large.
>>>>
>>>> I've been trying out an open-source reporter, which sends reports over
>>>> the internet with minimal user will needed:
>>>> http://crashrpt.sourceforge.net/
>>>> There are a couple other such reporters, like BreakPad (originally from
>>>> Mozilla, and used by Firefox and Chrome) but they seem pretty heavy-weight
>>>> and more difficult to use.
>>>>
>>>> CrashRpt requires a separate .dll (I doubt I would be able to
>>>> statically link) plus CrashSender.exe which does the actual work, and an
>>>> ini file.
>>>> Zipped, these are 300KB in total (I compiled a custom build to strip
>>>> out the extravagant ability to record ogg theora videos and jpgs!). So
>>>> that's fine for including with Custom in support/. But we wouldn't want
>>>> these extra files when distributing games. However, if you have a full
>>>> distribution of the OHRRPGCE, then any copy of game.exe can reuse the
>>>> CrashRpt files from support/. If CrashRpt isn't available, game.exe can
>>>> still easily save a minidump itself and ask the user to email it manually.
>>>>
>>>> == How this will work ==
>>>>
>>>> CrashRpt does not generate stack traces. It only collects data and
>>>> sends it off. You can tell it which files to attach, and it includes some
>>>> metadata and a minidump and can include a screenshot.
>>>>
>>>> fbc -gen gas generates STABS debug symbols, gcc generates DWARF
>>>> symbols, and Visual C++ generates PDB symbols, which live in a separate
>>>> .pdb file. PDB is undocumented, but recently LLVM folks have been creating
>>>> documentation by examining code Microsoft open-sourced. So GCC and GDB
>>>> don't support PDB or minidumps.
>>>>
>>>> We can use the cv2pdb tool to convert DWARF to .pdb files, which we
>>>> keep. The executables themselves get stripped, and also need to be kept; a
>>>> .pdb alone is useless. (Currently we partially strip exes, leaving in
>>>> function symbols so gcc can create a backtrace. I don't think that's worth
>>>> keeping anymore, except in the debug build. By fully stripping, we can
>>>> reduce each exe by another 300KB.) cv2pdb doesn't convert FB strings
>>>> correctly, but those are on the heap so not in the minidump anyway.
>>>>
>>>> To examine the minidumps, you can use Visual Studio (the free versions
>>>> will do) or windbg (CLI), or use a commandline tool or server-side to
>>>> generate a stracktrace and other info from it. CrashRpt has such a tool,
>>>> and scripts for handling large numbers of crash reports and sorting them.
>>>> BreakPad also has tools for it.
>>>>
>>>> == What we need to do ==
>>>>
>>>> Firstly the .pdb files should get uploaded somewhere. We will want to
>>>> keep the .pdb files and .exes for every nightly build - not just the weekly
>>>> archived nightlies! .pdb files alone are unfortunately useless.
>>>> I'm going to get rid of most of most of our windows nightly builds,
>>>> they're redundant now that you can switch graphics backend. So leaving just
>>>> music_sdl, music_native, music_native2 and music_sdl-debug. (In fact,
>>>> music_native and music_native2 appear broken and I couldn't find anyone who
>>>> uses them or even remembers using them.)
>>>> We should keep the files for at least every default (music_sdl) build
>>>> since at least the last stable release, probably longer since not everyone
>>>> upgrades promptly. Using xz, the game+custom exe+pdb files compress to
>>>> 2.4MB. So 4 builds, 3 months of backups is ~1000MB. **
>>>> An idea to save space: upload just the music_sdl files since the last
>>>> stable release, (~250MB) and keep all the other files on the build machine
>>>> in the unlikely event they're needed.
>>>>
>>>> (** however I just remembered that exes from the nightly build machine
>>>> are significantly larger than ones built on my machine)
>>>>
>>>> Secondly we need to collect reports - CrashRpt can send via HTTP,
>>>> HTTPS, SMTP and by opening an email client. I assume dreamhost will block
>>>> emails from random IPs, so we will want HTTP.
>>>> There is an example php script to accept reports and save them.
>>>>
>>>> http://crashrpt.sourceforge.net/docs/html/sending_error_reports.html#httpsend
>>>> I guess it would be easiest if we modify this script to email us the
>>>> zip file. And keep a copy on the server in case something goes wrong? Can
>>>> we put us up at hamsterrepublic.com?
>>>>
>>>>
>> _______________________________________________
>> Ohrrpgce mailing list
>> ohrrpgce at lists.motherhamster.org
>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.motherhamster.org/pipermail/ohrrpgce-motherhamster.org/attachments/20180906/de6b42ee/attachment-0001.html>