[Ohrrpgce] Importing bugzilla bugs into github

Ralph Versteegen teeemcee at gmail.com
Fri Mar 13 22:06:45 PDT 2020


I've adapted the script (which I will also upload to GH, but it's a fork of
a gist instead of a fork of the actual bugzilla2github repo; the history
needs cleaning) to also import from SourceForge to GitHub. I didn't like
the perl script I found to do the import, it wasn't flexible, I don't know
perl, and this way we get the same formatting for bugs from both sources.
That was pretty easy actually!

Bugs are in the test repo: https://github.com/ohrrpgce/testrepo/issues?q=

I made a change to the formatting: the original bug number is now at the
top of the description so that it shows in the mouse hover preview.

I added @arperry @Spencer-Zhang and @kveroneau to the usernames list.

I downloaded the (4) bug attachments from SF and uploaded them to the same
place as the bugzilla ones, so we're not reliant on SF at all.

I've updated the http://rpg.hamsterrepublic.com/bugzilla redirect to point
to GH. I think there were other redirects?
After the import I'll try make it redirect from to the original bugzilla
bug to the imported one, which should be easy because I will have the
lookup table for it.

On Sat, 14 Mar 2020 at 01:19, James Paige <Bob at hamsterrepublic.com> wrote:

> This looks fantastic! Very nice.
>
> I think the name mapping is a good idea.
>
> I see I reason not to proceed
>
> On Fri, Mar 13, 2020, 7:58 AM Ralph Versteegen <teeemcee at gmail.com> wrote:
>
>> Update! I have now completed making changes to and testing the
>> bugzilla2github tool. (I ended up making an awful lot.) I've imported all
>> the bugs into a test repository here:
>> https://github.com/ohrrpgce/testrepo/issues
>> This is the last chance to look over the results before I run it on the
>> real thing, which can't be undone.
>>
>> I have replaced names and emails for certain people with their github
>> accounts:
>> email2login = {
>>     "__name__": "email to GitHub login",
>>     "teeemcee": "rversteegen",
>>     "caron.mike": "pkmnfrk",
>>     "Bob": "bob-the-hamster",
>>     "Bob-bugzilla": "bob-the-hamster",
>>     "mkidder": "fyrewulff",
>>     "neota": "0ion9",
>>     "sonichedgehog_hyperblast00": "MirceaKitsune",
>>     "sorlok_reaves": "sorlok",
>>     "ziggy": "ziggythehamster",
>>     "alkabanfall": "raekuul",
>> }
>> Is this a bad thing? Otherwise you would see both email prefix and name,
>> e.g. "Keith Gable <ziggy>". (Note that noone receives notifications from
>> the imported bugs.)
>> I see I missed @arperry.
>>
>> I decided to squash "cosmetic" and "minor" severity levels into a "minor"
>> tag, and not convert the "major" and "critical" levels to tags, because I
>> found the application of those too dubious/spotty.
>>
>> I have an "os: unix" label, and no label for linux.
>>
>> There are bugs for OHRRPGCE FMF and Hamster Whisper (just the one).
>>
>> Note about the testrepo: I was testing out interrupting and resuming the
>> run, which exposed a bug, and also made a change to the script halfway, so
>> bugs before 529 have broken cross-reference links, and also show "bz#300
>> (#300)" instead of just "bz#300".
>>
>> I notice that links between duplicate bugs are one-way in the original
>> xml (unlike depends-on and blocked-by). Adding the reverse links to the bug
>> description might be the last change I make, since github doesn't generate
>> "mentioned by" cross-bug links if the mentioning bug hasn't been imported
>> yet.
>>
>> On Tue, 25 Feb 2020 at 23:25, Ralph Versteegen <teeemcee at gmail.com>
>> wrote:
>>
>>> Projects just seem like a handy way to maintain a todo list and an
>>> *indication* of goal for the next release without needing to actually file
>>> a bunch of feature requests in the bug tracker. The Plans on the wiki tend
>>> to be very long term and out of date. I would definitely continue to use
>>> the wiki (and emails) for detailed plans, not the bug tracker.
>>> My actual todo list is completely disorganised, unprioritised, too long,
>>> and often doesn't actually say what my todos for the next release are, so I
>>> don't even refer to it myself! A public todo list (people sometimes ask me)
>>> would be nice to have.
>>>
>>> Come to think of it, there's no real point putting projects for OHRRPGCE
>>> releases in the ohrrpgce organisation, because releases never depend on
>>> anything that might be in another repo (or in web or tools). And even if we
>>> had separate web/tools github repos, we may as well just keep issues for
>>> HWhisper, etc, all in the oe bug tracker. (There were bugs for HW and even
>>> Seth's OHR FMF on bugzilla anyway.)
>>>
>>> Splitting the git repos won't require any changes to svn. In fact, the
>>> web and tools branches could continue to be stored in the same git repo on
>>> the svn-git cron machine, and just get uploaded to separate github repos.
>>> And it's already possible for someone to clone the bitbucket/github repo
>>> three times locally, downloading just web or tools branches, or everything
>>> else. Complex to explain that to someone, though.
>>>
>>> On Mon, 24 Feb 2020 at 23:49, James Paige <Bob at hamsterrepublic.com>
>>> wrote:
>>>
>>>> I have no strong preference for whether we use projects or milestones.
>>>> I guess for bigger projects with more people there must be a use case for
>>>> both.
>>>>
>>>> Splitting web and tools into different git repos sounds good to me. If
>>>> that breaks anything regarding the subversion repo, I would be okay with
>>>> splitting them out of the svn repo also if needed.
>>>>
>>>> On Sun, Feb 23, 2020, 8:16 PM Ralph Versteegen <teeemcee at gmail.com>
>>>> wrote:
>>>>
>>>>> An update: I haven't worked on this for a couple weeks, but I was
>>>>> pretty close to having it working, just making tweaks. Want to get it done
>>>>> before I get distracted with this year's 7DRL. All the old attachments are
>>>>> uploaded (along with http://rpg.hamsterrepublic.com/bugzilla/quips.txt
>>>>> and http://rpg.hamsterrepublic.com/bugzilla/buglist.xml). Then I can
>>>>> look at Sourceforge.
>>>>>
>>>>> I'd never paid much attention to "Projects" on Github before, but they
>>>>> seem more useful than using Milestones for todo lists. Milestones let you
>>>>> reorder issues/PRs, but Projects let you also sort into columns, eg for
>>>>> priority. The project UI is very different. But the most important feature
>>>>> is that you can add note cards, not just issues. They can also contain
>>>>> issues from multiple repositories, although project-issue crosslinking only
>>>>> works if project and issue are in the same repo or it's a project in the
>>>>> 'ohrrpgce' organisation and the issue is in a repo under the ohrrpgce
>>>>> organisation**.
>>>>>
>>>>> Compare:
>>>>> https://github.com/ohrrpgce/ohrrpgce/milestone/1
>>>>> https://github.com/orgs/ohrrpgce/projects/1
>>>>>
>>>>> But milestones and projects seem pretty redundant to each other. I
>>>>> wonder whether to use milestones at all. (Maybe just to tag closed bugs,
>>>>> not as a todo list for a release?)
>>>>>
>>>>> ** Adding an issue to a project crosslinks it from the issue itself,
>>>>> see eg https://github.com/ohrrpgce/ohrrpgce/issues/5.
>>>>> We might want to add more repos to the ohrrpgce organisation, so could
>>>>> make sense to put the projects there instead of the repo. For example, I'm
>>>>> thinking of splitting 'web' and 'tools' branches out to separate git repos,
>>>>> since it's pretty inconvenient and strange to have them as orphaned
>>>>> branches.
>>>>> I'm also wondering whether a separate repo where crash reports are
>>>>> automatically uploaded to could be a handy way to manage them.
>>>>>
>>>>>
>>>>> On Sat, 8 Feb 2020 at 01:51, James Paige <Bob at hamsterrepublic.com>
>>>>> wrote:
>>>>>
>>>>>> That is even better!
>>>>>>
>>>>>> On Fri, Feb 7, 2020, 4:50 AM Ralph Versteegen <teeemcee at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> https://rpg.hamsterrepublic.com/bugzilla/ already exists (it's a
>>>>>>> redirect to the sf buglist, plus hosts
>>>>>>> https://rpg.hamsterrepublic.com/bugzilla/buglist.txt). So I guess
>>>>>>> the logical URL to use is
>>>>>>> https://rpg.hamsterrepublic.com/bugzilla/old-attachments/
>>>>>>>
>>>>>>> <https://rpg.hamsterrepublic.com/old-bugzilla-attachments/>
>>>>>>>
>>>>>>> On Fri, 7 Feb 2020 at 14:40, James Paige <Bob at hamsterrepublic.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Oops! Yes!
>>>>>>>>
>>>>>>>> How about putting them at
>>>>>>>> https://rpg.hamsterrepublic.com/old-bugzilla-attachments/ ?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Feb 6, 2020, 5:53 PM Ralph Versteegen <teeemcee at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Oh, right! The main reason I wrote that email was to ask about
>>>>>>>>> uploading the bug attachments somewhere. I see there are still various
>>>>>>>>> redirects and remnants of bugzilla left behind.
>>>>>>>>>
>>>>>>>>> On Fri, 7 Feb 2020 at 02:47, Ralph Versteegen <teeemcee at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I agree that too many labels aren't helpful. Labels are most
>>>>>>>>>> useful when you can't just do a text search (all crash bugs will be easily
>>>>>>>>>> found with a search for "crash"), eg for bug priorities.
>>>>>>>>>> I think maybe I'll combine "cosmetic" and "minor" into a single
>>>>>>>>>> "minor" tag, since we likely want that one anyway while "cosmetic" is too
>>>>>>>>>> specific. Wait, how much difference is there between "minor" and "needs
>>>>>>>>>> improvement"?  Plenty of outright bugs are merely cosmetic. Argh!
>>>>>>>>>> Really, "bug" is a region of the continuum (nearly the whole
>>>>>>>>>> continuum) from feature request to "needs improvement" to blocker.
>>>>>>>>>>
>>>>>>>>>> But... major/high priority bugs is possible the most likely thing
>>>>>>>>>> of all that we would want to categorise and search for. And they aren't
>>>>>>>>>> necessarily going to be put on the todo list for the next release
>>>>>>>>>> (milestone). So I guess I should map major/critical/blocker to "major".
>>>>>>>>>> Although the categorisation of bug severities on bugzilla is pretty
>>>>>>>>>> dubious, it is a useful starting point.
>>>>>>>>>>
>>>>>>>>>> 7d344553-34f0-0310-a9b1-970ce8f1c3a2 is the "Repository UUID"
>>>>>>>>>> you see when running "svn info".
>>>>>>>>>> It's also possible to give git-svn a file with mapping from
>>>>>>>>>> username to actual email, but that won't apply retroactively.
>>>>>>>>>>
>>>>>>>>>> On Fri, 7 Feb 2020 at 02:08, James Paige <Bob at hamsterrepublic.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Cool!
>>>>>>>>>>>
>>>>>>>>>>> I like the idea of rel: and resolved: prefixes.
>>>>>>>>>>>
>>>>>>>>>>> I vote for eliminating the "bug" tag completely. In general I
>>>>>>>>>>> say err on the side of fewer tags when the specific need for a tag is not
>>>>>>>>>>> obvious.
>>>>>>>>>>>
>>>>>>>>>>> That GitHub contributions trick is cool. I'll try it. Is
>>>>>>>>>>> 7d344553-34f0-0310-a9b1-970ce8f1c3a2 the internal ID of our svn
>>>>>>>>>>> repo?
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 6, 2020, 8:04 AM Ralph Versteegen <
>>>>>>>>>>> teeemcee at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, 7 Feb 2020 at 01:43, Ralph Versteegen <
>>>>>>>>>>>> teeemcee at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I've been putting this off for years (two years since my last
>>>>>>>>>>>>> attempt in fact) but I've finally managed to get bugzilla running locally
>>>>>>>>>>>>> and import the databases so that I can export the buglist as xml and use this
>>>>>>>>>>>>> script
>>>>>>>>>>>>> <https://gist.github.com/Zimmi48/d923e52f64fe17c72852d9c148bfcdc6#file-bugzilla2github>
>>>>>>>>>>>>> to import it into github.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm now in the process of customising the script and the
>>>>>>>>>>>>> mapping to labels.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Github doesn't have an API that allows attaching files to
>>>>>>>>>>>>> issue comments, so I suggest that we upload all the attachments to a
>>>>>>>>>>>>> directory on hamsterrepublic.com so that we can link to them.
>>>>>>>>>>>>> (Unfortunately I first have to write a script to extract them
>>>>>>>>>>>>> out of the bugzilla DB.)
>>>>>>>>>>>>>
>>>>>>>>>>>>> With all the labels for components, versions, severity,
>>>>>>>>>>>>> status, etc, we'll have a lot of labels. I think it makes sense to organise
>>>>>>>>>>>>> our labels a bit better. All version labels could be prefixed with "rel: ",
>>>>>>>>>>>>> like "rel: alectormancy". And maybe duplicate, wontfix, invalid,
>>>>>>>>>>>>> worksforme, tooconfusing
>>>>>>>>>>>>> could be prefixed with "resolved:" or "closed:"?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm going to translate the "feature request", "cosmetic",
>>>>>>>>>>>>> "blocker" severities into labels. I think I'll leave behind "major",
>>>>>>>>>>>>> "minor", "normal" and "critical", and the priority levels. They (if not
>>>>>>>>>>>>> default value) will still be in the bug description.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There isn't really anything that corresponds closely to our gh
>>>>>>>>>>>>> "bug" and "needs improvement" labels. I wonder whether we should just
>>>>>>>>>>>>> remove the "bug" label, since although it's informative and useful in
>>>>>>>>>>>>> searches, it's already a nuisance to accurately tag all the bugs** and
>>>>>>>>>>>>> we'll get an extra 1000 bugs missing it. "bug" is basically the absense of
>>>>>>>>>>>>> numerous special categories like "new feature", "invalid" and "tracking".
>>>>>>>>>>>>>
>>>>>>>>>>>> Or, I could just apply "bug" as a guess according to something
>>>>>>>>>>>> like that simple rule, and we can fix up any mistakes. Will be looking
>>>>>>>>>>>> through all the open imported bugs anyway.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> **See
>>>>>>>>>>>>> https://github.com/ohrrpgce/ohrrpgce/issues?utf8=%E2%9C%93&q=-label%3Abug+-label%3A%22needs+improvement%22++-label%3A%22new+feature%22
>>>>>>>>>>>>>
>>>>>>>>>>>>> Bugzilla hides people's email addresses (by stripping the
>>>>>>>>>>>>> domain) by default so user names will be something like "David Gowers
>>>>>>>>>>>>> <00ai99>". I think that's fine. I can (am planning to) also replace the
>>>>>>>>>>>>> username with a github @ mention (it shouldn't cause any notifications,
>>>>>>>>>>>>> because this is the mass import API), but the only people who I know the
>>>>>>>>>>>>> github username of are myself, James and Mike. Doesn't matter.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> BTW, I found that it's possible to make svn commits copied to
>>>>>>>>>>>>> the git-svn mirror on github count towards your "contributions" shown on
>>>>>>>>>>>>> your github profile page. To do so, add a new email address to your github
>>>>>>>>>>>>> account with the form svncommitername at 7d344553-34f0-0310-a9b1-970ce8f1c3a2,
>>>>>>>>>>>>> where svncommitername is e.g. teeemcee or james. The email can be private.
>>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Ohrrpgce mailing list
>>>>>>>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>>>>>>>>
>>>>>>>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Ohrrpgce mailing list
>>>>>>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>>>>>>>
>>>>>>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>> Ohrrpgce mailing list
>>>>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>>>>>
>>>>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Ohrrpgce mailing list
>>>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>>>>
>>>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Ohrrpgce mailing list
>>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>>>
>>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Ohrrpgce mailing list
>>>>>> ohrrpgce at lists.motherhamster.org
>>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>>>>
>>>>> _______________________________________________
>>>>> Ohrrpgce mailing list
>>>>> ohrrpgce at lists.motherhamster.org
>>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>>>
>>>> _______________________________________________
>>>> Ohrrpgce mailing list
>>>> ohrrpgce at lists.motherhamster.org
>>>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>>>
>>> _______________________________________________
>> Ohrrpgce mailing list
>> ohrrpgce at lists.motherhamster.org
>> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>>
> _______________________________________________
> Ohrrpgce mailing list
> ohrrpgce at lists.motherhamster.org
> http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.motherhamster.org/pipermail/ohrrpgce-motherhamster.org/attachments/20200314/ce3ca49f/attachment-0001.html>


More information about the Ohrrpgce mailing list