Rebasing a git branch that has a pull request

I recently wanted to merge a pull request in a git repository, where a lot had changed in master since the pull request was opened. The owner of the repository wanted me to do a rebase instead of merge, so I this is what I figured out:

Step 1) Update your branches.
Update all branches from the remote repository to make sure everything is up to date (newfeature is your branch with the pull request, develop is the branch it should be merged to):

$ git checkout develop
$ git pull
$ git checkout newfeature
$ git pull

Step 2) Perform the rebase:

git rebase develop
...
CONFLICT (content): Merge conflict in <bla>
error: Failed to merge in the changes.
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

After each time git stops with an error, resolve the conflicts and use git add. Do not commit anything during the rebase. Instead, just continue the rebase until everything is fine.

git rebase --continue

Now, you will have some modified files. You can make some final changes if necessary, and then commit your changes:

git add <bla> <blubb>
git commit -m "Resolve conflicts from rebase"

Step 3) Push your changes to the remote repository.
Unfortunately, using only git push will not work, but give the message “Updates were rejected because the tip of your current branch is behind hint: its remote counterpart. Integrate the remote changes (e.g. hint: ‘git pull …’) before pushing again.”
Do NOT use git pull at this moment!!
Instead, force push your changes onto the remote repository:

git push --force-with-lease

Now, you have overwritten the history on the remote repository with your local history. That is fine. Now you can merge the pull request and delete the branch and go on your merry way.

Step 4) BUT…
If you are not yet at the place where you merge the pull request, but someone else wants to work with the branch, that person is in trouble! A normal git pull on the branch will fail, because history was changed! What you need to do is:

git pull --rebase

See also: Gerald Versluis: Git Rebase: Don’t be Afraid of the Force (Push)

Version conflict in Google protobuf on EMR cluster

I have written a Spark application that uses a library that uses the Google protobuf library in version 3.3.1. On my computer I can run it with my local Spark and everything is fine. But now, I want to run it on an EMR cluster in on AWS. And I get this:

java.lang.NoSuchMethodError: com.google.protobuf.CodedInputStream.readStringRequireUtf8()Ljava/lang/String;

It seems that this is a common incopatibility issue (says stackoverflow). Java libraries are loaded from the classpath. The classpath is searched from front to back. And in the EMR cluster, Spark and Hadoop libraries are inserted before the user libraries. So the old protobuf version is found.

But there is the possibility to change this behaviour described in the Spark 2.4.4 documentation. So I added two corresponding options to spark-submit in the EMR cluster step:

--conf spark.driver.userClassPathFirst=true 
--conf spark.executor.userClassPathFirst=true

And it worked!

NB: The proposed solution at stackoverflow is to create a “shaded” fat jar. That probably works as well and should be simple for Maven users. Maybe my next post is about that 😉

Login to AWS with the Java SDK and role assumption

The following seems to work for me as contents of .aws/credentials:

[default]
region = eu-west-1
role_arn = arn:aws:iam::12345:role/blubb
source_profile = blabla
[blabla]
aws_access_key_id = XXX
aws_secret_access_key = YYY
aws_session_token = ZZZ
valid_until = DDD

The content of .aws/config does not seem to matter. So it can be whatever it needs to be for the CLI.

Throw away your change and reset a git branch to remote

Sometimes, you do something really stupid and just want to get rid of it. Or you are suddenly in the middle of a complicated merge and don’t really know why anymore (“But I didn’t change anything!”). In this case, if you are sure you want to throw away everything in your local branch and just want to be at the same status as the remote branch, this is your rescue:

git reset --hard origin/<branchname>

But be careful, the --hard option deletes all changes that you have made without recovery.

Jameica in aktueller Version mit jVerein verwenden (H2 DB Version)

Ich verwende ein lokal über Homebrew installiertes jameica für unsere Vereinsverwaltung mit jVerein:

brew cask install jameica

Damit sowohl das jameica Plugin hibiscus zur Anbindung an unsere Bank als auch das jVerein Plugin unter jameica 2.8.6 funktionieren, muss man momentan im jameica Anwendungsverzeichnis (Mac Show Package Contents jameica.app) im Verzeichnis lib/h2 das dort liegende neuere h2 jar durch h2-1.4.196.jar ersetzen.

Spendenbescheinigung mit jVerein erstellen

Um eine Spendenbescheinigung automatisch erzeugen zu können, muss die Buchung korrekt als Spende markiert sein und einem Mitglied zugeordnet sein. Das geht folgendermaßen:
1. Unter “JVerein – Buchführung – Buchungen” die entsprechende Buchung finden und in die Detailansicht gehen.
2. Buchungsart “Spende” (oder wie auch immer die entsprechende Kategorie heißt) auswählen. Bei “Mitgliedskonto” die Punkte klicken, auf den Tab “Soll und Ist” und unten in der Tabelle die korrekte Person auswählen (dort lief eine Suche über die Mitglieder nach dem Namen auf der Überweisung, falls da nicht das korrekt Mitglied gefunden wurde – oh je!). Als Ergebnis steht jetzt im Feld “Name, Sollbuchung erzeugen”

Wenn das bei allen Buchungen korrekt verbucht wurde, können die Bescheinigungen erzeugt werden. Das geht so:
1. “JVerein – Spendenbescheinigungen”
2. Unten “neu (automatisch)” klicken.
3. Korrektes Jahr und Vorlage auswählen. Dann sollte unten eine Liste der Spenden erscheinen.
3. Auf “erstellen” klicken.
4. Jetzt sollten die entsprechenden Einträge unter “Spendenbescheinigungen” auffindbar sein. Jeden Eintrag dort auswählen und einmal auf “speichern” klicken und den Eintrag wieder schließen (unbedingt nötig!!!)
5. Den Eintrag neu öffnen und mit “pdf (individuell)” ein pdf mit der Spendenbescheinigung erzeugen lassen.

Nun kann die Bescheinigung gedruckt, verschickt oder was auch immer werden.

Install freetz on FritzBox 7390 from a mac

A pretty silly reason made me try to switch our FritzBox from original AVM firmware to Freetz custom image today – I wanted to be able to use .local dns entries while connected via VPN which seems not be possible otherwise.

Well, I always wanted to play around with Freetz, the 7390 is anyhow old and no longer under active support by AVM, so let’s get started:

1) Get Freetz sources, because due to legal reasons we need to build this on our own
$ git clone https://github.com/Freetz/freetz.git

2) However this won’t build without extra fiddling from my mac. The official guidance is to use a VM, but I had docker ready and quickly found a docker image for that:
$ docker pull mtneug/freetz

3) Quite some fiddling/reading later I figured I might be best of to use it like this:
$ docker run -it -v $PWD:/freetz -v $PWD/images:/freetz/images -v /Users/hro/:/home/freetz mtneug/freetz /bin/bash
(the freetz mount is to just use the downloaded freetz source dir as base, freetz images because that’s where the final image is written and the home mount is to store a key used for signing in the users home dir instead of the freetz build dir)

4) When inside the docker container run
$ make menuconfig && make

5) Exit and now we can push the firmware via a cmd tool
$ tools/push_firmware images/<your_image.image> ip

Clean Spark environment in Zeppelin

When you have experimented a lot, there are many things in your notebook and quite often, you are not sure what is there and what not. Zeppelin does not really give you a good tool to do that, so I wrote my own function to clean up the global variables of Python:

def clear(keep=("__builtins__", "clear", 'completion', 'z', '__zeppelin_completion__', '_zsc_', '__zSqlc__', '__zeppelin__', '__zSpark__', 'sc', 'spark', 'sqlc', 'sqlContext')):
    keeps = {}
    for name, value in globals().iteritems():
        if name in keep: 
            keeps[name] = value
    globals().clear()
    for name, value in keeps.iteritems():
        globals()[name] = value

You can then call the function with clear().

Additionally, if you are dealing with libraries, you may want to explicitly remove them with del sys.modules['modulename'].