About swk

I am a software developr, data scientist, computational linguist, teacher of computer science and above all a huge fan of LaTeX. I use LaTeX for everything, including things you never wanted to do with LaTeX. My latest love is lilypond, aka LaTeX for music. I'll post at irregular intervals about cool stuff, stupid hacks and annoying settings I want to remember for the future.

Cloudformation templates for existing resources

You can use the AWS CLI to get a description of existing resources. This description is in JSON format, which can sometimes directly be used as a Cloudformation template.

Example:

aws glue get-job --job-name MyJobName

Makes the job easier! For example with the Glue job, we can see the undocumented options

"--enable-metrics": ""
"--TempDir": "s3://blablubbtest"
"--enable-continuous-cloudwatch-log": "true"

So since Cloudformation is a pain to debug, a possible way to write a template might be to click together the resources with the Console, then get the description with the CLI and use this to create the resources with Cloudformation next time.

Disable graphical prompt for ssh passphrase

When I open a ssh session in the terminal, it asks for my passphrase in a graphical prompt window. That would be ok in theory. But I don’t know my passphrase. So I need to copy it from my password manager. And unfortunately the stupid window doesn’t allow me to access anything else. So, I wanted to disable it.

The usual way is with the environment variable SSH_ASKPASS. To disable the graphical prompt, just remove the value of this variable:

unset SSH_ASKPASS

Unfortunately, in my case this did not work and I needed to remove also another variable:

unset SSH_AUTH_SOCK

Pip and custom prefixes… again! This time it’s Ubuntu’s fault

I wanted to install a Python library to a custom location. Thanks to a long fight with Python on that issue (I can’t believe I haven’t blogged about this!), I know that --prefix does the trick for pip. So I run pip and this happens:

> pip3 install --prefix tmp/ boto3
ERROR: Can not combine '--user' and '--prefix' 
as they imply different installation locations

Alternatively the error is:

distutils.errors.DistutilsOptionError: can't combine user
with prefix, exec_prefix/home, or install_(plat)base

It seems to be an option that Ubuntu adds by default. The magic solution comes from a GNU bug tracker thread:

> pip3 install -U pip

Basically, this installs pip into my user directory (you can find it now in .local/bin/pip). pip3 still fail afterwards with a version mismatch:

> pip3 install --prefix tmp/ boto3
Traceback (most recent call last):
  File "/usr/bin/pip3", line 9, in <module>
    from pip import main
ImportError: cannot import name 'main'

But now I can call my local pip (which is a pip3):

> pip install --prefix tmp/ boto3
Collecting boto3
...
Successfully installed boto3-1.9.206 botocore-1.12.206

To force a re-install, even if the library is already installed somewhere else, use the flag --ignore-installed.

Add shared Exchange calender to Evolution Mail

I just spent a lot of time searching, so I really need to make sure not to forget this. I want to see a colleague’s calendar. Everything is in Office 365, so I have no problem adding the calendar in the web interface. Naive me thought, this will cause Evolution Mail to show the calendar also. Nope.

I searched for a way to add a calendar in the calendar view of Evolution. With right click on the account or in some menu. Nope. I can add calendars for myself, but not get a colleague’s calendar. Of course. That would have been too easy!

So what does work? Thank you jldugger:

On the Mail view, right click your exchange account, select “Subscribe to folder of other EWS user.” Type in the name of the account you want, and choose “Calendar” from the dropdown.

For me “Free/Busy as Calendar” worked, “Calendar” did not.

Copy your pictures from iPhone with Linux

First, you will need to install iFuse:

sudo apt install iFuse

Then, it is really simple:
1. Connect your iPhone with USB to your laptop.
2. Unlock your iPhone.
2. Run idevicepair pair.
3. The phone will ask if you want to trust this computer, say yes.
3. Open the folder Dolphin/file browser. For me, I get a “open camera” notification that leads me to the picture folder in the file browser. You can also try camera://<id given by idevicepair> or afc://<id given by idevicepair>.
4. Copy the files!

When I did this, I had to reset the connection with idevicepair a few times, but other than that it worked very well.

If this does not work for you, try this very thorough guide by Dedoimedo/Igor Ljubuncic: How to access iPhone (6s) with iOS 11 in Linux

Tree view in Nautilus on Ubuntu 18.04

Someone thought it a good idea to remove the settings dialogue in Nautilus. Whatever. Here is how to set “list view” as default and enable tree view:

gsettings set org.gnome.nautilus.preferences default-folder-viewer 'list-view'
gsettings set org.gnome.nautilus.list-view use-tree-view true

The REAL internet is waiting!

Did you know, that you can be connected to the internet with a cable and still be in the wrong subnet? That’s what happened to me. I plugged in the cable and the internet worked. Or so I thought. Then I needed to download a file from a company internal location and the server was unreachable. But for my colleague it worked without problems. Mhm. Typo? My colleague sent me a link – no luck. DNS? I tried the IP directly – no luck. Restart? Advice from IT – but no luck.

After involving two more colleagues, we figured it out: There is a thing called 802.1x Security Authentication. Basically, after connecting to the internet, you still need to enter your user credentials to be allowed into the internal parts of the network. In my company’s case, they use PEAP. Apparently, Windows and Mac usually ask for the authentication automatically when connecting to a network that offers this method. Hence the advice by IT to restart. Well, Linux doesn’t ask. You need to know how to answer! When you know it, it’s easy: In Gnome activate the method under “Network Settings” – “Security”. You may need a certificate – ask your admins!

I learned something new today.

Microsoft Teams on Linux

I use Linux in a corporate environment where Microsoft Office 365 is the toolchain of choice. One of the first things I needed to do my job was Microsoft Teams. Here are two things that worked for me.

The first (obvious) idea is to use the web UI. Messaging works out of the box with Chromium or Google Chrome. Getting audio and video calls to work is not obvious anymore. Basically, you need to pretend that you are on a Windows machine. This can be done by setting the “User Agent” to “Edge – Windows” with in the developer tools – see Christian Hujer’s blog post on the topic. Unfortunately, this hack breaks the message history. And there seems to be no way to get screen sharing to work. So while this solution is very easy, it is not optimal.

The second thing I tried and which works fine for me, is the unofficial MS Teams client by Ismael Martinez. I have it running on Ubuntu 18.04. As far as I have seen, everything works. Thank you!

Delete data on a disk

If you want to give away a computer and you want to really delete the data on the hard drive, you can use dd under Linux. Start a live linux from a USB drive (for example Ubuntu) on the computer where you want to erase the disk.

First, find out which partitions you have. A basic way of doing this is by using fdsik:

sudo fdisk -l

Most distributions will have a graphical editor which makes it easier to see what is going on. In gnome this will be gparted, in KDE KDE partition manager. There are surely other tools around. But it does not really matter, all you need is to know the name of the partition which you want to erase. In my case it is /dev/sda4.

Now we will use dd (“disk dump”) to write random information to this partition on top of the existing information. This is the command:

sudo dd if=/dev/urandom of=/dev/sda4 bs=65536 status=progress

The parameter of (“output file”) is the hard disk partition we want to write to. You do not want to mess up and take the wrong partition. The is no “undo”. Check this parameter twenty times. The parameter if (“input file”) is used to set the data which should be written. In our case, we use /dev/urandom which is a generator for random numbers. status=progress will enable some output on the command line which tells us what is happening. If you forget this parameter, there will be no output and you will have no idea if 1 byte has been written in the last hour or 200 GB. Setting the block size (bs) to something larger than the default 512 bytes is also very highly recommended. The difference in the time the command needs to run may be huge. I use 64k in the example which worked fine for me. If you want to determine the optimal block size for your system, I suggest this article: Tuning dd block size by Danny Guinther.

Anaconda and environments (basics)

TLDR: Install Anaconda NOT into your path, then do:

ln -s ~/anaconda3/bin/activate ~/bin/activate
source activate
conda create --name myenv
conda activate myenv
python

Long version: Install Anaconda. In the process you will be asked whether you want to edit your .bashrc to setup Anaconda. Answer NO!!

You will still need to add the Anaconda executables into your path to make Anaconda work. The easiest solution that does not destroy your system is to link the activate script somewhere in your path. I do this in a folder ~/bin which is always on my path:

ln -s ~/anaconda3/bin/activate ~/bin/activate

Now call

source activate

The text (base) will be prepended to your prompt and the Anaconda binaries will be in your paths. This should be more or less equivalent to what happens when you call conda init, but without changes to your .bashrc. You could now call python and it should print “[GCC 7.3.0] :: Anaconda, Inc. on linux” instead of your default operating system python.

If you don’t do anything, you have the base environment activated. First thing you want to do, is create a new environment and activate it. Then you can install your few packages into it and work with this environment. The advantage is, that you can throw away this environment if anything goes wrong and start from scratch easily.

These are the most important commands for dealing with environments (in my examples myenv is used as a name for the environment, but of course you can use a better one):
– List all available environments: conda info --envs
– Create an environment: conda create --name myenv
– Delete an environment with all contained packages: conda remove --name myenv --all
– Activate an environment: conda activate myenv
– Deactivate the current environment: conda deactivate
– List all packages installed in the current environment: conda list
– Install a package into the current environment: conda install packagename
– Delete a package from the current environment: conda remove packagename

When you start python with python from an activated environment, you will have all packages in this environment available to you.