Robert Basic's blog

Installing Python2 with Ansible

by Robert Basic on June 29, 2016.

Ansible uses Python2 to run the provisioning commands on the host machines. At this time it does not support Python3, which is the default python version in Fedora releases for quite some time now.

So to be able to manage Fedora machines with Ansible, I need to install Python2, but how to install it when all the Ansible modules depend on Python2 being installed? Turns out it’s quite simple, by turning of the gathering of facts in Ansible and using the raw module to install the required packages:

- hosts: all
  gather_facts: no
  become: yes
  tasks:
    - name: Install python2 and python2-dnf
      raw: dnf -y install python2 python2-dnf
    - name: Gather facts
      setup:

Just remember this needs to be the very first thing that happens on all your Fedora hosts. After python2 is installed, gather the facts for all the hosts by running the setup module.

Happy hackin’!

Tags: ansible, fedora, provisioning, python, vagrant.
Categories: Development, Software.

Creating a PostgreSQL user in Vagrant with Ansible

by Robert Basic on June 28, 2016.

Lately I’ve been playing around with provisioning a PostgreSQL server with Ansible in a local Vagrant machine that runs a Fedora 23 image.

The first task after installing and starting the PostgreSQL server is to create a database user and a database. So far I have found an ugly way, a really ugly way and a nice way to do this.

How it should be done

The proper way to do this would be to use the postgresql_user Ansible module and the become, become_user and become_method directives, like so:

- name: Create a PostgreSQL database user
  postgresql_user: name=project password=project role_attr_flags=CREATEDB state=present
  become: yes
  become_user: postgres
  become_method: sudo

But this fails because sudo expects us to enter the password:

TASK [postgresql : Create user] ************************************************
fatal: [default]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "sudo: a password is required\r\n", "msg": "MODULE FAILURE", "parsed": false}

You can read more about privilege escalation in Ansible in their documentation.

The really ugly way

This solution is so bad I’m not even sure I should write it down. It depends on changing the default identification method for local connections from peer to the trust method, so we can use the default vagrant user to create new users without any checks, based only on, well, trust.

- name: Change peer identification to trust
  shell: /bin/sed -i '/^local/s/peer/trust/' /var/lib/pgsql/data/pg_hba.conf
  notify: restart dbserver

- meta: flush_handlers

- name: Create a PostgreSQL database user
  postgresql_user: name=project password=project role_attr_flags=CREATEDB state=present

- name: Change trust identification back to peer
  shell: /bin/sed -i '/^local/s/trust/peer/' /var/lib/pgsql/data/pg_hba.conf
  notify: restart dbserver

- meta: flush_handlers

This is just bad, there must be a better way.

The less ugly way

But still ugly. This is based on running a psql command using the shell Ansible module.

- name: Create a PostgreSQL database user
  shell: sudo -u postgres bash -c "psql -c \"CREATE USER project WITH CREATEDB PASSWORD 'project';\""

This one has an additional problem of that it only works when we run it for the first time, because we can’t create the same user twice. A possible solution would be to wrap the CREATE USER ... in an additional IF NOT EXISTS (SELECT * FROM pg_catalog.pg_user ... query, but that’s just… Ugh. No.

Back to square one

Let’s go back to the way how it should be done, by using the become and become_user directives. But how do we handle the sudo password? We tell sudo to not ask for a password by editing the /etc/sudoers files. The line to add is:

vagrant ALL=(postgres) NOPASSWD:/bin/sh

This tells sudo that the user vagrant on ALL hosts can run the /bin/sh program with NOPASSWD as the user postgres. I’m explicitly limiting the possible commands to /bin/sh as that is the only command we need to be able to run to make things work. I don’t want to add more if I don’t need to.

The Ansible tasks are now:

- name: Enable passwordless sudo
  lineinfile: dest=/etc/sudoers regexp=^vagrant line="vagrant ALL=(postgres) NOPASSWD:/bin/sh"

- name: Create a PostgreSQL database user
  postgresql_user: name=project password=project role_attr_flags=CREATEDB state=present
  become: yes
  become_user: postgres
  become_method: sudo

For added bonus we can cleanup the sudoers file after we are done by removing the line we added.

Happy hackin’!

Tags: ansible, postgresql, provisioning, vagrant.
Categories: Development, Programming, Software.

Helping juniors debug

by Robert Basic on June 23, 2016.

These days I spend most of my time reviewing code. Lots and lots of code. It’s mostly written by juniors and some of it is good, some of it is bad. I try to be patient, to be a good mentor, to hopefully teach (and be taught), while not letting out of sight that the most important thing is that the code does the right thing in the right way. The business and the users come first, after all.

When I encounter a piece of code that looks like a bug, I run the code myself to verify it. After that I usually ask a short question, something like “Hey, what does this code do? I think there’s a bug in it”. There are times when the author figures it out on their own and fixes it, or asks for help. If the former, great, if the latter, I either give them some time to figure it out on their own, or help them debug it — depending on the difficulty of the bug.

What I don’t do is outright tell them what the bug is, or even worse, how to fix it without giving them the chance to understand the bug. How are they going to learn like that?

Granted, guiding them through the entire debugging process can be slow and difficult, so this might not be a good approach when the deadline is close.

I do believe that taking the time to guide them can be rewarding for all parties involved. They get to learn by doing, get an insight on how a more senior developer works and thinks, and I, the very least, get to improve my communication skills by being forced to explain my thought process in words.

What are you trying to do?

That’s usually my first question. It’s important to understand what they think is going on, so I can guide them better. It also prevents me from assuming they know something that might be fundamental knowledge for a senior, but not so much for a junior.

My end goal is not just to fix a bug, but to actually teach, help them understand what is going on, so the next time they can fix the bug on their own, or prevent it from happening in the first place.

What does the code do?

Next thing I ask for is an explanation of the code, line by line if needed. This is probably the slowest part of this process, but I think it is necessary as it can point out places where their knowledge is lacking. They might don’t understand a language feature, misread how an internal API method needs to be called, think that the code does one thing, but actually it does something else, because hey! that’s how it works. And we’ve all been in a situation when we misunderstood something. It’s OK. Our job is to teach them, just as we were taught.

During this part I try to not interrupt them, I let them finish. I keep track of all the bits they got wrong and go back to those one by one once they are done with explaining the code.

Again I try not to flat out tell them what they got wrong, but to guide them to the correct answer. Asking simple questions like “Are you sure that it’s doing what you think it’s doing?”, or “Have you read the manual entry for that method call?” can be at times enough to make them see their mistake. “Oh, I’m passing a string and it should have been an array!” (side note: PHP 7 can’t come soon enough to all of our projects).

If they just don’t know what they got wrong, explain it to them. Don’t try to skip over any parts that might seem obvious to you — it might not be so obvious for them.

Rinse and repeat for all the parts they got wrong.

Let’s find that bug

Now when they have a better understanding of the code, ask them to find the root cause of the bug. I repeat myself, but guiding them to the moment of discovery is much better than just leading them straight to the answer. Let them have their “A-ha!” moment. I know those moments are the best moments for me.

At this point they should be able to tell you what be a good fix for the bug. If it’s a good fix, great. If not, ask them something like “Do you think doing X would be a better approach?” and explain why it would be.

This is a long process that can take up quite some time, even up to an hour or an hour and a half. You’ll probably need to do it more than once because, well, there are many types of bugs and not all bugs can be debugged the same way, but it is going to be rewarding for both you and the junior. And the more you do it, the faster it will be.

Until one day when they will be teaching the next junior on the team how to debug.

Tags: debugging, mentoring, teaching.
Categories: Development, Programming.

Import custom Python modules in Vim plugins

by Robert Basic on June 16, 2016.

This took me a while to figure out so I’m writing it down for future self and anyone else who needs it.

I started writing a new Vim plugin that will use the Python interface as most of the work will be done there, mostly to keep my sanity.

Having a plugin layout such as:

.
├── lib
│   └── mypymodule
│       └── ham.py
└── plugin
    └── my-vim-plugin.vim

I want to be able to do a

from mypymodule import ham

from within the my-vim-plugin.vim file.

For that to happen the <sfile> command line special comes to rescue. It is the file name of the sourced file in Vim, that is, the file name of the Vim plugin. Using the :p and :h file name modifiers it gives us the full path to the plugin directory of our plugin.

" This will give something like
" /home/robert/projects/my-vim-plugin/plugin
let g:plugin_path = expand('<sfile>:p:h')

And here comes the kicker: the <sfile> needs to be expanded outside of our Vim function where it is used, otherwise the <sfile> points to the path of the file that called the Vim function.

In code, getting the path to the sourced file from within the Vim function would be incorrect:

" ./plugin/my-vim-plugin.vim
function! MyVimPlugin()
python << endpython

import vim

vim.command("let a:plugin_path = expand('<sfile>:p:h')
plugin_path = vim.eval("a:plugin_path")
print plugin_path

endpython
endfunction

because it would end up printing the current working directory from where the MyVimPlugin function is called.

The correct way to do is to get the path to the sourced file outside of the Vim function:

" ./plugin/my-vim-plugin.vim
let g:plugin_path = expand('<sfile>:p:h')

function! MyVimPlugin()
python << endpython

import vim

vim.command("let a:plugin_path = expand('<sfile>:p:h')
plugin_path = vim.eval("a:plugin_path")
print plugin_path

endpython
endfunction

This way the path is set at the time the plugin is sourced and not at the time when function is called.

Finally, to be able to import the mypymodule from the lib, we need to point to the lib directory and add it to the system paths. Complete example:

" ./plugin/my-vim-plugin.vim
let g:plugin_path = expand('<sfile>:p:h')

function! MyVimPlugin()
python << endpython

import os
import sys
import vim

" Get the Vim variable to Python
plugin_path = vim.eval("g:plugin_path")
" Get the absolute path to the lib directory
python_module_path = os.path.abspath('%s/../lib' % (plugin_path))
" Append it to the system paths
sys.path.append(python_module_path)

" And import!
from mypymodule import ham

endpython
endfunction

By the way, here’s the documentation for the Vim Python module.

Happy hackin’!

Tags: plugin, python, vim.
Categories: Development, Programming.

GitHub flavoured code fences in Hugo

by Robert Basic on March 28, 2016.

This was an undocumented feature until today, so I missed it when I was converting my site to Hugo last week. It is also possible to highlight code examples with GitHub flavoured code fences, or, in other words, with triple backticks ```.

I like this a lot because it makes highlighting code in posts easier. Typing the {{< highlight >}} shortcode is just awkward and I always end up forgetting either the angle brackets or add too much or too little currly brackets. Backticks are much nicer.

The code fences are not enabled by default, though. We need to set PygmentsCodeFences to true in Hugo’s configuration file and that’s about it. Everything else about syntax highlighting stays the same.

Change to backticks in old posts

I used these two simple sed one-liners to change all the highlighting shortcodes to the code fences:

find -name "*.md" -print0 | xargs -0 sed -i 's/{{< highlight \([a-z]*\) >}}/``` \1/g'
find -name "*.md" -print0 | xargs -0 sed -i 's/{{< \/highlight >}}/```/g'

Now I don’t even need a custom Vim function to insert the highlighting shortcode. Sweet.

Tags: highlight, hugo, syntax.
Categories: Development, Programming, Software.