Archive for the 'Software' category

Docker nginx host not found in upstream error

published on January 30, 2018.

I’ve been toying around with Docker for the past couple of days, mostly to learn more about it, to understand it better. I just didn’t bother with it until now.

I started from scratch. Installing Docker, configuring it (I really don’t appreciate it filling my root partition with images), and, well, using it. I sort of figured out the docker command line interface, I get the difference between images and containers, I know how to write a Dockerfile, and when all the commands and options and flags start get confusing I know where to look in the help for help.

Happy with the progress I made, it was time to start connecting different containers so that they can talk to each other. Starting with a single container with nginx in it, and another container with php-fpm in it. Using their official images even.

To keep my sanity intact, as much as it is possible with software these days, I heeded Vranac’s advice and installed docker-compose for that.

No cheating, so I wrote my own docker-compose.yml file using just the documentation:

version: '3'
services:

    php:
        image: php:7.2-fpm
        expose:
            - "9000"
        volumes:
            - ./app:/app

    nginx:
        image: nginx:stable
        ports:
            - "8080:80"
        volumes:
            - ./site.conf:/etc/nginx/conf.d/default.conf
            - ./app:/app
        depends_on:
            - php

And now let’s build it and bring up the containers:

$ docker-compose build
$ docker-compose up

Aaaaaaand… It dies with an error like:

nginx: [emerg] host not found in upstream "php" in /etc/nginx/conf.d/default.conf:15

Blergh. I guess I missed something from the documentation. Fast-forward 2 hours, dozens of google searches and articles, countless rewrites of the docker-compose.yml file, and zero luck. Whatever I did, same error: “host not found in upstream”.

Then I finally remembered. What is the one thing that always causes me grief when trying to work with a web server? That’s right: SELinux!

Turn off selinux, restart docker, build && up, and it works. Sonofa. Works even with the very first version of docker-compose.yml I wrote.

OK, turning off selinux can’t be the solution, so I searched more… And no one, ever, recommends, or even mentions, that selinux might be the problem. I’ve installed the Docker SELinux package (it’s container-selinux on my Fedora 26). It should be working!

Another hour later, more searches and articles, I end up at the beginning, at the “Get Docker CE for Fedora” documentation page. Docker CE? What the fresh hell is this?

Well, it’s the docker version I should’ve installed in the first place.

Fedora’s repos have docker version 1.13.something. Docker-CE is at 17.12.something.

Remove old docker, re-enable selinux, install new docker, everything works just fine, and run the following:

sudo ausearch -c 'iptables' --raw | audit2allow -M my-iptables
sudo semodule -X 300 -i my-iptables.pp

I have no idea what that does, but it was required to make it work.

sigh

Happy hackin’!

Build and run Golang projects in VS Code

published on January 24, 2018.

I’ve been using VS Code for my Golang development needs for a few months now. Minor kinks here and there, nothing serious, and the development experience gets better with every update. I have also tried out IntelliJ Idea as the editor, and one feature that I’m missing in Code from Idea is the build-run-reload process. I thought to myself, that’s such a basic feature, it should be possible to have that.

And it is! VS Code Tasks to the rescue!

These tasks allow us to run different kind of tools and, well, tasks inside VS Code.

Go to Tasks -> Configure Default Build Task and then select the “Create tasks.json file from template” in the little pop-up window, and after that select the “Others” option. This tasks.json file will live inside the .vscode directory.

For my overcomplicated d20 roller, which is my first website built with Golang, I have the following content for the tasks:

.vscode/tasks.json

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "Build and run",
            "type": "shell",
            "command": "go build && ./d20",
            "group": {
                "kind": "build",
                "isDefault": true
            }
        }
    ]
}

What this one task does is that it runs go build to build the project and then runs the generated executable, which for this project is d20.

I guess providing a standardized name to go build with the -o flag this could be made more portable so that the command part reads something like go build -o proj && ./proj, but I’m ok with this for now.

And now just type Ctrl+Shift+b and Code will execute this “Build and run” task for us! Hooray! The terminal window in Code will pop-up saying something like:

> Executing task: go build && ./d20 <

By going to http://localhost:8080 I can see that my little website is up and running. Cool.

But if we want to restart this taks, running Ctrl+Shift+b again won’t work and Code will complain that the “Task is already active blablabla…”.

Looking at the Tasks menu, we can see that there’s a “Restart running task…” menu entry. Clicking that, it pops up a window with a list of running tasks. In this case there’s only one, our “Build and run” task. Clicking through the menu every time would be boring, so let’s add a keyboard shortcut for it.

Go to File -> Preferences -> Keyboard shortcuts (or just hit Ctrl+k Ctrl+s), search for “Restart running task” keybinding, and set it to whatever you like. I’ve set it to Ctrl+Alt+r.

Finally, the flow is Ctrl+Shift+b to start the taks for the very first time, code-code-code, Ctrl+Alt+r to re-build. Sweet. Now the only annoying bit is that I have to pick out that one running task from the list of running tasks when restarting, but I can live with that. For now.

Happy hackin’!

Static web pages in Hugo

published on January 24, 2018.

Last week I created a page on this site that holds all the talks I have prepared for meetups and conferences. As this site is powered by Hugo, the process wasn’t that straightforward. I want to write down the steps I did to make it easier in the future.

Oh, and when I say “static” in the title of this post, I mean pages whose content is not completely powered by a markdown content file.

I have tried different approaches, but what ended up working is the following.

In the configuration file, I added a new type of permalink:

config.toml

[permalinks]
    talks = "/talks/"

I created a new type of an archetype under the archetypes directory of my theme:

themes/robertbasic.com/archetypes/talks.md

+++
draft = false
date = {{ .Date }}
title = "{{ replace .TranslationBaseName "-" " " | title }}"
+++

I have also created a new template file for that talks type, which actually has all the content I want to display, but is also capable of using the partials I have created before:

themes/robertbasic.com/layouts/static/list.html

{{ partial "header.html" . }}
...
&lt;div class="post">
    &lt;h1>
        Talks
    &lt;/h1>
    ...
&lt;/div>
&lt;div class="column">
    {{ partial "sidebar.html" . }}
&lt;/div>
...
{{ partial "footer.html" . }}

And finally create a markdown file for it with hugo new talks/page.md, leaving it as is.

Happy hackin’!

Tags: hugo, blog, talks.
Categories: Blablabla, Software.

Mockery return values based on arguments

published on December 12, 2017.

Sometimes when working with Mockery mock objects, we want to tell a mocked method to return different values for different arguments. It is a rare occasion when I need this feature, but every time I need it, I’m happy it’s there.

The feature that allows us to return different values based on arguments is the andReturnUsing Mockery method, which takes a closure as an argument:

example.php

$dependencyMock = \Mockery::mock('SomeDependency');
$dependencyMock->shouldReceive('callDependency')
    ->andReturnUsing(function ($argument) {
        if ($argument <= 10) {
            return 'low';
        }

        return 'high';
    });

$dependencyMock->callDependency(10); // 'low'
$dependencyMock->callDependency(11); // 'high'

Any number of times we call our callDependency method on our mock object with a number 10 or less, it will return 'low', otherwise it will return 'high'.

Not much of an example, so let’s take a look at one a bit closer to a real world scenario.

Say we’re using Doctrine’s entity manager to get repositories for our entities in a service class:

src/ArticleService.php

<?php

class ArticleService
{
    public function __construct(EntityManager $em)
    {
        $this->articleRepo = $em->getRepository(Entity\Article::class);
        $this->authorRepo = $em->getRepository(Entity\Author::class);
    }
}

Not the best of the codes, but we’ll manage. The entity manager receives two calls to the getRepository method, once for the Article entity, once for the Author entity.

In a test case we could then set up the mocks like so:

tests/ArticleServiceTest.php

<?php

class ArticleServiceTest extends MockeryTestCase
{
    public function setup()
    {
        $this->authorRepositoryMock = \Mockery::mock(AuthorRepository::class);
        $this->articleRepositoryMock = \Mockery::mock(ArticleRepository::class);
        $this->entityManagerMock = \Mockery::mock(EntityManager::class);
    }

    public function testArticleService()
    {
        $repositoryMap = [
            'Entity\Author' => $this->authorRepositoryMock,
            'Entity\Article' => $this->articleRepositoryMock,
        ];
        $this->entityManagerMock->shouldReceive('getRepository')
            ->andReturnUsing(function($argument) use ($repositoryMap) {
                return $repositoryMap[$argument];
            });

        $articleService = new ArticleService($this->entityManagerMock);
    }
}

In the setup method we create the three mock objects that we need and then in the test method we create a $repositoryMap to help us map entities to repositories. The repository map could have been created in the andReturnUsing closure as well.

Now when we instantiate the ArticleService with the mocked entity manager, that mocked entity manager will receive two calls to the getRepository method in the ArticleServices constructor, and it will use the closure defined in andReturnUsing to return the correct repository mock objects.

More than one way to do it

Of course there is another way to achieve the same thing and that’s by using andReturn for the return value expectations, but it’s a bit more to write:

tests/ArticleServiceTest.php

<?php
    public function testArticleService()
    {
        $this->entityManagerMock->shouldReceive('getRepository')
            ->with('Entity\Author')
            ->andReturn($this->authorRepositoryMock);
        $this->entityManagerMock->shouldReceive('getRepository')
            ->with('Entity\Article')
            ->andReturn($this->articleRepositoryMock);

        $articleService = new ArticleService($this->entityManagerMock);
    }

It does the same thing as the previous thing. We might even argue that this second example is even clearer than the first example, sure, for a relatively small argument “map”. But if we need to handle a case with more than just two possible arguments, andReturnUsing can help us in those cases.

Happy hackin’!

P.S.: The proper way to do this actually would be to refactor that ArticleService to not get the two repositories from the entity manager, but to inject them directly instead.

CLI command to whitelist Composer packages

published on December 04, 2017.

James asked this question the other day on Twitter:

#LazyWeb is there a way to do a composer update of everything except a specific package or two? like `composer update --exclude doctrine/orm --exclude doctrine/dbal` or something? I don't want to have to whitelist everything all the time (there's bigger problems ofc)

Given that Composer has no --exclude flag or similar, the only other option is to create a list of packages we allow to be updated, excluding the ones we don’t want to be updated. We need to create a whitelist.

Creating it manually would be a PITA though, especially if there’s a lot of packages to include or exclude.

CLI to the rescue!

composer info | grep -v ^doctrine | sed 's/  \+/:/g' | cut -d: -f1 | paste -sd\ 

Note: There’s a single whitespace after the last backslash \.

This would result in a list of packages in a single line, something like:

beberlei/assert composer/ca-bundle container-interop/container-interop guzzlehttp/guzzle mockery/mockery

Let’s break it down

The composer info command shows information about the installed packages. The output is in the format of:

vendor1/package1      vx.y.z      Package 1 description
vendor1/package2      vx.y.z      Package 2 description
vendor2/package       vx.y.z      Package description

It’s all text so we can work with that.

The next step is to remove the packages we don’t want to be in our whitelist. We do that with grep -v ^package1 — search for and output anything that does not start with package1.

We are only interested in the vendor/package parts of the composer info output as that’s all we’ll need eventually for the composer update command.

When we have text that is formatted in columns, we can use the cut command to split these columns by a delimiter. There is a delimiter in the above output from composer info, but the delimiter is a varying number of whitespaces. That’s not really helpful.

What can we do now? Using sed we can replace those whitespaces to something that’s easier to use as a delimiter in cut, a colon : for example. sed 's/ \+/:/g' searches for two or more consecutive spaces and replaces them with a single : (not really visible, but the / \+/ part has two space characters between / and \).

The output at this point would look something like this:

vendor1/package1:vx.y.z:Package 1 description
vendor1/package2:vx.y.z:Package 2 description
vendor2/package:vx.y.z:Package description

Now we can use the cut command, tell it to use the colon as a delimiter with -d: and to take only the first field with -f1.

Finally, we use the paste command to merge lines together to get the final output. The s option is to merge horizontally and the d\ tells it to join using a single space character (again, it’s not really visible, but there is a single space character after the \ character).

Feel free to convert this one liner to a shell script that takes the package names as arguments so it’s a bit more reusable for future uses :)

Happy hackin’!

Tags: cli, shell, bash, composer.
Categories: Development, Software.
Robert Basic

Robert Basic

Software engineer, consultant, open source contributor.

Let's work together!

If you require outsourcing or consulting help on your projects, I'm available!

Robert Basic © 2008 — 2019
Get the feed