Archive for the 'Development' category

Five days of Advent of Gode

published on December 06, 2017.

A week or so ago, Luka mentioned this Advent of Code thing. I’ve been doing coding challenges and examples before, but never have I tried the AoC (this is the third year it’s running).

Advent of Code is a series of programming puzzles, where you get 2 puzzles a day for 25 days.

Given that I have started to learn golang again, I figured might as well learn more about it by joining this years challenge.

I keep a repository with my solutions to these puzzles and try to take notes for every day and now I want to look back at the first 5 days.

I started of the first day with just a mess of a code, just pushing for the first solution that gets the correct answer. On the second day I realised I won’t be learning much like this, so I decided to bring some order to the chaos: organised the code a bit nicer (even though it’s a far cry from good), added tests, and generally tried to get to a point where it’s easier to get started with a day’s challenge.

The AoC puzzles are relatively easy so far. Day 3 was the only day so far where I had problems wrapping my head around the problem. I’ve managed to figure out the solution for the first puzzle, but for the second puzzle I “cheated” and used the OEIS. shrug

As for golang… I don’t know enough of it to say if I like it or not. I mean, I do like it, sort of, but for these 5 days the most I did was toying around with strings and integers and slices and maps. That’s hardly enough time and usage to pass judgment on it.

On golang

I like how it fails to build if there’s an unused variable laying around, but then again I tend to save often, so I write:

for k, v := range slc {
}

Hit save and the IDE underlines that entire for line. What, why?! Hover over the line to see what the problem is… Oh, k and v are unused. Well, d-uh, I’m not done yet. But I still have to double check because the bug might be real, for example if I want to range over an integer the IDE will again underline the entire line.

For some reason I’m really bad at naming things in golang, quite often the variable and function names are just bad. I’m trying my best to follow the golang styles, but… I don’t know. I’m not sold on the whole “short rather than long” thing.

I keep mixing assignment = and assignment & declaration :=. But I’ll learn it, eventually.

I like everything about the types, even though I don’t understand everything about them, yet. Maps, for example, are not ordered even though I tried to use them as such once.

On slices

Except slices. Slices are weird. Well, were weird until I understood how golang treats and works with them.

I read four different articles to get to the point where I know what a slice is, only to get a succinct explanation of “slices are mutable views of an array”. For me, that one sentence explains it better than the four articles.

Here’s an example:

package main

import (
	"fmt"
)

func main() {
	x := []int{1, 2, 3, 4, 5}
	a := make([]int, 0)
	b := make([]int, len(x))

	y := x
	copy(a, x)
	copy(b, x)

	F(x)

	fmt.Println(x) // [2 3 4 5 6]
	fmt.Println(y) // [2 3 4 5 6]
	fmt.Println(a) // []
	fmt.Println(b) // [1 2 3 4 5]
}

func F(z []int) {
	for k, _ := range z {
		z[k]++
	}
}

Notice how both x and y are the same, even though we only modify x within the F() function, without even returning anything from that function. What happens is that the array on which the slice is built gets modified, which in turns modifies the slice(s) as well. Another thing worth remembering is to only append to a slice with the append() function.

On testing

I learned how to write tests. I guess there’s more to learn about them, but so far I’m doing OK. I’m writing a lot of repetitive things like this to get data providers:

for _, tt := range footests {
    r := Foo(tt.in)

    if r != tt.out {
        t.Errorf("Got %d for %s, expected %d", r, tt.in, tt.out)
    }
}

Even if there’s no shorter/better way to handle data providers in golang tests, I’ll just create some snippets and be done with all the copy/pasting.

I like that VS Code allows to run and debug a single test function. That’s really helpful. It shows small “run test” and “debug test” links above every test method. Click and off we go!

On packages

I haven’t yet learned how to properly work with my own packages, how to organize code and name packages to import one into the other, but I didn’t really had the need for it.

What I did learn, is that I can’t name a function like an existing package. For example, I wrote my own little function called sort, which made it impossible to import the sort package from the standard library.

All in all, I’m making good progress with both the AoC puzzles and with golang. I believe this will give me a good foundation for further learning and improving. There’s still a long road before me, but I feel I’m on the right track.

Happy hackin’!

CLI command to whitelist Composer packages

published on December 04, 2017.

James asked this question the other day on Twitter:

#LazyWeb is there a way to do a composer update of everything except a specific package or two? like `composer update --exclude doctrine/orm --exclude doctrine/dbal` or something? I don't want to have to whitelist everything all the time (there's bigger problems ofc)

Given that Composer has no --exclude flag or similar, the only other option is to create a list of packages we allow to be updated, excluding the ones we don’t want to be updated. We need to create a whitelist.

Creating it manually would be a PITA though, especially if there’s a lot of packages to include or exclude.

CLI to the rescue!

composer info | grep -v ^doctrine | sed 's/  \+/:/g' | cut -d: -f1 | paste -sd\ 

Note: There’s a single whitespace after the last backslash \.

This would result in a list of packages in a single line, something like:

beberlei/assert composer/ca-bundle container-interop/container-interop guzzlehttp/guzzle mockery/mockery

Let’s break it down

The composer info command shows information about the installed packages. The output is in the format of:

vendor1/package1      vx.y.z      Package 1 description
vendor1/package2      vx.y.z      Package 2 description
vendor2/package       vx.y.z      Package description

It’s all text so we can work with that.

The next step is to remove the packages we don’t want to be in our whitelist. We do that with grep -v ^package1 — search for and output anything that does not start with package1.

We are only interested in the vendor/package parts of the composer info output as that’s all we’ll need eventually for the composer update command.

When we have text that is formatted in columns, we can use the cut command to split these columns by a delimiter. There is a delimiter in the above output from composer info, but the delimiter is a varying number of whitespaces. That’s not really helpful.

What can we do now? Using sed we can replace those whitespaces to something that’s easier to use as a delimiter in cut, a colon : for example. sed 's/ \+/:/g' searches for two or more consecutive spaces and replaces them with a single : (not really visible, but the / \+/ part has two space characters between / and \).

The output at this point would look something like this:

vendor1/package1:vx.y.z:Package 1 description
vendor1/package2:vx.y.z:Package 2 description
vendor2/package:vx.y.z:Package description

Now we can use the cut command, tell it to use the colon as a delimiter with -d: and to take only the first field with -f1.

Finally, we use the paste command to merge lines together to get the final output. The s option is to merge horizontally and the d\ tells it to join using a single space character (again, it’s not really visible, but there is a single space character after the \ character).

Feel free to convert this one liner to a shell script that takes the package names as arguments so it’s a bit more reusable for future uses :)

Happy hackin’!

Tags: cli, shell, bash, composer.
Categories: Development, Software.

Reacting to promises

published on November 29, 2017.

I was working on something that includes the usage of ReactPHP promises. Given that I haven’t had the chance to take a closer look at it yet, I decided that this is the right time for it.

ReactPHP has several different components, with the end goal of providing a low-level library for event-driven programming in PHP. The one component I want to talk about today is the promise component, which is a Promises/A implementation for PHP.

What this promise library allows us is a nicer workflow with asynchronous code.

With promises, when we want to execute something asynchronously we defer the work that will be executed asynchronously. The Deferred unit of work will complete sometimes in the future, but we don’t know when. But it does promise that the work will be done, one way or the other.

The Promise is a sort of a placeholder for the result that will eventually be returned from our deferred work. This promise can then either be resolved or rejected by our deferred. When a promise is resolved successfully it has an associated value, and when it is rejected it has an associated reason for the rejection.

We use the then method on the promise to register handlers that will be called when the deferred is resolved or rejected.

To install the React/Promise component, run:

$ composer require react/promise

An example

Let’s say we have some code that does some asynchronous work. Checking the HTTP status code of a bunch of URLs, for example. We could create an invokable class that extends the Deferred:

FetchStatusCodes.php

<?php declare(strict_types=1);

use React\Promise\Deferred;

class FetchStatusCodes extends Deferred
{
    public function __invoke(array $urls)
    {
        $multiHandle = curl_multi_init();

        $handles = $this->getHandlesForUrls($urls, $multiHandle);

        $this->executeMultiHandle($multiHandle);

        $statusCodes = $this->getStatusCodes($handles);

        curl_multi_close($multiHandle);

        $successRate = $this->calculateSuccessRate($statusCodes);

        if ($successRate > 50) {
            $this->resolve($statusCodes);
        } else {
            $this->reject('Success rate too low: ' . $successRate);
        }
    }
}

I’ve left out here a bunch of code that deals with the actual fetching of the status codes, just to keep the “noise” down. The full example is available in this repository.

The important thing here is that we extend React\Promise\Deferred and that at the end we call the resolve() method to resolve this deferred if the success rate is over 50%, or that we call the reject() method if the success rate is below 50%.

The set up of the actual promise and its handlers would look something like this:

promise.php

<?php

$statusCodes = new FetchStatusCodes();
$promise = $statusCodes->promise();

$promise
    ->then(
        function($value) {
            var_dump($value);
        },
        function($reason) {
            echo $reason . PHP_EOL;
        }
    );

$urls = [
    'https://example.com/',
    'https://stackoverflow.com/',
    'https://www.google.com/',
    'https://www.google.com/no-such-url',
    'https://www.google.com:81'
];
$statusCodes($urls);

We create the FetchStatusCodes deferred object and get the promise. We setup the resolve and reject handler callbacks in the then method. They don’t do much for now:

  • the resolve handler dumps the value it got,
  • the reject handler prints out the reason of the rejection.

The output for a resolved promise would be something like this:

$ php promise.php
/home/robert/projects/react-promise-example/promise.php:32:
array(5) {
  'https://example.com/' => int(200)
  'https://stackoverflow.com/' => int(200)
  'https://www.google.rs/' => int(200)
  'https://www.google.com/no-such-url' => int(404)
  'https://www.google.com:81/' => int(0)
}

We’re not done yet!

The example above where we call the then method to set up our resolve/reject handlers, isn’t quite correct. Why?

When we call the then method it actually returns a new Promise. This feature of the Promises/A specification allows us to chain promises together.

On this second promise we can again set up our resolve/reject handlers calling the then method on it, same as we do for our first promise. The resolve handler of the second promise will be called with the return value of either the resolve or the reject handler of the first promise. The reject handler of the second promise will be called when either the resolve or the reject handler of the first promise throws an exception. And the then method of our second promise again returns a new, third promise.

Let’s see if an example makes it a bit more clearer:

promise.php

<?php

$statusCodes = new FetchStatusCodes();
$firstPromise = $statusCodes->promise();

$secondPromise = $firstPromise->then(
    function($statusCodes) {
        $successCodes = array_filter($statusCodes, function ($code) {
            if ($code >= 200 && $code < 300) {
                return true;
            }
            return false;
        });
        return $successCodes;
    },
    function($reason) {
        // handle rejected promise
        // gets called when Deferred gets reject-ed
    }
);

$thirdPromise = $secondPromise->then(
    function ($successCodes) {
        return json_encode($successCodes);
    },
    function ($reason) {
        // handle rejected promise
        // gets called when $firstPromise handlers throw an exception
    }
);

$urls = [
    'https://example.com/',
    'https://stackoverflow.com/',
    'https://www.google.com/',
    'https://www.google.com/no-such-url',
    'https://www.google.com:81'
];
$statusCodes($urls);

When our FetchStatusCodes deferred resolves, it will call the resolve handler of the $firstPromise. In that first resolve handler we get only the successful status codes and return them.

With this return from the resolve handler of the first promise, we “trigger” the resolve handler of the $secondPromise where we can, for example, json_encode our success codes. By returning this JSON string from the resolve handler of the second promise, we again “trigger” the resolve handler of the $thirdPromise, and so on.

Almost done!

When we call then, we make a new promise.

To actually be done with all the promises, we need to call the done method on the last promise in our chain. With done we stop making promises and use the result of our last promise:

promise.php

<?php
$thirdPromise->done(
    function ($jsonString) {
        echo $jsonString . PHP_EOL;
    },
    function ($reason) {
        // handle rejected promise
        // gets called when $secondPromise handlers throw an exception
    }
);

If we’d run the example now, we’d get something like this:

$ php promise.php | json_pp
{
   "https://example.com/" : 200,
   "https://www.google.rs/" : 200,
   "https://stackoverflow.com/" : 200
}

We additionally pipe the output of our example script to json_pp to pretty print the JSON string.

Now we’re done

ReactPHP promises have an ExtendedPromisesInterface that include additional shortcut and utility methods that are not part of the Promise/A specification. Their docs include some more examples, and Cees-Jan Kiewiet looks at examples using the react/dns component, among other things.

When we deal with asynchronous code in PHP, using ReactPHP promises gives us a way to deal with it in a much nicer, saner way.

Happy hackin’!

PHP FPM slow log

published on November 23, 2017.

The other day I was going through the configuration file for php-fpm, when I noticed a configuration directive I haven’t before: slowlog. I guess it’s been around for a while, I just never noticed it.

The php-fpm slow log is a pool configuration, meaning that we configure it in www.conf, and has two directives for it:

  • the slowlog, which is a path to a file where the slow requests will be logged,
  • and request_slowlog_timeout is a time unit after which PHP will dump a backtrace for that request in to the slow log file. We can configure it to be in seconds, minutes, hours, or days.

What’s in the box backtrace?

It has the date and time for when the slow request happened, the pool and PID for the php-fpm process. script_filename is the entry point to the request, and the backtrace includes a list of function calls up until the moment when the request_slowlog_timeout was hit.

[23-Nov-2017 15:28:21]  [pool www] pid 8992
script_filename = /var/www/example/web/app_dev.php
[0x00007efe32a14a40] sleep() /var/www/example/src/AppBundle/Controller/DefaultController.php:18
[0x00007efe32a149d0] indexAction() /var/www/example/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpKernel.php:153
[0x00007efe32a14960] call_user_func_array() /var/www/example/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpKernel.php:153
[0x00007efe32a14470] handleRaw() /var/www/example/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpKernel.php:68
[0x00007efe32a14320] handle() /var/www/example/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php:169
[0x00007efe32a14250] handle() /var/www/example/web/app_dev.php:29

Even though it doesn’t reveal too much, together with other profiling tools, like Xdebug and kcachegrind, it can help us a great deal on finding and fixing performance problems in web applications.

Happy hackin’!

Visual sudo for shell scripts

published on November 06, 2017.

The other day I was putting together a small shell script to do some administrative tasks on my Fedora workstation.

Even though I spend most of my time in a terminal, I wanted to have this script available from “everywhere”, that is to have it available to run it as a keyboard shortcut.

The script requires sudo privileges, and up until now, I thought that the only way to get sudo was from the terminal.

But then I remembered that some applications, like firewall-config, ask for the sudo password via a pop-up window. Surely it’s available to whatever application needs it, right?

The answer is pkexec.

To quote the man pages:

pkexec allows an authorized user to execute program as another user. If program is not specified, the default shell will be run. If username is not specified, then the program will be executed as the administrative super user, root.

Looks like that this pkexec is a part of, or at least relates to, something called Polkit. I honestly don’t understand that part yet, and what does it really do. Need to learn more about it, but as this is the first time I came across it, the learning more about it thing might not happen soon.

To make the shell script ask for the sudo password through pkexec we add it to the she-bang line:

#!/usr/bin/pkexec /bin/bash

touch /some/path/requiring/permissions.txt

Now when we run this script either through the terminal, or through an application finder/launcher applet, or by invoking it with a keyboard shortcut, it’ll ask for the sudo password first with the pop-up window.

Happy hackin’!

Tags: sudo, shell, pkexec, gksudo, fedora.
Categories: Development, Software.
Robert Basic

Robert Basic

Software engineer, consultant, open source contributor.

Let's work together!

If you require outsourcing or consulting help on your projects, I'm available!

Robert Basic © 2008 — 2019
Get the feed