Tag QA

Introducing pylint-django 2.0

Today I have released pylint-django version 2.0 on PyPI. The changes are centered around compatibility with the latest pylint 2.0 and astroid 2.0 versions. I've also bumped pylint-django's version number to reflact that.

A major component, class transformations, was updated so don't be surprised if there are bugs. All the existing test cases pass but you never know what sort of edge case there could be.

I'm also hosting a workshop/corporate training about writing pylint plugins. If you are interested see this page!

Thanks for reading and happy testing!

There are comments.

Upstream rebuilds with Jenkins Job Builder

I have been working on Weldr for some time now. It is a multi-component software with several layers built on top of each other as seen on the image below.

Weldr components

One of the risks that we face is introducing changes in downstream components which are going to break something up the stack! In this post I am going to show you how I have configured Jenkins to trigger dependent rebuilds and report all of the statuses back to the original GitHub PR. All of the code below is Jenkins Job Builder yaml.

bdcs is the first layer of our software stack. It provides command line utilities. codec-rpm is a library component that facilitates working with RPM packages (in Haskell). bdcs links to codec-rpm when it is compiled, bdcs uses some functions and data types from codec-rpm.

When a pull request is opened against codec-rpm and testing completes successfully I want to reuse that particular version of the codec-rpm library and rebuild/test bdcs with that.

YAML configuration

All jobs have the following structure: -trigger -> -provision -> -runtest -> -teardown. This means that Jenkins will start executing a new job when it gets triggered by an event in GitHub (commit to master branch or new pull request), then it will provision a slave VM in OpenStack, execute the test suite on the slave and destroy all of the resources at the end. This is repeated twice: for master branch and for pull requests! Here's how the -runtest jobs look:

- job-template:
    name: '{name}-provision'
    node: master
      - string:
          name: PROVIDER
        - git:
            url: 'https://github.com/weldr/{repo_name}.git'
            refspec: ${{git_refspec}}
              - ${{git_branch}}
      - github-notifier
      - shell: |
            #!/bin/bash -ex
            # do the openstack provisioning here
        # NB: runtest_job is passed to us via the -trigger job
      - trigger-builds:
          - project: '${{runtest_job}}'
            block: true
            current-parameters: true
            condition: 'SUCCESS'
            fail-on-missing: true

- job-template:
    name: '{name}-master-runtest'
    node: cinch-slave
    project-type: freestyle
    description: 'Build master branch of {name}!'
        - git:
            url: 'https://github.com/weldr/{repo_name}.git'
                - master
      - github-notifier
      - conditional-step:
          condition-kind: regex-match
          regex: "^.+$"
          label: '${{UPSTREAM_BUILD}}'
          on-evaluation-failure: dont-run
            - copyartifact:
                project: ${{UPSTREAM_BUILD}}
                which-build: specific-build
                build-number: ${{UPSTREAM_BUILD_NUMBER}}
                filter: ${{UPSTREAM_ARTIFACT}}
                flatten: true
      - shell: |
            #!/bin/bash -ex
            make ci
      - trigger-parameterized-builds:
          - project: '{name}-teardown'
            current-parameters: true
      - github-notifier

- job-template:
    name: '{name}-PR-runtest'
    node: cinch-slave
    description: 'Build PRs for {name}!'
        - git:
            url: 'https://github.com/weldr/{repo_name}.git'
            refspec: +refs/pull/*:refs/remotes/origin/pr/*
                # builds the commit hash instead of a branch
                - ${{ghprbActualCommit}}
      - github-notifier
      - shell: |
            #!/bin/bash -ex
            make ci
      - conditional-step:
          condition-kind: current-status
          condition-worst: SUCCESS
          condition-best: SUCCESS
          on-evaluation-failure: dont-run
            - shell: |
                #!/bin/bash -ex
                make after_success
      - archive:
          artifacts: '{artifacts_path}'
          allow-empty: '{artifacts_empty}'
      - conditional-publisher:
          - condition-kind: '{execute_dependent_job}'
            on-evaluation-failure: dont-run
              - trigger-parameterized-builds:
                - project: '{dependent_job}'
                  current-parameters: true
                  predefined-parameters: |
                  condition: 'SUCCESS'
      - trigger-parameterized-builds:
          - project: '{name}-teardown'
            current-parameters: true
      - github-notifier

- job-group:
    name: '{name}-tests'
    - '{name}-provision'
    - '{name}-teardown'
    - '{name}-master-trigger'
    - '{name}-master-runtest'
    - '{name}-PR-trigger'
    - '{name}-PR-runtest'

- job:
    name: 'codec-rpm-rebuild-bdcs'
    node: master
    project-type: freestyle
    description: 'Rebuild bdcs after codec-rpm PR!'
        - git:
            url: 'https://github.com/weldr/codec-rpm.git'
            refspec: +refs/pull/*:refs/remotes/origin/pr/*
                # builds the commit hash instead of a branch
                - ${ghprbActualCommit}
      - github-notifier
      - trigger-builds:
          - project: 'bdcs-master-trigger'
            block: true
            predefined-parameters: |
      - github-notifier

- project:
    name: codec-rpm
    dependent_job: '{name}-rebuild-bdcs'
    execute_dependent_job: always
    artifacts_path: 'dist/{name}-latest.tar.gz'
    artifacts_empty: false
      - '{name}-tests'

Publishing artifacts

make after_success is responsible for creating a tarball if codec-rpm test suite passed. This tarball gets uploaded as artifact into Jenkins and we can make use of it later!

Inside -master-runtest I have a conditional-step inside the builders section which will copy the artifacts from the previous build if they are present. Notice that I copy artifacts for a particular job number, which is the job for codec-rpm PR.

Making use of local artifacts is handled inside bdcs' make ci because it is per-project specific and because I'd like to reuse my YAML templates.

Reporting statuses to GitHub

For github-notifier to be able to report statuses back to the pull request the job needs to be configured with the git repository this pull request came from. This is done by specifying the same scm section for all jobs that are related and current-parameters: true to pass the revision information to the other jobs.

This also means that if I want to report status from codec-rpm-rebuild-bdcs then it needs to be configured for the codec-rpm repository (see yaml) but somehow it should trigger jobs for another repository!

When jobs are started via trigger-parameterized-builds their statuses are reported separately to GitHub. When they are started via trigger-builds there should be only one status reported.

Trigger chain for dependency rebuilds

With all of the above info we can now look at the codec-rpm-rebuild-bdcs job.

  • It is configured for the codec-rpm repository so it will report its status to the PR
  • It is conditionally started after codec-rpm-PR-runtest finishes successfully
  • It triggers bdcs-master-trigger which in turn will rebuild & retest the bdcs component. Additional parameters specify whether we're going to use locally built artifacts or attempt to download then from Hackage
  • It uses block: true so that the status of codec-rpm-rebuild-bdcs is dependent on the status of bdcs-master-runtest (everything in the job chain uses block: true because of this)

How this looks like in practice

I have opened codec-rpm #39 to validate my configuration. The chain of jobs that gets executed in Jenkins is:

--- console.log for bdcs-master-runtest ---
Started by upstream project "bdcs-jslave-1-provision" build number 267
originally caused by:
 Started by upstream project "bdcs-master-trigger" build number 133
 originally caused by:
  Started by upstream project "codec-rpm-rebuild-bdcs" build number 25
  originally caused by:
   Started by upstream project "codec-rpm-PR-runtest" build number 77
   originally caused by:
    Started by upstream project "codec-rpm-jslave-1-provision" build number 178
    originally caused by:
     Started by upstream project "codec-rpm-PR-trigger" build number 118
     originally caused by:
      GitHub pull request #39 of commit b00c923065e367afd5b7a7cc068b049bb1ed25e1, no merge conflicts.

Statuses are reported on GitHub as follows:

example of PR statuses

default is coming from the provisioning step and I think this is some sort of a bug or misconfiguration of the provisioning job. We don't really care about this.

On the picture you can see that codec-rpm-PR-runtest was successful but codec-rpm-rebuild-bdcs was not. The actual error when compiling bdcs is:

src/BDCS/Import/RPM.hs:110:24: error:
    * Couldn't match type `Entry' with `C8.ByteString'
      Expected type: conduit-
                       ([T.Text], [Maybe ObjectDigest])
        Actual type: conduit-
                       ([T.Text], [Maybe ObjectDigest])
    * In the second argument of `(.|)', namely
           ((,) <$> ZipConduit filenames <*> ZipConduit digests)'
      In the second argument of `($)', namely
               ((,) <$> ZipConduit filenames <*> ZipConduit digests)'
      In the second argument of `($)', namely
           $ src
                   ((,) <$> ZipConduit filenames <*> ZipConduit digests)'
110 |                     .| getZipConduit ((,) <$> ZipConduit filenames
    |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...

That is because PR #39 changes the return type of Codec.RPM.Conduit::payloadContentsC from Entry to C8.ByteString.

Thanks for reading and happy testing!

social image CC by https://pxhere.com/en/photo/226978

There are comments.

Introducing pylint-django 0.8.0

Since my previous post was about writing pylint plugins I figured I'd let you know that I've released pylint-django version 0.8.0 over the weekend. This release merges all pull requests which were pending till now so make sure to read the change log.

Starting with this release Colin Howe and myself are the new maintainers of this package. My immediate goal is to triage all of the open issue and figure out if they still reproduce. If yes try to come up with fixes for them or at least get the conversation going again.

My next goal is to integrate pylint-django with Kiwi TCMS and start resolving all the 4000+ errors and warnings that it produces.

You are welcome to contribute of course. I'm also interested in hosting a workshop on the topic of pylint plugins.

Thanks for reading and happy testing!

There are comments.

How to write pylint checker plugins

In this post I will walk you through the process of learning how to write additional checkers for pylint!


  1. Read Contributing to pylint to get basic knowledge of how to execute the test suite and how it is structured. Basically call tox -e py36. Verify that all tests PASS locally!

  2. Read pylint's How To Guides, in particular the section about writing a new checker. A plugin is usually a Python module that registers a new checker.

  3. Most of pylint checkers are AST based, meaning they operate on the abstract syntax tree of the source code. You will have to familiarize yourself with the AST node reference for the astroid and ast modules. Pylint uses Astroid for parsing and augmenting the AST.

    NOTE: there is compact and excellent documentation provided by the Green Tree Snakes project. I would recommend the Meet the Nodes chapter.

    Astroid also provides exhaustive documentation and node API reference.

    WARNING: sometimes Astroid node class names don't match the ones from ast!

  4. Your interactive shell weapons are ast.dump(), ast.parse(), astroid.parse() and astroid.extract_node(). I use them inside an interactive Python shell to figure out how a piece of source code is parsed and converted back to AST nodes! You can also try this ast node pretty printer! I personally haven't used it.

How pylint processes the AST tree

Every checker class may include special methods with names visit_xxx(self, node) and leave_xxx(self, node) where xxx is the lowercase name of the node class (as defined by astroid). These methods are executed automatically when the parser iterates over nodes of the respective type.

All of the magic happens inside such methods. They are responsible for collecting information about the context of specific statements or patterns that you wish to detect. The hard part is figuring out how to collect all the information you need because sometimes it can be spread across nodes of several different types (e.g. more complex code patterns).

There is a special decorator called @utils.check_messages. You have to list all message ids that your visit_ or leave_ method will generate!

How to select message codes and IDs

One of the most unclear things for me is message codes. pylint docs say

The message-id should be a 5-digit number, prefixed with a message category. There are multiple message categories, these being C, W, E, F, R, standing for Convention, Warning, Error, Fatal and Refactoring. The rest of the 5 digits should not conflict with existing checkers and they should be consistent across the checker. For instance, the first two digits should not be different across the checker.

I'm usually having troubles with the numbering part so you will have to get creative or look at existing checker codes.

Practical example

In Kiwi TCMS there's legacy code that looks like this:

def add_cases(run_ids, case_ids):
    trs = TestRun.objects.filter(run_id__in=pre_process_ids(run_ids))
    tcs = TestCase.objects.filter(case_id__in=pre_process_ids(case_ids))

    for tr in trs.iterator():
        for tc in tcs.iterator():


Notice the dangling return statement at the end! It is useless because when missing the default return value of this function will still be None. So I've decided to create a plugin for that.

Armed with the knowledge above I first try the ast parser in the console:

Python 3.6.3 (default, Oct  5 2017, 20:27:50) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> import astroid
>>> ast.dump(ast.parse('def func():\n    return'))
"Module(body=[FunctionDef(name='func', args=arguments(args=[], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[]), body=[Return(value=None)], decorator_list=[], returns=None)])"
>>> node = astroid.parse('def func():\n    return')
>>> node
<Module l.0 at 0x7f5b04621b38>
>>> node.body
[<FunctionDef.func l.1 at 0x7f5b046219e8>]
>>> node.body[0]
<FunctionDef.func l.1 at 0x7f5b046219e8>
>>> node.body[0].body
[<Return l.2 at 0x7f5b04621c18>]

As you can see there is a FunctionDef node representing the function and it has a body attribute which is a list of all statements inside the function. The last element is .body[-1] and it is of type Return! The Return node also has an attribute called .value which is the return value! The complete code will look like this:

import astroid

from pylint import checkers
from pylint import interfaces
from pylint.checkers import utils

class UselessReturnChecker(checkers.BaseChecker):
    __implements__ = interfaces.IAstroidChecker

    name = 'useless-return'

    msgs = {
        'R2119': ("Useless return at end of function or method",
                  'Emitted when a bare return statement is found at the end of '
                  'function or method definition'

    def visit_functiondef(self, node):
            Checks for presence of return statement at the end of a function
            "return" or "return None" are useless because None is the default
            return type if they are missing
        # if the function has empty body then return
        if not node.body:

        last = node.body[-1]
        if isinstance(last, astroid.Return):
            # e.g. "return"
            if last.value is None:
                self.add_message('useless-return', node=node)
            # e.g. "return None"
            elif isinstance(last.value, astroid.Const) and (last.value.value is None):
                self.add_message('useless-return', node=node)

def register(linter):
    """required method to auto register this checker"""

Here's how to execute the new plugin:

$ PYTHONPATH=./myplugins pylint --load-plugins=uselessreturn tcms/xmlrpc/api/testrun.py | grep useless-return
W: 40, 0: Useless return at end of function or method (useless-return)
W:117, 0: Useless return at end of function or method (useless-return)
W:242, 0: Useless return at end of function or method (useless-return)
W:495, 0: Useless return at end of function or method (useless-return)


  • If you contribute this code upstream and pylint releases it you will get a traceback:

    pylint.exceptions.InvalidMessageError: Message symbol 'useless-return' is already defined

    this means your checker has been released in the latest version and you can drop the custom plugin!

  • This is example is fairly simple because the AST tree provides the information we need in a very handy way. Take a look at some of my other checkers to get a feeling of what a more complex checker looks like!

  • Write and run tests for your new checkers, especially if contributing upstream. Have in mind that the new checker will be executed against existing code and in combination with other checkers which could lead to some interesting results. I will leave the testing to yourself, all is written in the documentation.

This particular example I've contributed as PR #1821 which happened to contradict an existing checker. The update, raising warnings only when there's a single return statement in the function body, is PR #1823.

Workshop around the corner

I will be working together with HackSoft on an in-house workshop/training for writing pylint plugins. I'm also looking at reviving pylint-django so we can write more plugins specifically for Django based projects.

If you are interested in workshop and training on the topic let me know!

Thanks for reading and happy testing!

There are comments.

The ARCS model of motivational design


The ARCS model is an instructional design method developed by John Keller that focuses on motivation. ARCS is based on a research into best practices and successful teachers and gives you tactics on how to evaluate your lessons in order to build motivation right into them.

I have conducted and oversaw quite a few trainings and I have not been impressed with the success rate of those so this topic is very dear to me. Success for me measures in the ability to complete the training and learn the basis of a technical topic. And then gather the initial momentum to continue developing your skills within the chosen field. This is what I've been doing for myself and this is what I'd like to see my students do.

In his paper (I have a year 2000 printed copy from Cuba) Keller argues that motivation is a product of four factors: Attention, Relevance, Confidence and Satisfaction. You need all of them incorporated in your lessons and learning materials for them to be motivational. I could argue that you need the same characteristics at work in order to motivate people to do their job as you wish.

Once you start a lesson you need to grab the audience Attention so they can listen to you. Then the topic needs to be relevant to the audience so they will continue listening to the end. This makes for a good start but is not enough. Confidence means for the audience to feel confident they can perform all the necessary tasks on their own, that they have what it takes to learn (and you have to build that). If they think they can't make it from the start then it is a lost battle. And Satisfaction means the person feels that achievements are due to their own abilities and hard work not due to external factors (work not demanding enough, luck, etc).

If all of the above 4 factors are true then the audience should feel personally motivated to learn because they can clearly understand the benefit for themselves and they realize that everything depends on them.

ARCS gives you a model to evaluate your target audience and lesson properties and figure out tactics by which to address any shortcomings in the above 4 areas.

Last Friday I hosted 2 training sessions: a Python and Selenium workshop at HackConf and then a lecture about test case management and demo of Kiwi TCMS before students at Pragmatic IT academy. For both of them I used the simplified ARCS evaluation matrix.

In this matrix the columns map to the ARCS areas while the rows map to different parts of the lesson: audience, presentation media, exercise, etc. Here's how I used them (I've mostly analyzed the audience).

Python & Selenium workshop

  • Attention
    • (+) this is an elective workshop
    • (+) the topic is clear and the curricula is on GitHub
    • (+) the title is catchy (Learn Python & Selenium in 6 hours)
    • (+) I am well known in the industry
  • Relevance
    • (+) Basic Python practical skills, being able to write small programs, knowing the basic building blocks
    • (+) Basic Selenium skills: finding and using elements
    • (+) Basic Python test automation skills: writing simple tests and asserts
  • Confidence
    • (+) each task has tests which need to report PASS at the end
    • (-) need to use PyCharm IDE, unfamiliar with IDEs
    • (-) not enough experience with programming or Linux
    • (-) not enough experience with (automation) testing
    • (-) all materials and exercises are in English
  • Satisfaction
    • (-) not being able to create a simple program

From the above it was clear that I didn't need to spend much time on building attention or relevance. The topic itself and the fact that these are skill which can be immediately applied at work gave the workshop a huge boost. During the opening part of my workshop I've stated "this training takes around 2 months, I've seen some of you forking my GitHub repo so I know you are prepared. Let's see how much you can do in 6 hours" which sets the challenge and was my attention building moment. Then I reiterated that all skills are directly applicable in daily work confirming the relevance part.

I did need a confidence building strategy though. So having all the tests ready meant evaluation was quick and easy. Anton (my assistant) and I promised to help with the IDE and all other questions to counter the other items on the list. During the course of the workshop I did quick code review of all participants that managed to complete their tasks within the hour giving them quick tips on how to perform or highlighting pieces of code/approaches that were different from mine or that I found elegant or interesting. This was my confidence building strategy. Code review and verbal praising also touches on the satisfaction area, i.e. the participant gets the feeling they are doing well.

My Satisfaction building strategy was kind of mixed. Before I read about ARCS I wanted to give penalty points to participants who didn't complete on time and then send them home after 3 fails. At the end I only said I will do this but didn't do it.

Instead I used the challenge statement from the attention phase and turned that into a competition. The first 3 participants to complete their module tasks on time were rewarded chocolates. With the agreement of the entire group the grand prize was set to be a small box of the same chocolates and this would be awarded to the person with the most chocolates (e.g. the one who's been in top 3 the most times).

I don't know if ARCS had anything to do with it but this workshop was the most successful training I've ever done. 40% of the participants managed to get at least one chocolate and at least 50% have completed all of their tasks within the hour. Normally a passing rate on such training is around 10 to 20 %.

During the workshop we had 5 different modules which consisted of 10-15 minutes explanation of Python basics (e.g. loops or if conditions), quick Q&A session and around 30 minutes for working alone and code review. I don't think I was following ARCS for each of the separate modules because I didn't have time to analyze them individually. I gambled all my money on the introductory 10 minutes!

TCMS lecture

My second lecture for the day was about test case management. The audience was students who are aspiring to become software testers and attending the Software Testing training at Pragmatic. In my lecture (around 1 hour) I wanted to explain what test management is, why it is important and also demo the tool I'm working on - Kiwi TCMS. The analysis looks like:

  • Attention
    • (+) the entire training was elective but
    • (-) that particular lecture was mandatory. Students were not able to select what they are going to study
  • Relevance
    • (-) it may not be clear what TCMS is and why we need it
    • (+) however students may sense that this is something work related since the entire training is
  • Confidence
    • (-) unknown UI, generally unfamiliar workflow
    • (-) not enough knowledge how to write a Test Plan document or test cases
  • Satisfaction
    • (-) how to make sure new skills can be applied in practice

So I was in a medium need of a strategy to build attention. My opening was by introducing myself to establish my professional level and introducing Kiwi TCMS by saying it is the best open source test case management system to which I'm one of the core maintainers.

Then I had a medium need of a relevance building strategy. I did this by explaining what test management is and why it is important. I've talked briefly about QA managers trying to indirectly inspire the audience to aim for this position. I finished this part by telling the students how a TCMS system helps the ordinary guy in their daily work - namely by giving you a dashboard where you can monitor all the work you need to do, check your progress, etc.

I was in a strong need to build confidence. I did a 20-30 minutes demonstration where I was writing a Test Plan and test cases and then pretending to execute them and marking bugs and test results in the system. I told the students "you are my boss for today, tell me what I need to test". So they instructed me to test the login functionality of the system and we agreed on 5 different test cases. I described all of these into Kiwi TCMS and began executing them. During execution I opened another browser window and did exactly what the test case steps were asking for. There were some bugs so I promptly marked them as such and I promised I will fix them.

To build satisfaction I was planning on having the students write one test plan and some test cases but we didn't have time for this. Their instructor promised they will be doing more exercises and using Kiwi TCMS in the next 2 months but this remains to be seen. I've wrapped my lecture by giving advise to use Kiwi TCMS as a portfolio building tool. Since these students are newcomers to the QA industry their next priority will be looking for a job. I've advised them to document their test plans and test cases into Kiwi TCMS and then present these artifacts to future employers. I've also told them they are more than welcome to test and report bugs against Kiwi TCMS on GitHub and add these bugs to their portfolio!

This is how I've applied ARCS for the first time. I like it and will continue to use it for my trainings and workshops. I will try harder to make the application process more iterative and apply the method not only to my opening speech but for all submodules as well!

One thing that bothers me is can I apply the ARCS principles when doing a technical presentation and how do they play together or clash with storytelling, communication style and rhetoric (all topics I'm exploring for my public speaking). If you do have more experience with these please share it in the comments below.

Thanks for reading and happy testing!

There are comments.

Storytelling for test professionals

This is a very condensed brief of an 8 hour workshop I visited earlier this year held by Huib Schoots. You can find the slides here.

Storytelling is the form in which people naturally communicate. Understanding the building blocks of a story will help us understand other people's motivations, serve as map for actions and emotions, help uncover unknown perspectives and serve as source for inspiration.

Stories stand on their own and have a beginning, middle and an end. There is a main character and a storyline with development. Stories are authentic and personal and often provocative and evoke emotions.

7 basic story plots

  1. Overcoming the Monster
  2. Rags to Riches
  3. The Quest
  4. Voyage and return
  5. Comedy
  6. Tragedy
  7. Rebirth

From these we can derive the following types of stories.

6 types of stories

  1. Who am I (identity stories)
  2. Why am I here (motive and mission stories)
  3. Vision stories (the big picture)
  4. Future scenarios (imagining the future)
  5. Product stories (branding)
  6. Culture stories (a sum of other stories)

12 Common Archetypes

Each story needs a hero and there are 12 common archetypes of heroes. More importantly you can also find these archetypes within your team and organization. Read the link above to find out what their motto, core desire, goals, fears and motives are. The 12 types are

  1. Innocent
  2. Everyman
  3. Hero
  4. Caregiver
  5. Explorer
  6. Rebel
  7. Lover
  8. Creator
  9. Jester
  10. Sage
  11. Magician
  12. Ruler

6 key elements of a story

  1. Who's the hero?
  2. What is their desire?
  3. What is stopping them?
  4. What is the turning point?
  5. What are their insights?
  6. What is the solution?

Dramatic structure and Freytag's pyramid

"Freytag's pyramid"

One of the most commonly used storytelling structures is the Freytag's Pyramid. According to it each story has an exposition, rising action, climax, falling action and resolution. I think this can be applied directly when preparing presentations even technical ones.

The Hero's journey

Successful stories follow the 12 steps of the hero's journey

  1. Ordinary world
  2. Call to adventure
  3. Refusal of the Call
  4. Meeting the mentor
  5. Crossing the threshold (after which the hero enters the Special world)
  6. Tests, allies and enemies
  7. Approach
  8. Ordeal, death & rebirth
  9. Rewards, seizing the sword
  10. The road back (to the ordinary world)
  11. Resurrection
  12. Return with elixir

As part of the workshop we worked in groups and created a completely made up story. Every person in the group was contributing couple of sentences from their own experiences, trying to describe the particular step in the hero's journey. At the end we told a story from the point of view of a single hero which was a complete mash-up of moments that had nothing to do with each other. Still it sounded very realistic and plausible.

Storytelling techniques

SUCCESS means Simple, Unexpected, Concrete, Credible, Emotional, Stories. To use this technique find the core of your idea, grab people's attention by surprising them and make sure the idea can be understood and remembered later. Find a way to make people believe in the idea so they can test it for themselves, make them feel something to understand why this idea is important. Tell stories and empower people to use an idea through narrative.

STAR means Something They will Always Remember. A STAR Moment should be Simple, Transferable, Audience-centered, Repeatable, and Meaningful. There are 5 types of STAR moments: memorable dramatization, repeatable sound bites, evocative visuals, emotive storytelling, shocking statistics.

To enhance our stories and presentations we should appeal to senses (smell, sounds, sight, touch, taste) and make it visual.

I will be using some of these techniques combined with others in my future presentations and workshops. I'd love to be able to summarize all of them into a short guide targeted at IT professionals but I don't know if this is even possible.

Anyway if you do try some of these techniques in your public speaking please let me know how it goes. I want to hear what works for you and your audience and what doesn't.

Thanks for reading and happy testing!

There are comments.

More tests for login forms

"Telenor's login form"

By now I probably have documented more test cases for login forms than anyone else. You can check out my previous posts on the topic here and here. I give you a few more examples.

Test 01 and 02: First of all let's start by saying that a "Remember me" checkbox should actually remember the user and login them automatically on the next visit if checked. The other way around if not checked. I don't think this has been mentioned previously!

Test 03: When there is a "Remember me" checkbox it should be selectable both with the mouse and the keyboard. On my.telenor.bg the checkbox changes its image only when clicked with the mouse. Also clicking the login button with Space doesn't work!

Interestingly enough when I don't select "Remember me" at all and close then revisit the page I am still able to access the internal pages of my account! At this point I'm not quite sure what this checkbox does!

Test 04: Testing two factor authentication. I had the case where GitHub SMS didn't arrive for over 24 hrs and I wasn't able to login. After requesting a new code you can see the UI updating but I didn't receive another message. In this particular case I received only one message with an already invalid code. So test for:

  • how long does it take for the codes to expire
  • is there a visual feedback indicating how many codes have been requested
  • do latest code invalidates all the previous ones or all that have been unused still work
  • what happens if I'm already logged in and somebody tries to access my account requesting additional codes which may or may not invalidate my login session?

Test 05: Check that confirmation codes, links, etc will actually expire after their configured time. Kiwi TCMS had this problem which has been fixed in version 3.32.

Test 06: Is this a social-network-login only site? Then which of my profiles did I use? Check that there is a working social auth provider reminder.

Test 07: Check that there is an error message visible (e.g. wrong login credentials). After the redesign Kiwi TCMS had stopped displaying this message and instead presents the user with the login form again!

Also checkout these testing challenges by Claudiu Draghia where you can see many cases related to input field validation! For example empty field, value too long, special characters in field, etc. All of these can lead to issues depending on how login is implemented.

Thanks for reading and happy testing!

There are comments.

Xiaomi's selfie bug

Recently I've been exploring the user interface of a Xiaomi Redmi Note 4X phone and noticed a peculiar bug, adding to my collection of obscure phone bugs. Sometimes when taking selfies the images will not be saved in the correct orientation. Instead they will be saved as if looking in the mirror and this is a bug!

"Samsung S5 front screen"

While taking the selfie the display correctly acts as a mirror, see my personal Samsung S5 (black) and the Xiaomi device (white).

"Xiaomi front screen"

However when the image is saved and then viewed through the gallery application there is a difference. The image below is taken with the Xiaomi device and there have been no effects added to it except scaling and cropping. As you can see the letters on the cereal box are mirrored!

"Xiaomi mirrored image"

The symptoms of the bug are not quite clear as of yet. I've managed to reproduce at around 50% rate so far. I've tried taking pictures during the day in direct sunlight and in the shade, also in the evening under bad artificial lighting. Taking photo of a child's face and then child plus varying number of adults. Then photo of only 1 or more adults, heck I even made a picture of myself. I though that lighting or the number of faces and their age have something to do with this bug but so far I'm not getting consistent results. Sometimes the images turn out OK and other times they don't regardless of what I take a picture of.

I also took a picture of the same cereal box, under the same conditions as above but not capturing the child's face and the image came out not mirrored. The only clue that seems to hold true so far is that you need to have people's faces in the picture for this bug to reproduce but that isn't an edge case when taking selfies, right?

I've also compared the results with my Samsung S5 (Android version 6.0.1) and BlackBerry Z10 devices and both work as expected: while taking the picture the display acts as a mirror but when viewing the saved image it appears in normal orientation. On S5 there is also a clearly visible "Processing" progress bar while the picture is being saved!

For reference the system information is below:

Model number: Redmi Note 4X
Android version: 6.0 MRA58K
Android security patch level: 2017-03-01
Kernel version: 3.18.22+

I'd love if somebody from Xiaomi's engineering department looks into this and sends me a root cause analysis of the problem.

Thanks for reading and happy testing! Oh and btw this is my breakfast, not hers!

There are comments.

Speeding up Rust builds inside Docker

Currently it is not possible to instruct cargo, the Rust package manager, to build only the dependencies of the software you are compiling! This means you can't easily pre-install build dependencies. Luckily you can workaround this with cargo build -p! I've been using this Python script to parse Cargo.toml:

#!/usr/bin/env python

from __future__ import print_function

import os
import toml

_pwd = os.path.dirname(os.path.abspath(__file__))
cargo = toml.loads(open(os.path.join(_pwd, 'Cargo.toml'), 'r').read())

for section in ['dependencies', 'dev-dependencies']:
    for dep, version in cargo[section].items():
        print('cargo build -p %s' % dep)

and then inside my Dockerfile:

RUN mkdir /bdcs-api-rs/
COPY parse-cargo-toml.py /bdcs-api-rs/

# Manually install cargo dependencies before building
# so we can have a reusable intermediate container.
# This workaround is needed until cargo can do this by itself:
# https://github.com/rust-lang/cargo/issues/2644
# https://github.com/rust-lang/cargo/pull/3567
COPY Cargo.toml /bdcs-api-rs/
WORKDIR /bdcs-api-rs/
RUN python ./parse-cargo-toml.py | while read cmd; do \
        $cmd;                                    \

It doesn't take into account the version constraints specified in Cargo.toml but is still able to produce an intermediate docker layer which I can use to speed-up my tests by caching the dependency compilation part.

As seen in the build log, lines 1173-1182, when doing cargo build it downloads and compiles chrono v0.3.0 and toml v0.3.2. The rest of the dependencies are already available. The logs also show that after Job #285 the build times dropped from 16 minutes down to 3-4 minutes due to Docker caching. This would be even less if the cache is kept locally!

Thanks for reading and happy testing!

There are comments.

Code coverage from Nightmare.js tests

In this article I'm going to walk you through the steps required to collect code coverage when running an end-to-end test suite against a React.js application.

The application under test looks like this

<!doctype html>
<html lang="en-us" class="layout-pf layout-pf-fixed">
    <!-- js dependencies skipped -->
    <div id="main"></div>
    <script src="./dist/main.js?0ca4cedf3884d3943762"></script>

It is served as an index.html file and a main.js file which intercepts all interactions from the user and sends requests to the backend API when needed.

There is an existing unit-test suite which loads the individual components and tests them in isolation. Apparently people do this!

There is also an end-to-end test suite which does the majority of the testing. It fires up a browser instance and interacts with the application. Everything runs inside Docker containers providing a full-blown production-like environment. They look like this

test('should switch to Edit Recipe page - recipe creation success', (done) => {
  const nightmare = new Nightmare();
    .wait(page => document.querySelector(page.dialogRootElement).style.display === 'block'
      , createRecipePage)
    .insert(createRecipePage.inputName, createRecipePage.varRecName)
    .insert(createRecipePage.inputDescription, createRecipePage.varRecDesc)
    .end() // remove this!
    .then((element) => {
      // here goes coverage collection helper
      done(); // remove this!
}, timeout);

The browser interaction is handled by Nightmare.js (sort of like Selenium) and the test runner is Jest.

Code instrumentation

The first thing we need is to instrument the application code to provide coverage statistics. This is done via babel-plugin-istanbul. Because unit-tests are executed a bit differently we want to enable conditional instrumentation. In reality for unit tests we use jest --coverage which enables istanbul on the fly and having the code already instrumented breaks this. So I have the following in webpack.config.js

if (process.argv.includes('--with-coverage')) {

and then build my application with node run build --with-coverage.

You can execute node run start --with-coverage, open the JavaScript console in your browser and inspect the window.__coverage__ variable. If this is defined then the application is instrumented correctly.

Fetching coverage information from within the tests

Remember that main.js from the beginning of this post? It lives inside index.html which means everything gets downloaded to the client side and executed there. When running the end-to-end test suite that is the browser instance which is controlled via Nightmare. You have to pass window.__coverage__ from the browser scope back to nodejs scope via nightmare.evaluate()! I opted to directly save the coverage data on the file system and make it available to coverage reporting tools later!

My coverage collecting snippet looks like this

  .evaluate(() => window.__coverage__) // this executes in browser scope
  .end() // terminate the Electron (browser) process
  .then((cov) => {
    // this executes in Node scope
    // handle the data passed back to us from browser scope
    const strCoverage = JSON.stringify(cov);
    const hash = require('crypto').createHmac('sha256', '')
    const fileName = `/tmp/coverage-${hash}.json`;
    require('fs').writeFileSync(fileName, strCoverage);

    done(); // the callback from the test
.catch(err => console.log(err));

Nightmare returns window.__coverage__ from browser scope back to nodejs scope and we save it under /tmp using a hash value of the coverage data as the file name.

Side note: I do have about 40% less coverage files than number of test cases. This means some test scenarios exercise the same code paths. Storing the individual coverage reports under a hashed file name makes this very easy to see!

Note that in my coverage handling code I also call .end() which will terminate the browser instance and also execute the done() callback which is being passed as parameter to the test above! This is important because it means we had to update the way tests were written. In particular the Nightmare method sequence doesn't have to call .end() and done() except in the coverage handling code. The coverage helper must be the last code executed inside the body of the last .then() method. This is usually after all assertions (expectations) have been met!

Now this coverage helper needs to be part of every single test case so I wanted it to be a one line function, easy to copy&paste! All my attempts to move this code inside a module have been futile. I can get the module loaded but it kept failing with Unhandled promise rejection (rejection id: 1): cov_23rlop1885 is not defined;`

At the end I've resorted to this simple hack


Shout-out to Krasimir Tsonev who joined me on a two days pairing session to figure this stuff out. Too bad we couldn't quite figure it out. If you do please send me a pull request!

Reporting the results

All of these coverage-*.json files are directly consumable by nyc - the coverage reporting tool that comes with the Istanbul suite! I mounted .nyc_output/ directly under /tmp inside my Docker container so I could

nyc report
nyc report --reporter=lcov | codecov

We can also modify the unit-test command to jest --coverage --coverageReporters json --coverageDirectory .nyc_output so it produces a coverage-final.json file for nyc. Use this if you want to combine the coverage reports from both test suites.

Because I'm using Travis CI the two test suites are executed independently and there is no easy way to share information between them. Instead I've switched from Coveralls to CodeCov which is smart enough to merge coverage submissions coming from multiple jobs on the same git commits. You can compare the commit submitting only unit-test results with the one submitting coverage from both test suites.

All of the above steps are put into practice in PR #136 if you want to check them out!

Thanks for reading and happy testing!

There are comments.

Faster Travis CI tests with Docker cache

For a while now I've been running tests on Travis CI using Docker containers to build the project and execute the tests inside. In this post I will explain how to speed up execution times.

A Docker image is a filesystem snapshot similar to a virtual machine image. From these images we build containers (e.g. we run the container X from the image Y). The construction of Docker images is controlled via Dockerfile which contains a set of instructions how to build the image. For example:

FROM welder/web-nodejs:latest
MAINTAINER Brian C. Lane <bcl@redhat.com>
RUN dnf install -y nginx

CMD nginx -g "daemon off;"

## Do the things more likely to change below here. ##

COPY ./docker/nginx.conf /etc/nginx/

# Update node dependencies only if they have changed
COPY ./package.json /welder/package.json
RUN cd /welder/ && npm install

# Copy the rest of the UI files over and compile them
COPY . /welder/
RUN cd /welder/ && node run build

COPY entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

docker build is smart enough to actually build intermediate layers for each command and store them on your computer. Each command is hashed and it is rebuilt only if it has been changed. Thus the stuff which doesn't change often goes first (like setting up a web server or a DB) and the stuff that changes (like the project source code) goes at the end. All of this is beautifully explained by Stefan Kanev in this video (in Bulgarian).

Travis and Docker

While intermediate layer caching is a standard feature for Docker it is disabled by default in Travis CI and any other CI service I was able to find. To be frank Circles CI offer this as a premium feature but their pricing plans on that aren't clear at all.

However you can enable the use of caching following a few simple steps:

  1. Make your Docker images publicly available (e.g. Docker Hub or Amazon EC2 Container Service)
  2. Before starting the test job do a docker pull my/image:latest
  3. When building your Docker images in Travis add --cache-from my/image:latest to docker build
  4. After successful execution docker tag the latest image with the build job number and docker push it again to the hub!


  • Everything you do will become public so take care not to expose internal code. Alternatively you may configure a private docker registry (e.g. Amazon EC2 CS) and use encrypted passwords for Travis to access your images;
  • docker pull will download all layers that it needs. If your hosting is slow this will negatively impact execution times;
  • docker push will upload only the layers that have been changed;
  • I only push images coming from the master branch which are not from a pull request build job. This prevents me from accidentally messing something up.

If you examine the logs of Job #247.4 and Job #254.4 you will notice that almost all intermediate layers were re-used from cache:

Step 3/12 : RUN dnf install -y nginx
 ---> Using cache
 ---> 25311f052381
Step 4/12 : CMD nginx -g "daemon off;"
 ---> Using cache
 ---> 858606811c85
Step 5/12 : EXPOSE 3000
 ---> Using cache
 ---> d778cbbe0758
Step 6/12 : COPY ./docker/nginx.conf /etc/nginx/
 ---> Using cache
 ---> 56bfa3fa4741
Step 7/12 : COPY ./package.json /welder/package.json
 ---> Using cache
 ---> 929f20da0fc1
Step 8/12 : RUN cd /welder/ && npm install
 ---> Using cache
 ---> 68a30a4aa5c6

Here the slowest operations are dnf install and npm install which on normal execution will take around 5 minutes.

You can check-out my .travis.yml for more info.

First time cache

It is important to note that you need to have your docker images available in the registry before you execute the first docker pull from CI. I do this by manually building the images on my computer and uploading them before configuring CI integration. Afterwards the CI system takes care of updating the images for me.

Initially you may not notice a significant improvement as seen in Job #262, Step 18/22. The initial image available on Docker Hub has all the build dependencies installed and the code has not been changed when job #262 was executed.

The COPY command copies the entire contents of the directory, including filesystem metadata! Things like uid/gid (file ownership), timestamps (not sure if taken into account) and/or extended attributes (e.g. SELinux) will cause the intermediate layers checksums to differ even though the actual source code didn't change. This will resolve itself once your CI system starts automatically pushing the latest images to the registry.

Thanks for reading and happy testing!

There are comments.

TransactionManagementError during testing with Django 1.10

During the past 3 weeks I've been debugging a weird error which started happening after I migrated KiwiTestPad to Django 1.10.7. Here is the reason why this happened.


After migrating to Django 1.10 all tests appeared to be working locally on SQLite however they failed on MySQL with

TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

The exact same test cases failed on PostgreSQL with:

InterfaceError: connection already closed

Since version 1.10 Django executes all tests inside transactions so my first thoughts were related to the auto-commit mode. However upon closer inspection we can see that the line which triggers the failure is


which is essentially a SELECT query aka User.objects.filter(username=username).exists()!

My tests were failing on a SELECT query!

Reading the numerous posts about TransactionManagementError I discovered it may be caused by a run-away cursor. The application did use raw SQL statements which I've converted promptly to ORM queries, that took me some time. Then I also fixed a couple of places where it used transaction.atomic() as well. No luck!

Then, after numerous experiments and tons of logging inside Django's own code I was able to figure out when the failure occurred and what events were in place. The test code looked like this:

response = self.client.get('/confirm/')

user = User.objects.get(username=self.new_user.username)

The failure was happening after the view had been rendered upon the first time I do a SELECT against the database!

The problem was that the connection to the database had been closed midway during the transaction!

In particular (after more debugging of course) the sequence of events was:

  1. execute django/test/client.py::Client::get()
  2. execute django/test/client.py::ClientHandler::__call__(), which takes care to disconnect/connect signals.request_started and signals.request_finished which are responsible for tearing down the DB connection, so problem not here
  3. execute django/core/handlers/base.py::BaseHandler::get_response()
  4. execute django/core/handlers/base.py::BaseHandler::_get_response() which goes through the middleware (needless to say I did inspect all of it as well since there have been some changes in Django 1.10)
  5. execute response = wrapped_callback() while still inside BaseHandler._get_response()
  6. execute django/http/response.py::HttpResponseBase::close() which looks like

    # These methods partially implement the file-like object interface.
    # See https://docs.python.org/3/library/io.html#io.IOBase
    # The WSGI server must call this method upon completion of the request.
    # See http://blog.dscpl.com.au/2012/10/obligations-for-calling-close-on.html
    def close(self):
        for closable in self._closable_objects:
            except Exception:
        self.closed = True
  7. signals.request_finished is fired

  8. django/db/__init__.py::close_old_connections() closes the connection!

IMPORTANT: On MySQL setting AUTO_COMMIT=False and CONN_MAX_AGE=None helps workaround this problem but is not the solution for me because it didn't help on PostgreSQL.

Going back to HttpResponseBase::close() I started wondering who calls this method. The answer was it was getting called by the @content.setter method at django/http/response.py::HttpResponse::content() which is even more weird because we assign to self.content inside HttpResponse::__init__()

Root cause

The root cause of my problem was precisely this HttpResponse::__init__() method or rather the way we arrive at it inside the application.

The offending view last line was

return HttpResponse(Prompt.render(
     next=request.GET.get('next', reverse('core-views-index'))

and the Prompt class looks like this

from django.shortcuts import render

class Prompt(object):
    def render(cls, request, info_type=None, info=None, next=None):
        return render(request, 'prompt.html', {
            'type': info_type,
            'info': info,
            'next': next

Looking back at the internals of HttpResponse we see that

  • if content is a string we call self.make_bytes()
  • if the content is an iterator then we assign it and if the object has a close method then it is executed.

HttpResponse itself is an iterator, inherits from six.Iterator so when we initialize HttpResponse with another HttpResponse object (aka the content) we execute content.close() which unfortunately happens to close the database connection as well.

IMPORTANT: note that from the point of view of a person using the application the HTML content is exactly the same regardless of whether we have nested HttpResponse objects or not. Also during normal execution the code doesn't run inside a transaction so we never notice the problem in production.

The fix of course is very simple, just return Prompt.render()!

Thanks for reading and happy testing!

There are comments.

Producing coverage report for Haskell binaries

Recently I've started testing a Haskell application and a question I find unanswered (or at least very poorly documented) is how to produce coverage reports for binaries ?

Understanding HPC & cabal

hpc is the Haskell code coverage tool. It produces the following files:

  • .mix - module index file, contains information about tick boxes - their type and location in the source code;
  • .tix - tick index file aka coverage report;
  • .pix - program index file, used only by hpc trans.

The invocation to hpc report needs to know where to find the .mix files in order to be able to translate the coverage information back to source and it needs to know the location (full path or relative from pwd) to the tix file we want to report.

cabal is the package management tool for Haskell. Among other thing it can be used to build your code, execute the test suite and produce the coverage report for you. cabal build will produce module information in dist/hpc/vanilla/mix and cabal test will store coverage information in dist/hpc/vanilla/tix!

A particular thing about Haskell is that you can only test code which can be imported, e.g. it is a library module. You can't test (via Hspec or Hunit) code which lives inside a file that produces a binary (e.g. Main.hs). However you can still execute these binaries (e.g. invoke them from the shell) and they will produce a coverage report in the current directory (e.g. main.tix).

Putting everything together

  1. Using cabal build and cabal test build the project and execute your unit tests. This will create the necessary .mix files (including ones for binaries) and .tix files coming from unit testing;
  2. Invoke your binaries passing appropriate data and examining the results (e.g. compare the output to a known value). A simple shell or Python script could do the job;
  3. Copy the binary.tix file under dist/hpc/vanilla/binary/binary.tix!

Produce coverage report with hpc:

hpc markup --hpcdir=dist/hpc/vanilla/mix/lib --hpcdir=dist/hpc/vanilla/mix/binary  dist/hpc/vanilla/tix/binary/binary.tix

Convert the coverage report to JSON and send it to Coveralls.io:

cabal install hpc-coveralls
~/.cabal/bin/hpc-coveralls --display-report tests binary


Check out the haskell-rpm repository for an example. See job #45 where there is now coverage for the inspect.hs, unrpm.hs and rpm2json.hs files, producing binary executables. Also notice that in RPM/Parse.hs the function parseRPMC is now covered, while it was not covered in the previous job #42!

.travis.yml snippet
  - ~/.cabal/bin/hlint .
  - cabal install --dependencies-only --enable-tests
  - cabal configure --enable-tests --enable-coverage --ghc-option=-DTEST
  - cabal build
  - cabal test --show-details=always

  # tests to produce coverage for binaries
  - wget https://s3.amazonaws.com/atodorov/rpms/macbook/el7/x86_64/efivar-0.14-1.el7.x86_64.rpm
  - ./tests/test_binaries.sh ./efivar-0.14-1.el7.x86_64.rpm

  # move .tix files in appropriate directories
  - mkdir ./dist/hpc/vanilla/tix/inspect/ ./dist/hpc/vanilla/tix/unrpm/ ./dist/hpc/vanilla/tix/rpm2json/
  - mv inspect.tix ./dist/hpc/vanilla/tix/inspect/
  - mv rpm2json.tix ./dist/hpc/vanilla/tix/rpm2json/
  - mv unrpm.tix ./dist/hpc/vanilla/tix/unrpm/

  - cabal install hpc-coveralls
  - ~/.cabal/bin/hpc-coveralls --display-report tests inspect rpm2json unrpm

Thanks for reading and happy testing!

There are comments.

What's the bug in this pseudo-code

Rails Girls Vratsa sticker

This is one of the stickers for the second edition of Rails Girls Vratsa which was held yesterday. Let's explore some of the bug proposals submitted by the Bulgarian QA group:

  1. sad() == true is ugly
  2. sad() is not very nice, better make it if(isSad())
  3. use sadStop(), and even better - stopSad()
  4. there is an extra space character in beAwesome( )
  5. the last curly bracket needs to be on a new line

Lyudmil Latinov

My friend Lu describes what I would call style issues. The style he refers to is mostly Java oriented, especially with naming things. In Ruby we would probably go with sad? instead of isSad. Style is important and there are many tools to help us with that this will not cause a functional problem! While I'm at it let me say the curly brackets are not the problem either. They are not valid in Ruby this is a pseudo-code and they also fall in the style category.

The next interesting proposal comes from Tsveta Krasteva. She examines the possibility of sad() returning an object or nil instead of boolean value. Her first question was will the if statement still work, and the answer is yes. In Ruby everything is an object and every object can be compared to true and false. See Alan Skorkin's blog post on the subject.

Then Tsveta says the answer is to use sad().stop() with the warning that it may return nil. In this context the sad() method returns on object indicating that the person is feeling sad. If the method returns nil then the person is feeling OK.

example by Tsveta
class Csad
  def stop()

def sad()

def beAwesome()

# notice == true was removed
  print("Yes, I am sad\n");
  beAwesome( );

While this is coming closer to a functioning solution something about it is bugging me. In the if statement the developer has typed more characters than required (== true). This sounds to me unlikely but is possible with less experienced developers. The other issue is that we are using an object (of class Csad) to represent an internal state in the system under test. There is one method to return the state (sad()) and another one to alter the state (Csad.stop()). The two methods don't operate on the same object! Not a very strong OOP design. On top of that we have to call the method twice, first time in the if statement, the second time in the body of the if statement, which may have unwanted side effects. It is best to assign the return value to some variable instead.

IMO if we are to use this OOP approach the code should look something like:

class Person
  def sad?()

  def stopBeingSad()

  def beAwesome()

p = Person.new
if p.sad?

Let me return back to assuming we don't use classes here. The first obvious mistake is the space in sad stop(); first spotted by Peter Sabev*. His proposal, backed by others is to use sad.stop(). However they didn't use my hint asking what is the return value of sad() ?

If sad() returns boolean then we'll get undefined method 'stop' for true:TrueClass (NoMethodError)! Same thing if sad() returns nil, although we skip the if block in this case.

In Ruby we are allowed to skip parentheses when calling a method, like I've shown above. If we ignore this fact for a second, then sad?.stop() will mean execute the method named stop() which is a member of the sad? variable, which is of type method! Again, methods don't have an attribute named stop!

The last two paragraphs are the semantic/functional mistake I see in this code. The only way for it to work is to use an OOP variant which is further away from what the existing clues give us.

Note: The variant sad? stop() is syntactically correct. This means call the function sad? with parameter the result of calling the method stop(), which depending on the outer scope of this program may or may not be correct (e.g. stop is defined, sad? accepts optional parameters, sad? maintains global state).

Thanks for reading and happy testing!

There are comments.

VMware's favorite login form

How do you test a login form? is one of my favorite questions when screening candidates for QA positions and also a good brain exercise even for experienced testers. I've written about it last year. In this blog post I'd like to share a different perspective on this same question, this time courtesy of my friend Rayna Stankova.

Login form

What bugs do you see above

The series of images above is from a Women Who Code Sofia workshop where the participants were given printed copies and asked to find as much defects as possible. Here they are (counting clock-wise from the top-left corner):

  1. Typo in "Registr" link at the bottom;
  2. UI components are not aligned;
  3. Missing "Forgot your password?" link
  4. Backend credentials validation with empty password; plain text password field; Too specific information about incorrect credentials;
  5. Too specific information about incorrect credentials with visual hint as to what exactly is not correct. In this case it looks like the password is OK, maybe it was one of the 4 most commonly used passwords, but the username is wrong which we can easily figure out;
  6. In this case the error handling appears to be correct, not disclosing what exactly is wrong. The placement is somewhat wrong, it looks like an error message for one of the fields instead for the entire form. I'd move that to the top and even slightly update the wording to be more like Login failed, bad credentials, try again.

How do you test this

Here is a list of possible test scenarios, proposed by Rayna. Notes are mine.

UI Layer

  • Test 1: Verify Email (User ID) field has focus on page load
  • Test 2: Verify Empty Email (User ID) field and Password field
  • Test 3: Verify Empty Email (User ID) field
  • Test 4: Verify Empty Password field
  • Test 5: Verify Correct sign in
  • Test 6: Verify Incorrect sign in
  • Test 7: Verify Password Reset - working link
  • Test 8: Verify Password Reset - invalid emails
  • Test 9: Verify Password Reset - valid email
  • Test 10: Verify Password Reset - using new password
  • Test 11: Verify Password Reset - using old password
  • Test 12: Verify whether password text is hidden
  • Test 13: Verify text field limits - whether the browser accepts more than the allowed database limits
  • Test 14: Verify that validation message is displayed in case user exceeds the character limit of the username and password fields
  • Test 15: Verify if there is checkbox with label "remember password" in the login page
  • Test 16: Verify if it’s allowed the username to contain non printable characters? If not, this is invalid on the 'create user' section.
  • Test 17: Verify if the user must be logged in to access any other area of the site.

Tests 10 and 11 are particularly relevant for Fedora Account System where you need a really strong password and (at least in the past) had to change it more often and couldn't reuse any of your old passwords. As a user I really hate this b/c I can't remember my own password but it makes for a good test scenario.

13 and 14 are also something I rarely see and could make a nice case for property based testing.

16 would have been the bread and butter of testing Emoj.li (the first emoji-only social network).

Keyboard Specific

  • Test 18: Verify Navigate to all fields
  • Test 19: Verify Enter submits on password focus
  • Test 20: Verify Space submits on login focus
  • Test 21: Verify Enter submits

These are all so relevant with beautifully styled websites nowadays. The one I hate the most is when space key doesn't trigger select/unselect for checkboxes which are actually images!


  • Test 22: Verify SQL Injections testing - password field
  • Test 23: Verify SQL Injections testing - username field
  • Test 24: Verify SQL Injections testing - reset password
  • Test 25: Verify Password/username not visible from URL login
  • Test 26: Verify For security point of view, in case of incorrect credentials user is displayed the message like "incorrect username or password" instead of exact message pointing at the field that is incorrect. As message like "incorrect username" will aid hacker in brute-forcing the fields one by one
  • Test 27: Verify the timeout of the login session
  • Test 28: Verify if the password can be copy-pasted or not
  • Test 29: Verify that once logged in, clicking back button doesn't logout user

22, 23 and 24 are a bit generic and I guess can be collapsed into one. Better yet make them more specific instead.

Test 28 may sound like nonsense but is not. I remember back in the days that it was possible to copy and paste the password out of Windows dial-up credentials screen. With heavily styled form fields it is possible to have this problem again so it is a valid check IMO.


  • Test 30: Verify that the password is in encrypted form when entered
  • Test 31: Verify the user must be logged in to call any web services.
  • Test 32: Verify if the username is allowed to contain non printable characters, the code handling login can deal with them and no error is thrown.

I think Test 30 means to validate that the backend doesn't store passwords in plain text but rather stores their hashes.

32 is a duplicate of 16. I also say why only the username? Password field is also a good candidate for this.

If you check how I would test a login form you will find some similarities but there are also scenarios which are different. I'm interested to see what other scenarios we've both missed, especially ones which have manifested themselves as bugs in actual applications.

Thanks for reading and happy testing!

There are comments.

Monitoring behavior via automated tests

In my last several presentations I briefly talked about using your tests as a monitoring tool. I've not been eating my own dog food and stuff failed in production!

What is monitoring via testing

This is a technique I coined 6 months ago while working with Tradeo's team. I'm not the first one to figure this out so if you know the proper name for it please let me know in the comments. So why not take a subset of your automated tests and run them regularly against production? Let's say every hour?

In my particular case we started with integration tests which interact with the product (a web app) in a way that a living person would do. E.g. login, update their settings, follow another user, chat with another user, try to deposit money, etc. The results from these tests are logged into a database and then charted (using Grafana). This way we can bring lots of data points together and easily analyze them.

This technique has the added bonus that we can cover the most critical test paths in a couple of minutes and do so regularly without human intervention. Perusing the existing monitoring infrastructure of the devops team we can configure alerts if need be. This makes it sort of early detection/warning system plus it gives a degree of possibility to spot correlations between data points or patterns.

As simple as it sounds I've heard about a handfull of companies doing this sort of continuous testing against production. Maybe you can implement something similar in your organization and we can talk more about the results?

Why does it matter

Anyway, everyone knows how to write Selenium tests so I'm not going to bother you with the details. Why does this kind of testing matter?

Do you remember a recent announcement by GitHub about Travis CI leaking some authentication tokens into their public log files? I did receive an email about this but didn't pay attention to it because I don't use GitHub tokens for anything I do in Travis. However as a safety measure GitHub had went ahead and wiped out my security tokens.

The result from this is that my automated upstream testing infrastructure had stopped working! In particular my requests to the GitHub API stopped working. And I didn't even know about it!

This means that since May 24th there have been at least 4 new versions of libraries and frameworks on which some of my software depends and I failed to test them! One of them was Django 1.11.2.

I have supplied a new GitHub token for my infra but if I had monitoring I would have known about this problem well in advance. Next I'm off to write some monitoring tests and also implement better failure detection in Strazar itself!

Thanks for reading and happy testing (in production)!

There are comments.

Semantically Invalid Input

Unsolvable Square example
. . 9 | . 2 8 | 7 . .
8 . 6 | . . 4 | . . 5
. . 3 | . . . | . . 4
6 . . | . . . | . . .
? 2 . | 7 1 3 | 4 5 .
. . . | . . . | . . 2
3 . . | . . . | 5 . .
9 . . | 4 . . | 8 . 7
. . 1 | 2 5 . | 3 . .

In a comment to a previous post Flavio Poletti proposed a very interesting test case for a function which solves the Sudoku game - semantically invalid input, i.e. an input that passes intermediate validation checks (no duplicates in any row/col/9-square) but that cannot possibly have a solution.

Until then I thought that Sudoku was a completely deterministic game and if input followed all validation checks then we always have a solution. Apparently I was wrong! Reading more on the topic I discovered these Sudoku test cases from Sudopedia. Their Invalid Test Cases section lists several examples of semantically invalid input in Sudoku:

  • Unsolvable Square;
  • Unsolvable Box;
  • Unsolvable Column;
  • Unsolvable Row;
  • Not Unique with examples having 2, 3, 4, 10 and 125 solutions

The example above cannot be solved because the left-most square of the middle row (r5c1) has no possible candidates.

Following the rule non-repeating numbers from 1 to 9 in each row for row 5 we're left with numbers: 6, 8 and 9. For (r5c1) 6 is a no-go because it is already present in the same square. Then 9 is a no-go because it is present in column 1. Which leaves us with 8, which is also present in column 1! Pretty awesome, isn't it?

Also check the Valid Test Cases section which includes other interesting examples and definitely not ones which I have considered previously when testing Sudoku.

On a more practical note I have been trying to remember a case from my QA practice where we had input data that matched all conditions but is semantically invalid. I can't remember of such a case. If you do have examples about semantically invalid data in real software please let me know in the comments below!

Thanks for reading and happy testing!

There are comments.

Learn Python & Selenium Automation in 8 weeks

Couple of months ago I conducted a practical, instructor lead training in Python and Selenium automation for manual testers. You can find the materials at GitHub.

The training consists of several basic modules and practical homework assignments. The modules explain

  1. The basic structure of a Python program and functions
  2. Commonly used data types
  3. If statements and (for) loops
  4. Classes and objects
  5. The Python unit testing framework and its assertions
  6. High-level introduction to Selenium with Python
  7. High-level introduction to the Page Objects design pattern
  8. Writing automated tests for real world scenarios without any help from the instructor.

Every module is intended to be taken in the course of 1 week and begins with links to preparatory materials and lots of reading. Then I help the students understand the basics and explain with more examples, often writing code as we go along. At the end there is the homework assignment for which I expect a solution presented by the end of the week so I can comment and code-review it.

All assignments which require the student to implement functionality, not tests, are paired with a test suite, which the student should use to validate their solution.

What worked well

Despite everything I've written below I had 2 students (from a group of 8) which showed very good progress. One of them was the absolute star, taking active participation in every class and doing almost all homework assignments on time, pretty much without errors. I think she'd had some previous training or experience though. She was in the USA, training was done remotely via Google Hangouts.

The other student was in Sofia, training was done in person. He is not on the same level as the US student but is the best from the Bulgarian team. IMO he lacks a little bit of motivation. He "cheated" a bit on some tasks providing non-standard, easier solutions and made most of his assignments. After the first Selenium session he started creating small scripts to extract results from football sites or as helpers to be applied in the daily job. The interesting fact for me was that he created his programs as unittest.TestCase classes. I guess because this was the way he knew how to run them!?!

There were another few students which had had some prior experience with programming but weren't very active in class so I can't tell how their careers will progress. If they put some more effort into it I'm sure they can turn out to have decent programming skills.

What didn't work well

Starting from the beginning most students failed to read the preparatory materials. Some of the students did read a little bit, others didn't read at all. At the times when they came prepared I had the feeling the sessions progressed more smoothly. I also had students joining late in the process, which for the most part didn't participate at all in the training. I'd like to avoid that in the future if possible.

Sometimes students complained about lack of example code, although Dive into Python includes tons of examples. I've resorted to sending them the example.py files which I produced during class.

The practical part of the training was mostly myself programming on a big TV screen in front of everyone else. Several times someone from the students took my place. There wasn't much active participation on their part and unfortunately they didn't want to bring personal laptops to the training (or maybe weren't allowed)! We did have a company provided laptop though.

When practicing functions and arithmetic operations the students struggled with basic maths like breaking down a number into its digits or vice versa, working with Fibonacci sequences and the like. In some cases they cheated by converting to/from strings and then iterating over them. Also some hard-coded the first few numbers of the Fibonacci sequence and returned it directly. Maybe an in-place explanation of the underlying maths would have been helpful but honestly I was surprised by this. Somebody please explain or give me an advise here!

I am completely missing examples of the datetime and timedelta classes which tuned out to be very handy in the practical Selenium tasks and we had to go over them on the fly.

The OOP assignments went mostly undone, not to mention one of them had bonus tasks which are easily solved using recursion. I think we could skip some of the OOP practice (not sure how safe that is) because I really need classes only for constructing the tests and we don't do anything fancy there.

Page Object design pattern is also OOP based and I think that went somewhat well granted that we are only passing values around and performing some actions. I didn't put constraints nor provided guidance on what the classes should look like and which methods go where. Maybe I should have made it easier.

Anyway, given that Page Objects is being replaced by Screenplay pattern, I think we can safely stick to the all-in-one functional based Selenium tests. Maybe utilize helper functions for repeated tasks (like login). Indeed this is what I was using last year with Rspec & Capybara!

What students didn't understand

Right until the end I had people who had troubles understanding function signatures, function instances and calling/executing a function. Also returning a value from a function vs. printing the (same) value on screen or assigning to the same global variable (e.g. FIB_NUMBERS).

In the same category falls using method parameters vs. using global variables (which happened to have the same value), using the parameters as arguments to another function inside the body of the current function, using class attributes (e.g. self.name) to store and pass values around vs. local variables in methods vs. method parameters which have the same names.

I think there was some confusion about lists, dictionaries and tuples but we did practice mostly with list structures so I don't have enough information.

I have the impression that object oriented programming (classes and instances, we didn't go into inheritance) are generally confusing to beginners with zero programming experience. The classical way to explain them is by using some abstractions like animal -> dog -> a particular dog breed -> a particular pet. OOP was explained to me in a similar way back in school so these kinds of abstractions are very natural for me. I have no idea if my explanation sucks or students are having hard time wrapping their heads around the abstraction. I'd love to hear some feedback from other instructors on this one.

I think there is some misunderstanding between a class (a definition of behavior) and an instance/object of this class (something which exists into memory). This may also explain the difficulty remembering or figuring out what self points to and why do we need to use it inside method bodies.

For unittest.TestCase we didn't do lots of practice which is my fault. The homework assignments request the students to go back to solutions of previous modules and implement more tests for them. Next time I should provide a module (possibly with non-obvious bugs) and request to write a comprehensive test suite for it.

Because of the missing practice there was some confusion/misunderstanding about the setUpClass/tearDownClass and the setUp/tearDown methods. Also add to the mix that the first are @classmethod while the later are not. "To be safe" students always defined both as class methods!

I have since corrected the training materials but we didn't have good examples (nor practiced) explaining the difference between setUpClass (executed once aka before suite) and setUp (possibly executed multiple times aka before test method).

On the Selenium side I think it is mostly practice which students lack, not understanding. The entire Selenium framework (any web test framework for that matter) boils down to

  • Load a page
  • Find element(s)
  • Click or hover (that one was tricky) element
  • Get element's attribute value or text
  • Wait for the proper page to load (or worst case AJAX calls)

IMO finding the correct element on the page is on-par with waiting (which also relies on locating elements) and took 80% of the time we spent working with Selenium.

Thanks for reading and don't forget to comment and give me your feedback!

Image source: https://www.udemy.com/selenium-webdriver-with-python/

There are comments.

Quality Assurance According 2 Einstein


Quality Assurance According 2 Einstein is a talk which introduces several different ideas about how we need to think and approach software testing. It touches on subjects like mutation testing, pairwise testing, automatic test execution, smart test non-execution, using tests as monitoring tools and team/process organization.

Because testing is more thinking than writing I have chosen a different format for this presentation. It contains only slides with famous quotes from one of the greatest thinkers of our time - Albert Einstein!

This blog post includes the accompanying links and references only! It is my first iteration on the topic so expect it to be unclear and incomplete, use your imagination! I will continue working and presenting on the same topic in the next few months so you can expect updates from time to time. In the mean time I am happy to discuss with you down in the comments.






This principle can be applied to any team/process within the organization. The above link is reference to a nice book which was recommended to me but the gist of it is that we always need to analyze, ask questions and change is we want to achieve great results. A practicle example of what is possible if you follow this principle is this talk Accelerate Automation Tests From 3 Hours to 3 Minutes.


The topic here is "using tests as a monitoring tool". This is something I started a while back ago, helping a prominent startup with their production testing but my involvement ended very soon after the framework was deployed live so I don't have lots of insight.

As the first few days this technique identified some unexpected behaviors, for example a 3rd party service was updating very often. Once even they were broken for a few hours - something nobody had information about.

Since then I've heard about 2 more companies using similar techniques to continuously validate that production software continues to work without having a physical person to verify it. In the event of failures there are alerts which are delath with accordingly.


That much must be obvious to us quality engineers. What about the future however?

I don't have anything more concrete here. Just looking towards what is coming next!


Thanks for reading and happy testing!

There are comments.

Automatic cargo update & pull requests for Rust projects

If you follow my blog you are aware that I use automated tools to do some boring tasks instead of me. For example they can detect when new versions of dependencies I'm using are available and then schedule testing against them on the fly.

One of these tools is Strazar which I use heavily for my Django based packages. Example: django-s3-cache build job.

Recently I've made a slightly different proof-of-concept for a Rust project. Because rustc and various dependencies (called crates) are updated very often we didn't want to expand the test matrix like Strazar does. Instead we wanted to always build & test against the latest crates versions and if that passes create a pull request for the update (in Cargo.lock). All of this unattended of course!

To start create a cron job in Travis CI which will execute once per day and call your test script. The script looks like this:


if [ -z "$GITHUB_TOKEN" ]; then
    echo "GITHUB_TOKEN is not defined"
    exit 1


git checkout -b $BRANCH_NAME
cargo update && cargo test

DIFF=`git diff`
# NOTE: we don't really check the result from testing here. Only that
# something has been changed, e.g. Cargo.lock
if [ -n "$DIFF" ]; then
    # configure git authorship
    git config --global user.email "atodorov@MrSenko.com"
    git config --global user.name "Alexander Todorov"

    # add a remote with read/write permissions!
    # use token authentication instead of password
    git remote add authenticated https://atodorov:$GITHUB_TOKEN@github.com/atodorov/bdcs-api-rs.git

    # commit the changes to Cargo.lock
    git commit -a -m "Auto-update cargo crates"

    # push the changes so that PR API has something to compare against
    git push authenticated $BRANCH_NAME

    # finally create the PR
    curl -X POST -H "Content-Type: application/json" -H "Authorization: token $GITHUB_TOKEN" \
         --data '{"title":"Auto-update cargo crates","head":"automated_cargo_update","base":"master", "body":"@atodorov review"}' \

A few notes here:

  • You need to define a secret GITHUB_TOKEN variable for authentication;
  • The script doesn't force push, but in practice that may be useful (e.g. updating the PR);
  • The script doesn't have any error handling;
  • If PR is still open GitHub will tell us about it but we ignore the result here;
  • DON'T paste this into your Makefile because the GITHUB_TOKEN variable will be expanded into the logs and your secrets go away! Always call the script from your Makefile to avoid revealing secrets.
  • I am using topic branches because this is a POC. Switch to master and maybe move all URLs as variables at the top of the script!
  • I run this cron build against a fork of the project because the team doesn't feel comfortable having automated commits/pushes. I also create the pull requests against my own fork. You will have to adjust the targets if you want your PR to go to the original repository.

Here is the PR which was created by this script: https://github.com/atodorov/bdcs-api-rs/pull/5

Notice that it includes previous commits b/c they have not been merged to the master branch!

Here's the test job (#77) which generated this PR: https://travis-ci.org/atodorov/bdcs-api-rs/builds/219274916

Here's a test job (#87) which bails out miserably because the PR already exists: https://travis-ci.org/atodorov/bdcs-api-rs/builds/220954269

This post is part of my Quality Assurance According to Einstein series - a detailed description of useful techniques I will be presenting very soon.

Thanks for reading and happy testing!

There are comments.

Page 1 / 7