atodorov.org


you can logoff, but you can never leave

What I Learned from EuRuKo 2016

EuRuKo 2016

As my frequent readers may know I try to summarize all the conferences and events I go to. This year's EuRuKo inspired me to take a different approach and instead of quickly summarizing the event I will try to highlight what I have learned from it! My intention is to use this as a tool to improve my skills and the work I do. It will probably be a long post so here we go.

Let me say that I don't consider myself a Ruby developer although I do write a small amount of Ruby code. I also don't really consider myself a developer although I have a formal degree in software engineering and do my fair share of open source contributions.

Being different and thinking differently has always been helpful to me in Quality Assurance and this time was no exception. Attending a conference I knew nothing about and meeting with people whose job is totally different than mine turned out to be my greatest experience on the conference circuit this year.

Lesson 1

Get out of the comfort zone, meet new people, exchange ideas and learn! The very fact that I am writing this post not following my usual summary style proves this is working.

Very early during the event I started to notice a recurring theme which grew stronger by the minute. The Ruby community is very open and inclusive to newcomers and they seem to be doing a very good job about on-boarding everyone who wants to learn. I already wrote about Ivan Nemitchenko's experience of organizing remote internships and there are also the Rails Girls local communities, the Rails Girls Summer of Code (didn't know about it) and the various local Ruby communities who pitched their cities to host the next EuRuKo. I really loved this feeling of community. In the broader Linux, Python and QA world I have not seen this being so pronounced.

Lesson 2

Open up (the open source) community even more. Make it easier for newcomers to join! Treat them as human and don't expect them to be like yourself. Do teach and mentor both to help newcomers but also to help yourself become better!

This is mostly on par with my community work but I think I can do better. I will take the time to evaluate what I've been doing in the past and identify areas for improvements. I also encourage my readers and students to send me feedback as well.

I've also learned that junior developers can make meaningful contributions to production grade code when they are given the appropriate set of tasks and guidance. Stephanie Nemeth argued that companies should hire (more) enthusiastic career changers as junior developers because they have very strong motivation for success.

Lesson 3

Re-evaluate how we look at junior developers, especially how we examine and hire them and how we on-board them.

Both lessons 2 and 3 are valid in the open source world and even more so in the corporate world.

I also liked the fact that some of the lightning talks were given by people who had no previous experience in Ruby. @TeamJoda2016 talked about what they did and learn throughout the summer and really cracked the room with their "oh and btw we are looking for a job" as their final slide!

Lesson 4

If you are new/inexperienced at something don't be afraid to try it out. Give it the best you've got and see how it goes. Worst case .... well nothing bad really happens, best case you end up doing the best job in your life. That's also been my personal experience with software testing.

Carina C. Zona's Consequences Of An Insightful Algorithm (old video here) dealt with the ethical responsibilities of us as developers and this is becoming more common with deep learning neural networks.

Lesson 5

We’re able to extract remarkably precise intuitions about an individual. But do we have a right to know what they didn’t consent to share, even when they willingly shared the data that leads us there?

Krissy's The HTT(Pancake) Request made a great analogy of consuming APIs with your customer experience when visiting a restaurant.

Lesson 6

Design APIs (software in general) as if that was a physical product where your customers happiness matters. We see this all the time in our daily jobs and we're guilty of doing it as well. Btw at the moment I'm in the middle of huge refactoring of django-chartit which breaks all backwards compatibility. I guess I will have to re-evaluate my design and approach.

By accident I've made good friends with Alex Georgiev and the folks at Fyber. I liked the fact that at the conference they had couple of people working in QA and we managed to have a nice talk about QA vs developers and the transformation between the two. That also touched on the bigger subject of testers not being able to code and testers not being available for hire.

Lesson 7

Driving people to improve their skills (learn to code, write tests, etc) is possible but needs to come from management, needs clear direction and also a little bit of peer pressure.

After all isn't that what an agile team is supposed to be ?

Now being the able to code, not entirely Ruby ignorant QA guy that I am I was immediately offered several positions in London and Berlin (and no I'm still staying in Sofia). As it turned out good QA engineers with good development skills are in greater demand than developers not only in Sofia but all around the world!

Lesson 8

Fellow QA guys, please do learn to program. Dear developers, please try thinking more like a tester the next time you write code (me included).

Hiring a barista and furnishing your company stand with the best coffee machine you can afford while having an ugly hand written sign saying "MAIN CONFERENCE COFFEE ->" is a marketing stunt that I really love. I'm not sure how well that worked for their hiring but it got them visibility. I'm definitely stealing this one!

Lesson 9

Conference coffee sucks. Provide better one and developers will queue at your stand. To a greater extent - research your target and their needs and provide a product that solves their problem.

What we gave back

indeed Monica it is. Here's the secret sauce

My personal contribution back was telling Yammer and Deliveroo about mutation testing and pointing them to the right tools and videos on the subject. I wish them good luck and happy testing.

NOTE: I will be speaking about mutation testing at several different events in Bulgaria in the next 2 months so make sure to find me if you want to chat.

There are comments.

What Ivan Learned from Organizing Internships

This is a summary of Ivan Nemytchenko's talk at EuRuKo yesterday (slides here). I'm writing this because that was the best talk both in terms of content and visual presentation I saw at the conference and because it is closely related to my work with HackBulgaria.

The short story is that at some point Ivan was mentoring several junior developers and saw the need to scale this effort so he did a call for interns and got back 60 replies.

What an Intern Gets

  1. Projects in their portfolio
  2. Working experience, including team work
  3. Developing an entire product from idea to production

Ivan wanted to find suitable interns who have basic Ruby on Rails knowledge and who could invest a minimum of 20 hours per week of their time so he devised an aptitude test of 3 parts.

Part 1 is developing basic functionality of the product. Part 2 was adding different user types which require different validation logic, etc. Part 3 was adding "purchasing" logic via external APIs. In Part 3 intentionally there was no code review!

The final result was shit! That was the purpose of the test. The reasoning being that there is no right or wrong way to solve the problems he presented to the interns. Instead he wanted to make them think and decide on a solution. Then feel the pain of their decision. Ivan argues that what made us senior developers are these pains we have experienced at some point in our careers, those fuck-ups that we did in some old project. All of them made us better in our job because we could learn from the mistakes we've made and more importantly understand the consequence of our decisions.

The common mistakes Ivan saw were:

  • Ignoring levels of abstraction;
  • Using too many gems without knowing or understanding their limitations;
  • Gems were treated as the only way to solve a problem. More importantly changing this way was out of the question;
  • Interns didn't know about service objects, well even some experienced developers seem to not know that;
  • Business logic was all around the place;
  • Bad naming all around

The next thing Ivan did was a group hangout code review followed by a short lecture about design patterns, a refactoring session and finally cross code review. At the end the product was delivered as expected.

Following these initial efforts Ivan continued (with even more interns, or the next group of them I think) by asking interns to develop internship automatization, that is a means for the system to distribute tasks based on git commits, tags, etc so it can scale. They've added an admin dashboard and started working on an open source alternative to NewRelic (if I got that correctly). He was also able to enlist 2 more mentors to help him.

Problems Ivan found:

  • Not enough mentors and external projects to work on for all of the interns;
  • Treating a project as not real (e.g. not a real world product) is a mistake;
  • A training project has the same management issues that a real product will have and they need to be resolved in pretty much the same way;
  • There was collective irresponsibility from the group of interns. They didn't do what they said they will do;
  • There were communication issues between the interns and the lack of enough mentors was an obvious problem.
  • There was also lack of motivation.

I'd say these are the typical problems one also sees in almost any teams. It doesn't matter if these are teams of students or teams of developers inside some company.

What a Junior Needs

  • A real project to work on;
  • A business context, a reason why something should be done and why it needs to be done in a particular way;
  • Some visible achievement for their portfolio;
  • Team work experience;
  • Whole cycle development experience.

Ivan thinks that the aptitude test worked great because his interns were able to find good jobs afterwards but he will change a few things. There will be even more tests and he will reject unfit/bad interns. He will also do call for mentors not only for interns. And he wants to turn mentors' experience into tests as well.

I particularly like the "business context" item. IMO even seasoned developers need to have this if they are expected to create a great product for their company. We're not just coders but sometimes companies forget that!

I am also wondering how can I apply a similar aptitude test in my work (both mentoring at HackBulgaria and otherwise).

How about Senior Developers

  • They all have routine tasks;
  • and research tasks;
  • Nice to have features and
  • Low priority features;
  • Side project ideas
  • Missing features in their favorite open source projects

Senior developers' tasks and desires will have to align with what a junior needs in order for the mentorship to work. As senior devs we often make a mistake and expect everyone else to think the same way we do and act as fast as we do. Ideally senior developers want to have multiple clones of ourselves to work with! I myself have been guilty of that and trying to change.

In the context of a for-profit company the above findings should be taken into deep consideration if you are about to have interns.

After the talk I was lucky to talk to Ivan and tell him more about the training sessions at HackBulgaria. I also proposed to him the sponsorship model which he hasn't considered. He then made a counter offer: ask interns for high payment upfront and let them recoup that based on their progress towards the end.

I am really happy to have heard this presentation and being able to talk to Ivan in person. I also have my notes about my "QA and Automation 101" training at HackBulgaria and I now have a better idea how to go about organizing and summarizing them (will try to publish that soon).

Last but not least, Ivan works at GitLab and promised to look at an issue I personally have so here it is GitLab #7953 :).

Related reading

There are comments.

Questers Beer'n'Code Day 2.0

Last weekend I've visited Questers Beer'n'Code Day which was an open air mini-conference held at the terrace of their office. As to organization the only drawback was the summer sun which made it impossible to see anything on the screen. Most speakers were OK with that although they wanted to show some code examples.

I have recorded all talks and they are available in my TECH TALKS YouTube play list. You can also hear me asking some questions from behind the camera. All of the talks are in Bulgarian though, so sorry for my English speaking readers.

The afternoon started with Lidiya Georgieva and her talk about clean code and code smells. I find the topic particularly interesting but she didn't go into more details. She said she had used SonarCube but couldn't recommend any other tools, except for the standard lint style ones. I have been using LandScape.io for all Python based code I've been working on recently and I think it is great.

Another talk I found interesting was by my fellow QA Petar Sabev on reporting bugs. It was more of an entry level talk, but still very informative for both less experienced QAs and other technical folks so I definitely recommend it.

The last one, and most interesting, was Bogoi Bogdanov with Scaling Agile. Despite the name he covered some basics about Agile and what it actually is. Afterwards we've stayed and talked for a good 2 hours more. I definitely would like to hear more from him in the future.

A big thanks to Questers for hosting this event and allowing me to record it. Happy watching.

There are comments.

Python 2 vs. Python 3 List Sort Causes Bugs

Can sorting a list of values crash your software? Apparently it can and is another example of my Hello World Bugs. Python 3 has simplified the rules for ordering comparisons which changes the behavior of sorting lists when their contents are dictionaries. For example:

Python 2.7.5 (default, Oct 11 2015, 17:47:16) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
>>> 
>>> [{'a':1}, {'b':2}] < [{'a':1}, {'b':2, 'c':3}]
True
>>>
Python 3.5.1 (default, Apr 27 2016, 04:21:56) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux
>>> [{'a':1}, {'b':2}] < [{'a':1}, {'b':2, 'c':3}]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: dict() < dict()
>>>

The problem is that the second elements in both lists have different keys and Python doesn't know how to compare them. In earlier Python versions this has been special cased as described here by Ned Batchelder (the author of Python's coverage tool) but in Python 3 dictionaries have no natural sort order.

In the case of django-chartit (of which I'm now the official maintainer) this bug triggers when you want to plot data from multiple sources (models) on the same chart. In this case the fields coming from each data series are different and the above error is triggered.

I have worked around this in commit 9d9033e by simply disabling an iterator sort but this is sub-optimal and I'm not quite certain what the side effect might be. I suspect you may end up with a chart where the order of values on the X axis isn't the same for the different models, e.g. one graph plotting the data in ascending order the other one in descending.

The trouble also comes from the fact that we're sorting an iterator (a list of fields) by telling Python to use a list of dicts to determine the sort order. In this arrangement there is no way to tell Python how we want to compare our dicts. The only solution I can think about is creating a custom class and implementing a custom __cmp__() method for this data structure!

There are comments.

PhantomJS 2.1.1 in Ubuntu different from upstream

For some time now I've been hitting PhantomJS #12506 with the latest 2.1.1 version. The problem is supposedly fixed in 2.1.0 but this is not always the case. If you use a .deb package from the latest Ubuntu then the problem still exists, see Ubuntu #1605628.

It turns out the root cause of this, and probably other problems, is the way PhantomJS packages are built. Ubuntu builds the package against their stock Qt5WebKit libraries which leads to

$ ldd usr/lib/phantomjs/phantomjs | grep -i qt
    libQt5WebKitWidgets.so.5 => /lib64/libQt5WebKitWidgets.so.5 (0x00007f5173ebf000)
    libQt5PrintSupport.so.5 => /lib64/libQt5PrintSupport.so.5 (0x00007f5173e4d000)
    libQt5Widgets.so.5 => /lib64/libQt5Widgets.so.5 (0x00007f51737b6000)
    libQt5WebKit.so.5 => /lib64/libQt5WebKit.so.5 (0x00007f5171342000)
    libQt5Gui.so.5 => /lib64/libQt5Gui.so.5 (0x00007f5170df8000)
    libQt5Network.so.5 => /lib64/libQt5Network.so.5 (0x00007f5170c9a000)
    libQt5Core.so.5 => /lib64/libQt5Core.so.5 (0x00007f517080d000)
    libQt5Sensors.so.5 => /lib64/libQt5Sensors.so.5 (0x00007f516b218000)
    libQt5Positioning.so.5 => /lib64/libQt5Positioning.so.5 (0x00007f516b1d7000)
    libQt5OpenGL.so.5 => /lib64/libQt5OpenGL.so.5 (0x00007f516b17c000)
    libQt5Sql.so.5 => /lib64/libQt5Sql.so.5 (0x00007f516b136000)
    libQt5Quick.so.5 => /lib64/libQt5Quick.so.5 (0x00007f5169dad000)
    libQt5Qml.so.5 => /lib64/libQt5Qml.so.5 (0x00007f5169999000)
    libQt5WebChannel.so.5 => /lib64/libQt5WebChannel.so.5 (0x00007f5169978000)

While building from the upstream sources gives

$ ldd /tmp/bin/phantomjs | grep -i qt

If you take a closer look at PhantomJS's sources you will notice there are 3 git submodules in their repository - 3rdparty, qtbase and qtwebkit. Then in their build.py you can clearly see that this local fork of QtWebKit is built first, then the phantomjs binary is built against it.

The problem is that these custom forks include additional patches to make WebKit suitable for Phantom's needs. And these patches are not available in the stock WebKit library that Ubuntu uses.

Yes, that's correct. We need additional functionality that vanilla QtWebKit doesn't have. That's why we use custom version.

Vitaly Slobodin, PhantomJS

At the moment of this writing Vitaly's qtwebkit fork is 28 commits ahead and 39 commits behind qt:dev. I'm surprised Ubuntu's PhantomJS even works.

The solution IMO is to bundle the additional sources into the src.deb package and use the same building procedure as upstream.

There are comments.

On Python Infinite Loops

How do you write an endless loop without using True, False, number constants and comparison operators in Python ?

I've been working on the mutation test tool Cosmic Ray and discovered that it was missing a boolean replacement operator, that is an operator which will switch True to False and vice versa, so I wrote one. I've also added some tests to Cosmic Ray's test suite and then I hit the infinite loop problem. CR's test suite contains the following code inside a module called adam.py

while True:
    break

The test suite executes mutations on adam.py and then runs some tests which it expects to fail. During execution one of the mutations is replace break with continue which makes the above loop infinite. The test suite times out after a while and kills the mutation. Everything fails as expected and we're good.

Adding my boolean replacement operator broke this function. All of the other mutations work as expected but then the loop becomes

while False:
    break

When we test this particular mutation there is no infinite loop so Cosmic Ray's test suite doesn't time out like it should and an error is reported.

job ID 25:Outcome.SURVIVED:adam
command: cosmic-ray worker adam boolean_replacer 2 unittest -- tests
--- mutation diff ---
--- a/home/travis/build/MrSenko/cosmic-ray/test_project/adam.py
+++ b/home/travis/build/MrSenko/cosmic-ray/test_project/adam.py
@@ -32,6 +32,6 @@
     return x
 
 def trigger_infinite_loop():
-    while True:
+    while False:
         break

So the question becomes how to write the loop condition in such a way that nothing will mutate it but it will still remain true so that when break becomes continue this piece of code will become an infinite loop ? Using True or False constants obviously is a no go. Same goes for numeric constants, e.g. 1 or comparison operators like >, <, is, not, etc. - all of them will be mutated and will break the loop condition.

So I took a look at the docs for truth value testing and discovered my solution:

while object():
    break

I'm creating an object instance here which will not be mutated by any of the existing mutation operators.

Thanks for reading and happy testing!

There are comments.

Bug in TuxCon Website

TuxCon bug

Here comes July 9th 2016 and the start of TuxCon ... with a bug on their website! The image above is taken during the first talk of the conference. Obviously the count down timer is completely off.

In init.js:100 there is this piece of code

var finalDate = '2016/07/09';

$('div#counter').countdown(finalDate)
.on('update.countdown', function(event) {
    $(this).html(event.strftime('<span>%D <em>days</em></span>' +
                                '<span>%H <em>hours</em></span>' +
                                '<span>%M <em>minutes</em></span>' +
                                '<span>%S <em>seconds</em></span>'));
});

It counts backwards and updates the HTML until finalDate is reached. Then the HTML is no longer updated and the default values are shown, which in this case are non zero. A simple patch fixes the problem.

Initialize your variables properly and happy testing!

There are comments.

Testing the 8-bit computer Puldin

Puldin creators

Last weekend I visited TuxCon in Plovdiv and was very happy to meet and talk to some of the creators of the Puldin computer! On the picture above are (left to right) Dimitar Georgiev - wrote the text editor, Ivo Nenov - BIOS, DOS and core OS developer, Nedyalko Todorov - director of the vendor company and Orlin Shopov - BIOS, DOS, compiler and core OS developer.

Puldin is 100% pure Bulgarian development, while the “Pravetz” brand was copy of Apple ][ (Pravetz 8A, 8C, 8M), Oric (Pravets 8D) and IBM-PC (Pravetz 16). The Puldin computers were build from scratch both hardware and software and were produced in Plovdiv in the late 80s and early 90s. 50000 pieces were made, at least 35000 of them have been exported to Russia and paid for. A typical configuration in a Russian class room consisted of several Puldin computers and a single Pravetz 16. According to Russian sources the last usage of these computers was in 2003 serving as Linux terminals and being maintained without any support from the vendor (b/c it ceased to exist).

Puldin 601

One of the main objectives of Puldin was full compatibility with IBM-PC. At the time IBM had been releasing extensive documentation about how their software and hardware works which has been used by Puldin's creators as their software specs. Despite IBM-PC using faster CPU the Puldin 601 had a comparable performance due to aggressive software and compiler optimizations.

Testing wise the guys used to compare Puldin's functionality with that of IBM-PC. It was a hard requirement to have full compatibility on the file storage layer, that means floppy disks written on Puldin had to be readable on IBM-PC and vice versa. Same goes for programs compiled on Puldin - they had to execute on IBM-PC.

Everything of course had been tested manually and on top of that all the software had to be burned to ROM before you can do anything with it. As you can imagine the testing process had been quite slow and painful compared to today's standards. I've asked the guys if they'd happened to find a bug in IBM-PC which wasn't present in their code but they couldn't remember.

What was interesting for me on the hardware side was the fact that you can plug the computer directly to a cheap TV set and that it's been one of the first computers which could operate on 12V DC, powered directly from a car battery.

Pravetz 8

There was also a fully functional Pravetz 8 with an additional VGA port to connect it to the LCD monitor as well as a SD card reader wired to function as a floppy disk reader (the small black dot behind the joystick).

For those who missed it (and understand Bulgarian) I have a video recording on YouTube. For more info about the history and the hardware please check-out Olimex post on Puldin (in English). For more info on Puldin and Pravetz please visit pyldin.info (in Russian) and pravetz8.com (in Bulgarian) using Google translate if need be.

There are comments.

Testing Data Structures in Pykickstart

When designing automated test cases we often think either about increasing coverage or in terms of testing more use-cases aka behavior scenarios. Both are valid approaches to improve testing and both of them seem to focus around execution control flow (or business logic). However program behavior is sometimes controlled via the contents of its data structures and this is something which is rarely tested.

In this comment Brian C. Lane and Vratislav Podzimek from Red Hat are talking about a data structure which maps Fedora versions to particular implementations of kickstart commands. For example

class RHEL7Handler(BaseHandler):
    version = RHEL7

    commandMap = {
        "auth": commands.authconfig.FC3_Authconfig,
        "authconfig": commands.authconfig.FC3_Authconfig,
        "autopart": commands.autopart.F20_AutoPart,
        "autostep": commands.autostep.FC3_AutoStep,
        "bootloader": commands.bootloader.RHEL7_Bootloader,
    }

In their particular case the Fedora 21 logvol implementation introduced the --profile parameter but in Fedora 22 and Fedora 23 the logvol command mapped to the Fedora 20 implementation and the --profile parameter wasn't available. This is unexpected change in program behavior although the logvol.py and handlers/f22.py files have 99% and 100% code coverage.

This morning I did some coding and created an automated test for this problem. The test iterates over all command maps. For each command in the map (that is data structure member) we load the module which provides all possible implementations for that command. In the loaded module we search for implementations which have newer versions than what is in the map, but not newer than the current Fedora version under test. With a little bit of pruning the current list of offenses is

ERROR: In `handlers/devel.py` the "fcoe" command maps to "F13_Fcoe" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_Fcoe".

ERROR: In `handlers/devel.py` "FcoeData" maps to "F13_FcoeData" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_FcoeData".

ERROR: In `handlers/devel.py` the "user" command maps to "F19_User" while in
`pykickstart.commands.user` there is newer implementation: "F24_User".

ERROR: In `handlers/f24.py` the "user" command maps to "F19_User" while in
`pykickstart.commands.user` there is newer implementation: "F24_User".

ERROR: In `handlers/f22.py` the "logvol" command maps to "F20_LogVol" while in
`pykickstart.commands.logvol` there is newer implementation: "F21_LogVol".

ERROR: In `handlers/f22.py` "LogVolData" maps to "F20_LogVolData" while in
`pykickstart.commands.logvol` there is newer implementation: "F21_LogVolData".

ERROR: In `handlers/f18.py` the "network" command maps to "F16_Network" while in
`pykickstart.commands.network` there is newer implementation: "F18_Network".

The first two are possibly false negatives or related to the naming conventions used in this module. However the rest appear to be legitimate problems. The user command has introduced the --groups parameter in Fedora 24 (devel is Fedora 25 currently) but the parser will fail to recognize this parameter. The logvol problem is recognized as well since it was never patched. And the Fedora 18 network command implements a new property called hostname which has probably never been available to be used.

You can follow my current work in PR #91 and happy testing your data structures.

There are comments.

Don't Upgrade Galaxy S5 to Android 6.0

Samsung is shipping out buggy software like a boss, no doubt about it. I've written a bit about their bugs previously. However I didn't expect them to release Android 6.0.1 and render my Galaxy S5 completely useless with respect to the feature I use the most.

Lockscreen

Tell me the weather for Brussels

So on Monday I've let Android upgrade to 6.0.1 to be completely surprised that the lockscreen shows the weather report for Brussels, while I'm based in Sofia. I've checked AccuWeather (I did go to Brussels earlier this year) but it displayed only Sofia and Thessaloniki. To get rid of this widget go to Settings -> Lockscreen -> Additional information and turn it off!

I think this weather report comes from GPS/Location based data, which I have turned off by default but did use a while back ago. After turning the widget off and back on it didn't appear on the lockscreen. I suspect they fall back to showing the last good known value when data is missing instead of handling the error properly.

Apps are gone

Some of my installed apps are missing now. So far I've noticed that the Gallery and S Health icons have disappeared from my homescreen. I think S Health came from Samsung's app store but still they shouldn't have removed it silently. Now I wonder what happened to my data.

I don't see why Gallery was removed though. The only way to view pictures is to use the camera app preview functionality which is kind of grose.

Grayscale in powersafe mode is gone

The killer feature on these higher end Galaxy devices is the Powersafe mode and Ultra Powersafe mode. I use them a lot and by default have my phone in Powersafe mode with grayscale colors enabled. It is easier on the eyes and also safes your battery.

NOTE: grayscale colors don't affect some displays but these devices use AMOLED screens which need different amounts of power to display different colors. More black means less power required!

After the upgrade grayscale is no more. There's not even an on/off switch. I've managed to find a workaround though. First you need to enable developer mode by tapping 7 times on About device -> Build number. Then go to Settings -> Developer options, look for the Hardware Accelerated Rendering section and select Simulate Color Space -> Monochromacy! This is a bit ugly hack and doesn't have the convenience of turning colors on/off by tapping the quick Powersafe mode button at the top of the screen!

It looks like Samsung didn't think this upgrade well enough or didn't test it well enough ? In my line of work (installation and upgrade testing) I've rarely seen such a big blunder. Thanks for reading and happy testing!

There are comments.

How To Hire Software Testers, Pt. 3

In previous posts (links below) I have described my process of interviewing QA candidates. Today I'm quoting an excerpt from the book Mission: My IT career(Bulgarian only) by Ivaylo Hristov, one of Komfo's co-founders.

Fedora pen

He writes

Probably the most important personal trait of a QA engineer is to
be able to think outside given boundaries and prejudices
(about software that is). When necessary to be non-conventional and
apply different approaches to the problems being solved. This will help
them find defect which nobody else will notice.

Most often errors/mistakes in software development are made due to
wrong expectations or wrong assumptions. Very often this happens because
developers hope their software will be used in one particular way
(as it was designed to) or that a particular set of data will be returned.
Thus the skill to think outside the box is the most important skill
we (as employers) are looking to find in a QA candidate. At job interviews
you can expect to be given tasks and questions which examine those skills.

How would you test a pen?

This is Ivaylo's favorite question for QA candidates. He's looking for attention to details and knowing when to stop testing. Some of the possible answers related to core functionality are

  • Does the pen write in the correct color
  • Does the color fades over time
  • Does the pen operate normally at various temperatures? What temperature intervals would you choose for testing
  • Does the pen operate normally at various atmospheric pressure
  • When writing, does the pen leave excessive ink
  • When writing, do you get a continuous line or not
  • What pressure does the user need to apply in order to write a continuous line
  • What surfaces can the pen write on? What surfaces would you test
  • Are you able to write on a piece of paper if there is something soft underneath
  • What is the maximum inclination angle at which the pen is able to write without problems
  • Does the ink dry fast
  • If we spill different liquids onto a sheet of paper, on which we had written something, does the ink stay intact or smear
  • Can you use pencil rubber to erase the ink? What else would you test
  • How long can you write before we run out of ink
  • How fat is the ink line

Then Ivaylo gives a few more non-obvious answers

  • Verify that all labels on the pen/ink cartridge are correctly spelled and how durable they are (try to erase them)
  • Strength test - what is the maximum height you can drop the pen from without breaking it
  • Verify that dimensions are correct
  • Test if the pen keeps writing after not being used for some time (how long)
  • Testing individual pen components under different temperature and atmospheric conditions
  • Verify that materials used to make the pen are safe, e.g. when you put the pen in your mouth

When should you stop ? According to the book there can be between 50 and 100 test cases for a single pen, maybe more. It's not a good sign if you stop at the first 3!

If you want to know what skills are revealed via these questions please read my other posts on the topic:

Thanks for reading and happy testing!

There are comments.

Capybara's find().click doesn't always fire onClick

Recently I've observed a strange behavior in one of the test suites I'm working with - a test which submits a web form appeared to fail at a rate between 10% and 30%. This immediately made me think there is some kind of race-condition but it turned out that Capybara's find().click method doesn't always fire the onClick event in the browser!

The test suite uses Capybara, Poltergeist and PhantomJS. The element we click on is an image, coupled to a hidden check-box underneath. When the image is clicked onClick is fired and the check-box is updated accordingly. In the failed cases the underlying check-box wasn't updated! Searching the web reveals a similar problem described by Alex Okolish so we've tried his solution:

div.find('.replacement', visible: true).trigger(:click)

How to Test

The failure behavior being somewhat flaky I've opted for running the test multiple times and see what happens when it fails. Initially I've executed the test in batches of 10 and 20 repetitions to get a feeling of how often does it fail before proceeding with debugging. Debugging was done by logging variables and state on the console and repeating multiple times. Once a possible solution was proposed we've run the tests in batches of 100 repetitions and counted how often they failed.

At the end, when Alex's solution was discovered we've repeated the testing around 1000 times just to make sure it works reliably. So far this has been working without issues!

I've spent around a week working on this together with a co-worker and we didn't really want to spend more time trying to figure out what was going wrong with our tools. Once we saw that trigger does the job we didn't continue debugging Capybara or PhantomJS.

There are comments.

DEVit Conf 2016

It's been another busy week after DEVit conf took place in Thessaloniki. Here are my impressions.

DEVit 2016

Pre-conference

TechMinistry is Thessaloniki's hacker space which is hosted at a central location, near major shopping streets. I've attended an Open Source Wednesday meeting. From the event description I thought that there was going to be a discussion about getting involved with Firefox. However that was not the case. Once people started coming in they formed organic groups and started discussing various topics on their own.

I was also shown their 3D printer which IMO is the most precise of 3D printers I've seen so far. Imagine what it would be like to click Print, sometime in the future, and have your online orders appear on your desk over night. That would be quite cool!

I've met with Christos Bacharakis, a Mozilla representative for Greece, who gave me some goodies for my students at HackBulgaria!

On Thursday I spent the day merging pull requests for MrSenko/pelican-octopress-theme and attended the DEVit Speakers dinner at Massalia. Food and drinks were very good and I even found a new recipe for mushrooms with ouzo, of which I think I had a bit too many :).

I was also told that "a full stack developer is a developer who can introduce a bug to every layer of the software stack". I can't agree more!

DEVit

The conference day started with a huge delay due to long queues for registration. The fist talk I attended, and the best one IMO was Need It Robust? Make It Fragile! by Yegor Bugayenko (watch the video). There he talked about two different approaches to writing software: fail safe vs. fail fast.

He argues that when software is designed to fail fast bugs are discovered earlier in the development cycle/software lifetime and thus are easier to fix, making the whole system more robust and more stable. On the other hand when software is designed to hide failures and tries to recover auto-magically the same problems remain hidden for longer and when they are finally discovered they are harder to fix. This is mostly due to the fact that the original error condition is hidden and manifested in a different way which makes it harder to debug.

Yegor made several examples, all of which are valid code, which he considers bad practice. For example imagine we have a function that accepts a filename as parameter:

def read_file_fail_safe(fname):
    if not os.path.exists(fname):
        return -1

    # read the file, do something else
    ...
    return bytes_read


def read_file_fail_fast(fname):
    if not os.path.exists(fname):
        raise Exception('File does not exist')

    # read the file, do something else
    return bytes_read

In the first example read_file_fail_safe returns -1 on error. The trouble is whoever is calling this method may not check for errors thus corrupting the flow of the program further down the line. You may also want to collect metrics and update your database with the number of bytes processed - this will totally skew your metrics. C programmers out there will quickly remember at least one case when they didn't check the return code for errors!

The second example read_file_fail_fast will raise an exception the moment it encounters a problem. It's not its fault that the file doesn't exist and there's nothing it can do about it, nor is its job to do anything about it. Raising an exception will surface back to the caller and they will be notified about the problem, taking appropriate actions to resolve it.

Yegor was also unhappy that many books teach fail safe and even IDEs (for Java) generate fail safe boiler-plate code (need to check this)! Indeed it is me who asks the first question Are there any tools to detect fail safe code patterns? and it turns out there aren't (for the majority of cases that is). If you happen to know such a tool please post a link in the comments below.

I was a bit disappointed by the rest of the talks. They were all high-level overviews IMO and didn't go deep technical. Last year was better. I also wanted to attend the GitHub Patchwork workshop but looking at the agenda it looked like this is for users who are starting with git and GitHub (which I'm not).

The closing session of the day was "Real time front-end alchemy, or: capturing, playing, altering and encoding video and audio streams, without servers or plugins!" by Soledad Penades from Mozilla. There she gave a demo about the latest and greatest in terms of audio and video capturing, recording and mixing natively in the browser. This is definitively very cool for apps in the audio/video space but I can also imagine an application for us software testers.

Depending on computational and memory requirements you should be able to record everything the user does in their browser (while on your website) and send it back home when they want to report an error or contact support. Definitely better than screenshots and having to go back and forth until the exact steps to reproduce are established.

There are comments.

Changing Rails consider_all_requests_local in RSpec fails

As many others I've been trying to change Rails.application.config.consider_all_requests_local and Rails.application.config.action_dispatch.show_exceptions inside my RSpec tests in order to test custom error pages in a Rails app. However this doesn't work. My code looked like this

feature 'Exceptions' do
  before do
    Rails.application.config.action_dispatch.show_exceptions = true
    Rails.application.config.consider_all_requests_local = false
  end

This works only if I execute exceptions_spec.rb alone. However when something else executes before that it fails. The config values are correctly updated but that doesn't seem to have effect.

The answer and solution comes from Henrik N.

action_dispatch.show_exceptions gets copied and cached in Rails.application.env_config, so even if you change Rails.application.config.action_dispatch.show_exceptions in this before block the value isn't what you expect when it's used in ActionDispatch::DebugExceptions.

In fact DebugExceptions uses env['action_dispatch.show_exceptions']. The correct code should look like this

before do
  method = Rails.application.method(:env_config)
  expect(Rails.application).to receive(:env_config).with(no_args) do
    method.call.merge(
      'action_dispatch.show_exceptions' => true,
      'action_dispatch.show_detailed_exceptions' => false,
      'consider_all_requests_local' => false
    )
  end
end

This allows the test to work regardless of the order of execution of spec files. I don't know why but I also had to leave show_detailed_exceptions otherwise I wasn't getting the desired results.

There are comments.

Mismatch in Pyparted Interfaces

Last week my co-worker Marek Hruscak, from Red Hat, found an interesting case of mismatch between the two interfaces provided by pyparted. In this article I'm going to give an example, using simplified code and explain what is happening. From pyparted's documentation we learn the following

pyparted is a set of native Python bindings for libparted. libparted is the library portion of the GNU parted project. With pyparted, you can write applications that interact with disk partition tables and filesystems.

The Python bindings are implemented in two layers. Since libparted itself is written in C without any real implementation of objects, a simple 1:1 mapping of externally accessible libparted functions was written. This mapping is provided in the _ped Python module. You can use that module if you want to, but it's really just meant for the larger parted module.

_ped       libparted Python bindings, direct 1:1: function mapping
parted     Native Python code building on _ped, complete with classes,
           exceptions, and advanced functionality.

The two interfaces are the _ped and parted modules. As a user I expect them to behave exactly the same but they don't. For example some partition properties are read-only in libparted and _ped but not in parted. This is the mismatch I'm talking about.

Consider the following tests (also available on GitHub)

diff --git a/tests/baseclass.py b/tests/baseclass.py
index 4f48b87..30ffc11 100644
--- a/tests/baseclass.py
+++ b/tests/baseclass.py
@@ -168,6 +168,12 @@ class RequiresPartition(RequiresDisk):
         self._part = _ped.Partition(disk=self._disk, type=_ped.PARTITION_NORMAL,
         self._part = _ped.Partition(disk=self._disk, type=_ped.PARTITION_NORMAL,
                                     start=0, end=100, fs_type=_ped.file_system_type_get("ext2"))
 
+        geom = parted.Geometry(self.device, start=100, length=100)
+        fs = parted.FileSystem(type='ext2', geometry=geom)
+        self.part = parted.Partition(disk=self.disk, type=parted.PARTITION_NORMAL,
+                                    geometry=geom, fs=fs)
+
+
 # Base class for any test case that requires a hash table of all
 # _ped.DiskType objects available
 class RequiresDiskTypes(unittest.TestCase):
diff --git a/tests/test__ped_partition.py b/tests/test__ped_partition.py
index 7ef049a..26449b4 100755
--- a/tests/test__ped_partition.py
+++ b/tests/test__ped_partition.py
@@ -62,8 +62,10 @@ class PartitionGetSetTestCase(RequiresPartition):
         self.assertRaises(exn, setattr, self._part, "num", 1)
         self.assertRaises(exn, setattr, self._part, "fs_type",
             _ped.file_system_type_get("fat32"))
-        self.assertRaises(exn, setattr, self._part, "geom",
-                                     _ped.Geometry(self._device, 10, 20))
+        with self.assertRaises((AttributeError, TypeError)):
+#            setattr(self._part, "geom", _ped.Geometry(self._device, 10, 20))
+            self._part.geom = _ped.Geometry(self._device, 10, 20)
+
         self.assertRaises(exn, setattr, self._part, "disk", self._disk)
 
         # Check that values have the right type.
diff --git a/tests/test_parted_partition.py b/tests/test_parted_partition.py
index 0a406a0..8d8d0fd 100755
--- a/tests/test_parted_partition.py
+++ b/tests/test_parted_partition.py
@@ -23,7 +23,7 @@
 import parted
 import unittest
 
-from tests.baseclass import RequiresDisk
+from tests.baseclass import RequiresDisk, RequiresPartition
 
 # One class per method, multiple tests per class.  For these simple methods,
 # that seems like good organization.  More complicated methods may require
@@ -34,11 +34,11 @@ class PartitionNewTestCase(unittest.TestCase):
         # TODO
         self.fail("Unimplemented test case.")
 
-@unittest.skip("Unimplemented test case.")
-class PartitionGetSetTestCase(unittest.TestCase):
+class PartitionGetSetTestCase(RequiresPartition):
     def runTest(self):
-        # TODO
-        self.fail("Unimplemented test case.")
+        with self.assertRaises((AttributeError, TypeError)):
+            #setattr(self.part, "geometry", parted.Geometry(self.device, start=10, length=20))
+            self.part.geometry = parted.Geometry(self.device, start=10, length=20)
 
 @unittest.skip("Unimplemented test case.")
 class PartitionGetFlagTestCase(unittest.TestCase):

The test in test__ped_partition.py works without problems, I've modified it for visual reference only. This was also the inspiration behind the test in test_parted_partition.py. However the second test fails with

======================================================================
FAIL: runTest (tests.test_parted_partition.PartitionGetSetTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/pyparted/tests/test_parted_partition.py", line 41, in runTest
    self.part.geometry = parted.Geometry(self.device, start=10, length=20)
AssertionError: (<type 'exceptions.AttributeError'>, <type 'exceptions.TypeError'>) not raised

----------------------------------------------------------------------

Now it's clear that something isn't quite the same between the two interfaces. If we look at src/parted/partition.py we see the following snippet

137     fileSystem = property(lambda s: s._fileSystem, lambda s, v: setattr(s, "_fileSystem", v))
138     geometry = property(lambda s: s._geometry, lambda s, v: setattr(s, "_geometry", v))
139     system = property(lambda s: s.__writeOnly("system"), lambda s, v: s.__partition.set_system(v))
140     type = property(lambda s: s.__partition.type, lambda s, v: setattr(s.__partition, "type", v))

The geometry property is indeed read-write but the system property is write-only. git blame leads us to the interesting commit 2fc0ee2b, which changes definitions for quite a few properties and removes the _readOnly method which raises an exception. Even more interesting is the fact that the Partition.geometry property hasn't been changed. If you look closer you will notice that the deleted definition and the new one are exactly the same. Looks like the problem existed even before this change.

Digging down even further we find commit 7599aa1 which is the very first implementation of the parted module. There you can see the _readOnly method and some properties like path and disk correctly marked as such but geometry isn't.

Shortly after this commit the first test was added (4b9de0e) and a bit later the second, empty test class, was added (c85a5e6). This only goes to show that every piece of software needs appropriate QA coverage, which pyparted was kind of lacking (and I'm trying to change that).

The reason this bug went unnoticed for so long is the limited exposure of pyparted. To my knowledge anaconda, the Fedora installer is its biggest (if not single) consumer and maybe it uses only the _ped interface (I didn't check) or it doesn't try to do silly things like setting a value to a read-only property.

The lesson from this story is to test all of your interfaces and also make sure they are behaving in exactly the same manner!

Thanks for reading and happy testing!

There are comments.

Capybara's within() Altering expect(page) Scope

When making assertions inside a within block the assertion scope is limited to the element selected by the within() function, although it looks like you are asserting on the entire page!

scenario 'Pressing Escape closes autocomplete popup' do
  within('#new-broadcast') do
    find('#broadcast_field').set('Hello ')
      start_typing_name('#broadcast_field', '@Bret')
      # will fail below
      expect(page).to have_selector('.ui-autocomplete')
      send_keys('#broadcast_field', :escape)
  end
  expect(page).to have_no_selector('.ui-autocomplete')
end

The above code failed the first expect() and it took me some time before I figured it out. Capybara's test suite itself gives you the answer

it "should assert content in the given scope" do
  @session.within(:css, "#for_foo") do
    expect(@session).not_to have_content('First Name')
  end
  expect(@session).to have_content('First Name')
end

So know your frameworks and happy testing.

There are comments.

Unix Stickers for Your Laptop

Last month I was asked to review stickers from UnixStickers. In return I would receive some of them. I've made them a counter offer - they send me stickers and I give them to students attending my QA-and-Automation-101 course.

Unix Stickers

Yesterday I gave away everything I was sent, some of which you can see on the picture above. All stickers were gone in minutes. The ones that were left were the yellow JS ones and the Fedora infinity logo. It turned out most students are not familiar with Fedora but otherwise liked the stickers.

If you haven't come across UnixStickers until now I definitely recommend it. It is a great source to purchase stickers, mugs and T-shirts branded with your favorite open source project(s). In return some of the money is donated back to the community to support their open source work. A great business model in my opinion.

There are comments.

3 Bugs in Grajdanite

Grajdanite is a social app that allows everyone (in Bulgaria) to photograph vehicles in breach of traffic rules or misbehaving drivers, upload them online and ask them to appologize. They also offer some functionality to report offenses to the authorities are are partnering with local municipalities and law enforcement agencies to make the process easier. And of course this is one of my favorite apps as of latest.

Missing Icon in My Profile

Missing icon

The more offenses you report the more points you get. Points lead to ranks (e.g. junior officer, senior officer, etc). The page showing your points and rank is missing an icon. If I had to guess this is the badge which comes with different ranks.

Preloading the Very First Form Value

Preloading gone wrong

Once you opt for reporting an offence to the authorities you need to specify the address where the action took place, your name, phone and e-mail address. The app correctly saves your details and pre-loads them later to speed-up data entry. However I typed my e-mail wrong the very first time. Now every time I want to report something the app pre-loads the wrong address. Even after I change it to the correct one, the next time I still see the very first, wrong value.

In code this is probably something like:

# pre-load
form.email = store.get("email", "")
form.show()

# save
if form.firstTime():
    store.save("email", form.email)

The fix is to save the form value every time (not expensive operation here) or check if the current value is different from the last time and only then save it.

DST and Time Sync

The last bug is in the app confirmation email. Once an offence is reported the user receives an email with the uploaded photo and the information they have provided. The email includes a timestamp. However the email timestamp is 1 hour off from the actual time. In particular it is 1 hour behind the current time and I think the email server doesn't follow summer time.

The result from this is:

  • Report an offense
  • Wait 1 minute for the email to be received;
  • The email says the offense happened 1 hour ago!

All of these bugs are in version 3.86.3, which is the latest one.

There are comments.

How To Hire Software Testers, Pt. 2

In my previous post I have described the process I follow when interviewing candidates for a QA position. The first question is designed to expose the applicant's way of thinking. My second question is designed to examine their technical understanding and to a lesser extent their way of thinking.

Sudoku

How do You Test a Sudoku Solving Function

You have implementation of a sudoku solver function with the following pseudocode:

func Sudoku(Array[2]) {
    ...
    return Array[2]
}
  • The function solves a sudoku puzzle;
  • Input parameter is a two-dimensional array with the known numbers (from 1 to 9) in the Sudoku grid;
  • The output is a two-dimensional array with the numbers from the solved puzzle.

You have 10 minutes to write down a list of all test cases you can think of!

Behind The Scenes

One set of possible tests is to examine the input and figure out if the function has been passed valid data. In the real-world programs interact with each other, they are not alone. Sometimes it happens that a valid output from one program isn't a valid input for the next one. Also we have malicious users who will try to break the program.

If a person manages to test for this case then I know they have a bit more clue about how software is used in the real-world. This also touches a bit on white-box testing, where the tester has full info about the software under test. In this example the implementation is intentionally left blank.

OTOH I've seen answers where the applicant blindly assumes that the input is 1-9, because the spec says so, and excludes the entire input testing from their scope. I classify this answer as immediate failure, because a tester should never assume anything and test to verify their initial conditions are indeed as stated in the documentation.

Another set of possible tests is to verify the correct work of the function. That is to verify the proposed Sudoku solution is indeed following the rules of the game. This is what we usually refer to black-box testing. The tester doesn't know how the SUT works internally, they only know the input data and the expected output.

If a person fails to describe at least one such test case they have essentially failed the question. What is the point of a SUT which doesn't crash (suppose that all previous tests passed) but doesn't produce the desired correct result ?

Then there are test cases related to the environment in which this Sudoku solver function operates. This is where I examine the creativity of the person, their familiarity with other platforms and to some extent their thinking out of the box. Is the Sudoku solver iterative or recursive ? What if we're on an embedded system and recursion is too heavy for it ? How much power does the function require, how fast it works, etc.

A person that provides at least one answer in this category has bonus points over the others who didn't. IMO it is very important for a tester to have experience with various platforms and environments because this helps them see edge cases which others will not be able to see. I also consider a strong plus if the person shows they can operate outside their comfort zone.

If we have time I may ask the applicant to write the tests using a programming language they know. This is to verify their coding and automation skills.

OTOH having the tests as code will show me how much the person knows about testing vs. coding. I've seen solutions where people write a for loop, looping over all numbers from 1 to 100 and testing if they are a valid input to Sudoku(). Obviously this is pointless and they failed the test.

Last but not least, the question asks for testing a particular Sudoku solver implementation. I expect the answers to be designed around the given function. However I've seen answers designed around a Sudoku solver website or described as intermediate states in an interactive Sudoku game (e.g. wrong answers shown in red). I consider these invalid because the question is to test a particular given function, not anything Sudoku related. If you do this in real-life that means you are not testing the SUT directly but maybe touching it indirectly (at best). This is not what a QA job is about.

What Are The Correct Answers

Here are some of the possible tests.

  • Test with single dimensional input array - we expect an error;
  • Test with 3 dimensional input array - we expect an error;
  • Then proceed testing with 2 dimensional array;
  • Test with number less than 1 (usually 0) - expect error;
  • Test with number greater than 9 (usually 10) - expect error;
  • Test how the function handles non-numerical data - chars & symbols (essentially the same thing for our function);
  • Test with strings which actually represent a number, e.g. "1";
  • Test with floating point numbers, e.g. 1.0, 2.0, 3.0 - may or may not work depending on how the code is written;
  • If floating point numbers are accepted, then test with a different locale. Is "1.0" the same as "1,0";
  • Test with null, nil, None (whatever the language supports) - this should be a valid value for unknown numbers and not cause a crash;
  • Test if the function validates that the provided input follows the Sudoku rules by passing it duplicate numbers in one row, column or square. It should produce an error;
  • Test if the input data contains the minimum number of givens, 17 for a general Sudoku, so that a solution can be found. Otherwise the function may go into an endless loop;
  • Verify the proposed solution conforms to Sudoku rules;
  • Test with a fully solved puzzle as input - output should be exactly the same;
  • If on mobile, measure battery consumption for 1 minute of operation. I've seen a game which uses 1% battery power for 1 minute of game play;
  • Test for buffer overflows;
  • Test for speed of execution (performance);
  • Test performance on single and multiple (core) CPUs - depending on the language and how the function is written this may produce a difference or not;

I'm sure I'm missing something so please use the comments below to tell me your suggestions.

Thanks for reading and happy testing!

There are comments.

How To Hire Software Testers, Pt. 1

Many people have asked me how do I make sure a person who applies for a QA/software tester position is a good fit ? On the opposite side people have asked online how do they give correct answers on test related questions at job interviews. I have two general questions to help me decide if a person knows about testing and if they are a good fit for the team or not.

Login form

How do You Test a Login Form

You are given the login form above and the following constraints:

  • Log in is possible with username and password or through the social networks;
  • After successful registration an email with the following content is sent to the user:

Helo and welcome to atodorov.org! Click _here_ to confirm your emeil address.

You have 10 minutes to write down a list of all test cases you can think of!

Behind The Scenes

The question looks trivial but isn't as easy to answer as you may think. If you haven't spent the last 20 years of your life in a cave, chances are that you will give technically correct answers but this is not the only thing I'm looking for.

The question is designed to simulate a real-world scenario, where the QA person is given a piece of software, or requirements document and tasked with creating a test plan for it. The question is intentionally vague because that's how real-world works, most often testers don't have all the requirements and specifications available beforehand.

The time constrain, especially when the interview is performed in person, simulates work under pressure - get the job done as soon as possible.

While I review the answers I'm trying to figure out how does the person think, not how much about technology they know. I'm trying to figure out what are their strong areas and where they need to improve. IMO being able to think as a tester and having attention to details, being able to easily spot corner cases and look at the problem from different angles is much more important than technical knowledge in a particular domain.

As long as a person is suited to think like a tester they can learn to apply their critical thinking to any software under test and use various testing techniques to discover or safeguard against problems.

  • A person that answers quickly and intuitively is better than a person who takes a long time to figure out what to test. I can see they are active thinkers and can work without micro-management and hand-holding.

  • A person that goes on and on describing different test cases is better than one who limits themselves to the most obvious cases. I can see they have an exploratory passion, which is the key to finding many bugs and making the software better;

  • A person that goes to explore the system in breadth is better than one who keeps banging on the same test case with more and more variations. I can see they are noticing the various aspects of the software (e.g. social login, email confirmation, etc) but also to some extent, not investing all of their resources (the remaining time to answer) into a single direction. Also in real-world testing, testing the crap out of something is useful up to a point. Afterwards we don't really see any significant value from additional testing efforts.

  • A person that is quick to figure out one or two corner cases is better than a person who can't. This tells me they are thinking about what goes on under the hood and trying to predict unpredictable behavior - for example what happens if you try to register with already registered username or email?

  • A person that asks questions in order to minimize uncertainty and vagueness is better than the one who doesn't. In real-world if the tester doesn't know something they have to ask. Quite often even developers and product managers don't know the answer. Then how are we developing software if we don't know what it is supposed to do ?

  • If given more time (writing interview), a person that organizes their answers into steps (1, 2, 3) is a bit better than one who simply throws at you random answers without context. Similar thought applies to people who write down their test pre-conditions before writing down scenarios. From this I can see that the person is well organized and will have no trouble writing detailed test cases, with pre-conditions, steps to execute and expected results. This is what QAs do. Also we have the, sometimes tedious, task of organizing all test results into a test case management system (aka test book) for further reference.

  • The question intentionally includes some mistakes. In this example 2 spelling errors in the email text. Whoever manages to spot them and tell me about it is better than others who don't spot the errors or assume that's how it is. QAs job is to always question everything and never blindly trust that the state of the system is the way it is. Also simple errors like typos can be embarrassing or generate unnecessary support calls.

  • Bravo if you tested not only the outgoing email but also social login. This shows attention to details, not to mention social is 1/3rd of our example system. It also shows that QA's job doesn't end with testing the core, perceived functionality of the system. QA tests everything, even interactions with external systems if that is necessary.

What Are The Correct Answers

I will document some of the possible answers as I recall them from memory. I will update the list with other interesting answers given by students who applied to my QA and Automation 101 course, answering this very same question.

  • Test if users can register using valid username, email and password;
  • Test if SUT gives an error message when email or password (or username) format doesn't follow a particular format (e.g. no special symbols);
  • After registration, test that the user can login successfully;
  • Depending on requirements test if the user can login before they have confirmed their email address;
  • Test that upon registration a confirmation email is actually sent;
  • Spell-check the email text;
  • Test if the click here piece of text is a hyperlink;
  • Verify that when clicked, the hyperlink successfully confirmed email/activates the account (depending on what confirmed/activated means per requirements);
  • Test what happens if the link is clicked a second time;
  • Test what happens if the link is clicked after 24 or 48 hrs;
  • Test that the social network icons, actually link to the desired SN and not someplace else;
  • Test if new user accounts can be created via all specified social networks;
  • Test what happens if there is an existing user, who registered with a password and they (or somebody else) tries to register via social with an account that has the same email address, aka account hijacking;
  • Same as previous test but try to register a new user, using email address that was previously used with social login;
  • Test what happens if users forget their password - intentionally we don't have the '[] Forgot my password' checkbox. This is both usability feature and missing requirements;
  • Test for simple SQL injections like Bobby Tables. btw I was given this image as an answer which scored high on the geek-o-meter;
  • Test for XSS - Tweetdeck didn't;
  • Test if non-activated/non-confirmed usernames expire after some time and can be used again;
  • Test of fields tab order - something I haven't done in 15 or more years but still valid and I've seen sites getting it wrong quite often;
  • When trying to login test what happens when username/password is wrong or empty;
  • Test if email is required for login - this isn't clear from the requirements so it is a valid answer. Better answer is to clarify that;
  • Test if username/email or password is case sensitive. Valid test and indeed I recently saw a problem where upon registration users entered their emails using some capital letters but they were lower-cased before saving to the DB. Later this broke a piece of code which forgot to apply the lowercase on the input data. The code was handling account reactivation;
  • Test if the password field shows the actual password or not. I haven't seen this in person but I'm certain there is some site which maybe used CSS and nice images instead of the default ugly password field and that didn't work on all browsers;
  • Test if you can copy&paste the masked password, probably trying to steal somebody else's password. Last time I saw this was on early Windows 95 with the modem connection dialog. Very briefly it allowed you to copy the text from the field and paste it into Notepad to reveal the actual password;
  • If we're on mobile (intentionally not specified) test for buffer overflows; Actually test that everywhere and see what happens;
  • Test if the social network buttons use the same action verb. In the example we have Log in, Connect and Sign in. This is sort of usability testing and helping have a unified look and feel of the product;
  • Test which of the Log in and Sign up tabs is active at the moment. The example is intentionally left to look like a wireframe but it is important for the user to easily tell where they are. Otherwise they'll call support or even worse, simply give up on us;
  • Test if all static files (images) will load if they are deployed onto CDN. Not surprisingly I've seen this bug;
  • In case we have a "[] Remember me" checkbox, test if it actually remembers the user credentials. Yesterday I saw this same functionality not working on a specialized desktop app in the corner case where you supply a different connection endpoint (server) instead of the ones already provided. The user defined value is accepted but not saved automatically;
  • Test if the "Remember me" functionality actually saves your last credentials or only the first ones you provided. There is a similar bug in Grajdanite, where once you enter a wrong email, it is remembered and every time the form is pre-filled with the previous value (which is wrong). I'm yet to report it though;
  • Cross-browser testing - hmm, login and registration should work on all browsers you say. It's not browser dependent, is it? Well yeah, login isn't browser dependent unless we did something stupid like pre-handling the form submit via non-cross-platform JavaScript or even accidentally doing so;
  • Test with Unicode characters, especially non Latin ones. It's been many years since we had Unicode but quite a few apps haven't learned how to deal with Unicode text properly.

I'm certain there are more answers and I will update the list as I figure them out. You can always post in the comments and tell me something I've missed.

How to Pass The Job Interview

This is a question I often see on Quora. I have a job interview tomorrow. How do I test a login form (or whatever) ?

If this section is what you're after I suspect you are a junior or wanna-be software tester. As you've seen the interviewer isn't really interested in what you know already, at least not as much. We're interested in getting to know how you think in the course of 30-60 minutes.

If you ever find yourself being asked a similar question just start thinking and answering and don't stop. Vocalize your thoughts, even if you don't know what will happen when testing a certain condition. Then keep going on and on. Look at the problem from all angles, explain how you'd test various aspects and features of the SUT. Then move on to the next bit. Always think about what you may have forgotten and revisit your answers - this is what real QAs do - learn from mistakes. Ask questions, don't ever assume anything. If something is unclear ask to be clarified. For example I've seen a person who doesn't use social networks and didn't know how social login/registration worked. They did good by asking me to describe how that works.

Your goal is to make the interviewer ask you to stop answering. Then tell them a few more answers.

However beware of cheating. You may cheat a little bit by saying you will test this and that or design scenarios you have no clue about. Maybe you read them in my blog or elsewhere. If the interviewer knows their job (which they should) they will instantly ask you another question to verify what you say. Don't forget the interviewer is probably an experienced tester and validating assumptions is what they do every day.

For example, if you told me something about security testing or SQL injection or XSS I will ask you to explain that in more details. If you forgot to mention, one of them, say XSS but only heard about SQL injection I will ask you about the other one. This will immediately tell me if you have a clue what you are talking about.

Feel free to send me suggestions and answers in the comments below. You can find the second part of this post at How do you test a Sudoku solving function.

Thanks for reading and happy testing!

There are comments.


Page 1 / 13