At the beginning of this year I've hosted the first QA related course at HackBulgaria. This is a long overdue post about how the course went, what worked well and what didn't. Brace yourself because it is going to be a long one.
The idea behind a QA course has been lurking in both RadoRado's (from HackBulgaria) and my heads for quite a while. We've been discussing it at least a year before we actually started. One day Rado told me he'd found a sponsor and we have the go ahead for the course and that's how it all started!
The first issue was that we weren't prepared to start at a moments notice. I literally had two weeks to prepare the curriculum and initial interview questions. Next we opened the application form and left it open until the last possible moment. I've been reviewing candidate answers hours before the course started, which was another mistake we made!
On the positive side is that I hosted a Q&A session on YouTube answering general questions about the profession and the course itself. This live stream helped popularize the course.
At the start we had 30 people and around 13 of them managed to "graduate" till the final lesson. The biggest portion of students dropped out after the first 5 lessons of Java crash course! Each lesson was around 4 hours with 20-30 minutes break in the middle.
With respect to the criteria find a first job or find a new/better job I consider the training successful. To my knowledge all students have found better jobs, many of them as software testers!
On the practical side of things students managed to find and report 11 interesting bugs against Fedora. Mind you that these were all found in the wild: fedora-infrastructure #5323, RHBZ#1339701, RHBZ#1339709, RHBZ#1339713, RHBZ#1339719, RHBZ#1339731, RHBZ#1339739, RHBZ#1339742, RHBZ#1339746, RHBZ#1340541, RHBZ#1340891.
Then students also made a few pull requests on GitHub (3 which I know off): commons-math #38, commons-csv #12, commons-email #1.
Lesson format
For reference most lessons were a mix of short presentation about theory and best practices followed by discussions and where appropriate practical sessions with technology or projects. The exercises were designed for individual work, work in pairs or small groups (4-5) on purpose.
By request from the sponsors I've tried to keep a detailed record of each student's performance and personality traits as much as I was able to observe them. I really enjoyed keeping such a journal but didn't share this info with my students which I consider a negative issue. I think knowing where your strong and weak areas are would help you become a better expert in your field!
Feedback from students
- There was little time (to work on the practical examples I guess);
- Not having particular practical tasks was a problem. For example we didn't have tasks of the sort do "X" then "Y";
- They needed more time and more attention to be given to them;
- Installing different pieces of software and tools took a lot of time and frustration. It was also quite problematic sometimes depending on whether they used Linux, Windows or Mac OS X;
- Working with Eclipse IDE was horrible. Nobody new what to do and the interface wasn't newbie friendly. Also it took quite a lot of time to install dependencies and/or import projects to run in Eclipse;
- There were a few problems with Selenium and its different versions being used.
I have to point out that while these are valid concerns and major issues students were at least partially guilty for the last 3 of them. It was my impression that most of them didn't prepare at home, didn't read the next lesson and didn't install prerequisite tools and software!
5x Java Crash Course
We've started with a Java crash course as requested by our sponsors which was extended to 5 instead of the original 3 sessions. RadoRado was teaching Java fundamentals while I was assisting him with comments.
On the good side is that Rado explains very well and in much details. He also writes code to demonstrate what he teaches and while doing so uses only the knowledge he's presented so far. For example if there's a repeating logic/functionality he would just write it twice instead of refactoring that into a separate function with parameters (assuming the students have not learned about functions yet). I think this made it more easier to understand the concepts being taught.
Another positive thing we did was me going behind Rado's computer and modifying some of the code while he was explaining something on screen. If you take the above example and have two methods with print out salutations, e.g. "Good morning, Alex" I would go and modify one of them to include "Mr." while the other will not. This introduced a change in behavior which ultimately results in a bug! This was a nice practical way to demonstrate how some classes of bugs get introduced in reality. We did only a few of these behind the computer changes and I definitely liked them! They were all ad-hoc, not planned for.
On the negative side Java seems hard to learn and after these 5 lessons half of the students dropped out. Maybe part of the reason is they didn't expect to start a QA course with lessons about programming. But that also means they didn't pay enough attention to the curriculum, which was announced in advance!
Lesson 01 - QA Fundamentals
I had made a point to assign time constraints to each exercise in the lessons. While that mostly worked in the first few lessons, where there is more theory, we didn't keep the schedule and were overtime.
Explaining testing theory (based on ISTQB fundamentals) took longer than I expected. it was also evident that we needed more written examples of what different test analysis techniques are (e.g. boundary value analysis). Here Petar Sabev helped me deliver few very nice examples.
One of the exercises was "when to stop testing" with an example of a Sudoku solving function and different environments in which this code operates, e.g. browser, mobile, etc. Students appeared to have a hard time understanding what a "runtime environment" is and define relevant tests based on that! I believe most of the students, due to lack of knowledge and experience, were also having a hard time grasping the concept of non-functional testing.
A positive thing was that students started explaining to one another and giving examples for bugs they've seen outside the course.
Lesson 02 - Software Development Lifecycle
This lesson was designed as role playing game to demonstrate the most common software development methodologies - waterfall and agile and discuss the QA role in both of them. The format by itself is very hard to conduct successfully and this was my first time ever doing this. I've also never taken part of such games until then, only heard about them.
During the waterfall exercise it was harder for the students to follow the game constraints and not exchange information with one another because they were sitting on the same table.
On the positive side all groups came with unique ideas about software features and how they want to develop them. Timewise we managed to do very well. On the negative side is that I was the client for all groups and didn't manage to pay enough attention to everyone, which btw is what clients usually do in real life.
Lesson 03 - Bug Tracking
This lesson was a practical exercise in writing bug reports and figuring out what information needs to be present in a good bug report. Btw this is something I always ask junior members at job interviews.
First we started with working in pairs to define what a good bug report is without actually knowing what that means. Students found it hard to brainstorm together and most of them worked alone during this exercise.
Next students had to write bug reports for some example bugs, which I've explained briefly on purpose and perform peer reviews of their bugs. Reviews took a long time to complete but overall students had a good idea of what information to include in a bug report.
Then, after learning from their mistakes and hearing what others had done, they've learned about some good practices and were tasked to rewrite their bug reports using the new knowledge. I really like the approach of letting students make some mistakes and then showing them the easier/better way of doing things. This is also on-par with Ivan Nemytchenko's methodology of letting his interns learn by their mistakes.
All bug reports can be found in students repositories, which are forked from the curriculum. Check out https://github.com/HackBulgaria/QA-and-Automation-101/network.
I should have really asked everyone to file bugs under the curriculum repository so it is easier for me to track them. On the other hand I wanted each student to start building their own public profile to show potential employers.
Lesson 04 - Test Case Management
This lesson started with an exercise asking students to create accounts for Red Hat's OpenShift cloud platform in the form of a test scenario. The scenario intentionally left out some details. The idea being that missing information introduces inconsistencies during testing and to demonstrate how the same steps were performed slightly differently.
We had some troubles explaining exactly "how did you test" because most inexperienced people would not remember all details about where they clicked, did they use the mouse or the keyboard, was the tab order correct, etc. Regardless students managed to receive different results and discover that some email providers were not supported.
The homework assignment was to create test plans and test cases in Nitrate at https://nitrate-hackbg.rhcloud.com/. Unfortunately the system appears to be down ATM and I don't have time to investigate why. This piece of infrastructure was put together in 2 hours and I'm surprised it lasted without issues during the entire course.
2x Introduction to Linux
This was a crash course in Linux fundamentals and exercise with most common commands and text editors in the console. Most of the students were not prepared with virtual machines. We've also used a cloud provider to give students remote shell but the provider API was failing and we had to deploy docker containers manually. Overall infrastructure was a big problem but we somehow managed. Another problem was with ssh clients from Windows who generated keys in a format that our cloud provider couldn't understand.
Wrt commands and exercises students did well and managed to execute most of them on their own. That's very good for people who've never seen a terminal in their lives (more or less).
Lesson 05 - Testing Fedora 24(25) Changes
Once again nobody was prepared with a virtual machine with Fedora and students were installing software as we go. Because of that we didn't manage to conduct the lesson properly and had to repeat it on the next session.
Rawhide being the bleeding edge of Fedora means it is full of bugs. Well I couldn't keep up with everyone and explain workarounds or how to install/upgrade Fedora. That was a major setback. It also became evident that you can't move quickly if you have no idea what to do and no instructions about it either.
Lesson 05 - Again
Once prepared with the latest and greatest from Rawhide the task was to analyze the proposed feature changes (on the Fedora wiki) and create test plans and design test cases for said changes. Then execute the tests in search for bugs. This is where some of the bugs above came from. The rest were found during upgrades.
This lesson was team work (4-5 students) but the results were mixed. IMO Fedora changes are quite hard to grasp, especially if you lack domain knowledge and broader knowledge about the structure and operation of a Linux distribution. I don't think most teams were able to clearly understand their chosen features and successfully create good plans/scenarios for them. On the other hand in real life software you don't necessarily understand the domain better and know what to do. I've been in situations where whole features have been defined by a single sentence and requested to be tested by QA.
One of the teams didn't manage to install Fedora (IIRC they didn't have laptops capable of running a VM) and were not able to conduct the exercise.
Being able to find real life bugs, some of them serious, and getting traction in Bugzilla is the most positive effect of this lesson. I personally wanted to have more output (e.g. more bugs, more cases defined, etc) but taking into account the blocking factors and setbacks I think this is a good initial result.
Lesson 06 - Unit Testing and Continuous Integration
Here we had a few examples of bad stubs and mocks which were not received very well. The topic is hard in itself and wasn't very well explained with practical examples.
Another negative thing is that students took a lot of time to fiddle around with Eclipse, they were mistyping commands in the terminal and generally not paying enough attention to instructions. This caused the exercises to go slowly.
We've had an exercise which asks the student to write a new test for a non-existing method. Then implement the method and make sure all the tests passed. You guessed it this is Test Driven Development. IIRC one of the students was having a hard time with that exercise so I popped up my editor on the large screen and started typing what she told me, then re-running the tests and asking her to show me the errors I've made and tell me how to correct them. The exercise was received very well and was fun to do.
Due to lack of time we had to go over TravisCI very quickly. The other bad thing about TravisCI is that it requires git/GitHub and the students were generally inexperienced with that. Both GitHub for Windows and Mac OS suck a big time IMO. What you need is the console. However none of the students had any practical experience with git and knew how to commit code and push branches to GitHub. git fundamentals however is a separate one or two lessons by itself which we didn't do.
Lesson 07 - Writing JUnit tests for Apache Commons
Excluding the problems with Eclipse and the GitHub desktop client and missing instructions for Windows the hardest part of this lesson was actually selecting a component to work on, understanding what the code does and actually writing meaningful tests. On top of that most students were not very proficient programmers and Java was completely new to them.
Despite having 3 pull requests on GitHub I consider this lesson to be a failure.
Lesson 08 - Integration Testing with Selenium
This lesson starts with an example of what a flaky test it. At the moment I don't think this lesson is the best place for that example. To make things even more difficult the example is in Python (because that way was the easiest for me to write it) instead of Java. Students had problems installing Python on Windows just to make this example work. They also were lacking the knowledge how to execute a script in the terminal.
One of the students proposed a better flaky example utilizing dates and times and executing it during various hours of the day. I have yet to code this and prepare environment in which it would be executed. Btw recently I've seen similar behavior caused by inconsistent timezone usage in Ruby which resulted in unexpected time offset a little after midnight :).
Once again I have to point out that students came generally unprepared for the lesson and haven't installed prerequisite software and programming languages. This is becoming a trend and needs to be split out into a preparation session, possibly with a check list.
On the Selenium side, starting with Selenium IDE, it was a bit unclear how to use it and what needs to be done. This is another negative trend, where students were missing clear instructions what they are expected to do. At the end we did resort to live demo using Selenium IDE so they can at least get some idea about it.
Lesson 09 and 10 - Writing Selenium tests for Mozilla Add-ons website
IMO these two lessons are the biggest disaster of the entire course. Python & virtualenv on Windows was a total no go but on Linux things weren't much easier because students had no idea what a virtualenv is.
Practice wise they haven't managed to read all the bugs on the Mozilla bug tracker and had a very hard time selecting bugs to write tests for. Not to mention that many of the reported bugs were administrative tasks to create or remove add-on categories. There weren't many functional related bugs to write tests for.
The product under test was also hard to understand and most students were seeing it for the first time, let alone getting to know the devel and testing environments that Mozilla provides. Mozilla's test suite being in Python is just another issue to make contribution harder because we've never actually studied Python.
Between the two lessons there were students who've missed the Selenium introduction lesson and were having even harder time to figure things out. I didn't have the time to explain and go back to the previous lesson for them. Maybe an attendance policy is needed for dependent lessons.
Before the course started I've talked to some guys at Mozilla's IRC channel and they agreed to help but at the end we didn't even engage with them. At this point I'm skeptical that mentoring over IRC would have worked anyway.
Lesson 11 - Introduction to Performance Testing
This was a more theoretical lesson with less practical examples and exercises. I have provided some blog posts of mine related to the topic of performance testing but in general they are related to software that I've used which isn't generally known to a less experienced audience (Celery, Twisted). These blog posts IMO were hard to put into perspective and didn't serve a good purpose as examples.
The practical part of the lesson was a discussion with the goal of creating a performance testing strategy for GitHub's infrastructure. It was me who was doing most of the talking because students have no experience working on such a large infrastructure like GitHub and didn't know what components might be there, how they might be organized (load balancers, fail overs, etc) and what needs to be tested.
There was also a more practical example to create a performance test in Java for
one of the classes found in commons-codec/src/main/java/org/apache/commons/codec/digest.
Again the main difficulty here was working fluently with Eclipse, getting the projects to
build/run and knowing how the software under test was supposed to work and be executed.
Lesson 12 - How to find 1000 bugs in 30 minutes
This was a more relaxed lesson with examples of simple types of bugs found on a large scale. Most examples came from my blog and experiments I've made against Fedora.
While amusing and fun I don't think all of the students understood me and kept their attention. Part of that is because Fedora tends to focus on low level stuff and my examples were not necessarily easy to understand.
Feedback from the sponsor
Experian Bulgaria was the exclusive sponsor for this course. At the end of the summer Rado and I met with them to discuss the results of the training. Here's what they say
- Overall technical knowledge they consider to be weak. What they need are people with basic knowledge in the field of object oriented programming, databases, operating systems and networking. These are skills candidates need in order to work for Experian. It must be noted that most of them were not taught at this course!
- English language proficiency for all students is low on average;
- User level experience with Linux was fine but students were missing deeper knowledge about the operating system. Once again something we didn't teach;
- The programming languages favored at Experian are Java and Ruby and students had poor knowledge of them. They also had weak understanding of OOP principles;
- According to Experian students were more ingrained with the development mindset and were trying their luck in the QA profession instead of genuinely being interested in the field. While this is generally the case I have to point out that as sponsor Experian did a poor job at promoting their company and the QA profession as a whole. What isn't known to the general public is their big in-house QA community which could have served as a source of inspiration for the students!
- Low motivation and lack of fundamental knowledge from university are other traits interviewers at Experian have identified. They argued for a stronger acceptance process and a requirement of minimum 2 years of university education in computer science.
On the topic of testing knowledge candidates did mostly OK, however we don't have enough information about this. Also the hiring process at Experian is more focused on the broader knowledge areas listed above so substantial improvement in the testing knowledge of candidates doesn't given them much head start.
While to my knowledge they didn't hire anyone few people received an offer but declined due to various personal reasons. I view this as poor performance on our side but Experian thinks otherwise and are willing to sponsor another round of training.
Summary
Here is a list of all the things that could be improved
- Take time to develop the curriculum and have it pass QA review by other experienced testers (in particular such that also teach students);
- Make the application process harder to include people with broader IT knowledge;
- Allow time to review all applicants;
- Find a co-trainer and additional mentors for some of the exercises;
- Minimize the number of students accepted in the course so we can handle everyone with more care;
- Give homework assignments and examine them before each lesson;
- Collect and provide performance review at the end of the course;
- Minimize the technology stack and tools used;
- Host a technology preparation session at the beginning paired with a check list of all the things that need to be done. Very likely make the chosen technology stack a hard requirement to avoid setbacks later in the schedule;
- We need more fundamental lessons like practical git tutorial, practical Linux tutorial, databases, networking, etc.
- Same goes for Java or any other programming language. Each of these technologies easily makes a course by itself. Possibly include technology skill assessment in the application form and reject candidates who don't meet the minimum level. This is to ensure the group is able to move at the same pace.
- We need to improve the infrastructure used during the course, especially exercise bug tracking and test case management systems;
- Students need clear instructions for every exercise - what is required from them, what the expected result is and what they need to be doing, etc;
- Provide more examples for just about everything. Also make the examples easier to understand/simple with the harder examples left for further reading;
- Provide easier projects to work on. This means applications whose domain is easier to understand and closer to what experiences the students might have. Also projects need clear instructions how to join and how to contribute with tests.
- The same application/software needs to be used for manual bug finding, unit test writing and integration test with Selenium. This will minimize both context and technology switches and allow to view various testing activities in the context of a single software under test;
- While using open source projects for the above listed purposes sounds great the reality is that they are hard to join and will not move according to our schedule. Maybe less focus on open source per-se and more focus on a particular application under test. Instructors can then proxy all the tests forward to the upstream community;
- Focus more on practice and less on exotic topics like performance testing and large scale bug finding;
- Seek more active participation from sponsors!
If you have suggestions please comment below, especially if you can tell me how to implement them in practice.
Thanks for reading and happy testing!
Comments !