atodorov.org

In the last few weeks I've been working together with Tim Flink and Kamil Paral from the Fedora QA team on bringing some installation testing expertise to Fedora and establishing an open source test lab to perform automated testing in. The infrastructure is already in relatively usable condition so I've decided to share this information with the community.

Beaker is Running Our Test Lab

Beaker is the software suite that powers the test lab infrastructure. It is quite complex, with many components and sometimes not very straight-forward to set up. Tim has been working on that with me giving it a try and reporting issues as they have been discovered and fixed.

In the process of working on this I've managed to create couple of patches against Beaker and friends. They are still pending release in a future version because of more urgent bug fixes which need to released first.

SNAKE is The Kickstart Template Server

SNAKE is a client/server Python framework used to support Anaconda installations. It supports plain text ks.cfg files, IIRC those were static templates, no variable substitution.

The other possibility is Python templates based on Pykickstart:

from pykickstart.constants import KS_SCRIPT_POST
from pykickstart.parser import Script
from installdefaults import InstallKs

def ks(**context):
    '''Anaconda autopart'''

    ks=InstallKs()
    ks.packages.add(['@base'])

    ks.clearpart(initAll=True)
    ks.autopart(autopart=True)

    script = '''
cp /tmp/ks.cfg /mnt/sysimage/root/ks.cfg || \
cp /run/install/ks.cfg /mnt/sysimage/root/ks.cfg
'''
    post = Script(script, type=KS_SCRIPT_POST, inChroot=False)
    ks.scripts.append(post)

    return ks

At the moment SNAKE is essentially abandoned but feature complete. I'm thinking about adopting the project just in case we need to make some fixes. Will let you know more about this when it happens.

Open Source Test Suite

I have been working on opening up several test cases for what we (QE) call a tier #1 installation test suite. They can be found in git. The tests are base on beakerlib and the legacy RHTS framework which is now part of Beaker.

This effort has been coordinated with Kamil as part of a pilot project he's responsible for. I've been executing the same test suite against earlier Fedora 20 snapshots but using an internal environment. Now everything is going out in the open.

Executing The Tests

Well you can't do that - YET! There are command line client tools for Fedora but Beaker and SNAKE are not well suited for use outside a restricted network like LAN or VPN. There are issues with authentication most notably for SNAKE.

At the moment I have to ssh through two different systems to get proper access. However this is been worked on. I've read about a rewrite which will allow Beaker to utilize a custom authentication framework like FAS for example. Hopefully that will be implemented soon enough.

I will also like to see the test systems have direct access to the Internet for various reasons but this is not without its risks either. This is still to be decided.

If you are interested anyway see the kick-tests.sh file in the test suite for examples and command line options.

Test Results

The first successfully completed test jobs are jobs 50 to 58. There's a failure in one of the test cases, namely SELinux related RHBZ #1027148.

From what I can tell the lab is now working as expected and we can start doing some testing against Fedora development snapshots.

Ping me or join #fedora-qa on irc.freenode.net if you'd like to join Fedora QA!

Posted by Alexander Todorov on Tue 19 November 2013

There are comments.

Today I was looking for a command sequence to cut a string in two by predefined delimiter (e.g. like cut does). I wanted to get the last field only and all fields but the last as separate variables.

The proposed solutions I've found suggested using awk but I don't like it. Here's a simple solution using cut and rev which can extract arbitrary field counts from the end of the string.

$ echo 'buildvm-08.phx2.fedoraproject.org' | rev | cut -f1 -d. | rev
org
$ echo 'buildvm-08.phx2.fedoraproject.org' | rev | cut -f-2 -d. | rev
fedoraproject.org
$ echo 'buildvm-08.phx2.fedoraproject.org' | rev | cut -f-3 -d. | rev
phx2.fedoraproject.org
$ echo 'buildvm-08.phx2.fedoraproject.org' | rev | cut -f2- -d. | rev
buildvm-08.phx2.fedoraproject
$ echo 'buildvm-08.phx2.fedoraproject.org' | rev | cut -f3- -d. | rev
buildvm-08.phx2
$ echo 'buildvm-08.phx2.fedoraproject.org' | rev | cut -f4- -d. | rev
buildvm-08

The magic here is done by rev which reverses the order of characters in every line. It comes with the util-linux-ng package.

Note to Self: util-linux-ng appears to contain more useful commands which I wasn't aware of. Need to RTFM a little bit.

Posted by Alexander Todorov on Tue 19 November 2013

There are comments.

Consider the following scenario:

• I'm using SNAKE templates as part of my installation testing work;
• SNAKE has a dependency on pykickstart;
• To test the latest and greatest kickstart features in Fedora we need the latest version of pykickstart;
• Latest pykickstart needs Python 2.7
• Python 2.7 is not available on RHEL 6 which is used to host the test infrastructure.

Just yesterday I hit an issue with the above setup and figured Fedora QA is in a kind of strange situation - we always need the latest but need it conservative enough to run on RHEL 6. See the original thread at kickstart-list.

In this particular case the solution will be to remove the offending code and implement the same functionality in backward-compatible manner. Also add more tests. I will be working on this tomorrow (there's an older patch already).

The BIG question remains though - how do you manage software evolution and still keep it compatible with older execution stacks? Please share your experience in the comments section.

PP: Spoiler - this is part of an ongoing effort to bring open source installation testing expertise (my domain) into Fedora world, plus establish a community supported test infrastructure. More info TBA soon.

Posted by Alexander Todorov on Wed 13 November 2013

There are comments.

image by opensource.com

Recently OpenSource.com published my first article User guide for open source project bug submissions which was inspired by real events.

The comments at the bottom are very interesting. It looks like Fedora needs to make some changes in the bug reporting process and tools.

I've started a discussion on Fedora QA mailing list about this. Follow the thread if you are interested in the topic.

Posted by Alexander Todorov on Fri 01 November 2013

There are comments.

HackFMI starts tonight! During the previous edition of the hackathon I gave one team a special personal gift based on my estimation of their hacker level! Thanks to Vihren Ganev (one of the gift winners) for reminding me about that!

This year I have even bigger items to give to those who think out of the box, who hack their way around and are brave enough to try and change the world through software! I have one Lenovo Thinkpad Business Backpack and one Lowepro D-Res 20 AW Digital Camera Bag (a bit bigger than the one on Amazon, easily fits a BlackBerry Z10) as seen on the pictures below. Plus some small Red Hat branded items.

How To Get The Gifts

There are no rules, I will make a decision as I go along. I can give away all items or none. It's up to you! Surprise me!

Tomorrow I will be mentoring the teams who decided to work with Django but I'm happy to talk to anyone who needs help or just wants to grab the cool stuff! See you there!

Posted by Alexander Todorov on Fri 25 October 2013

There are comments.

I was testing Fedora 20 inside a KVM guest this week when the disk space run out. The system was configured to use Btrfs filesystem and this is how to extend it.

First you have to extend the underlying guest storage. On the host I'm using LVM so this is a no brainer:

# pvs
  PV                                                    VG              Fmt  Attr PSize   PFree  
  /dev/mapper/luks-f3f6cea1-baba-4aaf-bca8-33a0ec540369 vg_redbull_mini lvm2 a--  289,11g 134,11g

# lvs
  LV            VG              Attr      LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  vm_fedora     vg_redbull_mini -wi-ao---  15,00g                                             

# lvextend -L +5G /dev/mapper/vg_redbull_mini-vm_fedora 
  Extending logical volume vm_fedora to 20,00 GiB
  Logical volume vm_fedora successfully resized

# lvs
  LV            VG              Attr      LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  vm_fedora     vg_redbull_mini -wi-ao---  20,00g                                             

# pvs
  PV                                                    VG              Fmt  Attr PSize   PFree  
  /dev/mapper/luks-f3f6cea1-baba-4aaf-bca8-33a0ec540369 vg_redbull_mini lvm2 a--  289,11g 129,11g

On the VM we have a default Btrfs layout:

# blkid
/dev/vda1: UUID="410ee563-e701-42ff-9d5f-5805dd103e35" TYPE="ext4" PARTUUID="0000330f-01" 
/dev/vda2: UUID="f4addad4-a0fc-482e-ad5a-240864b76f09" TYPE="swap" PARTUUID="0000330f-02" 
/dev/vda3: LABEL="fedora" UUID="f0b589ce-061c-4ac3-826e-7f3f8c8a6d30" UUID_SUB="11aa8414-3ce1-4fe7-a506-9a4f91ba5c30" TYPE="btrfs" PARTUUID="0000330f-03" 

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3        13G   11G  1.4G  89% /
devtmpfs        996M     0  996M   0% /dev
tmpfs          1002M   80K 1002M   1% /dev/shm
tmpfs          1002M  668K 1002M   1% /run
tmpfs          1002M     0 1002M   0% /sys/fs/cgroup
tmpfs          1002M   16K 1002M   1% /tmp
/dev/vda3        13G   11G  1.4G  89% /home
/dev/vda1       477M   72M  376M  17% /boot

Now power-off (not reboot) and power-on the VM guest so that it sees the new size of the underlying storage. See the fdisk header (line 9 below), vda is now 20GiB!

Before extending the filesystem you have to extend the underlying disk partition! This is the trickiest part. Using fdisk or parted you have to delete the partition and add it again. Make sure to use the SAME starting sector for the new partition (line 33)!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

# fdisk /dev/vda

Welcome to fdisk (util-linux 2.24-rc1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/vda: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0000330f

Device    Boot     Start       End   Blocks  Id System
/dev/vda1 *         2048   1026047   512000  83 Linux
/dev/vda2        1026048   5253119  2113536  82 Linux swap / Solaris
/dev/vda3        5253120  31457279 13102080  83 Linux

Command (m for help): d
Partition number (1-3, default 3): 3

Partition 3 is deleted

Command (m for help): n

Partition type:
   p   primary (2 primary, 0 extended, 2 free)
   e   extended
Select (default p): p
Partition number (3,4, default 3): 3
First sector (5253120-41943039, default 5253120): 
Last sector, +sectors or +size{K,M,G,T,P} (5253120-41943039, default 41943039): 

Created a new partition 3 of type 'Linux' and of size 17,5 GiB.

Command (m for help): p
Disk /dev/vda: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0000330f

Device    Boot     Start       End   Blocks  Id System
/dev/vda1 *         2048   1026047   512000  83 Linux
/dev/vda2        1026048   5253119  2113536  82 Linux swap / Solaris
/dev/vda3        5253120  41943039 18344960  83 Linux

Command (m for help): w

The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

# partprobe
Error: Partition(s) 3 on /dev/vda have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.

# reboot

See lines 36 and 49 above. The new partition has a greater size. After reboot just resize the filesystem and verify the new space has been added

# btrfs filesystem resize max /
Resize '/' of 'max'

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3        18G   11G  6.4G  63% /
devtmpfs        996M     0  996M   0% /dev
tmpfs          1002M   80K 1002M   1% /dev/shm
tmpfs          1002M  660K 1002M   1% /run
tmpfs          1002M     0 1002M   0% /sys/fs/cgroup
tmpfs          1002M   16K 1002M   1% /tmp
/dev/vda3        18G   11G  6.4G  63% /home
/dev/vda1       477M   72M  376M  17% /boot

This is it, more disk space available for the virtual machine. Let me know how it works for you.

Posted by Alexander Todorov on Sun 13 October 2013

There are comments.

Every once in a while ABRT will tell you that reporting is disabled because backtrace is unusable. What it means is that it can't read some of the debugging symbols and the most likely reason for that is debuginfo packages are missing.

To install them first locate the directory containing the files for that particular crash. Use the executable file to find out if you are looking into the correct directory. Then use this one liner to install the missing debuginfo packages.

# pwd
/var/tmp/abrt/ccpp-2013-10-10-15:55:18-15533
# cat backtrace | grep lib | tr -s ' ' | cut -f4 -d' ' | sort | uniq | grep "/" | xargs rpm -qf --qf "%{name}\n" | xargs debuginfo-install -y

Posted by Alexander Todorov on Sat 12 October 2013

There are comments.

Here is my summary of the second Fedora Test Day hosted at init Lab yesterday.

Local attendance was a total disaster, in fact I was testing once again by my own. This time there were more people in the lab, all busy with their daily routines and tasks. There were no people who came for the testing :(. I will have to try different venues in the future and see if the situation improves.

On the testing front I managed to score 5 bugs against GNOME and Fedora. You can see the other test results and bugs on the wiki.

There are two things I didn't like in particular

• GNOME 3 as well as its classic mode - simply not the environment I'm used to;
• Having to record test results in the wiki! I'm writing to the Fedora QA mailing list about that as we speak.

At one time I was engaged in a discussion about which Bulgarian keyboard layout should be the default in GNOME simply because of GNOME #709799. The default keyboard layout will be Bulgarian (traditional phonetic) aka bg+phonetic.

(16,13,26) rtcm: atodorov: are you bulgarian and/or live in bulgaria?
(16,14,20) rtcm: atodorov: if yes, I wanted to know which keyboard layout most people expect there to be the default
(16,16,34) atodorov: rtcm: I'm a Bulgarian, however I can't tell which one. Both Phonetic and standard (BDS) are common
(16,16,54) atodorov: a safe bet is to go with phonetic I guess. 
(16,19,46) rtcm: atodorov: can you tell me which one is it in XKB terms? is it "bg", "bg+bas_phonetic" or "bg+phonetic" ?
(16,20,51) rtcm: they're labeled as "Bulgarian", "Bulgarian (new phonetic)" and "Bulgarian (traditional phonetic)"
(16,21,30) atodorov: bg+phonetic is the traditional phonetic
(16,22,04) atodorov: bg labeled as "Bulgarian" is the standard one I guess. Here we call it BDS after the standardization institute
(16,22,34) atodorov: bg+bas_phonetic is created from the Bulgarian Academy of Science and is not very popular as far as I know. I've never seen it in use
(16,23,15) rtcm: atodorov: all I want to know is what most people would expect? like what does windows do by default? that's a good bet
(16,27,46) atodorov: rtcm: I'm just being told that new Windows releases use yet another layout by default, which is like phonetic but with some characters in new places and people don't like that
(16,27,53) atodorov: my safe bet goes to bg+phonetic
(16,29,05) rtcm: ok, thanks

It is a rare occasion when you get to make a decision that affects a large group of people and I hope you don't hate me for that!

Do you want to see more Fedora Test Days happening in Sofia? Join me and I will organize some more!

Posted by Alexander Todorov on Fri 11 October 2013

There are comments.

Here is a random collection of links I came across this week which appear interesting to me but I don't have time to blog about in details.

Making a Multi-branch Test Server for Python Apps

If you are wondering how to test different feature branches of your Python application but don't have the resources to create separate test servers this is for you: http://depressedoptimism.com/blog/2013/10/8/making-a-multi-branch-test-server!

Kudos to the python-django-bulgaria Google group for finding this link!

OpenSource.com Interview with Chris DiBona

Just read it at http://opensource.com/business/13/10/interview-chris-dibona.

I particularly like the part where he called open source "brutal".

You once called open source “brutal”. What did you mean by that?

...

I think that it is because open source projects are able to only work with the productive people and ignore everyone else. That behavior can come across as very harsh or exclusionary, and that's because it is that: brutally harsh and exclusionary of anyone who isn't contributing.

...

So, I guess what I'm saying is that survival of the fittest as practiced in the open source world is a pretty brutal mechanism, but it works very very well for producing quality software. Boy is it hard on newcomers though...

OpenShift Finally Introduces Environment Variables

Yes! Finally!

    rhc set-env VARIABLE1=VALUE1 -a myapp

No need for my work around anymore! I will give the new feature a go very soon.

Read more about it at the OpenShift blog: https://www.openshift.com/blogs/taking-advantage-of-environment-variables-in-openshift-php-apps.

Have you found anything interesting this week? Please share in the comments below! Thanks!

Posted by Alexander Todorov on Fri 11 October 2013

There are comments.

Here is a quick summary of the first Fedora Test Day in Sofia I hosted at init Lab today.

Attendance was quite poor, actually nobody else except me participated but almost nobody else visited the hackespace as well. I get that it is a working day and Test Days conflict with regular business hours but this is not going to change anyway. On the other hand where were all the freelancers and non-office job workers who usually hang around in the Lab? I have no idea!

On IRC there was much better activity, 5 or 6 people were testing across Asia, Europe and USA time zones. You can see the test results here. I've started filing quite a few bugs in the morning and continued well into the afternoon. I've managed to file a total of 10 bugs. Some of them were not related to virtualization and some of them turned out to be duplicates or not a bug. I even managed to file 2 duplicate bugs which likely have the same root cause myself :).

I've also experienced two bugs filed by other people: RHBZ #967371 for MATE desktop and RHBZ #1015636 for virt-manager's Save/Restore functionality.

I've tried ARM on x86_64 but that didn't get anywhere near a running system. I will make another post about ARM and what I've discovered there.

The one thing I liked is the test results application. It is not what I'm used to when dealing with RHEL, has far less features but is very fast and easy to use and suits the Test Days participants just fine. And is definitely much easier to use compared to filing results in the wiki.

Overall Fedora 20 virtualization status according to me is pretty good.

I hope to see more attendance on Thursday when we're going to test GNOME 3.10.

Posted by Alexander Todorov on Tue 08 October 2013

There are comments.

Fedora 20 Virtualization and GNOME test days will be tomorrow (8th Oct) and on Thursday (10th Oct)! Local community in Sofia will gather at init Lab! We start at 10:00 and everybody is welcome.

If you have no idea what I'm talking about check my previous posts and the announcement at init Lab's website:

• http://initlab.org/event/testing-fedora-20-virtualization-test-day
• http://initlab.org/event/gnome-test-day

See you there!

Posted by Alexander Todorov on Mon 07 October 2013

There are comments.

Does this look familiar to you ?

No? See SofiaValley and opensource.com.

Either this is a very common front-end mistake (I would blame CSS) or Facebook have screwed up their buttons.

Posted by Alexander Todorov on Mon 23 September 2013

There are comments.

As I mentioned earlier I will organize some Fedora testing in Sofia. Here's an outline of my talk this Saturday and a general guide for anyone who wants to participate.

What are Fedora Test Days

Fedora Test Days are day long events focused on testing a particular feature in the upcoming Fedora release. At the time of writing they are focused on Fedora 20.

What You Need Before Joining Fedora

• A bugzilla.redhat.com account;
• A Fedora account. Please complete the onboarding process in advance as it takes time;
• Create a wiki page for your profile. Mine is here. This will verify you can edit the wiki;
• Know how to use IRC. There is a web chat client at https://webchat.freenode.net/.

What You Need Before The Test Day

• Know how to properly report a bug;
• Installation ISO (DVD) or Live CD depending on what you want to test;
• Nightly LiveCD ISOs can be found here;
• Another alternative is official test compose (TC) images or yum update to the latest release;
• Extra hardware or a virtual machine for testing.

What You Need On The Test Day

• Join the #fedora-test-day IRC channel on freenode.net;
• Have the current test day wiki page handy;
• Execute some test cases and file bugs;
• Report your test results on the wiki;

Time and place for two test days will be announced at the end of the week and earlier next week so stay tuned!

Posted by Alexander Todorov on Mon 23 September 2013

There are comments.

To all my readers - sorry for not being able to blog more frequently lately. Here's an easy read about my favourite laptop brand Lenovo and some of their design flaws I've found.

X220 and T60 Batterries are ALMOST Identical

As you can see the X220 and T60 batteries are nearly identical with the notable exception of the connector placement. The end result - I have to purchase yet another battery as a backup for long travel/work on the go. Not what I want.

I wish all Lenovo models had the same batteries so people can swap them around as they wish. Is this too much to ask for? Have you seen another brand which got this right?

ThinkPad Mini Dock Design Flaw

I'm using a ThinkPad Mini Dock Series 3 docking station with my X220 laptop. Being a QA engineer for so long I immediately noticed something that wasn't quite right. The buttons on the left and the mechanism next to them are blocking the hot air exhaust from the CPU fan. This model of docking station is made to fit several models of laptops and those which dock in position 2 are less affected from those which dock in possition 1. Mine was not a lucky one.

On the pictures below it is clearly visible that most of the hot air coming out of the CPU fan is blocked.

In order to reduce laptop heating and provide better cooling I decided to remove the 1/2 position switch mechanism. To do that you have to unscrew all screws from the docking station and carefully split the top and bottom halves. The offending piece of plastic is screwed with two tiny screws at the bottom. Once they are removed everything comes off.

Even with this piece removed my laptop still hets up too much! I guess 80 C is just normal for the Core i7 processors :(.

Have you found something not quite right in your hardware design? Please share in the comments.

Posted by Alexander Todorov on Sun 15 September 2013

There are comments.

On September 28th I will be giving a short talk about Fedora Test Days at the regular Linux for Bulgarians conference. I will explain what these are and how one can participate. I will also announce my plans and schedule to organize some Fedora 20 test days locally in Sofia. If you are a fan of Fedora and want to file bugs and kick some developers' ass this is the way to do it!

Other talks include Alexander Shopov's „Oracle's take on NoSQL“ which I wanted to hear since this summer and TBA talks about MicroTik routers.

The conference will take place on September 28th at the French Institute at Sofia University (see map). It starts at 13:00 and my talk should be at the beginning. See you there!

PS: this post was initially written on my BlackBerry Z10 with Penzus Editor - another small step in retiring my laptop.

Posted by Alexander Todorov on Sat 14 September 2013

There are comments.

In my previous post I asked the readers of this blog to pick a bug number from Red Hat's Bugzilla so I can analyze it later.

Radoslav Georgiev decided to step up and selected the Leet bug https://bugzilla.redhat.com/show_bug.cgi?id=1337

This is a rather old bug against kernel, in particular against the token ring driver. There isn't much info on the bug but it seems the issue is hardware dependent and doesn't reproduce reliably.

Looking at the bug status and history it looks like it was closed without fixing it. Most likely the reason for this was there was no hardware to test, bug was not reproduced and no customers were seeing the issue or were willing to test and work with devel!

If you'd like to see my comments on other interesting bugs just post a link to them in the comments section.

Posted by Alexander Todorov on Mon 02 September 2013

There are comments.

Red Hat's Bugzilla passed the 1 million bugs milestone yesterday! RHBZ #1000000 has been filed by Anton Arapov, a kernel engineer and a very nice guy (I know him btw). I've filed several bugs yesterday but the last one was #999941. A bit too short!

To celebrate this event I dare you to pick some bugs from Bugzilla that you find interesting or frustrating and I will try to analyze and explain them from a QA engineer's point of view. Since I've reported over 1000 bugs and been involved in another close to 5000 I think I will be able to answer almost any question.

Challenge accepted!

Posted by Alexander Todorov on Fri 23 August 2013

There are comments.

I've been having troubles with Twitter lately but this bug is just annoying! Where on Earth is the "code below" ? I've already reported it. Let's see how long it takes for them to fix it!

Posted by Alexander Todorov on Fri 23 August 2013

There are comments.

One thing you need to know when ordering Red Hat subscriptions is your account number. It is available in the drop down menu at the top right corner after you login at https://access.redhat.com.

New accounts don't have an account number. It will be generated after the first subscription activation.

If you need to know more about Red Hat's ordering process just ask me!

Posted by Alexander Todorov on Fri 23 August 2013

There are comments.

Back in February 2013 Russell Beattie has published an article on his blog titled Artisanal Websites: The rise of .io domains for well crafted web services . He has compiled a great deal of sites in the .io domain space with screenshots and links to each individual service. There are lots of interesting ones, so go and check them out!

I'm glad Russell has added Difio as well!

Posted by Alexander Todorov on Tue 13 August 2013

There are comments.