Tag Django

SMS PIN Verification with Twilio and Django

This is a quick example of SMS PIN verification using Twilio cloud services and Django which I did for a small site recently.

SMS PIN form

The page template contains the following HTML snippet

<input type="text" id="mobile_number" name="mobile_number" placeholder="111-111-1111" required>
<button class="btn" type="button" onClick="send_pin()"><i class="icon-share"></i> Get PIN</button>

and a JavaScript function utilizing jQuery:

function send_pin() {
    $.ajax({
                url: "{% url 'ajax_send_pin' %}",
                type: "POST",
                data: { mobile_number:  $("#mobile_number").val() },
            })
            .done(function(data) {
                alert("PIN sent via SMS!");
            })
            .fail(function(jqXHR, textStatus, errorThrown) {
                alert(errorThrown + ' : ' + jqXHR.responseText);
            });
}

Then create the following views in Django:

def _get_pin(length=5):
    """ Return a numeric PIN with length digits """
    return random.sample(range(10**(length-1), 10**length), 1)[0]


def _verify_pin(mobile_number, pin):
    """ Verify a PIN is correct """
    return pin == cache.get(mobile_number)


def ajax_send_pin(request):
    """ Sends SMS PIN to the specified number """
    mobile_number = request.POST.get('mobile_number', "")
    if not mobile_number:
        return HttpResponse("No mobile number", mimetype='text/plain', status=403)

    pin = _get_pin()

    # store the PIN in the cache for later verification.
    cache.set(mobile_number, pin, 24*3600) # valid for 24 hrs

    client = TwilioRestClient(settings.TWILIO_ACCOUNT_SID, settings.TWILIO_AUTH_TOKEN)
    message = client.messages.create(
                        body="%s" % pin,
                        to=mobile_number,
                        from_=settings.TWILIO_FROM_NUMBER,
                    )
    return HttpResponse("Message %s sent" % message.sid, mimetype='text/plain', status=200)

def process_order(request):
    """ Process orders made via web form and verified by SMS PIN. """
    form = OrderForm(request.POST or None)

    if form.is_valid():
        pin = int(request.POST.get("pin", "0"))
        mobile_number = request.POST.get("mobile_number", "")

        if _verify_pin(mobile_number, pin):
            form.save()
            return redirect('transaction_complete')
        else:
            messages.error(request, "Invalid PIN!")
    else:
        return render(
                    request,
                    'order.html',
                    {
                        'form': form
                    }
                )

PINs are only numeric and are stored in the CACHE instead of the database which I think is simpler and less expensive in terms of I/O operations.

There are comments.

Mocking Django AUTH_PROFILE_MODULE without a Database

Difio is a Django based service which uses a profile model to provide site-specific, per-user information. In the process of open sourcing Difio its core functionality becomes available as a Django app. The trouble is that the UserProfile model contains site-specific and proprietary data which doesn't make sense to the public nor I want to release it.

The solution is to have a MockProfile model and work with that by default while www.dif.io and other implementations override it as needed. How do you do that without creating useless table and records in the database but still have the profiles created automatically for every user?

It turns out the solution is quite simple. See my comments inside the code below.

class AbstractMockProfile(models.Model):
    """
        Any AUTH_PROFILE_MODULE model should inherit this
        and override the default methods.

        This model provides the FK to User!
    """
    user = models.ForeignKey(User, unique=True)

    def is_subscribed(self):
        """ Is this user subscribed for our newsletter? """
        return True

    class Meta:
        # no DB table created b/c model is abstract
        abstract = True

class MockProfileManager(models.Manager):
    """
        This manager creates MockProfile's on the fly without
        touching the database. It is needed by User.get_profile()
        b/c we can't have an abstract base class as AUTH_PROFILE_MODULE.
    """
    def using(self, *args, **kwargs):
        """ It doesn't matter which database we use! """
        return self

    def get(self, *args, **kwargs):
        """
            User.get_profile() calls .using(...).get(user_id__exact=X)
            so we instrument it here to return a MockProfile() with
            user_id=X parameter. Anything else may break!!!
        """
        params = {}
        for p in kwargs.keys():
            params[p.split("__")[0]] = kwargs[p]

        # this creates an object in memory. To save it to DB
        # call obj.save() which we DON'T do anyway!
        return MockProfile(params)


class MockProfile(AbstractMockProfile):
    """
        In-memory (fake) profile class used by default for
        the AUTH_PROFILE_MODULE setting.
    """
    objects = MockProfileManager()

    class Meta:
        # DB table is NOT created automatically
        # when managed = False
        managed = False

In Difio core the user profile is always used like this

profile = request.user.get_profile()
if profile.is_subscribed():
    pass

and by default

AUTH_PROFILE_MODULE = "difio.MockProfile"

Voila!

There are comments.

Skip or Render Specific Blocks from Jinja2 Templates

I wasn't able to find detailed information on how to skip rendering or only render specific blocks from Jinja2 templates so here's my solution. Hopefully you find it useful too.

With below template I want to be able to render only kernel_options block as a single line and then render the rest of the template excluding kernel_options.

base.j2
{% block kernel_options %}
console=tty0
    {% block debug %}
        debug=1
    {% endblock %}
{% endblock kernel_options %}

{% if OS_MAJOR == 5 %}
key --skip
{% endif %}

%packages
@base
{% if OS_MAJOR > 5 %}
%end
{% endif %}

To render a particular block you have to use the low level Jinja API template.blocks. This will return a dict of block rendering functions which need a Context to work with.

The second part is trickier. To remove a block we have to create an extension which will filter it out. The provided SkipBlockExtension class does exactly this.

Last but not least - if you'd like to use both together you have to disable caching in the Environment (so you get a fresh template every time), render your blocks first, configure env.skip_blocks and render the entire template without the specified blocks.

jinja2-render
#!/usr/bin/env python

import os
import sys
from jinja2.ext import Extension
from jinja2 import Environment, FileSystemLoader


class SkipBlockExtension(Extension):
    def __init__(self, environment):
        super(SkipBlockExtension, self).__init__(environment)
        environment.extend(skip_blocks=[])

    def filter_stream(self, stream):
        block_level = 0
        skip_level = 0
        in_endblock = False

        for token in stream:
            if (token.type == 'block_begin'):
                if (stream.current.value == 'block'):
                    block_level += 1
                    if (stream.look().value in self.environment.skip_blocks):
                        skip_level = block_level

            if (token.value == 'endblock' ):
                in_endblock = True

            if skip_level == 0:
                yield token

            if (token.type == 'block_end'):
                if in_endblock:
                    in_endblock = False
                    block_level -= 1

                    if skip_level == block_level+1:
                        skip_level = 0


if __name__ == "__main__":
    context = {'OS_MAJOR' : 5, 'ARCH' : 'x86_64'}

    abs_path  = os.path.abspath(sys.argv[1])
    dir_name  = os.path.dirname(abs_path)
    base_name = os.path.basename(abs_path)

    env = Environment(
                loader = FileSystemLoader(dir_name),
                extensions = [SkipBlockExtension],
                cache_size = 0, # disable cache b/c we do 2 get_template()
            )

    # first render only the block we want
    template = env.get_template(base_name)
    lines = []
    for line in template.blocks['kernel_options'](template.new_context(context)):
        lines.append(line.strip())
    print "Boot Args:", " ".join(lines)
    print "---------------------------"

    # now instruct SkipBlockExtension which blocks we don't want
    # and get a new instance of the template with these blocks removed
    env.skip_blocks.append('kernel_options')
    template = env.get_template(base_name)
    print template.render(context)
    print "---------------------------"

The above code results in the following output:

$ ./jinja2-render ./base.j2 
Boot Args: console=tty0 debug=1 
---------------------------

key --skip

%packages
@base
---------------------------

Teaser: this is part of my effort to replace SNAKE with a client side kickstart template engine for Beaker so stay tuned!

There are comments.

Django Template Tag Inheritance How-to

While working on open-sourcing Difio I needed to remove all hard-coded URL references from the templates. My solution was to essentially inherit from the standard {% url %} template tag. Here is how to do it.

Background History

Difio is not hosted on a single server. Parts of the website are static HTML, hosted on Amazon S3. Other parts are dynamic - hosted on OpenShift. It's also possible but not required at the moment to host at various PaaS providers for redundancy and simple load balancing.

As an easy fix I had hard-coded some URLs to link to the static S3 pages and others go link to my PaaS provider.

I needed a simple solution which can be extended to allow for multiple domain hosting.

The Solution

The solution I came up with is to override the standard {% url %} tag and use it everywhere in my templates. The new tag will produce absolute URLs containing the specified protocol plus domain name and view path. For this to work you have to inherit the standard URLNode class and override the render() method to include the new values.

You also need to register a tag method to utilize the new class. My approach was to use the existing url() method to do all background processing and simply casting the result object to the new class.

All code is available at https://djangosnippets.org/snippets/3013/.

To use in your templates simply add

{% load fqdn_url from fqdn_url %}
<a href="{% fqdn_url 'dashboard' %}">Dashboard</a>

There are comments.

Idempotent Django Email Sender with Amazon SQS and Memcache

Recently I wrote about my problem with duplicate Amazon SQS messages causing multiple emails for Difio. After considering several options and feedback from @Answers4AWS I wrote a small decorator to fix this.

It uses the cache backend to prevent the task from executing twice during the specified time frame. The code is available at https://djangosnippets.org/snippets/3010/.

As stated on Twitter you should use Memcache (or ElastiCache) for this. If using Amazon S3 with my django-s3-cache don't use the us-east-1 region because it is eventually consistent.

The solution is fast and simple on the development side and uses my existing cache infrastructure so it doesn't cost anything more!

There is still a race condition between marking the message as processed and the second check but nevertheless this should minimize the possibility of receiving duplicate emails to an accepted level. Only time will tell though!

There are comments.

Django Tips: Using Cache for Stateful HTTP

How do you keep state when working with a stateless protocol like HTTP? One possible answer is to use a cache back-end.

I'm working on an IVR application demo with Django and Twilio. The caller will make multiple choices using the phone keyboard. All of this needs to be put together and sent back to another server for processing. In my views I've added a simple cache get/set lines to preserve the selection.

Here's how it looks with actual application logic omitted

def incoming_call(request):
    call_sid = request.GET.get('CallSid', '')
    caller_id = request.GET.get('From', '')

    state = {'from' : caller_id}
    cache.set(call_sid, state)

    return render(request, 'step2.xml')

def step2(request):
    call_sid = request.GET.get('CallSid', '')
    selection = int(request.GET.get('Digits', 0))

    state = cache.get(call_sid, {})
    state['step2_selection'] = selection
    cache.set(call_sid, state)

    return render(request, 'final_step.xml')


def final_step(request):
    call_sid = request.GET.get('CallSid', '')
    selection = int(request.GET.get('Digits', 1))

    state = cache.get(call_sid, {})
    state['final_step_selection'] = selection

    Backend.process_user_selections(state)

    return render(request, 'thank_you.xml')

At each step Django will update the current state associated with this call and return a TwiML XML response. CallSid is a handy unique identifier provided by Twilio.

I am using the latest django-s3-cache version which properly works with directories. When going to production that will likely switch to Amazon ElastiCache.

There are comments.

Using Django built-in template tags and filters in code

In case you are wondering how to use Django's built-in template tags and filters in your source code, not inside a template here is how:

>>> from django.template.defaultfilters import *
>>> filesizeformat(1024)
u'1.0 KB'
>>> filesizeformat(1020)
u'1020 bytes'
>>> filesizeformat(102412354)
u'97.7 MB'
>>>

All built-ins live in pythonX.Y/site-packages/django/template/defaultfilters.py.

There are comments.

Tip: Renaming Model Fields in Django

Did you ever have to re-purpose a column in your database schema? Here's a quick and easy way to do this if you happen to be using Django.

Scenario

I had an integer field in my model called lines which counted the lines of code in a particular tar.gz package. I figured the file size is a better indicator so decided to start using it. I was not planning to use the old field anymore and I didn't care about the data it was holding. So I decided to re-purpose it as the size field.

Possible methods

Looking around I figured several different ways to do this:

  1. Continue using the existing lines field and keep referencing the old name in the code. This is no-brainer but feels awkward and is a disaster waiting to happen;
  2. Add new size field and remove the old lines field. This involves modification to the DB schema and requires at least a backup with possible down time. Not something I will jump at right away;
  3. Add a size property in the model class which will persist to self.lines. This is a quick way to go but I'm not sure if one can use the property with the Django QuerySet API (objects.filter(), objects.update(), etc.) I suspect not. If you don't filter by the property or use it in bulk operations this method is fine though;
  4. Change the field name to size but continue to use the lines DB column; Mind my wording here :);
  5. Rename the column in the DB schema and then update the model class field.

How I did it

I decided to go for option 4 above: change the field name to size but continue to use the lines DB column.

diff --git a/models.py b/models.py
index e06d2b2..18cad6f 100644
--- a/models.py
+++ b/models.py
@@ -667,7 +667,7 @@ class Package(models.Model):
-    lines = models.IntegerField(default=None, null=True, blank=True)
+    size  = models.IntegerField(default=None, null=True, blank=True, db_column='lines')
  1. Removed all references to lines from the code except the model class. This served as clean up as well.
  2. Renamed the model field to size but continued using the lines DB column as shown above. Django's db_column option makes this possible.
  3. From the Django shell (./manage.py shell) reset size to None (NULL) for all objects;
  4. Finally implement my new code and functionality behind the size field.

The entire process happened for under 10 minutes. I will also opt for renaming the DB column at a later time. This is to sync the naming used in Python code and in MySQL in case I ever need to use raw SQL or anything but Django.

If you were me, how would you do this? Please share in the comments below.

There are comments.

Django QuerySet tip - Search and Order By Exact Match

How do you order Django QuerySet results so that first item is the exact match if using contains or icontains ? Both solutions were proposed on the django-users mailing list.

Solution by Tom Evans, example is mine:

>>> from django.db.models import Q
>>> Package.objects.filter(
        Q(name=Django) | Q(name__icontains=Django)
    ).extra(
        select={'match' : 'name = "Django"'}
    ).order_by('-match', 'name')
[<Package: Django>, <Package: appomatic_django_cms>, <Package: appomatic_django_filer>,
<Package: appomatic_django_vcs>, <Package: BabelDjango>, <Package: BDD4Django>,
<Package: blanc-django-admin-skin>, <Package: bootstrap-django-forms>,
<Package: capistrano-django>, <Package: ccnmtldjango>, <Package: collective.django>,
<Package: csdjango.contactform>, <Package: cykooz.djangopaste>,
<Package: cykooz.djangorecipe>, <Package: d51.django.virtualenv.test_runner>,
<Package: django-4store>, <Package: django-503>, <Package: django-absolute>,
<Package: django-abstract-templates>, <Package: django-account>,
'...(remaining elements truncated)...']
>>>

Another one:

I'm not sure this is the right way, but you could drop the Q objects, use only icontains and sort by the length of 'name'

Gabriel https://groups.google.com/d/topic/django-users/OCNmIXrRgag/discussion

>>> packages = [p.name for p in Package.objects.filter(name__icontains='Dancer')]
>>> sorted(packages, key=len)
[u'Dancer', u'Dancer2', u'breakdancer', u'Task::Dancer', u'App::Dancer2', u'Dancer::Routes',
u'DancerX::Routes', u'DancerX::Config', u'Task::DWIM::Dancer', u'Dancer::Plugin::CDN',
u'Dancer::Plugin::Feed', u'Dancer::Plugin::LDAP', u'Dancer::Plugin::Lucy', 
'...(remaining elements truncated)...']
>>>

That's all folks. If you have other more interesting sorting needs please comment below. Thanks!

There are comments.

django-social-auth tip: Reminder of Login Provider

Every now and then users forget their passwords. This is why I prefer using OAuth and social network accounts like GitHub or Twitter. But what do you do when somebody forgets which OAuth provider they used to login to your site? Your website needs a reminder. This is how to implement one if using django-social-auth.

Back-end

Create a similar view on your Django back-end

def ajax_social_auth_provider_reminder(request):
    """
        Remind the user which social auth provider they used to login.
    """
    if not request.POST:
        return HttpResponse("Not a POST", mimetype='text/plain', status=403)

    email = request.POST.get('email', "")
    email = email.strip()
    if not email or (email.find("@") == -1):
        return HttpResponse("Invalid address!", mimetype='text/plain', status=400)

    try:
        user = User.objects.filter(email=email, is_active=True).only('pk')[0]
    except:
        return HttpResponse("No user with address '%s' found!" % email, mimetype='text/plain', status=400)

    providers = []
    for sa in UserSocialAuth.objects.filter(user=user.pk).only('provider'):
        providers.append(sa.provider.title())

    if len(providers) > 0:
        send_templated_mail(
            template_name='social_provider_reminder',
            from_email='Difio <reminder@dif.io>',
            recipient_list=[email],
            context={'providers' : providers},
        )
        return HttpResponse("Reminder sent to '%s'" % email, mimetype='text/plain', status=200)
    else:
        return HttpResponse("User found but no social providers found!", mimetype='text/plain', status=400)

This example assumes it is called via POST request which contains the email address. All responses are handled at the front-end via JavaScript. If a user with the specified email address exists this address will receive a reminder listing all social auth providers associated with the user account.

Front-end

On the browser side I like to use Dojo. Here is a simple script which connects to a form and POSTs the data back to the server.

require(["dojo"]);
require(["dijit"]);

function sendReminderForm(){
    var form = dojo.byId("reminderForm");

    dojo.connect(form, "onsubmit", function(event){
        dojo.stopEvent(event);
        dijit.byId("dlgForgot").hide();
        var xhrArgs = {
            form: form,
            handleAs: "text",
            load: function(data){alert(data);},
            error: function(error, ioargs){alert(ioargs.xhr.responseText);}
        };
        var deferred = dojo.xhrPost(xhrArgs);
    });
}
dojo.ready(sendReminderForm);

You can try this out at Difio and let me know how it works for you!

There are comments.

Python Twitter + django-social-auth == Hello New User

I have been experimenting with the twitter module for Python and decided to combine it with django-social-auth to welcome new users who join Difio. In this post I will show you how to tweet on behalf of the user when they join your site and send them a welcome email.

Configuration

In django-social-auth the authentication workflow is handled by an operations pipeline where custom functions can be added or default items can be removed to provide custom behavior. This is how our pipeline looks:

settings.py
SOCIAL_AUTH_PIPELINE = (
    'social_auth.backends.pipeline.social.social_auth_user',
    #'social_auth.backends.pipeline.associate.associate_by_email',
    'social_auth.backends.pipeline.user.get_username',
    'social_auth.backends.pipeline.user.create_user',
    'social_auth.backends.pipeline.social.associate_user',
    'social_auth.backends.pipeline.social.load_extra_data',
    'social_auth.backends.pipeline.user.update_user_details',
    'myproject.tasks.welcome_new_user'
)

This is the default plus an additional method at the end to welcome new users.

You also have to create and configure a Twitter application so that users can login with Twitter OAuth to your site. RTFM for more information on how to do this.

Custom pipeline actions

This is how the custom pipeline action should look:

myproject/tasks.py
from urlparse import parse_qs

def welcome_new_user(backend, user, social_user, is_new=False, new_association=False, *args, **kwargs):
    """
        Part of SOCIAL_AUTH_PIPELINE. Works with django-social-auth==0.7.21 or newer
        @backend - social_auth.backends.twitter.TwitterBackend (or other) object
        @user - User (if is_new) or django.utils.functional.SimpleLazyObject (if new_association)
        @social_user - UserSocialAuth object
    """
    if is_new:
        send_welcome_email.delay(user.email, user.first_name)

    if backend.name == 'twitter':
        if is_new or new_association:
            access_token = social_user.extra_data['access_token']
            parsed_tokens = parse_qs(access_token)
            oauth_token = parsed_tokens['oauth_token'][0]
            oauth_secret = parsed_tokens['oauth_token_secret'][0]
            tweet_on_join.delay(oauth_token, oauth_secret)

    return None

This code works with django-social-auth==0.7.21 or newer. In older versions the new_association parameter is missing as I discovered. If you use an older version you won't be able to distinguish between newly created accounts and ones which have associated another OAuth backend. You are warned!

Tweet & email

Sending the welcome email is out of the scope of this post. I am using django-templated-email to define how emails look and sending them via Amazon SES. See Email Logging for Django on RedHat OpenShift With Amazon SES for more information on how to configure emailing with SES.

Here is how the Twitter code looks:

myproject/tasks.py
import twitter
from celery.task import task
from settings import TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET

@task
def tweet_on_join(oauth_token, oauth_secret):
    """
        Tweet when the user is logged in for the first time or
        when new Twitter account is associated.

        @oauth_token - string
        @oauth_secret - string
    """
    t = twitter.Twitter(
            auth=twitter.OAuth(
                oauth_token, oauth_secret,
                TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET
            )
        )
    t.statuses.update(status='Started following open source changes at http://www.dif.io!')

This will post a new tweet on behalf of the user, telling everyone they joined your website!

NOTE: tweet_on_join and send_welcome_email are Celery tasks, not ordinary Python functions. This has the advantage of being able to execute these actions async and not slow down the user interface.

Are you doing something special when a user joins your website? Please share your comments below. Thanks!

There are comments.

Tip: Delete User Profiles with django-social-auth

Common functionality for websites is the 'DELETE ACCOUNT' or 'DISABLE ACCOUNT' button. This is how to implement it if using django-social-auth.

views.py
delete_objects_for_user(request.user.pk) # optional
UserSocialAuth.objects.filter(user=request.user).delete()
User.objects.filter(pk=request.user.pk).update(is_active=False, email=None)
return HttpResponseRedirect(reverse('django.contrib.auth.views.logout'))

This snippet does the following:

  • Delete (or archive) all objects for the current user;
  • Delete the social auth profile(s) because there is no way to disable them. DSA will create new objects if the user logs in again;
  • Disable the User object. You could also delete it but mind foreign keys;
  • Clear the email for the User object - if a new user is created after deletion we don't want duplicated email addresses in the database;
  • Finally redirect the user to the logout view.

There are comments.

Email Logging for Django on RedHat OpenShift with Amazon SES

Sending email in the cloud can be tricky. IPs of cloud providers are blacklisted because of frequent abuse. For that reason I use Amazon SES as my email backend. Here is how to configure Django to send emails to site admins when something goes wrong.

settings.py
# Valid addresses only.
ADMINS = (
    ('Alexander Todorov', 'atodorov@example.com'),
)

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}
 
# Used as the From: address when reporting errors to admins
# Needs to be verified in Amazon SES as a valid sender
SERVER_EMAIL = 'django@example.com'

# Amazon Simple Email Service settings
AWS_SES_ACCESS_KEY_ID = 'xxxxxxxxxxxx'
AWS_SES_SECRET_ACCESS_KEY = 'xxxxxxxx'
EMAIL_BACKEND = 'django_ses.SESBackend'

You also need the django-ses dependency.

See http://docs.djangoproject.com/en/dev/topics/logging for more details on how to customize your logging configuration.

I am using this configuration successfully at RedHat's OpenShift PaaS environment. Other users have reported it works for them too. Should work with any other PaaS provider.

There are comments.


Page 2 / 2