Lesson 13 - Magic Methods in Python - Collections and Descriptors

In the previous lesson, Magic Methods in Python - Math methods, we were dealing with mathematical functions and operators. In this Python tutorial, we're going to look at the magic collection methods and their abstract base classes.

Magic methods

__len__(self)

Returns the number of items in the collection.

__getitem__(self, key)

Returns an item from the collection by a specified key, this syntax applies to classes similar to dictionaries. If it fails, it should raise KeyError.

__missing__(self, key)

The method is called by the __getitem__() method if the given key is missing in the dictionary subclass.

__getitem__(self, index)

Returns an item from the collection at a given position. In case of failure, it should raise IndexError (in the for loop, IndexError indicates the end of the sequence).

__setitem__(self, key, value)

Sets the value of an item specified by a given key in the collection, this syntax applies to classes similar to dictionaries. If it fails, he should raise KeyError.

__delitem__(self, key)

Deletes an entry with a given key from the collection, this syntax applies to classes similar to dictionaries. If it fails, he should raise KeyError.

__iter__(self)

Returns an iterator for the collection. We can create an iterator using the iter() function from an object that has a magical __iter__() or __getitem__() method.

__reversed__(self)

Returns an inverted iterator for the collection, we should re-implement this method if there's a faster implementation than:

for i in reversed(range(len(self))):
    yield self[i]

Abstract base classes

Abstract base classes provide us with a list of interfaces that we should implement. Interfaces are all public methods, including magic methods. If we implement the required minimum, other methods are derived from them.

Class	Inherits from	Abstract methods	Derived methods
Container		__contains__
Hashable		__hash__
Iterable		__iter__
Iterator	Iterable	__next__	__iter__
Sized		__len__
Callable		__call__
Sequence	Sized, Iterable, Container	__getitem__, __len__	__contains__, __iter__, __reversed__, index, and count
MutableSequence	Sequence	__getitem__, __setitem__, __delitem__, __len__, insert	Sequence methods + append, reverse, extend, pop, remove and __iadd__
Set	Sized, Iterable, Container	__contains__, __iter__, __len__	__le__, __lt__, __eq__, __ne__, __gt__, __ge__, __and__, __or__, __sub__, __xor__, and isdisjoint
MutableSet	Set	__contains__, __iter__, __len__, add, discard	The Set method + clear, pop, remove, __ior__, __iand__, __ixor__, and __isub__
Mapping	Sized, Iterable, Container	__getitem__, __iter__, __len__	__contains__, keys, items, values, get, __eq__ a __ne__
MutableMapping	Mapping	__getitem__, __setitem__, __delitem__, __iter__, __len__	The methods Mapping + pop, popitem, clear, update, and setdefault
MappingView	Sized		__len__
ItemsView	MappingView, Set		__contains__, __iter__
KeysView	MappingView, Set		__contains__, __iter__
ValueView	MappingView		__contains__, __iter__

Attribute methods

__getattr__(self, name)

This method is called if the attribute isn't found using the usual approach. The method should return the attribute's value or invoke AttributeError.

__setattr__(self, name, value)

It's called when setting attributes instead of assigning variables to __dict__. If the class wants to save the attribute, it should call the __setattr__() superclass method.

__delattr__(self, name)

We should only implement this method if:

del obj.name

makes some meaningful sense for the class.

An example:

class Foo:

    def __setattr__(self, name, value):
        print("Setting")
        super().__setattr__(name, value)

    def __getattr__(self, name):
        print("Getting")
        return 1

foo = Foo()
print(foo.x)
print(foo.__dict__)
foo.x = 2
print(foo.x)
print(foo.__dict__)

Descriptors

Descriptors allow us to override attribute access using another class. We can avoid redundant properties like this. The class must define at least one of the __get__(), __set__(), or __delete__() methods. With the descriptor, we can improve the validation of attribute values. We'll create a sample Typed descriptor that will ensure that the attribute is of a certain type:

from weakref import WeakKeyDictionary

class Typed:

    dictionary = WeakKeyDictionary()

    def __init__(self, ty=int):
        self.ty = ty

    def __get__(self, instance, owner=None):
        print("Getting instance")
        return self.dictionary[instance]

    def __set__(self, instance, value):
        print("Setting instance", instance, "to", value)
        if not isinstance(value, self.ty):
            raise TypeError("Value must be type {}".format(self.ty))
        self.dictionary[instance] = value

__get__(self, instance, owner)

The method accepts an instance and a class as parameters. The instance's class can be obtained as follows:

owner = type(instance)

In the code, we return the property value stored in the descriptor.

__set__(self, instance, value)

The method accepts an instance as a parameter and the value to which the attribute should be set. When setting, we check if the attribute is of the given type. It's normally set to int (integer). If not, we raise the TypeError exception.

Creating a class

Now we'll create a class that will represent a human. We set him three attributes: name, age, and weight. The name will have to be str (string), age of the int (integer) type, and weight will be float (decimal number):

class Person:

    name = Typed(ty = str)
    age = Typed()
    weight = Typed(ty = float)

    def __init__(self, name, age, weight):
        self.name = name
        self.age = age
        self.weight = weight

We create the descriptors in the way that each property name will be a descriptor. We create descriptors at the class level. Each descriptor then contains a dictionary with instances and property values.

When calling attributes, Python recognizes that they are descriptors and calls the __get__(), __set__(), or __delete__() methods.

Of course, we can also create subclasses without redefining the properties.

For example, a subclass with an additional iq property:

class CleverPerson(Person):

    iq = Typed(ty = int)

    def __init__(self, name, age, weight, iq):
        super().__init__(name, age, weight)
        self.iq = iq

Of course, we can create subclasses of descriptors as well:

class Ranged(Typed):

    def __init__(self, ty=int, min=0, max=100):
        super().__init__(ty)
        self.min = min
        self.max = max

    def __set__(self, instance, value):
        if value < self.min:
            raise TypeError("Value must be greater than {}".format(self.min)
        elif value > self.max:
            raise TypeError("Value must be lower than {}".format(self.max)
        super().__set__(instance, value)

Weak-reference dictionaries

We use a weak reference dictionary (WeakKeyDictionary) from the weakref module to store the attributes. It works like a normal dictionary but differs in one detail. If there are no more references to an object in the dictionary than the one in the dictionary, the object is deleted from the dictionary. This will prevent memory leaks. An ordinary dictionary would prevent instances from being deleted even if they were no longer used, because it would be the dictionary using them. If we use a __slots__ class, we need to add weakref to __slots__ to support weak links.

Getting the descriptor

If we want to modify the descriptor, we need to access the object class and slightly modify the __get__() method. For example:

from weakref import WeakKeyDictionary

class Typed:

    dictionary = WeakKeyDictionary()

    def __init__(self, ty=int):
        self.ty = ty

    def __get__(self, instance, owner=None):
        if instance is None: # we access through the class
            return self
        print("Getting instance")
        return self.dictionary[instance]

    ...


CleverPerson.name # gets the descriptor

This way we can create and call our own methods on the descriptor.

In the next lesson, Datetime Library for Python, we'll look at the datetime library in Python.

 

Did you have a problem with anything? Download the sample application below and compare it with your project, you will find the error easily.