Python Pandas library: the tricks behind DataFrame filtering

(*As a matter of fact, pandas use a structure that is column oriented more than row oriented, that is to say you can consider df.population as a list of index-value pairs)

3) DataFrame filtering use predicates

In programming, a predicate is a function that take one parameter and returns either True or False.

What happens when we write df.population > 13000000?

1. we get an instance of the object that will allow us to work with the content of the population column,
2. then the __gt__(self, obj) method of this object is called (the one responsible for overloading the > operator) and it will give us a predicate that will return True if the population value for a given row is strictly superior to 13 million and False otherwise

The same things is made for df.density > 4000

As for the & in the middle of (df.population > 13000000) & (df.density > 4000) it will allow to have access to a combining logical “and” predicate between the first (population based) and the second (density based) one. We can’t use directly the logical “and” keyword as no operator overloading is available for it, so the bitwise and operator __and__(self, obj) is used instead.

Lets write a simple examples using the same principles

The source code behind Pandas is, of course, more complex than the one I will propose you, as it must handle more use cases and contains securities.

We will consider a list of dictionaries as our input data and will only write a source code corresponding to what we need for our famous new_df = df[(df.population … 4000)] use case.

Code for the predicates

Illustration by the author.

• Predicate, it has a match method always returning True, and it overrides the & operator to return a AndPredicate,
• GTPredicate, it uses both a value_accessor and a value to compare with. It will match when the value retrieved for a given row is greater than (gt) the given value
• AndPredicate, it uses two predicate. It will match when both sub-predicates matches.

Value accessor

Illustration by the author.

We will define a RowColumnAccessor, it takes a column_name, that is to say the name of key in a dictionary.

It will be used to retrieve the value associated with the given key for a row (__call__ method)

It overrides the > operator to return a GTPredicate (greater than predicate)

The DataContainer

Illustration by the author.

The DataContainer store the raw_data (the list of dictionaries).

It define __getitem__ to override the [] operator. You can see there that it will work only with a Predicate instance. It will use it to construct a new, with filtered data, DataContainer.

It also define the __getattr__ method, to create RowColumnAccessor instances, and an iterrows method to enumerate the content.

Testing it

Testing what we wrote so far, illustration by the author

1. We start by constructing the source_data, here it is a list with data about 5 cities.
2. We print the DataContainer content using iterrows.
3. We then filter cities using the population and the density.
4. If 3 cities from our dataset have a population greater than 13 million, only Tokyo and Delhi also match the density criteria.

Full source code is accessible below:

---

That’s all folks!

I hope you will have learn one trick or two with this.

You can also read more about operator overloading with my previous article about them.

https://medium.com/@jerome.o.diaz/python-cheatsheet-operator-overloading-247bfce6a16f