Latest Blog Posts

Vector graphics wallpapers with python

2023-02-11T12:00:00Z

Graphic by author

TL;DR

If you are not interested in any of the code or math but only want to see pretty graphics - jump straight to the graphics section.

Introduction

In the past months I've been working with a lot with vector graphics. Although I wouldn't call myself a graphic designer, I was mistaken for one a couple of times at tech events. I quite enjoy creative design work since it's a really refreshing mode change from writing code or looking at data. I've always enjoyed making things look aesthetically pleasing but in my work that was mainly limited around creating charts and data visualizations, arranging those into dashboards and laying it out in some app. Occasionally I'm creating diagrams, but that's mostly for technical purposes.

Most of the work with vector graphics I've done to date was in Inkscape which I fell most comfortable with since I've been using on and off for a few good years. Although I've tried more modern, web based tools like Canva or Figma but whenever I had something a bit more advanced I always went back to Inkscape. All tools have their strengths, but they all require some learning investment before becoming productive. Inkscape turned out to be a great investment for me, but I feel I could do much more with vector graphics if I could tap into my python skills.

For some time I wanted to learn how to create vector graphics with python. I've watched a few conference talks on creative coding and the whole concept resonated with me instantly. I started exploring what's possible around the stack I'm familiar with. A good place to start was learning a bit more about SVG itself which stands for Scalable Vector Graphics, and it is an XML-based markup language that describes 2D vector graphics. For our purposes we won't need to go very deep and only explore a few elements and attributes and that'll be sufficient to start creating graphics.

What I would like to do is to be able to create repeated patterns (or tilings) of simple geometric shapes and apply a few effects to make it a bit less boring. I am aware that that Inkscape has a tool to create tilings based on repeating patterns, but I couldn't fully do what I had in mind and I wanted to try creating graphics programmatically. There is also a practical aspect to this project since I would like to come up with a design to cover a few windows with in a restaurant to increase privacy and comfort for guests.

I'll start with a simple square lattice¹ and insert simple shapes like squares on the lattice positions. The plan is to start modifying the shapes to create interesting effects. In the end I'll repeat the whole set of designs on a hexagonal to compare the results.

Creating SVG graphics with python

As you might've expected, there are python packages out there, that can help you create SVGs without having to write raw XML and worrying about matching tags and debugging typos. I'll use svgwrite since it works out of the box with JupyterLab. Alternatively you can try drawSvg, they both seem to have a very similar API.

I'm doing most of my coding here in a jupyter notebook it might be worth mentioning that notebooks can display SVGs natively. They can display SVG content directly in a cell either from a file or from SVG source with the handy %%SVG cell magic

%%SVG
<svg height="210" width="500">
<polygon
    points="200,10 300,180 100,180"
    style="fill:darkcyan;stroke:tomato;stroke-width:2"
/>
</svg>

should give you the following in the output cell:

Alternatively you can also use the builtin display system of IPython:

from IPython.core.display import SVG
SVG("""<svg height="210" width="500">
<polygon
    points="100,200 200,200 150.0,113.39745962155614"
    style="fill:darkcyan;stroke:tomato;stroke-width:2"
/>
<polygon
    points="200,200 300,200 250.0,113.39745962155614"
    style="fill:darkcyan;stroke:tomato;stroke-width:2"
/>
</svg>""")

should in turn produce:

As you can see SVG is quite straightforward. The drawing in enclosed in the <svg>...</svg> tags, and I'm using the polygon element to draw the shapes by specifying the coordinates of all vertices. The appearance is then modifies with the style attribute in a CSS-like syntax.

Turning to python and making graphics with svgwrite requires creating python representation of the drawing and adding shapes to it. I'm only going to use rect, circle and polygon elements with a few standard attributes like: fill, stroke and opacity. Here's an example that illustrates basic functionality:

import svgwrite
dwg = svgwrite.Drawing(filename="example.svg", debug=True, size=(400, 140))
dwg.add(dwg.rect(
    insert=(50, 20), size=(100, 50), fill="darkcyan", stroke="none", opacity=0.7))
dwg.add(dwg.rect(
    insert=(180, 20), size=(50, 100), fill="darkcyan", stroke="none", opacity=1.0))
dwg.add(dwg.circle(
    center=(290, 60), r=40, fill="tomato", stroke="none"))
dwg.add(dwg.polygon(
    points=[(350, 120), (400, 120), (375.0, 13.5)], fill="tomato", stroke="none",
    opacity=0.8))

That's it! We now know enough about SVG to start producing more advanced designs programmatically.

Square lattice

Square lattice is the best place to start since it's the simplest. What I want, specifically is a regular grid of points whose coordinates will be expressed as the SVG canvas coordinates². Once we have the coordinates we can start putting shapes in those locations. To define a square lattice it is sufficient to define a single parameter a which is the lattice parameter³. Defining the lattice and size of our graphics will allow us to determine how many tiles we'll be able to fit on the graphic of a given size. Here's how it might look in code:

@dataclass
class SquareLattice:
    "Square Lattice in 2D"
    a: float

    def coords(self, row: int, col: int, center: bool = True) -> npt.ArrayLike:
        "Coordinates for a lattice position (row, col)"
        if center:
            return np.array([
                self.a * (row + 0.5),
                self.a * (col + 0.5)], dtype=float)
        else:
            return np.array([self.a * row, self.a * col], dtype=float)

    def size(self, width: int, height: int) -> npt.ArrayLike:
        "Size of the lattice in lattice postions"
        s = math.ceil(width / self.a) + 1
        return np.array([s, s], dtype=int)

SquareLatttice takes a single parameter a and with the coords method will generate (x, y) coordinates for each of the integer lattice positions (i, j). To see how it works let's generate a few coordinate pairs:

lat = SquareLattice(a=10)
print(f"coordiantes for position (0, 0): {lat.coords(0, 0)}")
print(f"coordiantes for position (0, 0): {lat.coords(1, 0)}")
print(f"coordiantes for position (0, 0): {lat.coords(0, 1)}")
print(f"coordiantes for position (0, 0): {lat.coords(2, 2)}")

coordiantes for position (0, 0): [5. 5.]
coordiantes for position (0, 0): [15.  5.]
coordiantes for position (0, 0): [ 5. 15.]
coordiantes for position (0, 0): [25. 25.]

By default, we get coordinates for the center of each cell. It will only be important later on, but it's good to notice.

Now that we have the lattice positions we can put some elements there. For a square lattice, squares are a good starting point. Here's how we can create the drawing and repeat squares on the predefined lattice:

def draw_squares(
    lattice: Lattice,
    side: int = 15,
    width: int = 800,
    height: int = 800,
    fill: str = "darkcyan",
    stroke: str = "none",
    opacity: float = 1.0,
):
    "Draw SVG squares on a lattice"
    dwg = svgwrite.Drawing(debug=True, size=(width, height))

    xsize, ysize = lattice.size(width, height)
    for i in range(xsize):
        for j in range(ysize):
            dwg.add(
                dwg.rect(
                    insert=lattice.coords(i, j),
                    size=(side, side),
                    fill=fill,
                    stroke=stroke,
                    opacity=opacity,
                )
            )
    return dwg

Since we can independently control the lattice spacing and size of the squares let's see a few variants:

Lattice a=40, square size=5

Lattice a=40, square size=20

Lattice a=90, square size=80

Nothing spectacular yet but we need this foundation to be able to quickly modify the designs.

Making things more interesting

There are many ways to modify the grids we've produced so far to make it more interesting. Obvious example is probably changing the color channel but before going there I will try changing the size of the squares. The effect that I think it will have to imitate brightness where smaller shapes will be less intense and bigger elements more intense. Alternatively I would try to modify opacity in a similar fashion, but I think encoding brightness with size will be a nicer aesthetic.

I want to simulate a fade effect with a square size gradient. I would like to try out a few different gradient functions, but I'll start with the linear gradient along the y axis, that we'll use to modify elements.

With the function below I'm going to be able to create an array with coefficients [0.0, 1.0] to scale the squares by. Since the gradient will be applied along the y axis the size of the gradient array is the same as the number of lattice points in the y direction.

def linear_gradient(
    size: int,
    offset: float = 0.0,
    geom: bool = False,
    start: float = 0.01
    ):
    c = math.floor(size * offset)
    if geom:
        return np.hstack((
            np.geomspace(start, 1, size - c, endpoint=False),
            np.ones(c)
        ))
    else:
        return np.hstack((
            np.linspace(start, 1, size - c, endpoint=False),
            np.ones(c)
        ))

To make it work we'll need a small modification to the drawing function to take gradient argument and scale the squares with it. Here's the updated function:

def draw_squares(
    lattice: Lattice,
    side: int = 15,
    width: int = 800,
    height: int = 800,
    fill: str = "darkcyan",
    stroke: str = "none",
    opacity: float = 1.0,
    gradient: npt.ArrayLike = None,
):
    "Draw SVG squares on a lattice with gradient"
    dwg = svgwrite.Drawing(debug=True, size=(width, height))

    xsize, ysize = lattice.size(width, height)
    for i in range(xsize):
        for j in range(ysize):
            size = side * gradient[j]
            dwg.add(
                dwg.rect(
                    insert=lattice.coords(i, j),
                    size=(size, size),
                    fill=fill,
                    stroke=stroke,
                    opacity=opacity,
                )
            )
    return dwg

Putting it all together and generating the grid with the gradient gives the following result

Lattice a=40, square size=15 with a linear gradient

If we draw squares that are a bit larger we'll notice a slight problem with the current approach since the default rect element is drawn with respect to top left corner which results in an alignment to the left.

Lattice a=80, square size=75 with a linear gradient

Although it might be desirable is some cases I would rather have all the squares centered with respect to the lattice positions. To make that happen I created a custom Square class that I'll use with the polygon element instead of rect. Here's the Square class and updated draw_squares function

@dataclass
class Square:
    "Square"
    side: int

    def coords(self):
        "Vertex coordinates of a square with repect to its center (0, 0)"
        return np.array([
            (-self.side / 2, -self.side / 2),            
            (-self.side / 2, self.side / 2),
            (self.side / 2, self.side / 2),
            (self.side / 2, -self.side / 2),
        ])

def draw_squares(
    lattice: Lattice,
    side: int = 15,
    width: int = 800,
    height: int = 800,
    fill: str = "darkcyan",
    stroke: str = "none",
    opacity: float = 1.0,
    gradient: npt.ArrayLike = None,
):
    "Draw SVG squares on a lattice"
    dwg = svgwrite.Drawing(debug=True, size=(width, height))

    xsize, ysize = lattice.size(width, height)
    for i in range(xsize):
        for j in range(ysize):
            square = Square(side=side * gradient[j])
            points = square.coords() + lattice.coords(i, j)
            dwg.add(
                dwg.polygon(
                    points = points,
                    fill=fill,
                    stroke=stroke,
                    opacity=opacity,
                )
            )
    return dwg

Redrawing the previous graphic with the updates gives the expected result where all the squares are now centered around lattice positions.

Lattice a=80, square size=75 with a linear gradient centered

Ridge

With a slight modification to the linear_gradient function we can make the gradient symmetric with respect to the center. I'm calling it ridge gradient, and it should create a fade effect going both up and down.

def ridge(size: int, geom: bool = False, start: float = 0.01):
    c = round(size / 2)
    if geom:
        return np.hstack((
            np.geomspace(start, 1, size - c),
            np.geomspace(start, 1, c)[::-1]
        ))
    else:
        return np.hstack((
            np.linspace(start, 1, size - c),
            np.linspace(start, 1, c)[::-1]
        ))

Applying the ridge gradient give a nicely symmetric pattern.

Lattice a=20, square size=15 with a ridge gradient

I guess it's easy to see how to modify the current code to change the direction or produce a different fade pattern. In the next step we're going to update the gradient from 1D to 2D to allow for more sophisticated patterns.

Radial gradient

So far we've applied gradient along a single axis, but there is nothing preventing us from extending the gradient to 2D. One familiar concept we can try to implement is radial gradient. We'll need to modify the function for drawing the squares to modify the size of squares based on both i and j lattice positions, and we'll need a 2D array with sizes representing brightness. The radial_gradient function evaluates the distance between the origin of the gradient and given lattice position and scales the value to [0.0, 1.0] range within the specified radius.

def radial_gradient(
    lattice: Lattice,
    origin: np.ArrayLike,
    radius: float,
    width: int,
    height: int) -> npt.ArrayLike:
    "Compute brightness for a radial gradient"
    def radial_distance(origin, coords, radius):
        distance = euclidean(origin, coords)
        return min(distance / radius, 1.0)

    xsize, ysize = lattice.size(width, height)
    gradient = np.ones((xsize, ysize))
    for i in range(xsize):
        for j in range(ysize):
            gradient[i, j] = min(
                gradient[i, j],
                radial_distance(origin, lattice.coords(i, j), radius)
            )
    return gradient

Now the returned array is 2D and can be applied to our grid. The modification to draw_squares is straightforward, so I'll not repeat the code here. Since the number in the gradient array are scaled to [0.0, 1.0] we can also compute the inverse gradient by subtracting the gradient from 1. Here are the two variants we get:

Lattice a=20, square size=15, radial gradient

Lattice a=20, square size=15, inverse radial gradient

A nice effect here is that side by side these look like the right hand side one is "taken out" from the left hand side one.

Hexagonal lattice

Square lattice is not the only choice for a lattice. It's the most symmetric looking, and therefore it has an artificial or dry aesthetic. To check the effect that a lattice might have on the overall graphic I'll try also a hexagonal lattice also known as honeycomb structure.

Just as in the case of a square lattice we need only a single parameter a to define the lattice. Since for the hexagonal lattice the translation vectors are no longer parallel we need to take that into account when computing lattice coordiantes⁴.

@dataclass
class HexLattice:
    "Hexagonal lattice"
    a: float

    @property
    def vectors(self):
        return np.array([[self.a, 0], [self.a / 2, self.a * np.sqrt(3) / 2]])

    def coords(self, row: int, col: int) -> npt.ArrayLike:
        "Coordinates for a lattice position (row, col)"
        return self.vectors[0] * row + self.vectrors[1] * col

With the lattice defined I would like to have regular hexagons in cell positions in analogy to having squares on a square lattice. For a regular hexagon with the side a here is a diagram with the coordinates of the vertices with respect to the center of the hexagon⁵.

Coordinates of the hexagon vertices with respect to its center

Based on that we can code up the Hexagon class to be drawn on the canvas as the polygon element.

@dataclass
class Hexagon:
    "Regular Hexagon"
    side: int

    @property
    def height(self):
        "Height of a flat top regular hexagon"
        return math.sqrt(3) * self.side

    def coords(self):
        "Coordinates of vertices with repect to hexagon center (0, 0)"
        return np.array([
            (-self.side / 2, -self.height / 2),            
            (-self.side, 0),
            (-self.side / 2, self.height / 2),
            (self.side / 2, self.height / 2),
            (self.side, 0),
            (self.side / 2, -self.height / 2),
        ])

With that we should be able to recreate the analogues of the graphics we created with squares on a square lattice. We'll also need a draw_hexagons function, but that again is pretty analogous to draw_squares, so I'll leave it for you to figure adapt.

Graphics

With all things in place here is a side by side comparison of the graphics with and without different gradient effects. You might notice that there is an extra wavy pattern generated which is also at the head of this post. It's essentially a sin wave with a fade based on distance from the curve. If you're curious about it let me in the comments and I'll more.

Square lattice a=40, square side=20

Hex lattice padding=5, hex side=20

Sq lattice a=33, square side=25, linear gradient

Hex lattice padding=5, hex side=20, linear gradient

Sq lattice a=33, square side=25, ridge gradient

Hex lattice a=20, hex side=20, ridge gradient

Sq Lattice a=33, square size=25, sin gradient

Hex lattice a=20, hex side=20, sin gradient

Sq lattice a=33, square side=25, radial gradient

Hex lattice a=20, hex side=20, radial gradient

Sq lattice a=33, square side=25, inverse radial gradient

Hex lattice a=20, hex side=20, inverse radial gradient

Summary

Generating vector graphics programmatically was a lot more exciting than I expected and I was surprised how much I got sucked into it. Once it was clear how to create shapes and position them on the canvas create variations on the designs was quite easy. I am quite happy with the results so far and definitely prefer the hex-on-hex variant, but I would like to explore further and try out a few more things to make the designs a bit more organic and less perfect.

To take it further I would probably explore ways of introducing randomness in various channels and to encode different attributes. So far all the designs were quite simple and deterministic and randomness would take them to next level. Breaking the perfect symmetries either by deforming the lattice of making the shapes a bit irregular (or both) might also produce an interesting result.

On top of that I think that introducing various kinds of defects and imperfections inspired by real crystal structures where those phenomena occur would add yet another dimension to explore.

Footnotes

This is going to be a nice journey back in to my crystallgraphy lectures. ↩
By default SVG coordinates start in the top-left corner at (0, 0) and go to (width, height) with the pixels as unit. ↩
In general there are 5 types Bravais lattices in 2D, and they can be specified by defining the parameters of the unit cell. ↩
If you want to know learn more about hexagonal coordinates check out this post. ↩
I choose the flat top orientation of the hexagon, an alternative would be the pointy top hexagon orientation. They differ by a 90-degree rotation of all vertices, so it's quite straightforward to switch between them. ↩

Better select forms with django and Bootstrap 5

2022-10-16T23:55:00Z

Photo by Kelly Sikkema on Unsplash

I enjoy working with Django for many reasons but mostly since I can focus on what I want my app to do and django happily takes care of the rest. In 2022 it's probably not the sexiest choice when it comes to web frameworks, but there are good reasons it is still as popular as it is. There is a recent wave of excitement about server side rendering frameworks thanks to a new batch of tools like HTMX, django-unicorn and django-sockpuppet that give django superpowers and put into question the React based paradigm for building apps. But that's a digression for another time.

Django takes care of a lot of stuff behind the scenes and that is a solid productivity boost. Being a mature framework means that is has opinions (mostly for very good reasons) and provides you with defaults that are sensible in most o the cases. There are however cases when it's worth to make some changes to improve end-user experience.

One such aspect is the default UI widget for selecting items. I am of course talking about dropdowns. They have been around since forever and although being familiar they are usually not the best choice when it comes to modern UX standards. Out in the wild you can find professionals suggesting caution when using dropdowns and calls to stop using drop-downs at all.

Personally I would like to adhere to good UX practices as far as I can and although my level of comfort increases with proximity to backend side of things I cannot be annoyed by all the low hanging fruit that sometimes plague otherwise good products. Wanting things to look good without a massive design effort is one of the reasons I usually reach for a frontend framework, most often bootstrap. With django that means using django crispy forms to get the forms looking sharp and consistent with an extra line in the template.

Making choices

One of the things I was recently working on is a django-based dashboard where I wanted to have the option to choose the aggregation level between: daily, weekly, monthly aggregates. Of course my first pass was the default forms.Select widget that is a dropdown. Once I saw the result on the screen though, I felt something is off, and that this should probably be improved. Usually I try to move on until I have a working prototype that I can iterate over. Following the Kent Beck's principle

make it work, make it right, make it fast

When I got everything wired up and working it seemed like a good time get back to the select widget and scratch that itch.

On the django side I had a form to represent the choices

from django import forms

class FrequencyForm(forms.Form):
    choices = [
        ("1D", "Daily"),
        ("1W", "Weekly),
        ("1M", "Monthly")
    ]
    freq = forms.ChoiceField(label="Frequency", choices=choices)

which was being rendered in a template with pretty standard django

{% load crispy_forms_tags %}
...
{{ form|crispy }}

The effect isn't surprising either

Frequency*

I didn't want for the options to be hidden, so the next best thing was to switch to a radio select by changing the for widget to RadioSelect and keeping the template as is

from django import forms

class FrequencyForm(forms.Form):
    choices = [
        ("1D", "Daily"),
        ("1W", "Weekly),
        ("1M", "Monthly")
    ]
    freq = forms.ChoiceField(
        label="Frequency",
        choices=choices,
        widget=forms.RadioSelect(),
    )

This gives

Frequency*

Daily

Weekly

Monthly

Bettter but I still wasn't happy with the result. When looking for inspiration I found this post suggesting a segmented button that reminded me of what I saw in latest Bootstrap 5 docs so I decided to give it a go since I was already using Bootstrap.

Looking at the example from Bootstrap it became clear that I would have to render the form manually. The example I started from was this:

<div class="btn-group" role="group" aria-label="Basic radio toggle button group">
<input type="radio" class="btn-check" name="btnradio" id="btnradio1" autocomplete="off" checked>
<label class="btn btn-outline-primary" for="btnradio1">Radio 1</label>

<input type="radio" class="btn-check" name="btnradio" id="btnradio2" autocomplete="off">
<label class="btn btn-outline-primary" for="btnradio2">Radio 2</label>

<input type="radio" class="btn-check" name="btnradio" id="btnradio3" autocomplete="off">
<label class="btn btn-outline-primary" for="btnradio3">Radio 3</label>
</div>

which looks like

Radio 1 Radio 2 Radio 3

Pretty nice! I went ahead and updated my template and instead of the {{ form|crispy }} elements I went ahead and unfolded the form and added the classes and attributes to render the button group:

<div class="btn-group" role="group" aria-label="Basic radio toggle button group">
    {% for choice in form.freq %}
        {{ choice.tag }}
        <label class="btn btn-outline-primary" for="{{ choice.id_for_label }}">{{ choice.choice_label }}</label>
    {% endfor %}
</div>

Everything looked nice when rendered however when clicking on the buttons they wouldn't stay selected as they should. They would change back to unselected which was annoying. I had to compare the original HTML with my rendered version to realize that the <input> tags in my case were missing s class="btn-check" attribute, so I modified the form to have that added

class FrequencyForm(forms.Form):
    choices = [
        ("1D", "Daily"),
        ("1W", "Weekly),
        ("1M", "Monthly")
    ]
    freq = forms.ChoiceField(
        label='Category',
        choices=choices,
        widget=forms.RadioSelect(attrs={'class':'btn-check'}),
    )

Adding that attribute to the form did the trick. With that little tweak I was able to achieve way better UX and eliminate and unnecessary dropdown.

Daily Weekly Monthly

This was it for me but in case you have a few more choices to select from (maybe 5-9 items) and you still would like to present them to the user this might not be a good choice since button group will most likely be too wide for small screens and overflow.

In this case you can switch to splitting choices into individual buttons and arranging them in a grid rather than a single row. Then your CSS framework can take over and manage the responsive layout for smaller screens where you probably want to either switch to a more vertical layout or a grid layout.

Let me add two more choices to my original form to illustrate how could it look like

class FrequencyForm(forms.Form):
    choices = [
        ("1D", "Daily"),
        ("1W", "Weekly),
        ("1M", "Monthly")
        ("1Q", "Quarterly")
        ("1Y", "Yearly")
    ]
    freq = forms.ChoiceField(
        label='Category',
        choices=choices,
        widget=forms.RadioSelect(attrs={'class':'btn-check'}),
    )

In the template we need to wrap all choice fields in a row and each input + label tag in a col with appropriate widths for each display breakpoint. This is bootstrap specific so check out bootstrap layout if you want to know more.

<form>
<div class="row">
{% for choice in form.freq %}
    <div class="col-4 col-md-3 col-lg-2">
    {{ choice.tag }}
    <label class="btn btn-outline-primary" for="{{ choice.id_for_label }}">{{ choice.choice_label }}</label>
    </div>
{% endfor %}
</form>

This is now rendered as independent radio buttons

Summary

If you're using django and a css framework like bootstrap it does not take much effort to adapt your forms and improve overall user experience and at the same time conform to modern standard for look and feel of your web app. If you have similar quick improvements please share below since I'm always on the lookout for how to improve usability.

Datasets beat Models

2021-06-24T12:00:00Z

I know it’s easy to get carried away by all the shiny, sophisticated ML models out there, but we need to have a serious conversation about datasets. In most industrial applications of ML it’s the datasets that limit performance not model complexity.

Lack of high quality training datasets is the biggest challenge for industrial machine learning right now. That should not be surprising if you consider how much bigger the supply of models is compared to datasets. A big portion of the ML algorithms development happens thanks to open source and academic communities which gladly publish their work in exchange for reputation points. In contrast, developing and curating datasets is often less valued and not as glamorous.

My experience from working on data-driven solutions for industrial problems is that there is no shortage of data and huge volumes are available in raw form; however converting them into training and validation datasets is what usually blocks ML/AI progress.

I won't be controversial if I attribute 80% of a data scientist's time to what is collectively termed preprocessing, cleaning or structuring the data. The result of all of those activities is a dataset that hopefully accurately represents reality and can serve as ground truth for model development and evaluation. Creating high quality datasets however takes effort, skills and a right blend of domain knowledge and experience with data.

This is exactly what we are passionate about at unifai. We are approaching dataset development as an engineering practice and we are creating high quality datasets to increase adoption of machine learning in the industry. Our experience in data science, data engineering, software development and heavy asset industry puts us in a unique position to deliver datasets with a perfect mix of domain knowledge.

Unifai is not alone in betting on datasets, one of the biggest names in AI: Andrew Ng recently started promoting a more data-centric view of AI instead of the model-centric view. In a recent announcement together with landing.ai and deeplearning.ai he launched a competition that focuses on improving data while keeping the model fixed to inspire new methods for improving data.

Get in touch with me if you share similar challenges or if you're sitting on piles of raw data and your organization wants to get some quick wins with machine learning.

This post was originally published on LinkedIn.

Zeolite pore size oddity

2017-10-31T12:00:00Z

One of the ways to classify zeolites is usually by the size of their pore openings given in the peculiar unit of T atom count. T atom stands for tetrachedrally coordinated atoms, usually the term lumps together silicon and aluminum in case of zeolites or aluminum and phosphorus in case of aluminumphpsphates. The structural units characterising pores are usually rings from which the tube-like channels are formed (as shown below) and therefore the as described as n membered rings or nMR in short. Where n can be as small as 4 and as large as 20.

[picture of 8MR, 10MR and 12MR with a corresponding tube]

By inspecting the figure below showing the distribution of ring sizes among the known zeolite structures it is easy to see that 8, 10, and 12 MR structures dominate over the remaining topologies. Which justifies the why this parameter might serve as a good classification. It should also be taken into account that for most of the interesting applications topologies with MR < 8 are not intersting since molecules cannot enter the small channels and channels larger than 14 would have very similar properties.

IUPAC classification into nano, micro and mesoporous structures in a technical report.

It is customary to call 8, 10 and 12MR, small, medium and large pore zeolites suggesting that this classification provides distinct groups of similar characteristing being size. While intuitively and qualitatively correct, applying it directly can lead to incorrect insights since this MR measure provides limited structural information. A closer inspection of each class revelas that in each class there is a distribution of sizes as evaluated by different criteria actually associated with a more functional definition of pore size.

apply clustering based on size largst cavity diameter (LCD) and some other features related to functional size and compare them to the classification based on MR and see which are misclassified.

Superimpose chemical structures

2017-09-28T12:00:00Z

I was struggling recently how to present multiple chemical structures that differed slightly in their geometry but

import os
import re
from collections import OrderedDict
from ase import Atoms
import ase.io


def read_structures(path, fpatt, verbose=False):
    '''
    Read all the structures from `path` that match the `fpatt` pattern
    '''

    fpatt = re.compile(fpatt)

    out = {'II': OrderedDict(), 'III': OrderedDict(), 'IV': OrderedDict()}

    for fname in os.listdir(path):
        m = fpatt.match(fname)
        if m:
            symbol, oxs = m.group(1), m.group(2)
            out[oxs][symbol] = ase.io.read(os.path.join(path, fname))
        else:
            if verbose:
                print('ERROR ', fname)

    print('Read: ', ', '.join(['{}: {}'.format(k, len(v)) for k, v in out.items()]))

    return out

if __name__ == '__main__':

    path = 'AlPO-5-H'
    patt = r'([A-Za-z]+)\(([IV]+)\)-AlPO-5-O2-H\.traj'

    halpo = read_structures(path, patt)

    atoms = Atoms()

    for me, a in halpo['IV'].items():
        atoms = atoms + a

    transmittances = [0.8] * len(atoms)

    ase.io.write('Me-AlPO-5-H.pov', atoms,
        rotation='-15y',
        show_unit_cell=0,
        run_povray=True,
        display=False,
        pause=True,
        canvas_width=1024,
        camera_type='perspective',
        transmittances=transmittances)

    ase.io.write('Me-AlPO-5-H_0y.pov', atoms,
        rotation='',
        show_unit_cell=0,
        run_povray=True,
        display=False,
        pause=True,
        canvas_width=1024,
        camera_type='perspective',
        transmittances=transmittances)

Rendering the scenes for multiple structures was a bit too slow for my laptop so I had to use a supercomputer to speed things up.

This is a bit of a special case since the structures are taken from a periodic crystals and therefore all have the same unit cell size. It simpifies things a lot since then all the coordinates are given with respect to origin of the unit cell which is the same for all the materials. For (non-periodic) molecules we would have to align the molecules first since their coordinates can be defined with respect to different origins. One possible solution is to minimize the root mean square deviation between the structures in order to realign them which is usually done using the Kabsch algorithm. For those curious to see it in practice e is also a python available rmsdpy.