Author: J. Pablo Fernández

Converting a Python data into a ReStructured Text table

This probably exist but I couldn’t find it. I wanted to export a bunch of data from a Python/Django application into something a non-coder could understand. The data was not going to be a plain CSV, but a document, with various tables and explanations of what each table is. Because ReStructured Text seems to be the winning format in the Python world I decided to go with that.

Generating the text part was easy and straightforward. The question was how to export tables. I decided to represent tables as lists of dicts and thus, I ended up building this little module:

def dict_to_rst_table(data):
    field_names, column_widths = _get_fields(data)
    with StringIO() as output:
        output.write(_generate_header(field_names, column_widths))
        for row in data:
            output.write(_generate_row(row, field_names, column_widths))
        return output.getvalue()


def _generate_header(field_names, column_widths):
    with StringIO() as output:
        for field_name in field_names:
            output.write(f"+-{'-' * column_widths[field_name]}-")
        output.write("+\n")
        for field_name in field_names:
            output.write(
                f"| {field_name} {' ' * (column_widths[field_name] - len(field_name))}"
            )
        output.write("|\n")
        for field_name in field_names:
            output.write(f"+={'=' * column_widths[field_name]}=")
        output.write("+\n")
        return output.getvalue()


def _generate_row(row, field_names, column_widths):
    with StringIO() as output:
        for field_name in field_names:
            output.write(
                f"| {row[field_name]}{' ' * (column_widths[field_name] - len(str(row[field_name])))} "
            )
        output.write("|\n")
        for field_name in field_names:
            output.write(f"+-{'-' * column_widths[field_name]}-")
        output.write("+\n")
        return output.getvalue()


def _get_fields(data):
    field_names = []
    column_widths = defaultdict(lambda: 0)
    for row in data:
        for field_name in row:
            if field_name not in field_names:
                field_names.append(field_name)
            column_widths[field_name] = max(
                column_widths[field_name], len(field_name), len(str(row[field_name]))
            )
return field_names, column_widths

It’s straightforward and simple. It currently cannot deal very well with cases in which dicts have different set of columns.

Should this be turned into a reusable library?

WordPress.com new editor handles titles right

I like my text properly formatted and with pretty much every CMS editor out there I always have some confusion when it comes to titles. The post or page has a title and then sections inside it also have titles and they are second level titles. On most CMS you have the option of titles or headers starting at level 1 through 6 (that maps to h1, h2, through h6 in HTML).

For example, this is Confluence, a tool that I really like:

The confusion that I get here is whether a subsection to this page should have Heading 1 or Heading 2. Sometimes Heading 1 will be displayed with the same font, size, etc as the title of the page, so, by using Heading 1 you are almost creating two pages in one. But in other CMS, Heading 1 is the top level section heading and the title of the page is a special title that will always sit above it.

The new WordPress.com editor does this correctly by being very clear that your first option for section headers, after setting the title of the page or post, is h2:

I think that was a very neat solution to the problem. Bravo WordPress.

Turning a list of dicts into a ReStructured Text table

I recently found myself having to prepare a report of some mortgage calculations so that non-technical domain experts could read it, evaluate it, and tell me whether my math and the way I was using certain APIs was correct.

Since I’m using Python, I decided to go as native as possible and make my little script generate a ReStructured Text file that I would then convert into HTML, PDFs, whatever. The result of certain calculations ended up looking like a data table expressed as list of dicts all with the same keys. I wrote a function that would turn that list of dicts into the appropriately formatted ReStructured Text.

For example, given this data:

creators = [{"name": "Guido van Rossum", "language": "Python"}, 
            {"name": "Alan Kay", "language": "Smalltalk"},
            {"name": "John McCarthy", "language": "Lisp"}]

when you call it with:

dict_to_rst_table(creators)

it produces:

+------------------+-----------+
| name             | language  |
+==================+===========+
| Guido van Rossum | Python    |
+------------------+-----------+
| Alan Kay         | Smalltalk |
+------------------+-----------+
| John McCarthy    | Lisp      |
+------------------+-----------+

The full code for this is:

from collections import defaultdict

from io import StringIO


def dict_to_rst_table(data):
    field_names, column_widths = _get_fields(data)
    with StringIO() as output:
        output.write(_generate_header(field_names, column_widths))
        for row in data:
            output.write(_generate_row(row, field_names, column_widths))
        return output.getvalue()


def _generate_header(field_names, column_widths):
    with StringIO() as output:
        for field_name in field_names:
            output.write(f"+-{'-' * column_widths[field_name]}-")
        output.write("+\n")
        for field_name in field_names:
            output.write(f"| {field_name} {' ' * (column_widths[field_name] - len(field_name))}")
        output.write("|\n")
        for field_name in field_names:
            output.write(f"+={'=' * column_widths[field_name]}=")
        output.write("+\n")
        return output.getvalue()


def _generate_row(row, field_names, column_widths):
    with StringIO() as output:
        for field_name in field_names:
            output.write(f"| {row[field_name]}{' ' * (column_widths[field_name] - len(str(row[field_name])))} ")
        output.write("|\n")
        for field_name in field_names:
            output.write(f"+-{'-' * column_widths[field_name]}-")
        output.write("+\n")
        return output.getvalue()


def _get_fields(data):
    field_names = []
    column_widths = defaultdict(lambda: 0)
    for row in data:
        for field_name in row:
            if field_name not in field_names:
                field_names.append(field_name)
            column_widths[field_name] = max(column_widths[field_name], len(field_name), len(str(row[field_name])))
    return field_names, column_widths

Feel free to use it as you see fit, and if you’d like this to be a nicely tested reusable pip package, let me know and I’ll turn it to one. One thing that I would need to add is making it more robust to malformed data and handle more cases of data that looks differently.

If I turn it into a pip package, it would be released from Eligible, as I wrote this code while working there and we are happy to contribute to open source.

Book Review: Scrum: The Art of Doing Twice the Work in Half the Time by Jeff Sutherland

scrum coverI first came in contact with Scrum when I was working at Google and since then, I’ve been applying it to the startups I co-founded with good outcomes. Since I was searching for a job, I kept seeing “Scrum Master” come up over and over and I thought it was about time that I learned all the details of Scrum to be able to be a proper Scrum master. Well, in only a couple of ways I finished the book and discovered I was already a proper Scrum master, having learned all the details about it from my time at Google and blog posts.

About the book itself, it’s short and entertaining with enough story telling to keep you engaged even if you only have a passing interest in Scrum. The system is rather simple, with only a few moving pieces and I’m glad of that. Simple systems tend to work better. The testimonials of how much productive a team is with Scrum feel exaggerated completely out of proportion, but then again, some companies are so terrible at producing anything at all, being the cradle of dysfunction, that is no surprise their productivity can be doubled or quadrupled.

★★★★☆

Buy The Art of Doing Twice the Work in Half the Time in USA

Buy The Art of Doing Twice the Work in Half the Time in UK

Book Review: Build the Fort: Why 5 Simple Lessons You Learned as a 10 year-old Can Set You Up for Startup Success by Chris Heivly

build the fort coverThis is a short and sweet book that compares the act of starting a company, a startup, with that of building a fort. It’s a very enjoyable read with the childhood stories of building a fort and it reminded me of those days in which it seemed easier to find a co-founder.

I’m not sure the book had anything new to me having already read so many other books about starting companies but it was still an enjoyable read and it’s good to get concepts refreshed. This book puts a lot of emphasis in socializing the idea and indeed that gave me a lot of food for though regarding how I’m going about my own startup.

I think it makes a very good first book about startups or a refresher for someone that’s been doing it for a while and needs a refresher.

★★★☆☆

Buy Build the Fort: Why 5 Simple Lessons You Learned as a 10 year-old Can Set You Up for Startup Success in USA

Buy Build the Fort: Why 5 Simple Lessons You Learned as a 10 year-old Can Set You Up for Startup Success in UK

Book Review: The Better Angels of Our Nature: Why Violence Has Declined by Steven Pinker

Better Angels of our NatureIf you are one of those people constantly watching news and worrying about the sorry state of the world. Read this book. It’ll convince you with data, instead of baseless sensationalism, that life has never been more peaceful, safer and better than right now.

I procrastinated on starting reading this book because I already fully buy Steven Pinker’s proposition that we are living the most peaceful time in human kind, but I’m glad I’m reading it now. This book is full of well researched nuggets of information that are very interesting.

I’m fascinated by the explanation of why USA is so violent compared to all other developed countries (having murder rates similar to third world countries). I’m not sure I buy all of it, but it’s interesting to see how things were different there compared to western Europe, Canada, etc.

Having been raised in one of the dysfunctional governments he describes as promoting violence through corruption and negligence, I was thrilled when he called them crappy governments, because that’s the category they have in my mind. I was a little bit miffed by him using the word “negro”. He used it to refer to groups of people back when those groups of people were referred to as negros and in the context of their class struggle, but it was still weird.

Another interesting aspect was how people that live in crappy governments do differently in some psychological tests showing how deep a dysfunctional authority can affect people and why countries that are at the top remain at the top and countries that are at the bottom, remain at the bottom.

This book is long, and in a good way. It feels it covers everything there’s to cover about the subject, so, it would be impossible for me to do a full review. All I’m going to say is this: read it. It’ll surprise you in at least one way. It may change how you see the world entirely.

★★★★★

Buy The Better Angels of Our Nature: Why Violence Has Declined in USA

Buy The Better Angels of Our Nature: Why Violence Has Declined in UK

The $10 weekend (or why selling to techies is sometimes so hard)

The first version of Dashman didn’t have a centralized configuration system to control a lot of displays at once. It was a one time fee software to run in one computer. Its price was $10.

I started networking, going to events, trying to find customers. I’m a techie, but I cleaned up, trimmed my beard, combed my hair, put on a suit (well, most of one… I still have issues with ties) and off I went, looking like this:

pupeno6

Techies stopped identifying me as part of their crowd which was enlightening. I was the target of so much condescension. It was funny. I still remember the face of a coder that told me he was building his application on Lisp and I asked “Which one?”. His face lit up when we discussed Common Lisp vs others, or SBCL vs other Common Lisp compilers. We became friends, but I digress.

One of my most enlightening discussions was with a developer that told me that he didn’t need my product because he could hack something together in a weekend. Let’s assume this is correct. He preferred to work for a weekend over spending $10. Or put in a different way, his weekend rate was $0.42 per hour. “Excuse me, can I hire you?”

Building something because it’s fun or educational is obviously a very rewarding activity and everybody that wants to do it should do it. When it comes to business, the equation of whether to build or buy should be a financial one.

Weird interaction with Google, was it Duplex?

For a few weeks I’ve been receiving this email from Google:

birth control

My first question is why is birth control an issue. Is Google limiting the advertisement of birth control? Why? Did we somehow slipped into the 19th century and nobody told me?

My second question was… how could they think I have anything to do with birth control. I guess that was nothing more than AI failing miserably, so, I decided to go and fix it. The important caveat here is that I haven’t run any ads in months, maybe more than a year.

I click the Fix button and and it took me to a black page with the “Unknown Business” title. There was no way to text support, so, when the emails got annoying enough, I called them.

The support guy was nice, but he couldn’t do much about it. I explained that I wasn’t running any ads and I didn’t plan on running any for now, but maybe in the future. He told me to ignore the emails.  I asked if there was a way to stop them and he told me to search for an unsubscribe link at the bottom of the email. That’s not what I meant, I want to fix the problem. All right, that was enough time on the phone, I’ll just ignore the emails.

Now is when it got weirder. The support guy said that his supervisor was there and wanted to talk to me. Ok… Click! Someone else starts speaking:

Supervisor: Hello Hos (their way of pronouncing José), did so-and-so answer your query today?

I’m always careful here. He didn’t solve the problem but I’m sure it wasn’t his fault. Most of the times I have an issue, it’s their system being broken and a support specialist shouldn’t be punished for that.

Me: Sort-of…

And as I was trying to explain the situation, the supervisor interrupt me:

Supervisor: Ok then. If you have any other questions, feel free to call us between 9 am and 5 pm.

Click! Hung up. Wow… that was rude… and odd. And now I’m thinking, did I just talk to Duplex and it failed at managing my answer?

When it comes to technology such as Duplex, my take is this: it’s going to happen no matter what, fighting it is futile, let’s try to figure out how to make the most out of it. But I have to admit having interacted with what I suspect was Duplex gives me an odd feeling (even if it was it). It makes me want to rebel, it makes me want to test it the next time I call to try to figure out if it’s a human or not. This is obviously useless; the only thing that matters is getting my issue resolved. What concerns me here is that if a technology-loving person such as myself is getting this strong reaction, how will the general population react?

I think we are going to have some interesting growing pains in the next couple of decades.

Book Review: WiX 3.6: A Developer’s Guide to Windows Installer XML by Nick Ramirez

This book was instrumental in me managing to package Dashman. It had pretty much everything I needed and most if not everything I learned from the book worked with the latest WiX 3.X.
One issue that I have with most WiX information out there is that it assumes you are working in a .Net language and using Visual Studio. I understand this is true for most WiX users but it’s not for me, as I was building a Java application. If you are in the same situation as me, I recommend reading chapter 1 and then immediately jumping to chapter 9 that explains how to use all the different parts of WiX from the command line. After that go back to chapter 2 and work yourself over to chapter 8.
I haven’t read a lot past chapter 9 because it gets into a lot of very advanced stuff that I didn’t need. I feel that a lot of computer books these days are just introductions. It’s nice to see one that goes so deep into a subject that I find myself not needing to read it, instead of wondering where’s the next more advanced book.
I wish more computer books were like this (but I do understand why they are not).

★★★★★

Buy WiX 3.6: A Developer’s Guide to Windows Installer XML in USA

Buy WiX 3.6: A Developer’s Guide to Windows Installer XML in UK

I had to stop watching Lost in Space

I’m referring to the new series, the Netflix one. I’ve never really watched the old one but I’ve seen enough clips to know: “Danger Will Robinson”. There are several reasons why I stopped watching even though I’m starving for non-pessimistic future space science fiction.

The first one is that I don’t enjoy seeing people make obvious bad decisions. When it was the kids making bad decisions it was annoying but it made sense, they are kids after all. But when it’s adults, I can’t stand it. These are supposed to be a selection of the most intelligent and capable adults on earth, and yet, they constantly make bad decisions.

Some spoilers ahead. Stop reading here if you don’t want to be exposed to them.

For example, Maureen Robinson decided to go do an experiment with a high altitude balloon, on her own, without telling anybody, on a planet of unknown levels danger. She should have taken a few people with her and notified other people where she was going to be. This is what we do on earth when we go hiking or some other wilderness adventure. At the very least her ex-husband would have jumped into the opportunity as he’s hungry for her acceptance but also being a former military man, he’s a good asset. This almost cost her life in a very stupid way.

Ignacio Serricchio, the mechanic, not notifying everybody what happened with Dr. Smith. Judy Robinson not notifying everybody what happened with Dr. Smith. See a pattern? There’s a lot of information hoarding. When you are in a life-or-death situation in such a small community of survivals, you don’t hoard information. You share it so that if they are part of a bigger puzzle, someone can put it together.

I had to stop watching when Maureen Robinson discovered their impending doom and didn’t immediately tell everybody. I don’t know if she’ll tell them afterwards but just considering not doing was the straw that broke the camel’s back. I sort of understand keeping doom-predicting information from the general population of Earth, to avoid panic, specially if the average person can’t do anything about it. This is not the case. They are a elite, they are a small community of highly capable people that should know if they are doomed to work their asses off to solve the problem, using resources that maybe they would have been saving for the future otherwise. A future that doesn’t exist.

Oh… and electricity, physics, astrophysics and medicine are all horribly wrong. Not a little wrong, not a little exaggerated: horribly wrong.