Helmut.dev

Understanding the difference between .get and .first in Django

2021-05-21T07:30:00+02:00

When retrieving a single record in Django, the ORM offers two possible methods: .get() and .first(). Let's learn how each one works and when to use them.

Get

Django's most basic way to retrieve a single record is the .get() method. It is used as follows:

burger = Menu.objects.get(name="Beef Burger")

Quick facts about .get():

It retrieves only one record.
If no record exists that meets the given criteria, it raises a DoesNotExist exception.
If more than one record with the given criteria exists, it raises a MultipleObjectsReturned exception.

First

A different way to get one record is to use .first() on a QuerySet. For example:

burger = Menu.objects.filter(name="Beef Burger").first()

Quick facts about .first():

It will return the first item of a recordset.
If no record exists, it returns None.

Handling unexpected situations

Get

If Django's ORM can't return one record (i.e. there is either no record, or there are more than one), you have to catch the relevant exception. For example:

try:
    return Menu.objects.get(name="Beef Burger")
except Menu.DoesNotExist:
    raise Exception("Could not find a burger")
except Menu.MultipleObjectsReturned:
    raise Exception("Found too many burgers")

First

When using .first(), the only unexpected situation is if no record was present in the QuerySet. Since it's not a problem if multiple records exist that meet the criteria, we have to deal only with the one case. For example:

burger = Menu.objects.filter(name="Beef Burger").first()
if not burger:
    raise Exception("Could not find a burger")

return burger

Subtle differences

Did you notice the differences between the two above? For .get() we have to deal with more "error" scenarios. But why is that? The .get() method expects no more and no less than one record to be returned, whereas .first() assumes there might be multiple records, and it returns the first. So we have a fundamental difference in the assumptions each of these makes.

Should we care? Does it really make a difference which of the two methods we use if we get the same result?

I used to think it didn't matter, but I was wrong.

Using .get() might point to a problem with our Model that we are unaware of. If we are expecting only one record to be returned, but we get back multiple, we have failed to make sure that one of the fields (or multiple ones) enforce a unique constraint. Handling this at a database layer is crucial. If we had used .first(), and had not added a unique constraint, we might have never noticed if there were multiple records where we are expecting only one.
What about readability? I never thought about how the code reads in each case, and what it tells the reader about the data models. When you use .get(), the reader will (or should) know immediately that the record should be unique for the fields you filtered on. However, if you use .first(), it implies to the reader that multiple records might exist, but you only care about the first one.

When to use which

Get

Use .get() when you are expecting only one (or a maximum of one) record to be returned. The code makes your intentions clear, and there's less room for undetected errors.

First

Use .first() when you know there might be multiple records (and that's ok), but you only care about the first. In most cases you should also have some order defined on the model (or specified on your query), otherwise the first record might be random. In some cases you might not even care about the order, but just be aware of it.

Gotchas

If you use .first(), you run the risk of using the None (in the case of no record found) as a different return data type to signify the fact that nothing was found. It's not very likely, but I've caught myself doing this. I wrote about that trap here.
The SQL query generated by .get() does not have a limit. So if you have multiple records, before the MultipleObjectsReturned exception is raised, your database will return all the records matching that criteria. But using .first() adds a LIMIT 1 to your query, so it will only ever return one record. Be careful which one you choose.

Summary

If you query the database and only ever expect one record (maximum) to be returned, don't use .first(), but rather .get() instead. You'll thank yourself later.

Using Exceptions instead of Dynamic Typing

2021-05-17T22:00:00+02:00

Python's dynamic typing is great, right? You can just do whatever you want with variables. FREEDOM! ;-) This comes at a price though, and sometimes one we're not always aware of.

Trying not to fail

For years I took advantage of Python's dynamic typing by having functions return an error state via a different data type. For example:

def double(int_input):
    if not isinstance(int_input, int):
        return "An integer must be provided"

    return int_input * 2

And consuming this function would have looked like this:

doubled_int = double(user_input)
if isinstance(doubled_int, str):
    print("You did not provide an integer")
    return

print(f"Your doubled value: {doubled_int}")

Seems kind of legit. Performance wise there shouldn't be any problems, right?

But what we're ignoring here is maintainability. This is an over-simplified example, but if you have longer functions with multiple return statements, and you're using more than just one data type to signify different things, this can quickly become a problem.

If another developer has to use or modify this function somewhere down the line, it becomes more work to understand what the function is returning in which cases. They will need to read through the function in order to know which return types to expect, and what they mean.

The Catch

Can we improve on this? Can we make it easier for that developer to use or modify that function? I think there is a way, and that is using exceptions.

Exceptions are a built-in part of Python, and for some reason, over the years, I made a habit of trying to avoid raising of exceptions. But in this case they can serve us pretty well. Our goal is to have a function that has a consistent return data type. And any problems are caught by said exceptions. If we now refactor the function above, we get this:

def double(int_input):
    if not isinstance(int_input, int):
        raise ValueError("An integer must be provided.")

    return int_input * 2

Simple, right? Instead of returning a string, we just raise an exception. So far we haven't complicated anything, but the benefit will become apparent in the next step. We are now going to consume this refactored function and catch the exception.

try:
    doubled_int = double(user_input)
    print(f"Your doubled value: {doubled_int}")
except ValueError:
    print("You did not provide an integer")

Finally

So, what exactly happened? We use a try/catch block to catch this new exception that we raised. If everything goes well, our integer is doubled and printed to the screen. However, if we provided a wrong value, the exception is caught and we print an error to the screen.

Have we gained anything? It has come at the cost of an extra line of code, but I think it's clear that we have gained in readability. For any Python developer it is now immediately clear that this function might result in an error, and it's also clear what happens in that case.

Maintainability has also improved. Any future consumer of this function doesn't need to wonder about the different return types that they could/should be expecting. And if the function is modified, one can now just raise other exceptions if new error cases are added. Again, the return type stays consistent.

Write better Django views

2021-03-12T07:00:00+02:00

Should you use class-based views (CBV's) or function-based views (FBV's) in Django? It seems a lot of people are pushing CBV's, touting them to be the "standard" way of writing views in Django. But why do FBV's still exist then? Just for backward-compatibility? This is my highly opinionated "view" (see what I did there??) on this matter.

The topic is hotly debated. Why are some people so passionate about CBV's? CBVs are said to be better because they abstract a lot of boiler-plate code into base classes and mixins. This is true. As a very basic example, instead of writing code to generate and return a rendered template, you can just use an existing base class for your view, and all you have to specify is the template name. And boom, Bob's your mother's cousin's uncle, twice removed. Seems like a no-brainer, right? We're saving at least 1-2 lines of code! (Yes, really)

But not all views are that simple. Soon you have CBVs that inherit from multiple base classes and mixins, you've overridden 7 methods of those classes, but your view still doesn't work. To figure out why, you read the documentation of all those classes, some of which have only their source code as documentation. You have 17 tabs open in your browser and it's a mess. Eventually you figure it out, and the problem was that the order in which those methods are run is different than what you expected. Is this all really worth it? Maybe you can figure these things out once, and then you just profit! But what about the next developer. They will read the code, and will have to go through the same process to figure out what is happening when and how. And if you don't touch that code again for the next months, will you remember these things next time?

I think you can see where I'm heading. One very important thing we're forgetting is the Zen of Python. In particular, I'm thinking of "Simple is better than complex", and "Explicit is better than implicit". CBVs are certainly not explicit. There's hidden stuff going on. A lot. And they are also not simple - there's a huge tree hierarchy behind the different base classes that Django provides. They hide tons of logic that is not obvious to the developer, and if you don't go and read that internal logic, you might have a hard time using those classes. But FBVs address these issues pretty well. They are explicit: you have one method that shows you all the logic, sequentially. And they are simple: there is no "API" that you need to know - no base classes that have complex, hidden logic. Ok, but what about the down-sides? What if you have many views with common logic? Then surely CBVs are the way to go? Yes and no. In a few cases I admit that CBVs could possibly, maybe, conceivably be beneficial. But for most cases FBVs will give you more advantages. And to reduce the repetition of common logic, there are alternative means.

I think the big issue is what we've been taught about abstraction and "DRY". Apart from solving a problem, those are the main things we do when coding. We actively look for things to abstract. And often (preaching to myself here mainly) we pull the trigger way too early. "Hey, this seems like a generic bit of code that we'll surely use again later!" is something I think to myself regularly. But at this point DRY is not even applicable yet, because we haven't repeated ourselves yet. But even if you find code in multiple places that is similar, we should be very careful to immediately create an abstraction. Instead, we should be thinking about how readable and maintainable that code would become as a result. And arguably, readability and maintainability will be more profitable to our employers than saving a few lines of code. Sometimes we should indeed abstract in order to reduce boiler-plate code, but when we do, we must do it veeeewy cawefuwy (my Elmer Fudd impression).

I get it. There's hardly a better feeling for developers than when you've written a piece of very clever code, and it just works. It might enable other developers (or your future self) to handle similar cases with just a line or two of code, and the thought of that appeals to us. But what we're forgetting is that often "similar" cases are just different enough that the abstraction needs to be tweaked ever so slightly. And this carries on, until the abstraction is nothing short of a Frankenstein. We've all been there, right? So why do we still go down that path?

Abstraction and DRY are at the heart of CBVs - and in most cases we save just a few lines of code. But it hurts the maintainability of the code in other ways. Future developers will spend hours figuring out the logic of the views, just so that there can be less code. Yes, fewer lines means less code to maintain. But often those fewer lines are harder to maintain because of the abstractions.

There. I've said it. You might not agree, especially if you've used CBV's for many years, and you're so used to them that they are not an issue for you at all. Often, when we're used to something, we don't see the down-sides anymore. So I hope this has helped in that sense.

Now, what I haven't done is SHOW you the differences between the two approaches, to prove the things I've claimed. But there's a resource that does a very, very excellent job at that, and you should go and read ALL of it, as soon as possible. It seriously blew my mind. My eternal thanks go to Luke Plant for spending hours upon hours to create this.

Django Views — The Right Way

For extra credit, go and read up about the pitfalls of abstractions. Abstractions have their place, but sometimes we think that place is "everywhere". At least that's how I code at times. Anyway, enough now. Go forth and code simply and explicitly. ;-)

Learn the subtle differences between Save and Update in Django

2021-02-12T07:10:00+02:00

To save data in Django, you normally use .save() on a model instance. However the ORM also provides a .update() method on queryset objects. So which one should you use? Let's take a look at each, and then decide which one to use in which situations.

Save

The .save() method is used to write a model instance to the database. It can be an existing record or even a new one. For an existing record, Django will run a SQL UPDATE statement on the database. For a new record, Django will run an INSERT. Before Django 3.0, it would also do a SELECT before the INSERT, to make sure that the primary key doesn't exist (if one was given), and do an UPDATE instead if it found something.

Examples:

new_book = Book(title="How to code", author="Mike Lambda", price=9)
new_book.save()

book = Book.objects.get(title="Debugging like a pro")
book.price += 5
book.save()

When updating an existing record, Django will update every field. That means, if you changed only one field on the model instance (like the price in the example above), all the fields will be updated (with the values staying the same). This is rather inefficient, right? Especially if you model has many, many fields. And there's another problem. If there's a significant amount of time in between the fetching of the record and the saving of it, it is possible that another request will have updated one of the fields on that record, and your model instance still has the old value. If you call .save() now, it will overwrite the new value that someone else saved. These two problems can be prevented by using the update_fields parameter. It expects a list of fields (as strings) that you want to save. So if you have only a few fields to save, use this parameter to prevent such data overwrite problems. The UPDATE that is run against the database then only updates those fields that you have specified.

In the example above, it would change to:

book.save(update_fields=["price"])

When to use Save

You are creating a new record (an alternative is using the .create() method).
You have a model instance already, and you need to change some field values.
You've overridden the save method to include some custom model logic.
You've defined pre_save and post_save signals for that model.
The changes you need to make are dynamic, based on current field values (e.g. incrementing a value - see the example above).
You want to update fields from a related model (like a ForeignKey).

Update

The .update() method is available on each queryset object of the ORM. Using it will typically look like this:

Book.objects.filter(price__lt=10).update(price=10)

This would update the price of all books that are below a certain price. While this is for a range of records, it could also be used as in our first example, to update a single record:

Book.objects.filter(title="Debugging like a pro").update(price=15)

You'll notice that this doesn't increment the price like it does in the first save example. You would need to use an F expression for that, but that's beyond the scope for now.

Update cannot be used to create a new record, like save can.

When to use Update

You are updating fields of one or many records
You don't need to re-use a model instance after updating
You don't need your overridden save method or any pre_save or post_save signals to run
You know what your new field values must be without having retrieved the record(s)

Summary

If you need to update field values, without any other model logic needing to be run, and you know what the changes will be without fetching the record(s) first, use update. The SQL that is run against the database will be more efficient. However, if you need to, or have already, loaded the record from the database, and you possibly need the change to be dynamic based on existing data, or there's attached model logic that needs to run, use save.

I hope this has been helpful to you. If you have any questions or comments, please don't hesitate to reach out to me. I'd be glad to have a chat.

Results - VS Code vs PyCharm

2021-02-02T07:00:00+02:00

I asked Reddit about text editors vs IDEs. In particular, I asked VS Code users why they preferred it over IDEs like PyCharm, and then I asked PyCharm users why they preferred it over text editors like VS Code. The results were very interesting.

VSCode users: Why do you prefer it over PyCharm?

PyCharm users: Why do you prefer it over VSCode or other editors?

Results: VS Code

Among VS Code users, the following reasons were listed most:

Results: PyCharm

And among PyCharm users, the following reasons were most common:

Conclusion

Looking at the above, I can conclude a few things:

Users who work with multiple languages frequently prefer VS Code, whereas users who work mainly with Python prefer PyCharm.
Users who don't want to spend much time on set up and configuration prefer PyCharm, because of it's many out-of-the-box features.
Users who like flexibility prefer VS Code because of its vast plugin ecosystem that lets them add virtually any needed functionality.
The price of PyCharm Professional seems to be a significant factor.
PyCharm users say that such deep language integration cannot be found in any text editor.

I myself am a PyCharm Pro user. The price is worth it to me, and my top reasons why I prefer it are:

Deep language integration: Even though Python support in VS Code is very good, PyCharm has even better integration. This ranges from smart autocomplete to stellar refactoring tools. To a limited degree, VS Code has these as well (with plugins), but PyCharm takes it to another level.
I like not having to look for and install multiple plugins before having the functionality I need. Plugins also make you dependent on 3rd parties, whereas in PyCharm everything is built-in and always works seamlessly.

Speed up bulk-exists check with python sets

2020-08-01T08:00:00+02:00

The .exists() function of Django's ORM is very useful. However, if you need to do this in bulk (think hundreds of thousands or more), this becomes a strain on your database. Let's say you are fetching records from an external system, and if the record doesn't exist locally, you need to do something:

for item in external_records:
    if not Data.objects.filter(external_id=item.id).exists():
        # Do something

If you are working with a large quantity of records, this will flood your DB with queries. Instead, you could first pull all external_id values from the DB:

existing_ids = Data.objects.all().values_list("external_id", flat=True)
for item in external_records:
    if item.id not in existing_ids:
        # Do something

But again, for large volumes of data, this will be slow in terms of computing time. The trick is to use a set instead of a list. The python in operator performs much, much better for sets, since items in sets are guaranteed to be unique. The above code thus becomes:

existing_ids = set(Data.objects.all().values_list("external_id", flat=True))
for item in external_records:
    if item.id not in existing_ids:
        # Do something

A very simple change that increases performance dramatically.

(Code above can be found on GitHub)

Early Exit

2020-07-12T07:46:00+02:00

You will have definitely come across the following pattern often:

if post_data:
    thing = post_data.get("thing")
    if thing is not None:
        setting = get_user_setting(user, thing)
        if setting is not None:
            permission = get_user_permission(user)
            if permission is True:
                if user.is_superuser:
                    return DataPoint.objects.all()
                elif user.is_staff:
                    return DataPoint.objects.filter(user=user)
                else:
                    return DataPoint.objects.filter(user=user, thing=thing)
            else:
                return "Permission denied"
        else:
            return "Setting not found"
    else:
        return "Thing not specified"
else:
    return "No data posted"

What is good about the code above? It follows the thought process that most developers would have: which conditions must be met for me to execute the core logic of this feature? So I check each of those conditions and proceed if it is met. Then, in the middle of the code block, I have my core logic bundled up nicely. And at the end I deal with the cases where the conditions were not met.

But are there any downsides?

The condition checks are quite disjointed. If you want to see what happens when a condition is not met, you have to carefully follow the indentation down to where the corresponding else is. This is not ideal, but often accepted by a lot of developers.
The indentation can become very acute if there are many layers of conditions.

Is there a better way? Yes, there is: the early exit. Instead of following the thought process of "Which conditions must be met?", we rather ask "Which conditions can I eliminate right off the bat?" Here's how we would rewrite the code above:

if not post_data:
    return "No data posted"

thing = post_data.get("thing")
if thing is None:
    return "Thing not specified"

setting = get_user_settig(user, thing)
if setting is None:
    return "Setting not found"

permission = get_user_permission(user)
if permission is False:
    return "Permission denied"

if user.is_superuser:
    return DataPoint.objects.all()

if user.is_staff:
    return DataPoint.objects.filter(user=user)

return DataPoint.objects.filter(user=user, thing=thing)

For each condition that is not met, we return immediately, since no further processing is necessary. And then at the end we have our core logic bundled up again.

This does a few things for us.

It still keeps the core logic in one place.
The conditions are also bundled up with their corresponding logic.
We don't have the indentation problem any more.
This may be just me, but I find the refactored code easier to read.

There is one caveat to this pattern: it mostly makes sense if you return something when a conditions is not met. If your application logic needs to not return anything, but for example append an error message to a list instead (and then continue), the above might have to be structured differently.

But you'll find tons of places where this pattern is applicable, and if it makes sense in your scenario, go ahead and use it liberally. It's awesome.

(Code above can be found on Github)

Faster updating of dictionaries

2020-07-09T17:18:00+02:00

When updating a dictionary, no one would object to this:

ages["Pete"] = 42

But what about this?

ages["Tom"] = 29
ages["Linda"] = 37
ages["Pam"] = 62
ages["Coraline"] = 21

Still fine, right? There is another way of writing this, using the .update() method.

ages.update(
    {
        "Tom": 29,
        "Linda": 37,
        "Pam": 62,
        "Coraline": 21,
    }
)

Which one do you prefer? Would you say it's a matter of preference? I always preferred using the .update() method when there were at least 3 updates being done. But one day I wondered about the performance of each method. Let's see if one is slower than the other.

Using Python's timeit, we run the following code to test our first method. This updates the dictionary three times.

ages = {}; ages["Tom"] = 29; ages["Linda"] = 37; ages["Pam"] = 62

Then, using the same method, we test .update().

ages = {}; ages.update({"Tom": 29, "Linda": 37, "Pam": 62})

The results?

Straight assignment: 225 nsec per loop

Using .update(): 522 nsec per loop

And the pattern stays linear with more keys.

So we see that straight assignment is about twice as fast. Does that mean we never use .update()? No. Like so many other things, it depends on your specific case.

The Case for Assignment

If you're updating a few keys, and your new data is not already gathered in a second dictionary, it probably makes sense to just use normal assignment.

The Case for Update

If you've got new data in a second dictionary (either a few items or many), and you'd like to update the first dictionary with these items, I would use the .update() method. It makes for cleaner code.