The Rule Everyone Misses: How to Stop Confusing loc and iloc in Pandas

with pandas, you’ve in all probability came upon this basic confusion: do you have to use loc or iloc to extract information? At first look, they appear virtually an identical. Each are used to slice, filter, and retrieve rows or columns from a DataFrame — but one tiny distinction in how they work can utterly change your outcomes (or throw an error that leaves you scratching your head).

I keep in mind the primary time I attempted choosing a row with df.loc[0] and questioned why it didn’t work. The rationale? Pandas doesn’t at all times “assume” when it comes to positions — typically it makes use of labels. That’s the place the loc vs iloc distinction is available in.

On this article, I’ll stroll via a easy mini mission utilizing a small scholar efficiency dataset. By the top, you’ll not solely perceive the distinction between loc and iloc, but in addition know precisely when to make use of every in your individual information evaluation.

Introducing the dataset

The dataset comes from ChatGPT. It incorporates some primary scholar examination rating information. Right here’s a snapshot of our dataset

import pandas as pd
df = pd.read_csv(‘student_scores.csv’)
df

Output:

I’ll attempt to carry out some information extraction duties utilizing loc and iloc, like

• Extracting a single row from the DataFrame
• Extracting a single worth
• Extracting a number of rows
• Slicing a spread of rows
• Extracting particular columns and
  Boolean Filtering

First, let me briefly clarify what loc and iloc are in Pandas.

What’s loc and iloc

Loc and iloc are information extraction strategies in Pandas. They’re fairly useful for choosing information from information.

Loc makes use of labels to retrieve information from a DataFrame, so I discover it simpler to make use of. Iloc, nevertheless, are useful for a extra exact retrieval of information, as a result of iloc selects information primarily based on the integer positions of the rows and columns, much like how you’ll index a Python listing or array.

However in the event you’re like me, you is likely to be questioning. If loc is clearly simpler due to row labels, why trouble utilizing iloc? Why trouble attempting to determine row indexes, particularly in the event you’re coping with giant datasets? Listed below are a few causes.

• A number of occasions, datasets don’t include neat row indexes (like 101, 102, …). As an alternative, you’ve a plain index (0, 1, 2, …), otherwise you would possibly misspell row labelling when retrieving information. On this case, you’re higher off utilizing iloc. Later on this article, it’s one thing we’ll be addressing additionally.
• In some situations, like machine studying preprocessing, labels don’t actually matter. You solely care a couple of snapshot of the information. As an illustration, the primary or final three information. iloc is basically useful on this state of affairs. iloc makes the code shorter and fewer fragile, particularly if labels change, which may break your machine studying mannequin
• A number of datasets have duplicate row labels. On this case, iloc at all times works since positions are distinctive.
• The underside line is, use loc when your dataset has clear, significant labels and also you need your code to be readable.
• Use iloc if you want position-based management, or when labels are lacking/messy.

Now that I’ve cleared the air, right here’s the essential syntax for loc and iloc under:

df.loc[rows, columns]
df.iloc[rows, columns]

The syntax is just about the identical. With this syntax, let’s attempt to retrieve some information utilizing loc and iloc.

Extracting a single row from the DataFrame

To make a correct demonstration, let’s first change the column index and make it student_id. At present, pandas is auto-indexing:

# setting student_id as index
df.set_index('student_id', inplace=True)

Right here’s the output:

Seems higher. Now, let’s attempt to retrieve all of Bob’s information. Right here’s tips on how to method that utilizing loc:

df.loc[102]

All I’m doing right here is specifying the row label. This could retrieve all of Bob’s information.

Right here’s the output:

title   Bob
math    58
english 64
science 70
Title: 102, dtype: object

The cool factor about that is that I can drill down, kinda like a hierarchy. As an illustration, let’s attempt to retrieve particular data about Bob, like his rating on math.

df.loc[102, ‘math’]

The output can be 58.

Now let’s do this utilizing iloc. Should you’re acquainted with lists and arrays, indexing at all times begins at 0. So if I wish to retrieve the primary report within the DataFrame, I’ll need to specify the index 0. On this case, I’m attempting to retrieve Bob, which is the second row in our DataFrame — so, on this case, the index can be 1.

df.iloc[1]

We’d get the identical output as above:

title   Bob
math    58
english 64
science 70
Title: 102, dtype: object

And if I attempt to drill down and retrieve the maths rating of Bob. Our index would even be 1, provided that math is on the second row

df.iloc[1, 1]

The output can be 58.

Alright, I can wrap this text up right here, however loc and iloc provide some extra spectacular options. Let’s speed-run via a few of them.

Extract A number of Rows (Particular College students)

Pandas means that you can retrieve a number of rows utilizing loc and iloc. I’m gonna make an illustration by retrieving the information of a number of college students. On this case, as an alternative of storing a single worth in our loc/iloc technique, we’d be storing a listing. Right here’s how you are able to do that with loc:

# Alice, Charlie and Edward's information
df.loc[[101, 103, 105]]

Right here’s the output:

And right here’s how to do this with iloc:

df.iloc[[0, 2, 4]]

We’d get the identical output:

I hope you’re getting the hold of it.

Slice a Vary of Rows

One other useful characteristic Python Pandas provides is the flexibility to slice a spread of rows. Right here, you possibly can specify your begin and finish place. Right here’s the syntax for loc/iloc slicing:

df.loc[start_label:end_label]

In loc, nevertheless, the top label can be included within the output — fairly totally different from the default Python slicing.

The syntax is similar for iloc, with the exception that the top label can be excluded from the output (identical to the default Python slicing).

Let’s stroll via an instance:

I’m attempting to retrieve a spread of scholars’ information. Let’s attempt that utilizing loc:

df.loc[101:103]

Output:

As you possibly can see above, the top label is included within the end result. Now, let’s attempt that utilizing iloc. Should you recall, the primary row index can be 0, which might imply the third row can be 2.

df.iloc[0:3]

Output:

Right here, the third row is excluded. However in the event you’re like me (somebody who questions issues so much), you is likely to be questioning, why would you need the final row to be excluded? In what situations would that be useful? What if I informed you it truly makes your life simpler? Let’s clear that up actual fast.

Assuming you wish to course of your DataFrame in chunks of 100 rows every.

If slicing had been inclusive, you’d need to do some awkward math to keep away from repeating the final row.

However as a result of slicing is unique on the finish, you are able to do this fairly simply, like so.

df.iloc[0:100] # first 100 rows
df.iloc[100:200] # subsequent 100 rows
df.iloc[200:300] # subsequent 100 rows

Right here, there will probably be no overlaps, and there will probably be constant chunk sizes. Another excuse is the way it’s much like how ranges work in Pandas. Normally, if you wish to retrieve a spread of rows, it additionally begins at 0 and doesn’t embrace the final row. Having this identical logic in iloc slicing is basically useful, particularly if you’re engaged on some net scraping or looping via a spread of rows.

Extract Particular Columns (Topics)

I’d additionally like to introduce you to the colon : signal. This lets you retrieve all information in your DataFrame utilizing loc. Just like the * in SQL. The cool factor about that is you could filter and extract a subset of columns.

That is often the place I discover myself beginning. I exploit it to get an summary of a specific dataset. From there, I can begin to filter and drill down. Let me present you what I imply.

Let’s retrieve all information:

df.loc[:]

Output:

From right here, I can extract particular columns like so. With loc:

df.loc[:, [‘math’, ‘science’]]

Output:

With iloc:

df.iloc[:, [2, 4]]

The output can be the identical.

I like this characteristic as a result of it’s so versatile. Let’s say I wish to retrieve Alice and Bob’s math and science scores. It’ll go one thing like this. I can simply specify the vary of information and columns I need.

With loc:

df.loc[101:103, ['name', 'math', 'science']]

Output:

With iloc:

df.iloc[0:3, [0, 1, 3]]

We’d get the identical output.

Boolean Filtering (Who scored above 80 in Math?)

The ultimate characteristic I wish to share with you is Boolean filtering. This permits for a extra versatile extraction. Let’s say I wish to retrieve the information of scholars who scored above 80 in Math. Normally, in SQL, you’ll have to make use of the WHERE and HAVING clauses. Python makes this really easy.

# College students with Math > 80.
df.loc[df['math'] > 80]

Output:

It’s also possible to filter on a number of circumstances utilizing the AND(&), OR(|), and NOT(~) operators. As an illustration:

# Math > 70 and Science > 80
df.loc[(df[‘math’] > 70) & (df[‘science’] > 80)]

Output:
P.S. I wrote an article on filtering with Pandas. You may learn it here

Normally, you’ll end up utilizing this characteristic with loc. It will possibly get a bit sophisticated with iloc, because it doesn’t assist Boolean circumstances. To do that with iloc, you’ll need to convert the Boolean filtering into a listing, like so:

# College students with Math > 80.
df.iloc[list(df['math'] > 80)]

To keep away from the headache, simply go together with loc.

Conclusion

You’ll in all probability use the loc and iloc strategies so much if you’re engaged on a dataset. So it’s essential to understand how they work and distinguish the 2. I like how simple and versatile it’s to extract information with these strategies. Everytime you’re confused, simply keep in mind loc is all about labels whereas iloc is about positions.

I hope you discovered this text useful. Strive working these examples by yourself dataset to see the distinction in motion.

I write these articles as a solution to take a look at and strengthen my very own understanding of technical ideas — and to share what I’m studying with others who is likely to be on the identical path. Be at liberty to share with others. Let’s study and develop collectively. Cheers!

Be at liberty to say hello on any of those platforms

Medium

LinkedIn

Twitter

YouTube