How to Split a String by Whitespace in Python: Brute Force and split()

Once again, I’m back with another look at some ways to solve a common Python problem. This time, we’ll be looking at how to split a string by whitespace (and other separators) in Python.

If you’re in a rush, here’s the key takeaway. You could write your own whitespace splitting function, but they’re often slow and lack robustness. Instead, you should probably opt for Python’s builtin split() function. It works for any string as follows: "What a Wonderful World".split(). If done correctly, you’ll get a nice list of substrings without all that whitespace (e.g. ["What", "a", "Wonderful", "World"]).

In the remainder of this article, we’ll look at the solution described above in more detail. In addition, we’ll try writing our own solution. Then, we’ll compare them all by performance. At the end, I’ll ask you to tackle a little challenge.

Let’s get started!

Problem Description

When we talk about splitting a string, what we’re really talking about is the process of breaking a string up into parts. As it turns out, there are a lot of ways to split a string. For the purposes of this article, we’ll just be looking at splitting a string by whitespace.

Of course, what does it mean to split a string by whitespace? Well, let’s look at an example:

Here, the only two whitespace characters are the two spaces. As a result, splitting this string by whitespace would result in a list of three strings:

Of course, there are a ton of different types of whitespace characters. Unfortunately, which characters are considered whitespace are totally dependent on the character set being used. As a result, we’ll simplify this problem by only concerning ourselves with Unicode characters (as of the publish date).

In the Unicode character set, there are 17 “separator, space” characters. In addition, there are another 8 whitespace characters which include things like line separators. As a result, the following string is a bit more interesting:

With the addition of the line break, we would expect that splitting by whitespace would result in the following list:

In this article, we’ll take a look at a few ways to actually write some code that will split a string by whitespace and store the result in a list.

Solutions

As always, there are a lot of different ways to split a string by whitespace. To kick things off, we’ll try to write our own solution. Then, we’ll look at a few more practical solutions.

Split a String by Whitespace Using Brute Force

If I were given the problem description above and asked to solve it without using any libraries, here’s what I would do:

Here, I decided to build up a few variables. First, we need to track the end result which is items in this case. Then, we need some sort of string to work with (e.g. my_string).

To perform the splitting, we’ll need to track a couple indices: one for the front of each substring (e.g. start_index) and one for the back of the substring (e.g. end_index).

On top of all that, we need some way to verify that a character is in fact a whitespace. To do that, we created a list of whitespace characters called whitespace_chars. Rather than listing all of the whitespace characters, I cheated and showed two examples with a little ellipses. Make sure to remove the ellipsis before running this code. For some reason, Python gives those three dots meaning, so it won’t actually error out (although, it likely won’t cause any harm either).

Using these variables, we’re able to loop over our string and construct our substrings. We do that by checking if each character is a whitespace. If it is, we know we need to construct a substring and update start_index to begin tracking the next word. Then, when we’re done, we can grab the last word and store it.

Now, there’s a lot of messiness here. To make life a bit easier, I decided to move the code into a function which we could modify as we go along:

Now, this solution is extremely error prone. To prove that, try running this function as follows:

Notice how having two spaces in a row causes us to store empty strings? Yeah, that’s not ideal. In the next section, we’ll look at a way to improve this code.

Split a String by Whitespace Using State

Now, I borrowed this solution from a method that we ask students to write for a lab in one of the courses I teach. Basically, the method is called “nextWordOrSeparator” which is a method that looks like this:

One way to implement this method is to check whether or not the first character is a separator. If it is, loop until it’s not. If it’s not, loop until it is.

Typically, this is done by writing two separate loops. One loop continually checks characters until a character is in the separator set. Meanwhile, the other loop does the opposite.

Of course, I think that’s a little redundant, so I wrote my solution using a single loop (this time in Python):

Here, we track a couple variables. First, we need an end_index, so we know where to split our string. In addition, we need to determine if we’re dealing with a word or separator. To do that, we check if the character at the current position in text is in separators. Then, we store the result in is_separator.

With is_separator, all there is left to do is loop over the string until we find a character that is different. To do that, we repeatedly run the same computation we ran for is_separator. To make that more obvious, I’ve stored that expression in a lambda function:

At any rate, this loop will run until either we run out of string or our test_separator function gives us a value that differs from is_separator. For example, if is_separator is True then we won’t break until test_separator is False.

Now, we can use this function to make our first solution a bit more robust:

Unfortunately, this code is still wrong because we don’t bother to check if what is returned is a word or a separator. To do that, we’ll need to run a quick test:

Now, we have a solution that is slightly more robust! Also, it gets the job done for anything we consider separators; they don’t even have to be whitespace. Let’s go ahead and adapt this one last time to let the user enter any separators they like:

Then, when we run this, we’ll see that we can split by whatever we like:

How cool is that?! In the next section, we’ll look at some builtin tools that do exactly this.

Split a String by Whitespace Using split()

While we spent all this time trying to write our own split method, Python had one built in all along. It’s called split(), and we can call it on strings directly:

In addition, we can provide our own separators to split the string:

However, this method doesn’t work quite like the method we provided. If we input multiple separators, the method will only match the combined string:

In the documentation, this is described as a “different algorithm” from the default behavior. In other words, the whitespace algorithm will treat consecutive whitespace characters as a single entity. Meanwhile, if a separator is provided, the method splits at every occurrence of that separator:

But, that’s not all! This method can also limit the number of splits using an additional parameter, maxsplit:

How cool is that? In the next section, we’ll see how this solution stacks up against the solutions we wrote ourselves.

Performance

To test performance, we’ll be using the timeit library. Essentially, it allows us to compute the runtime of our code snippets for comparison. If you’d like to learn more about this process, I’ve documented my approach in an article on performance testing in Python.

Otherwise, let’s go ahead and convert our solutions into strings:

For this first set of tests, I decided to start with a string that has no spaces:

Looks like our next_word_or_separator() solution is very slow. Meanwhile, the builtin split() is extremely fast. Let’s see if that trends continues. Here are the results when we look at one space:

Again, Python’s split() method is pretty quick. Meanwhile, our robust method is terribly slow. I can’t imagine how much worse our performance is going to get with a larger string. Let’s try the many_spaces string next:

This very quickly became painful to wait out. I’m a bit afraid to try the long_string test to be honest. At any rate, let’s check out the performance for the first_space string (and recall that the bugged solution doesn’t work as expected):

At this point, I’m not seeing much difference in the results, so I figured I’d spare you the data dump and instead provide a table of the results:

Clearly, the builtin method should be the goto method for splitting strings.

Challenge

At this point, we’ve covered just about everything I want to talk about today. As a result, I’ll leave you with this challenge.

We’ve written a function which can be used to split any string we like by any separator. How could we go about writing something similar for numbers? For example, what if I wanted to split a number every time the number 256 appears?

This could be a cool way to create a fun coding scheme where ASCII codes could be embedded in a large number:

We could then delineate each code by some separator code—in this case 256 because it’s outside of ASCII range. Using our method, we could split our coded string by the separator and then make sense of the result using chr():

If you read my article on obfuscation, you already know why this might be desirable. We could essentially write up an enormous number and use it to generate strings of text. Anyone trying to reverse engineer our solution would have to make sense of our coded string.

Also, I think something like this is fun thought experiment; I don’t expect it to be entirely useful. That said, feel free to share your solutions in the comments.

A Little Recap

And with that, we’re done! As always, here are all the solutions from this article in one convenient location:

If you liked this article, and you’d like to read more like it, check out the following list of related articles:

• How to Convert a String to Lowercase in Python
• How to Compare Strings in Python
• How to Check If a String Contains a Substring in Python

If you’d like to go the extra mile, check out my article on ways you can help grow The Renegade Coder. This list includes ways to get involved like hopping on my mailing list or joining me on Patreon.

Once again, thanks for stopping by. Hopefully, you found value in this article and you’ll swing by again later! I’d appreciate it.

Do you want to know how to develop your skillset to become a Web Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. Building web apps with Node.js

2. HTML5 Programming Cookbook

3. CSS Programming Cookbook

4. AngularJS Programming Cookbook

5. jQuery Programming Cookbook

6. Bootstrap Programming Cookbook

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.