Vagrant on Windows

There are many different ways to install python and work with python on Windows. You can install Canopy or Anaconda to have an entire python ecosystem self-contained or you can install python directly onto your machine and configure all the bits and bytes yourself. My current recommendation is to use Vagrant on Windows combined with Virtualbox to virtualize your development environment.

While I use a mac or the majority of my development, I do find myself using Windows 10 more and more, and may be moving to a Windows machine in the future for my day-to-day laptop.  I have and do use Canopy and/or Anaconda but I’ve recently moved the majority of my python development on Windows into Linux (Ubuntu) virtual machines using Vagrant and Virtualbox. You can use other products like VMWare’s virtual machine platform, but Virtualbox is free and does a good enough job for day-to-day development.1

One Caveat: if you’re doing processor / memory intensive development with python, this may not be the best option for you. That said, it can work for those types of development efforts if you configure your virtual machine with enough RAM and processors.

To get started, you’ll need to download and install Vagrant and Virtualbox for your machine.   I am using Vagrant 1.90 and Virtualbox 5.1.10 at the time of this post.

Feel free to ‘run’ either of the programs, but there’s no need to enter either program just yet.    To really use the Vagrant and the linux virtual machine, you’ll need to download a *nix emulator to allow you to do the things you need to with vagrant.

I use Git’s “bash” to interface with my virtual machines and Vagrant.  You could use putty or CygWin or any other emulator, but I’ve found Git’s bash to be the easiest and simplest to install and use.  Jump over and download git for your machine and install it. At the time of writing, I’m using Git 2.11.0.

While installing Git, I recommend leaving everything checked on the ‘select’ components window if you don’t currently have any git applications installed. If you want to use other git applications, you can uncheck the “associate .git* configuration files…” option.  There is one ‘gotcha’ when installing git that you should be aware of.

On the “adjusting your path” section (see figure 1), you’ll need to think about how you want to use git on the command line.

git command line path
Figure 1: Adjusting your path

I selected the third option when installing git. I do not use git from the windows command line though…I use a git GUI along with git from the command line within my virtual environment.

Another screen to consider is the “Configuring the terminal emulator…” screen (figure 2).  I selected and use the MinTTY option because it gives me a much more *nix feel. This is personal preference. If you are going to be doing a lot of interactive python work in the console, you might want to select the 2nd option to use the windows default console window.

Configuring your terminal emulator
Figure 2: Configuring your terminal emulator

During the remainder of the installation, I left the rest of the options at the defaults.

Now that git (and bash) is installed, you can launch Git Bash to start working with Vagrant. You should see a window similar to Figure 3.

Git Bash
Figure 3: Git Bash

From this point, you can do your ‘vagrant init’, ‘vagrant up’ and ‘vagrant ssh’ to initialize, create and ssh into your vagrant machine.

Setting up Vagrant on Windows

For those of you that haven’t used Vagrant in the past, here’s how I set it up and use it. I generally use vagrant in this way to run jupyter, so I’ll walk you through setting things up for jupyter, pandas, etc.

First, set up a directory for your project. At the Bash command line, change into the directory you want to work from and type “mkdir vagrant_project” (or whatever name you want to use). Now, initialize your vagrant project by typing:

This will create a Vagrantfile in the directory you’re in. This will allow you to set the configuration of your virtual machine and Vagrant. The Vagrantfile should look something like this:

VagrantFile Example
Figure 4: VagrantFile Example

Before we go any further, open up your Vagrantfile and change the following line:

change “base” to “ubuntu/xenial64” to run Ubuntu 16.04. The line should now read:

If you want to run other flavors of linux or other OS’s, you can find others at https://atlas.hashicorp.com/search.

Since I’m setting this VM up to work with jupyter, I also want to configure port forwarding in the Vagrantfile. Look for the line that reads:

and add a line directly below that line to read:

This addition creates a forwarded port on your system from port 8888 on your host (your windows machine) to port 8888 on your guest (the virtual machine). This will allow you to access your jupyter notebooks from your Windows browser.

At this point, you could also configure lots of other vagrant options, but these are the bare minimums that you need to get started.

At the Bash command line, you can now type “vagrant up” to build your virtual machine. Since this is the first time you’ve run the command on this directory, it will go out and download the ubuntu/xenial64 ‘box’ and then build the virtual machine with the defaults.  You might see a Windows alert asking to ‘approve’ vagrant to make some changes…go ahead and allow that.

Once the ‘vagrant up’ command is complete, you should see something similar to Figure 5 below.

Output of Vagrant Up
Figure 5: Output of ‘vagrant up’

Now, you can ‘vagrant ssh’ to get into the virtual machine.  You should then see something similar to Figure 6. Now your running vagrant on windows!

Vagrant SSH - Vagrant on Windows
Figure 6: Output of ‘vagrant ssh’

One of the really cool things that vagrant does by default is set up shared folders. This allows you to do your development work in your favorite IDE or editor and have the changes show up automatically in your vagrant virtual machine.

At the Bash command line, type:

You should see a directory listing that has your Vagrantfile and a log file. If you visit your project directory using windows explorer, you should see the same two files. Shared folders for the win! I know its just a small thing, but it makes things easier for initial setup.

You now have vagrant on windows!

Configure the Python Environment

Time to set up your python environment.

First, install pip.

Even though you’ve set up a virtual machine for development, it is still a good idea to use virtualenv to separate multiple projects requirements.  Install install virtualenv  with the following command:

In your project directory, set up your virtual environment by typing:

Note: You may run unto an error while running this command. It will be something like like the message below:

If this happens, delete the ‘env’ folder and then add ‘–always-copy’ to the command and re-run it. See here for more details.

Activate your virtualenv by typing:

We’re ready to install pandas and jupyter using the command below. This will install both modules as well as their dependencies.

Now you’re ready to run jupyter.

In the above command, we start jupyter notebook with an extra config line of ‘–ip=0.0.0.0’. This tells jupyter to listen on any IP address. It may not always be necessary, but I find it cuts out a lot of issues when I’m running it in vagrant like this.

In your windows browser, visit ‘http://localhost:8888/tree’ and  – assuming everything went the way it should – you should see your jupyter notebook tree.

Jupyter via Vagrant VM
Figure 7: Jupyter via Vagrant VM

From here, you can create your notebooks and run them just like you would with any other platform.

 

Installing Python on OSX (and the necessary modules)

If you need help installing python on OSX, read on.

For the last three years, I’ve used a mac for all my development. I love the fact that everything ‘just works’ on the platform. That said, when you get into scientific computing and data analytics, especially with python, you can  run into some issues.

Just like linux, python is included with the operating system. Unlike linux, this can cause problems long-term for you due to upgrades and changes that Apple may make to the python ecosystem.

On OS X, I recommend those of you starting out to go with Anaconda or Enthought Canopy.  As I said in “Installing python on Windows“, I prefer Canopy over Anaconda for scientific computing / data analytics but either will work for you.  Installing Canopy on the mac is very similar to installing it on Windows…so I’ll let this post be your guide for installing Canopy.

If you want to get into the nitty-gritty and install and configure python and the modules yourself, you can easily do so, but be prepared to spend some time on the command line.

Before we get started installing python on your Mac, we need to install homebrew, which is a package manager for OS X (it acts similar to the ‘apt’ package manager on ubuntu / debian).

To install homebrew, open a terminal and paste the following:

This command installs the homebrew ecosystem onto your machine and preps your machine to be ready to install various packages, including python.

Installing Python on OSX

Step 1: Let’s get python installed via homebrew.  In your terminal, type:

This will install a version of python onto your machine and set up your environment to use that version. This helps mitigate any issues you might have down the road if / when Apple makes changes to the system provided python.   Additionally, brew installs pip into the system to make it easy to get the necessary modules onto your machine.

From this point on, we are generally going to follow exactly the same steps that I outline in Installing Python on Linux except we don’t need to install any additional tools.

Step 2: Not required, but highly recommended – install a virtual environment.  I recommend virtualenv. Install it with this command:

When you are ready to get started on a new project, type the below command to install python into a new virtual environment (the ‘env’ is the name of the environment). You only have to do this once per project. Note: You should use a folder per project to keep your virtual environments separated.

Whenever you want to work on a specific project, change into that folder and type the following. This will set up your environment with all of your installed python modules:

For the purpose of this walk-through let’s create a new directory, set up a new virtual environment and then install the necessary modules.

  • Create a folder in your home directory called ‘projects’.
  • Type “mkdir projects” to do this from the command line.
  • Change into that folder and then type “mkdir install_example” to create another folder inside the projects folder.
  • Type “virtualenv env” to create your virtual environment.
  • Type “source env/bin/activate” to begin using this environment

Now that we have our environment ready to go, we need to install some of the modules that are most often used when doing data work inside python. These modules are:

The above modules can be installed with one pip command.

You’re ready to start working with python for data analysis on your mac. Just remember, for each virtualenv you create, you’ll need to reinstall these modules if you wish to use them.

Check back here often for more information on using the above modules to actually DO something.

Data Analytics & Python

data analytics & pythonSo you want (or need) to analyze some data. You’ve got some data in an excel spreadsheet or database somewhere and you’ve been asked to take that data and do something useful with it. Maybe its time for data analytics & Python?

Maybe you’ve been asked to build some models for predictive analytics. Maybe you’ve been asked to better understand your customer base based on their previous purchases and activity.  Perhaps you’ve been asked to build a new business model to generate new revenue.

Where do you start?

You could go out and spend a great deal of money on systems to help you in your analytics efforts, or you could start with tools that are available to you already.  You could open up excel, which is very much overlooked by people these days for data analytics. Or…you could install open source tools (for free!) and begin hacking away.

When I was in your shoes in my first days playing around with data, I started with excel. I quickly moved on to other tools because the things I needed to do seemed difficult to accomplish in excel. I then installed R and began to learn ‘real’ data analytics (or so I thought).

I liked (and still do like) R, but it never felt like ‘home’ to me.  After a few months poking around in R, I ran across python and fell in love. Python felt like home to me.

With python, I could quickly cobble together a script to do just about anything I needed to do. In the 5+ years I’ve been working with python now, I’ve not found anything that I cannot do with python and freely available modules.

Need to do some time series analysis and/or forecasting? Python and statsmodels (along with others).

Need to do some natural language processing?  Python and NLTK (along with others).

Need to do some machine learning work? Python and sklearn (along with others).

You don’t HAVE to use python for data analysis. R is perfectly capabale of doing the same things python is – and in some cases, R has more capabilities than python does because its been used an analytics tool for much longer than python has.

That said, I prefer python and use python in everything I do. Data analytics & python go together quite well.