Pulling Your Data From Pocket: The Command Line Style

In 2013, I received an email from Pocket. Apparently, I was one of the Top 5% of Pocket users that year, having read close to 750,000 words within Pocket. In 2014, I made it into the Top 5% again, with 1.5 million words read in Pocket. That got me interested. I wanted to quantify what I was reading within Pocket. With that mission in mind, I set up an IFTTT recipe that added what I’d archived after reading in Pocket to a Google spreadsheet. After a year of checking and adding, the spreadsheet was finally filled with the results on one year of reading. You can check out the spreadsheet here.

Everything was good, but there was just a small problem. I couldn’t get the number of words in each article. IFTTT, though it uses the Pocket API, doesn’t provide users with the word count of an article. And to manually enter the number of words in 800+ articles in a spreadsheet would be a crazy thing to do. Indeed, I tried copy-pasting the article text into an online word-counter tool, and gave up after about 9 items.

So, after being one of the Top 1% of Pocket users in 2015, </humblebrag>, I thought of using Pocket’s API to accomplish what I wanted: word counts for the items I’d saved. This little pet project soon morphed into a sort of subset-archive of my data on Pocket. The recommended way to get around to doing this is to develop your own application, but I used cURL on Linux, with the standard bash terminal, to make requests and pull the data off Pocket. If you have experience developing your own programs, you can do this pretty easily, but I’m just a noobie programmer, so using cURL to make the necessary POST requests was the way to go for me. This little tutorial will help you pull your account data from Pocket through the recommended API requests, in a clean way. Before you begin, please copy the code provided throughout this post and save them as .sh files. This will make it easier to execute them later on.

You’ll need to save the commands provided here in a text file as a .sh shell file. Then you’ll need to chmod +x the file, to make it executable. For instance, let’s say you’ve saved this .sh file to your Desktop, as PocketRequest.sh. You’ll need to type

chmod +x ~/Desktop/PocketRequest.sh

to make the file executable. Once you’ve done that, simply type in the location of the file in the Terminal to run it.

~/Desktop/PocketRequest.sh

The first thing to do is to get your Consumer Key from Pocket. This is fairly simple. Just visit the Pocket Developer Console and register a new application. Fill in the details and you’ll be provided with your Consumer Key, like in the screenshot below.

Pocket_Consumer_Key
The Consumer Key is a secret token that shouldn’t be shared with anyone else.

Open up a text editor and type in the following lines of code. You’ll need to enter your own Consumer Key where it says your_consumer_key_here.

curl \
--header "Content-type: application/json" \
--request POST \
--data '{"consumer_key": "your_consumer_key_here",
"redirect_uri": "https://google.co.in"}' \
https://getpocket.com/v3/oauth/request

Assuming you’ve already saved this code to a shell file, you can execute it in the Terminal, as

~/location/file.sh

This will get you a Request Token. Note that I’ve provided the URL to Google as the Redirect URI. This is because I don’t really have an application that the user can return back to. It’s just a hackey fix.

Once you’ve made the request, you get a response from the Pocket server, straight in your Terminal. It will look something like this:

code=dcba4321-dcba-4321-dcba-4321dc

Note this Request Token down somewhere, because this is what you will be using to authorize this ‘app‘ with your account. Next up, open your browser and go to

https://getpocket.com/auth/authorize?request_token=YOUR_REQUEST_TOKEN&redirect_uri=https%3A%2F%2Fgoogle%2Ecom

Remember to substitute YOUR_REQUEST_TOKEN with the code you received in the previous step. Again, we are using Google as the Redirect URI. This means that, after you authorize the application you created to access your Pocket account, it will take you to the Google homepage. If we had an actual app working here, the Redirect URI would be a way to get back to the app/website we have.

On visiting this URL, you’ll be presented with the Pocket authorization page. Here you can allow your app to access your account. Click on Allow and then return to the Terminal. You’ll need to execute the second file now. The commands are as follows, which you can save to a .sh file for easier execution.

curl \
--header "Content-type: application/json" \
--request POST \
--data '{"consumer_key": "your_consumer_key_here",
"code": "request_token_you_received"}' \
https://getpocket.com/v3/oauth/authorize

This will give you a response like so

access_token=5678defg-5678-defg-5678-defg56&username=your_pocket_account_username

For all future requests, you will use this Access Token and your Consumer Key that you received in the first step.

Now that we’ve finished authenticating your application with your account, we can get to the actual process of getting your data. In case you want to download all of your items from Pocket, with their URL and word counts, save the following code in a .sh file.

curl \
--header "Content-type: application/json" \
--request POST \
--data '{"state": "all",
"consumer_key": "your_consumer_key_here",
"access_token": "your_access_token_here"}' \
https://getpocket.com/v3/get

A problem you might face when executing this command is that the response is dumped straight into the Terminal. This means that if you have lots of items in your Pocket list and archive, your Terminal screen will be filled with tons of gibberish data. To prevent that, when executing this file, we pipe the output into a temporary JSON file, like so.

~/location/file.sh > Temp.json

If you check your Home directory after executing this file, you’ll notice a new file named, obviously, Temp.json. Depending on the number of items you have in your Pocket, the file might be large and be filled with loads of relevant data, like the URL of the article or video, its word count, title, website it’s from, time it was added and so on. For now, we only need the title of the article, its URL and the word count.

You will need to install a tiny utility called jq for our next step. To install jq, type in

sudo apt-get install jq

in your Terminal. Once it’s downloaded and installed, type the following code in a text editor and save it as a .sh

 cat Temp.json | jq '.list | .[] | .resolved_title, .resolved_url, .word_count' 

Again, executing the code snippet above, will just flood your Terminal window with the output, so we’ll pipe the output to a final document file. The following is how you execute this last shell file.

~/location/file.sh > Pocket_Data.doc

This will create a .doc file with your whole Pocket library and the word counts of all items inside it. There you go, you’ve successfully authorized and downloaded your Pocket Data. There are several other combinations of parameters that you can use to download your data, which you can read about at the Pocket API Documentation pages.

If you have any suggestions to improve this hackey solution, please do let me know. Also, I’m not an expert on the Linux command line or about APIs in general. I created this whole solution with a little bit of Google-fu and lurking over at Stack Overflow. Nevertheless, I’d be happy to know how this little experiment worked out for you. Let’s discuss in the comments!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s