September 12, 2015

Searching for a flat in Abu Dhabi, the hacker way

Pre-story (skip this)

I like comfort, so I'm searching for a flat in Abu Dhabi with very specific properties, in a very specific location with balcony and sea view and stuff. After looking at most of the property websites my path ended on www.dubizzle.com, which is the local Craigslist for UAE. Logically, it was created with the best clich├ęs in mind and async JS requests were never never heard of in the world of developers. Of course I had to write my own parser and searching script for that. The script allows to search for a property in a location, defined by a bounding box, rather than an district.

The path

Dubizzle has a mobile app, a little bit more modern. Knowing the only thing know to do best practices, I fired up mitm and had a closer look at their API.

Access token

First request goes to https://kombi.dubizzle.com/v1/kombi_access_token/ and funny thing, it does not work without the trailing slash (huh, I wonder who wrote that). That route gives us an access token that should be used with the rest of the calls.

Searching for the item needed

In my case, the second call goes to https://kombi.dubizzle.com/v1/countries/4/listings/?categories=%2Fcountries%2F4%2Fcategories%2F24%2F&neighbourhoods=%2Fcountries%2F4%2Fneighbourhoods%2F57394%2F&order_by=added_desc&cities=%2Fcountries%2F4%2Fcities%2F3%2F&price__lte=120000&bedrooms__gte=1&bedrooms__lte=1&language=en
There are several params but a little browsing will make you familiar with it. Look at the source for that.
This will give us the first batch. The script goes over the paging records all the items and them stores them in json files.

Displaying results

For quite some time I wanted to try node-webkit. it allways to create desktop apps using html/js/node at one place, writing the node.js code right in script part of the html.
I created the index.html file, so to run and see the crawler results just run run.sh or type nwjs .

Source code

The source code for this is available at https://bitbucket.org/rahar/flatfinder/
to run first do
npm install
node crawler.js
nwjs .

small disclaimer, the is just a story, not a call to action. They might get angry if you create a better ui

  • LinkedIn
  • Tumblr
  • Reddit
  • Google+
  • Pinterest
  • Pocket
Comments powered by Disqus