Searching for a flat in Abu Dhabi, the hacker way
Pre-story (skip this)
I like comfort, so I'm searching for a flat in Abu Dhabi with very specific properties, in a very specific location with balcony and sea view and stuff. After looking at most of the property websites my path ended on www.dubizzle.com, which is the local Craigslist for UAE. Logically, it was created with the best clichés in mind and async JS requests were never never heard of in the world of developers. Of course I had to write my own parser and searching script for that. The script allows to search for a property in a location, defined by a bounding box, rather than an district.
Dubizzle has a mobile app, a little bit more modern. Knowing the
only thing know to do best practices, I fired up mitm and had a closer look at their API.
First request goes to
https://kombi.dubizzle.com/v1/kombi_access_token/ and funny thing, it does not work without the trailing slash (huh, I wonder who wrote that). That route gives us an access token that should be used with the rest of the calls.
Searching for the item needed
In my case, the second call goes to
There are several params but a little browsing will make you familiar with it. Look at the source for that.
This will give us the first batch. The script goes over the paging records all the items and them stores them in json files.
For quite some time I wanted to try node-webkit. it allways to create desktop apps using html/js/node at one place, writing the node.js code right in script part of the html.
I created the index.html file, so to run and see the crawler results just run run.sh or type
The source code for this is available at https://bitbucket.org/rahar/flatfinder/
to run first do
small disclaimer, the is just a story, not a call to action. They might get angry if you create a better ui