This weeks update : Bye Bye MongoDB

So it is exciting times! I have made some progress with TRTLExchange, however, due to things outside of my control it been slower than expected.  So I have turned my spare time to, my new search engine project and while there is no website for it yet (will be by the weekend), the actual search technology(code) has come along leaps and bounds.  Overnight it managed to index over 500,000 pages which for a single server, was pretty cool.  It did get up to 1.3 million pages but MongoDB has erm, shit the bed(many many times).  This could be a hardware limit (Harddrive speed) or some performance thing I need to do however it gets to the point where I can’t even insert more records without timeouts.  This concerns me quite a bit as I have a HP Blade Server on way to somewhat up the crawling rate by a factor of 8.  I am going to try and give it one last go today however its taken 12 hours to delete the data from the DB (I did remove instead of drop 🙁 ).  It has been a very interesting learning curve on learning MongoDB.  I think unless some magic happens I am going to try out Postgres next.

On the Swift front I did start building the frontend for Burf, first I was going to do this in VueJS, however, I have now found that Swift’s server-side framework Perfect supports templating via Mustache.  I think I will make faster progress writing it all in Swift than switching back and forth.   I still want to continue learning VueJS on the side (used for the TRTLExchange) as Javascript is such a good thing to know nowadays.

Writing this blog post has also just raised the point that I was trying to learn Kotlin about a month ago (facepalm).  Damn!


Experimenting with MongoKitten

As mentioned in my previous post, I have started looking at Server Side Swift with the aim to build a search engine (  To store my crawled data I decided to try and use MongoDB as it supports full-text search out of the box.  The original used Equinox (made by Compsoft) and then later used Microsoft Indexing Service.  This time round I wanted to be a little more scalable.  Now there are probably better DB solutions for what I plan to do, but MongoDB seemed really simple to get up and running with.  Later on, I should be able to switch out the database layer if needed.


Now that I had decided to use Swift, and MongoDB, I needed to find a framework that connects them, my friend (who knows his stuff) recommended MongoKitten!  I got up and running with it fairly quickly even though I don’t know MongoDB too well. Life was good, however, there were a few things I did struggle with:


So, search a field for a partial string requires you to use Regex (it seems).  


db.users.findOne({“username” : {$regex : “.*eBay.*”}});


let query: Query = [

           “url”: RegularExpression(pattern: “.\(eBay).”)


let matchingEntities: CollectionSlice<Document> = try pages.find(query)

Sorting results on $meta textScore

MongoDB allows you to setup full text searching across your data, it can be across an entire record, or just certain fields (name, address etc).  When you perform a full-text search, MongoDB returns the relevant records with an accuracy score ($meta.textScore).  MongoDB lets you change how it creates these scores by allowing you to adjust the weights each field receives e.g name is more important than address.


db.pages.find( {$text: {$search: “ebay”}},{score: {$meta: “textScore” }}).sort({score: {$meta:”textScore”}})


let query: Query = [

           “$text”: [“$search”: str ],

           “lang” : [“$eq”: “en”],


let projection: Projection = [

            “_id”: .excluded,

           “url”: “url”,

           “title”: “title”,

           “score”: [“$meta” : “textScore”]


let sort: Sort = [

           “score”: .custom([

               “$meta”: “textScore”



let matchingEntities: CollectionSlice<Document> = try pages.find(query, sortedBy: sort, projecting: projection, readConcern:nil,  collation:nil, skipping: 0, limitedTo: Settings.searchResultLimit )

Getting Help

I found the best way to get help was to contact the creator(Joannis) of MongoKitten via Slack, he is pretty busy but super helpful!

This weeks update : Server side Swift and updates to the site!

So, in terms of coding, the last week has been rather busy, actually in terms of progress, it’s looking good on all front (except Hack24)!

First off, there have been some updates to the website to finally include a basic list of historic Robotics projects.  I had planned to do that last year!  I have also updated the design and content a little, which frankly no one will notice!

Next, I started yet another new project, Burf Search Engine (The Return).  It had been on the cards for a while, however, I have now started coding it.  First in NodeJS, then in Server Side Swift using the Perfect framework.  It went swimmingly well (thanks to Ad) until I had to deploy it to Ubuntu.  There, you start to learn the differences between Apple frameworks and the open source versions of the same framework. Randomly I am expecting the Apple framework to have less //todo and //to implement comments 🙂

The Crypto project didn’t process as far as I hoped this week purely due to my friend (and other coder) being ill, however, momentium has now picked back up.  As your also probably aware Cryptocurrencies took a massive dive this week!

No update on Hack24, it will resume after phase 1 of the Crypto project has been completed and the search engine crawler for is live 🙂

On the side note, really enjoying Altered Carbon on Netflix’s and Thanks to Ford for sending me this epic LEGO set 75885 (Ford Fiesta M-Sport WRC) to review at the weekend.