Burf.co and Elasticsearch
So, as you can guess by the title, I decided to check out Elasticsearch (and do a couple of courses on it) to see if I could use it with Burf.co Search Engine. Elasticsearch is built to be a search engine, whereas previously I was using MongoDB’s full test indexing and well, it just wasn’t up to the job of quickly returning results in a few seconds.
I still use the CommonCrawl data, which is filtered and put into MongoDB (the actual HTML document is stored), I then parse this and chuck it into Elasticsearch. The program is all written in Java and seems to work pretty well.
The only issues I had were around updating my Java Spring Boot API’s to talk to Elasticsearch. The Spring Boot Elasticsearch component uses an older version of Elasticsearch (current is v6.4.2, Spring Boot I believe is v5.x). The other issue I had which seems silly is that most examples I found use Maven as the build pipeline, whereas I use Gradle.
The result is that search on Burf.co now works and is a lot quicker than before (disclaimer, I am still adding data to Elasticsearch, last looked it was around 55m pages). I still need to learn about how to optimise it, how to do better more accurate searches, however, I am happy that I moved to use the correct tool for the job 🙂
Last Friday, a few of my old Compsoft work colleagues met up for a GameJam which was fun. I decided to focus on finishing Hack24 (finally), or to at least evaluate how much work was required. Sadly the GameJam only lasted 7 hours (2 am) however I did get quite a lot of bugs and features implemented into Hack. An Android MVP version is coming really soon!
I read a book!!!
So, it appears that you don’t just need to watch tv and play games while you’re on the exercise bike, you can READ, and I did so. I read a book on flipping businesses (random I know) and quite enjoyed it. So I have the $100 startup to read next 🙂