So it is exciting times! I have made some progress with TRTLExchange, however, due to things outside of my control it been slower than expected. So I have turned my spare time to Burf.co, my new search engine project and while there is no website for it yet (will be by the weekend), the actual search technology(code) has come along leaps and bounds. Overnight it managed to index over 500,000 pages which for a single server, was pretty cool. It did get up to 1.3 million pages but MongoDB has erm, shit the bed(many many times). This could be a hardware limit (Harddrive speed) or some performance thing I need to do however it gets to the point where I can’t even insert more records without timeouts. This concerns me quite a bit as I have a HP Blade Server on way to somewhat up the crawling rate by a factor of 8. I am going to try and give it one last go today however its taken 12 hours to delete the data from the DB (I did remove instead of drop 🙁 ). It has been a very interesting learning curve on learning MongoDB. I think unless some magic happens I am going to try out Postgres next.
Writing this blog post has also just raised the point that I was trying to learn Kotlin about a month ago (facepalm). Damn!