Burf.co paused, Hack24 Resumed…..

So I been on holiday driving across America, it was great fun!  I was hoping to be inspired on what to do with Burf.co, what’s the business plan, the focus etc.  Sadly absolutely nothing came to mind!  Not a dime!  Technically I can now index a lot of records pretty fast!  I have enough hardware to heat up a street of houses (more than my ring-main could handle) however I still don’t know what the point is.  So for the moment, it’s being paused!

With that in mind, I thought I would carry on with Hack24 and see if I can get it out the door.  I am still using LibGDX however, I have to moved to Multi-OS Engine for iOS and I am going to see if I can use FireBase for the backend.

Hopefully, by the time I have an MVP built, I should then know what I am going to do with Burf.co

 

 

Tumbleweed and that damn CommonCrawl

I hate by starting a post apologising for not updating my blog so I won’t do that!

I have been a bit busy with the new job I started 3 weeks ago, so most of my side projects have been paused!  However, work on Burf.co has gone 2 steps forward, a couple to the left and then a couple steps backwards, this is largely due to the awesome site CommonCrawl.org having a huge part of the Internet crawled and open for anyone to use! They have petabytes of web data open for anyone to use and there are some really cool examples of how to use it, most involving a huge amount of cloud power!! I did ponder for quite a while how I would store so much data!!! I found an interesting Java project that scans the index of the CommonCrawl for interesting file types (https://github.com/centic9/CommonCrawlDocumentDownload).

I took this project, hacked it about a bit and changed it so that it would only return URLs that are mine type HTML and that had a response status of 200. This gave me around 50 million URLs to play with which all had file pointers to the actual web page data. Because this data is compressed, it’s far quicker to download them from the CommonCrawl than actually scrapping the website itself. CommonCrawl also follows the Robot.txt which is far more than I have ever done :). So far the end result is that I can get around 5 million pages of data a day (from my home internet) compared to around 500k on a good day!  That’s a pretty good increase!

This weeks update : Bye Bye MongoDB

So it is exciting times! I have made some progress with TRTLExchange, however, due to things outside of my control it been slower than expected.  So I have turned my spare time to Burf.co, my new search engine project and while there is no website for it yet (will be by the weekend), the actual search technology(code) has come along leaps and bounds.  Overnight it managed to index over 500,000 pages which for a single server, was pretty cool.  It did get up to 1.3 million pages but MongoDB has erm, shit the bed(many many times).  This could be a hardware limit (Harddrive speed) or some performance thing I need to do however it gets to the point where I can’t even insert more records without timeouts.  This concerns me quite a bit as I have a HP Blade Server on way to somewhat up the crawling rate by a factor of 8.  I am going to try and give it one last go today however its taken 12 hours to delete the data from the DB (I did remove instead of drop 🙁 ).  It has been a very interesting learning curve on learning MongoDB.  I think unless some magic happens I am going to try out Postgres next.

On the Swift front I did start building the frontend for Burf, first I was going to do this in VueJS, however, I have now found that Swift’s server-side framework Perfect supports templating via Mustache.  I think I will make faster progress writing it all in Swift than switching back and forth.   I still want to continue learning VueJS on the side (used for the TRTLExchange) as Javascript is such a good thing to know nowadays.

Writing this blog post has also just raised the point that I was trying to learn Kotlin about a month ago (facepalm).  Damn!

 

Humble Bundle: Mobile Development Bundle

So a new Humble Bundle has appeared that’s aimed at mobile devs which is cool!  It contains a large range of books by Packt which will help any Mobile Dev improve their skills, details below:

https://www.humblebundle.com/books/mobile-app-development-books

I am sure I have done it before, but I would also like to point out that Packt also gives away a free book every day!!!  Claiming it is one of my daily tasks!

https://www.packtpub.com/packt/offers/free-learning

 

Where is Hack24?

So Hack24 is becoming a bit more delayed than hoped, however, this is for good reasons (Well I think so).

I have been introduced to cryptocurrencies, via my friend sending me a link to https://turtlecoin.lol/ and telling me to mine the sh*t out of it!

Knowing a little (and I mean a little) about BitCoin (and its huge growth in value), I thought it was a great opportunity to find out more about this technology without investing any money.

Since I started mining it about 2 weeks ago, I have watched several documentaries with my wife about digital currencies plus the actual TRTL coin has gone up 100x in value!

If nothing else it’s been a fun learning experience and I feel a bit more comfortable around the subject.  I am not working on a very short-term project in this space 🙂

Hack24 will resume very shortly 🙂

Hack24 V2 started :)

So, it may end up being called something else but I have now started writing a new (well kinda) game.  The last couple of weeks I have been bug fixing and improving the LibGDX framework / demo app that I made a few years ago.  Code can be found here: https://github.com/burf2000/BurfEngine

Yes, it has a terrible name!  Anyway it was my attempt at a simple game like Minecraft where you can place and remove cubes.  I added things like chunking, gravity, and culling.  It also has a database and network layer, custom collision code and works on iOS, Android, and Desktop.  It has been converted to Kotlin, which I am really liking 🙂

Now, after a few long nights of fixing, that code is being parked and anyone can use it!  I now plan to rip it apart and use the best bits to form the engine for Hack24 (v2).  Made aim is to make a 3D game that has the same look and feel as Hack24 that’s cross-platform and easy to develop further.

I will focus on a MVP first, which should not take too long, I will try and rewrite the server in NodeJS (Well KotlinJS).

So why am I doing this?  A few reasons:

I really liked Hack24 but it had some performance issues and really needs rewriting to use OpenGL VBO / VAO.  It also makes sense to make it cross-platform while I am at it.  I don’t fancy doing it in Unity and really want to learn Kotlin for work (which I can for LibGDX).  It would be nice if people enjoy playing it too, so I hope to make it bigger with more content 🙂

Watch this space!

 

 

VEX Worlds 2017 – Robotics Competition

Sorry for the late post about VEX Worlds, I thought I would have more time after worlds to catch up with stuff, sadly (well not really), the kids have been mega active.  My eldest son played his first football tournament, had a holiday, lots of family stuff!

So, VEX Worlds, what an amazing experience, I went along for the VEX EDR side of the competition (this year it was split EDR / IQ) as I was showing off the EDR Tank.  Sadly I had to leave the US early as my son, Max was ill.  Still a very cool experience!

So, the EDR tank, well it performed really really well in remote control mode.  I mean the thing was fairly slow but must have covered MILES!  The batteries never died on me, nor did any motors!  I did kill a few Omniwheels, however, that’s to be expected.   Even though I left early, the EDR tank did not and so others drove it around.  I have not received it back yet to see how bad it is now, but I am sure it will be fine.

The autonomous side was a bit of a failure, to be honest, and looking back I had set up myself to fail and I will explain why.  The autonomous side was using ROS (Robotic operating system) which is an industry standard.  I was using a Neato Lidar system which is awesome however it only had a range of 5 meters and SLAM (simultaneous localization and mapping) to work out where I was, and where I needed to go via building up a map.  SLAM works by detecting features of the surrounding area to work out where it is.  When you’re in a hall that’s hundreds of meters wide with very little features, a sensor with a range of 5 meters is practically useless.  In the end, I just showed kids how it worked on my laptop using RVIZ.  If I had to do this properly I would need to invest in a proper LIDAR system with a much greater range.  Another aspect which makes this very hard is all the people moving around, how can SLAM pick up features if they are constantly moving!

Overall, the EDR tank was hugely popular, I gave tons of fist bumps, high fives, etc, people just thought it was cool, just a little slow.

Next year, if I did a vehicle again, I would have to make it a lot faster and forget about advance sensors etc!

Here are some videos of VEX World and the EDR TANK:

This weeks update : It’s a lie : Metabase, more VEX and some data science!

So, let’s get the lie out of the way, this week’s update could cover more or less than a week!  It is whatever I am thinking of at the time, that may or may not be happening.  So apologies for that bombshell.

Software

So, at work (O2’s Innovation Lab) I am currently learning data science stuff, for anyone who knows me, this is an extremely hard task as I have the focus of hamster on Redbull.  I am usually doing more than 1 thing (usually 5) and so it can be a struggle to learn a new skill, let alone one as difficult as data science.  This week, I would say I am starting to get somewhere.  I been using different classifiers across my data, checked its score and then looked at the confusion matrix.  What that told me was that my data sucked badly, however, the upside was I could prove that my data was terrible.

Another thing I am doing at work around data (oh look at my focus) needed me to take some data and put a GUI over the top for people to be able to easy “ask the data questions”, I found a really cool free tool called Metabase which worked really nicely.  All I needed to do was take an MS Access DB (oh boy who uses MS Access), convert it to CSV, and chuck it in a Postgres DB.  Would have taken 5 mins on a PC, a Mac took a little bit longer!

Robots

So what’s new on the robot front this week? well VEX Worlds is in less than 25 days and the software is erm…. still in development.  The EDR Tank should be on way to the US, so I made a mini version of it so that I can carry on with the development.  I have written some safety features into the software so that I don’t mow down innocent kids, mouthy kids, will, of course, run over!  The nex thing I need to do is finish the bridge between the VEX Cortex and the ROS software

ROS

I have a new friend on Facebook, (whoop whoop) who has been helping me with the ROS stuff, it’s useful to have a sounding board on learning new stuff, especially something as complex as ROS.  I have a fear that the VEX Tank may not work too well with all the people moving about.  Slam and autonomous driving works (very simple form) by identifying features in the environment to try and locate itself. when you have no real features (e.g a long corridor) or lots of things changing (e.g people moving about), it can get very confused.  I am sure robotics engineers have a good solution to this, but being a beginner and using Hector Slam for the first time,  I am not holding my breath.  My mini raspberry Pi / LEGO version got confused if I farted near it, let alone 10,000 kids running around!

Ending Notes

I started a statistics course as its the precursor to the Udacity Machine Learning Course.

I finished a Sentiment analysis course, pretty interesting, showed how to work out if a review to a film was positive or negative.

I watched Logan, was very good and rather violent and definitely not for the kids

I watched Kong, was pretty good but preferred the previous one, which to be fair is nothing like the new one.

I started printing the Inmoov project 🙂 THE BEST 3D Printed project in the world!

This weeks update :) Big Bang, VEX Worlds.

So, last week I was at the Big Bang Science Fair, I was asked to be a judge for the VEX Robotics National Championship 🙂


I also managed to get VEX the VEX Tank ready to ship it to the US for VEX WORLDS, build-wise, it’s complete, software wise, it’s not 🙁

My flights have been booked for VEX WORLD for next month, so full speed ahead to finish the software!  Anyone know ROS?