Busy week, nothing achieved!

The title of this weeks post is probably a little unfair on myself.  Lots of stuff got worked on last week but no milestones to talk about.  After reading a blog post on productivity, I think I need to start focusing on fewer things to achieve more.  My constant context switching is fun but less productive.

Vibe Innovation

So this is my idea of starting a proper business and learning the ropes of a limited company.  The idea is to provide a technology prototyping service, basically making proof of concepts for people.  Progress this week has been around getting a site up and running (VibeInnovation.com), talking to accountants and stuff and preparing my business PayPal account.

Burf.co

I am trying to automate as much of this as possible and just make it run itself.  After that, I need to design a service or product that it offers that people will want and pay for.

Echo

Progress on this has been pretty good, for the MVP I have decided to use Google’s Firebase to handle the backend communication and data.  Once the paperwork is done for Vibe, my primary focus will be on Echo.

Hack24

Paused 🙁

Keyword Research Ninja

So I have updated the costs of the product, linked it to my business PayPal account and created a product page.  All I need to do next is link it to the product and the site is done.

Robotics

Finally started playing around with the VEX EDR v5 hardware, ready to make a large epic robot 🙂

Crazy times with Innovation and Ninja’s

I have a plan, it involves learning, innovation and contains the word Ninja? Confused, read on!

So, this weeks blog post will probably be more confusing than most.  I had a few ideas floating around in my head that have formed into a kind of business idea (maybe even a plan).

Ninja!

First off, I was looking to buy or start a proper business, what I mean by proper is that it is a limited company registered with Companies House. Why you ask?  Well, I thought it would be a good thing to learn about.   I Thought a good place to start was buying something random off eBay that was already set up,  I hadn’t really worked out what to buy but something online, maybe marketing, SEO or something around that area.  I randomly brought the site Keyword Research Ninja for less than a Dominos pizza.  It sells keyword research software written in C#.  For me, the worst case scenario of buying this site is that I have some fun with the software.  However, this does not fulfill my idea of getting a limited company and the site does need some work.

Vibe Marketing Limited

So, also on eBay, they sell dormant companies (again not sure why I just didn’t set up a new one), and I came across someone selling Vibe Marketing Limited, they wanted quite a bit of money for a name,  after some strong negotiation skills, I brought it for less than a Dominos pizza!  I thought, hey I can get back into SEO (I used to love doing that in my spare time) and do it part-time for fun, just to learn the ropes of running a company.

Coffee with an old friend

I had been in contact with an old friend who used to work at Compsoft when I first joined.  He was a member of R&D and went off to finish his degree, start a company and do pretty well in life.  Anyway, we met up for Costa (Starbucks was too far away) and had a chat about life, work, and code.  He had set up a successful HR company that his wife runs, and he now does PoC work for fun to keep him actively coding.  I mentioned to him to him that I really enjoyed my time at O2’s Lab doing innovation and PoC work and that I missed the crazy times of making stuff work in new tech and then chucking it over the wall for a dev team to properly implement it if the business thought it was viable.

The penny drops.

So, I decided that my next plan of action was to take this limited company (in the process of being renamed to Vibe Innovation) and set it up as a consultancy company that creates proof of concepts for people.   I want to start really small with really small goals, this is still a learning process.  This year’s aim (November / December) is to create the website, set up email and show some of the projects I have done in the past.  Next years goal is simple, do 1 paid bit of work and work out how to process it through the books.

Other news

So work on Echo Music Group is actually progressing nicely.  It’s great fun getting back into doing iOS work.  I am currently working on the signup process.

Elasticsearch definitely seems to be the way to go with Burf.co which has been working well as a search engine.  I need to improve the quality of the dataset (which is in progress) and fix the site to work on mobile.  It has had over 16,000 sites manually submitted to it in the last week which is pretty epic.

I have also upped my reading for a second week in a row, I am now currently reading the $100 startup…. can’t imagine why 🙂

This weeks update: Just keep spinner!

So as the wife is in Portugal getting some sun and I am home alone with the kids who are now finally asleep.  I thought I would review the week(s)

Burf.co

There is not much to report except a realisation that creating a new Burf Search Engine is gonna be a lot of work!  Even taking a fairly small chunk of it (100 million pages), I just can’t produce results fast enough using MongoDB.  So I am thinking about a few key points I want to focus.

  • Must be fast
  • Must be fairly useful and produce useful results
  • Must update itself
  • Must have a niche

So at the moment, I take the first 100,000,000 URLs from the CommonCrawl that returned HTTP status of 200 and are marked as English.  The CommonCrawl contains I believe around 2.3 billion URLs and so what I may do is filter which ones I want.  I could also build up a simple list of top sites (BBC, Wikipedia, MSN etc) and just index them once a week.

I also need to think about the people who submit their sites,  I need to at least action them.  I am thinking of moving to Elasticsearch just to speed everything up.  Everything seems slow in MongoDB past 1 million records even on an SSD.

I have also found myself regressing a bit to my old Burf.com ways of buying domains, looking at turnkey websites and SEO tips and tricks.  I used to love this and had over 100 sites at one point.

On a side note, Burf.co has a new site design and is now hosted remotely 🙂

Echo MG

So iOS development has started on the MVP which is good, more would have happened if Burf.co had played ball.  This is far more important than Burf.co, so if it comes to it, I will turn Burf.co off to focus.  It’s nice to do some iOS development again.

Hack24

This had been parked however this is a game jam coming up this Friday and I think with a lot of RedBull and sugar, I may be able to actually complete the MVP and get it uploaded which would be great.

Robots….

So some bits have been printed but I really need to sort the garage out so that I can then sort out all the EDR parts.  Too many jobs, not enough time 🙁

New Burf.co, Business Success Diploma and building furniture!

So, I am trying my best to make sure I update my blog regularly so that I can look back and see what I have achieved.

Burf.co, now with no search button

So between spending over 12 hours building furniture for the wife (2 new bunk beds, 2 desks, 2 chairs and a bookcase) I also managed to build a new MVP of Burf.co.  The previous version was using Swift and was all in one (frontend/ backend).  The new one is properly structured and uses VueJS for the frontend hosted externally, and the backend is in Java Spring Boot.  It is just a prototype at the moment and doesn’t even have a search button, you just type and it starts getting the results (and smashing the db) but it was a fun prototype to build and is definitely the way I want to go.

Next Steps

  • Increase dataset from 10 million to 100 million and make it fast as f*ck
  • Add site pages (contact, about, submit etc)
  • Add tag cloud and routing to make it better for SEO

Business Success Diploma

So ages ago I signed up to Shaw Academy as it has some great courses and reviews.  I got a lifetime membership so I wasn’t in a rush.  Since I finished the bodybuilding show, I have been caining through the course at super speed and hope to take my exam this week:). It was one of my new year’s resolutions to do (so was Hack24 🙁 ).  I am learning a lot and hope to carry on with more advanced training around business.

Echo

The above projects are the fluff as I call it to get out of the way so that I can focus on Echo and getting this startup off the ground.  I already started building a basic website for them.  Once that’s complete, its time to focus on the MVP

Robotics

When it rains, it pours!  So before I had even thought about the above 3 projects, I had decided I was going to build a humanoid out of VEX EDR, which is a brilliant building platform.  They had just brought out their new V5 system which looks epic (more powerful motors, more motors, vision control, touch screen etc).  VEX was very nice in sending me some V5 hardware to start building this 3D Printed / VEX EDR humanoid!  This will be my chilled weekend project 🙂

 

A busy week : CTO, 70 million and a Java backend!

As I do my final prep for my bodybuilding show tomorrow, I thought I would do a quick post of what’s happened over the week.  I had planned to do very little due to the show however people who know me, know I don’t like to stay still for long!

CTO

So, a while back I got invited to become the part-time CTO of a small startup called Echo MG (Music Global) who have big plans to change how entertainment is booked, artists etc.  The role would include everything from designing their infrastructure to help pitch their MVP to investors.  It all sounded very exciting, however, I had to turn it down at the time due to changing my main job (from O2 to Reach).  Now I have settled in and got approval, I have gone for the role.  I hope to post more about this as it develops but it should be a good learning experience.

70 Million

So, I kicked Burf.co off again on Friday and its been going like the clappers! (and not even using the blade server).  It’s currently sitting at 70 million pages and once it gets to 100, I plan to stop it and start the NLP parsing which should be super interesting.

Java Sprint Boot(kotlin)

So I have had to think long and hard what to write my search engine, AI/ML platform and robotics stuff in.  The obvious choice is Python but I just don’t like the syntax (space, no brackets etc).  So after looking around and seeing what would also help me for work, I decided to learn Spring Boot using Kotlin.  Kotlin is a great language, it’s very like Swift which is brilliant.  I can use it with Android so helps work and there is not much you can’t do with Java.  Spring Boot seems to be like black magic, you go to their site, tell them the frameworks, build tools etc and press a button and it makes a project for you 🙂

Chromebook

So I brought a Chromebook, it’s actually the 2nd one I have owned maybe 3rd but that was before they could run Android apps which is sweet!  Why? you ask!  Well, I wanted a cheap, light laptop with good battery life and no noise (e.g I cant dev on it) to focus me to actually plan and write stuff down.  So by being fairly limiting, and running off the Google ecosystem, it should make me more productive!

 

Lets kick it off again!

So one of the only plus points of not getting any sleep due to the effects of the extreme diet for this bodybuilding show is insomnia, my mind gets very creative and forces me to start kicking off new ideas, projects, missions et!

So, if you saw my last post, I said I was gonna finish Hack24, fix Burf.co and sort the garage!   So far, Burf.co is back up, but about to completely change, the garage is nearly finished being geared up as a robotics lab and Hack24 has not moved.  I do want to finish hack24 but I don’t want to rush it and I want to harness my energy on some crazy robotics ideas while my brain still works 🙂

So the plan v2!  Warning it’s a little bonkers, even for me!

Build a backend set of machine learning API’s that Burf.co, mobile devices, and my robots use to send and retrieve data.  The idea is I could send it a question, a command or an image and it does some magic and responds.

  • So for mobile devices, they would send images and text to speech, it would return ImageNet classification or answers to questions.
  • Burf.co would become more of a knowledge base system using NLP to feed into other systems.
  • There would also be a public facing chatbot which would hopefully learn off of all of this.  Planning a system POC using AIML to test the waters
  • This would all somehow be also brought together to add some usefulness to my future robotics projects (image classification, knowledge base, etc)

I brought some odd bits of hardware, upgraded the Burf.co server, brought some domains, and started rewriting Burf.co in Java.  I decided I want to try and use a common language and randomly Java seemed the best fit (client, server, mobile etc)

It’s gonna be a slow progress but I think its gonna be exciting.

Let’s kick it all off!

Sorry for the last few months there has been little update. I decided that before I got too old I should enter a bodybuilding contest, honestly seemed like a good idea at the time. Well it’s been the hardest 3 months of my life to be honest. First off it isn’t the cheapest thing to do (I have spent £500 on chicken alone), you have to be super disciplined (up at 5, 1 hour cardio, training even if ill), you become very moody, tired and even get insomnia (writing this at 3am) because your constantly hungry! However with less than 2 weeks to go, I am still chuffed I decided to do it. On a positive side, because you don’t sleep as much, your brain seems to be come very creative!!!

So what’s the plan Batman?

So, I have many plans, lots of ideas and have been researching lots of stuff but I need to do some house keeping first!

1) Release a v1 of Hack24 cross platform to prove the framework works.

2) Fix Burf.co Search Engine because I turned it off.

3) Finish the garage so that large projects are possible 🙂

All of the above are in progress and I hope to have them done very soon. Then it is full speed ahead for some cool robotics / machine learning project that I will discuss in my next post 🙂

Burf.co paused, Hack24 Resumed…..

So I been on holiday driving across America, it was great fun!  I was hoping to be inspired on what to do with Burf.co, what’s the business plan, the focus etc.  Sadly absolutely nothing came to mind!  Not a dime!  Technically I can now index a lot of records pretty fast!  I have enough hardware to heat up a street of houses (more than my ring-main could handle) however I still don’t know what the point is.  So for the moment, it’s being paused!

With that in mind, I thought I would carry on with Hack24 and see if I can get it out the door.  I am still using LibGDX however, I have to moved to Multi-OS Engine for iOS and I am going to see if I can use FireBase for the backend.

Hopefully, by the time I have an MVP built, I should then know what I am going to do with Burf.co

 

 

Tumbleweed and that damn CommonCrawl

I hate by starting a post apologising for not updating my blog so I won’t do that!

I have been a bit busy with the new job I started 3 weeks ago, so most of my side projects have been paused!  However, work on Burf.co has gone 2 steps forward, a couple to the left and then a couple steps backwards, this is largely due to the awesome site CommonCrawl.org having a huge part of the Internet crawled and open for anyone to use! They have petabytes of web data open for anyone to use and there are some really cool examples of how to use it, most involving a huge amount of cloud power!! I did ponder for quite a while how I would store so much data!!! I found an interesting Java project that scans the index of the CommonCrawl for interesting file types (https://github.com/centic9/CommonCrawlDocumentDownload).

I took this project, hacked it about a bit and changed it so that it would only return URLs that are mine type HTML and that had a response status of 200. This gave me around 50 million URLs to play with which all had file pointers to the actual web page data. Because this data is compressed, it’s far quicker to download them from the CommonCrawl than actually scrapping the website itself. CommonCrawl also follows the Robot.txt which is far more than I have ever done :). So far the end result is that I can get around 5 million pages of data a day (from my home internet) compared to around 500k on a good day!  That’s a pretty good increase!

This weeks update : Bye Bye MongoDB

So it is exciting times! I have made some progress with TRTLExchange, however, due to things outside of my control it been slower than expected.  So I have turned my spare time to Burf.co, my new search engine project and while there is no website for it yet (will be by the weekend), the actual search technology(code) has come along leaps and bounds.  Overnight it managed to index over 500,000 pages which for a single server, was pretty cool.  It did get up to 1.3 million pages but MongoDB has erm, shit the bed(many many times).  This could be a hardware limit (Harddrive speed) or some performance thing I need to do however it gets to the point where I can’t even insert more records without timeouts.  This concerns me quite a bit as I have a HP Blade Server on way to somewhat up the crawling rate by a factor of 8.  I am going to try and give it one last go today however its taken 12 hours to delete the data from the DB (I did remove instead of drop 🙁 ).  It has been a very interesting learning curve on learning MongoDB.  I think unless some magic happens I am going to try out Postgres next.

On the Swift front I did start building the frontend for Burf, first I was going to do this in VueJS, however, I have now found that Swift’s server-side framework Perfect supports templating via Mustache.  I think I will make faster progress writing it all in Swift than switching back and forth.   I still want to continue learning VueJS on the side (used for the TRTLExchange) as Javascript is such a good thing to know nowadays.

Writing this blog post has also just raised the point that I was trying to learn Kotlin about a month ago (facepalm).  Damn!