Machine Learning 101

#ML #Diary

Google has just released a series of videos to teach machine learning.

The first step is, however, installing and playing with Anaconda — a completely free Python distribution (including for commercial use and redistribution). It includes more than 400 of the most popular Python packages for science, math, engineering, and data analysis.

Choose the command line installer (on OSX) — it will save you a LOT of bother.

Installing Anaconda also means getting to know and love Conda — a package manager application that quickly installs, runs, and updates packages and their dependencies. It seems to be like pip, but better?

Conda has a test drive, which I am now trying out. Notes as I go along —

  1. Step one failed. I needed to try reinstalling using the command line installer. Chrome blocks the download as malicious, so I got the file using curl. Now running the installation. I had to edit .bash_profile to  edit the PATH variable to include the conda directory. Everything seems to be working now.
  2. I ran through the test drive in about half the suggested time. The most useful thing was this conda cheat sheet I downloaded.Key commands:
    Create an environment

    conda create -n snowflakes biopython

    Switch to the environment

    source activate snowflakes

    Remove an environment

    conda remove -n snowflakes --all

    Install a new package to an environment

    conda install -n snowflakes beautiful-so up
  3. Now creating an environment — calling it datalab –and installing the scikit-learn package

    conda create -n datalab scikit-learn

PyFactStream — Analysing news

Github: https://github.com/narendranag/PyFactStream

Just playing around with a very interesting library — Newspaper — https://github.com/codelucas/newspaper/

It solves the problem of scraping news content (though it doesn’t necessarily bring in full text on a lot of sites).

I piped this text through the OpenCalais text processing engine — http://www.opencalais.com/

Let’s see where I get with this 🙂

The Perfect Mac Dev Setup

Fire up your terminal.

  1. Install Xcode Command Line tools
    xcode-select --install 
  2. Install Homebrew
    ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" 
  3. Install Git
    brew install git 
  4. Install OhMyZSH
    sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)" 
  5. Download and Install Virtual Box
    Website 
  6. Download and Install Vagrant
    Website
  7. A LAMP EnvironmentNow to set up a LAMP environment. We’re using Scotch Box. More HereYou will get Ruby, PHP, Node, NPM, Composer along with MySQL, Postgres .. you get the picture. Visit their site for a complete download.
  8. Init the Scotch Box repo in your new project folder
    vagrant init scotch/box
  9. Get into the folder and start the box
    vagrant up

Predicting The Future

#NotesFromSXSW2016

By Kevin Kelly

AI and robots will create new jobs for humans – jobs that are focused on productivity over all else will be done by AI/robots. Jobs that need creativity and the willingness to fail will be done by robots. People with the ability to work with AI will be valued.

Virtual Reality and Mixed/Augmented Reality will lead us to an Internet of experiences. In VR you no longer watch scenes but viscerally experience the environment leading to memory. The hard problem is tracking your body and providing tactile feedback, not creating the world. Best example Void (redirected walking). For AR Mega

VR will become the most social of social media. It is inherently a social experience.

Personalisation and Tracking: anything that can be tracked, will be tracked. In VR your whole self is being tracked. The tracking will become more civilised because of co-veillance. We can track the trackers. There is a correlation between privacy and generic experiences, and transparency and personalisation

The great products of the next 20 years haven’t been invented yet.

The State of Blockchain 2016

#NotesFromSXSW2016

Background: https://en.m.wikipedia.org/wiki/Block_chain_(database)

HyperLedger – Blockchain project supported by the Linux Foundation. The objective is to create a kernel for the tech (like the Linux kernel). Companies will be able to build their offerings on top of a shared kernel. This gets launched at the end of the year and it will be the most significant software launch of the year.

Ethereum – Blockchain for contracts (and everything else). Andrew Keys calls it Facebook vs MySpace where Bitcoin is MySpace to Ethereum’s Facebook. In six months, the startup has hit a billion dollar valuation.

Blockchain is a new network of contracts and agreements. The smart contract is the killer app. The litigators of tomorrow will understand both Blockchain as well as the law to be able to handle disputes in the era of smart contracts.

The legal profession is about to be disrupted massively over the next 5-10 years as dispute resolution is baked into code.

 In the next 10-15 years, all payment systems are going to be replaced by Blockchain based systems. This will require, and drive, massive changes in financial policy at the local, national and global level. 

Blockchain is opening up a Pandora’s box of questions around the nature of a transaction, any transaction – from person to person to securities transactions.

This technology solves the trust problem: trust between people traditionally has been provided by a middleman (a bank or a notary for example). The Blockchain now provides that trust technologically – removing the middleman. 

This clearly has massive regulatory consequences.

As companies retool themselves for this world and a new regulatory environment, it represents a massive business opportunity for IT services companies that can bring themselves up to speed quickly enough to be able to provide transition services for everything from banks to airlines and schools.

Serious challenges exist for governments as they try to impose financial controls on a completely decentralized financial technology layer.

Programming a new life language 

#NotesFromSXSW2016

By Aaron Kimball

The panel submission deck:  http://www.slideshare.net/WhitneyJencks/zymergen-sxsw-2016-submission

Additional Background: http://www.pbs.org/wgbh/nova/next/evolution/crispr-gene-drives/

 

The problem: DNA is incredibly complex and it presents a data challenge. It is not possible to test every combination in which the base pairs arrange themselves.

While the cost of sequencing has fallen, it is not simple to figure out what any particular sequence of symbols does. Interpreting DNA is hard – the conventional method sees scientists try out different hypotheses, experimenting in wet labs, with potential combinations. The downside – of 10,000 attempts we might see a minor result in one. This makes this whole process hard, expensive, and time consuming.

Zymergen has built out a robotic process to automate this, allowing for many experiments to be run in parallel.

The problem is the amount, and kind, of data being generated. 

Fun fact: 93% of all chemicals in use comes from petroleum. Only 6% come from industrial fermentation. However as oil runs out, microbe-based chemical production processes becomes super important. But that needs us to be able to manipulate genes in microbes, designing better microbes.

Four phases:  

A new suite of software is allowing for high throughput microbe design and testing:  

Codon is a language that allows scientists to define a design idea – a gene manipulation within a microbe.  

A sequence looks like this – promoter + gene + terminator. The promoter defines how much the gene expresses itself. For example, how blonde will your hair be – platinum or just a dirty yellow.

The language allows scientists to very quickly create experiments that can test multiple permutations

The rest of the process allows for automation, speed and quick analysis on the data using a sophisticated software stack.

There are inbuilt decision trees based off previous non-machine test results.

The expected outcome is better chemicals that can lead to safer pesticides, plastics that break down, even better medicines.

Planning for Moments

#NotesFromSXSW2016

By Kiip

The real impact of IoT on marketing
Connected devices allows us to plan for and measure moments

Allows us to model intent on the basis of usage of devices – moments are directly connected to an action/series of actions by a consumer – typically triggered through app activity

Passive Moments leads to automation leads to detectable intent triggers/ moments which are non intrusive with permission built in

Moment Types: 
By adding proximity to the equation we start to build connected moments (how does connections planning change when we move from connecting funnel activations to moment activations)

Example: Rewards based on understanding of upcoming moments

Example: Oral B at MWC

Predict intent / upcoming moment using narrow AI that is digesting device data signals.

Understanding the new nature of instant gratification – The “connect” and “instant” generation: postmates/instacart/uber

Messaging gives us moments driven by human phrases – machines understanding and responding to natural language prompts from a human

The Connected Generation: the death of traditional segment (by age etc). Segment by intent in the moment.

We are moving to a Moments based CRM: 
Far more powerful than content based intent modeling. As long as brands add value to the moment, instead of abusing it for push-based messaging.

The World Without Glasses

Without my glasses on, the world immediately becomes less familiar
The warts are hidden, the world’s minor flaws can’t be seen
In the evenings, every light in every building twinkles in the haze
Cars flit by on the distant road like red and yellow fireflies
The manmade river in this manicured city glitters pretentiously
People dart around, it seems everyone has plans and places to go
Except for me, staring out of my window without my glasses on

Essential Free Software For The Mac

A new Mac buyer recently asked me for the best starter set of software for a new Mac. I am very Apple — which means I have an iPhone, a Mac, an Apple TV and an iPad, so I like things that let me work across all my devices seamlessly. For all office work I highly recommend Apple’s Pages, Numbers and Keynote apps — they are very very good, are free, and work everywhere (except the Apple TV). You won’t find a note-taking app in my list because I’m using Apple’s inbuilt Notes. I also use Mail, Calendar and Address Book — all default apps. And I can’t even begin to talk about how good iMovie is for video editing (I have been editing video since 2005, and while my professional video editing days are behind me, I love how easy iMovie makes life for me).

Anyway, here’s my list (and I would recommend installing in this order):

Essential Software:

Onyx
System Cleaner
Run it once and then set it up to auto-execute on a weekly/monthly basis. It will clear your system of un-needed files.
http://www.titanium.free.fr/onyx.html

Dropbox (iOS + OSX)
Cloud Storage
I used only free storage for around a year, now I pay for a Terabyte of storage. I cannot imagine life without Dropbox.
https://www.dropbox.com/install

Cyberduck
FTP Client
And not just FTP, supports S3, Azure etc.
https://cyberduck.io/

Transmission
Torrent Client
My favorite way to download files — uTorrent is a very good alternative, but for some reason I’ve always preferred Transmission.
http://www.transmissionbt.com/download/

The Unarchiver
File Compression
Zip, Rar and almost every other file compression format is supported
https://itunes.apple.com/app/the-unarchiver/id425424353?mt=12&ls=1

VLC
Media Player
Plays every media file out there. I use it instead of Quicktime.
http://www.videolan.org/vlc/index.html

Handbrake
Video Format Transcoder
This is what I use to convert video files into formats supported by iTunes so I copy stuff over to my phone/ipad.
https://handbrake.fr/downloads.php

Brackets
Text Editor
If you code, you’re going to love this. If you like text-editors, you are going to love this.
http://brackets.io/

Gimp
Photoshop Replacement
As powerful as Photoshop and ridiculous overkill for most people.
https://www.gimp.org/macintosh/

Pixelmator (Especially, if Gimp feels like too much)
Image Editing
Powerful image editing — most of you will be happy with this.
http://www.pixelmator.com/mac/

Inkscape
Illustrator Replacement
Ridiculously good. Seriously.
https://inkscape.org/en/download/mac-os/

Pocket (iOS + OSX)
Article Reader / Saver
My favorite way to save articles I live/love — I have a pocket button in my browser, the app on all my devices and I love it. And I love the clean interface it gives me to read articles.
https://itunes.apple.com/us/app/pocket/id568494494?mt=12

Reeder (iOS + OSX)
RSS Reader
Old school 🙂 Still use an RSS Reeder and send almost everything I like to Pocket to read later.
http://reederapp.com/

Monolingual
Removes unwanted language files (saved 3+ GB of space when I ran it recently)
https://ingmarstein.github.io/Monolingual/

Choices; Places

What is any place
Is it a promise
dreamt in the night
of golden days ahead

Is it the people
that brush past you
when you need them most
to stop and say they love you

Or is it the secret life
hidden on every street
that seduces you and makes
you feel like you belong

What is any place
but a bookmark on life’s journey
sometimes a whisper in the wind
sometimes a stain on the soul

How far do I travel
Where do I rest
Who do I trust
What choices do I make

I want to hang my hat
but I don’t own one
I want to put up my feet
but am scared of doing it alone

Is a place a choice
or is it an inevitable
cruel joke, unsolicited advice
nonsense so incredible

That life itself rebels,
rejects, tears asunder
any hope, any happiness
any chance of a better tomorrow.