Awesome Machine Learning Repo

 A curated list of awesome machine learning frameworks, libraries and software (by language).

https://github.com/josephmisiti/awesome-machine-learning

Awesome Python Repo

A curated list of awesome Python frameworks, libraries, software and resources.

https://github.com/vinta/awesome-python

https://github.com/AhmadIbrahiim/awesome-python

Build your own X (repo)

 

https://github.com/danistefanovic/build-your-own-x

Repo with many web dev resources (save for future reference)

https://github.com/bradtraversy/design-resources-for-developers

Create Sphinx based documentation with Markdown

Introduction

Our latest project documentation was created with Sphinx. 

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation.

Sphinx is natively using reStructuredText format, but we will use markdown, as it is more widely known and used.

Installation

Sphinx is a python package, so create a new python project and install the following packages:

• Sphinx - sphinx package
• recommonmark - package for markdown support
• sphinx-markdown-tables - package for markdown tables support
• sphinx-rtd-theme - a theme package

Initialisation

Sphinx comes with a script called sphinx-quickstart that sets up a source directory and creates a default conf.py with the most useful configuration values from a few questions it asks you. To initialise a new project just run this:

sphinx-quickstart

Then answer a few questions (choose to have source and build separate) to complete the setup.

Project structure

Quick start command will generate a few folders and files.

• source directory - is where sll the content should be stored.
• conf py file - this is the file where all the project configurations are stored
• index.rst - this is the main documentation page. It is the only file that we will keep in rst format.
• make.bat - called to generate output

Configuration

All configuration is done in conf.py file. The file contains comments explaining each section. We will configure the following:

Support for markdown format (in addition to native rst support):

source_suffix = {
    '.rst': 'restructuredtext',
    '.md': 'markdown'
}

Packages we need to support markdown:

extensions = [
    'recommonmark',
    'sphinx_markdown_tables'
]

Theme and theme settings:

html_theme = 'sphinx_rtd_theme'

html_theme_options = {
    'logo_only': True,
    'display_version': True,
    'prev_next_buttons_location': 'bottom',
    'style_external_links': True,
    # Toc options
    'collapse_navigation': False,
    'sticky_navigation': False,
    'navigation_depth': 3,
    'includehidden': True,
    'titles_only': False
}
html_title = "My amazing documentation"
html_logo = "path/to/logo.png"
html_favicon = "path/to/favicon.ico"

Index setup

Index.rst file is the master document. The main function of the master document is to serve as a welcome page, and to contain the root of the “table of contents tree” (or toctree). By default it contains a single toctree entry, but in order to have nice sidebar sections, we use multiple toctrees.

.. toctree::
   :maxdepth: 1
   :hidden:
   :caption: Section 1:

   docs/MyDoc1.md
   docs/MyDoc1.md

.. toctree::
   :maxdepth: 2
   :caption: Section 2:

   docs/MyDoc3.md
   docs/MyDoc4.md
   docs/MyDoc5.md

• Documents path specified as relative to source directory
• :maxdepth: - how many sub-section levels to show (relevant only for the welcome page, not the sidebar)
• :hidden: - in case we want to hide the section from the main page. (in our plugin configuration we stated that we want to use hidden section in sidebar: ‘includehidden’:True)

Markdown pages

All the documentation content is stored in .md files in markdown standard format (same format that is used for wikis on github/gitlab/etc) The only format requirement that is needed for Sphinx in order to generate a consistent menu bar is to have # Title in the file. It will be used by the toctree. 

Generating HTML

When all the changes are completed on the markdown files, or after each change, run the following command to generate html output:

make html

It will output the html website to /build/html directory. Console output is very useful for identifying any issues with content (for example it will alert you if a document exists that is not included in toctree).

You can now view your documentation in your favorite browser.

Result Example

Hope this helps someone.

AWS EMR some basic learnings (python and spark)

As a new user of this service it was a bit confusing to start with, especially as there seem to be endless number of contradicting articles about how to add steps and what they should execute.

My main issue was with this part of the documentation:

Steps=[
        {
            'Name': 'string',
            'ActionOnFailure': 'TERMINATE_JOB_FLOW'|'TERMINATE_CLUSTER'|'CANCEL_AND_WAIT'|'CONTINUE',
            'HadoopJarStep': {
                'Properties': [
                    {
                        'Key': 'string',
                        'Value': 'string'
                    },
                ],
                'Jar': 'string',
                'MainClass': 'string',
                'Args': [
                    'string',
                ]
            }
        },
    ],

It looks like Jar is not an optional parameter, so how am I supposed to run python? I don’t have any jars...

So here is what I discovered:
AWS EMR provides two jars:

script-runner.jar and command-runner.jar

script-runner.jar can execute scripts, so it will get a script file as a parameter.
command-runner.jar is simillar to ssh connection and running commands.

For my use-case i think the command-runner was the best fit, so for a simpliest command of running spark-submit command with a python file my step becomes:

{
"Name": “Python Step",
"ActionOnFailure": "CONTINUE",
'HadoopJarStep': {
"Properties":[],
"Jar":"command-runner.jar",
“Args": [
'spark-submit',
's3://buket/my_spart_python_file.py',
]
}
}

Args tranlates each , to space, so the command that is going to be executed is: "spark-submit s3://buket/my_spart_python_file.py"

Any additional parameter I would like to execute, I would just add it with , and construct the command just as if I’m running bash commands.

For example '--executor-memory’, ‘5g’, ‘—something-else’, ‘else'

Also, to enable logging and debuging need to add another step:
{
'Name': 'Setup Hadoop Debugging',
'ActionOnFailure': 'TERMINATE_CLUSTER',
'HadoopJarStep': {
'Jar': 'command-runner.jar',
'Args': ['state-pusher-script']
}
}

OnePlus 6T Windows 10 file access issue (Shows Drivers CD only)

OnePlus 6T – when connecting to USB port shows only the CD Drive that contains the drivers.

Tried:

1. Installing the drivers from CD drive

2. Enabled Developer options and File Access.

3. Installing and uninstalling MTP from device manager (shows as issue).

4. Rebooting both the device and the PC

5. Many more – nothing helped.

Solution:

Installing “Media Feature Pack for N versions of Windows 10” and rebooting.

Now the device shows as expected both in the device manager and Windows explorer"

Jenkins Groovy issue with disappearing quotes

One of the really annoying issues in Jenkins I’ve encountered recently is disappearing double and single quotes.
There are solutions out there with using double tripple or even more quotes, but that didn’t work for us.
My colegue David found a nice soluton which works 100% of times for us:

Surround your string with $/ {my string with quotes} /$ instead of outer quotes. Now we can use any combination of quotes inside the string without any issues.

Thanks David!

AWS Cognito - Get token with ADMIN_NO_SRP_AUTH (Python)

try:
    return boto3.Session(profile_name=’[PROFILE_NAME]’)

.client('cognito-idp', [REGION]).admin_initiate_auth(
        UserPoolId=[USER_POOL_ID],
        ClientId=[CLIENT_ID],
        AuthFlow='ADMIN_NO_SRP_AUTH',
        AuthParameters={
            'USERNAME': [USER_NAME],
            'PASSWORD': [PASSWORD],
            'SECRET_HASH': get_secret_hash([UserName])
        }
    )
except botocore.exceptions.ClientError as e:
    return e.response

# creating secret hash

def get_secret_hash(self, username):
    message = username + self.client_id
    dig = hmac.new(bytes(self.client_secret, encoding='utf-8'), msg=message.encode('UTF-8'),
                   digestmod=hashlib.sha256).digest()
    hash_is = base64.b64encode(dig).decode()
    return hash_is

Default Angular 6 (and earlier) doesn’t support Internet Explorer 11 (IE11) by default

It is actually something that was apparently planned.

In order to resolve this issue, just open polyfills.ts and uncomment the relevant section.

For me it was:

iPhone is not recognized by Windows 10 1803 April 2018 Update

After an update to 1803 iPhone 6S+ was not visible in the Windows Explorer.

Device manager showed driver unrecognized.

All the Apple support articles suggested updating drivers, etc.., but nothing helped.

For example:

1. This one

2. And this

3. And this

I’ve lost all hope, but then Firefox (I’m giving it a new chance to replace Chrome) suggested to download and install Media Feature Pack from Microsoft to improve video playback.

Miraculously, after a reboot, my iPhone was recognized.

Git – merging conflicts

git commit -am <message>       # add & commit your local changes
git pull origin master         # pull the latest codes
git status                     # see the files that have conflicted (untracked files)

# If you want to keep your changes then
git checkout . --ours

# Or, if you want to keep remote changes
git checkout . --theirs

git commit -am <message>           # add & commit with a new message
git push origin master             # push to remote  

source: https://stackoverflow.com/a/40800624/260238

Angular (First Blood) Resources

https://cli.angular.io/ – Angular CLI

https://update.angular.io/ – a wizard for upgrading Angular Versions

https://material.angularjs.org – Angular Material (also a nice guide here: https://medium.com/@tomastrajan/the-complete-guide-to-angular-material-themes-4d165a9d24d1)

Firefox 35–Fails to login to twitter

Had this issue where Firefox fails to login to twitter, just retuning me to the login screen. Tested with Chrome – no issue there.

The solution was to clear the cookies for Twitter in Firefox.

Options->Privacy Tab –>Remove individual Cookies->Search for Twitter.com and remove all the relevant cookies.

Issue resolved.

Fast way to open CMD or Powershell from Windows Explorer

 

This is an amazing tip I found as a comment on the Rick Strahl’s Web Log (by peSHIr).

So, I’ve usually used Shift + Right Click to get to the “Open Command Window Here” option in the menu. It works, but the following tip is much nicer.

Just type cmd or powershell (overwrite the existing path) in the path field of the Windows Explorer, and hit Enter.

That is it! The cmd will just open directly in the path you were browsing in your Windows Explorer.

Great tip!

 

Thanks peSHIr