Friday, June 24, 2022

MacOS - take screenshots in udemy courses

It appears Udemy is blocking screenshots.
Taking screenshots of important points in the courses was, until now, my way to quickly take notes.
 
Now Udemy block screenshots and all you get is a black screenshot. I have not seen any anouncement about this “feature” and was taken by surprise. Spent too much time assuming it was an issue with my Mac...
 
Anyhow, there seem to be a way to overcome this limitation, at least for now.
 
I tested this on Chrome and Edge, and I assume that Firefox would work the same.
 
Here is what to do:
Go to settings of your browser, and search for hardware acceleration configuration.
Turn it off.
 
That’s it!
 
Now you can continue taking screenshots on Udemy courses, and you should not get black screen on your screenshot.
 
Good luck and productive learning!

Wednesday, April 7, 2021

McFly - Mac terminal shell history search (Ctrl+R) replacement

With my bad memory I constantly use Ctrl+R to find previously typed commands in the terminal.
McFly makes this option much more pleasant by providing an intellignet search eninge that is much smarter than the default.
 
Looks like this:
 
Install on Mac:
 
1. 
brew tap cantino/mcfly https://github.com/cantino/mcfly

2.

brew install mcfly

3. Zsh config:

eval "$(mcfly init zsh)”


 
 
 
 
 
 

Friday, April 2, 2021

VSCode debugging Django (python) website and javascript with breakpoints in both

I was looking for a way to make VSCode debugger to stop on breakpoints both in python and javascript while debugging Django app.
Python and templates breakpoints work well with django launch.json settings, but javascript breakpoints are ignored.
 
 
What I found is a workaround (I don’t know if this is just a standard way to do this, but feels like workaround), by using the `compouns` option in the launch.json.
It is basically an ability of VSCode to debug two apps in parallel. For example when you have a server/client app and you want to debug both simultaneously.
 
Anyway, for Django, I used a regular django config, and a chrome web debug for javascript:
 
 
 
 
A new option is then available in the Run and Debug dorpdown menu: "Django/Web"
This way it works great.
 
 

Wednesday, March 31, 2021

AWS - Retry messages from Dead Letter Queue

I have a SQS queue and a DLQ defined. 
Some messages were moved to DLQ because of some temoporary issue, so I wanted to move them back to the original queue for processing after the issue resolved.
The fastest way was:
1. Set the original queue as the DLQ on the DLQ queue with max receives 1
2. View the messages on DLQ, which moves the messages.
3. Remove the setting in 1.
 
I know there are some tools like this: https://github.com/garryyao/replay-aws-dlq , but in my case the above was the fastest solution.

Sunday, March 28, 2021

pip install issue - Retrying (Retry..

On my home Windows 10 I was getting an error when calling `pip install` command:

Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out.

I’ve seen posts where others had the same issue on Linux systems.

The solution for this issue was to disable IPv6.

On Windows, go to Properties of the network adapter, and find “Internet Protocol Version 6 (TCP/IPv6). Uncheck the checkbox and reboot (in my case reboot was not needed, but it is advised).

Saturday, March 27, 2021

Cunningham’s Law–Ask Dumb Things to Get Smart Answers

Cunningham's Law: "The best way to get the right answer on the Internet is not to ask a question, it's to post the wrong answer."

Sherlock Holmes in The Great Game says, “people do not like telling you things; they love to contradict you”. Therefore if you want smart answers, do not ask a question. Instead, give a wrong answer or ask a question in such a way that it already contains the wrong information. It is highly likely that people will correct you.

Thursday, March 18, 2021

Terminal to Finder and back

On Mac it is usefull to switch between the Finder and a terminal window.
 
To open current Terminal path in Finder type:
 
open ./
 
To navigate in Terminal to some folder shown in the Finder, just drop it on the Terminal window.
 
 
 

Python profiling on Mac (Big Sur)

 I needed to profile a python app on my mac. Was it too much to ask? Apparently it was...

Anyhow for the most basic stuff the following articles helped a lot:

Python profiling

Handy Python profiling analysis with pstats interactive browser

I prefer not to change my code so call cProfile from the terminal like this:

python -m cProfile -o prof_results my_code.py

which would generate the prof_results file with the results.

Then call:

python -m pstats prof_results

to view the results.

`pstats` can filter and sort the output, but I couldn't find a way to show a nested view.


It seems that there are a few packages out there that are supposed to help better visualize the profiling results, but all/most of them are outdated or couldn't deal with the complexity of my output.

The only/best result I finally settled on was to use Spyder. 

Installation on Big Sur was not pleasant.  dmg installation is not working for me, and other options were problematic.

pip install spyder worked!

In order to profile in spyder follow this: Spyder Profiler



 

Tuesday, September 29, 2020

Awesome Machine Learning Repo

 A curated list of awesome machine learning frameworks, libraries and software (by language).

https://github.com/josephmisiti/awesome-machine-learning

Awesome Python Repo

A curated list of awesome Python frameworks, libraries, software and resources.

https://github.com/vinta/awesome-python

https://github.com/AhmadIbrahiim/awesome-python



Monday, August 3, 2020

Create Sphinx based documentation with Markdown

Introduction


Our latest project documentation was created with Sphinx. 

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation.

Sphinx is natively using reStructuredText format, but we will use markdown, as it is more widely known and used.

Installation

Sphinx is a python package, so create a new python project and install the following packages:

  • Sphinx - sphinx package
  • recommonmark - package for markdown support
  • sphinx-markdown-tables - package for markdown tables support
  • sphinx-rtd-theme - a theme package


Initialisation


Sphinx comes with a script called sphinx-quickstart that sets up a source directory and creates a default conf.py with the most useful configuration values from a few questions it asks you. To initialise a new project just run this:

sphinx-quickstart

Then answer a few questions (choose to have source and build separate) to complete the setup.

Project structure

Quick start command will generate a few folders and files.

  • source directory - is where sll the content should be stored.
  • conf py file - this is the file where all the project configurations are stored
  • index.rst - this is the main documentation page. It is the only file that we will keep in rst format.
  • make.bat - called to generate output


Configuration

All configuration is done in conf.py file. The file contains comments explaining each section. We will configure the following:

Support for markdown format (in addition to native rst support):

source_suffix = {
    '.rst': 'restructuredtext',
    '.md': 'markdown'
}

Packages we need to support markdown:

extensions = [
    'recommonmark',
    'sphinx_markdown_tables'
]

Theme and theme settings:

html_theme = 'sphinx_rtd_theme'

html_theme_options = {
    'logo_only': True,
    'display_version': True,
    'prev_next_buttons_location': 'bottom',
    'style_external_links': True,
    # Toc options
    'collapse_navigation': False,
    'sticky_navigation': False,
    'navigation_depth': 3,
    'includehidden': True,
    'titles_only': False
}
html_title = "My amazing documentation"
html_logo = "path/to/logo.png"
html_favicon = "path/to/favicon.ico"


Index setup

Index.rst file is the master document. The main function of the master document is to serve as a welcome page, and to contain the root of the “table of contents tree” (or toctree). By default it contains a single toctree entry, but in order to have nice sidebar sections, we use multiple toctrees.

.. toctree::
   :maxdepth: 1
   :hidden:
   :caption: Section 1:

   docs/MyDoc1.md
   docs/MyDoc1.md

.. toctree::
   :maxdepth: 2
   :caption: Section 2:

   docs/MyDoc3.md
   docs/MyDoc4.md
   docs/MyDoc5.md
  • Documents path specified as relative to source directory
  • :maxdepth: - how many sub-section levels to show (relevant only for the welcome page, not the sidebar)
  • :hidden: - in case we want to hide the section from the main page. (in our plugin configuration we stated that we want to use hidden section in sidebar: ‘includehidden’:True)


Markdown pages

All the documentation content is stored in .md files in markdown standard format (same format that is used for wikis on github/gitlab/etc) The only format requirement that is needed for Sphinx in order to generate a consistent menu bar is to have # Title in the file. It will be used by the toctree. 



Generating HTML

When all the changes are completed on the markdown files, or after each change, run the following command to generate html output:

make html

It will output the html website to /build/html directory. Console output is very useful for identifying any issues with content (for example it will alert you if a document exists that is not included in toctree).

You can now view your documentation in your favorite browser.

Result Example



Hope this helps someone.

Sunday, September 8, 2019

AWS EMR some basic learnings (python and spark)

As a new user of this service it was a bit confusing to start with, especially as there seem to be endless number of contradicting articles about how to add steps and what they should execute.

My main issue was with this part of the documentation:

Steps=[
{
'Name': 'string',
'ActionOnFailure': 'TERMINATE_JOB_FLOW'|'TERMINATE_CLUSTER'|'CANCEL_AND_WAIT'|'CONTINUE',
'HadoopJarStep': {
'Properties': [
{
'Key': 'string',
'Value': 'string'
},
],
'Jar': 'string',
'MainClass': 'string',
'Args': [
'string',
]
}
},
],


It looks like Jar is not an optional parameter, so how am I supposed to run python? I don’t have any jars...

So here is what I discovered:
AWS EMR provides two jars:

script-runner.jar and command-runner.jar

script-runner.jar can execute scripts, so it will get a script file as a parameter.
command-runner.jar is simillar to ssh connection and running commands.

For my use-case i think the command-runner was the best fit, so for a simpliest command of running spark-submit command with a python file my step becomes:


{
"Name": “Python Step",
"ActionOnFailure": "CONTINUE",
'HadoopJarStep': {
"Properties":[],
"Jar":"command-runner.jar",
“Args": [
'spark-submit',
's3://buket/my_spart_python_file.py',
]
}
}

Args tranlates each , to space, so the command that is going to be executed is: "spark-submit s3://buket/my_spart_python_file.py"

Any additional parameter I would like to execute, I would just add it with , and construct the command just as if I’m running bash commands.

For example '--executor-memory’, ‘5g’, ‘—something-else’, ‘else'

Also, to enable logging and debuging need to add another step:
{
'Name': 'Setup Hadoop Debugging',
'ActionOnFailure': 'TERMINATE_CLUSTER',
'HadoopJarStep': {
'Jar': 'command-runner.jar',
'Args': ['state-pusher-script']
}
}

Thursday, August 22, 2019

OnePlus 6T Windows 10 file access issue (Shows Drivers CD only)

OnePlus 6T – when connecting to USB port shows only the CD Drive that contains the drivers.

Tried:

1. Installing the drivers from CD drive

2. Enabled Developer options and File Access.

3. Installing and uninstalling MTP from device manager (shows as issue).

4. Rebooting both the device and the PC

5. Many more – nothing helped.


Solution:

Installing “Media Feature Pack for N versions of Windows 10” and rebooting.

Now the device shows as expected both in the device manager and Windows explorer"