VSCode debugging Django (python) website and javascript with breakpoints in both

I was looking for a way to make VSCode debugger to stop on breakpoints both in python and javascript while debugging Django app.

Python and templates breakpoints work well with django launch.json settings, but javascript breakpoints are ignored.

 

 

What I found is a workaround (I don’t know if this is just a standard way to do this, but feels like workaround), by using the `compouns` option in the launch.json.

It is basically an ability of VSCode to debug two apps in parallel. For example when you have a server/client app and you want to debug both simultaneously.

 

Anyway, for Django, I used a regular django config, and a chrome web debug for javascript:

 

 

 

 

A new option is then available in the Run and Debug dorpdown menu: "Django/Web"

This way it works great.

 

 

pip install issue - Retrying (Retry..

On my home Windows 10 I was getting an error when calling `pip install` command:

Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out.

I’ve seen posts where others had the same issue on Linux systems.

The solution for this issue was to disable IPv6.

On Windows, go to Properties of the network adapter, and find “Internet Protocol Version 6 (TCP/IPv6). Uncheck the checkbox and reboot (in my case reboot was not needed, but it is advised).

Python profiling on Mac (Big Sur)

 I needed to profile a python app on my mac. Was it too much to ask? Apparently it was...

Anyhow for the most basic stuff the following articles helped a lot:

Python profiling

Handy Python profiling analysis with pstats interactive browser

I prefer not to change my code so call cProfile from the terminal like this:

python -m cProfile -o prof_results my_code.py

which would generate the prof_results file with the results.

Then call:

python -m pstats prof_results

to view the results.

`pstats` can filter and sort the output, but I couldn't find a way to show a nested view.

It seems that there are a few packages out there that are supposed to help better visualize the profiling results, but all/most of them are outdated or couldn't deal with the complexity of my output.

The only/best result I finally settled on was to use Spyder. 

Installation on Big Sur was not pleasant.  dmg installation is not working for me, and other options were problematic.

pip install spyder worked!

In order to profile in spyder follow this: Spyder Profiler

 

AWS EMR some basic learnings (python and spark)

As a new user of this service it was a bit confusing to start with, especially as there seem to be endless number of contradicting articles about how to add steps and what they should execute.

My main issue was with this part of the documentation:

Steps=[
        {
            'Name': 'string',
            'ActionOnFailure': 'TERMINATE_JOB_FLOW'|'TERMINATE_CLUSTER'|'CANCEL_AND_WAIT'|'CONTINUE',
            'HadoopJarStep': {
                'Properties': [
                    {
                        'Key': 'string',
                        'Value': 'string'
                    },
                ],
                'Jar': 'string',
                'MainClass': 'string',
                'Args': [
                    'string',
                ]
            }
        },
    ],

It looks like Jar is not an optional parameter, so how am I supposed to run python? I don’t have any jars...

So here is what I discovered:
AWS EMR provides two jars:

script-runner.jar and command-runner.jar

script-runner.jar can execute scripts, so it will get a script file as a parameter.
command-runner.jar is simillar to ssh connection and running commands.

For my use-case i think the command-runner was the best fit, so for a simpliest command of running spark-submit command with a python file my step becomes:

{
"Name": “Python Step",
"ActionOnFailure": "CONTINUE",
'HadoopJarStep': {
"Properties":[],
"Jar":"command-runner.jar",
“Args": [
'spark-submit',
's3://buket/my_spart_python_file.py',
]
}
}

Args tranlates each , to space, so the command that is going to be executed is: "spark-submit s3://buket/my_spart_python_file.py"

Any additional parameter I would like to execute, I would just add it with , and construct the command just as if I’m running bash commands.

For example '--executor-memory’, ‘5g’, ‘—something-else’, ‘else'

Also, to enable logging and debuging need to add another step:
{
'Name': 'Setup Hadoop Debugging',
'ActionOnFailure': 'TERMINATE_CLUSTER',
'HadoopJarStep': {
'Jar': 'command-runner.jar',
'Args': ['state-pusher-script']
}
}

AWS Cognito - Get token with ADMIN_NO_SRP_AUTH (Python)

try:
    return boto3.Session(profile_name=’[PROFILE_NAME]’)

.client('cognito-idp', [REGION]).admin_initiate_auth(
        UserPoolId=[USER_POOL_ID],
        ClientId=[CLIENT_ID],
        AuthFlow='ADMIN_NO_SRP_AUTH',
        AuthParameters={
            'USERNAME': [USER_NAME],
            'PASSWORD': [PASSWORD],
            'SECRET_HASH': get_secret_hash([UserName])
        }
    )
except botocore.exceptions.ClientError as e:
    return e.response

# creating secret hash

def get_secret_hash(self, username):
    message = username + self.client_id
    dig = hmac.new(bytes(self.client_secret, encoding='utf-8'), msg=message.encode('UTF-8'),
                   digestmod=hashlib.sha256).digest()
    hash_is = base64.b64encode(dig).decode()
    return hash_is