Shift remote Jupyter Notebooks to run locally on MacOS with Pycharm

This year I've been working with IntelliJ so much that I decided to pay a little more and get the "All Packs" edition so I could use JetBrains IDEs with Python and .Net.  When I completed a Coursera course on Machine Learning, I wanted to have the Jupyter notebooks on my laptop so I'd have access to the materials in case I stopped my Coursera subscription.  The Jupyter notebooks for the course are a very good resource as I develop my own products that use machine learning, and I wanted them available on my Mac and Pycharm.  

Below are the steps and resources I used to: download the Jupyter notebooks and execute them on my MacBook.  Mac or Windows won't impact the steps much.  The I also included references to the resources I used work out what I needed to do but viewing them are not necessary as I'm showing the steps.  It took an hour to work out how to do this so hopefully this article will speed you along.

Tool chain: Web browser, Pycharm.  (Community edition of Pycharm doesn't allow execution of notebooks. I have the paid/Pro edition, which isn't very much for an individual developer.)

1) Download the entire lab files necessary to run the notebook

It's possible to just download and run a notebook but in my case, each notebook had dependencies with files outside of the notebook file (.ipynb file).  This required me to downloaded all the lab files.  This is likely a common situation so you'll probably need to do this too.  You'll know this is your case if when you look at the import statements in your notebook, you'll see items it's getting from custom modules that are available only as part of the notebook environment and cannot be pip installed later.  

An example of a notebook using custom modules

The following steps will produce a Zip file of the lab. Unzip the files to a location where you wish to keep this work.

Optional Reading: 

https://community.deeplearning.ai/t/downloading-your-notebook-downloading-your-workspace-and-refreshing-your-workspace/475495#1.2

2) Create a Pycharm project at that location. I used the "create venv" for interpreter.

Open Pycharm and create a project for the directory containing the zip file you downloaded. I didn't like the "Files" directory that wrapped the files in the zip so I moved those files out and removed the "Files" directory. I kept the "Files.zip" but renamed it to "Original Download Files.zip" so I'd later know what the contents are and I could have a backup if I later mucked up the notebooks too much.

Expanded the Zip file and moved the contents of the Files directory up one level.

Create the Pycharm project

Optional viewing:

https://youtu.be/4Ji-r455FVk

https://youtu.be/uiIKaacMGoE

3) Add dependencies that you know of into the Requirements.txt

Although the local dependencies needed by the notebook are available, there will be some others that were pip installed by the lab.  You can find out what those are by looking at the import statements at the top of each notebook. What I did is then add them to the Pycharm project's Requirements.txt. Or you can use the Pycharm's Python Packages tool.

 

This is an alternative to hand editing the Requirements.txt file.

4) Operate the notebook

Navigate to a notebook and execute it and see how it does. You may discover runtime errors at which point you'll need to determine what dependency you missed in which case look through the error message to discover what is missing, or click on the Pycharm AI button and it will help.  If you forget to install the dependencies listed in Requirements.txt, you'll be reminded when opening the notebook.

You'll know you've got your notebook sorted when all the cells execute without errors.

This is an example.

AndroidStudio Development with an M1 Mac

 

I got to use an M1 Mac as a loaner while my other Mac (2019, 32GB, MBP) was in the shop. So I used Time Machine to restore my work on the M1 MBP with 16GB. This M1 really is impressive in performance and using less power.  Having half the RAM of the Intel Mac wasn't noticeable at all.

Everyone should ditch their Intel and get one of these! That said, there are two impediments I had with getting work done: Android Studio and Open Broadcast Studio.  In this article I'll share how I resolved my issue with Android Studio.

The Issue 

Everything worked "out of the box" with Arctic Fox patch 4 (not a native M1 app), except I still couldn't get the emulator to launch.  But it actually was pretty easy to resolve once I learned a bit more about how Android emulation works.  

When I chose to launch my app, nothing would happen.  What was happening in my case is that it was trying to launch an emulator based on the x86 architecture.  So I needed to change the emulator.  The following is how. There are articles on StackOverflow, Reddit, and Medium but as AndroidStudio frequently releases new versions, a lot of the information is educational but the steps are quickly out of date.  (We are speeding toward the Singularity after all.)

Changing the Virtual Device

Note: As you go through the below images, you'll notice they are a tiny bit fuzzy. This is a compromise to get these big screenshots to fit in this webpage's layout. If you can't read something clearly, click on the image and it'll expand to full size and everything will be clear.

Go into the Android Vertical Device manager (AVD).

Create a new virtual device. 

Select the same phone type you were using before (I'm using Pixel 3a).

Download the version of Android OS you need (notice the ABI is for ARM).

Select the ARM emulator.  (Remember, Apple Silicon is an ARM architecture.)

After clicking finish you're ready to go back to AndroidStudio and launch the app in the emulator. (It will take more time on the first launch as it doesn't have things locally cached.)

Cleanup

When you're happy with the results of the new virtual device, go back into the Android Virtual Device manager and delete your old, unusable device and free up those GBs of storage.

Zoom Screen Sharing freezing on MacOS

You're all set. You've got the best presentation ready. Your green screen is shiny and your new pinstripe T-Shirt displaying the M1X logo on the chest is well pressed and lightly starched and you're totally killing it on your 4K webcam, but then one of your participants says, "Oh?  Did you change slides? Your screen sharing is frozen."  

OK.  So you close the sharing and turn it on again and it goes even worse: "Now it's a blank screen."

A NIGHTMARE! And you KNOW your hard wired connection to gigabit ethernet can't be the problem. And they say your are looking GOOD on the webcam. That means this is a screen sharing problem.

Here are some things that worked for me if you're living this nightmare:

Of the following two steps, doing the steps in #1 didn't resolve the issue.  Doing the steps for #2 did.

1. learn about your Mac's ability to switch video card, know how to identify what card it's using, and configure it to not switch when connected to power.
2. configure Zoom to avoid the situation

I included the steps of #1 because:

• sometimes more knowledge will save your bacon, 
• I wanted to capture these notes, and 
• if you're a live streamer like me (TDD.Academy Youtube, TDD.Academy Twitch), you may need to "amp things up" to increase your frame rates for your live stream.

It's possible after some time and experience, that I'll go back to #1 and allow my machine to automatically switch video chips while connected to power.  But for now, I've some experiments to try out.

Did you know your Mac could have more than one graphics processor?

Go to your battery settings to see if your Mac has "Automatic graphics switching." If it does, you've got more than one graphics processor.  Here are the steps:

For more background, read this article: https://it.umn.edu/services-technologies/how-tos/zoom-black-screen-during-screen-sharing

If you're curious about which GPU your machine is using at any time, either open Activity Monitor or click on the apple icon in the menu bar and select "About this mac."  Activity Monitor will tell you if they are using the "high performance" card or not.  About this mac will tell you which GPU (Graphics Processing Unit) is active. Typically the low power GPU is from Intel and the high performance one is from Radeon or Invidea.

To change or turn off the ability to switch GPUs, your can deselect the "Automatic graphics switching." I did this for when my machine is running on power.  I want it to select to the low power GPU when it's running on battery.  When I'm presenting my best face to the world, I'm usually connect to battery and hardwired to the LAN.

Zoom settings that solved this problem

Zoom has two interesting settings.  One of which, I'm sure resolved my problem.  Go into Zoom preferences and turn them both on if your having issues.