Thi's avatar
HomeAboutNotesBlogTopicsToolsReading
About|My sketches |Cooking |Cafe icon Support Thi
πŸ’Œ [email protected]

Jupyter notebook

Anh-Thi Dinh
SkillsData ScienceDeep LearningMachine LearningPythonTools
Left aside
☝
If you use VSCode, you should use its Jupyter Notebook extension, it's quick, clean and very easy to use.

Installation

Jupyter notebook

Or read more in Python Installation.
If you meet error OSError: [Errno 99] Cannot assign requested address, try

Setting up a password

Inside notebook:
With docker

R with jupyter notebook

Read more here.

Run JS codes

Import local package

Suppose that we have a folder structure like this,
I want to call function get_youtube_video_metadata() inside transcribe.py. However, in that file, I import from app.constant import ....
In the jupyter notebook,
Don’t forget to add __init__.py inside the app folder.

Use the running kernel in VSCode

Suppose that you are running a kernel at
In VSCode β†’ Open a notebook β†’ Click Select Kernel on the top right of the notebook β†’ Existing jupyter server β†’ … β†’ paste the above url and hit Enter.

Environment variables

Other tips

  • Running 2 tasks in the same cell TAKE LONGER TIME than running each on different cells.
  • Download a folder in jupyter notebook:
    • Inside notebook, use:
    • Or using nbzip (only working on current server).

Check the info

Check where command executed from (in your $path)?

Multiline commands

You CANNOT put # comments at the end of each line break!

Hotkeys / Shortcuts

There are 2 modes: command mode (pres ESC to activate) and edit mode (Enter to activate). Below are the most useful ones (for me).
You can edit / add more shortcuts in Help > Edit Keyboard Shortcuts.

Jupyter notebook on remote server

Open jupyter notebook in local browser but the backend-server is on remote.
  • If jupyter server is already running on remote at http://192.168.0.155:9889
      • Open browser: http://localhost:9888 (type password if needed).
  • If jupyter server is not running on remote yet,
      • On remote,
      It's running and there are somethings like that,
      Open another terminal window and type,
      Open browser:
You can choose any port number you wanna instead of 9888 and 9889 (they can be the same), note that, you need to use a port number GREATER THAN 8000!

Install new python package inside Jupyter Notebook

Using conda (ref)
Using pip
Check version and update/upgrade,

Display dataframes side-by-side

Get previous outputs

Display 2 figures side-by-side markdown cell

Put below codes in the markdown cell of Jupyter Notebook.

Magic Functions

  • Check the full list (in examples) here or their docs here.
  • You can define your custom magic functions here.
Auto update the new updated modules (put at the beginning of the notebook)
Check more settings of %autoreload here.
Show the plots inside the notebook:
Get the commands from 1 to 4:
With the bash command line + and using also curl
πŸ‘‰ Note: REST API with cURL.
☝
We can run bash script inside a cell with ! pip install numpy.

Extensions

Table of contents

  1. Install npm and nodejs.
  1. Install this extension.
  1. Enable in jupyter lab view.
  1. Refresh the page.

Debugger

  1. Install xeus-python, jupyterlab
  1. Install this extension.
  1. Refresh the page, you have to choose kernel xpython (instead of Python 3) to use the debugger.

Convert notebook to HTML

β—†Installationβ—‹Jupyter notebookβ—‹Setting up a passwordβ—‹R with jupyter notebookβ—†Run JS codesβ—†Import local packageβ—‹Use the running kernel in VSCodeβ—†Environment variablesβ—†Other tipsβ—†Check the infoβ—†Multiline commandsβ—†Hotkeys / Shortcutsβ—†Jupyter notebook on remote serverβ—†Install new python package inside Jupyter Notebookβ—†Display dataframes side-by-sideβ—†Get previous outputsβ—†Display 2 figures side-by-side markdown cellβ—†Magic Functionsβ—†Extensionsβ—‹Table of contentsβ—‹Debuggerβ—†Convert notebook to HTML
About|My sketches |Cooking |Cafe icon Support Thi
πŸ’Œ [email protected]
1# BY PIP
2pip install --upgrade pip
3pip install --upgrade ipython jupyter
1# BY CONDA
2conda install ipython jupyter
1jupyter notebook --ip=127.0.0.1 --port=8080
2# or
3jupyter notebook --ip=127.0.0.1 --port=8080 --allow-root
1# create a juputer notebook config file
2# it can be used for other settings
3# <https://jupyter-notebook.readthedocs.io/en/stable/public_server.html#prerequisite-a-notebook-configuration-file>
4jupyter notebook --generate-config
5
6# create a new password
7# note: sha1 cannot be reverted!!
8jupyter notebook password
1from notebook.auth import passwd
2passwd()
1# create a sha1 password
2# download file create_sha1.py from <https://github.com/dinhanhthi/scripts>
3# run ./create_sha1.py
4
5# docker-compose.yml
6environment:
7  - PASSWD='sha1:d03968479249:319e92302e68d601392918f011d6c9334493023f'
8
9# Dockerfile
10CMD /bin/bash -c 'jupyter lab --no-browser --allow-root --ip=0.0.0.0 --NotebookApp.password="$PASSWD" "$@"'
1# install jupyter
2sudo apt-get install libzmq3-dev libcurl4-openssl-dev libssl-dev jupyter-core jupyter-client
3
4# install R on linux
5sudo apt install r-base
6
7# R kernel for Jupyter Notebook
8R # enter R environnement
9# install R kernel
10install.packages(c('repr', 'IRdisplay', 'IRkernel'), type = 'source')
11# or
12install.packages(c('repr', 'IRkernel'), type = 'source')
13# make jupyter see r kernel
14IRkernel::installspec() # current user
15IRkernel::installspec(user = FALSE) # global
1# embedded R
2# use by cell magic %%R
3pip install rpy2
4
5# in a notebook
6%load_ext rpy2.ipython
7
8# then use
9%%R
10# R's codes
1%%script node
2
3// js codes
4const { inspect } = require("util");
5console.log(inspect(result, { depth: null, colors: true }))
1- app/transcribe.py
2- app/constants.py
3- playground/notebok.ipynb
1import sys
2from pathlib import Path
3
4sys.path.append(str(Path.cwd().parent))
5
6from app.transcribe import get_youtube_video_metadata
7
8video_url = "https://www.youtube.com/watch?v=ry9SYnV3svc"
9video_metadata = get_youtube_video_metadata(video_url)
10print(video_metadata)
1http://localhost:8888/lab?token=85e5ed15a9d9024b358abe1b42e66e563502b66cd69b5d59
1from dotenv import load_dotenv
2import os
3load_dotenv()
4azure_api_key = os.getenv('AZURE_API_KEY')
1%%bash
2tar -czf archive.tar.gz foldername
1# function's info
2?<func-name>
1# function's shortcode
2??<func-name>
1# get the list of current variables
2whos
1!type python
1python is /Users/thi/anaconda/envs/python3.6/bin/python
1# Using '\\'
2df.columns = df.columns.str.replace('.', ' ')\\
3                       .str.replace('\\s+', ' ')\\
4                       .str.strip().str.upper()
1ssh -N -L localhost:9888:192.168.0.155:9899 <username-remote>@<remote-host> -p <port>
2# if there is no port, remove `-p <port>`
1# connect to remote
2ssh <username-remote>@<remote-host> -p <port>
3# if there is no port, remove `-p <port>`
1# run juputer with custom port
2jupyter notebook --no-browser --port=9899
3
4# if there is error `OSError: [Errno 99] Cannot assign requested address`
5jupyter notebook --ip=0.0.0.0 --no-browser --port=9899
6
7# if there is error `Running as root is not recommended`
8jupyter notebook --ip=0.0.0.0 --no-browser --port=9899 --alow-root
1http://127.0.0.1:9889/?token=717d9d276f0537a9...831793df6319ad389accd
1ssh -N -L localhost:9888:localhost:9889 <username-remote>@<remote-host> -p <port>
2# if there is no port, remove `-p <port>`
3# there is nothing but it's running
1<http://localhost:9888/?token=717d9d276f0537a9...831793df6319ad389accd>
1# Install a conda package in the current Jupyter kernel
2import sys
3!conda install --yes --prefix {sys.prefix} numpy
4
5# DON'T DO THIS
6!conda install --yes numpy
1# Install a pip package in the current Jupyter kernel
2import sys
3!{sys.executable} -m pip install numpy
4
5# DON'T DO THIS
6!pip install numpy
1!pip show pandas
1from IPython.display import display_html
2def display_side_by_side(*args):
3    html_str=''
4    for df in args:
5        html_str+=df.to_html()
6    display_html(html_str.replace('table','table style="display:inline; margin-right: 5px;"'),raw=True)
1display_side_by_side(df1,df2,df1)
1_ # previous output
2__ # second-to-last output
3___ # third-to-last output
1<tr>
2  <td> <img src="Nordic_trails.jpg" alt="Drawing" style="width: 250px;"/> </td>
3  <td> <img src="Nordic_trails.jpg" alt="Drawing" style="width: 250px;"/> </td>
4</tr>
1%load_ext autoreload
2%autoreload 2 # Reload all modules every time before executing
3
4%autoreload 0 # disable autoreloader
1%matplotlib inline
1%history -n 1-4 # get commands 1 to 4
1%%bash -s $APP_NAME
2
3APP_NAME=$1
4
5cat > ./predictor/instances.json <<END
6{
7   "instances": [
8     {
9       "data": {
10         "b64": "WW91IGFyZW4ndCBraW5kLCBpIGhhdGUgeW91Lg=="
11       }
12     }
13   ]
14}
15END
16
17curl -s -X POST \\
18  -H "Content-Type: application/json; charset=utf-8" \\
19  -d @./predictor/instances.json \\
20  <http://localhost:7080/predictions/$APP_NAME/>
1# errors
2# UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 23: ordinal not in range(128)
3npm config set unicode false
1pip install xeus-python
2pip install jupyterlab
1pip install jupyterlab
2jupyter nbconvert --to html <notebook>