Jupyter Notebook

Last modified 4 months ago / Edit on Github
Idea icon

If you use VSCode, you should use its Jupyter Notebook extension, it's quick, clean and very easy to use.

Installation

Jupyter notebook

# BY PIP
pip install --upgrade pip
pip install --upgrade ipython jupyter
# BY CONDA
conda install ipython jupyter

Or read more in this note.

If you meet error OSError: [Errno 99] Cannot assign requested address, try

jupyter notebook --ip=127.0.0.1 --port=8080
# or
jupyter notebook --ip=127.0.0.1 --port=8080 --allow-root

Setting up a password

# create a juputer notebook config file
# it can be used for other settings
# https://jupyter-notebook.readthedocs.io/en/stable/public_server.html#prerequisite-a-notebook-configuration-file
jupyter notebook --generate-config

# create a new password
# note: sha1 cannot be reverted!!
jupyter notebook password

Inside notebook:

from notebook.auth import passwd
passwd()

With docker

# create a sha1 password
# download file create_sha1.py from https://github.com/dinhanhthi/scripts
# run ./create_sha1.py

# docker-compose.yml
environment:
- PASSWD='sha1:d03968479249:319e92302e68d601392918f011d6c9334493023f'

# Dockerfile
CMD /bin/bash -c 'jupyter lab --no-browser --allow-root --ip=0.0.0.0 --NotebookApp.password="$PASSWD" "[email protected]"'

R with jupyter notebook

Read more here.

# install jupyter
sudo apt-get install libzmq3-dev libcurl4-openssl-dev libssl-dev jupyter-core jupyter-client

# install R on linux
sudo apt install r-base

# R kernel for Jupyter Notebook
R # enter R environnement
# install R kernel
install.packages(c('repr', 'IRdisplay', 'IRkernel'), type = 'source')
# or
install.packages(c('repr', 'IRkernel'), type = 'source')
# make jupyter see r kernel
IRkernel::installspec() # current user
IRkernel::installspec(user = FALSE) # global
# embedded R
# use by cell magic %%R
pip install rpy2

# in a notebook
%load_ext rpy2.ipython

# then use
%%R
# R's codes

Other tips

  • Running 2 tasks in the same cell TAKE LONGER TIME than running each on different cells.
  • Download a folder in jupyter notebook:
    • Inside notebook, use:

      %%bash
      tar -czf archive.tar.gz foldername
    • Or using nbzip (only working on current server).

Check the info

# function's info
?<func-name>
# function's shortcode
??<func-name>
# get the list of current variables
whos

Check where command executed from (in your $path)?

!type python
python is /Users/thi/anaconda/envs/python3.6/bin/python

Multiline commands

# Using '\'
df.columns = df.columns.str.replace('.', ' ')\
.str.replace('\s+', ' ')\
.str.strip().str.upper()

You CANNOT put # comments at the end of each line break!

Hotkeys / Shortcuts

There are 2 modes: command mode (pres ESC to activate) and edit mode (Enter to activate). Below are the most useful ones (for me).

You can edit / add more shortcuts in Help > Edit Keyboard Shortcuts.

Click to see the full list

For both modes,

  • Shift + Enter run the current cell, select below.
  • Ctrl + Enter run selected cells.
  • Alt + Enter run the current cell, insert below.
  • Ctrl + S save and checkpoint.

Command modes,

  • Enter take you into edit mode.
  • H show all shortcuts.
  • Up / Down select cell above / below.
  • Shift + Up / Down extend selected cells above / below.
  • A / B insert cell above / below.
  • X cut selected cells.
  • C copy selected cells.
  • V / Shift + V paste cells below / above.
  • D, D (press the key twice) delete selected cells.
  • Z undo cell deletion.
  • S Save and Checkpoint.
  • Y change the cell type to Code.
  • M change the cell type to Markdown.

Edit mode,

  • Esc take you into command mode.
  • Tab code completion or indent.
  • Ctrl + ] indent.
  • Ctrl + [ dedent.
  • Ctrl + A select all.
  • Ctrl + Z undo.
  • Ctrl + Shift + Z or Ctrl + Y redo.

Jupyter notebook on remote server

Open jupyter notebook in local browser but the backend-server is on remote.

  • If jupyter server is already running on remote at http://192.168.0.155:9889,

    ssh -N -L localhost:9888:192.168.0.155:9899 <username-remote>@<remote-host> -p <port>
    # if there is no port, remove `-p <port>`

    Open browser: http://localhost:9888 (type password if needed).

  • If jupyter server is not running on remote yet,

    # connect to remote
    ssh <username-remote>@<remote-host> -p <port>
    # if there is no port, remove `-p <port>`

    On remote,

    # run juputer with custom port
    jupyter notebook --no-browser --port=9899

    # if there is error `OSError: [Errno 99] Cannot assign requested address`
    jupyter notebook --ip=0.0.0.0 --no-browser --port=9899

    # if there is error `Running as root is not recommended`
    jupyter notebook --ip=0.0.0.0 --no-browser --port=9899 --alow-root

    It's running and there are somethings like that,

    http://127.0.0.1:9889/?token=717d9d276f0537a9...831793df6319ad389accd

    Open another terminal window and type,

    ssh -N -L localhost:9888:localhost:9889 <username-remote>@<remote-host> -p <port>
    # if there is no port, remove `-p <port>`
    # there is nothing but it's running

    Open browser:

    http://localhost:9888/?token=717d9d276f0537a9...831793df6319ad389accd

You can choose any port number you wanna instead of 9888 and 9889 (they can be the same), note that, you need to use a port number GREATER THAN 8000!

Install new python package inside Jupyter Notebook

Using conda[ref],

# Install a conda package in the current Jupyter kernel
import sys
!conda install --yes --prefix {sys.prefix} numpy

# DON'T DO THIS
!conda install --yes numpy

Using pip,

# Install a pip package in the current Jupyter kernel
import sys
!{sys.executable} -m pip install numpy

# DON'T DO THIS
!pip install numpy

Check version and update/upgrade,

!pip show pandas

Display dataframes side-by-side

from IPython.display import display_html
def display_side_by_side(*args):
html_str=''
for df in args:
html_str+=df.to_html()
display_html(html_str.replace('table','table style="display:inline; margin-right: 5px;"'),raw=True)
display_side_by_side(df1,df2,df1)

Get previous outputs

_ # previous output
__ # second-to-last output
___ # third-to-last output

Display 2 figures side-by-side markdown cell

Put below codes in the markdown cell of Jupyter Notebook.

<tr>
<td> <img src="Nordic_trails.jpg" alt="Drawing" style="width: 250px;"/> </td>
<td> <img src="Nordic_trails.jpg" alt="Drawing" style="width: 250px;"/> </td>
</tr>

Magic Functions

  • Check the full list (in examples) here or their docs here.
  • You can define your custom magic functions here.

Auto update the new updated modules (put at the beginning of the notebook)

%load_ext autoreload
%autoreload 2 # Reload all modules every time before executing

%autoreload 0 # disable autoreloader

Check more settings of %autoreload here.

Show the plots inside the notebook:

%matplotlib inline

Get the commands from 1 to 4:

%history -n 1-4 # get commands 1 to 4

With the bash command line + and using also curl,

👉 Note: curl.

%%bash -s $APP_NAME

APP_NAME=$1

cat > ./predictor/instances.json <<END
{
"instances": [
{
"data": {
"b64": "WW91IGFyZW4ndCBraW5kLCBpIGhhdGUgeW91Lg=="
}
}
]
}
END

curl -s -X POST \
-H "Content-Type: application/json; charset=utf-8" \
-d @./predictor/instances.json \
http://localhost:7080/predictions/$APP_NAME/
Explain $APP_NAME above

%%bash -s $APP_NAME will take the value of APP_NAME somewhere else and pass to $1.

APP_NAME=$1 will set the APP_NAME inside this block to the value of $1 which is set in previous step.

If you wanna use 2 variables?

%%bash -s $VAR_1 $VAR_2

VAR_1 = $1
VAR_2 = $2
Info icon

We can run bash script inside a cell with ! pip install numpy.

Extensions

Table of contents

  1. Install npm and nodejs.
  2. Install this extension.
  3. Enable in jupyter lab view.
  4. Refresh the page.
# errors
# UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 23: ordinal not in range(128)
npm config set unicode false

Debugger

  1. Install xeus-python, jupyterlab

    pip install xeus-python
    pip install jupyterlab
  2. Install this extension.

  3. Refresh the page, you have to choose kernel xpython (instead of Python 3) to use the debugger.

Convert notebook to HTML

pip install jupyterlab
jupyter nbconvert --to html <notebook>

💬 Comments

Support Thi Support Thi