{ "cells": [ { "cell_type": "markdown", "id": "5da1b555-956f-4a4f-b414-411fc615a580", "metadata": {}, "source": [ "## How to take advantage of the computational cluster?" ] }, { "cell_type": "markdown", "id": "803d1298-297e-4e90-9dd3-8e1b40120d71", "metadata": {}, "source": [ "The cluster offers storage, cores, memory and convenience." ] }, { "cell_type": "markdown", "id": "7c384098-775e-456e-bf55-885d0370b9bd", "metadata": {}, "source": [ "### Using the Cores - Multiprocessing" ] }, { "cell_type": "markdown", "id": "55893669-5d7f-43e4-af48-3399483fbcd6", "metadata": {}, "source": [ "#### Prelude" ] }, { "cell_type": "markdown", "id": "b87c9985-8441-4b7c-bb32-8781c343e887", "metadata": {}, "source": [ "Before going into the multiprocessing I would like to look a little bit into a few tips for efficient programming." ] }, { "cell_type": "markdown", "id": "3c3cb5db-4e12-403e-a18a-77f5a7c1eb1a", "metadata": {}, "source": [ "So let's assume that we want to make a calculation for each row in a dataset:" ] }, { "cell_type": "code", "execution_count": 152, "id": "c7b6ad3e-ca8b-4a9e-ad92-790ce61f518e", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np" ] }, { "cell_type": "code", "execution_count": 153, "id": "697afa2f-5a26-4063-a094-85a68dda7cc8", "metadata": {}, "outputs": [], "source": [ "df0 = pd.read_csv(\"files/infl_unemp.csv\").dropna().reset_index(drop=True)\n", "df = df0.dropna()" ] }, { "cell_type": "code", "execution_count": 154, "id": "520ea925-75b1-4cf5-8ba0-af118598c91e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0countrydateyearmonthhicpunempfirst_yearinflationlag_inflationweights
048IE2022M0220222107.104.81996-2.20-1.000.235249
164AT2022M0120221113.934.81996-1.47-2.600.343967
265BE2022M0120221116.995.61996-2.50-0.790.269843
366BG2022M0120221114.804.52000-1.36-2.430.593467
467CY2022M0120221103.336.52000-0.89-1.910.734424
....................................
1005210122NO1996M011996169.204.81996-0.10-0.200.358546
1005310123PT1996M011996165.768.01996-0.33-0.140.110814
1005410124SE1996M011996175.418.71996-0.19-0.540.820810
1005510125SI1996M011996143.887.21996-0.55-0.640.947574
1005610126UK1996M011996167.808.21996-0.30-0.300.300982
\n", "

10057 rows × 11 columns

\n", "
" ], "text/plain": [ " Unnamed: 0 country date year month hicp unemp first_year \\\n", "0 48 IE 2022M02 2022 2 107.10 4.8 1996 \n", "1 64 AT 2022M01 2022 1 113.93 4.8 1996 \n", "2 65 BE 2022M01 2022 1 116.99 5.6 1996 \n", "3 66 BG 2022M01 2022 1 114.80 4.5 2000 \n", "4 67 CY 2022M01 2022 1 103.33 6.5 2000 \n", "... ... ... ... ... ... ... ... ... \n", "10052 10122 NO 1996M01 1996 1 69.20 4.8 1996 \n", "10053 10123 PT 1996M01 1996 1 65.76 8.0 1996 \n", "10054 10124 SE 1996M01 1996 1 75.41 8.7 1996 \n", "10055 10125 SI 1996M01 1996 1 43.88 7.2 1996 \n", "10056 10126 UK 1996M01 1996 1 67.80 8.2 1996 \n", "\n", " inflation lag_inflation weights \n", "0 -2.20 -1.00 0.235249 \n", "1 -1.47 -2.60 0.343967 \n", "2 -2.50 -0.79 0.269843 \n", "3 -1.36 -2.43 0.593467 \n", "4 -0.89 -1.91 0.734424 \n", "... ... ... ... \n", "10052 -0.10 -0.20 0.358546 \n", "10053 -0.33 -0.14 0.110814 \n", "10054 -0.19 -0.54 0.820810 \n", "10055 -0.55 -0.64 0.947574 \n", "10056 -0.30 -0.30 0.300982 \n", "\n", "[10057 rows x 11 columns]" ] }, "execution_count": 154, "metadata": {}, "output_type": "execute_result" } ], "source": [ "countries = ['BE', 'BG', 'CY', 'CZ', 'DE', 'DK', 'EA19', 'EE', 'EL', 'ES',\n", " 'EU27_2020', 'FI', 'FR', 'HR', 'HU', 'IE', 'IS', 'IT', 'LT', 'LU',\n", " 'LV', 'MT', 'NL', 'PT', 'RO', 'SE', 'SI', 'SK', 'AT', 'NO', 'PL',\n", " 'CH', 'TR', 'US', 'UK']\n", "weights = np.random.rand(len(countries))\n", "weight_dict = dict(zip(countries,weights/sum(weights)))\n", "weights_frame = pd.DataFrame(weights,columns=[\"weights\"])\n", "weights_frame[\"country\"] = countries\n", "df = pd.merge(df, weights_frame, on=\"country\", how=\"left\")\n", "df" ] }, { "cell_type": "code", "execution_count": 155, "id": "5f87ddeb-49c8-4b48-9811-d5e4b81fd997", "metadata": {}, "outputs": [], "source": [ "# I want to perform the following operation on each row:\n", "def operation(row):\n", " return row[\"weights\"]*(1.5*row[\"inflation\"]+0.3*row[\"lag_inflation\"]+0.2*row[\"unemp\"])" ] }, { "cell_type": "code", "execution_count": 156, "id": "81016ffc-a7c3-48fb-9608-a0197cc57316", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The line_profiler extension is already loaded. To reload it, use:\n", " %reload_ext line_profiler\n" ] } ], "source": [ "# There is a lot of different ways to do this and I want to time my results for each option\n", "from qgrid import show_grid\n", "# this only works on jupyter hub, magic commands\n", "%load_ext line_profiler " ] }, { "cell_type": "code", "execution_count": 157, "id": "23b57653-81b6-498b-836f-1188bd4409ea", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0countrydateyearmonthhicpunempfirst_yearinflationlag_inflationweights
048IE2022M0220222107.104.81996-2.20-1.000.235249
164AT2022M0120221113.934.81996-1.47-2.600.343967
265BE2022M0120221116.995.61996-2.50-0.790.269843
366BG2022M0120221114.804.52000-1.36-2.430.593467
467CY2022M0120221103.336.52000-0.89-1.910.734424
....................................
1005210122NO1996M011996169.204.81996-0.10-0.200.358546
1005310123PT1996M011996165.768.01996-0.33-0.140.110814
1005410124SE1996M011996175.418.71996-0.19-0.540.820810
1005510125SI1996M011996143.887.21996-0.55-0.640.947574
1005610126UK1996M011996167.808.21996-0.30-0.300.300982
\n", "

10057 rows × 11 columns

\n", "
" ], "text/plain": [ " Unnamed: 0 country date year month hicp unemp first_year \\\n", "0 48 IE 2022M02 2022 2 107.10 4.8 1996 \n", "1 64 AT 2022M01 2022 1 113.93 4.8 1996 \n", "2 65 BE 2022M01 2022 1 116.99 5.6 1996 \n", "3 66 BG 2022M01 2022 1 114.80 4.5 2000 \n", "4 67 CY 2022M01 2022 1 103.33 6.5 2000 \n", "... ... ... ... ... ... ... ... ... \n", "10052 10122 NO 1996M01 1996 1 69.20 4.8 1996 \n", "10053 10123 PT 1996M01 1996 1 65.76 8.0 1996 \n", "10054 10124 SE 1996M01 1996 1 75.41 8.7 1996 \n", "10055 10125 SI 1996M01 1996 1 43.88 7.2 1996 \n", "10056 10126 UK 1996M01 1996 1 67.80 8.2 1996 \n", "\n", " inflation lag_inflation weights \n", "0 -2.20 -1.00 0.235249 \n", "1 -1.47 -2.60 0.343967 \n", "2 -2.50 -0.79 0.269843 \n", "3 -1.36 -2.43 0.593467 \n", "4 -0.89 -1.91 0.734424 \n", "... ... ... ... \n", "10052 -0.10 -0.20 0.358546 \n", "10053 -0.33 -0.14 0.110814 \n", "10054 -0.19 -0.54 0.820810 \n", "10055 -0.55 -0.64 0.947574 \n", "10056 -0.30 -0.30 0.300982 \n", "\n", "[10057 rows x 11 columns]" ] }, "execution_count": 157, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "code", "execution_count": 158, "id": "8af38289-48f1-4934-b3fd-7d7843d7e7aa", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1.39 s ± 14.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" ] } ], "source": [ "%%timeit\n", "# crude looping\n", "res = []\n", "for i in range(len(df)):\n", " res.append(operation(df.iloc[i,:]))\n", "df[\"int\"]=res" ] }, { "cell_type": "code", "execution_count": 159, "id": "32b8a269-4e23-43fd-ba39-c2d62cc476b1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "677 ms ± 7.35 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" ] } ], "source": [ "%%timeit\n", "# loop with iterrows\n", "res = []\n", "for _, row in df.iterrows():\n", " res.append(operation(row))\n", "df[\"int\"]=res" ] }, { "cell_type": "code", "execution_count": 160, "id": "0fbdc0f8-ad63-49b9-b8f3-fc87dbc187f5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "195 ms ± 2.28 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n" ] } ], "source": [ "%%timeit\n", "# loop using apply\n", "df.apply(operation, axis=1)" ] }, { "cell_type": "code", "execution_count": 161, "id": "653f8302-aa7a-4801-9414-10cd8262525e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Timer unit: 1e-06 s\n", "\n", "Total time: 0.436895 s\n", "File: /tmp/ipykernel_796523/3911977210.py\n", "Function: operation at line 2\n", "\n", "Line # Hits Time Per Hit % Time Line Contents\n", "==============================================================\n", " 2 def operation(row):\n", " 3 10057 436895.0 43.4 100.0 return row[\"weights\"]*(1.5*row[\"inflation\"]+0.3*row[\"lag_inflation\"]+0.2*row[\"unemp\"])" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# analyze the process \n", "%lprun -f operation df.apply(operation, axis=1) # this is still going through the loop" ] }, { "cell_type": "code", "execution_count": 162, "id": "f10ccd52-0fcc-412f-9ebd-33aec356c7e2", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0countrydateyearmonthhicpunempfirst_yearinflationlag_inflationweightsint
048IE2022M0220222107.104.81996-2.20-1.000.235249-0.621058
164AT2022M0120221113.934.81996-1.47-2.600.343967-0.696534
265BE2022M0120221116.995.61996-2.50-0.790.269843-0.773641
366BG2022M0120221114.804.52000-1.36-2.430.593467-1.109191
467CY2022M0120221103.336.52000-0.89-1.910.734424-0.446530
.......................................
1005210122NO1996M011996169.204.81996-0.10-0.200.3585460.268909
1005310123PT1996M011996165.768.01996-0.33-0.140.1108140.117795
1005410124SE1996M011996175.418.71996-0.19-0.540.8208101.061307
1005510125SI1996M011996143.887.21996-0.55-0.640.9475740.400824
1005610126UK1996M011996167.808.21996-0.30-0.300.3009820.331080
\n", "

10057 rows × 12 columns

\n", "
" ], "text/plain": [ " Unnamed: 0 country date year month hicp unemp first_year \\\n", "0 48 IE 2022M02 2022 2 107.10 4.8 1996 \n", "1 64 AT 2022M01 2022 1 113.93 4.8 1996 \n", "2 65 BE 2022M01 2022 1 116.99 5.6 1996 \n", "3 66 BG 2022M01 2022 1 114.80 4.5 2000 \n", "4 67 CY 2022M01 2022 1 103.33 6.5 2000 \n", "... ... ... ... ... ... ... ... ... \n", "10052 10122 NO 1996M01 1996 1 69.20 4.8 1996 \n", "10053 10123 PT 1996M01 1996 1 65.76 8.0 1996 \n", "10054 10124 SE 1996M01 1996 1 75.41 8.7 1996 \n", "10055 10125 SI 1996M01 1996 1 43.88 7.2 1996 \n", "10056 10126 UK 1996M01 1996 1 67.80 8.2 1996 \n", "\n", " inflation lag_inflation weights int \n", "0 -2.20 -1.00 0.235249 -0.621058 \n", "1 -1.47 -2.60 0.343967 -0.696534 \n", "2 -2.50 -0.79 0.269843 -0.773641 \n", "3 -1.36 -2.43 0.593467 -1.109191 \n", "4 -0.89 -1.91 0.734424 -0.446530 \n", "... ... ... ... ... \n", "10052 -0.10 -0.20 0.358546 0.268909 \n", "10053 -0.33 -0.14 0.110814 0.117795 \n", "10054 -0.19 -0.54 0.820810 1.061307 \n", "10055 -0.55 -0.64 0.947574 0.400824 \n", "10056 -0.30 -0.30 0.300982 0.331080 \n", "\n", "[10057 rows x 12 columns]" ] }, "execution_count": 162, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "code", "execution_count": 163, "id": "f6226635-1595-4036-a9c7-8b169149b11c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "584 µs ± 14.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n" ] } ], "source": [ "%%timeit\n", "# use vectorised pandas processes\n", "df[\"int\"] = operation(df)" ] }, { "cell_type": "code", "execution_count": 164, "id": "719be959-fce5-42cf-9ce5-0a69a23a27f2", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Timer unit: 1e-06 s\n", "\n", "Total time: 0.0019 s\n", "File: /tmp/ipykernel_796523/3911977210.py\n", "Function: operation at line 2\n", "\n", "Line # Hits Time Per Hit % Time Line Contents\n", "==============================================================\n", " 2 def operation(row):\n", " 3 1 1900.0 1900.0 100.0 return row[\"weights\"]*(1.5*row[\"inflation\"]+0.3*row[\"lag_inflation\"]+0.2*row[\"unemp\"])" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%lprun -f operation operation(df) # this is only running once" ] }, { "cell_type": "code", "execution_count": 165, "id": "c956a3cf-7f2e-4a37-91b3-b939a99622e6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "584 µs ± 13.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n" ] } ], "source": [ "%%timeit\n", "# another way to do the same thing\n", "df[\"int\"] = df[\"weights\"]*(1.5*df[\"inflation\"]+0.3*df[\"lag_inflation\"]+0.2*df[\"unemp\"])" ] }, { "cell_type": "code", "execution_count": 166, "id": "bbde0479-3ee7-49b6-8654-ce2b13b94f99", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "162 µs ± 1.51 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n" ] } ], "source": [ "%%timeit\n", "# and if we need a lot of speed we go to numpy arrays\n", "df[\"int\"] = df[\"weights\"].values*(1.5*df[\"inflation\"].values+0.3*df[\"lag_inflation\"].values+0.2*df[\"unemp\"].values)" ] }, { "cell_type": "code", "execution_count": 167, "id": "6e33fba2-1f50-437c-8713-209bed5b8ef8", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0countrydateyearmonthhicpunempfirst_yearinflationlag_inflationweightsint
048IE2022M0220222107.104.81996-2.20-1.000.235249-0.621058
164AT2022M0120221113.934.81996-1.47-2.600.343967-0.696534
265BE2022M0120221116.995.61996-2.50-0.790.269843-0.773641
366BG2022M0120221114.804.52000-1.36-2.430.593467-1.109191
467CY2022M0120221103.336.52000-0.89-1.910.734424-0.446530
.......................................
1005210122NO1996M011996169.204.81996-0.10-0.200.3585460.268909
1005310123PT1996M011996165.768.01996-0.33-0.140.1108140.117795
1005410124SE1996M011996175.418.71996-0.19-0.540.8208101.061307
1005510125SI1996M011996143.887.21996-0.55-0.640.9475740.400824
1005610126UK1996M011996167.808.21996-0.30-0.300.3009820.331080
\n", "

10057 rows × 12 columns

\n", "
" ], "text/plain": [ " Unnamed: 0 country date year month hicp unemp first_year \\\n", "0 48 IE 2022M02 2022 2 107.10 4.8 1996 \n", "1 64 AT 2022M01 2022 1 113.93 4.8 1996 \n", "2 65 BE 2022M01 2022 1 116.99 5.6 1996 \n", "3 66 BG 2022M01 2022 1 114.80 4.5 2000 \n", "4 67 CY 2022M01 2022 1 103.33 6.5 2000 \n", "... ... ... ... ... ... ... ... ... \n", "10052 10122 NO 1996M01 1996 1 69.20 4.8 1996 \n", "10053 10123 PT 1996M01 1996 1 65.76 8.0 1996 \n", "10054 10124 SE 1996M01 1996 1 75.41 8.7 1996 \n", "10055 10125 SI 1996M01 1996 1 43.88 7.2 1996 \n", "10056 10126 UK 1996M01 1996 1 67.80 8.2 1996 \n", "\n", " inflation lag_inflation weights int \n", "0 -2.20 -1.00 0.235249 -0.621058 \n", "1 -1.47 -2.60 0.343967 -0.696534 \n", "2 -2.50 -0.79 0.269843 -0.773641 \n", "3 -1.36 -2.43 0.593467 -1.109191 \n", "4 -0.89 -1.91 0.734424 -0.446530 \n", "... ... ... ... ... \n", "10052 -0.10 -0.20 0.358546 0.268909 \n", "10053 -0.33 -0.14 0.110814 0.117795 \n", "10054 -0.19 -0.54 0.820810 1.061307 \n", "10055 -0.55 -0.64 0.947574 0.400824 \n", "10056 -0.30 -0.30 0.300982 0.331080 \n", "\n", "[10057 rows x 12 columns]" ] }, "execution_count": 167, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "markdown", "id": "3cf99000-d0ce-4a2e-b4dd-ba36db37d870", "metadata": {}, "source": [ "#### Now actual multiprocessing" ] }, { "cell_type": "markdown", "id": "7675c928-cd61-451f-8e74-c5c1efd2d6c0", "metadata": {}, "source": [ "The standard library used in python is multiprocessing. However, I will jump directly to something that I find more convenient, namely concurrent.futures." ] }, { "cell_type": "code", "execution_count": 169, "id": "fdbb9358-36af-4df5-a918-d2825e71a9d8", "metadata": {}, "outputs": [], "source": [ "# import multiprocessing\n", "import concurrent.futures" ] }, { "cell_type": "code", "execution_count": 170, "id": "33f37960-23fc-4bd7-9002-b50cd31e5dd7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1e+03 ms ± 96.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" ] } ], "source": [ "%%timeit\n", "chunk_size = 1000\n", "with concurrent.futures.ProcessPoolExecutor() as executor:\n", " fs = [executor.submit(operation, df.iloc[x:x+chunk_size,:]) for x in range(0,df.shape[0],chunk_size) ]\n", " results = [f.result() for f in fs]\n", "\n", "df[\"int\"]=pd.concat(results)" ] }, { "cell_type": "code", "execution_count": null, "id": "c736a930-e592-457d-8e39-de7521726eb6", "metadata": {}, "outputs": [], "source": [ "df" ] }, { "cell_type": "markdown", "id": "87f73c83-eb66-475a-96a2-caf2ae5af47e", "metadata": {}, "source": [ "* This library is also handy because it unifies the syntax for multiprocessing and multithreading. If you are not constrained by multiprocessing but by input/output operations then multithreading may be superior." ] }, { "cell_type": "markdown", "id": "30552748-f617-4e54-849e-2eccb4b79508", "metadata": {}, "source": [ "So, it does not really make sense to introduce parallel computation for relatively simple tasks. It costs in coding and the computational overveads can actually make it slower." ] }, { "cell_type": "markdown", "id": "273f4127-344e-4176-9b4c-6379f1793865", "metadata": {}, "source": [ "#### Let's look at a case when it could help." ] }, { "cell_type": "code", "execution_count": null, "id": "a65e9a76-06cb-4221-882e-83fb94e8d7c2", "metadata": {}, "outputs": [], "source": [ "# def reg_unemp_on_lag_inf(frame):\n", "# \"This gives the slop coefficient of a regression of unemployment on lag_inflation\"\n", "# # print(\"bob\")\n", "# # # print(np.dot(frame[\"unemp\"], frame[\"lag_inflation\"]))\n", "# # print(np.dot(frame[\"unemp\"], frame[\"lag_inflation\"])/np.dot(frame[\"lag_inflation\"], frame[\"lag_inflation\"]))\n", "# # assert False\n", "# return np.dot(frame[\"unemp\"], frame[\"lag_inflation\"].T)/np.dot(frame[\"lag_inflation\"], frame[\"lag_inflation\"])" ] }, { "cell_type": "code", "execution_count": 171, "id": "e281d4b8-58ca-41e7-9c41-c05288100731", "metadata": {}, "outputs": [], "source": [ "import statsmodels.api as sm\n", "\n", "def get_reg_results(frame):\n", " temp = frame.copy(deep=False)\n", " temp[\"const\"] = 1\n", " model = sm.OLS(temp[\"unemp\"], temp[[\"const\",\"inflation\",\"lag_inflation\"]])\n", " results = model.fit()\n", " for _ in range(20):\n", " results = model.fit()\n", " return results.params\n", "\n", "\n", "def get_reg_results2(frame):\n", " results = [[],[],[]]\n", " for _ in range(100):\n", " temp = frame.sample(frac=2/3) \n", " # temp = temp.sample(int(df.shape[0]*2/3))\n", " temp[\"const\"] = 1\n", " # print(frame)\n", " # print(\"lala\")\n", " # print(temp)\n", " model = sm.OLS(temp[\"unemp\"], temp[[\"const\",\"inflation\",\"lag_inflation\"]])\n", " result = model.fit()\n", " results[0].append(result.params[0])\n", " results[1].append(result.params[1])\n", " # results[2].append(result.params[2])\n", " \n", " return pd.Series([(sum(series)/len(series) if len(series) != 0 else None) for series in results] ,index=[\"const\",\"inflation\",\"lag_inflation\"])" ] }, { "cell_type": "code", "execution_count": 172, "id": "c4bed68a-d86b-4a07-b056-79a6bdc6fa53", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "const 8.407389\n", "inflation 0.115055\n", "lag_inflation NaN\n", "dtype: float64" ] }, "execution_count": 172, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_reg_results2(df)# df" ] }, { "cell_type": "code", "execution_count": null, "id": "f06cc516-4e4e-441a-81e3-f054b027afab", "metadata": {}, "outputs": [], "source": [ "reg_unemp_on_lag_inf(df)" ] }, { "cell_type": "code", "execution_count": null, "id": "39ddc737-8c09-4030-8d94-23afe9312072", "metadata": {}, "outputs": [], "source": [ "df.shape[0]" ] }, { "cell_type": "code", "execution_count": 175, "id": "04f318ff-1511-48e0-9dc0-97df8c16e888", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1.33 s ± 63.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" ] } ], "source": [ "%%timeit\n", "chunk_size = 1000\n", "with concurrent.futures.ProcessPoolExecutor() as executor:\n", " countries = df.loc[:,\"country\"].unique()\n", " fs = [executor.submit(get_reg_results2, df.loc[df.loc[:,\"country\"]==country]) for country in countries]\n", " results = np.column_stack((countries,np.array([f.result() for f in fs])))\n", "temp = pd.DataFrame(results, columns=[\"country\", \"c_const\",\"c_infl\",\"c_lag_infl\"])\n", "df_temp = pd.merge(df, temp, on=\"country\", how=\"left\")" ] }, { "cell_type": "code", "execution_count": null, "id": "c53ead91-e275-44a3-8535-50d624e6805c", "metadata": {}, "outputs": [], "source": [ "# %%timeit\n", "# chunk_size = 1000\n", "# with concurrent.futures.ProcessPoolExecutor() as executor:\n", "# countries = df.loc[:,\"country\"].unique()\n", "# fs = [executor.submit(reg_unemp_on_lag_inf, df.loc[df.loc[:,\"country\"]==country]) for country in df.loc[:,\"country\"].unique()]\n", "# results = np.column_stack((countries,np.array([f.result() for f in fs])))\n", "# temp = pd.DataFrame(results, columns=[\"country\", \"beta\"])\n", "# df_temp = pd.merge(df, temp, on=\"country\", how=\"left\")" ] }, { "cell_type": "code", "execution_count": null, "id": "355f23b2-d3bd-4ef7-bfb8-5a4b22d03d9e", "metadata": {}, "outputs": [], "source": [ "df" ] }, { "cell_type": "markdown", "id": "e69f79ed-ac4e-4723-9a71-308e037a6876", "metadata": {}, "source": [ "#### What about without multiprocessing?" ] }, { "cell_type": "code", "execution_count": null, "id": "eec59f8b-b191-4471-bae6-7d14719c57bf", "metadata": {}, "outputs": [], "source": [ "# %%timeit\n", "# df_temp = pd.merge(df, df.groupby(\"country\").apply(reg_unemp_on_lag_inf).rename(\"beta\").to_frame(), on=\"country\",how=\"left\") " ] }, { "cell_type": "code", "execution_count": 177, "id": "c669bfb1-59dd-4e75-b29f-4274afee71f6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "58.4 ms ± 1.19 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n" ] } ], "source": [ "%%timeit\n", "df_temp = pd.merge(df, df.groupby(\"country\").apply(get_reg_results).rename(columns={\"const\":\"c_const\",\"inflation\":\"c_infl\",\"lag_inflation\":\"c_lag_infl\"}), on=\"country\",how=\"left\") \n", "df_temp" ] }, { "cell_type": "code", "execution_count": null, "id": "c6b887ac-438d-419f-85b0-89d38671ea9e", "metadata": {}, "outputs": [], "source": [ "df" ] }, { "cell_type": "markdown", "id": "5b8cd6c6-842a-4138-a9d5-9af3d0d6cb13", "metadata": {}, "source": [ "### The Computational Cluster also offers a lot of memory!" ] }, { "cell_type": "markdown", "id": "c4535e50-5ac6-4ee3-b5a0-705cb808d3fd", "metadata": {}, "source": [ "There is not much that I can actaully show you here as the title actually says it all." ] }, { "cell_type": "markdown", "id": "b19a1858-7e0c-4a58-be94-efe4789e6147", "metadata": {}, "source": [ "However, even if the cluster offers a lot of memory it would also be good to know some practises in order to economise on the size of the data in memory. So, let me load a relatively large dataset:" ] }, { "cell_type": "code", "execution_count": null, "id": "6b1c487f-442e-480c-a094-a123e80b6228", "metadata": {}, "outputs": [], "source": [ "# how to find the files that I am interested in:\n", "# cd /SAFE_NAS" ] }, { "cell_type": "markdown", "id": "38ccf13c-b9cb-48c4-8cfc-caa077556407", "metadata": {}, "source": [ "- You can use option iterator and/or chunksize in pandas.read_csv\n", "- You can change the data types of your variables in order to save space." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 5 }