{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Topics: Analytics Tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import ibis\n",
    "\n",
    "ibis.options.interactive = True\n",
    "\n",
    "connection = ibis.sqlite.connect(os.path.join('data', 'geography.db'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frequency tables\n",
    "\n",
    "Ibis provides the `value_counts` API, just like pandas, for computing a frequency table for a table column or array expression. You might have seen it used already earlier in the tutorial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries = connection.table('countries')\n",
    "countries.continent.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be customized, of course:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq = (countries.group_by(countries.continent)\n",
    "                 .aggregate([countries.count().name('# countries'),\n",
    "                             countries.population.sum().name('total population')]))\n",
    "freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binning and histograms\n",
    "\n",
    "\n",
    "Numeric array expressions (columns with numeric type and other array expressions) have `bucket` and `histogram` methods which produce different kinds of binning. These produce category values (the computed bins) that can be used in grouping and other analytics.\n",
    "\n",
    "Some backends implement the `.summary()` method, which can be used to see the general distribution of a column.\n",
    "\n",
    "Let's have a look at a few examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright then, now suppose we want to split the countries up into some buckets of our choosing for their population:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "buckets = [0, 1e6, 1e7, 1e8, 1e9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `bucket` function creates a bucketed category from the prices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucketed = countries.population.bucket(buckets).name('bucket')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the value counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucketed.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The buckets we wrote down define 4 buckets numbered 0 through 3. The `NaN` is a pandas `NULL` value (since that's how pandas represents nulls in numeric arrays), so don't worry too much about that. Since the bucketing ends at 100000, we see there are 4122 values that are over 100000. These can be included in the bucketing with `include_over`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucketed = (countries.population\n",
    "            .bucket(buckets, include_over=True)\n",
    "            .name('bucket'))\n",
    "bucketed.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `bucketed` object here is a special **_category_** type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucketed.type()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Category values can either have a known or unknown **_cardinality_**. In this case, there's either 4 or 5 buckets based on how we used the `bucket` function.\n",
    "\n",
    "Labels can be assigned to the buckets at any time using the `label` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket_counts = bucketed.value_counts()\n",
    "\n",
    "labeled_bucket = (bucket_counts.bucket\n",
    "                  .label(['< 1M', '> 1M', '> 10M', '> 100M', '> 1B'])\n",
    "                  .name('bucket_name'))\n",
    "\n",
    "expr = (bucket_counts[labeled_bucket, bucket_counts]\n",
    "        .sort_by('bucket'))\n",
    "expr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice, huh?\n",
    "\n",
    "Some backends implement `histogram(num_bins)`, a linear (fixed size bin) equivalent."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
