src/Notebook/01_Scrape.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extracción de datos a través de *Beautiful Soap 4*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.- Obtención de datos general de los datos.\n",
    "\n",
    "En un primer paso vamos a hacer llamadas a través de la librería de BeautifulSoap4 para obtener datos generales desde la primera pagina."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Como siempre lo primero es importar las librerías "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup as bs\n",
    "import requests\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'d:\\\\Data_science\\\\Javier\\\\Repositorios\\\\Proyecto_tienda_online\\\\src'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.chdir(os.path.split(os.getcwd())[0])\n",
    "folder=os.getcwd()\n",
    "folder"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conectamos la pagina web para acceder a ella y *arañar* los datos de su página web."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://www.amantis.net/productos-amantis/\"              # lista productos\n",
    "url_principal=\"https://www.amantis.net/\"                        # productos\n",
    "response = requests.get(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "html = response.content\n",
    "soup = bs(html, \"lxml\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a.- Vamos a obtener información de los datos existente en la URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<h3 class=\"t2sDiv-titulo hidden text-left color-corporativo\">Top ventas en amantis</h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/tobogane-hot-rabbit-el-superventas-amantis-mejorado/\">\n",
       " <span>TOBOGANE HOT RABBIT, el superventas de amantis ¡mejorado!</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/ballenato-tu-vibrador-distancia-aleta-movil-sumergible/\">\n",
       " <span>BALLENATO, tu vibrador a distancia con aleta móvil y sumergible...</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/tobogane-el-vibrador-doble-mas-vendido-ahora-efecto-hot/\">\n",
       " <span>TOBOGANE, el vibrador doble más vendido</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/meneo-sube-baja-realista-control-remoto/\">\n",
       " <span>MENEO sube y baja, placer realista con control remoto</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/fresh-girl-6-kilos-40cm-piel-real-disfrutable/\">\n",
       " <span>FRESH GIRL, 6 Kilos y 40cm de piel real disfrutable</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/lizo-2-dildo-suave-silicona-3-tamanos/\">\n",
       " <span>LIZO 2, Dildo de suave silicona en 3 tamaños</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/reggia-masturbador-masculino-doble/\">\n",
       " <span>REGGIA, masturbador masculino doble</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/pro-anal-2-vibrador-anal/\">\n",
       " <span>Pro ANAL, vibrador anal progresivo</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/tandem-2-flex-vibrador-doble-flexible-mando/\">\n",
       " <span>TANDEM 2 flex, vibrador doble flexible con mando</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/foxtail-plug-anal-cola-zorro/\">\n",
       " <span>FOXTAIL, plug anal cola de zorro de 35cm</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/bisou-besos-ondas-succion-la-mejor-vibracion/\">\n",
       " <span>BISOU, besos por ondas de succión con la mejor vibración</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/tok2-bala-vibradora-mando-mas-potente-recargable/\">\n",
       " <span>TOK2 bala vibradora con mando, más potente y recargable</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/sazzia-vagina-hiperrealista-amantis/\">\n",
       " <span>SAZZIA, masturbador hiperrealista</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/cristalino-xl-gran-dildo-transparente-22cm/\">\n",
       " <span>CRISTALINO XL, gran dildo transparente de 22cm</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/tris-anilla-vibrador-doble-penetracion-trinity/\">\n",
       " <span>TRIS-TRAS, anilla con vibrador para doble penetración TRINITY...</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/desliz-vibragel-liquido-vibrador-30ml/\">\n",
       " <span>Vibrador Líquido con sabor Desliz! VIBRAGEL 30ml, hormigueo...</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/kit-3-plugs-anales-diamante-black-star-plugress/\">\n",
       " <span>Kit de 3 Plugs anales con diamante BLACK STAR Plugress</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/idyllic-boy-grosso-plug-anal-distancia-mas-intenso/\">\n",
       " <span>IDYLLIC BOY, Plug anal a distancia más intenso</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/cuore-trio-kit-3-plugs-corazon/\">\n",
       " <span>CUORE TRÍO, Kit de 3 plugs con corazón</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/atame-facil-kit-cama-universal-amantis/\">\n",
       " <span>Átame-Fácil, kit de cama universal de amantis</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/yola-suave-texturizada-preciosista-masturbacion-definitiva/\">\n",
       " <span>YOLA. Suave, Texturizada y Preciosista, masturbación definitiva...</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/vulvanic-vibrador-distancia-ergonomico-entraras-erupcion/\">\n",
       " <span>VULVANIC, Vibrador a distancia ergonómico. Entrarás en erupción...</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/passion-pump-bomba-succion-automatica-amantis/\">\n",
       " <span>PASSION PUMP, bomba de succión automática de amantis</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/pulso-air-vibrador-palpitador-succionador/\">\n",
       " <span>PULSO AIR - Vibrador palpitador con succionador</span>\n",
       " </a>\n",
       " </h3>]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_h3 = soup.find_all(\"h3\")\n",
    "all_h3"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b.- Extraemos los nombres desde esta URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24\n",
      "['TOBOGANE HOT RABBIT', 'BALLENATO', 'TOBOGANE', 'MENEO sube y baja', 'FRESH GIRL', 'LIZO 2', 'REGGIA', 'Pro ANAL', 'TANDEM 2 flex', 'FOXTAIL', 'BISOU', 'TOK2 bala vibradora con mando', 'SAZZIA', 'CRISTALINO XL', 'TRIS-TRAS', 'Vibrador Líquido con sabor Desliz! VIBRAGEL 30ml', 'Kit de 3 Plugs anales con diamante BLACK STAR Plugress', 'IDYLLIC BOY', 'CUORE TRÍO', 'Átame-Fácil', 'YOLA. Suave', 'VULVANIC', 'PASSION PUMP', 'PULSO AIR - Vibrador palpitador con succionador']\n"
     ]
    }
   ],
   "source": [
    "titulos=soup.find_all(\"h3\")\n",
    "name=[]\n",
    "for titulo in titulos[1:]:\n",
    "    nombre=titulo.get_text(strip=True).split(',')[0]\n",
    "    name.append(nombre)\n",
    "print(len(name))\n",
    "print(name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### c.- Con este codigo extraigo la descripción del producto."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24\n",
      "[['el superventas de amantis ¡mejorado!'], ['tu vibrador a distancia con aleta móvil y sumergible...'], ['el vibrador doble más vendido'], ['placer realista con control remoto'], ['6 Kilos y 40cm de piel real disfrutable'], ['Dildo de suave silicona en 3 tamaños'], ['masturbador masculino doble'], ['vibrador anal progresivo'], ['vibrador doble flexible con mando'], ['plug anal cola de zorro de 35cm'], ['besos por ondas de succión con la mejor vibración'], ['más potente y recargable'], ['masturbador hiperrealista'], ['gran dildo transparente de 22cm'], ['anilla con vibrador para doble penetración TRINITY...'], ['hormigueo...'], ['No hay datos'], ['Plug anal a distancia más intenso'], ['Kit de 3 plugs con corazón'], ['kit de cama universal de amantis'], ['Texturizada y Preciosista', 'masturbación definitiva...'], ['Vibrador a distancia ergonómico. Entrarás en erupción...'], ['bomba de succión automática de amantis'], ['No hay datos']]\n"
     ]
    }
   ],
   "source": [
    "titulos=soup.find_all(\"h3\")\n",
    "desc=[]\n",
    "\n",
    "for titulo in titulos[1:]:\n",
    "    description=titulo.get_text(strip=True).split(', ')[1:]\n",
    "    if description==[]:\n",
    "        description=[\"No hay datos\"]\n",
    "    desc.append(description)\n",
    "print(len(desc))\n",
    "print(desc)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vemos que en algunos casos no hay información. \n",
    "\n",
    "En otros casos la separación de la descripción, no se ha separado.\n",
    "\n",
    "Por este motivo, haremos la extracción unicamente de todo y posteriormente lo trataremos con la librería de *pandas*."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### d.- Con esto extraigo los links de los productos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "productos = soup.find_all(class_='caption')\n",
    "lista_URLs = []\n",
    "for producto in productos[8:]:\n",
    "    URL_producto = producto.find('a')['href']\n",
    "    lista_URLs.append(URL_producto)\n",
    "\n",
    "len(lista_URLs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lista_URLs.pop(0)\n",
    "len(lista_URLs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.amantis.net/tobogane-hot-rabbit-el-superventas-amantis-mejorado/'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lista_URLs[0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Esta información es importante ya que nos permitirá obtener las direcciones de cada producto para hacer posteriormente la información desde cada producto."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### e.- Obtenemos la información de precio rebajado de cada producto."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_price=soup.find_all(\"span\",class_=\"productSpecialPrice\")\n",
    "price=[]\n",
    "\n",
    "for precio in all_price:\n",
    "    item_price=precio.get_text(strip=True).replace(\",\", \".\").split('€')[0]\n",
    "    price.append(item_price)\n",
    "len(price)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Este es el codigo completo para extraer la información de cada página."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Description</th>\n",
       "      <th>Price</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>TOBOGANE HOT RABBIT</td>\n",
       "      <td>[el superventas de amantis ¡mejorado!]</td>\n",
       "      <td>39.99</td>\n",
       "      <td>https://www.amantis.net/tobogane-hot-rabbit-el...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BALLENATO</td>\n",
       "      <td>[tu vibrador a distancia con aleta móvil y sum...</td>\n",
       "      <td>49.99</td>\n",
       "      <td>https://www.amantis.net/ballenato-tu-vibrador-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TOBOGANE</td>\n",
       "      <td>[el vibrador doble más vendido]</td>\n",
       "      <td>36.99</td>\n",
       "      <td>https://www.amantis.net/tobogane-el-vibrador-d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MENEO sube y baja</td>\n",
       "      <td>[placer realista con control remoto]</td>\n",
       "      <td>44.99</td>\n",
       "      <td>https://www.amantis.net/meneo-sube-baja-realis...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>FRESH GIRL</td>\n",
       "      <td>[6 Kilos y 40cm de piel real disfrutable]</td>\n",
       "      <td>99.99</td>\n",
       "      <td>https://www.amantis.net/fresh-girl-6-kilos-40c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>LIZO 2</td>\n",
       "      <td>[Dildo de suave silicona en 3 tamaños]</td>\n",
       "      <td>17.99</td>\n",
       "      <td>https://www.amantis.net/lizo-2-dildo-suave-sil...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>REGGIA</td>\n",
       "      <td>[masturbador masculino doble]</td>\n",
       "      <td>29.99</td>\n",
       "      <td>https://www.amantis.net/reggia-masturbador-mas...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pro ANAL</td>\n",
       "      <td>[vibrador anal progresivo]</td>\n",
       "      <td>34.99</td>\n",
       "      <td>https://www.amantis.net/pro-anal-2-vibrador-anal/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>TANDEM 2 flex</td>\n",
       "      <td>[vibrador doble flexible con mando]</td>\n",
       "      <td>59.99</td>\n",
       "      <td>https://www.amantis.net/tandem-2-flex-vibrador...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>FOXTAIL</td>\n",
       "      <td>[plug anal cola de zorro de 35cm]</td>\n",
       "      <td>9.99</td>\n",
       "      <td>https://www.amantis.net/foxtail-plug-anal-cola...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>BISOU</td>\n",
       "      <td>[besos por ondas de succión con la mejor vibra...</td>\n",
       "      <td>44.99</td>\n",
       "      <td>https://www.amantis.net/bisou-besos-ondas-succ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TOK2 bala vibradora con mando</td>\n",
       "      <td>[más potente y recargable]</td>\n",
       "      <td>39.99</td>\n",
       "      <td>https://www.amantis.net/tok2-bala-vibradora-ma...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>SAZZIA</td>\n",
       "      <td>[masturbador hiperrealista]</td>\n",
       "      <td>29.99</td>\n",
       "      <td>https://www.amantis.net/sazzia-vagina-hiperrea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>CRISTALINO XL</td>\n",
       "      <td>[gran dildo transparente de 22cm]</td>\n",
       "      <td>17.99</td>\n",
       "      <td>https://www.amantis.net/cristalino-xl-gran-dil...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>TRIS-TRAS</td>\n",
       "      <td>[anilla con vibrador para doble penetración TR...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>https://www.amantis.net/tris-anilla-vibrador-d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Vibrador Líquido con sabor Desliz! VIBRAGEL 30ml</td>\n",
       "      <td>[hormigueo...]</td>\n",
       "      <td>9.99</td>\n",
       "      <td>https://www.amantis.net/desliz-vibragel-liquid...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Kit de 3 Plugs anales con diamante BLACK STAR ...</td>\n",
       "      <td>[No hay datos]</td>\n",
       "      <td>24.99</td>\n",
       "      <td>https://www.amantis.net/kit-3-plugs-anales-dia...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>IDYLLIC BOY</td>\n",
       "      <td>[Plug anal a distancia más intenso]</td>\n",
       "      <td>54.99</td>\n",
       "      <td>https://www.amantis.net/idyllic-boy-grosso-plu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>CUORE TRÍO</td>\n",
       "      <td>[Kit de 3 plugs con corazón]</td>\n",
       "      <td>24.99</td>\n",
       "      <td>https://www.amantis.net/cuore-trio-kit-3-plugs...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Átame-Fácil</td>\n",
       "      <td>[kit de cama universal de amantis]</td>\n",
       "      <td>24.99</td>\n",
       "      <td>https://www.amantis.net/atame-facil-kit-cama-u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>YOLA. Suave</td>\n",
       "      <td>[Texturizada y Preciosista, masturbación defin...</td>\n",
       "      <td>22.99</td>\n",
       "      <td>https://www.amantis.net/yola-suave-texturizada...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>VULVANIC</td>\n",
       "      <td>[Vibrador a distancia ergonómico. Entrarás en ...</td>\n",
       "      <td>49.99</td>\n",
       "      <td>https://www.amantis.net/vulvanic-vibrador-dist...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>PASSION PUMP</td>\n",
       "      <td>[bomba de succión automática de amantis]</td>\n",
       "      <td>59.99</td>\n",
       "      <td>https://www.amantis.net/passion-pump-bomba-suc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>PULSO AIR - Vibrador palpitador con succionador</td>\n",
       "      <td>[No hay datos]</td>\n",
       "      <td>59.99</td>\n",
       "      <td>https://www.amantis.net/pulso-air-vibrador-pal...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 Name  \\\n",
       "0                                 TOBOGANE HOT RABBIT   \n",
       "1                                           BALLENATO   \n",
       "2                                            TOBOGANE   \n",
       "3                                   MENEO sube y baja   \n",
       "4                                          FRESH GIRL   \n",
       "5                                              LIZO 2   \n",
       "6                                              REGGIA   \n",
       "7                                            Pro ANAL   \n",
       "8                                       TANDEM 2 flex   \n",
       "9                                             FOXTAIL   \n",
       "10                                              BISOU   \n",
       "11                      TOK2 bala vibradora con mando   \n",
       "12                                             SAZZIA   \n",
       "13                                      CRISTALINO XL   \n",
       "14                                          TRIS-TRAS   \n",
       "15   Vibrador Líquido con sabor Desliz! VIBRAGEL 30ml   \n",
       "16  Kit de 3 Plugs anales con diamante BLACK STAR ...   \n",
       "17                                        IDYLLIC BOY   \n",
       "18                                         CUORE TRÍO   \n",
       "19                                        Átame-Fácil   \n",
       "20                                        YOLA. Suave   \n",
       "21                                           VULVANIC   \n",
       "22                                       PASSION PUMP   \n",
       "23    PULSO AIR - Vibrador palpitador con succionador   \n",
       "\n",
       "                                          Description  Price  \\\n",
       "0              [el superventas de amantis ¡mejorado!]  39.99   \n",
       "1   [tu vibrador a distancia con aleta móvil y sum...  49.99   \n",
       "2                     [el vibrador doble más vendido]  36.99   \n",
       "3                [placer realista con control remoto]  44.99   \n",
       "4           [6 Kilos y 40cm de piel real disfrutable]  99.99   \n",
       "5              [Dildo de suave silicona en 3 tamaños]  17.99   \n",
       "6                       [masturbador masculino doble]  29.99   \n",
       "7                          [vibrador anal progresivo]  34.99   \n",
       "8                 [vibrador doble flexible con mando]  59.99   \n",
       "9                   [plug anal cola de zorro de 35cm]   9.99   \n",
       "10  [besos por ondas de succión con la mejor vibra...  44.99   \n",
       "11                         [más potente y recargable]  39.99   \n",
       "12                        [masturbador hiperrealista]  29.99   \n",
       "13                  [gran dildo transparente de 22cm]  17.99   \n",
       "14  [anilla con vibrador para doble penetración TR...  24.99   \n",
       "15                                     [hormigueo...]   9.99   \n",
       "16                                     [No hay datos]  24.99   \n",
       "17                [Plug anal a distancia más intenso]  54.99   \n",
       "18                       [Kit de 3 plugs con corazón]  24.99   \n",
       "19                 [kit de cama universal de amantis]  24.99   \n",
       "20  [Texturizada y Preciosista, masturbación defin...  22.99   \n",
       "21  [Vibrador a distancia ergonómico. Entrarás en ...  49.99   \n",
       "22           [bomba de succión automática de amantis]  59.99   \n",
       "23                                     [No hay datos]  59.99   \n",
       "\n",
       "                                                 link  \n",
       "0   https://www.amantis.net/tobogane-hot-rabbit-el...  \n",
       "1   https://www.amantis.net/ballenato-tu-vibrador-...  \n",
       "2   https://www.amantis.net/tobogane-el-vibrador-d...  \n",
       "3   https://www.amantis.net/meneo-sube-baja-realis...  \n",
       "4   https://www.amantis.net/fresh-girl-6-kilos-40c...  \n",
       "5   https://www.amantis.net/lizo-2-dildo-suave-sil...  \n",
       "6   https://www.amantis.net/reggia-masturbador-mas...  \n",
       "7   https://www.amantis.net/pro-anal-2-vibrador-anal/  \n",
       "8   https://www.amantis.net/tandem-2-flex-vibrador...  \n",
       "9   https://www.amantis.net/foxtail-plug-anal-cola...  \n",
       "10  https://www.amantis.net/bisou-besos-ondas-succ...  \n",
       "11  https://www.amantis.net/tok2-bala-vibradora-ma...  \n",
       "12  https://www.amantis.net/sazzia-vagina-hiperrea...  \n",
       "13  https://www.amantis.net/cristalino-xl-gran-dil...  \n",
       "14  https://www.amantis.net/tris-anilla-vibrador-d...  \n",
       "15  https://www.amantis.net/desliz-vibragel-liquid...  \n",
       "16  https://www.amantis.net/kit-3-plugs-anales-dia...  \n",
       "17  https://www.amantis.net/idyllic-boy-grosso-plu...  \n",
       "18  https://www.amantis.net/cuore-trio-kit-3-plugs...  \n",
       "19  https://www.amantis.net/atame-facil-kit-cama-u...  \n",
       "20  https://www.amantis.net/yola-suave-texturizada...  \n",
       "21  https://www.amantis.net/vulvanic-vibrador-dist...  \n",
       "22  https://www.amantis.net/passion-pump-bomba-suc...  \n",
       "23  https://www.amantis.net/pulso-air-vibrador-pal...  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titulos=soup.find_all(\"h3\")\n",
    "name=[]\n",
    "desc=[]\n",
    "price=[]\n",
    "lista_URLs = []\n",
    "\n",
    "for titulo in titulos:\n",
    "    nombre=titulo.get_text(strip=True).split(',')[0]\n",
    "    name.append(nombre)\n",
    "    description=titulo.get_text(strip=True).split(', ')[1:]\n",
    "    if description==[]:\n",
    "        description=[\"No hay datos\"]\n",
    "    desc.append(description)\n",
    "\n",
    "all_price=soup.find_all(\"span\",class_=\"productSpecialPrice\")\n",
    "\n",
    "for precio in all_price:\n",
    "    item_price=precio.get_text(strip=True).replace(\",\", \".\").split('€')[0]\n",
    "    price.append(item_price)\n",
    "\n",
    "productos = soup.find_all(class_='caption')\n",
    "for producto in productos[8:]:\n",
    "    URL_producto = producto.find('a')['href']\n",
    "    lista_URLs.append(URL_producto)\n",
    "\n",
    "\n",
    "name.pop(0)\n",
    "desc.pop(0)\n",
    "lista_URLs.pop(0)\n",
    "\n",
    "df_productos = pd.DataFrame({\"Name\": name,\"Description\": desc,\"Price\":price,\"link\":lista_URLs})\n",
    "df_productos"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vamos a sacar la información de todas las páginas posibles.\n",
    "\n",
    "Para esto vamos a obtener las URLs, como en el punto d. \n",
    "\n",
    "Para extraer los links de los productos y ver si podemos sacar la información de su página concreta."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.amantis.net/tobogane-hot-rabbit-el-superventas-amantis-mejorado/'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lista_URLs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hacemos un nuevo request para el primer libro: \n",
    "r = requests.get(lista_URLs[0])\n",
    "\n",
    "# Creamos una sopa específica con la info de cada libro\n",
    "soup_producto = bs(r.text, \"lxml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup_producto"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TOBOGANE HOT RABBIT, el superventas de amantis ¡mejorado!\n"
     ]
    }
   ],
   "source": [
    "name = soup_producto.find('h1').text\n",
    "print(name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a.- Vamos a entrar en una página general para extraer información de ella.\n",
    "\n",
    "Principalmente vamos a ver como podemos, con un bucle, sacar todos los datos de cada producto."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<h3 class=\"t2sDiv-titulo hidden text-left color-corporativo\">Top ventas en amantis</h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/tobogane-hot-rabbit-el-superventas-amantis-mejorado/\">\n",
       " <span>TOBOGANE HOT RABBIT, el superventas de amantis ¡mejorado!</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/ballenato-tu-vibrador-distancia-aleta-movil-sumergible/\">\n",
       " <span>BALLENATO, tu vibrador a distancia con aleta móvil y sumergible...</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/tobogane-el-vibrador-doble-mas-vendido-ahora-efecto-hot/\">\n",
       " <span>TOBOGANE, el vibrador doble más vendido</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/meneo-sube-baja-realista-control-remoto/\">\n",
       " <span>MENEO sube y baja, placer realista con control remoto</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/fresh-girl-6-kilos-40cm-piel-real-disfrutable/\">\n",
       " <span>FRESH GIRL, 6 Kilos y 40cm de piel real disfrutable</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/lizo-2-dildo-suave-silicona-3-tamanos/\">\n",
       " <span>LIZO 2, Dildo de suave silicona en 3 tamaños</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/reggia-masturbador-masculino-doble/\">\n",
       " <span>REGGIA, masturbador masculino doble</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/pro-anal-2-vibrador-anal/\">\n",
       " <span>Pro ANAL, vibrador anal progresivo</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/tandem-2-flex-vibrador-doble-flexible-mando/\">\n",
       " <span>TANDEM 2 flex, vibrador doble flexible con mando</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/foxtail-plug-anal-cola-zorro/\">\n",
       " <span>FOXTAIL, plug anal cola de zorro de 35cm</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/bisou-besos-ondas-succion-la-mejor-vibracion/\">\n",
       " <span>BISOU, besos por ondas de succión con la mejor vibración</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/tok2-bala-vibradora-mando-mas-potente-recargable/\">\n",
       " <span>TOK2 bala vibradora con mando, más potente y recargable</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/sazzia-vagina-hiperrealista-amantis/\">\n",
       " <span>SAZZIA, masturbador hiperrealista</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/cristalino-xl-gran-dildo-transparente-22cm/\">\n",
       " <span>CRISTALINO XL, gran dildo transparente de 22cm</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/tris-anilla-vibrador-doble-penetracion-trinity/\">\n",
       " <span>TRIS-TRAS, anilla con vibrador para doble penetración TRINITY...</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/desliz-vibragel-liquido-vibrador-30ml/\">\n",
       " <span>Vibrador Líquido con sabor Desliz! VIBRAGEL 30ml, hormigueo...</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/kit-3-plugs-anales-diamante-black-star-plugress/\">\n",
       " <span>Kit de 3 Plugs anales con diamante BLACK STAR Plugress</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/idyllic-boy-grosso-plug-anal-distancia-mas-intenso/\">\n",
       " <span>IDYLLIC BOY, Plug anal a distancia más intenso</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/cuore-trio-kit-3-plugs-corazon/\">\n",
       " <span>CUORE TRÍO, Kit de 3 plugs con corazón</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/atame-facil-kit-cama-universal-amantis/\">\n",
       " <span>Átame-Fácil, kit de cama universal de amantis</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/yola-suave-texturizada-preciosista-masturbacion-definitiva/\">\n",
       " <span>YOLA. Suave, Texturizada y Preciosista, masturbación definitiva...</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/vulvanic-vibrador-distancia-ergonomico-entraras-erupcion/\">\n",
       " <span>VULVANIC, Vibrador a distancia ergonómico. Entrarás en erupción...</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/passion-pump-bomba-succion-automatica-amantis/\">\n",
       " <span>PASSION PUMP, bomba de succión automática de amantis</span>\n",
       " </a>\n",
       " </h3>,\n",
       " <h3 class=\"h3 group inner list-group-item-heading tdd_listado_nombre\">\n",
       " <a href=\"https://www.amantis.net/pulso-air-vibrador-palpitador-succionador/\">\n",
       " <span>PULSO AIR - Vibrador palpitador con succionador</span>\n",
       " </a>\n",
       " </h3>]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page=3\n",
    "url = \"https://www.amantis.net/productos-amantis/\"              # lista productos\n",
    "URL = url+'page' + str(page)+'/'\n",
    "response = requests.get(URL)\n",
    "titulos=soup.find_all(\"h3\")\n",
    "titulos"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Este es el bucle para obtener información anterior de las 5 primeras paginas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pagina 2\n",
      "Pagina 3\n",
      "Pagina 4\n",
      "Nombres:  72\n",
      "Descripción:  72\n",
      "URL:  72\n",
      "Precios:  72\n"
     ]
    }
   ],
   "source": [
    "url = \"https://www.amantis.net/productos-amantis/\"              # lista productos\n",
    "url_principal=\"https://www.amantis.net/\"                        # productos\n",
    "response = requests.get(url)\n",
    "\n",
    "\n",
    "pages= np.arange(2,5)                                   # La primera pagina tiene una serie de datos que no debemos de recoger.\n",
    "name=[]\n",
    "desc=[]\n",
    "price=[]\n",
    "lista_URLs = []\n",
    "\n",
    "for page in pages:\n",
    "    if page == 1:\n",
    "        print(\"Pagina\",page)\n",
    "        URL = url\n",
    "        response = requests.get(url)\n",
    "        soup = bs(response.text, 'lxml')\n",
    "        titulos=soup.find_all(\"h3\")\n",
    "\n",
    "        for titulo in titulos:\n",
    "            nombre=titulo.get_text(strip=True).split(',')[0]\n",
    "            name.append(nombre)\n",
    "            description=titulo.get_text(strip=True).split(', ')[1:]\n",
    "            if description==[]:\n",
    "                description=[\"No hay datos\"]\n",
    "            desc.append(description)\n",
    "            print(nombre)\n",
    "\n",
    "        all_price=soup.find_all(\"span\",class_=\"productSpecialPrice\")\n",
    "\n",
    "        for precio in all_price:\n",
    "            item_price=precio.get_text(strip=True).replace(\",\", \".\").split('€')[0]\n",
    "            price.append(item_price)\n",
    "\n",
    "\n",
    "        productos = soup.find_all(class_='caption')\n",
    "\n",
    "        for producto in productos[9:]:\n",
    "            URL_producto = producto.find('a')['href']\n",
    "            lista_URLs.append(URL_producto)\n",
    "            print(URL_producto)\n",
    "            \n",
    "            \n",
    "    else:\n",
    "        print(\"Pagina\",page)\n",
    "        URL = url+'page' + str(page)+'/'\n",
    "        response = requests.get(URL)\n",
    "        soup = bs(response.text, 'lxml')\n",
    "        titulos=soup.find_all(\"h3\")\n",
    "\n",
    "        for titulo in titulos:\n",
    "            nombre=titulo.get_text(strip=True).split(',')[0]\n",
    "            name.append(nombre)\n",
    "            description=titulo.get_text(strip=True).split(', ')[1:]\n",
    "            if description==[]:\n",
    "                description=[\"No hay datos\"]\n",
    "            desc.append(description)\n",
    "            # print(nombre)\n",
    "\n",
    "        for precio in all_price:\n",
    "            item_price=precio.get_text(strip=True).replace(\",\", \".\").split('€')[0]\n",
    "            price.append(item_price)\n",
    "\n",
    "        productos = soup.find_all(class_='caption')\n",
    "\n",
    "        for producto in productos[9:]:\n",
    "            URL_producto = producto.find('a')['href']\n",
    "            lista_URLs.append(URL_producto)\n",
    "            # print(URL_producto)\n",
    "\n",
    "print(\"Nombres: \",len(name))\n",
    "print(\"Descripción: \",len(desc))\n",
    "print(\"URL: \",len(lista_URLs))\n",
    "print(\"Precios: \",len(price))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dado que vemos que hay incongruencias en la longitud del tamaño de cada lista, vamos a realizar un Dataframe unicamente de los datos con la misma longitud.\n",
    "\n",
    "Esta problemática la solucionaremos más adelante."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Description</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MAGARAC</td>\n",
       "      <td>[Kit de arnés + esposas by amantis]</td>\n",
       "      <td>https://www.amantis.net/magarac-kit-arnes-espo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>PULSO AIR - Vibrador palpitador con succionador</td>\n",
       "      <td>[No hay datos]</td>\n",
       "      <td>https://www.amantis.net/pulso-air-vibrador-pal...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TYPHOON</td>\n",
       "      <td>[masturbador masculino giroscópico]</td>\n",
       "      <td>https://www.amantis.net/typhoon-masturbador-ma...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>POWER UP METER - Bomba de succión con manómetro</td>\n",
       "      <td>[No hay datos]</td>\n",
       "      <td>https://www.amantis.net/power-up-meter-bomba-s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AVENTURE -  Vibrador con imán y mando a distancia</td>\n",
       "      <td>[No hay datos]</td>\n",
       "      <td>https://www.amantis.net/aventure-vibrador-iman...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                Name  \\\n",
       "0                                            MAGARAC   \n",
       "1    PULSO AIR - Vibrador palpitador con succionador   \n",
       "2                                            TYPHOON   \n",
       "3    POWER UP METER - Bomba de succión con manómetro   \n",
       "4  AVENTURE -  Vibrador con imán y mando a distancia   \n",
       "\n",
       "                           Description  \\\n",
       "0  [Kit de arnés + esposas by amantis]   \n",
       "1                       [No hay datos]   \n",
       "2  [masturbador masculino giroscópico]   \n",
       "3                       [No hay datos]   \n",
       "4                       [No hay datos]   \n",
       "\n",
       "                                                link  \n",
       "0  https://www.amantis.net/magarac-kit-arnes-espo...  \n",
       "1  https://www.amantis.net/pulso-air-vibrador-pal...  \n",
       "2  https://www.amantis.net/typhoon-masturbador-ma...  \n",
       "3  https://www.amantis.net/power-up-meter-bomba-s...  \n",
       "4  https://www.amantis.net/aventure-vibrador-iman...  "
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_productos = pd.DataFrame({\"Name\": name,\"Description\": desc,\"link\":lista_URLs})\n",
    "df_productos.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b.- Verifico que he conseguido las URLs de cada producto, esto es importante para obtener la información de cada producto."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extraer la información de producto, descripción, enlace y precio tomando los datos desde las URLs de cada producto.\n",
    "\n",
    "Queda pendiente extraer información de los ratings y los comentarios para establecer un estudio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://www.amantis.net/productos-amantis/\"              # lista productos\n",
    "url_principal=\"https://www.amantis.net/\"                        # productos\n",
    "response = requests.get(url)\n",
    "\n",
    "\n",
    "pages= np.arange(1, 25)\n",
    "name=[]\n",
    "desc=[]\n",
    "price=[]\n",
    "lista_URLs = []\n",
    "\n",
    "for page in pages:\n",
    "    if page == 1:\n",
    "        print(\"Pagina\",page)\n",
    "        URL = url\n",
    "        response = requests.get(url)\n",
    "        soup = bs(response.text, 'lxml')\n",
    "        titulos=soup.find_all(\"h3\")\n",
    "\n",
    "        for titulo in titulos[1:]:\n",
    "            nombre=titulo.get_text(strip=True).split(',')[0]\n",
    "            name.append(nombre)\n",
    "            description=titulo.get_text(strip=True).split(', ')[1:]\n",
    "            if description==[]:\n",
    "                description=[\"No hay datos\"]\n",
    "            desc.append(description)\n",
    "\n",
    "        all_price=soup.find_all(\"span\",class_=\"productSpecialPrice\")\n",
    "\n",
    "        for precio in all_price:\n",
    "            item_price=precio.get_text(strip=True).replace(\",\", \".\").split('€')[0]\n",
    "            price.append(item_price)\n",
    "\n",
    "\n",
    "        productos = soup.find_all(class_='caption')\n",
    "\n",
    "        for producto in productos[9:]:\n",
    "            URL_producto = producto.find('a')['href']\n",
    "            lista_URLs.append(URL_producto)\n",
    "            \n",
    "            \n",
    "    else:\n",
    "        print(\"Pagina\",page)\n",
    "        URL = url+'page' + str(page)+'/'\n",
    "        response = requests.get(URL)\n",
    "        soup = bs(response.text, 'lxml')\n",
    "        titulos=soup.find_all(\"h3\")\n",
    "\n",
    "        for titulo in titulos:\n",
    "            nombre=titulo.get_text(strip=True).split(',')[0]\n",
    "            name.append(nombre)\n",
    "            description=titulo.get_text(strip=True).split(', ')[1:]\n",
    "            if description==[]:\n",
    "                description=[\"No hay datos\"]\n",
    "            desc.append(description)\n",
    "\n",
    "        for precio in all_price:\n",
    "            item_price=precio.get_text(strip=True).replace(\",\", \".\").split('€')[0]\n",
    "            price.append(item_price)\n",
    "\n",
    "        productos = soup.find_all(class_='caption')\n",
    "\n",
    "        for producto in productos[9:]:\n",
    "            URL_producto = producto.find('a')['href']\n",
    "            lista_URLs.append(URL_producto)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# df_productos = pd.DataFrame({\"Name\": name,\"Description\": desc,\"Price\":price,\"link\":lista_URLs})\n",
    "# df_productos.head()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verificamos que cantidad de datos tenemos al recorrer todos las paginas con productos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nombres: 576\n",
      "descrip: 576\n",
      "precio: 576\n",
      "URL: 576\n"
     ]
    }
   ],
   "source": [
    "print(\"Nombres:\" ,len(name))\n",
    "print(\"descrip:\" ,len(desc))\n",
    "print(\"precio:\" ,len(price))                    #  Se ve que hay un desajuste en el precio al extraer la información de 1 pagina\n",
    "print(\"URL:\" ,len(lista_URLs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Nombres:\\n\" ,name[:5])\n",
    "print(\"descrip:\\n\" ,desc[:5])\n",
    "print(\"precio:\\n\" ,price[:5])\n",
    "print(\"URL:\\n\" ,lista_URLs[:3])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### c.- Extrayendo la información de los comentarios de cada producto."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos a entrar en un producto para obtener la información de los comentarios que hay:\n",
    "- Usuario\n",
    "- Fecha\n",
    "- Comentario\n",
    "- Rating "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prueba=lista_URLs[1]\n",
    "response = requests.get(prueba)\n",
    "soup_prueba = bs(response.text, 'lxml')\n",
    "\n",
    "titulo=soup_prueba.get_text(strip=True).split(',')[0]\n",
    "print(titulo)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Fechas**\n",
    "\n",
    "Estos datos son en este momento, *string*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rating=[]\n",
    "all_ratings = soup_prueba.find_all(\"span\", class_=\"date\")  \n",
    "sep_1=(' ')\n",
    "sep_2=(', ')\n",
    "# pattern = re.compile(sep)\n",
    "for ratings in all_ratings:\n",
    "    rating_coment_1=ratings.get_text(strip=True).split(sep_2)[1]\n",
    "    rating_coment_2=ratings.get_text(strip=True).split(sep_2)[0]\n",
    "    rating_coment_3=rating_coment_2.split(sep_1)[1:]\n",
    "    date_1=rating_coment_3[0]+\"/\"+rating_coment_3[1]\n",
    "    date_2=date_1+\"/\"+rating_coment_1                                       # Estamos pendientes de convertir a fechas, teniendo en cuenta\n",
    "    # date_object = datetime.strptime(date_2,'%d%m%Y')                        # que esta en español\n",
    "    # print(type(date_1))\n",
    "    # print(date_1)\n",
    "    rating.append(date_2)\n",
    "    # print(type(date_2))\n",
    "    # print(date_2)\n",
    "\n",
    "print(len(rating))\n",
    "rating"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Usuarios**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_comments=[]\n",
    "all_user_comments = soup_prueba.find_all(\"span\", class_=\"name-user\")  \n",
    "\n",
    "for user_comment in all_user_comments:\n",
    "    user_comments.append(user_comment.get_text(strip=True))\n",
    "\n",
    "print(len(user_comments))\n",
    "user_comments"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Comentarios**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comment=[]\n",
    "all_comments = soup_prueba.find_all(\"p\")\n",
    "for formats in all_comments[-len(rating):]:                     # Creo que me he cargado 'rating'\n",
    "    comment.append(formats.get_text(strip=True))\n",
    "\n",
    "print(comment[1])\n",
    "len(comment)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hasta aquí los datos que estamos obteniendo se pasan a una lista por atributo."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### d.- Este es el código bueno para iterar todos los productos extraer la información siguiente:\n",
    "\n",
    "- Nombre\n",
    "- Descripción\n",
    "- Precio sin rebaja (regular_price)\n",
    "- Precio rebajado (new_price)\n",
    "- Información de cada producto\n",
    "- Lista de usuarios que han comentado\n",
    "- Lista de comentarios\n",
    "- Lista de Fecha de comentarios\n",
    "- Lista de Ratings.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pagina 1\n",
      "Pagina 2\n",
      "Pagina 3\n",
      "Pagina 4\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n"
     ]
    }
   ],
   "source": [
    "url = \"https://www.amantis.net/productos-amantis/\"              # lista productos\n",
    "url_principal=\"https://www.amantis.net/\"                        # productos\n",
    "pages= np.arange(1,5)\n",
    "# pages= np.arange(1, 25)\n",
    "\n",
    "'''Listas a generar con la información de los productos'''\n",
    "lista_URLs = []\n",
    "name=[]\n",
    "subname=[]\n",
    "regular_prices=[]\n",
    "new_price=[]\n",
    "# lista_URLs = []\n",
    "info=[]\n",
    "charac=[]\n",
    "user_comments=[]\n",
    "comment=[]\n",
    "date=[]\n",
    "ratings=[]\n",
    "\n",
    "''' Obtenemos las URLs de los productos para entrar luego en sus URLS y extraer la información'''\n",
    "\n",
    "for page in pages:\n",
    "    if page == 1:\n",
    "        print(\"Pagina\",page)\n",
    "        URL = url\n",
    "        response = requests.get(url)\n",
    "        soup = bs(response.text, 'lxml')\n",
    "        productos = soup.find_all(class_='caption')\n",
    "        for producto in productos[9:]:\n",
    "            URL_producto = producto.find('a')['href']\n",
    "            lista_URLs.append(URL_producto)\n",
    "        \n",
    "    else:\n",
    "        print(\"Pagina\",page)\n",
    "        URL = url+'page' + str(page)+'/'\n",
    "        response = requests.get(URL)\n",
    "        soup = bs(response.text, 'lxml')\n",
    "        productos = soup.find_all(class_='caption')\n",
    "        for producto in productos[9:]:\n",
    "            URL_producto = producto.find('a')['href']\n",
    "            lista_URLs.append(URL_producto)\n",
    "\n",
    "'''Extraemos la información de cada producto existente'''\n",
    "\n",
    "for URL in lista_URLs:\n",
    "    url_product=URL\n",
    "    response_product = requests.get(url_product)\n",
    "    soup_product = bs(response_product.text, 'lxml')\n",
    "    user_comments_product=[]\n",
    "    date_comments_product=[]\n",
    "    comments_product=[]\n",
    "    rating=[]\n",
    "    \n",
    "\n",
    "    titulos=soup_product.find_all(\"h1\",class_=\"h3\")\n",
    "    for titulo in titulos:\n",
    "        nombre=titulo.get_text(strip=True).split(',')[0]\n",
    "        name.append(nombre)\n",
    "        sub_title=titulo.get_text(strip=True).split(', ')[1:]\n",
    "        if sub_title==[]:\n",
    "            sub_title=[\"No hay datos\"]\n",
    "        subname.append(sub_title)\n",
    "\n",
    "    all_price = soup_product.find_all(\"div\", class_=\"productoPrecio pull-right tdd_precio\")                        \n",
    "    for price_container in all_price:                                                                    \n",
    "        try:\n",
    "            special_price = price_container.find(\"span\", class_=\"productSpecialPrice\")\n",
    "            if special_price:\n",
    "                item_price = float(special_price.get_text(strip=True).replace(\",\", \".\").split('€')[0])\n",
    "                new_price.append(item_price)\n",
    "                regular_price = price_container.find(\"del\").get_text(strip=True)\n",
    "                item_regular_price = float(regular_price.replace(\",\", \".\").split('€')[0])\n",
    "                regular_prices.append(item_regular_price)\n",
    "            else:\n",
    "                regular_price = price_container.find(\"span\").get_text(strip=True)\n",
    "                item_regular_price = float(regular_price.replace(\",\", \".\").split('€')[0])\n",
    "                new_price.append(item_regular_price)\n",
    "                regular_prices.append(None)\n",
    "        except:\n",
    "            new_price.append(None)\n",
    "            regular_prices.append(None)\n",
    "\n",
    "    description=soup_product.find(\"div\", class_=\"description\") \n",
    "    information=description.get_text().split('\\n')[1:5]\n",
    "    info.append(information)\n",
    "    characteristic=description.get_text().split('\\n')[5:-3]\n",
    "    charac.append(characteristic)\n",
    "    '''Vamos a obtener los datos de los comentarios de los usuarios'''\n",
    "\n",
    "    all_user_comments = soup_product.find_all(\"span\", class_=\"name-user\") \n",
    "    for user_comment in all_user_comments:\n",
    "        user_comments_product.append(user_comment.get_text(strip=True))\n",
    "    user_comments.append(user_comments_product)\n",
    "\n",
    "    all_dates = soup_product.find_all(\"span\", class_=\"date\")  \n",
    "    for dates in all_dates:\n",
    "        dates_text=dates.get_text(strip=True)\n",
    "        # dates=datetime.strftime(dates, '%dd/%mm/%Y')\n",
    "        date_comments_product.append(dates_text)\n",
    "        # date_object = datetime.strptime(date_comments_product)\n",
    "    date.append(date_comments_product)\n",
    "\n",
    "    all_comments = soup_product.find_all(\"p\")\n",
    "    for formats in all_comments[-len(date_comments_product):]:\n",
    "        comments_product.append(formats.get_text(strip=True))\n",
    "    comment.append(comments_product)\n",
    "\n",
    "    hearts = soup_product.find_all('div', class_= 'box-description')\n",
    "    for heart in hearts:\n",
    "        heart_rating = heart.find_all('span', class_= 'fas fa-heart')\n",
    "        num_hearts = len(heart_rating)\n",
    "        rating.append(num_hearts)\n",
    "    ratings.append(rating)\n",
    "\n",
    "\n",
    "for i, regular_price in enumerate(regular_prices):\n",
    "    if regular_price is None:\n",
    "        regular_prices[i] = new_price[i]\n",
    "\n",
    "\n",
    "# print('-'*20)\n",
    "# print(name)\n",
    "print('-'*20)\n",
    "print(len(name))\n",
    "# print('-'*20)\n",
    "# print(subname)\n",
    "print('-'*20)\n",
    "print(len(subname))\n",
    "print('-'*20)\n",
    "# print(regular_prices)\n",
    "# print('-'*20)\n",
    "print(len(regular_prices))\n",
    "# print('-'*20)\n",
    "# print(new_price)\n",
    "print('-'*20)\n",
    "print(len(new_price))\n",
    "print('-'*20)\n",
    "print(len(lista_URLs))\n",
    "# print('-'*20)\n",
    "# print(lista_URLs)\n",
    "print('-'*20)\n",
    "print(len(info))\n",
    "# print('-'*20)\n",
    "# # print(charac)\n",
    "print('-'*20)\n",
    "print(len(charac))\n",
    "# print('-'*20)\n",
    "# # print(user_comments)\n",
    "print('-'*20)\n",
    "print(len(user_comments))\n",
    "# print('-'*20)\n",
    "# # print(comment)\n",
    "print('-'*20)\n",
    "print(len(comment))\n",
    "# # print('-'*20)\n",
    "# # print(date)\n",
    "print('-'*20)\n",
    "print(len(date))\n",
    "print('-'*20)\n",
    "print(len(ratings))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizamos los datos para ver qué se obtiene y lo guardamos en un fichero .csv.\n",
    "\n",
    "La intención es depurar el codigo para un mejor manejo de los datos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Subname</th>\n",
       "      <th>Description</th>\n",
       "      <th>Characteristics</th>\n",
       "      <th>Price</th>\n",
       "      <th>Reduced Price</th>\n",
       "      <th>date</th>\n",
       "      <th>User</th>\n",
       "      <th>Ratings</th>\n",
       "      <th>Comment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>TOBOGANE HOT RABBIT</td>\n",
       "      <td>[el superventas de amantis ¡mejorado!]</td>\n",
       "      <td>[Vuelve nuestro vibrador de doble estimulación...</td>\n",
       "      <td>[, , Medidas: 19cm (11cm insertables) y 3,3cm/...</td>\n",
       "      <td>14.99</td>\n",
       "      <td>39.99</td>\n",
       "      <td>[martes 22 noviembre, 2022, jueves 07 julio, 2...</td>\n",
       "      <td>[Rossi, Marina, Jennifer, Noa, Karen, Lorena, ...</td>\n",
       "      <td>[5, 5, 5, 4, 3, 5, 5, 5, 5, 5, 4, 5, 5, 3, 5, ...</td>\n",
       "      <td>[Mi primera compra. Me encantó la textura, los...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BALLENATO</td>\n",
       "      <td>[tu vibrador a distancia con aleta móvil y sum...</td>\n",
       "      <td>[De las profundidades más húmedas llega BALLEN...</td>\n",
       "      <td>[¡FREE BALLENATO! Ver características y medida...</td>\n",
       "      <td>14.99</td>\n",
       "      <td>49.99</td>\n",
       "      <td>[sábado 18 febrero, 2023, miércoles 01 febrero...</td>\n",
       "      <td>[Almudena, Tomabel, andrea, Carlos, maria, Mar...</td>\n",
       "      <td>[5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, ...</td>\n",
       "      <td>[Sigo temblando con este juguete, menudas vibr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TOBOGANE</td>\n",
       "      <td>[el vibrador doble más vendido]</td>\n",
       "      <td>[Por favor, desabróchense los cinturones de se...</td>\n",
       "      <td>[, La última atracción exclusiva de amantis es...</td>\n",
       "      <td>14.99</td>\n",
       "      <td>36.99</td>\n",
       "      <td>[lunes 06 marzo, 2023, sábado 04 marzo, 2023, ...</td>\n",
       "      <td>[Teresa, Alicia, María, Sara, Raquel, Sara, Da...</td>\n",
       "      <td>[5, 5, 3, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 5, ...</td>\n",
       "      <td>[Fue el primer vibrador que compré. Me lo reco...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MENEO sube y baja</td>\n",
       "      <td>[placer realista con control remoto]</td>\n",
       "      <td>[Si te gusta que te metan un buen meneo, hazte...</td>\n",
       "      <td>[Ya lo ves, Meneo’s Cock puede ir contigo a do...</td>\n",
       "      <td>14.99</td>\n",
       "      <td>44.99</td>\n",
       "      <td>[miércoles 14 diciembre, 2022, miércoles 09 no...</td>\n",
       "      <td>[Francisco, Maria, Jose Javier, Carlos, Pedro,...</td>\n",
       "      <td>[5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4]</td>\n",
       "      <td>[Quería saber cuantos cm tiene la longitud que...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>FRESH GIRL</td>\n",
       "      <td>[6 Kilos y 40cm de piel real disfrutable]</td>\n",
       "      <td>[FRESH GIRL es una bocanada de aire fresco en ...</td>\n",
       "      <td>[FRESH GIRL es tan pesada que tendrás que usar...</td>\n",
       "      <td>14.99</td>\n",
       "      <td>99.99</td>\n",
       "      <td>[sábado 31 diciembre, 2022, viernes 21 octubre...</td>\n",
       "      <td>[Adrian, Daniel, victor, Guillermo, Jesús, Dav...</td>\n",
       "      <td>[4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, ...</td>\n",
       "      <td>[Este juguete es bastante bueno, muy muy place...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>ARMOUR ALAS - Armadura sexual BDSM con alas</td>\n",
       "      <td>[No hay datos]</td>\n",
       "      <td>[Las prendas de la cultura BDSM han inundado l...</td>\n",
       "      <td>[Este modelo no es una única pieza si no que e...</td>\n",
       "      <td>14.99</td>\n",
       "      <td>29.99</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>[Parece que tu navegador está bloqueando JavaS...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>LAZZO</td>\n",
       "      <td>[Atar tu pene para desatar tu pasión]</td>\n",
       "      <td>[Te agarra la base del pene pero para dar rien...</td>\n",
       "      <td>[Ver características y medidas, , \\r, Funda pa...</td>\n",
       "      <td>14.99</td>\n",
       "      <td>12.99</td>\n",
       "      <td>[sábado 30 abril, 2022, martes 23 noviembre, 2...</td>\n",
       "      <td>[Javier, Miguel]</td>\n",
       "      <td>[4, 4]</td>\n",
       "      <td>[Está hecho de un buen material, es muy elásti...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>MYSTERY</td>\n",
       "      <td>[body de rejilla super sexy]</td>\n",
       "      <td>[El poder de la seducción está en tus manos co...</td>\n",
       "      <td>[Siéntente más sexy que nunca, sin renunciar a...</td>\n",
       "      <td>14.99</td>\n",
       "      <td>4.99</td>\n",
       "      <td>[viernes 09 diciembre, 2022, lunes 28 marzo, 2...</td>\n",
       "      <td>[andrea, María, María, Javier, Lucia, Anonimo]</td>\n",
       "      <td>[4, 3, 5, 4, 5, 5]</td>\n",
       "      <td>[es totalmente precioso, queda estupendo y a m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>Arnés sexual TRIANGLE</td>\n",
       "      <td>[100% Vegano con brillo y sujeción firme]</td>\n",
       "      <td>[En los juegos del amarse y disfrutarse todo p...</td>\n",
       "      <td>[Puedes usar este arnés con dildos de base anc...</td>\n",
       "      <td>14.99</td>\n",
       "      <td>12.99</td>\n",
       "      <td>[sábado 10 diciembre, 2022, jueves 06 octubre,...</td>\n",
       "      <td>[Sara, Sergio, Silvia, Eme, Ramón, Alba, Anne,...</td>\n",
       "      <td>[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]</td>\n",
       "      <td>[Es la segunda vez que lo compro y estoy encan...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>Bacanal</td>\n",
       "      <td>[gel lubricante anal de acción extrema con aloe]</td>\n",
       "      <td>[Bacanal es un lubricante en gel de base de ag...</td>\n",
       "      <td>[]</td>\n",
       "      <td>14.99</td>\n",
       "      <td>9.99</td>\n",
       "      <td>[sábado 13 mayo, 2023, martes 02 mayo, 2023, l...</td>\n",
       "      <td>[Jonatan, Joana, Mariola, Irene, Marta, Carlos...</td>\n",
       "      <td>[5, 5, 5, 5, 5, 5, 5, 4, 5, 5, 5, 5, 5, 5, 4, ...</td>\n",
       "      <td>[El mejor lubricante del mercado, con diferenc...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>96 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           Name  \\\n",
       "0                           TOBOGANE HOT RABBIT   \n",
       "1                                     BALLENATO   \n",
       "2                                      TOBOGANE   \n",
       "3                             MENEO sube y baja   \n",
       "4                                    FRESH GIRL   \n",
       "..                                          ...   \n",
       "91  ARMOUR ALAS - Armadura sexual BDSM con alas   \n",
       "92                                        LAZZO   \n",
       "93                                      MYSTERY   \n",
       "94                        Arnés sexual TRIANGLE   \n",
       "95                                      Bacanal   \n",
       "\n",
       "                                              Subname  \\\n",
       "0              [el superventas de amantis ¡mejorado!]   \n",
       "1   [tu vibrador a distancia con aleta móvil y sum...   \n",
       "2                     [el vibrador doble más vendido]   \n",
       "3                [placer realista con control remoto]   \n",
       "4           [6 Kilos y 40cm de piel real disfrutable]   \n",
       "..                                                ...   \n",
       "91                                     [No hay datos]   \n",
       "92              [Atar tu pene para desatar tu pasión]   \n",
       "93                       [body de rejilla super sexy]   \n",
       "94          [100% Vegano con brillo y sujeción firme]   \n",
       "95   [gel lubricante anal de acción extrema con aloe]   \n",
       "\n",
       "                                          Description  \\\n",
       "0   [Vuelve nuestro vibrador de doble estimulación...   \n",
       "1   [De las profundidades más húmedas llega BALLEN...   \n",
       "2   [Por favor, desabróchense los cinturones de se...   \n",
       "3   [Si te gusta que te metan un buen meneo, hazte...   \n",
       "4   [FRESH GIRL es una bocanada de aire fresco en ...   \n",
       "..                                                ...   \n",
       "91  [Las prendas de la cultura BDSM han inundado l...   \n",
       "92  [Te agarra la base del pene pero para dar rien...   \n",
       "93  [El poder de la seducción está en tus manos co...   \n",
       "94  [En los juegos del amarse y disfrutarse todo p...   \n",
       "95  [Bacanal es un lubricante en gel de base de ag...   \n",
       "\n",
       "                                      Characteristics  Price  Reduced Price  \\\n",
       "0   [, , Medidas: 19cm (11cm insertables) y 3,3cm/...  14.99          39.99   \n",
       "1   [¡FREE BALLENATO! Ver características y medida...  14.99          49.99   \n",
       "2   [, La última atracción exclusiva de amantis es...  14.99          36.99   \n",
       "3   [Ya lo ves, Meneo’s Cock puede ir contigo a do...  14.99          44.99   \n",
       "4   [FRESH GIRL es tan pesada que tendrás que usar...  14.99          99.99   \n",
       "..                                                ...    ...            ...   \n",
       "91  [Este modelo no es una única pieza si no que e...  14.99          29.99   \n",
       "92  [Ver características y medidas, , \\r, Funda pa...  14.99          12.99   \n",
       "93  [Siéntente más sexy que nunca, sin renunciar a...  14.99           4.99   \n",
       "94  [Puedes usar este arnés con dildos de base anc...  14.99          12.99   \n",
       "95                                                 []  14.99           9.99   \n",
       "\n",
       "                                                 date  \\\n",
       "0   [martes 22 noviembre, 2022, jueves 07 julio, 2...   \n",
       "1   [sábado 18 febrero, 2023, miércoles 01 febrero...   \n",
       "2   [lunes 06 marzo, 2023, sábado 04 marzo, 2023, ...   \n",
       "3   [miércoles 14 diciembre, 2022, miércoles 09 no...   \n",
       "4   [sábado 31 diciembre, 2022, viernes 21 octubre...   \n",
       "..                                                ...   \n",
       "91                                                 []   \n",
       "92  [sábado 30 abril, 2022, martes 23 noviembre, 2...   \n",
       "93  [viernes 09 diciembre, 2022, lunes 28 marzo, 2...   \n",
       "94  [sábado 10 diciembre, 2022, jueves 06 octubre,...   \n",
       "95  [sábado 13 mayo, 2023, martes 02 mayo, 2023, l...   \n",
       "\n",
       "                                                 User  \\\n",
       "0   [Rossi, Marina, Jennifer, Noa, Karen, Lorena, ...   \n",
       "1   [Almudena, Tomabel, andrea, Carlos, maria, Mar...   \n",
       "2   [Teresa, Alicia, María, Sara, Raquel, Sara, Da...   \n",
       "3   [Francisco, Maria, Jose Javier, Carlos, Pedro,...   \n",
       "4   [Adrian, Daniel, victor, Guillermo, Jesús, Dav...   \n",
       "..                                                ...   \n",
       "91                                                 []   \n",
       "92                                   [Javier, Miguel]   \n",
       "93     [andrea, María, María, Javier, Lucia, Anonimo]   \n",
       "94  [Sara, Sergio, Silvia, Eme, Ramón, Alba, Anne,...   \n",
       "95  [Jonatan, Joana, Mariola, Irene, Marta, Carlos...   \n",
       "\n",
       "                                              Ratings  \\\n",
       "0   [5, 5, 5, 4, 3, 5, 5, 5, 5, 5, 4, 5, 5, 3, 5, ...   \n",
       "1   [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, ...   \n",
       "2   [5, 5, 3, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 5, ...   \n",
       "3             [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4]   \n",
       "4   [4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, ...   \n",
       "..                                                ...   \n",
       "91                                                 []   \n",
       "92                                             [4, 4]   \n",
       "93                                 [4, 3, 5, 4, 5, 5]   \n",
       "94                     [5, 5, 5, 5, 5, 5, 5, 5, 5, 5]   \n",
       "95  [5, 5, 5, 5, 5, 5, 5, 4, 5, 5, 5, 5, 5, 5, 4, ...   \n",
       "\n",
       "                                              Comment  \n",
       "0   [Mi primera compra. Me encantó la textura, los...  \n",
       "1   [Sigo temblando con este juguete, menudas vibr...  \n",
       "2   [Fue el primer vibrador que compré. Me lo reco...  \n",
       "3   [Quería saber cuantos cm tiene la longitud que...  \n",
       "4   [Este juguete es bastante bueno, muy muy place...  \n",
       "..                                                ...  \n",
       "91  [Parece que tu navegador está bloqueando JavaS...  \n",
       "92  [Está hecho de un buen material, es muy elásti...  \n",
       "93  [es totalmente precioso, queda estupendo y a m...  \n",
       "94  [Es la segunda vez que lo compro y estoy encan...  \n",
       "95  [El mejor lubricante del mercado, con diferenc...  \n",
       "\n",
       "[96 rows x 10 columns]"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# dataframe= pd.DataFrame({\"Name\": name,\"Subname\": subname,\"Description\": info, \"Characteristics\": charac,\"Price\":regular_price,\"Reduced Price\":new_price,\n",
    "#                          \"date\":date,\"User\": user_comments,\"Ratings\": ratings,\"Comment\": comment})\n",
    "# dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# file=folder+'\\\\Data\\\\scrapped_data.csv'\n",
    "# file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dataframe.to_csv(file,header=True,index=False)           # Tengo que generar el path correcto\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solucionando inconvenientes de los datos obtenidos."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- *Subname* puede generar problemas por lo eliminaremos del codigo. Lo generaremos después a partir de *Name*.\n",
    "- *Characteristics* puede generar problemas, ya que serían datos de dimensiones, duración, etc. Lo generaremos después a partir de *Description*.\n",
    "- Hay que pasar las listas a registros individuales."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lista_URLs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# url = \"https://www.amantis.net/productos-amantis/\"              # lista productos\n",
    "# url_principal=\"https://www.amantis.net/\"                        # productos\n",
    "# # pages= np.arange(1,5)\n",
    "# pages= np.arange(1, 25)\n",
    "\n",
    "# '''Listas a generar con la información de los productos'''\n",
    "# lista_URLs = []\n",
    "\n",
    "# ''' Obtenemos las URLs de los productos para entrar luego en sus URLS y extraer la información'''\n",
    "\n",
    "# for page in pages:\n",
    "#     if page == 1:\n",
    "#         print(\"Leyendo paginas\")\n",
    "#         # print(\"Pagina\",page)\n",
    "#         URL = url\n",
    "#         response = requests.get(url)\n",
    "#         soup = bs(response.text, 'lxml')\n",
    "#         productos = soup.find_all(class_='caption')\n",
    "#         for producto in productos[9:]:\n",
    "#             URL_producto = producto.find('a')['href']\n",
    "#             lista_URLs.append(URL_producto)\n",
    "        \n",
    "#     else:\n",
    "# #        print(\"Pagina\",page)\n",
    "#         URL = url+'page' + str(page)+'/'\n",
    "#         response = requests.get(URL)\n",
    "#         soup = bs(response.text, 'lxml')\n",
    "#         productos = soup.find_all(class_='caption')\n",
    "#         for producto in productos[9:]:\n",
    "#             URL_producto = producto.find('a')['href']\n",
    "#             lista_URLs.append(URL_producto)\n",
    "# print(\"Terminando lectura.\\nRecabando información.\")\n",
    "\n",
    "# list_name=[]\n",
    "# list_regular_prices=[]\n",
    "# list_new_price=[]\n",
    "# list_info=[]\n",
    "# user_comments=[]\n",
    "# comment=[]\n",
    "# date=[]\n",
    "# ratings=[]\n",
    "# datas=[]\n",
    "# diccionario_URL={}\n",
    "\n",
    "\n",
    "\n",
    "# '''Extraemos la información de cada producto existente'''\n",
    "\n",
    "# for URL in lista_URLs:\n",
    "#     url_product=URL\n",
    "#     response_product = requests.get(url_product)\n",
    "#     soup_product = bs(response_product.text, 'lxml')\n",
    "#     user_comments_product=[]\n",
    "#     date_comments_product=[]\n",
    "#     comments_product=[]\n",
    "#     rating=[]\n",
    "#     data=[]\n",
    "#     name=[]\n",
    "#     regular_prices=[]\n",
    "#     new_price=[]\n",
    "#     info=[]\n",
    "\n",
    "#     diccionario_comments={}\n",
    "\n",
    "#     titulos=soup_product.find_all(\"h1\",class_=\"h3\")\n",
    "#     for titulo in titulos:\n",
    "#         nombre=titulo.get_text(strip=True)\n",
    "#         name.append(nombre)\n",
    "\n",
    "#     description=soup_product.find(\"div\", class_=\"description\") \n",
    "#     information=description.get_text().split('\\n')[1:]\n",
    "#     documentation = ''.join(information)\n",
    "#     info.append(documentation)\n",
    "\n",
    "\n",
    "#     all_price = soup_product.find_all(\"div\", class_=\"productoPrecio pull-right tdd_precio\")                        \n",
    "#     for price_container in all_price:                                                                    \n",
    "#         try:\n",
    "#             special_price = price_container.find(\"span\", class_=\"productSpecialPrice\")\n",
    "#             if special_price:\n",
    "#                 item_price = float(special_price.get_text(strip=True).replace(\",\", \".\").split('€')[0])\n",
    "#                 new_price.append(item_price)\n",
    "#                 regular_price = price_container.find(\"del\").get_text(strip=True)\n",
    "#                 item_regular_price = float(regular_price.replace(\",\", \".\").split('€')[0])\n",
    "#                 regular_prices.append(item_regular_price)\n",
    "#             else:\n",
    "#                 regular_price = price_container.find(\"span\").get_text(strip=True)\n",
    "#                 item_regular_price = float(regular_price.replace(\",\", \".\").split('€')[0])\n",
    "#                 new_price.append(item_regular_price)\n",
    "#                 regular_prices.append(None)\n",
    "#         except:\n",
    "#             new_price.append(None)\n",
    "#             regular_prices.append(None)\n",
    "\n",
    "#     for i, regular_price in enumerate(regular_prices):\n",
    "#         if regular_price is None:\n",
    "#             regular_prices[i] = new_price[i]\n",
    "\n",
    "\n",
    "#     '''Vamos a obtener los datos de los comentarios de los usuarios'''\n",
    "#     all_user_comments = soup_product.find_all(\"span\", class_=\"name-user\") \n",
    "#     for user_comment in all_user_comments:\n",
    "#         user_comments_product.append(user_comment.get_text(strip=True))\n",
    "#     all_dates = soup_product.find_all(\"span\", class_=\"date\")  \n",
    "#     for dates in all_dates:\n",
    "#         dates_text=dates.get_text(strip=True)\n",
    "#         date_comments_product.append(dates_text)\n",
    "#     all_comments = soup_product.find_all(\"p\")\n",
    "#     for formats in all_comments[-len(date_comments_product):]:\n",
    "#         comments_product.append(formats.get_text(strip=True))\n",
    "#     hearts = soup_product.find_all('div', class_= 'box-description')\n",
    "#     for heart in hearts:\n",
    "#         heart_rating = heart.find_all('span', class_= 'fas fa-heart')\n",
    "#         num_hearts = str(len(heart_rating))\n",
    "#         rating.append(num_hearts)\n",
    "    \n",
    "#     # diccionario={}\n",
    "#     user_comments.append(user_comments_product)\n",
    "#     date.append(date_comments_product)\n",
    "#     comment.append(comments_product)\n",
    "#     ratings.append(rating)\n",
    "    \n",
    "#     # datos = list(zip(nombre,user_comments_product,comments_product, date_comments_product, rating))\n",
    "#     # for dato in datos:\n",
    "#     #     nombre=dato[0]\n",
    "#     #     comentarios=dato[1:]\n",
    "#     #     if nombre in diccionario_comments:\n",
    "#     #         diccionario_comments[nombre].append(comentarios)\n",
    "#     #     else:\n",
    "#     #         diccionario_comments[nombre] = [comentarios]\n",
    "#     # # diccionario_URL[name] = name\n",
    "#     # diccionario_URL[URL] = diccionario_comments\n",
    "    \n",
    "    \n",
    "# # diccionario_URL\n",
    "\n",
    "\n",
    "# dataframe= pd.DataFrame({\"Name\": name,\"Description\": info,\"Price\":regular_price,\"Reduced Price\":new_price,\n",
    "#                          \"date\":date,\"User\": user_comments,\"Ratings\": ratings,\"Comment\": comment\n",
    "#                         #  ,\"diccionario\":diccionario\n",
    "#                         })\n",
    "# dataframe"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generando 2 diccionarios para separar la información y analizar la información dentro de estos diccionarios.\n",
    "\n",
    "Vamos a generar a partir de estos dos diccionarios, 2 dataframes para su estudio, pasarlo a un fichero .csv para guardarlo en una BBDD para su tratamiento posterior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Empezando a recoger datos de las paginas\n",
      "Terminando de recoger los datos de los links de las paginas\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n",
      "96\n",
      "--------------------\n"
     ]
    }
   ],
   "source": [
    "url = \"https://www.amantis.net/productos-amantis/\"              # lista productos\n",
    "url_principal=\"https://www.amantis.net/\"                        # productos\n",
    "pages= np.arange(1,5)\n",
    "# pages= np.arange(1, 25)\n",
    "\n",
    "'''Listas a generar con la información de los productos'''\n",
    "lista_URLs = []\n",
    "name=[]\n",
    "regular_prices=[]\n",
    "new_price=[]\n",
    "info=[]\n",
    "# user_comments=[]\n",
    "# comment=[]\n",
    "# date=[]\n",
    "# ratings=[]\n",
    "id=[]\n",
    "comentarios=[]\n",
    "\n",
    "\n",
    "\n",
    "'''Generamos 2 diccionarios con los datos importantes para ingresar en una BBDD'''\n",
    "\n",
    "diccionario_datos_productos={\"ID\":id,\"NAME\":name,\"INFO\":info,\"LISTA_URL\":lista_URLs,\"REGULAR_PRICE\":regular_prices,\"DISCOUNT_PRICE\":new_price}\n",
    "\n",
    "diccionario_comentarios_productos={\"ID\":id,\"COMENTARIOS\":comentarios}\n",
    "\n",
    "\n",
    "\n",
    "''' Obtenemos las URLs de los productos para entrar luego en sus URLS y extraer la información'''\n",
    "\n",
    "print(\"Empezando a recoger datos de las paginas\")\n",
    "for page in pages:\n",
    "    \n",
    "    if page == 1:\n",
    "        # print(\"Pagina\",page)\n",
    "        URL = url\n",
    "        response = requests.get(url)\n",
    "        soup = bs(response.text, 'lxml')\n",
    "        productos = soup.find_all(class_='caption')\n",
    "        for producto in productos[9:]:\n",
    "            URL_producto = producto.find('a')['href']\n",
    "            lista_URLs.append(URL_producto)\n",
    "        \n",
    "    else:\n",
    "        # print(\"Pagina\",page)\n",
    "        URL = url+'page' + str(page)+'/'\n",
    "        response = requests.get(URL)\n",
    "        soup = bs(response.text, 'lxml')\n",
    "        productos = soup.find_all(class_='caption')\n",
    "        for producto in productos[9:]:\n",
    "            URL_producto = producto.find('a')['href']\n",
    "            lista_URLs.append(URL_producto)\n",
    "print(\"Terminando de recoger los datos de los links de las paginas\")\n",
    "\n",
    "for i in range(len(lista_URLs)):\n",
    "    id.append(i)\n",
    "\n",
    "    \n",
    "'''Extraemos la información de cada producto existente'''\n",
    "\n",
    "for URL in lista_URLs:\n",
    "    url_product=URL\n",
    "    response_product = requests.get(url_product)\n",
    "    soup_product = bs(response_product.text, 'lxml')\n",
    "    user_comments_product=[]\n",
    "    date_comments_product=[]\n",
    "    comments_product=[]\n",
    "    rating=[]\n",
    "\n",
    "    titulos=soup_product.find_all(\"h1\",class_=\"h3\")\n",
    "    for titulo in titulos:\n",
    "        nombre=titulo.get_text(strip=True)\n",
    "        name.append(nombre)\n",
    "\n",
    "    all_price = soup_product.find_all(\"div\", class_=\"productoPrecio pull-right tdd_precio\")                        \n",
    "    for price_container in all_price:                                                                    \n",
    "        try:\n",
    "            special_price = price_container.find(\"span\", class_=\"productSpecialPrice\")\n",
    "            if special_price:\n",
    "                item_price = float(special_price.get_text(strip=True).replace(\",\", \".\").split('€')[0])\n",
    "                new_price.append(item_price)\n",
    "                regular_price = price_container.find(\"del\").get_text(strip=True)\n",
    "                item_regular_price = float(regular_price.replace(\",\", \".\").split('€')[0])\n",
    "                regular_prices.append(item_regular_price)\n",
    "            else:\n",
    "                regular_price = price_container.find(\"span\").get_text(strip=True)\n",
    "                item_regular_price = float(regular_price.replace(\",\", \".\").split('€')[0])\n",
    "                new_price.append(item_regular_price)\n",
    "                regular_prices.append(None)\n",
    "        except:\n",
    "            new_price.append(None)\n",
    "            regular_prices.append(None)\n",
    "\n",
    "    description=soup_product.find(\"div\", class_=\"description\") \n",
    "    information=description.get_text().split('\\n')[1:]\n",
    "    documentation = ''.join(information)\n",
    "    info.append(documentation)\n",
    "\n",
    "\n",
    "    '''Vamos a obtener los datos de los comentarios de los usuarios'''\n",
    "\n",
    "    all_user_comments = soup_product.find_all(\"span\", class_=\"name-user\") \n",
    "    for user_comment in all_user_comments:\n",
    "        user_comments_product.append(user_comment.get_text(strip=True))\n",
    "\n",
    "        \n",
    "    # user_comments.append(user_comments_product)\n",
    "\n",
    "\n",
    "    all_dates = soup_product.find_all(\"span\", class_=\"date\")  \n",
    "    for dates in all_dates:\n",
    "        dates_text=dates.get_text(strip=True)\n",
    "        # dates=datetime.strftime(dates, '%dd/%mm/%Y')\n",
    "        date_comments_product.append(dates_text)\n",
    "        # date_object = datetime.strptime(date_comments_product)\n",
    "    # date.append(date_comments_product)\n",
    "\n",
    "    all_comments = soup_product.find_all(\"p\")\n",
    "    for formats in all_comments[-len(date_comments_product):]:\n",
    "        comments_product.append(formats.get_text(strip=True))\n",
    "    # comment.append(comments_product)\n",
    "\n",
    "    hearts = soup_product.find_all('div', class_= 'box-description')\n",
    "    for heart in hearts:\n",
    "        heart_rating = heart.find_all('span', class_= 'fas fa-heart')\n",
    "        num_hearts = len(heart_rating)\n",
    "        rating.append(num_hearts)\n",
    "    # ratings.append(rating)\n",
    "\n",
    "    datos = list(zip(date_comments_product,rating, user_comments_product,comments_product ))\n",
    "    comentarios.append(datos)\n",
    "\n",
    "for i, regular_price in enumerate(regular_prices):\n",
    "    if regular_price is None:\n",
    "        regular_prices[i] = new_price[i]\n",
    "\n",
    "\n",
    "print('-'*20)\n",
    "print(len(name))\n",
    "print('-'*20)\n",
    "print(len(regular_prices))\n",
    "print('-'*20)\n",
    "print(len(new_price))\n",
    "print('-'*20)\n",
    "print(len(lista_URLs))\n",
    "print('-'*20)\n",
    "print(len(info))\n",
    "print('-'*20)\n",
    "# print(len(user_comments))\n",
    "# print('-'*20)\n",
    "# print(len(comment))\n",
    "# print('-'*20)\n",
    "# print(len(date))\n",
    "# print('-'*20)\n",
    "# print(len(ratings))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 96 entries, 0 to 95\n",
      "Data columns (total 6 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   ID              96 non-null     int64  \n",
      " 1   NAME            96 non-null     object \n",
      " 2   INFO            96 non-null     object \n",
      " 3   LISTA_URL       96 non-null     object \n",
      " 4   REGULAR_PRICE   96 non-null     float64\n",
      " 5   DISCOUNT_PRICE  96 non-null     float64\n",
      "dtypes: float64(2), int64(1), object(3)\n",
      "memory usage: 4.6+ KB\n"
     ]
    }
   ],
   "source": [
    "productos=pd.DataFrame(diccionario_datos_productos)\n",
    "productos.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Buscando duplicados para eliminarlos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AVATAR FIRST LOVER, amante realista de suave silicona            2\n",
       "DULCE TORMENTO, pala de bambú para azotes                        2\n",
       "TRIS-TRAS, anilla con vibrador para doble penetración TRINITY    2\n",
       "TROMPI, vibrador sumergible ideal para Punto-G                   2\n",
       "GOTA-SOFT Medium, plug anal con brillante                        2\n",
       "                                                                ..\n",
       "TOBOGANE, el vibrador doble más vendido                          1\n",
       "EXPLORE PERFECT, posiblemente el dildo realista perfecto         1\n",
       "MYSTERY, body de rejilla super sexy                              1\n",
       "LINEAS INFINITAS, medias con liguero para piernas de infarto     1\n",
       "Bacanal, gel lubricante anal de acción extrema con aloe          1\n",
       "Name: NAME, Length: 90, dtype: int64"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "productos.NAME.value_counts()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 90 entries, 0 to 95\n",
      "Data columns (total 6 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   ID              90 non-null     int64  \n",
      " 1   NAME            90 non-null     object \n",
      " 2   INFO            90 non-null     object \n",
      " 3   LISTA_URL       90 non-null     object \n",
      " 4   REGULAR_PRICE   90 non-null     float64\n",
      " 5   DISCOUNT_PRICE  90 non-null     float64\n",
      "dtypes: float64(2), int64(1), object(3)\n",
      "memory usage: 4.9+ KB\n"
     ]
    }
   ],
   "source": [
    "noduplicated_product = productos.drop_duplicates(subset='NAME', keep='first')\n",
    "noduplicated_product.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determinamos los ID de los registros a eliminar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.series.Series'>\n",
      "Int64Index: 6 entries, 25 to 80\n",
      "Series name: ID\n",
      "Non-Null Count  Dtype\n",
      "--------------  -----\n",
      "6 non-null      int64\n",
      "dtypes: int64(1)\n",
      "memory usage: 96.0 bytes\n"
     ]
    }
   ],
   "source": [
    "removed_id = productos[productos.duplicated(subset='NAME', keep='first')]['ID']\n",
    "removed_id.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25    25\n",
       "27    27\n",
       "49    49\n",
       "50    50\n",
       "52    52\n",
       "80    80\n",
       "Name: ID, dtype: int64"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "removed_id"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aqui genero un dataFrame con los datos de los comentarios.\n",
    "\n",
    "En este dataframe creado no están los id de los productos que no tienen comentarios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4345 entries, 0 to 4344\n",
      "Data columns (total 5 columns):\n",
      " #   Column   Non-Null Count  Dtype \n",
      "---  ------   --------------  ----- \n",
      " 0   ID       4345 non-null   int64 \n",
      " 1   DATE     4345 non-null   object\n",
      " 2   RATIO    4345 non-null   int64 \n",
      " 3   USERS    4345 non-null   object\n",
      " 4   COMMENT  4345 non-null   object\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 169.9+ KB\n"
     ]
    }
   ],
   "source": [
    "comentarios_productos=pd.DataFrame(diccionario_comentarios_productos)\n",
    "\n",
    "id=[]\n",
    "comments=[]\n",
    "date=[]\n",
    "ratio=[]\n",
    "users=[]\n",
    "comment=[]\n",
    "\n",
    "comentarios=pd.DataFrame()\n",
    "diccionario={\"id\":id,\"comments\":comments}\n",
    "\n",
    "for id_product,n_comments in enumerate (comentarios_productos['COMENTARIOS']):\n",
    "    # print(\"Imprimiendo texto del indice\",id_product)\n",
    "    # print(\"Imprimiento n_comentarios\",len(n_comments))\n",
    "    for i in n_comments:\n",
    "        # print(\"id\",id_product,\"coments\",comments)\n",
    "        id.append(id_product)\n",
    "        comments.append(i)\n",
    "\n",
    "\n",
    "for j in range(len(diccionario['comments'])):\n",
    "    date.append(diccionario['comments'][j][0])\n",
    "    ratio.append(diccionario['comments'][j][1])\n",
    "    users.append(diccionario['comments'][j][2])\n",
    "    comment.append(diccionario['comments'][j][3])\n",
    "\n",
    "\n",
    "comentarios['ID']=pd.Series(id)\n",
    "comentarios['DATE']=pd.Series(date)\n",
    "comentarios['RATIO']=pd.Series(ratio)\n",
    "comentarios['USERS']=pd.Series(users)\n",
    "comentarios['COMMENT']=pd.Series(comment)\n",
    "comentarios.info()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>RATIO</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10930.000000</td>\n",
       "      <td>10930.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>199.476121</td>\n",
       "      <td>4.669808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>145.982488</td>\n",
       "      <td>0.686413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>86.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>171.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>308.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>575.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ID         RATIO\n",
       "count  10930.000000  10930.000000\n",
       "mean     199.476121      4.669808\n",
       "std      145.982488      0.686413\n",
       "min        1.000000      1.000000\n",
       "25%       86.000000      5.000000\n",
       "50%      171.000000      5.000000\n",
       "75%      308.000000      5.000000\n",
       "max      575.000000      5.000000"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comentarios.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>DATE</th>\n",
       "      <th>RATIO</th>\n",
       "      <th>USERS</th>\n",
       "      <th>COMMENT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11370</th>\n",
       "      <td>575</td>\n",
       "      <td>viernes 22 enero, 2016</td>\n",
       "      <td>5</td>\n",
       "      <td>Luis</td>\n",
       "      <td>Ya te vayan los juegos de dominación o no...co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11371</th>\n",
       "      <td>575</td>\n",
       "      <td>viernes 26 diciembre, 2014</td>\n",
       "      <td>4</td>\n",
       "      <td>Ana</td>\n",
       "      <td>Hola, es la primera vez que adquiero artículos...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11372</th>\n",
       "      <td>575</td>\n",
       "      <td>martes 09 diciembre, 2014</td>\n",
       "      <td>4</td>\n",
       "      <td>Robert</td>\n",
       "      <td>Muy provocativo, y estimulante, con muy buena ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11373</th>\n",
       "      <td>575</td>\n",
       "      <td>martes 09 diciembre, 2014</td>\n",
       "      <td>5</td>\n",
       "      <td>Jose Antonio</td>\n",
       "      <td>Conjunto perfecto para que mi chica se ponga e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11374</th>\n",
       "      <td>575</td>\n",
       "      <td>jueves 04 diciembre, 2014</td>\n",
       "      <td>5</td>\n",
       "      <td>Hilario</td>\n",
       "      <td>mi novia me sorprendio con este conjunto y a p...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        ID                        DATE  RATIO         USERS  \\\n",
       "11370  575      viernes 22 enero, 2016      5          Luis   \n",
       "11371  575  viernes 26 diciembre, 2014      4           Ana   \n",
       "11372  575   martes 09 diciembre, 2014      4        Robert   \n",
       "11373  575   martes 09 diciembre, 2014      5  Jose Antonio   \n",
       "11374  575   jueves 04 diciembre, 2014      5       Hilario   \n",
       "\n",
       "                                                 COMMENT  \n",
       "11370  Ya te vayan los juegos de dominación o no...co...  \n",
       "11371  Hola, es la primera vez que adquiero artículos...  \n",
       "11372  Muy provocativo, y estimulante, con muy buena ...  \n",
       "11373  Conjunto perfecto para que mi chica se ponga e...  \n",
       "11374  mi novia me sorprendio con este conjunto y a p...  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comentarios.tail()\n",
    "# productos.tail()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Elimino los registros de ID que ya he eliminado en el anterior dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 10479 entries, 0 to 10929\n",
      "Data columns (total 5 columns):\n",
      " #   Column   Non-Null Count  Dtype \n",
      "---  ------   --------------  ----- \n",
      " 0   ID       10479 non-null  int64 \n",
      " 1   DATE     10479 non-null  object\n",
      " 2   RATIO    10479 non-null  int64 \n",
      " 3   USERS    10479 non-null  object\n",
      " 4   COMMENT  10479 non-null  object\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 491.2+ KB\n"
     ]
    }
   ],
   "source": [
    "noduplicated_comments = comentarios[~comentarios['ID'].isin(productos[productos['ID'].isin(removed_id)]['ID'])]\n",
    "noduplicated_comments.info()\n",
    "# Imprimir los resultados"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "noduplicated_product.to_csv('./Data/productos_scrape.csv',header=True,index=False)           # Tengo que generar el path correcto\n",
    "noduplicated_comments.to_csv('./Data/comentarios_scrape.csv',header=True,index=True)           # Tengo que generar el path correcto"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "afde6861a040563f15a2ec1b440faf84809f9a7bcc3c75cfd11a60e7dd448719"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}