Python scripts to import database schema information from different data sources into a custom database.

Prerequisites

• Python 3.8+
• pip
• Source db (postgres, snowflake, etc.)
• Destination db (your custom db)

Installing

1. Clone the repository to your local machine.

   git clone https://github.com/betocmn/LLM-import-and-document-postgres-schema.git

2. Install all requirements

   pip3 install -r requirements.txt

3. Use the env.example file to create a .env and fill it out.

Usage

Import from a Postgres source

1. Start by generating a new organization_id and importing all basic schema from the source. This command will output an "organization_id" that you will need next.

   python3 scripts/import_from_postgres.py

2. Generate the AI documentation for all tables and columns. The first argument is the "organization_id" from before and the second is the number of tables to generate (because openAI requests take a while, you might want to do in chunks).

   python3 scripts/document_from_postgres.py 8 30

3. Generate the LLM embeddings for this new org. The argument is the "organization_id" from before.

   python3 scripts/generate_embeddings.py 8