• Introduction to SQL
• SQL: Tables and Basic Structure
• SQL: Data Types
• SQL: Syntax

SQLite is a database that comes pre-installed with Python, you just have to import it just like any other module and start using it.

Another great advantage of SQLite for learning is that it only has 5 types:

• null: indicates a non-value
• integer: represents integer numbers
• real: represents floating numbers
• text: represents strings
• blob: anything that doesn’t fit in the other types, like images, music files, videos, etc.

To start using SQLite, you have to import it with import sqlite3.

Then you open a connection and call the database anything you want, if the database doesn’t exist yet, it will be created.

We are going to call our database company.db.

For SQLite, the database is just a single file.

After that, you create a cursor, which is the object we are going to call the execute the commands in the database.

Since most commands in SQL are pretty extensive, we will use triple double-quotes for multiline strings, it will help with overall readability.

For the command to be actually executed you have to commit it using commit().

Finally, when you are done, you close() the connection with the database.

Let’s see how this works with an example.

Create Database and Table

Create a file ‘company_database.py’ with the following:

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

sql = """ CREATE TABLE employees (
        name text,
        job_title text,
        salary real,
        hire_date text
    )"""

cursor.execute(sql)

connection.commit()

connection.close()

This structure covers everything described before.

Import sqlite3, create a database named ‘company.db’ to store data about our company, create a cursor from connection, execute the command to create the table ’employees’ with four fields, then commit the transaction and finally, we close the connection.

The four fields are name, job_title, salary, and hire_date. All of them are of the text type, except for salary which has the type real.

We are going to use the structure throughout this article repeatedly.

Execute the code with:

python company_database.py

If no error occurred, you will find a new file named ‘company.db’ alongside your program’s file ‘company_database.py’ in the same folder.

We are going to use the same file for every example from now on, keeping the basic structure.

Insert Data

In this next piece of code, we are going to ‘INSERT’ a record in our ’employees’ tables.

Notice how we have to use a ‘?’ to represent every field after the VALUES keyword.

The values are passed in the form of a tuple, notice that the format used for dates follows the pattern ‘yyyy-MM-dd HH:mm:ss’.

When calling execute(), the first argument is the SQL command, and the second is the tuple with the values of the new record.

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

sql = """ INSERT INTO employees (
        name,
        job_title,
        salary,
        hire_date
    ) VALUES (?, ?, ?, ?)"""

values = ('Bob', 'Engineer', 80000, '2007-09-22 00:00:00')

cursor.execute(sql, values)

connection.commit()

connection.close()

To insert many records at once, you can create a list of tuples, one tuple for every new record, and instead of calling execute(), you call executemany().

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

sql = """ INSERT INTO employees (
        name,
        job_title,
        salary,
        hire_date
    ) VALUES (?, ?, ?, ?)"""

values = [
    ('Mary', 'Designer', 60000, '2012-09-05 00:00:00'),
    ('Sarah', 'Sales Manager', 98000, '2017-06-21 00:00:00'),
    ('Peter', 'IT Manager', 95000, '2013-09-05 00:00:00'),
    ('Brian', 'HR Manager', 92000, '2010-09-05 00:00:00')
    ]

cursor.executemany(sql, values)

connection.commit()

connection.close()

Query Data

Now we have a database, a table, and a few records.

To see the records stored in a table, we use the SELECT command.

We use ‘*’ to retrieve all the fields at once.

To bring all the records from the query, we use fetchall().

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

sql = "SELECT * FROM employees"

cursor.execute(sql)

print(cursor.fetchall())

connection.commit()

connection.close()

The result will come as a list of tuples, one tuple for each record.

[('Bob', 'Engineer', 80000.0, '2007-09-22 00:00:00'), 
('Mary', 'Designer', 60000.0, '2012-09-05 00:00:00'), 
('Sarah', 'Sales Manager', 98000.0, '2017-06-21 00:00:00'), 
('Peter', 'IT Manager', 95000.0, '2013-09-05 00:00:00'), 
('Brian', 'HR Manager', 92000.0, '2010-09-05 00:00:00')]

Every record has a primary, an unique id.

SQLite creates these id’s automatically and autoincrements them for every new record.

They are implicit and are called ‘rowid’ by default.

To bring only some fields in a query, you can specify every field separated by commas.

Here we are using only three fieds: rowid, name, and salary.

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

sql = "SELECT rowid, name, salary FROM employees"

cursor.execute(sql)

print(cursor.fetchall())

connection.commit()

connection.close()

The output has less information, as requested:

[(1, 'Bob', 80000.0), (2, 'Mary', 60000.0), 
(3, 'Sarah', 98000.0), (4, 'Peter', 95000.0), 
(5, 'Brian', 92000.0)]

Since the result is a tuple, we can use brackets to reference each item in the tuple and format the output while iterating the list of tuples with a for loop.

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

sql = "SELECT name, salary FROM employees"

cursor.execute(sql)

records = cursor.fetchall()

for record in records:
    print(f'Name: {record[0]}, Salary: {record[1]}')

connection.commit()

connection.close()

The output will be:

Name: Bob, Salary: 80000.0
Name: Mary, Salary: 60000.0
Name: Sarah, Salary: 98000.0
Name: Peter, Salary: 95000.0
Name: Brian, Salary: 92000.0

The WHERE clause allows you to filter the results of a query.

You can also combine it with operators like >=, <, AND, OR, and others to create more complex filters.

This query lists only employees hired after 2011.

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

sql = """SELECT * FROM employees
        WHERE hire_date > '2011-01-01' """

cursor.execute(sql)

records = cursor.fetchall()

for record in records:
    print(record)

connection.commit()

connection.close()

You can see everyone on this list was hired after 2011:

('Mary', 'Designer', 60000.0, '2012-09-05 00:00:00')
('Sarah', 'Sales Manager', 98000.0, '2017-06-21 00:00:00')
('Peter', 'IT Manager', 95000.0, '2013-09-05 00:00:00')

This query lists only employees hired after 2011 and whose salary is below 96,000.

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

sql = """SELECT * FROM employees
        WHERE hire_date > '2011-01-01' 
        AND salary < 96000"""

cursor.execute(sql)

records = cursor.fetchall()

for record in records:
    print(record)

connection.commit()

connection.close()

You can see Sarah was excluded because her salary is 98,000.

('Mary', 'Designer', 60000.0, '2012-09-05 00:00:00')
('Peter', 'IT Manager', 95000.0, '2013-09-05 00:00:00')

Update Data

Now that you know how to query your database, let’s see how to UPDATE a record.

The general syntax is UPDATE <table name> SET <field> = <new value> WHERE <filter>.

The WHERE clause is actually optional, but get used to always use it to avoid update every single row of your table by mistake, which is very dangerous and may seriously harm your data. This way, your update will only affect the records filtered by the WHERE.

Here we are saying that everyone whose salary is below 90,000 will get an increase of 10,000.

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

sql = """UPDATE employees SET salary = salary + 10000
            WHERE salary < 90000
     """

cursor.execute(sql)

connection.commit()

cursor.execute("SELECT * FROM employees")

records = cursor.fetchall()

for record in records:
    print(record)

connection.commit()

connection.close()

Only Bob and Mary had an increase of 10000:

('Bob', 'Engineer', 90000.0, '2007-09-22 00:00:00')
('Mary', 'Designer', 70000.0, '2012-09-05 00:00:00')
('Sarah', 'Sales Manager', 98000.0, '2017-06-21 00:00:00')
('Peter', 'IT Manager', 95000.0, '2013-09-05 00:00:00')
('Brian', 'HR Manager', 92000.0, '2010-09-05 00:00:00')

Order By

You can use ORDER BY to choose how to order the results of a query

The default is to sort records by rowid and ascending, i.e, from the smalller to the higher id.

Here we are going to sort by salary descending, that is, from the higher to the smaller salary.

We are also to list rowid explicitly to see how the result is ordered by the salary column, not the id.

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

cursor.execute("SELECT rowid, * FROM employees ORDER BY salary DESC")

records = cursor.fetchall()

for record in records:
    print(record)

connection.commit()

connection.close()

You can easily see Sarah makes the most with 98,000.

(3, 'Sarah', 'Sales Manager', 98000.0, '2017-06-21 00:00:00')
(4, 'Peter', 'IT Manager', 95000.0, '2013-09-05 00:00:00')
(5, 'Brian', 'HR Manager', 92000.0, '2010-09-05 00:00:00')
(1, 'Bob', 'Engineer', 90000.0, '2007-09-22 00:00:00')
(2, 'Mary', 'Designer', 70000.0, '2012-09-05 00:00:00')

Limit

We have a very small database, but it is not uncommon to have thousands of records, and sometimes you just want to see a few of them to check the structure of the data.

Use LIMIT followed by the number of records you want.

In this case, we are just listing the first 3.

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

cursor.execute("SELECT rowid, * FROM employees LIMIT 3")

records = cursor.fetchall()

for record in records:
    print(record)

connection.commit()

connection.close()

You can see that records 4 and 5 were not included.

(1, 'Bob', 'Engineer', 90000.0, '2007-09-22 00:00:00')
(2, 'Mary', 'Designer', 70000.0, '2012-09-05 00:00:00')
(3, 'Sarah', 'Sales Manager', 98000.0, '2017-06-21 00:00:00')

Delete

To delete a record, the general syntax is DELETE FROM <table name> WHERE <filter>.

The WHERE clause is actually optional, but get used to always use it to avoid deleting records by mistake, which is very dangerous and may seriously harm your data. This way, your DELETE will only affect the records filtered by the WHERE.

Here we are deleting the record with Id 1, which is Bob.

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

cursor.execute("""DELETE FROM employees 
        WHERE rowid = 1
     """)

connection.commit()

cursor.execute("SELECT rowid,* FROM employees")

records = cursor.fetchall()

for record in records:
    print(record)

connection.commit()

connection.close()

You can see Bob was deleted.

(2, 'Mary', 'Designer', 70000.0, '2012-09-05 00:00:00')
(3, 'Sarah', 'Sales Manager', 98000.0, '2017-06-21 00:00:00')
(4, 'Peter', 'IT Manager', 95000.0, '2013-09-05 00:00:00')
(5, 'Brian', 'HR Manager', 92000.0, '2010-09-05 00:00:00')

Drop Table

To delete a whole table, use the DROP TABLE command indicating the name of the table you want to drop.

import sqlite3

connection = sqlite3.connect('company.db')

cursor = connection.cursor() 

cursor.execute("DROP TABLE employees")

connection.commit()

connection.close()

Other Databases

How to connect to a MySQL database in Python

How to connect to a PostgreSQL database in Python

The content SQLite in Python is from Renan Moura - Software Engineering.

For the sake of clarity and readability, it is advised to use the reserved keywords in uppercase, while the referenced tables and fields are written in lowercase.

Some databases do not require semicolon ; at the end of a statement, but as a general rule, always use it.

For instance, a SELECT statement has the following general look based on what was specified before:

SELECT column_name FROM table_name;

Notice that you can easily identify that SELECT and FROM are reserved keywords in SQL, while column_name and table_name reference a field and a table respectively.

The most fundamental commands are:

• SELECT: reads/extracts the data
• INSERT: inserts new data
• UPDATE: updates the existing data
• DELETE: removes the data

You will see them commonly referred to as CRUD, an acronym for Create, Read, Update, and Delete.

Together they form the basis of any program that needs to interact with a database.

The content SQL: Syntax is from Renan Moura - Software Engineering.

Each field in a Table has a Type.

A Data Type is used to specify what kind of object that particular field will store.

When creating your own structures, using the right type for the right data is essential to prevent data loss and to work better with the information you have.

It is important to note that each database has different types.

They might be similar, but they are not exactly the same and you should be aware of this when dealing with different products like Oracle or MySQL, for instance.

I will list PostgreSQL Data Types in this article to give you an understanding of the main types.

If you are working with other databases, just check the documentation of each one and you should see lots of similarities to among them.

A few popular examples are:

• MysSQL Data Types
• SQL Server Data Types
• Oracle Data Types

Let’s learn about PostgreSQL Data Types.

Data Types in PostgreSQL

If you check the oficial documentation for PostgreSQL Data Types, you will see some very specific types for very specific cases.

Let’s keep things simple and use some Pareto’s 80/20 to show you the types you are going to use most of the time, they are:

• Numeric Types: integer and floating-point
• Character types: char, varchar and text
• Boolean
• Date/Time Types: date, time, timestamp, and interval

These types are very generic and you can use them to store numbers, text, and dates, which is enough for most use cases.

Numeric Types

You can work with either integers of float-pointing numbers.

Integer

For integers you can work with:

• smallint: it stores 2 bytes and can store values in the range of -32768 to +32767.
• integer: it stores 4 bytes and can store values in the range of -2147483648 to +2147483647
• bigint: it can store 8 bytes and can store values in the range of -9223372036854775808 to +9223372036854775807

All of them have their serial counterparts, which is a specific type that automatically increments for every new row, it is very common for ID columns where you store values sequentially for each new record like 1, 2, 3, 4, 5 and so on.

For instance, the serial integer is:

• serial: stores 4 bytes and ranges from 1 to 2147483647

Notice its range starts on 1, not on a negative number since it doesn’t make much sense to have and ID of, say, -20 for an employee, for instance.

Floating-point numbers

For floating-point numbers, in general you can choose to use numeric for most cases.

It supports a very large number of digits and it is easy to work with when you need to deal with monetary values or simply needs a good precision.

Character types

These types are used to store strings.

• varchar: can store a string of the size you pre-define, and it does not pad spaces if the string to be stored is shorter than the declared length
• char: can store a string of the size you pre-define, but it pads spaces if the string to be stored is shorter than the declared length
• text: can store a string of any size, no need to predefine the number of characters

Boolean

The boolean type is by far the most simple one.

It stores 1 byte that represents true, false or null value for when you have an unknown state.

If you insert either of these in you database, PostgreSQL will understand them as true: true, yes, on, and 1.

If you insert either of these in you database, PostgreSQL will understand them as false: false, no, off, and 0.

Date/Time

You can basically store time, date, date and time simultaneously, and data/time with time zone:

• timestamp: you can store the date and the time
• date: you can store only the date
• time: you can store only the time
• timestamptz: is the same as timestamp with the addition of time zone
• interval: you can store a period of time, a time interval

The content SQL: Data Types is from Renan Moura - Software Engineering.

SQL works on a structure with four components: Table, Field, Row, and Column.

You can think of those components exactly like the ones in spreadsheets like Excel.

A database can have one or more tables in it.

Table

A table is a very common and simple way to organize data.

You have columns describe the kind of information and rows/records consisting of the information itself.

The following two tables Employees and Departments are the ones I will use to demonstrate the capabilities of SQL and how to manage data.

The Employees table contains data about the employees of a certain company.

Employees

| employee_id | name  |   job_title   | salary | hire_date  | department_id |
|-------------|-------|---------------|--------|------------|---------------|
| 1           | Bob   | Engineer      | 80000  | 2015-04-12 | 2             |
| 2           | Mary  | Designer      | 60000  | 2017-06-21 | 2             |
| 3           | Sarah | Sales Manager | 98000  | 2013-09-05 | 1             |
| 4           | Peter | IT Manager    | 95000  | 2010-09-05 | 2             |
| 5           | Brian | HR Manager    | 92000  | 2012-09-05 | 3             |

The Departments table describes de departments the company has.

Departments

| department_id | department_name | city      | country | manager_id |
|---------------|-----------------|-----------|---------|------------|
| 1             | Sales           | São Paulo | Brazil  | 3          |
| 2             | IT              | Barcelona | Spain   | 4          |
| 3             | Human Resources | New York  | USA     | 5          |

Field

A field specifies the kind of information a particular column has and its type.

The Employees table has the following fields:

• employee_id: de unique ID that identifies an employee
• name: the name of the employee
• job_title: the formal title the employee has inside the company
• salary: how much the employee makes yearly
• hire_date: when the employee was hired
• department_id: the unique ID of the department where the employee works

The Departments table has the following fields:

• department_id: the unique ID for the Department
• department_name: the name of the Department
• city: the city where the Department is located
• country: the country where the Department is located
• manager_id: the ID of the manager responsible for the department, the manager is an employee

Row

A row is a record of data inside the Table.

The Employee table has 5 rows.

The first row is Bob\’s record.

Notice the row with the identifications of the fields does not count as "first" row, the counting begins with the first record.

| employee_id | name  |   job_title   | salary | hire_date  | department_id |
|-------------|-------|---------------|--------|------------|---------------|
| 1           | Bob   | Engineer      | 80000  | 2015-04-12 | 2             |

Column

A column is a field and all of its information.

The column employee_id in the Employees table has all the unique IDs of all the employees.

| employee_id |
|-------------|
| 1           |
| 2           |
| 3           |
| 4           |
| 5           |

The content SQL: Tables and Basic Structure is from Renan Moura - Software Engineering.

What is SQL and why you should learn it

SQL stands for Structured Query Language.

It is pronounced SEQUEL.

SQL is a language designed to handle databases.

It allows you to manage and access data in a relational database using a standard approach.

Relational Database Management Systems or simply RDMS are the industry standard for storing data.

RDMS store data in tables, much like the ones you make in your Excel spreadsheets with columns and rows, each column is called a field and each row is a data entry called a record.

Some of the most common RDMS are PostgreSQL, MySQL, Oracle, and SQL Sever.

There are also NoSQL Databases like Mongo that store data in different ways such as documents, they are growing in use year after year, but are still far from becoming as widely used as relational ones.

SQL Usage

SQL is used in a variety of ways to manage your application’s data, such as to:

• Retrieve, Update, Insert and Delete data
• Create and Delete databases and tables
• Manage permissions to certain data, so some users can access it and others don’t
• Guarantee the integrity of the data
• Automate actions in the database according to some change in the data

SQL Dialects

Be aware that there are some variations of SQL depending on the RDMS you are using that make SQL even more powerful for those specific systems.

After learning the standard SQL, it is common for some people to specialize in a particular RDMS and its specific dialect like Oracle’s PL/SQL or PostgreSQL’s PL/pgSQL.

It is also common to see people referring to standard SQL as ANSI SQL to differentiate it from the specific dialects.

ANSI stands for American National Standards Institute, the institute standardized SQL in 1986.

The content Introduction to SQL is from Renan Moura - Software Engineering.